Skip to end of metadata
Go to start of metadata

Introduction

This is based on the work done by Will B: http://civicrm.org/blog/6885, http://mywillbrownsberger.com/node/41 and CiviEngage

You can read the IRC meeting logs here

Sponsorship

if you can help support this work, please make a donation here

Estimate

Phase 1 Feature Set

Project

Number of Hours

Port Will's Stuff to CiviCRM and use the schema defined below (one campaign, one custom field)

30 hours

Add GOTV interface

30 hours

Campaign Support for canvassing

50 hours

Custom Groups for questions and answers

50 hours

Reporting

25 hours

Unit Tests

25 hours

Misc Issues

25 hours

Total Hours

235 hours

Total Cost @ $75/hour

$17,625

Funds Raised so far

$1,150

Phase 2 Feature Set

Project

Number of Hours

Mobile Phone App (iPhone/Android)

200 hours

BarCode Support

50 hours

Scalability Optimization for high volume

50 hours

Click to Call w/o VOIP integration

100 hours

Asterisk integration with Click-2-Call

100 hours?

CiviCRM Consulting rates and contract are on this wiki page

Features

We have decided to use activities as a base to record all interactions with the constituents. This gives us a lot of flexibility down the road and allows us to expand this similar to CiviCase

  • Allows the admin to set up one or more "campaigns". In this context a campaign is either a "Constituent Interview" Campaign or a "Constituent has cast his ballot" Campaign (for GOTV)
  • Allows the admin to set various fields in the campaign and survey tables.
  • The admin associates one or more groups (static or smart) to this campaign. [Joe: of contacts to be canvassed, or of contacts who will be doing the canvassing, or both?]
  • The admin configures a 'script' for volunteers/interviewers for this campaign. Scripts can have instructions for the volunteer, and then one or more questions for the constituent (which are custom fields). Each question can have associated text (custom field "help"). For this iteration, script branching is not supported.
  • Allows a volunteer (with the right permissions), to "check" out a set of contacts from the campaign based on an address and district search (profile search in future?). [Joe: are the permissions set by by admin based on volunteer being part of a group? That is my preference.]
  • Allows admins/staff (with the right permissions), to "check" out a set of contacts from the campaign based on an address and district search and assign them to a volunteer
  • Can generate a walklist or phonelist report for the volunteer with their group of assigned contacts for use in field-based canvasses (i.e. where volunteer is not sitting in front of a computer).
  • Allows the above volunteer to record the interview responses and the status of this interview. Statuses may be used to control 'recontact interval'. This can be done 'realtime' for phone-banking canvasses (or any use-case where volunteer is online while canvassing). Alternatively, interview responses can be entered in batches (from filled-in paper reports) using an input table grid (jQuery dataTables). This mode is intended to cover both post-field-canvass input as well as poll-watcher ('GOTV') input (more details below).
  • Allows the volunteer to reclaim / add / return back his set of constituents.
  • Gives admin stats on the current status of the overall campaign and the volunteer stats on whats she/he has done for the campaign.
  • Allow admin to rollback work of a specific volunteer on any day / for all time. [Joe: What does this involve? I would suggest a 'cancelled' status be set up on the activity record that would be used to exclude the constitutent responses from being included in summary and detailed reports.]
  • Hook to allow sites to have more complex recontact intervals.
  • Hook to allow sites to have different search criteria. [Joe: are these criteria for searching the contacts to be canvassed, or the contacts who are allowed to canvass, or both?]

GOTV work

  • This is used on voting day to record who has cast a ballot and who should be reminded to go out and vote.
  • An attendance screen on a "voting place" basis, so a volunteer can mark all the constituents who have voted. This is the set of all constituents who are registered to vote at this voting place AND have not yet voted
    • This needs to be sorted by: Street Name, Street Number and Street Unit in that order
    • Search boxes for Street name and within that street unit and name would be great
    • Cross off constituents who have voted
    • This screen is a customized version of the input table grid
  • A custom search to give us all the people who are "for" the campaign but have not yet voted. This should be done via a GOTV constituent campaign using a smart group. So we can follow the same process as above.

Database Tables

Campaign Table

  • id
  • name
  • title of campaign
  • description
  • campaign start date
  • campaign end date
  • campaign type (option group, checkbox?)
  • campaign status
  • campaign external identifier
  • campaign parent id
  • is_active
  • created_id
  • created_date
  • last_modified_id
  • last_modified_date

Campaign Groups Table

  • id
  • campaign id
  • group type - enum('Include', 'Exclude')
  • entity_table (we can have groups, campaigns, mailings as the table)
  • entity_id

Survey Table

When a survey is created, the admin has a choice of either creating a very specific activity type for this survey or reusing a generic survey activity type, or a bit more specific "survey type" activity type. The custom group(s) (questions) that the admin can associate to this survey is limited on the activity type chosen for this survey. It can be very specific (i.e. different per survey), or categorized based on survey type or very generic (i.e. across all surveys)

  • id
  • campaign_id (optional)
  • activity_type_id - the activity type to be used for this survey (Canvass, PhoneBank, WalkList etc). This is useful to share custom group data. These activity types will be restricted by component id of campaign
  • result_group_id - option/value group for the "summary" result for this survey
  • serialized array for recontact intervals for each value in above group
  • script (script instructions for volunteers to use for the campaign)
  • release_interval - interval when contacts are released back to the pool (so it can be assigned to other volunteers)
  • max_number_of_contacts
  • default_number_of_contacts
  • is_active
  • is_default
  • created_id
  • created_date
  • last_modified_id
  • last_modified_date

Activity Table - stores the interaction between a "surveyor" (can be null, i.e. online) and the person "surveyed"

This reuses CiviCRM's current activity table. When an "admin" manually assigns people to be "surveyed" to a "surveyor", we can record this in the source_contact_id

  • id
  • survey_id - stored in the source_record_id
  • interviewer_id - contact id of interviewer - this is stored in the activity_assignee table
  • subject_id - contact id of constituent interviewed - this is stored in the activity_target table
  • subject_address - this is stored in the location field
  • subject_phone - this is stored in the phone_number field
  • notes - free form note re interview to be picked up in nightly batch and transmitted by email to campaign administrators', - this is stored in the details field
  • activity_date_time - stores the time when the survey took place
  • status_id - Completed for interview complete, Scheduled for Pending. Activities will be deleted when released or expired
  • result - (new field) - result of activity. For surveys, this will be the Label of the option value of survey result_group_id

Activity Custom Group for a specific Survey

This custom group stores all the questions (and the value table stores the answers ) for each survey. This custom group extends activities. We create activity types for Survey (and for each survey type). When a survey is created, we can create a very specific activity type for that survey, and have a custom group that extends that survey.

The CiviEngage model will move to using the survey object to link the questions and answers rather than CiviEvent

Profile to use for surveys

This is stored in the uf_join table. The current implementation will only support Profiles of type Activity.

Permissions and Operations allowed

administer campaign

  • CRUD on campaign
  • CRUD on survey
  • everything below

manage campaign

  • reserve/release/interview set of contacts restricted by survey to one (or all) Contacts

reserve campaign contacts

  • reserve set of contacts restricted by survey

release campaign contacts

  • release set of contacts assigned to me

interview campaign contacts

  • interview set of contacts assigned to me

Survey Search Log

This can potentially be done as a hook OUTSIDE of the engage module. might keep things a bit cleaner

  • id
  • campaign_id
  • interviewer_id - contact id of person searching
  • interviewer_display_name
  • interviewer_email
  • interviewer_ip
  • sql [Joe: I'm unclear what is stored here.]
  • search date
Labels
  • None
  1. Mar 04, 2010

    Is this to reimplement the Drupal stuff done by Will B more fully within CiviCRM?

    Some questions about clarification, and maybe a suggestion or two:

    1. Voter Campaign Table, uf_id - is this civicrm_uf_group.id? It needs to be identifying a structure like a profile that can store an interview script with several possible questions. One kind of voter ID script typically would include the following kinds of questions:
      1. Hello, my name is Joe, and I am calling on behalf of Z, your Amazing Party candidate. [Framing stuff.] Can Z of the Amazing Party count on your support on Election Day? [encode responses from an option group]
      2. [If a supporter:] Oh, great. Would you be willing to take a sign for your lawn, balcony or window? [Encode response from an option group]
      3. [If a supporter:] There's a lot of work to be done between now and election day. Would you be able to volunteer? Y/N
      4. Other questions that may be used ask about likelihood to vote, and second party preference in races with 3 or more strong parties/candidates.
      5. A simple simple script structure without conditional branches based on response values is a good starting place.
    2. Voter Campaign Table, uf_result_id: I don't understand the purpose of this field. Is this referring to a structure that will be used to set a recontact interval per subject based on their responses? How does this interact with the serialized array for recontact intervals for each status? Is it a reference to civicrm_uf_field.id?
    3. It may be appropriate to take contacts back from an interviewer who has had them for too long without completing the interviews. The period that is too long will vary by campaign. So perhaps add a field to support this in the Voter Campaign Table.
    4. Voter Interview Table, answers. Might be a good idea to normalize this a bit more into another table since reports will need to tally these values extensively. I'd prefer something more along the lines of civicrm_value_xxx custom field group tables. It's reasonable to expect to need to support more than one version of a script, but it might be better to require setting up a different campaign. It's not clear that the flexibilty in storage suggested will meet the user needs for determinate, consistent questions and response encoding. A different model might be to allow modifications to the script (eg questions asked, possible responses encoded) during a campaign, but for each change, a different version of the custom field group table would be created (an answers entity table field in would need to be added to the Voter Interview Table).

    I hope this helps.

    1. Mar 04, 2010

      clarified a few things and switched to a multiple record custom group to archive all answers. I dont think having a script (simple or conditional) is part of the current plan. You might want to download Will B's code and read the related blog posts for some field definitions. this is just a starting point while we start thinking about migrating it to CiviCRM

      While comments are good and valued, we definitely would appreciate if folks could step up and help sponsor the work and or contribute developer hours

      lobo

      1. Mar 04, 2010

        Makes sense - I'll see what I can do to rustle up some funds. I realize increasing scope is not that appreciated (wink) but sometimes assistance with use cases can help generalize requirements without adding (too much) to development costs. Anyway, that was my hope. I'll research more before commenting more.

        Cheers,

        Joe

  2. Mar 04, 2010

    Note that some of these features can already be accomplished via the CiviAssign Drupal module discussed here: http://forum.civicrm.org/index.php/topic,4628.0.html

    It's built on top of the existing 'Activity' system, so it requires no changes to CiviCRM's database.

    This was successfully used for canvasing by a political campaign last year.

    1. Mar 13, 2010

      CiviAssign only appears to work with Drupal 5. Have you heard anything about it being updated for Drupal 6?

      1. May 09, 2010

        Sorry, missed this reply. CiviAssign would be very trivial to upgrade for Drupal 6. I'll probably get around to it one of these days, unless CiviCanvass makes it unnecessary.

  3. Mar 07, 2010

    I have tried adding an attachment that is a tidied up script from the IRC discussion this morning. It seems to appears only as a wee paperclip at top left. Anyhow may be of use for those who missed the chat.

    I will be coming back with some input on UK + NZ + Aus polling booth practices in case that can help establish generic approaches - eg in NZ the polling booth staff read out the Page Number and Line Number of the voter - this then needs to be looked up on a paper version of the roll, to find the Name and Address of the voter.

    EDIT: of course in Aus voting is compulsary - so the issue of chasing voters to go to the polling booth isn't such an issue there!

    1. Mar 05, 2010

      Peter, thanks so much for the editted irc log - really helped fill me in on stuff.

      1. Mar 07, 2010

        No worries - it drops in to neooffice (and probably excel) really easily and improves the readability no end.

  4. Mar 11, 2010

    1. Could the status of the interview be just another question / response in the script? For a phone canvass, for example, I'm imagining these statuses would be things like Left Message, Answering Machine, Not In Service, Wrong Number, Not Home (when someone like a child answers and says voters are not home), Did Not Complete call (eg caller hung up before responding, sometimes put in with hostile responses which are about getting phoned rather than candidate/campaign), Completed. (I think the call centre business terms this the call disposition.)
      I also imagine it might make sense to have a field for when to call back, or at least a notes section. This is particularly useful on E-Day, when people indicate when they intend to go to vote.
    2. Going with this idea of simplifying things, I'm wondering about how the recontacting / reclaiming workflow happens. I would imagine that in some cases one wants to phone back all numbers with certain statuses (eg LM, AM, NH), while in other cases one wants to phone back people with a certain response to a question (eg people who are undecided about who to vote for or who are leaning towards the campaign's candidate, or just supporters during GOTV canvassing). Making the status just another script question from a system point of view might be advantageous.
    3. I'm just wondering if there needs to be two interfaces rather than one that can handle both GOTV data entry and responses to a script. I think there are good reasons to make it very easy to enter yes someone has voted. What is less clear to me is that this interface wouldn't be useful in other situations. In the GOTV case of what we term an Inside Scrutineer (ie the person in the polling station recording who comes to vote, as opposed to an Outside Scrutineer who knocks on supporters' doors to get people out), I imagine a column with a checkbox for a yes/no response, with a filter that either hides those with Yes or displays them with strikeout font through their names. In other cases, like data entry of responses from a walklist, there might be two or four columns, and no filtering.
    4. In many but not all of Canada's federal, provincial, and municipal elections, voters on the voters list have a number associated with them either for the election or permanently. In some cases this number is used to order the list, and is used as an easier way to uniquely identify voters than their name (often father and son in a house have same name, for example). Certain jurisdictions have the lists in polling station ordered alphabetically by name, some by street address then alphabetically by name within the household, some by street walk order (ie odd numbers for a street then even numbers for the same street) then alphabetically by name. To go back to the numbers case, voters added to the voters list during the campaign sometimes get numbers that are distributed to campaigns before E-Day, and sometimes are treated differently, just being listed at end of the voters list alphabetically wihtout numbers. So it would be useful to allow the sort order of lists to be configurable via a hook.
    5. Some jurisdictions allow voters who are not officially registered to vote by getting sworn in on E-Day. This means campaigns need to track supporters who are not on the list both during campaign and during GOTV efforts. In most cases the focus is just on whether the campaigns' supporters have turned out to vote, so just having their names in CiviCRM and available on the list displayed in the polling station should normally be enough. Occasionally in tight races there may also be a concern to document any potential justifications for a recount, which can include people voting who are not eligible, people voting multiple times in different polling stations, etc. So there might need to be an ability to add contacts from inside the polling station during GOTV efforts, though the focus needs to remain on identifying whether a supporter has voted.
    6. In many cases in Canada, 'marked' voters lists are not allowed in polling stations. This means not bringing in / letting anyone see any list that identifies who is or is not a supporter of a campaign. So it would be unacceptable to have only supporters show up in a filtered list on a screen, or to be identified in different font or colour or have their voting intentions displayed in a column inside the polling station.

    I hope this provides a richness to possible use cases, and perhaps allows the system design to be simplified and generalized.

  5. Mar 16, 2010

    We gave CiviCRM our first test drive for GOTV during our recent municipal elections.  Looking forward to the Canvassing and GOTV modules; maybe our recent experience will help with the design:

    1. GOTV

    We created separate organizations for each of our campaigns (candidate, office, and year specific).  We tracked voter bias (s=supporter, l=leaner, o=opposed, u=undecided, r=refused to answer) by creating a relationship between the voter and the campaign.  We also imported a good amount of historial bias data, so we had SLOUR relationships for a couple dozen prior campaigns.  Candidates were collecting bias data as they went door-to-door, and we set up about two weeks of phonebanking.  We (in the phonebank) focused on contacting voters who were never IDed for a past campaign.

    For our GOTV effort, we created a sidecheck list (the list we use at the polls to mark off people as they vote), that contained any voter who ever supported any candidate of ours, excluding any voter IDed as opposed to the current candidate.  We talked about tweaking our use of the old ID data a bit (why turn out a voter who has supported one campaign but opposed others?) but didn't have time to fine tune.

    To creating these lists we set up a smart group for: all voters IDed as supporting any Progressive's campaign ["(S)upported Any Prog"]; one for all "(L)eaners to Any Prog"; and one for all who ever "(O)pposed Any Prog".  We ignored U & R for the GOTV list creation.  We then had to create smart groups for voters opposed to our current candidates ["(O)pposed Jane Doe"].  I used the include/exclude custom search to create another smart group ("Jane Doe GOTV") that included "(S)upported Any Prog" and "(L)eaner to Any Prog" and excluded "(O)pposed Jane Doe".

    In VT, we can vote by mail or in person for several weeks prior to election day.  As that data came in, I created an event for the current election, a participant role of voter, and a participant status of voted.  I then created a smart group of contacts who had already returned ballots ("Voted March 2009").

    Finally, for the sidecheck list, I created a smart group from the include/exclude custom search that included "Jane Doe GOTV" and excluded "Voted March 2009". It got the data we needed, but I am worried about "group creep," if we need to create four separate groups (supporter, leaner, opposed, voted) for each candidate each election.

    What we didn't figure out is how to easily generate a report of people who have requested absentee/mail in ballots, but have not yet returned them.  I figured that again I could do that through a participant status field ("requested absentee") but ran out of time. There is advantage in having a record of absentee requests, so I don't think replacing "requested absentee" with "voted" is the answer.

    2. Phone Lookup

    For phone number lookup, we created a smart group that included all voters in the districts we had campaigns where we did not have phone numbers.  We generated excel lists for people to use to look up numbers, which didn't work for two reasons. 1) A few decided to print the list and write the numbers down, and given the sluggishness of civi we could have someone looking up and entering numbers as fast as entering hand written numbers. 2) We didn't tag what numbers were out where, so by t-7 days we really didn't know who was looking up what.  We also had some people entering this directly in civi (searching on that neednumber group and plucking them out one at a time).  This worked well.

    The only bonehead error we made was that we might have called voter X a couple weeks before the campaign, found the phone number was bad and deleted it, returning that contact to the neednumber smart group.  Our excellent volunteers then saw the number was needed, looked it up in the phone book, and entered the same, still incorrect number.  Going forward, instead of making a "bad phone" tag or flagging "do not phone" we are entering "XXX" for the area code, so that we can still see what the bad number was.  We will update the smart group to include numbers that start with "XXX".  Don't know if that is smart, but it should mean we don't have volunteers saying "I already entered this number!"

    Our phone number lookers aren't regular civiCRM users, so I would also like to get them to the list of needed numbers within one click of login.  I'd also like to have them edit in a table view (batch update via profile) in order to speed up data entry, assuming contacts can be "checked out" so that we don't have two people looking for the same contacts at the same time.

    3. Walklists

    For generating walklists, we primarily exported all voters in a district to excel, and then created a few columns to try to split house numbers and street names into separate fields.  We then sorted by street name, then odd/even house number, then by house number to get a list that could be used on one side of the street.  There was a lot of manual correction that needed to be done, primarily due to apartment numbers that got left in the street name field, or that were picked up in the house number field, or addresses such as "25 ½ Maple Street".  Other than that, it worked OK, and we could include other fields (like date last voted or year of birth) so that candidates could prioritize what houses they stopped at.

    BTW, running civiCRM 3.0.4.

  6. Mar 23, 2010

    Some Questions/Comments and a Facebook Causes Page to support this project

    I am in the process of building a community organizing website using Drupal and I am highly intrigued by this project.  I had a couple of questions/comments that I hoped I could run by the community here.  I also ran this by Lobo and got a quick and very helpful response.  I thought I'd post my comments/questions here as well to hear any other ideas.  I also think it is very important for people to make financial contributions to support this very admirable project and I discuss that below as well.  I've even made a Facebook Causes Page to support this project (http://apps.facebook.com/causes/causes/463499/about), which I'll mention again below.

    My goal is to create a community organizing website and I've been searching for useful modules, which brought me to CiviCRM and this project to add greater canvassing and GOTV functionality.  Needless to say, I think this may be exactly what I'm looking for.
    Part of what I want the website to enable is keeping track of contact data while running a phone and walk canvass.  It seems pretty clear that the features being worked on here would cover this, which is awesome.

    I am also interested in enabling site users to contact their elected representatives by phone for different campaigns and to track each time that a call is made.  It may also be helpful to keep track of how the call went (whether the representative is receptive to the campaign or not for example).  Advomatic has a module they call a "Political Click-To-Call Tool," which I think goes further than is necessary.  A better example of what I have in mind is being implemented by whipcongress.com.

    Basically, users would log-in to the site, go to a political calling campaign page, enter a zip code, get a list of their representatives and numbers for their area, and then record that they made a call and what the response was if any.

    >>>Do you think this canvassing project may also be adapted for this kind of campaign or do you know of any modules that support a campaign for calling elected representatives?

    I am wondering if it might work with this "CiviCanvass" project this way:

    All elected representatives (senators and house representatives in the U.S. for example) would be made part of a calling group on CiviCRM.  Then a campaign could be connected with this group (for example, encouraging senators to vote for X legislation).  Site users would go to the campaign page and enter a zip code to find the representatives for their area.  They then would call them and note how it went in some way.  The system would track number of calls and how the calls went.

    >>>Do you think that by using CiviCRM and with the added functionality of this canvass project, I could create a contact group of legislators to do a political call campaign (i.e. allow users to go to a campaign page, enter a zip code, and access legislator contact pages to call them and record results for the campaign?  While this would include data entry of all legislators' contact info into contact pages up front, do you think it could work?  Or would other features need to be added?

    I ran the above scenario of a campaign for calling elected representatives by Lobo and he thought that it could work.  He said it would depend on the exact feature set and my [specific] needs.  I think that the features already planned could be adapted for this kind of campaign.  Lobo also mentioned that there are not currently any plans to integrate with legislative databases, but that it could be done later.  Again, I don't think linking with a legislative database would be necessary for this kind of campaign; instead you could enter legislator contact info into contact pages up front.  Lobo also said that he plans to add click-2-call stuff for a future release (which again, I don't believe is necessary for these kinds of campaigns, while it may make them more efficient).

    Any thoughts on the possibility of doing a legislator calling campaign with this project or any other modules out there that could do this is much appreciated!

    Even without this added use, this project is incredibly important.  Creating an open-source and free module that has these features is a true gift to humanity.

    I worked for 5 years at a community organizing non-profit and I recognize the power of door-to-door and phone canvassing to organize people and raise contributions.  Technology to support grassroots organizing should be free and open source.  CiviCRM is an amazing project for promoting community organizing and this canvass project is a critical expansion.  It is this kind of technology that I believe can connect with on the ground work to make transformative change.

    I am currently a public school teacher in Washington, DC and I am making a small contribution.  I hope that you will make a contribution that is comfortable for you as well.

    >>Can you support this project and encourages others to do the same?!  I also made a Facebook Causes Page to encourage people to contribute to the project as well.

    Please also let me know what you think of the page, any suggestions, and also promote the page if you think it's worthwhile.

    Thank you all very much for your work.

    Best Regards!

  7. Mar 26, 2010

    Hey,

    Are we going to implement Constituent Canvass Table with as an activity type?  Seems like it has all the fields that we need.

    Michael


Creative Commons License
Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-Share Alike 3.0 United States Licence.