CiviMail

CiviMail Component Overview

Front End
- Enter Name of Mailing
- Choose Target Set
  - choose upto 5 include groups / saved searches
  - choose upto 5 exclude groups / saved searches
  - choose upto 5 exclude recipients from previous mailing
  - (mail only sent to primary email address for now)
- Choose Header and Footer
- Upload text and/or html contents
- Test mailing (i.e. generator is included in the run here, no job creation etc)
- Schedule / Send the mailing

Mail generator
- For all scheduled mail
  - Figure out the valid contact_id set and generate the emails for that mailing and put in the mailing queue
  - Let postfix handle the queuing for now

Mail processor (ie. postfix + amavis)
- Deliver email
- Process incoming mail (subscribe/unsubscribe/bounce/spam complaints). Sub/Unsub is both at the org level and the group level
- Write into DB various statistics regarding jobs and deliveries etc
- Add Mailgraph support
- Apply to AOL etc for whitelist support

Reporting
- Framework to generate reports for EN based on various events

Mailing Object Table (Mailing)
- name
- from name
- from email
- reply to email
- auto response?
- header id
- footer id
- response id
- subject
- body of text
- body of html
- is_template (this will not create a job)
- open_tracking (boolean)
- url_tracking (boolean)
- forward_replies (boolean)
- is_completed (boolean)

Join table to associate groups with Mailing Object (Mailing_Groups)
- Mailing id
- entity name (group or mailing object, eg for exclusion of previous mailings)
- entity id
- type (include or exclude)

Job table
- Mailing id
- scheduled date
- start date
- end date
- status (enum: scheduled, running, completed, paused, canceled)
- is_retry

Events
- Email Queue (was EventEmailSent in EN2)
  - Job ID
  - Email ID
  - Contact ID
  - Security hash
- Email Delivered
  - Event Email Queue ID
  - Timestamp
- Email Bounced
  - Event Email Queue ID
  - Bounce type id
  - Bounce reason (substring that matched regexp)
  - Timestamp
- Email Forwarded
  - Event Email Queue ID
  - Destination Queue ID
  - Timestamp
- Email Opened
  - Event Email Queue ID
  - Timestamp
- Trackable URL Open
  - Event Email Queue ID
  - Trackable URL ID
  - Timestamp
- Reply
  - Event Email Queue ID
  - Timestamp
- Unsubscribe
  - Event Email Queue ID
  - Timestamp
  - Org Unsubscribe (boolean, tells whether the contact wants out of all org emails, or just this group(s))
- Subscribe
  - Group ID
  - Contact ID
  - Security hash
  - Timestamp
- Subscribe-Confirm
  - Event Email Subscribe ID
  - Timestamp

Trackable URLs
- URL
- Mailing id

Bounce Type
- Name (AOL, Away, etc..)
- Description
- Required bounce count for hold

Bounce Patterns
- Bounce type id
- Regexp

List (Group) and Organization Subscriptions

Overview

Contacts are 'subscribed' and 'unsubscribed' to one or more Groups. They also have an opt-in or opt-out global status at the domain level (e.g. the organization 'owning/maintaining' the site data).

Each contact's domain 'opt-in' status is stored in civicrm_contact.is_opt_out (boolean). Contacts are opted IN by default when they are created (is_opt_out is FALSE). Opt-out is set to TRUE when a global opt-out request is received from a contact. This prevents them from receiving any future CiviMail mailings. This mechanism is particularly important given the Smart Groups feature - in that it creates a 'hard opt-out' for a given contact who says "I no longer want to receive any kind of mail from this organization."

'Explicit' group subscriptions (membership) statuses are stored in civicrm_group_contact. For 'smart' groups (where membership is primarily based on saved search criteria), explicit unsubscribes are also stored in civicrm_group_contact. Available status values are ADDED ('subscribed'), PENDING ('awaiting double opt-in confirmation), REMOVED ('unsubscribed').

History of subscribe/unsubscribe activity (domain and group) - e.g. status changes - is stored in civicrm_subscription_history.

User Action -> Processing Steps

1. Create new contact

Set contact.is_subscribed = TRUE (1) for new record
Insert subscription_history record: contact_id = $cid, group_id = NULL, status = IN, datetime = now(), method = $method

2. Add contact to group (w/o double opt-in)

Insert group_contact record (status = ADDED)
Insert subscription_history record: contact_id = $cid, group_id = $gid, status = ADDED, datetime = now(), method = $method
API = crm_add_group_contacts()

3. Subscribe via email (with double opt-in)

When subscribe request email received:
- Insert group_contact record (status = PENDING)
- Insert subscription_history record: contact_id = $cid, group_id = $gid, status = PENDING, datetime = now(), method = 'Email'
  API = crm_subscribe_group_contacts()

When confirmation email is received:
- Check that current group_contact status = PENDING.
  - if true, update group_contact record (status = ADDED)
  - if false, return error: "Can not confirm subscription. Current group status is NOT Pending."
- Insert subscription_history record: contact_id = $cid, group_id = $gid, status = ADDED, datetime = now(), method = 'Email'
  API = crm_confirm_group_contacts()

4. Unsubscribe contact from group (UI or via email/web-form request from contact)

Update group_contact record (status = REMOVED)
Insert subscription_history record: contact_id = $cid, group_id = $gid, status = REMOVED, datetime = now(), method = $method
API = crm_delete_group_contacts()

5. Opt-out contact from organization (global opt-out for all groups in the domain)

For each group_contact record
- Update record (status = REMOVED)
- Insert subscription_history record: contact_id = $cid, group_id = $gid status = REMOVED, datetime = now(), method = $method}
Insert subscription_history record: contact_id = $cid, group_id = NULL status = REMOVED, datetime = now(), method = $method
Set contact.is_opt_out = TRUE (1)

6. Delete contact

Additional step is needed to delete all subscription_history records

7. Exclusions when building recipient list for a mailing to a group:

contact.is_subscribed = FALSE OR contact.do_not_email automatically excludes from mailing

ISSUE: To clarify the enum values for subscription_history.method. Current values are: Admin', 'Email', 'Web', 'API'.

CiviCRM 'user' adds contact to a group or removes them via our UI
- method = Admin (should we also record the logged in user ID?)
Contact subs/unsubs by sending in email request
- method = Email
Contact subs/unsubs via web form
- method = Web
Contact is subscribed/unsubbed via API call
- method = API

Inbound processing

The inbound mail engine (postfix and amavis) will need to perform many core operations, including subscriptions, unsubscriptions, bounce handling, automated reply, etc. Rather than duplicate a large amount of DB code (as in EN2), the system should work via API calls from inbound (perl) to CiviCRM (php). See SOAP Access to CiviCRM API.

Changes to Current Schema

add mail_preferences enum( 'html', 'text', 'both' ) to crm_contact (remember to never display enum values directly and use them just as keys! That's i18n requirement.)

add is_subscribed boolean to civicrm_contact (org level opt-in/out)

Implement is_active model for civicrm_group (we can no longer delete Groups because we would have FK constraint issues w/ subscription_history records).

Modify civicrm_group_contact table
- Remove _date and _method columns
- Add location_id (nullable: if defined, location's primary email trumps contact)
- Add email_id (nullable: if defined, email id trumps contact and location primary emails)

New table = civicrm_subscription_history
- contact_id
- group_id (optional)
- date (datetime type column)
- method enum('Admin', 'Email', 'Web', 'API')
- status enum('Pending', 'Added', 'Removed')
- IP address and/or any other tracking information
- agent ??? (store login of user if done via CiviCRM UI or API, 'self' if by contact themselves)

Add bounce status fields to crm_email:
- on_hold
- hold_date
- reset_date

Add a group_id fkey to associate a static group with a saved search. This is hidden and invisible to the user. However internally all saved searches are treated as groups and we use group ids from a user perspective. We need to auto create a group on creation of a saved search

Add table for header / footer / auto-response
- id
- domain_id
- type enum( 'Header', 'Footer', 'Response' )
- body_text
- body_html
- is_primary

Create UI to add multiple header / footer with one default

Add fields to civicrom_domain:
- email_domain (@address for outgoing email)
- display/first/middle/last names of contact

Reporting

Here are some screenshots of the reporting interface.

	The main reporting screen for a mailing
	The main reporting screen continued
	An example page for browsing events such as click-throughs, deliveries, bounces, etc.