This documentation refers to an older version of CiviCRM (3.4 / 4.0). View latest version.

Skip to end of metadata
Go to start of metadata
This page refers to CiviCRM 3.4 and 4.0, current STABLE version.

Documentation Search


CiviCRM 3.4 and 4.0 Documentation

Support and Participation

Developer Resources


CiviCRM books!

Make sure to check out the FLOSS Manual Understanding CiviCRM as well! You can also support this project by ordering a hard copy.

Or support us by buying an eBook or hard copy of Using CiviCRM from Packt Publishing.

This page describes how the CiviMail mailer queues jobs, composes messages, and handles events.

Some of the information in this document is obsolete. Refer to CiviMail Action Tokens for updated information on token replacement.

Jobs

Queuing

Given a job, find all email addresses that should receive the mailing (job.mailing_id). 'email address' is taken to mean the primary email of the primary location of a contact, unless one (or both) of the email and location fields in the contact-group join table are non-null (see CiviMail). For inclusion, only contacts where do_not_email = 0 and bounce_hold = 0 are considered.

If a job has is_retry = 1, the queue is generated by finding all jobs with the same mailing_id and joining to bounce event (through queue). Otherwise, the queue is generated as follows:

  • Included recipients (group_type = Include)
    • Static groups
      • All contacts belonging to (status = In) any group in crm_mailing_group where mailing_id = job.mailing_id.
    • Saved searches
      • All contacts matching the where clause of a saved search linked from groups in crm_mailing_group.
    • Previous mailings
      • All email addresses in Mailing_Event_Queue with the job.mailing_id keyed from the mailing-group join table. Only jobs with Complete status should be considered.
  • Excluded recipients (group_type = Exclude)
    • Static groups
      • All contacts belonging to (status = In) any group in crm_mailing_group where mailing_id = job.mailing_id.
    • Saved searches
      • All contacts matching the where clause of a saved search linked from groups in crm_mailing_group.
    • Previous mailings
      • All email addresses in Mailing_Event_Queue with the job.mailing_id keyed from the mail-group join table.
    • Successful deliveries in previous jobs of the same mailing
      • All email addresses in Mailing_Event_Queue where job.mailing_id = mailing_id inner join to Mailing_Event_Delivery and left join to Mailing_Event_Bounce where Mailing_Event_Delivery.id <> null and Mailing_Event_Bounce.id is null.

Location/Email selection

By default, contacts are joined to a group with no location or email preference. In this case, the destination email is chosen by taking the primary email of the contact's primary location. If the contact has only one email or location, this works as would be expected.

If the contact has multiple locations, a location preference may be set for each group to which that contact belongs. If the membership has a location preference, the primary email address of that location is chosen for the destination. For advanced users, an email address preference may also be specified at the group membership level, overriding the location preference.

Note that a contact may only have one subscription record for the group, so the mailing will go to at most one of the contact's email addresses.

Status

Job status can take one of 5 states.

  1. Scheduled: All jobs start in this state, and stay there until the start_date has passed.
  2. Running: Jobs are marked Running only after the entire recipient queue (Mailing_Event_Queue) has been constructed. If the queuing process is interrupted, the job will remain in Scheduled and the queue will be reconstructed the next time the mailer is run.
  3. Complete: Jobs are marked complete after the mailer has attempted to send the message to every recipient in the queue (ie. every queue event has a coresponding bounce or delivery event). Note that slow bounce events may continue well after the job has been marked Complete.
  4. Paused: A job can only be marked paused by the admin interface. The mailer will not act on paused jobs.
  5. Canceled: Like paused, but cannot be placed back in the Running state.

Composition

VERP Headers

All outgoing mail uses the following associated VERP addresses:

Address

Use

bounce.JOB.QUEUE.HASH=EMAIL@FIXME.ORG

Return-path/bounce handling

reply.JOB.QUEUE.HASH=EMAIL@FIXME.ORG

Reply to the author of the mailing, if configured

unsubscribe.JOB.QUEUE.HASH=EMAIL@FIXME.ORG

Unsubscribe from the group(s) of this mailing

optOut.JOB.QUEUE.HASH=EMAIL@FIXME.ORG

Unsubscribe from the domain

Where JOB is the ID of the Job, QUEUE is the ID of the queue event for this particular message, and HASH is the SHA-1 of the job ID, contact's email ID, and contact ID. The email and contact IDs are never exposed to the recipient within the message. The recipient's email address is VERP-encoded (=EMAIL) in the address.

Additionally, the following addresses are used only for inbound processing:

subscribe.DOMAIN.GROUP@FIXME.ORG

Subscribe a contact to a group

confirm.CONTACT.SUBSCRIBE.HASH=EMAIL@FIXME.ORG

Confirm double opt-in for group subscription

Token replacement

Users will be able to customize outgoing mails by using tokens. Tokens fall into four categories: domain, contact, action, and mailing.

Required Tokens

The following tokens must be present in the TEXT or HTML of your message. Send Mailing will throw an error if any are missing:

  • {domain.address}
    
  • {action.optOut}
    

Domain-level tokens are substituted with values pulled from the civicrm_domain table, and are only generated once for all messages in the mailing. These tokens are prefixed by domain.

Token

Value

{domain.name}

Name of the domain

{domain.address}

Meta-token constructed by merging the various address components from civicrm_domain

{domain.phone}

Phone number for the domain

{domain.email}

Email address to contact the domain

add more tokens here...

Contact-level tokens are replaced with values specific to each individual contact receiving the mailing. These tokens are prefixed by contact.

Token

Value

{contact.display_name}

The contact's display_name (also used in the To: header)

{contact.first_name}
{contact.last_name}
{contact.custom_ID}
...

List of available contact-level tokens

Additionally, there are contact-specific tokens for several actions. These tokens are prefixed by action.

Token

Value

{action.forward}

Link to forward this mailing to an unsubscribed user

{action.donate}

Link to make a donation

{action.reply}

mailto: link to reply

{action.unsubscribe}

mailto: link to unsubscribe

{action.optOut}

mailto: link to opt out of the domain

add more tokens here...

There are also mailing-specific tokens:

Token

Value

{mailing.groups}

The list of target groups for this mailing

{mailing.name}

The name of the mailing

Other mailing components have tokens as well.

Unsubscribe:

Token

Value

{unsubscribe.group}

A bulleted list of groups from which the contact has been unsubscribed, along with web links to resubscribe.

HTML vs Text

Since we support both HTML and Text formatting of outgoing mail, we will need rules for how the tokens are used in both cases. This is mainly an issue for action tokens, but certain other tokens (such as domain.address) may be formatted differently in HTML mode.

In HTML content, action tokens should always be used as if they were URLs. If the action is an email address, a mailto: prefix will be added automatically.

Action Example.html

In text content, action tokens will be substituted directly.

Action Example.txt

If the action is a URL (eg forward or donate), the URL will be substituted directly in either case.

I18n

In order to support locale-specific token codes, a separate class will be created to map tokens to their localized equivalent. Instead of matching directly on a token, the token replacement code will match on the localized version of the code. Substitution then continues as before.

Null values and defaults

Many non-required contact fields are exposed to the token processor, and not every contact will have values for every token. By default, if there is no value for a token field, it is replaced with a blank string. This can be overridden in the document as follows:

Default Override Example.txt

URL Tracking

When URL Tracking is enabled in a mailing, all links in the body of the message will be inserted into the civicrm_mailing_trackable_url table. The links will then be translated to point to a redirect script which is passed the civicrm_mailing_event_queue.id and the trackable URL ID as GET parameters, and registers the event. 

One limitation of the current URL tracking is that the link text in HTML messages should not contain "http" otherwise it will be replaced by the tracking link.  So instead this

use a format without the http such as

Events

  • Delivery
    • Registered after a succesful SMTP transaction.
    • Action
      • Add a new row in Mailing_Event_Delivery with the queue_id.
  • Bounce
    • Registered after an unsuccesful SMTP transaction (fast bounce), or by the inbound processor (slow bounce, see: CiviMail Mailer Settings)
    • Action:
      • Add a new row in Mailing_Event_Bounce with the queue_id, bounce_type and bounce_reason returned by the bounce processor
      • Count the bounce events for email_id and compare with the hold_threshold for the matching bounce type. If the email address has more than the threshold of any type of bounce, place it on bounce hold.
Labels
  • None