This configuration is a community contribution. It is also community supported. The core team does not have any experience or detailed knowledge of this configuration
Alternative return path configuration with imap2soap.pl (Experimental!)
The method described on this page may be useful for you if you are trying to install CiviMail and any of the following hold true:
- you are on a shared web hosting service, so do not have the ability to set up the CiviCRM version of Amavis per the CiviMail installation guide;
- you do have a dedicated server or virtual private server, but you are having difficulties understanding or implementing the Amavis and MTA configuration;
- you are running on aqmail (or other non-Postfix) system;
- too many incoming messages are swamping your server's capabilities;
- you simply want a simpler mechanism than Amavis.
- you are uneasy about installing the rather elderly civicrm amavis package (Amavis version 2.4.2, current ubuntu (=8.04) version is Amavis 2.5.3)
Note that this is an addendum to the setup instructions for CiviMail. You still need to follow everything in those instructions that does not relate to Amavis/Postfix. This includes Domain setup, SMTP configuration, and setting up the cron job. Keep in mind that CiviMail requires PHP 5 compiled with the SOAP and DOM extensions; it won't work with PHP 4. If you have a SOAP-less PHP 5, there is a section below on installing SOAP::Lite that you can follow.
Imap2soap.pl is currently being maintained by Fen Labalme; you can find the most recent version at https://svn.civicactions.net/repos/civicrm/scripts/ (the versions attached to this page are older and out-of-date).
imap2soap.pl Feature Summary:
- Run interactively or called by a cron script
- Set message processing rate, maximum message count and high load average quit trigger
- Support for IMAPS (SSL/TLS encrypted IMAP); see the comments in the code if you need this support
- Interactive IMAP mailbox viewer and trimmer - also helps with connection troubleshooting
- Perldoc documentation (try
- Better overall error handling and reporting
Installation - Five Simple Steps:
This method uses the imap2soap Perl script which is attached to this page. To set this up you need to follow these five steps:
- Have a working IMAP email account somewhere for your domain: e.g. firstname.lastname@example.org. Your mail host may well be your web host (your website sends emails from "localhost") or any SMTP host you use to send email from CiviCRM. Forward all messages for processing to this address. The easy way is to set up a 'catch-all' email address on your domain, which receives all messages that do not match any other existing address on your domain. This could catch a lot of junk mail too, but the script moves any messages that do not conform to the CiviMail VERP format to the "Junk Folder" where you can inspect and delete them with your normal IMAP email viewer.
- Create a basic user account on your CMS (Drupal or Joomla) e.g. username: civimail password: whatever. This is your SOAP user (see next step). For Drupal this can be any authorized user account. For Joomla, this needs to be a 'Manager' account (at least).
- Download the imap2soap script and sample configuration file from https://svn.civicactions.net/repos/civicrm/scripts/
imap2soap.pl somewhere you store executables, like
~/bin/ and ensure it's executable (e.g.,
chmod 755 imap2soap.pl. Note the Perl modules required in the script and make sure they are installed on the computer you will be running the script on.*
imap2soap.conf someplace easy to edit from the user that will be calling it, say
~/civicrm/imap2soap.conf and for security purposes, protect it from being world readable (e.g.,
chmod 600 imap2soap.conf). Note that if you invoke
imap2soap.pl without including the
imap2soap.pl looks for the configuration file in the current directory.
imap2soap.conf and enter the usernames, passwords and servers from steps 1. and 2. There is a clearly marked section in the file where you need to insert your own settings for SOAP server and email server.
- Run the script to process messages in the Inbox.
- Manually - especially useful for testing IMAP connectivity - just invoke
- Set up a Cron job to run the script:
|Example Cron line|
23,53 * * * * /usr/local/bin/imap2soap.pl -q -w 1 -l 4 -L 1 /PATH/TO/imap2soap.conf
What's going on
When the script runs, it will:
* connect to your mail server using IMAP (or IMAPS),
* create imap folders to store processed messages if necessary,
* send any bounce/unsubscribe/etc. messages to CiviCRM via SOAP, and
* file processed messages in IMAP folders.
You can log into this mail account using
imap2soap.pl -i /PATH/TO/imap2soap.conf (or with a normal imap email client) to see the progress of the operation and perhaps to trim (remove old messages from) the Junk folder. A future version or imap2soap.pl will allow moving messages between folders, say from the Failed folder to the Inbox to try processing failed messages again. (If you want to do that now, you'll have to use an external IMAP mail client such as Mozilla Thunderbird.) If you run a lot of large mailings, you might want periodically to clear out these folders, using your email client, and save any emails locally that you want to archive.
Known Compatible Hosts
List of hosts known to support the required modules for this installation (please add / edit!)
Open Social Sites
Hosting service designed specifically to support CiviMail. A working CiviCRM / CiviMail is pre-installed for you to customize. (N.B. In the interests of full disclosure, this section and script is contributed by Open Social Sites!).
CivicSpace On Demand
If setting up CiviMail is all too much for you, CSOD could be for you. This is a CiviCRM 'ASP' that gives an 'instant', managed CiviCRM installation.
You may need to request that the following PERL modules be installed on your server:
You can use CPAN to install modules SOAP::Lite, Mail::IMAPClient, Mail::Verp, HTTP::Cookies, MIME::Entity, and dependencies into local directories (on model of SOAP::Lite below) and tell the .pl script to look for them there.
You will also need a version of PHP configured with --enable-soap. You can "roll-your-own" (compile from source a) PHP binary in a local directory and redirect server to it following these detailed instructions.
What's this return path thing and why do I need it?
The "return path" is the method by which CiviMail receives and processes messages via email: bounces from unsuccessful email deliveries, replies and requests for subscriptions to or from mailing lists. Without it, CiviMail will not know about bad email addresses in your mailing database, nor will it be able to process requests to unsubscribe, or forward replies to the appropriate address. At best you will not be able to use many of CiviMail's features. At worst, you will quickly find yourself (and probably your hosting / server provider) on a spam blacklist. These concepts are covered elsewhere in the wiki, but for convenience a summary of the return path system is provided here:
- When you create a mailing in CiviMail, you select one or more CiviCRM Groups to send the mailing to. (You can also exclude members of particular groups, and include or exclude recipients of previous mailings.).
- When CiviMail sends the mailing out, the From address in the email header is set to whatever address you elected while setting up that mailing (the default email address, unless you specifie otherwise). This is who the email appears to be from for recipients.
- However the Reply To address is set by CiviMail to an email address that encodes information about the recipient in a so-called VERP string. This is of the form: type.domain-id.job-id.queue.id.hash-recipient=recipient-domain@yourCiviMail-mail-domaine.g. email@example.com
where "type" could be "bounce", "reply", "optOut", "unsubscribe" or "confirm". There is an additional VERP format used in links for users to request a subscription to a mailing list:
Your incoming mail server must be set up to accept all emails with a valid CiviMail VERP format. Note this does not correspond to an actual valid mailbox on the mail server, hence the need to either set up a virtual-regexp or equivalent filter to specifically forward emails with a valid CiviMail VERP format to a valid e-mail address on the receiving server that is used to receive and process incoming messages for CiviMail, or use a "catch-all" to forward all incoming emails for which there is no valid recipient email address to an email account your use for CiviMail inbound processing.
The imap2soap script (like Amavis in the standard CiviMail configuration) intercepts these VERP encoded email addresses, extract the data encoded into them, and use the "SOAP interface" to CiviCRM to transfer this data to your server, where CiviCRM act on them accordingly eg adding or removing a user from a mailing group, forwarding a reply, etc.
imap2soap vs. mail2soap and new-mail2soap
The imap2soap script has a few significant improvements over new-mail2soap:
- You no longer need to have read / write access to the Maildir for the in-bound e-mail account
- Logging into the inbound email account with a mail client no longer messes up the location of the mail files (i.e moving them from New to Cur no longer a problem)
- Messages are filed in IMAP folders only after successful processing. Messages that were not successfully processed either remain in the Inbox (for processing on the next run) or end up in the Failed folder. You can manually inspect these folders with your IMAP client, and copy messages back into Inbox if you want to try reprocessing. This is resilient against the CiviCRM SOAP server being down for any reason.
- The filing folders are created automatically by the script if they don't already exist.
- The script can run on ANY computer that has the required PERL modules. Therefore, if your web host does not have the right PERL modules (the method below for installing PERL modules in your home directory will fail if the main Perl installation does not match the requirements for the modules you are trying to install), you could potentially run the processing script from your desktop computer.
Using IMAP connections may be slower than the method below which manipulates the Maildir files directly. However you may find in any case that the limiting factor is the rate at which your web server can handle connections (i.e. thousands of near-sumultaneous SOAP calls to handle bounces could drag your web server down, so the less aggressive rate of message handling through IMAP could be a benefit for some set-ups).
Alternative (earlier) implementation using new-mail2soap script:
The earlier new-mail2soap script acts on email files directly where they are stored. It is therefore potentially fast, but requires read and write access to your Maildir folders. To use ne-mail2soap:
- Catch all email with VERP address if you have POSTFIX as per CiviMail Mailer Settings (using virtual regexp) OR set up a catch-all email address of the form *@yourdomain.org and forward to a dedicated email account eg firstname.lastname@example.org.
- Set up a user account on your CMS/CRM eg civimail for SOAP to access your CMS. If your want to archive all bounces and replies etc, create a Bounces folder in the email@example.com account (assuming it's imap) or any other folder where you want to archive them.
- Download the new-mail2soap.pl file attached to this page. Use your favorite editor to open . Set the following variables. (Note the file as attached has the trace option enabled for SOAP::Lite, so will produce verbose output - useful for debugging.)
- soap user variables
- new mail directory
- bounce directory
- Upload the new-mail2soap.pl file somewhere that you have execute rights (eg could be your cgi-bin directory on a shared host)
- Make sure that the new-mail2soap.pl script executes correctly. If you have command line access to your hosted account just run the script. If you use a web hosting provider, SOAP::Lite and Mail::Verp perl modules may not be available by default. See below to install SOAP::Lite and Mail::Verp in your home directory via command line.
- Set up cron to run new_mail2soap.pl periodically.
- You're done.
The new_mail2soap.pl script looks in the new emails folder of your email account, loads any email files it finds there and processes them through the soap interface. This is a first-attempt version and could use some refinement! In particular, the message archiving should probably be moved into the process loop so only successfully processed messages are archived and others stay in the New folder until the next attempt (or forever!). Also, a sanity check on VERP headers to make sure they are legitimate messages might be useful. It does, however, avoid one potential problem of mail2soap.pl, as it only needs to be compiled once per cron run, which could help with large loads. Note in most imap daemons eg Courier, if you access this email account with a mail client (or webmail) your checked mail may be moved from the New to the Cur folder.
N.B. The VERP string in the email to line changed somewhere between early 1.4 and 1.5, with an extra field (domain_id) being added. To account for this, if the $verpinfo matching function in mail2soap-new.pl needs to be updated to
($verpInfo =~ m/(\w+).(\d+).(\d+).(\d+).(0-9A-F+)/i);
This is included in the 1.5 version of new-mail2soap_1.5.pl attached to this page
The following text was excerpted from: http://cookbook.soaplite.com
Install SOAP::Lite in a custom directory
You want to install SOAP::Lite, but don't have root/administrator privileges.
Install SOAP::Lite into a custom directory using CPAN module:
# perl -MCPAN -e shell
> o conf make_arg -I~/lib
> o conf make_install_arg -I~/lib
> o conf makepl_arg LIB=~/lib PREFIX=~ INSTALLMAN1DIR=~/man/man1 INSTALLMAN3DIR=~/man/man3
> install SOAP::Lite
Setup PERL5LIB environment variable. Depending on your shell it may look like: PERL5LIB=/you/home/directory/lib; export PERL5LIB
lib here is the name of directory where all libraries will be installed under your home directory.
Run CPAN module with perl -MCPAN -e shell
and run three commands from CPAN shell
> o conf make_arg -I~/lib
> o conf make_install_arg -I~/lib
> o conf makepl_arg LIB=~/lib PREFIX=~ INSTALLMAN1DIR=~/man/man1 INSTALLMAN3DIR=~/man/man3
LIB will specify directory where all libraries will reside.
PREFIX will specify prefix for all directories (like lib, bin, man, though it doesn't work in all cases for some reason).
INSTALLMAN1DIR and INSTALLMAN3DIR specify directories for manuals (if you don't specify them, install will fail because it'll try to setup it in default directory and you don't have permissions for that).
Then run: > install SOAP::Lite
Now in your scripts you need to specify: use lib '/your/home/directory/lib';
somewhere before 'use SOAP::Lite;'