Aller directement à la fin des métadonnées
Aller au début des métadonnées




Work in Progress
This is a planning document for new functionality.

Purpose

CiviMail allows non-profit organizations to send newsletters and other email blasts to their constituents. Unfortunately, for designers who prepare custom layouts, it is hard to predict how a given message will appear across different clients. The project aims to provide a mechanism for rendering a draft email in multiple clients.

Goals

  • Open source: Provide the core system and standard renderers under a free, open license.
  • Rendering: Webmail: Support rendering draft emails using common webmail clients (Gmail, Yahoo Mail, etc).
    • Rationale: Webmail clients are extremely popular. The tools for automating web-based interactions tend to work well across platforms.
  • Rendering: Extensible: Allow developers to contribute additional renderers for alternative clients (Outlook, Mac Mail, etc).
    • Rationale: Given the range of technologies involved (web-based, Windows-based, OSX-based, Android-based, iOS-based, etc), it's unlikely that one open-source contributor can maintain support for a comprehensive range of renderers (over the long term). However, a collective effort (with different contributors bringing different skillsets and managing separate renderers) would be much more viable.
  • Composition UI: CiviMail: Integrate preview functionality with the CiviMail email composition screen.
    • Rationale: CiviCRM is great, and it serves a large community of great non-profit organizations.
  • Composition UI: Extensible: Allow other open-source email composition tools to use the same preview/rendering infrastructure.
    • Rationale: Given the range of technologies involved (web-based, Windows-based, OSX-based, Android-based, iOS-based, etc), it would be difficult for CiviCRM (or any other existing FOSS community) to maintain a comprehensive rendering system on its own (over the long term).
  • Minimize setup difficulty: Setting up a cluster will not be easy, but we should minimize the difficulty by minimizing the number of dependencies and providing clear setup steps.

Terminology

  • Email Composer A frontend application in which a user composes a draft mailing. As part of the composition process, the user may want to preview the draft using a range of email clients. There may be multiple, alternative composition UIs (CiviMail, Drupal Newsletter, phplist, etc).
  • Email Renderer: A backend agent which is able to take a draft mailing (Subject + Body (HTML) + Body (Text)) and prepare a screenshot of how the message would appear in a particular email client. There may be multiple, alternative renderers (for Gmail, Yahoo Mail, Microsoft Outlook, Mac Mail, etc).
  • Email Preview Manager: The middle tier which manages the queue of pending previews.
  • Email Preview Batch and Task: To preview one mailing with 5 email clients, the composition UI would create one batch with 5 tasks.

Typical Data Flows

  • (Composer <=> Manager) Get a list of supported Renderers.
  • (Composer <=> Manager) Submit a PreviewBatch (with tasks for different renderers).
  • (Composer <=> Manager) Lookup past/pending PreviewBatches.
  • (Composer <=> Manager) Check the status of a PreviewBatch or PreviewTask. Fetch any available screenshots.
  • (Renderer <=> Manager) Claim/assign a pending PreviewTask.
  • (Renderer <=> Manager) Finish a pending PreviewTask – either with success or with a failure. (For successes, provide the screenshot.)
  • (Manager) Cleanup ancient or stalled PreviewBatches or PreviewTasks.

Misc: Design of the PreviewManager (Informal narrative)

I found myself going back and forth on (a) the network interface and (b) the implementation platform (language/library/framework) for the PreviewManager.

Network interface:

  • REST CRUD (with Polling): The PreviewManager provides a REST/CRUD API for managing batches and tasks, which is used by the composers and renderers. REST CRUD has broad understanding and tooling in the open-source community. If we're going to seek renderers in diverse environments, REST CRUD will likely be the most portable interface and easiest to explain. However, it doesn't provide job-management out-of-the-box – one must define the schema, calling patterns, etc.
  • Specialized Queue Protocol: The PreviewManager is basically a distributed queue based on one of the existing queue services or protocols (amqp, stomp, zeromq, ad nauseum). These existing queue services come with richer job-management functionality out-of-the-box. However, they generally come with exacting dependencies (which cannot be managed via, say, PHP's composer and which require special setup).
    • Variant: In NodeJS, there are several libraries available in npm which can manage bidirectional, real-time message passing on top of HTTP (eg Socket.IO, WebSocket). If you've already committed to node/npm, then adding support for real-time queuing has less onerous effect on the setup process.

Implementation platform (language/library/framework)

  • PHP: CiviCRM's backend is mostly implemented in PHP – as are many other open-source newsletter/mail-blast tools. PHP is quite suitable for REST CRUD (with polling), but specialized queues are uncommon in PHP and require special setup/administration.
  • JS: The CiviMail UI is based in JS, and the webmail renderer will be based in Javascript/NodeJS, and Civi already depends on NodeJS for core development (in v4.6+). The JS developer community is generally bigger (because webdevs in PHP/Python/Ruby/etc also learn JS). NodeJS can work with REST CRUD, and you can also embed a specialized queuing service in NodeJS apps (using one of its bidirectional, real-time libraries).
  • Everything else: Civi's community doesn't systematically use Ruby, Python, Java, Scheme, Haskell, C, C++, C#, Go, Scala, Erlang, or Rust (although there are individuals who use each).

Resolution:

  • Implement the manager in NodeJS. Try to use a NodeJS framework (eg loopback.io, sails, or just express+mysql). In the foreseeable future, define a network interface based on REST CRUD. However, in the future, we could augment the conventional REST CRUD transport with Socket.IO (or another real-time message library).

Misc: Delivering test emails (informal narrative)

When rendering a test email with a specific email program (Gmail, Thunderbird, etc), there must be some code which takes the test email and loads it in the client. How should that work? Where should it be located – in the "Manager" or in the "Renderer"?

This depends on whether you believe that SMTP is the best/only way to send a test mailing to an email client. If it is the best/only way, then we could put it in the "Manager" (so that it's shared by all renderers). However, if there are other ways to send a test mailing to an email client, then we should make it easy to swap them on a case-by-case basis.

For previews on Gmail or Yahoo Mail, SMTP is fairly convenient. All you need know are the username/password for the webmail service, and you can do everything (connect over SMTP, connect through web-browser, etc).

For previews in Mac Mail, Thunderbird, or Outlook (desktop), these programs don't natively accept messages from SMTP. Messages take a longer path, relayed over SMTP and then POP/IMAP and finally showing up on the desktop. For a sysadmin to setup this path, he needs to setup each part of the chain (SMTP server and IMAP server and desktop client), and that creates several points-of-failure. Fortunately, it's not really necessary replicate the entire chain - we can skip SMTP entirely and deliver the test message over POP/IMAP. This reduces the components/setup-steps/risk.

I'm not saying that we should support all these delivery mechanisms today. For now, we "keep it simple" (KISS) and focus on what works for the webmail-renderer. SMTP should be fine for the time being, but we shouldn't lock ourselves into it by making it part of the Manager.

 

Étiquette
  • Aucun

Creative Commons License
Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution-Share Alike 3.0 United States Licence.