Skip to end of metadata
Go to start of metadata

Status

This project was completed on 2017-08-22. The rest of this page is retained only for archival purposes

Vision

As of mid 2016, we have a new Developer Guide (aka Dev Guide) written with MkDocs and intended to replace most of the content within the Developer Documentation section of this wiki.

As of early 2017, we are actively migrating pages from the wiki to the Developer Guide.

This migration project will be completed when all of the following are true: 
  • When you look at the wiki's page tree, the "Develop" item will no longer exist
  • All wiki pages currently listed under "Develop" with either be (a) migrated to the Developer Guide (and eventually removed from the wiki), or (b) kept on the wiki long-term with a new spot in the wiki's page tree (not under "Develop"). 
  • URLs that point to existing pages within the wiki, will redirect to the corresponding page in the Developer Guide. This process is already happening, as we migrate pages. Here is an example redirect.

Target project completion date: Fall, 2017

Sean Madsen is leading this project. Find him on Mattermost if you have questions. 

Process for migrating other areas of the wiki

This project only covers the Developer Documentation section of the wiki.

There are many other wiki pages outside the scope of this project. A notably separate project is documentation on administering CiviCRM installations (e.g. installation, upgrades, etc.) and for this content we have plans to create a separate Administrator Guide. (But first we'd like to finish the Dev Guide project.)

Even still, there are other areas of the wiki not encompassed by the topics of "development" or "administration". As we continue to move content out of the wiki the fate of this other, more miscellaneous content is currently unknown.

How to edit existing documentation during the migration

First a foremost: go forth and edit! No need to hold back. We have systems in place to perform this migration in parallel with the ongoing maintenance of existing documentation.

Most wiki edits will be captured during page migration

If you find a wiki page that you would like to improve, then please do so! Your ability simply to view the page on the wiki means that the page has not been migrated – once pages are migrated, redirects are automatically put in place to bump visitors over to the new page in the Developer Guide. So any changes you make to that not-yet-migrated page will almost certainly be captured as part of its migration. 

Wiki edits not captured in migration fall into the "safety net"

There are a few edge cases when edits to the wiki are not captured by normal page migration:

  1. Timing edge case: When the edit is made within the (hopefully short) time window between the point when page migration begins and ends. For example, let's say Marge copies wiki content to markdown on Monday, in preparation for migration. Then on Tuesday Ed edits the wiki page. Then on Wednesday Marge completes her cleanup and migration. Ed's edits will not be reflected in the final page in the Developer Guide. 
  2. Status edge case: When the edit brings an existing page from a state of lacking content worthy of migration to now having some content worth migrating, and when this happens after the page has been evaluated in the spreadsheet (described in the migration process section below). 
  3. New page edge case: When the edit adds a new page

Safety net process

This is a manual process (but quick) and performed regularly by Sean Madsen.

It was last performed on: 2017-08-22T12:02-06:00

  1. Look at the recent updates to the wiki, one by one, since the "last performed date"
  2. Look at each edit using "view change" when necessary. Immediately disregard edits that meet any of the following criteria: 
    1. spam
    2. pages outside of CRMDOC/Develop
  3. Look for edge cases, as described above. 
    1. For a timing edge case, manually port the changes made in the wiki into the Developer Guide by looking at the diff in the wiki.
    2. For a status edge case, update the status in the wiki pages spreadsheet.
    3. For a new page edge case, add the page to the wiki pages spreadsheet.
  4. Update the "last performed date".

When to edit the Developer Guide

If you get redirected to the Developer Guide after viewing a wiki page you seek to edit, then please read about contributing to documentation in the Developer Guide so that you can make your edit there. If you're short on time and don't want to dig into editing content in an unfamiliar system just yet, then you may submit an issue describing changes that should be made, 

If you have brand new content to add (e.g. new page), then please add it to the Developer Guide if possible because that will save other people the time needed to migrate it – but even if you add it to the wiki, it will still get caught by the safety net described above. 

About MkDocs

MkDocs is the platform on which we have chosen to focus our documentation efforts, going forward. It has already been used to create the User Guide, with great success. It's a python-based tool that generates a static website from content written in markdown files. The files are then kept in GitHub for collaborative editing.

Why migrate to MkDocs?

The wiki has served us well. But MkDocs offers the following improvements:

  • Atomic changesets -- update APIs and docs at the same time (when the Developer Guide source code is eventually integrated into the CiviCRM Core source code)
  • Easier inline code blocks
  • Pull-requests / review / antispam
  • Offline access
  • Real branch/merge mechansim (CRMDOCxx is ucky)

  • Bulk changes are easier (e.g. with sed, macros, etc.)
  • Contributing to documentation becomes a pathway for aspiring developers to learn more about Git, GitHub, and their workflows within the Civi community

Differentiation between the wiki and the Developer Guide:

Yes, we are moving a lot of pages off the wiki. But this doesn't mean the wiki will cease to be useful. Here is a rubric to clarify the intended purpose of the wiki, going forward.

  • Developer Guide
    • Official documentation for existing and aspiring developers.
    • Explains development tools and methodologies, generally.
    • Examples are given minimally, and only when necessary to explain larger theories.
    • Topics are covered comprehensively.
    • Can be read linearly. Cumulative skills ordered sequentially for optimal reader comprehension.
    • Content held to high standards.
    • Written with MkDocs
  • Wiki – Reqs, Specs, & Recipes 
    • A place for requirements and specifications to be stored and reviewed.
    • A place for recipes – i.e. specific and detailed examples of accomplishing a task that will be uncommon to most developers
    • Sparse.
    • Little thought given to presenting content in a didactic order.
    • Minimal content standards.
    • Written with Confluence.

Migration Process

Here's the grand vision for the project:

  1. Pick pages on-by-one to move from the wiki to the Dev Guide
    1. In progress
    2. This process is being managed by this spreadsheet of wiki pages (in the same document as the above spreadsheet). Read more details about this spreadsheet by choosing the "README" tab at the bottom.
    3. The process of choosing pages to migrate can be done in parallel by many people.
    4. Where disagreements arise about the fate of a specific wiki page, discussion can happen on Mattermost. A page status even exists "to-discuss".
  2. Migrate pages one-by-one
    1. In progress
    2. For a given page to be migrated, see the steps below "how to migrate one page"
    3. Separate pages can be migrated in parallel by many people. Use the "Assignee" column in the spreadsheet is to prevent someone else from stepping on your toes as you begin the work to migrate a page or set of pages.
    4. The migration process of specific pages can begin before the status of all pages is decided.
  3. Re-organize wiki after all pages are migrated
    1. Not yet started
    2. To better usher-in the "specs & recipes" era of the wiki and allow it to thrive, we'd like to do the following
    3. Delete the wiki pages that have been migrated to the Dev Guide
    4. Re-organize pages that will remain on the wiki.
      1. The process for this is to-be-determined.
  4. --- At this point the project is deemed "Complete" ---
  5. Eventually, move the Developer Guide repository into the CiviCRM core repository so that future changes to CiviCRM core can be committed alongside documentation updates
    1. Not yet started

How to migrate one wiki page to the Dev Guide

  1. Learn how to edit the Dev Guide.
  2. Use the spreadsheet of wiki pages to pick a page to migrate.
  3. Assign the page to yourself in the spreadsheet by writing your name in the "Assignee" column.
  4. Identify a target location in the Dev Guide
    1. Keep in mind that we want cumulative skills to be presented sequentially.
    2. If you need to create a new page in the Dev Guide, use the mkdocs.yml file to add it to the navigation menu.
  5. Copy-paste content into the file(s) in the Dev Guide
    1. Keep your markdown clean by following the standards on the Markdown page within the Dev Guide
    2. Optionally, follow instructions below for automatically converting a wiki page to markdown
  6. Add a redirect from the wiki page to the Dev Guide page (see instructions below)
  7. Make it final
    1. Commit your changes and submit a pull request
    2. Update the status of the wiki page in the spreadsheet to "done-migrated"
      1. (You don't need to wait until your PR is merged to update this status)

How to use webpage2md to convert one wiki page to markdown

  1. Ensure you have the following packages installed (which are all available via apt-get and brew):
    curl pandoc html-xml-utils
  2. Get the wrapper script, webpage2md
    git clone https://github.com/seanmadsen/webpage2md
    cd webpage2md
  3. Then run like this:
    ./webpage2md 'https://wiki.civicrm.org/confluence/display/CRMDOC/The+codebase' > codebase.md
  4. Or:
    1. You can have it automatically pick a filename for you and save the file in your current directory
      ./webpage2md -a 'https://wiki.civicrm.org/confluence/display/CRMDOC/The+codebase'
    2. See all the options
      ./webpage2md --help
    3. If you pull pages from CRM (instead of CRMDOC), you need to set the -c option as follows to change how it picks content from the page
      ./webpage2md -c 'div#main-content' 'https://wiki.civicrm.org/confluence/display/CRM/Book+style+guide' > style.md

How to redirect one wiki page to the Dev Guide

As an example, to redirect:

from:   wiki.civicrm.org/confluence/display/CRMDOC/The+developer+community
to:   docs.civicrm.org/dev/en/master/basics/community

add the following line in civicrm-dev-docs/redirects/wiki-crmdoc.txt

The+developer+community basics/community

Commit the change to this txt file along with your commit that adds the page. Then when your PR is merged, the redirect will be automatically added to the wiki.

How to view an old wiki page after it has been redirected

If this wiki page has been redirected (as described above), 

wiki.civicrm.org/confluence/display/CRMDOC/The+developer+community

then add an extra slash after CRMDOC as follows, and you'll be able to view the page: 

                                           ↓
wiki.civicrm.org/confluence/display/CRMDOC//The+developer+community

 

 

 

Labels
  • None
  1. Jan 04, 2017

    As noted in https://github.com/civicrm/civicrm-dev-docs/pull/20#issuecomment-270247118  I am in favour of changing the User Guide to # for Heading 1 and ## for Heading 2.

    Regarding two or three levels of folders, we have two levels of folders in the User guide because the basic migration was done en masse (by Michael McAndrew) with a script and the Gitbook document being converted only supported two levels of "folders".  I didn't even realise that rtd supported a third level of folders.  I presume going to three levels of folders will let the menu expand an extra level, in which case I am in favour of allowing it as there are some places in the use guide that would benefit from that extra level of menu. 

  2. Jan 05, 2017

    Hey there,

    Great effort starting this migration documentation (smile). Here are a few thoughts from me. Apologies that this is a bit of a brain dump but hopefully some useful stuff below. Am back at work now and happy to help out from here on in...

    What does complete look like?

    When taking about the wiki, it is important to distinguish between the spaces CRM and CRMDOC. Background: The original intention was that CRM could be the place for specs, roadmaps, etc. and CRMDOC for documentation. In reality, a lot of people haven't made the distinction, and we a not insignification amount of documentation in CRM and visa versa.

    It would be good to define what complete looks like for the migration. One definition would be

    1. All documentation in CRMDOC has been moved to mkdocs
    2. All documentation that is mistakenly in CRM has been moved to mkdocs
    3. Non documentation content in CRMDOC is moved to CRM.

    At this point, CRM (i.e. https://wiki.civicrm.org/confluence/display/CRM/CiviCRM+Wiki) will probably be even uglier than it is now, but that is a project for another day.

    Standards and structure

    Over the past few years, we have come up with some pretty decent standards for the user documentation, many of which would be applicable to the developer documentation, though there are certainly some differences.

    One overarching theme that I think we should adhere to is 'keep it simple', both in terms of formatting within pages, and the structure of the book. This helps in a few ways:

    • it means makes it easier for us to have the content consumed in different channels (mobiles, ebooks, print, screenreaders, etc.)
    • it forces us to make an effort to make content easy to understand (in my experience, three levels of chapter and 7 levels of heading means that you haven't thought hard enough about how to organise your content)

    I would argue for two levels: 'Parts' and 'Chapters'. That is the 'industry standard' that you see in computer books, and in science books, etc. So following the for example: Part 1 API, Chapter: Chaining.

    Then each chapter (heading 1) can be broken down into sections (heading 2) and sub-sections (heading 3). In my experience, feeling the urge to use a chapter 4 means that you have overcomplicated the structure.

    Also, the more link levels we use, the more brittle the navigation becomes.  The more levels we have, the more likely it is that we'll break links when we move stuff around and create difficult to maintain redirects.

    I checked out the style guide that Sean wrote and I think it looks spot on. I'd caution against using too many bulleted lists, especially indented ones. They can often be better rewritten as paragraphs. And I agree on the comments re: tables. I think tables are good for small matrices, with 2 or 3 columns max and 1-5 words in each cell. Any bigger than that and they present problems when being consumed on different devices and can often be rewritten more intelligably as paragraphs.

    Agree that # is better than ======. Pandoc (the tool I used to convert from flossmanuals to markdown translated headings that way - there was probably an option I could have chosen to make them # which in hindsight would have been better

    Three books

    The other opinion I have (if you can handle any more!) is that I think we should have three books:

    A developer guide - how develop extensions and core code, etc. (basically what we have in the developer guide)

    A system administrator guide - requirements, installation, upgrades, email configuration, cron configuration, etc. - basically everything that someone needs to know to install and administer stable releases of civicrm and its extensions.

    A user guide - configuration and day to day use of civicrm - everything you can do via the user interface (no technical skills required)

    The reason that I suggest spliting out the system administrator guide from the user guide is because we want to avoid any alienating language in the user guide, and we want to have somewhere for users to be able to point the technical people who are helping them (the alternative is to use a civicrm hosting provider who understands all the system admin)

    The reason that I suggest spliting out the system administrator guide from the developer guide is because (although there is inevitable overlap and there will be cross references) there are two distinct audiences. 1) people that just want to set up an out of the box civicrm on a cms and then do the upgrading, etc. to support their users (they can just use the system admin guide) and 2) people that want to develop civicrm, who want to set up a developer environment and start coding etc.

    Tools

    If you want to do mass conversion of wiki pages to markdown to help with the migration, then pandoc (http://pandoc.org/installing.html) is a pretty powerful tool that I used when doing the book. I appreciate that there'll be a lot of manual manipulation of chapters etc. but every little (bit of automation) helps!