Minutes of Accounting Integration Steering Committee meetings

September 15, 2011 8:30pm EDT: Andrew, Dave G, Dave S, Joe (regrets from Paul)

Agreed that we will divide effort to produce documentation for implementation along the lines of tables. CiviAccounts Data Schema - Project Plan shows who is the lead in three initial areas. Dave G will review documents produced in each area and work with each lead on issues. These documents do not need to be as detailed as http://issues.civicrm.org/jira/browse/CRM-8425. Joe will give WebAccess notice that we expect to be ready to work with them again next week sometime.

We intend to validate the core contribution tables for a sample simple and complex transaction at CiviAccounts Data Schema - Sample Before and After Transactions. That page reflects the current database changes associated with each browser level click, and a spec concentrating on records and keys for what will happen after the schema is changed.

We will also validate the data model by inspection (or more) for the new use cases that need to be supported as documented in the spec pages under Accounting integration improvements.

Andrew, Dave and Joe will write blog posts and provide a day for the steering committee to comment before publishing them. These posts will be high level, context setting posts inviting community feedback.

Andrew, Dave and Joe will work with WebAccess on each area of implementation.

Andrew will provide Joe by Sunday with a WorkBench file with new schema, diagram(s), and a diff file of 4.0 versus proposed schema.

Next mtg Sep 21, 2011 8:30pm.

week of August 8, 2011

Several work sessions and meetings. Consensus at end was to try to reduce scope and risk by reducing changes to core Contribute table, and to take some of the new ideas from Andrew's approach. Extensive documentation created under Accounting integration improvements page.

August 3, 2011 9am EDT: Andrew, Dave G, Dave S, Paul, Joe

Reviewed Orders section of http://wiki.civicrm.org/confluence/display/CRM/CiviAccounts+-+Data+Flow+Examples

Discussion around financial_account, financial_type, and associated tables/notions

Discussion around recording of qty, unit price, total price. Consensus to return to existing model.

Decisions:

• Joe, Dave G and Andrew with occasional assistance from Dave S to spend a couple of days working through tech spec for new approach on Mon Aug 8, Tue Aug 9 N. Am. time. Andrew to do invites, with mtg to have overlapping segments with 2, 3 and possibly 4 people if Dave S can make some periods.
• As preprepation for meeting:
  • Joe to:
    • circulate expectations of funders for functionality that initiative is to support
    • develop one or more data flows for existing schema for price sets, recurring payments, pledges, payment to pledge
  • Dave G to:
    • review data flows that Joe gets done
  • Andrew to:
    • circulate a new version of workbench file that shows / highlights only relevant tables, eliminates scope creep as much as possible, and perhaps considers name changes for some tables.
    • develop data flows for proposed schema recurring payments, price sets, pledges, payment to pledge
• Agenda of design sprint meeting will be to cut back scope to what is necessary for 4.1, and to produce design documentation that can be provided to Web Access for development.

July 27, 2011 9am EDT: no minutes

July 20, 2011 9am EDT: Andrew, Dave G, Dave S, Joe, Paul Not present: Claudian, Pradeep

Status:

Paul & Dave S have done but not posted a bunch of use case work for batches and payments

Andrew has begun use cases for adjustments

Lin has done work on detailed accounts use cases

General discussion:

• a $100 donation with a $2.50 payment processor charge would result in two financial items, $100 associated with revenue account, $2.50 associated with banking expenses
• civicrm_entity_financial_account allows accounts for deductible, non_deductible, and other sorts of accounts (eg taxes) to be specified for items in place of fields in the tables of those items
• as a general pattern, it would be good to have revisions for every financial_account, and to have all links from items go to financial_account_revision (of some sort, perhaps a view of the union of the common fields in all revision tables), and have no links directly from items to financial_account

Process: complete and review use cases, then data flow diagrams, then review workplan, funding, and scope.

Decisions and Actions:

Use cases will be user oriented, not implmentation oriented. Next step will be developing data flow diagrams that detail how the use cases are reflected in the database at each step. Andrew to take lead in posting data flow diagrams for all use cases. Dave G will take lead in reviewing data flow diagrams.

BOT to post complete use cases for batches and payments on wiki by Friday, July 22.

Joe to review batches and payments use cases by Wednesday, July 27.

Joe to work with Lin and Andy @ CivicActions to complete use cases for detailed accounts by Friday, July 22.

Joe to complete use cases for receipts by Friday, July 22.

Andrew to post complete use cases for adjustments by Sunday, July 24.

Andrew to post data flow diagrams for adjustment use cases by Tuesday, July 26 to allow people to examine before next mtg.

Joe to check if his clients put non-deductible amounts into different accounts or not: idea is that difference may only be relevant for receipting purposes, not for other accounting, and would thus allow a simplification to be made.

July 13, 2011 9am EDT: Andrew, Claudian, Dave S, Joe, Paul, Pradeep. Not present: Dave G

Status: Currently 3/4 done item 1 financial_account table (CRM-8425). Spent 8 days of WA budget so far. Joe has posted initial spec for item 3 upgrade/migration.

Issue outstanding: API Team has not responded to question of whether we need to still support API v2?

Joe has held off on drafting specs on item 2 contribution table, etc awaiting outcome of schema discussion. Also held off on detailed conversations with API team for same reason.

Specs for use cases have not progressed. BOT will do use case / workflow specs for batching and payments by next week. Andrew will do specs for changes, refunds etc. by next Wed. Agreed last week that for batching phase 1, only export of financial transactions will be included. For payments, use cases will include functionality to allow a payment for part of an obligation and for multiple obligations; no refunds or changes to orders will be covered. Joe will take responsibility for Detailed Accounting codes, receipts. Joe and Paul to coordinate with Cathy on her work.

Group discussed generally whether to move towards using Community Builders v 0.10 design as the basis for further work and agreed it was sound in principal. Impacts: Delivery date will be extended. Need to try to avoid rework by WA implies stop working until new schema design is validated by detailed use cases. [NB: Joe and Andrew agreed after meeting to try to maximize usability of existing work on financial_account table.]

Short discussion of tables for handling accounts led to agreement that it would be nice to have revision / version history for Chart of Accounts. Agreed that we did not want CiviCRM to have to handle General Journal entries for changes resulting from wanting existing transactions posted to one account (eg Events Income) to be posted to another (eg AGM Event Income). Agreed that there were various problems and oversights with fields in the three tables under review, eg handling of account for payment processor, header accounts.

BackOffice will validate the schema against the use cases they have in their practice, as well as those they are specifying as part of Spec development itemized above. Community Builders will review complete design now they have agreed interest of Steering Committee. They will also review against 0.92 design before circulating to JMA. JMA will review against ideas that led to 0.92 design. WebAccess to review for implementation scope issues for coding, and will consult with core team on this.

WebAccess will allow a pause in the development. [Joe will work with WA to tie off current efforts.]

Community Builders agreed after meeting to discuss with JMA taking on more responsibility for spec development.

July 6, 2011 9am EDT: Andrew, Claudian, DaveG, DaveS, Pradeep. Regrets: Joe

Paul took minutes

General

• Pradeep had a few questions and they resolved.  The work will begin shortly.  JMA and WebAccess have started daily meetings -- notes from the meeting will be placed on gthe Wiki and project details on JIRA.
• Andrew:  concern in growth in tables with similar tables.  The changes are not clear. Andrew is not comfortable with the schema, will send an email to the team Wednesday July 6th with his concerns.

Review of questions:

• I think we should change the UI so that Contribution Type is replaced by Financial Account everywhere. Comments?
  • Do not replace Contribution Type in the UI, replace text field with a drop down for Financial accounts. We agree that there should be a seperate  Financial accounts table with 1 Financial Account linked to N contribution type.  Must remember that users will have accounts in the contribution type table and those will need to dealt with in the upgrade through a migration script.
• How should we maintain backwards compatibility yet keep our system as cruft-free as possible in terms of theming?
  • No change required based on the above answer.
    *Do we need to worry about code in .tools/drupal/modules/civicrm_giftaid_alpha or .tools/drupal/modules/civicrm_giftaid?
  • Who developed? Core team.  a UK based change.  Dave G to raise with the team.  There may be other modules that have a similar situation.
  • Who will take the lead on design of certain areas:
  • BOT will take the lead on the batch.  
  • Andrew: will take the lead on revisions (credits/adjustments/reversals/refunds) 
  • Other things on the list with a lead: BOT and Andrew will review this items on our call...
• Do we need to worry about code in .tools/drupal/modules/civicrm_giftaid_alpha or .tools/drupal/modules/civicrm_giftaid?
  • Not sure.....
• Are we ok with Default Financial Account Types will exclude: Equity, Other Revenue and Expenses?
  • Yes, but BOT and Andrew will review.
• There are no contributions on the QB integration MIH besides those for Data Schema work. Ideas on getting that piece funded? Suggestion: discuss with Eileen, and include in new MIH's coming out next week for CiviAccounts.
  • We believe that the generic export will be important to develop first.  Hopefully the export to QBs and other accounting packages can use this generic code or it can be easily extended. 
• Do we need make it happen projects for other enhancements like partial payments?  
  • Yes, but we should wait till they are better defined.
    Other:
• Andrew and BOT will review the table structure to ensure that "double entry" accounting is being correctly considered. Will do as soon as possible. We anticipate some changes. Get the key points to Joe as soon as possible. (Meeting scheduled for Thursday at 8:00 am)

June 29, 2011 9AM EDT: Claudian, Dave G, Dave S, Joe, Paul (regrets from Andrew)

Action items:

1. Claudian to provide an indication of a) minimum input on specs to get started, and b) what is desired
   1. aim is to have enough specs of a good enough level of detail to get a programmer working by early to mid next week
   2. once things are running smoothly with one developer, a second will be brought on
2. Dave S to work with Joe on Project Plan for data schema contract
   1. start with Contribution flow
   2. get dummy data working
   3. gradually build out to other functionality in following iterations
   4. we want to get this complete in time for 4.1 code freeze late August
   5. later user-facing initiatives can be part of minor releases if they don't make it into 4.1 eg 4.1.1
3. Joe to revise GenCode page to include more focus on migration of data
4. Joe to add a page for documentation to be produced
5. Dave S to produce a schema diff and post as attachment to Accounting integration improvements
   1. this will be used to fill in CiviAccounts Data Schema - files 
6. Andrew Perry has offered to validate data model - let's integrate this with 7.b.i as a key piece
7. Work on detailed use case pages will assist in validating the data model as well as estimating and raising funds for later pieces
   1. Dave G to provide URL to reasonable samples of use case documentation to use as prototypes
   2. by July 1, Joe to allocate work on these pages to people who volunteer perhaps along the following lines, assisting as necessary:
      1. Andrew: CiviAccounts Specifications-Changes, Reversals, Refunds
      2. Dave S: CiviAccounts Specifications-Batching of Contributions
      3. Cathy (SwitchBackCMS): CiviAccounts Specifications-Partial Payments - backoffice staff only
      4. Lin/Joe (SACNAS): CiviAccounts Specifications-Detailed Account Codes
      5. Joe: CiviAccounts Specifications-Official Receipting
      6. Eileen/Paul: CiviAccounts Specifications-Flexible User Payments
      7. other initiatives?
   3. by July 8, start re-estimating the work to complete these pieces/phases/initiatives
   4. by July 12, put up MIH pages for each initiative
8.  Joe and Claudian to begin meeting one-on-one this week, moving to daily developer scrum next week
9. Steering Committee will meet Wednesday 9am Eastern.

Status as of June 28, 2011

Support agreement has been agreed:

On June 23rd, JMA and WebAccess agreed to a contract in which WebAccess will provide development services to JMA for this initiative, and JMA will provide engineering design and supervision.

To assist in estimating the work for future phases of the Accounting Integration initiative, Joe has set up pages including functional descriptions of the enhancements. Detailed use case descriptions will be worked out for each area, an estimate will be created, and an MIH will be put up for each one.

Other items of note:

Joe will identify all API and BAO calls that will be affected, and how the parts of GenCode.php that produce the upgrade scripts will need to be enhanced. The intent is to keep changes for this data schema piece invisible to higher level presentation code by making these calls backward compatible, and if we're successful that will also help keep the scope of work more constrained. I'll also search for affected table names and changed field names in the codebase to find other places higher up the stack that will also be affected (e.g. Reports, Searches). For each identified piece of code, I'll then propose how it needs to be changed. We'll also spec out up front how test suites should be developed for this.

For project management, we'll convert these specs into JIRA issues. Paul and Joe will develop a draft project plan using a scrum sort of approach with work packages that focus on completing core pieces of work first, leaving peripheral pieces aside for possible future projects if our funding / hours start running low. We should plan on having a third of WebAccess hours reserved for support.

Potential major new funders who may assist with developing the specifications for later phases include SwitchBackCMS and SACNAS.

April 13 Mtg 4pm EDT Present: Andrew, Dave G, Dave S, Joe, Lobo, Paul

Agenda: Dave G's Apr 6 email on CiviAccount implementation

Agreed:

0. Responsibilities of participating organizations will be clarified and budget allocated by percentage early on. Responsibility for dealing with support bugs will be divided up the same as the original work into, say, project mgmt, design, coding, testing. Core team responsibilities drafted below. WebAccess will be used for development, but not using core team members employed by WebAccess. Andrew sees Community Builder's interests as ensuring the project happens and participating in meetings to provide constructive feedback and to understand rationales for design decisions that will affect their accounting code. JMA and BOT to discuss their roles and relationships offline after Paul back from vacation. Joe to draft and circulate proposal on project structure after that.

1. Development should occur on a separate branch in the repository. All development will be done for Drupal 7 / Joomla 1.6 / CiviCRM 4.1. This includes
* Modification and creation of new xml files for the schema
* Migrating all the features of the current code base to handle the new schema including import, export, search, ..
* Migrating the tests if needed
* Writing the upgrade php / sql from 4.0.x -> 4.1.x

2. Unit tests and web tests covering new functionality should be implemented as part of the development cycle. We will put some metrics on this at a later stage (number of tests and code coverage).

3. Prior to the account branch being merged with the current core development branch, the CiviAccounts development team needs to verify a successful run of your new tests as well as the entire existing unit test and web test suites on a local merged codebase (with the main 4.1 branch). Weekly merges of trunk into branch will keep the work on this down, and responsibility for fixing breakages will be shared as appropriate with trunk submitters.

4. The implementation team commits to support and bug fix that portion of the code for 12 months. Responsibilities for this divided as per agreement in 0 above.

5. Given the size and impact of these changes, the development and merge should be completed relatively early in the release cycle. Given a 4 month cycle for 4.1 core team would like to see a target code completion date of June 30 for this work. Team agrees a completion date will be developed following detailed spec and planning, crucially including input from implementors at WebAccess.

We anticipate that the core team will be involved in the project in the following ways:
* Ongoing design advice and participation in project meetings as needed. Ideally we should have weekly or bi-monthly status check meetings.
* Occasional technical advice / assistance.
* Development of an additional set of unit and web tests to ensure compatibility and catch regression bugs. This set of tests will focus on current transactional and reporting functionality to ensure that things work in a similar manner as today.
* Managing the final merge, test and upgrade process.

Core team suggests that 20-25% of the overall project budget should be allocated to them to cover the above activities.

Other points:

6. The scope is becoming clearer now that the schema design is settling down. Estimates will be able to be made once detailed use cases are developed. There is a sense that the budget allocated may be low.

7. Selenium tests can be used to specify technical requirements in terms of indicating what database changes should result from particular user actions. These tests can be developed against the 4.0 tarball so they test that current db changes occur ('before' tests). The desired project outcomes can be indicated by proposed db changes to occur for 4.1 for same user actions ('after' tests). Business and accounting use case analysis will need to validate the proposed 'after' db results.

Next meeting: Andrew, Dave G, Joe to meet on Skype April 20 4pm EDT. Rescheduled to April 27 4pm.

Joe and Paul to meet on GoToMeeting April 25 4pm EDT.

Agenda: Sort out roles, responsibility and budget.

Joe will be technical lead on project, Paul will be business lead, and will assist with optimizing interactions with WebAccess for maximum success.

Paul will be seeking funding for future pieces of work in the CiviAccounts initiative. For pieces where BackOffice Thinking has contributing clients, the expectation is that BackOffice will be able to devote more resources to developing specs, and will bill time against the project. For this initial data schema project, BackOffice Thinking will not be billing against its budget, though JMA will.

Various functional enhancements to CiviCRM's financial subsystems are listed in the CiviAccounts Project Estimates|confluence/display/CRM/CiviAccounts+Project+Estimates||||||\. BOT and JMA have a joint interest in 1, 3, 4, 5, 6, and at some point 7.

April 6 Mtg 10am EDT Present: Andrew, Dave G, Dave S, Joe, Paul

Agenda: a) review of 0.6 data schema and 0.5/0.6 data flow models, b) review of project finances

Agreed:

1. Focus on 0.6 not 0.5-2.
2. Various suggestions for refining schema. 
3. Joe to send revised to Dave for review and circulation
4. BOT (BackOffice Thinking) and CB (CommunityBuilders) to develop use cases.
5. Next Mtg April 13, 2011 4pm - 5pm EDT. Agenda: Finances, schema, use cases.

Minutes of Project Mtg Mar 10 2011, 10 - 11 am EDT

Present: Dave Greenberg, Kurund Jalmi, Paul Keogan, Joe Murray, Dave Schafer

Agenda: Review the current 0.5 data schema model regarding phasing and ability to meet use case needs

We agreed:

1. The functionality to be supported by the data schema changes is:
   1. Detailed account codes
   2. Changes and reversals
   3. Refunds
   4. Batches
   5. Partial and multiple payments for obligations
   6. Receipts by payment, obligation or period; and for amounts that are deductible, non-deductible or both
   7. Not explicitly discussed during meeting: Intra-organization distribution of revenues
   8. Out of scope: non-accounts receivable transactions such as payroll, capital depreciation
2. We will use the standard accounting design pattern of 'invoice', with the caveat that public contribution pages for example should not expose this terminology.
3. The entity_financial_trxn table will link to the invoice level record. It will also allow links to invoice line item and invoice line sub item records so that support for tracking which items are paid for can be added pending funding. We will develop advice to community on how support various use cases regarding ticket purchases/refunds. (This point revised Apr 29 2011).
4. The existing contribution table is probably closer to an invoice table than to an invoice detail. 0.5 changes its semantics to invoice detail (or sub-detail even for price set items for each participant on a multi-participant ticket purchase invoice).
   1. Joe will develop an alternative data schema model (0.6) for next week that gives the current contribution table the role of invoice header, which will spill over into changes to other tables that will take on responsibility for providing invoice details (and sub-details). The new model will only display proposed fields (a 0.5a model removing commentary on fields will also be put together).
   2. Joe will develop data flow diagrams to model common workflows for 0.5 and 0.6 data schemas. For example, purchase of 3 tickets with price sets, followed by payment, followed by cancellation of 1 of the tickets, followed by refund. The hope is that 0.6 would be less expensive to implement.
5. There is concern that the scope may be larger than the budget. It is not clear that the data schema work can be easily phased according to the colours in the existing 0.5 data model.
6. BackOffice Thinking will take role of Product Manager and Project Manager. JMA Consulting will be responsible for development. JMA's client, the RNAO, and perhaps others will share domain knowledge with BackOffice. The intent is to ensure the changes enable CiviCRM to support larger organizations' needs. Core team will provide support and feedback.
7. Same team will meet next week at same time. Agenda will be a) review of 0.6 data schema and 0.5/0.6 data flow models, b) review of project finances.