← You're reading a post on Code for America's Blog Archive. Head back to our main site.

Recapping the California Child Welfare Technology Vendor Forums: Part 2

On Friday December 18th, California held its second vendor forum for the Child Welfare Services project. Part 1 of my recap covers the first vendor forum.

The materials for the second vendor forum are now up at the Child Welfare Services – New System website, as well as links to the first two RFPs:

California has also published the in-person attendee list of the first forum [pdf].

The 2nd vendor forum presentation covers, as I understand it, questions that were emailed to the California team at CWS-NSP@osi.ca.gov after the first vendor forum.

I’m not going to recap the whole presentation – this time it clocks in at 54(!) pages. Instead, I’ll pick out from each section of the presentation the bits that I think are good/interesting/worth commenting on.

Doing business with the state

This section included issues like whether companies that aren’t based in California would be able to bid. Good news! You don’t have to be a California-based business to bid, but you do have to register to do business with the state.

I think, if vendors have issues with the registration process, they should be able to contact cws-nsp@osi.ca.gov for help.

About the procurement

Most of the questions about the RFPs are covered now that first two RFPs have been published on Bidsync (Intake RFP, API/Platform RFP), but here’s an interesting slide:

A question about whether the RFPs will ask for working code or prototypes.

Slide 7: Responding to the RFP

Some points to note:

  • No, the state isn’t requiring working code or prototypes in vendors’ bids, but
  • Bidders can demonstrate capability and skill by submitting working code or prototypes

This is in contrast to something like 18F’s Agile Blanket Purchase Agreement process that required vendors to demonstrate agile development capability by, er, actually doing some agile development. I *imagine* that, all things being equal, a vendor who demonstrates capability and skill with working code/prototypes will be more persuasive than a vendor who doesn’t. [Note: while I have been working with California and their Federal partners on this effort, I’m not a State employee and so can’t be and will not be on the evaluation team]

On when the additional RFPs are going to come out:

A question about when additional RFPs will be issued.

Slide 11: Issuing More RFPs

Two more RFPs – extensions for the Platform/API module and an RFP for the Children’s Residential Licensing module are expected to be issued in the first half of 2016.

Open government

This section was super exciting to me. It’s a great example of the direction that government technology is moving and is an area in which California is definitely taking the lead. Other states, take note!

First off:

A question about the process committing to open source and open standards for development and delivery.

Slide 14: Open Source and Open Standards

This is a big deal. I got a bit carried away when I tweeted this on the day (the clarification is that this project in California is committing to open source and open standards – it wasn’t a state-wide technology policy announcement), but the main points are that the Child Welfare Service will use:

  • open standards
  • commodity open source components
  • commodity open source tools

when they’re available.

Also, “all new source code will be made open and reusable and published with an appropriate license.

To make sure the point wasn’t missed, “we intend to release the code created under this effort as an open-source product.”

This was followed up with a slide about intellectual property and ownership:

A slide about ownership of the product created through this effort.

Slide 15: Ownership, licensing and open source

California and the federal government will own the products (including, for example, API and Intake module work and documentation like user interface guidelines, design standards and pattern libraries) and “intends to provide an open source license to the products created or modified as part of this procurement”.

To me, this means that as part of this procurement a lot of useful, valuable information is going to become available for others to build upon and use, which is a fantastic win for taxpayers and the government community.

The next slide, though, was one of my favourites:

A question about how much work will be done in public, e.g. with a public git repository with working demos online.

Slide 16: Working in the open

“The State will default to open” was a remark that I saw someone in a Code for America Slack channel say that they’d like on a t-shirt, and it’s an amazing, progressive sentiment and policy. To me, this indicates that California expects that most – if not all! – of the work on this project will be done on the project. The wording in the slide that:

The State is not opposed to working in public, provided appropriate and reasonable privacy and security requirements have been met, given the sensitive nature of Child Welfare. The State believes that we should share the progress and work of this project whenever we can, with colleagues, users and the world. This extends to sharing code, designs, ideas, intentions and failures. The more eyes there are on a service, the better it gets: mistakes are spotted, better alternatives can be suggested and the bar is raised for everyone. 

might be familiar to people who’ve read through the United Kingdom’s Government Digital Service Design Principles. The next and last slide in this section covered the issue that existing non-disclosure agreements are still in force until they’re changed and that some information will obviously (and sensibly) need to be kept confidential given the nature of the project.

User research

Peter Kelly, who was the voice of the California Team during this vendor forum, opened this section by saying that there was a surprising lack of questions about user research, considering how the team had emphasized how important user research and meeting user needs was in the first vendor forum.

That said, this part was good and worth noting:

A question about ensuring consistent look and feel.

Slide 20: Design and User Interface

I like how California answered this one. The question was relatively narrow:

Q: Is it the goal of the State to ensure a consistent look and feel across the multiple modules?

If I were being uncharitable, I’d view this question as a possible fishing expedition from a vendor who would’ve preferred a monolithic approach. This isn’t quite as bad as a “have you stopped beating your wife yet?” for which the question is a trap and there is no good answer, but the answer in this case is clearly “yes, of course the state wants a consistent look and feel across multiple modules”. You can imagine the argument that the best way to achieve a consistent look and feel would for the entire service to be supplied by one vendor. But! That’s a trap: relying on a single vendor is just one way to ensure a consistent look and feel.

Instead, California zooms out and tries to consider what the intent of the question is. A consistent look and feel is just one way to ensure that users have their needs met: that they can rely on predictable behaviour and interaction methods across different services. So it’s great to see that the state answers the question like this:

A: It is important to remember that the State has an explicit goal that the new Child Welfare Service be simple and intuitive enough that users succeed first time unaided. This means that a consistent look and feel across multiple modules is desirable. [my emphasis]

For example, we expect that vendors will collaborate with the State to produce a design and user interface standard, guidelines, pattern library to be used across modules. [my emphasis]

This is a great answer. It takes inspiration from the direction established by the U.S. Digital Service, the UK government and the Australian government in saying that services need to be simple and intuitive enough that users succeed first time unaided.

Secondly, it looks like California is using the Child Welfare Service project as a way to kickstart a design standard and design guidelines by saying that the State is going to collaborate with vendors in producing a design and user interface standard, guidelines [and] pattern library to be used across modules.

All of this gets to the answer of the actual question, as well as providing wider context: yes, a consistent look and feel across multiple modules is desirable. But, in the manner of a user story, a consistent look and feel across modules is desirable so that users can succeed first-time unaided because the service is being designed to meet the needs of its users.

The next few slides in this section clarified that users of the Intake module will be county and state employees/users as well a reporting function for the public to use. Also, the scope of the Intake module is covered in existing State regulations at Div. 31-100 and 31-110.

Another question and answer set was like this:

Q: Are you targeting an API for system-to-system integration, integration with mobile applications, or both?

Again, the state’s answer was less about the actual question and more about the overall context. It’s my understanding that the API isn’t just being used for system to system integration (and I’m not sure what that means, exactly: perhaps multiple back-end systems?) or with “mobile applications” – because the latter betrays thinking about the role that a modern application programming interface can fulfill. An API can be used by many things – whether it’s badly or well designed. Instead, California answered like this:

A: Both. We are targeting an API for system-to-system integration. The API does not specify in particular the types of systems that the API will be used with on the client side (e.g. “mobile applications”) other than the API be open and RESTful. For example, the API developed for Intake may provide functionality for other back-end services, or for front-end services.

There were some specific questions about how Intake module development would work with API development. The forum made clear that Intake module work (other than user research and discovery phases) would be dependent upon functionality provided by the API.

A question about platforms and dependencies showed how California is thinking how the platform is expected to work in the short-term and in the future:

A question about platform dependencies and restrictions, the database that  the Intake module will work with etc.

Slide 20: Platform and API (cont.)

Most of the answers to these types of questions were along the lines of saying that there will be a “high-level product vision and short-term user needs that must be met”, and that the RFP when released should answer questions about platform dependencies and restrictions. The Intake module will need to work with the legacy system and database, but through the Platform API. California expects “the intake module team to collaborate with the Platform module team to agree and deliver API functionality iteratively, on a sprint basis”.

Questions about the legacy, enterprise mainframe system were mainly answered by referring to information and documentation, like entity relationship diagrams, that are part of the RFP and the bidder’s library.

There was a specific question about using software-as-a-service:

A question about being open to licensing software as a service (SaaS) solutions

Slide 33: Software as a Service solutions

California’s answer about being open to using software as a service was that:

There may be components of the new CWS-NS could be serviced by SaaS solutions. However, any proposed system must be flexible enough for the State to be able to regularly and frequently improve and change its functionality and operation in order to better meet its users’ needs, whether through configuration, customization or new code.

To me, this means that the priority is making sure that any new service is able to continually evolve and improve to meet users’ needs. My interpretation would be that proposed solutions to modules could certainly make use of software-as-a-service or subscription/cloud software, but that the state expects regular and frequent improvements.

That attitude is repeated again in the following slide:

A question about whether California is procuring an already-existing system, services to build a new one, or modifying an existing system to fit needs.

Q: Are you procuring an existing system, procuring services to build a new system, or procuring services to modify an existing system to fit your needs?

A: We are procuring the development of a long-term service that is designed to meet our users’ needs. Any proposed system must be flexible enough for the State to be able to regularly and frequently improve and change its functionality and operation in order to better meet its users’ needs whether through configuration, customization or new code. [my emphasis]

My interpretation: California isn’t specifying a particular approach. In other words, California isn’t saying: that they want an existing, off-the-shelf system; they’re not saying that they’re just procuring design and development services to build a complete new system from scratch; and they’re not saying that they just want new services to modify an existing system (it’s not clear whether the question is about California’s existing system, or any other existing system) to meet needs. Instead, California is again focussing on long-term outcomes: a service that is designed to meet users’ needs and is flexible enough [to] regularly and frequently improve and change its functionality and operation.

Another big point for me was the question about standards and tooling:

A question about whether the first vendors will be establishing standards and tools for subsequent RFPs and vendors to follow.

Slide 36: Standards and Tooling

Q: Will the vendors for the first two modules establish the standards and tools for the remainder of the project?

A: The State will specify a number of standards and standard tools in the RFPs. We expect that the first two vendors will collaborate with the State to establish common standards that will be required to implement the first two modules. We expect that standards and choice of tools will change during the course of development of the new Child Welfare Service. For example, the standard product backlog task tracker may be changed if a better tool is found. Similarly, we expect that user interface standards and design pattern libraries will be constantly updated during the course of development. [my emphasis]

A good question: given how the overall service is being broken up into smaller, more manageable pieces, there’s understandably a question in vendors’ minds about whether an early vendor has the chance to set standards and tools that might (in one view) constrain other vendors in the future. California’s answer is a good one, too: they will specify standards and standard tools in RFPs but they also acknowledge that standards and choice of tools will change during the course of development. In other words, California is going to be iterative in how it treats standards and tools. Excitingly, there’s a hint that California will produce user interface standards and design pattern libraries and make them public.

Here’s an excerpt of some relevant sections from the now-released API RFP, though:

Exhibit A: Statement of Work

Section 5: Scope of Services

Sprint Planning and Execution

5.12.  The Contractor shall use Pivotal Tracker to manage the Product Backlog, User Story acceptance, and maintain a scrum board.

5.13.  The Contractor shall use Slack as the primary mechanism for project-related communication and real-time messaging, archiving and search for all project teams.


Version Control System

5.20. The Contractor shall manage all assets (e.g., source code, automated tests, user stories, configuration files, knowledge transfer material, etc.) using GitHub.

[my emphasis]

Ok, so maybe I buried the lede. This is a big, big deal for how government technology is designed, developed and delivered and another example of California following the lead of the good work done by the U.S. Digital Service, 18F and UK’s GDS.

While this may seem oddly specific, I think the intent here is for the state to set standard at the outset – rather than, for example, having to deal with a number of different product backlog and issue trackers. The above doesn’t look like it precludes the state from changing its mind as better tools or services become available, or if there are better ways of meeting the needs of the state and vendors.

Moving on… (we have around 20-odd slides left to go through).


There were a lot of questions in this section about what would be in or out of scope of the RFPs, like training, implementation and organizational change management.

Here’s one question about pilots:

A question about whether the state would be piloting the first two modules to counties.

Slide 39: Piloting and County Roll-Out

Q: Is the State planning to implement the first two modules to a pilot group of counties, followed by statewide rollout?

To me, this question sounds like it comes from a background of software/technology development and delivery where a solution is developed, it goes through user acceptance testing, then into a pilot phase, and then it’s rolled out to all users. I’d characterise this as a fairly  traditional waterfall software rollout. The problem with this approach is that it really leaves the issue of testing software with users to see if it actually does what they need it to do, or if it solves problems that need solving and really works far too late.

Instead, California answers like this:

A: The user-centered, agile, iterative method means that all services will go through development, alpha and beta phases in close collaboration with our county partners. Only after alpha and beta phases with live county users will services go live and be made available across the state.

We are still working with our partners to determine which counties will be involved in alpha, beta, etc.

In other words, California re-frames the question and says, roughly, all services will be used by users before they’re rolled out statewide. 

Next, a couple questions about organizational change and user research:

Q: How does the state plan for organizational change?

Organizational change is a big deal. Effectively, what the Department for Social Services is doing (along with the Office for Systems Integration) is a big digital government transformation effort. This isn’t so much about new technology as it is about looking at and improving how people do their work so they can do their jobs better. So, this is important.

A: The user-centered, agile, iterative approach means that user research is now an important and continuous process of delivering a new Child Welfare Service that meets its user’s needs. This user research capability is intended to be long- term so that the Child Welfare Service continues to improve over time. [my emphasis]

The State also has an explicit goal that the new Child Welfare Service be simple and intuitive enough that users succeed first time unaided. [my emphasis]

An existing organization change management contractor is assisting us with planning organizational change, and we expect them to work closely with vendors, module teams and researchers.

In this answer, we’ve got a few important points: that there’s a new emphasis on user research as a part of organizational change to make sure that the needs of state and county workers are understood; that this user-research capability is intended to be a long-term, internal capability, and again that regardless of organizational change management, the delivered services are expected to be simple and intuitive. Finally, this is a team effort: an existing organizational change management contractor will work closely with the project team.

A follow-up question also gives some more background:

Q: Will the Design and Development vendor for each module be responsible for related organization change management activities?

A: No, but module Design and Development vendors are part of each module service team. Each team has responsibility for redesigning and improving how their service meets its users’ needs and to improve outcomes.

To me, this means that each team is being empowered with the responsibility and ability to drive organizational change as it needs, against goals set with the project team.

“Other Stuff”

This was the grab-bag of other questions that didn’t fit into the above categories and also includes questions that were emailed during the vendor forum. Here’s my bullet point pick of short interesting ones:

  • The project team will have a collaborative space for the entire service delivery team – including vendors – in Sacramento so that the entire team will be co-located.
  • Vendor teams for the first two modules must include a scrum master.
  • Phasing out the existing system will follow the “strangler pattern“.

And here’s two that I picked out for more commentary. First, maintenance and operations:

A question about who'll be responsible for maintenance and operations for the first two modules

Slide 46: Maintenance and operations; DevOps

Q: Who will be responsible for maintenance and operations (M&O) of the first two modules of the software? How long does the State envision the M&O contract to be for the first two modules and for each subsequent module?

Like the above question about pilot phases, this question and use of the “maintenance and operations” phase comes, I think, from a traditional view of software design, development and delivery in government. What normally happens is something like this:

  • A design, development and initiation phase which is made up of a short initiation phase, then cycles of solution development, a period of end-to-end testing, a period of user acceptance testing and data conversion and getting ready for a pilot, then a pilot, then statewide rollout, then system acceptance
  • After that, a period of maintenance and operation which is where bugfixes and changes in functionality come in (in this case, for example, changes in functionality due to regulatory changes)

Instead, California says that it will be using a  “DevOps” model:

A: The vendors providing the first two modules will be responsible for the DevOps of those modules for the period of the contract.

It is important to note that the State does not anticipate a “traditional” government technology maintenance and operations phase. Instead, we expect to continuously, regularly and frequently release more reliable and more useful software and services, even post 1.0 release. Vendors will be expected to help transition the State to a DevOps culture. [my emphasis]

In this answer, California is signaling that development will continue with operations when services reach 1.0. Instead of behaving like they’re purchasing a product that will be finished and tweaked, California is acknowledging and signaling that the Child Welfare Service is just that – a set of digital services that are critical to providing child welfare in California and that will continue to improve and change as needed.

Second, two questions about managing integration and coordinating development work:

Q: Does the State plan to act as the overarching integrator, responsible for oversight and management across all contracts awarded through the multiple RFP process? Or does the State plan to hire one vendor to oversee the work being performed as the result of the various RFP’s released and awarded through this new approach.

This question is pretty much asking if there will be a system integrator role for a vendor. An alternate and more traditional approach for this kind of government technology (a big system made up of many services that does many jobs) is to hire a system integrator: one vendor who will look after everything for you and whose job it is to make sure that everything works. A disadvantage with that approach is that it means that the customer loses significant control, understanding and direction of the system that’s being delivered. If you believe that knowledge of and control of technology that’s critical to your business is something that shouldn’t be outsourced, then it doesn’t make sense to go with an outside vendor to such an extent. So this question is asking: who’s going to do that integration work?

For this Child Welfare Service project, California answers like this


A: The State plans to act as the systems integrator, in part by closely collaborating with the Platform module vendor, and by managing collaboration between module vendors. Ultimately the goal is that the State be able to understand, direct and control the evolution of both the new Child Welfare Service Platform and the user-facing services. We expect that we will need significant vendor assistance to accomplish this effort; however, that support will be as an extension of our State team. [my emphasis]

which is a clear signal to the vendor community that California intends to step up and into the role of system integrator – to understand and direct how everything fits together, but that stepping into that role will be a process over time and that the state will need help.


Q: How does the state envision assigning responsibility for coordination between different suppliers and end-to-end performance of the different system components as they are delivered?

OK, so given the state is going to have the overarching plan, how is it going to make sure the different suppliers co-operate? Here’s California’s answer:


A: The State expects that each vendor will provide highly automated testing and work to performant service expectations. The State will also work with each module vendor and the Platform/API vendor(s) to agree performance requirements. We expect the Platform vendor to be responsible for meeting the Platform and API performance requirements that are agreed for each user-facing module. The State will work with the Platform vendor and other vendors as necessary to measure the performance and health of the overall system. 

So for starters, there’s a clear expectation that vendors perform test-driven development and that those tests are run automatically and frequently as development proceeds. It also looks like there’s more responsibility and direction going to the Platform/API vendor, as that part of the system is the part that all other services will be relying upon.

OK, that’s it! A very long recap, of very many questions!

I’m going to take a break for the holidays, and when I come back, I’ll take a look through the RFPs and do a writeup of interesting parts in both the Intake and API modules.