Introducing PRODigy Phase II
The Consumer Product Safety Program (CPSP, the Program) is excited to be collaborating once again with Experimentation Works (EW2) to run PRODigy Phase II. In addition to partnering with EW2, we are also working in collaboration with Health Canada’s Solutions Fund. The Solutions Fund is a departmental initiative that gives employees the opportunity to lead experimentation projects designed to make the Department more efficient and better serve Canadians.
Let’s take a step back and revisit our last experiment: PRODigy Phase I.
About PRODigy Phase I
CPSP is the federal regulatory program that protects the public from unsafe consumer products and cosmetics in Canada. The Program monitors trends, identifies hazards, measures severity, and mitigates risks associated with those items. To accomplish this, the Program relies heavily on reports that are submitted by both consumers and industry members which contain details about product-related incidents and injuries. PRODigy is a multi-phased project focused on improving consumer incident reporting to the Program and designed to enhance our surveillance tools.
The first phase of the project was conducted in 2018 and consisted of a randomized controlled experiment using an A/B testing approach. The experiment demonstrated that we can better direct the public to our incident reporting portal by improving our understanding of consumer behaviour and modifying the information on our website accordingly. More details on the project description, design and results can be found in our previous blog post series (first, second and third post).
Building on this initial success, we are taking PRODigy to the next phase of development.
The need for a second experiment
In PRODigy Phase I, we were able to direct more users to the incident reporting form, but the form completion rate remained unchanged. PRODigy Phase II is aimed at improving this rate.
The Program regularly receives about 200 reports per month on unsafe consumer products and cosmetics in Canada. The majority of these reports are submitted to us using our online incident reporting form. In PRODigy Phase II, our objective is to implement user-experience (UX) and user-interface (UI) changes to our existing consumer reporting form. We hypothesize that these form redesign changes will lead to an increase in form completion.
We are working with colleagues within Health Canada at the Business Information and Data Service Division (BIDS-CPSP), Communications and Public Affairs Branch (CPAB) and Information Management Services Directorate (IMSD) to develop and implement the experimental form. Our initial plan was to conduct a single experiment where we would compare the existing form with the modified one (which contained enhanced UX and UI design features). However, after discussing this approach with our colleagues, we were advised to make changes to our experiment design due to the wide variety of changes that would be required to the form (and the time needed to make those modifications). Therefore, we will conduct a two-part A/B experiment to compare completion rates between the control group (Group A) where the form has not been modified, with two distinct experimental groups (Group B and Group C) where the form has been redesigned to different degrees to enhance user experience with UX approaches.
The figures below show the two experimental phases.
The intervention
At the start of the project, we conducted a literature review on different UX and UI techniques shown to be effective in improving consumer experience. We compiled a list of potential changes that we could apply to the intervention forms and we’re now working with our colleagues at Health Canada to identify the best approach in applying these changes over two experiments.
Below is a partial list of the proposed UX and UI changes:
- Re-order of form fields: at the start of our existing form, consumers are asked to provide personal information, including their full name, contact information, and address. We learned from our literature review that it is more effective to ask this information at the end, therefore the intervention form will request information on the incident/injury and product information first.
- Use of plain language: certain titles and questions in our existing form use jargon and are difficult to understand. We will be reviewing our form to include plain language writing throughout with the goal of improving the overall consumer experience.
- Application of conditional logic: the existing form is long and contains a great deal of questions (many of which with low response rate). We will be using conditional logic to tailor the form to the consumer’s specific needs and avoid asking questions that may not be applicable to their needs. For example, our form would only ask for details about the location of the retailer (like address and contact information) if the consumer knows and recalls where the item was bought.
How to plan and conduct an effective stakeholder consultation: the PRODigy experience
In addition to the literature review, we also held multiple stakeholder consultations prior to finalizing the experiment. Adding a stakeholder engagement component to our project was important to increase stakeholder support, understand the various perspectives, reduce risks associated with the proposed intervention, and improve the project outcomes. In total, we held six consultations with over 100 participants.
Our stakeholders in this project were wide and varied, since all Canadians are users of consumer products (and may one day use our incident reporting form). Due to that scope, we focused in on organizations/groups that currently use our reporting form and who could also represent large portions of the Canadian consumer population.
Planning for these consultations was not an easy process and we definitely learned a lot. Here is a step-by-step guide to help you hit the ground running when you decide to conduct your own:
1. Plan Early
Time is an important component when planning consultations. We recommend to start planning these sessions early on to scope key issues and to have the opportunity to implement the feedback received into the project plan.
2. Outline your goal(s)
An informed and meaningful consultation often comes with a clear purpose statement. Establishing overarching goal(s) at the early stages of planning is key to holding a beneficial and worthwhile consultation.
3. Define your audience
Stakeholder engagement is more effective when you gather the right people together. Take a step back and think of who would benefit from learning about your project and be able to provide insight and feedback. As we said above, we had a wide range of stakeholders including: our external experts, colleagues responsible for web posting and web development, partners who provided funds and financial support, analysts that currently use the form involved in the experiment, our counterparts in the US, and other external partners with an interest in consumer product safety.
4. Prepare and share documents
Just like steps 1 and 2 above, early engagement with our partners helped in setting the right time and place for those discussions. Share the meeting agenda, presentation slides, and any background material prior to your consultation so attendees have a chance to review the project proposal and come prepared with questions. Remember that you don’t have to meet everyone all at once. Think about the best size for your meetings and plan multiple consultations if you have lots of stakeholders.
5. Plan a Q&A session
It is important to allocate sufficient time for questions and discussions throughout your consultation. It is also a good idea to gather feedback through a survey following the consultation. We used different online tools (e.g. Slido) to help collect and manage stakeholder feedback during sessions.
6. Report back
Following your consultation, report back to stakeholders in a timely manner and be transparent on consultation outcomes and next steps. For instance, we compiled a list of questions asked during all of our consultations and survey responses, grouped them by common themes, and created a ‘Phase II PRODigy Q&A’ document. We shared this document with all the participants and we will provide an updated project plan that includes our final intervention/design.
Next blog post…
Following the consultations, we have been working with multiple partners to develop the intervention form and launch the first experiment in the fall. We are excited to share more in the next blog post including: our intervention, the launch of the experiment and any successes and lessons learned.
* * *
Post by Health Canada’s EW2 Team
Article également disponible en français ici: Présentation de la phase II de PRODigy
