Results and the Path forward for Health Canada’s PRODigy experiment
Since our last blog post we have implemented the experiment and have been collecting data on users visiting our incident reporting landing page. To recap: this experiment entailed a live, randomized A/B test using the existing consumer incident reporting landing page as well as a modified ‘intervention’ landing page (Figure 1). The intent was to determine if changes in language and presentation could successfully drive online users to continue through to the consumer incident reporting form. That experiment ran between December 12, 2018 and March 11, 2019.
Figure 1: Experimental design flow diagram.
What we found?
During the experimental period, there were 1,999 visits to the intervention landing page and 2,592 to the existing one. After reaching the landing page, visitors would click on a link to enter the consumer incident reporting form. A greater number of visitors from the intervention page entered the consumer incident reporting form (61%), compared to the existing page (27%). This difference in proportions (34%) is statistically significant (p<0. 01). This is an indication that optimizing the landing page content with consumer-focused changes played an important role in retaining consumers within the incident reporting pathway. Figure 2 below shows the daily visits to the incident reporting form from the two different landing pages.
Figure 2: Daily visits to the incident reporting forms between December 12, 2018 and March 11, 2019.
We used Google Analytics TM to compute these and other web metrics such as the geographical location of the user as well as other details that provide insight into the user’s experiences. Results from the experiment showed that most of the visits to the incident reporting landing pages occurred during the weekdays — although this finding could be confounded by the fact that industry members use those same landing pages. In addition, most of the visitors were from Ontario and Quebec — likely due to large proportion of the population residing in those Provinces. However, we also know that the form was accessed by Canadians from coast to coast-to-coast, from British Columbia, Northwest Territories, to Newfoundland and Labrador.
What did we learn about experimenting in our department?
The Internet has revolutionized the way our Federal Government does business. In the past, the Internet was seen as a tool for disseminating information. Today, it goes beyond knowledge translation and serves as an essential conduit for collecting product intelligence needed to support the mandate of the Program. However, competing for web presence is becoming increasingly challenging given the tremendous amount of content on the Internet. As such, this experiment and all the work that went into it compelled our Program to engage in a series of self-assessment activities. These led to the development of a number of lessons about experimenting including:
Lesson 1: Listen to your users
The data collected through the incident reporting system have many applications and are used by staff to conduct risk identification, assessment, and management in support of the Canada Consumer Product Safety Act. In this context, enlisting the help of these users (i.e., internal stakeholders within the Program) was essential to bridging the gap between subject matter experts, programmers, and communications officers.
Lesson 2: Adapting to user behaviour
From our results, the proportion of those who progressed from the intervention landing page to the consumer incident reporting form was 34 percent higher than those from the existing landing page. Using similar approaches in presenting/modifying information within other online spaces may help in creating web-content that is used, valued, and trusted by Canadians.
Lesson 3: Keeping it Simple
There is beauty in simplicity; consumers are over-exposed to so much information nowadays that the concept of simplicity stands out as elegant, refined, and credible. The addition of a ‘call-to-action’ button and rewording existing language kept the intervention landing page clean and free from clutter and special effects which seems to have enhanced the user experience (based on the greater number of form visits).
Lesson 4: Openness Drives Innovation
With the early support from Program senior management, various brainstorming sessions were held allowing for creative thinking on different ways to experiment with the current website. Fortunately, there exists a diverse pool of expertise and backgrounds within CPSD. As a team of uninhibited technical specialists, many fruitful discussions took place on ways to experiment with the current landing page of the incident reporting portal. This resulted in a landing page that was informative while enhancing the user-experience.
What’s next?
Results from our experiment have shown us that web traffic can be easily modified by making simple changes to the web interface; user experience is a crucial factor in this process. We are now updating the existing landing page to reflect the lessons learned. Longer term, we anticipate continuing with the experimental process to further engage the Canadian public and industry in reporting consumer product and cosmetic incidents.
The consumer incident report form is long and may appear intimidating. If we are considering form reform then we need to work with collaborators from different parts of our Department as well as other Federal partners such as Shared Services Canada. To work around our practical constraints, we will need to carry out this project in phases. While the current phase is targeted at bringing more users from the landing page to the form itself, the next phase is to increase completion rate of that form.
Increased rate of reporting allows us to collect more data which in turn increases our ability to identify hazards and manage risks with consumer products in the marketplace. Future modifications will focus on user experience and provide more context on why the data is being collected and how it will be used.
Post by Health Canada’s EW Team
Article également disponible en français ici: https://medium.com/@exp_oeuvre
