Review of the Experimentation Works initiative (May 2019)
Note: Please use this link to download the Review in .docx or .pdf formats. To see the EW team’s response to the Review, please see this link.
Introduction
The review of the Experimentation Works Initiative (the initiative) was conducted at the request of the Experimentation Works team (nb: the team has since re-branded as the Innovation and Experimentation team) in Strategic Policy in the Priorities and Planning Sector to support future design and delivery modifications. This review focused on three main questions:
1. How appropriate is Experimentation Works initiative’s design to supporting experiments?
2. To what extent did this initiative enhance experimentation capacity in participating departments?
3. To what extent did Experimentation Works’ team contribute to the successful implementation of the experiments?
This review is part of the Treasury Board Secretariat of Canada’s (TBS) five-year departmental evaluation plan. The research methods and limits of this review are presented in Appendix A.
Experimentation background
Experimentation within the federal government is a priority articulated in the President of the Treasury Board of Canada’s Mandate Letter. In 2015, the President was tasked to ensure that departments and agencies devoted a fixed percentage of program funding to the implementation of new solutions to existing problems.
In 2016, in response to the mandate letter, TBS and the Privy Council Office clarified this commitment with the publication of the Experimentation Direction for Deputy Heads (the Directive). The purpose of this Directive is to instill a working environment that fosters experimentation and the sharing of positive, negative or neutral results. These experimental projects are expected to resolve persistent and existing problems that traditional approaches have failed to solve and generate data on results.
In this context, the Directive defines experimentation as “testing new approaches to learn what works and what does not work using a rigorous method” whereas innovation is defined as the implementation of new ideas, approaches, solutions, methods or services. These experimental projects can take the form of comparisons between interventions and a control group, causality assessments on program impacts or impact assessment strategies for a new approach.
Initiative profile
In 2018, TBS implemented the Experimentation Works Initiative to respond to the requirements of the Experimentation Direction for Deputy Heads. This initiative was designed to take place over a period of one and half years. The theory of change is depicted below, with the logic model illustrated in Appendix B (nb: this is the broader team’s theory of change, not the EW initiative; the logic model presented in Appendix B is the logic model specific to the EW initiative, not that of the broader team).
Cohort Model
In April 2018, the initiative launched its first cohort of five experiments from four departments, one of which withdrew from the initiative.
The cohort model was selected in order for participants to benefit from each other’s experience. The initiative provided the cohort a variety of tools and resources, learning materials, events and sessions, access to experts and senior management exposure.
Governance structure
The initiative is led by the Experimentation Works team in Strategic Policy of the Priorities and Planning Sector of TBS. It used existing resources and allocated approximately one full-time equivalent.
The Assistant Deputy Minister Committee on Experimentation serves as an interdepartmental forum for the initiative. The initiative uses this forum to provide updates to members on participants’ experimental projects and to share the needs of the initiative for discussion and action when needed.
Stakeholder Roles and Responsibilities
A memorandum of understanding was signed by TBS and each stakeholder. The roles and responsibilities are as follows:
Treasury Board Secretariat
The Experimentation Works team is responsible for leading the overall initiative. It supports the participants through documents, learning events, collaboration activities and senior-management exposure. The Experimentation Works team is also responsible for monitoring, reporting and distributing lessons learned to the stakeholders and forums.
Participating departments
Departments are responsible for developing and leading their experimental projects by using existing resources. They also commit to publicly sharing outcomes of their projects through various forums.
Expert partners
Four (nb: the Review was completed prior to the EW team adding a fifth department to this list) departments provided expertise in experimentation to the participants through consulting sessions and collaboration. Participants were also required to write at least one blog post describing their experience in this initiative.
Cohort 1 Experimental Project Participants
The Paul Yuzyk Youth Initiative for Multiculturalism (a Canadian Heritage project)
In 2015, the Paul Yuzyk Award was converted to a micro-grant initiative which would award micro-grants worth $250, $500 and $1000 respectively to young applicants who propose ideas to promote diversity and inclusion. This experimental project’s purpose was to gather descriptive and qualitative data from applicants to measure the program’s effectiveness. The information gathered, using a survey and telephone interviews with beneficiaries, enabled the program to increase its understanding of the impacts of its grants and the potential for scalability.
The PRODigy Initiative (a Health Canada project)
Only a small percentage of Canadian consumers report product safety issues to Health Canada. This experimental project aimed to increase the consumer incident reporting rate of consumer or cosmetic products by comparing the effectiveness of two incident reporting forms randomly presented to consumers on the departmental website. An increase in the reporting rate would generate additional information for Health Canada and would help the department in identifying problems and trends in Canada. The project would support the next program iteration.
EnerGuide (a Natural Resources Canada project)
The EnerGuide label provide information about the energy rating demonstrating the energy performance of a home. The current label is provided to the homeowner in a paper format and while Canadians understand the information on the label, the value proposition of the label alone is not clear. This experimental project aimed to improve Canadians’ experiences with the label and to increase understanding of their home’s energy efficiency. The owners randomly received one of 36 EnerGuide labels through the Carrot Insights app in order to enable comparisons among labels to determine which were most and least effective.
Energy Efficiency of Homes (a Natural Resources Canada project)
This experimental project built on previous work from Natural Resources Canada and aimed to nudge homeowners to go beyond searching for an energy advisor to getting a home evaluation. Three different messages were sent out using the Carrot Insights app to encourage homeowners to make use of an energy-efficiency advisor.
Effectiveness of the initiative
Appropriateness of initiative design
The review’s findings on what worked well with the design:
- This GC wide initiative is well situated within TBS. As a central agency, it is able to exert leadership and influence over departments.
- The pairing of projects with experts is a component that stakeholders found useful in helping them refine the approach of their projects.
What could have worked better with the design:
- The definition of “experimentation” adopted by the initiative differs from the definition of the Directive which led to some confusion among participants.
- Coordinating timelines across initiatives was a challenge for at least one project. Others wondered if delivery flexibility was an option.
- An invitation to participate in the cohort was only sent to the members of the Assistant Deputy Minister Committee on Experimentation. Sending a call letter across federal government would have expanded the initiative’s reach.
- Helping initiatives apply their results at the end of their experiment.
The definition of experimentation
The stakeholder interviews revealed some confusion between “experimentation” and “innovation”. Participants found that the definition adopted by the initiative differed from that of academics, the Directive and the Management Accountability Framework. The initiative clarified experimentation as follows:
“Experimentation” refers to activities that seek to examine, test and compare the effects and repercussions (i.e. what works) of policies, actions and approaches in order to clarify evidence-based decision-making.
Although the Directive allows for the use of quasi-experimental methods, the participants noted that the EW team and the experts were focused on randomized control trials and A/B tests. The misunderstanding may have been a result of:
- The interchangeable use of innovation and experimentation in documents.
- The absence of definitions or criteria related to the “resolution of persistent problems and the testing of new approaches in order to determine what works and what does not”.
- The recommended approach by the EW team and experts (i.e. the randomized controlled trials) being more rigorous than that identified by the Directive.
Governance
Key informants felt that it was appropriate that TBS coordinated the initiative because of its central leadership position. They also felt that the involvement of a central agency facilitates the establishment of a government-wide culture of experimentation.
Project selection
The EW team anticipated that several departments would participate in the initiative and they developed a selection process with associated criteria. The team invited members of the Assistant Deputy Minister Committee on Experimentation to launch and discuss with each member the projects demonstrating an “overall level-of-readiness”. Once they amassed an adequate number of proposals members helped finalize the cohort. However, given the low number of project proposals received, the discussion process replaced the formal project selection process (the Innovation and Experimentation team received a total of 6 projects).
Cohort model and timeline
The initiative introduced a cohort model and experts observed that this approach encouraged the sharing of lessons learned, difficulties, practices and clarifications regarding issues between the cohort members.
The EW team intended that all of the participants would begin their projects (experimental phase), share their experiences (results phase) and report the results (impact phase) in accordance with a specific timeline (Appendix C presents the initiative timeline). However, all stakeholders mentioned that their projects did not follow the same pace and that flexibility in the timeline would have been desirable.
According to interviews, the required timelines complicated matters and could potentially have had negative repercussions on the usefulness of their projects. For example, TBS’s Canadian Digital Service withdrew from the initiative because their project timeline was faster than the initiative timeline and could not be delayed due to a memorandum of understanding with a client department.
Expert pairing
Pairing experts with participant departments was a key component of the initiative. The document review and stakeholder interviews revealed that the pairing began earlier than expected in order to refine the proposals and the design of the experimental projects. In fact, the departments did not have sufficient capacity to satisfy the selection criteria and required assistance in defining their projects.
The sharing of practical experience, feedback and dialogue allowed participants to better understand the concepts of experimentation, to identify problems, to clarify the research questions, to improve the thoroughness of their projects and to adopt new solutions when an obstacle appeared.
All stakeholders found this approach extremely useful and the experts were unanimous in saying that they would participate again if the opportunity presented itself. One interviewee observed that the initiative had allowed them to put the academic side of experimentation into practice within a federal context.
Use of the results
One of the objectives of the initiative is to generate evidence to inform decision-making in order to reduce the risk of investing in poorly-performing programs, products or services.
The review of the memoranda of understanding revealed the presence of provisions regarding the sharing of results, obstacles and lessons learned. However, there was no requirement or guidance regarding the department’s use of results following project completion.
The interviews indicated a lack of direction regarding next steps once the projects were completed. While there is an appetite for using the results from experimental projects for decision-making, there was no answer to the question, “what next?”, or what decisions would be informed by the project results. For example:
- Should the same project be attempted with a new approach?
- Following negative results, should the project be redefined?
- Should the results be shared with senior management?
Experimentation capacity building
One of the results targeted by the initiative is the building of experimentation capacity, skills and practices among the participants through sharing results and exchanges between the cohort members.
The experts indicated that the initiative and cohort model fostered capacity building; however, some stakeholders indicated that its value was mixed among participants who had experience with experimental projects.
Stakeholders mentioned that this first cohort was better positioned to conduct new experimental projects because of what they have learned and their understanding of the experimentation concepts. The majority of stakeholders felt there is a need for a community of practice in experimentation across the government in order to share, create a synergy between the resources and strengthen the practice of experimentation.
The opportunity to experiment in departments, the access to expertise in experimentation and the availability of appropriate tools are still needed so that departments can continue to conduct experimental projects.
However, experimentation requires a specific skill set, some of which already exists within the federal government. This raises the question of the efficiency and effectiveness of having non-experts conducting experiments versus using dedicated teams that already have experimental research skills.
Canadian Heritage
Stakeholder interviews revealed that this project was more of an exploratory rather than an experimental project due to its research questions which were:
1) Would a youth-focused micro-grant initiative increase public engagement when compared to previous versions of the Paul Yuzyk Award?
2) Can the subjective data generated by the post-impact survey of successful Paul Yuzyk Youth Initiative applicants be used to create objective performance indicators for similar programs?
The experts worked closely with Canadian Heritage team to better define the project’s design and to improve their understanding of experimentation concepts. They were unanimous in terms of the positive dynamics and openness of the team to the advice provided as part of this project. According to the interviews with Canadian Heritage, the initiative increased their capacity to undertake experimentation.
Health Canada
The experts assigned to Health Canada noted that the experimental project required a stronger design to ensure the implementation of randomized control trials. According to Health Canada, this exchange with the experts increased their understanding of experimentation concepts and their ability to conduct successful experimental projects in the future.
Health Canada faced IT issues but was unable to hire IT support due to the centralization of IT within Shared Services Canada (SSC) and the difficulty in accessing their services in a timely manner. This required Health Canada to modify their plan with support from the EW and TBS web teams. The participants noted that a dedicated SSC resource or in-house IT resource for their project would have helped the implementation of their project.
Natural Resources Canada
Natural Resources Canada conducted two experimental projects as part of this initiative. Stakeholder interviews revealed that the Natural Resources Canada team had a high level of experimentation maturity and were able to complete experimental projects without the initiative’s support. More specifically, the two projects had already been started “with partners within and outside government to stimulate innovation and experimentation on energy efficiency policies” before joining the initiative.
Given the level of maturity within the department, the experts assigned to these projects were re-assigned to the Health Canada project which had a greater need for experimentation support.
Canadian Digital Service
The case of Canadian Digital Service was similar to that of Natural Resources Canada. In fact, discussions with the experts revealed that the experimental project at Canadian Digital Service had already begun before its inclusion in the initiative. Given the level of experimentation expertise at Canadian Digital Service, the experts wondered about the value of including the project in the EW initiative.
Initiative’s Contribution to the Implementation of Experimental Projects
Without the initiative, two experimental projects would have been conducted as planned given the maturity of the departments, and one project would have been conducted on a smaller scale. The fourth project was considered more an exploratory project than an experimental project because it was expected to gather information on the impact of a grant program.
The EW team developed tools and collaboration platforms that were useful to the stakeholders. However, there were few opportunities for the participants and experts to work together to develop government-wide experimentation capacity.
Appropriateness of initiative tools
The EW team developed several tools, collaboration platforms to engage in person, by teleconference or digitally, (e.g. Trello and Medium) as well as devising experimentation learning activities. In addition, articles, examples and case studies were posted on the initiative’s website.
The purpose of these collaboration tools, specifically the blogs on the Medium platform, was for initiatives to share their lessons from their experimental projects. They also mentioned that it would be necessary to use other digital platforms in order to reach a wider audience to share results. In one case a department’s firewall blocked access to these digital platforms.
Some stakeholders noted that telephone or teleconference collaboration and exchange mechanisms were not conducive to a candid and spontaneous exchange. They suggested that such collaboration should be done in person in order to strengthen collaboration between the stakeholders. However one participant noted that it is the cohort members that generate collaboration, not the tools.
All stakeholders found the learning activities to be useful because they supported the experimental projects. However, it was suggested that the initiative explore the opportunity to integrate experts from outside the public service or experts with a specific expertise in order to better meet the needs of participants.
Absence of the initiative
The review asked interviewees how differently their experimentation project would have gone without the EW’s support.
Through its support and pairing with experts, the initiative expected to build the capacity and knowledge among cohort participants, however only two participants strengthened their knowledge and capacity. There was little added value of the initiative among participating departments that already had experimentation expertise.
The initiative was a catalyst for experimental projects at Health Canada and Heritage Canada, because these projects would not have taken place or would have been undertaken on a smaller scale without the EW initiative. In both cases, the teams would not have conducted their projects using an experimental approach.
However, the initiative did not impact the implementation of the Natural Resources Canada experimental project because their projects would have taken place despite the absence of the initiative. Interviews with stakeholders confirmed that Natural Resources Canada already have the capacity in experimentation and they conducted experimental projects in the past.
Considerations for the future
Basic knowledge of the experimentation concept
The initiative assumed that expert assistance would be needed only to implement projects. However, the review shows that for those with little experimentation experience, assistance was needed much earlier. Experimental concepts such as problem definition, hypothesis formulation, experimentation framework design, statistical methods, randomized control trials or the A/B tests require specialized expertise. It may be more effective for participants to have a basic understanding of experimentation approaches.
There are niches of expertise in experimentation across the government, decentralized within departments in existing teams dedicated to research, evaluation or experimental hubs. It would be useful for the EW team to explore ways for departments to benefit from their existing internal expertise.
Project selection
There appeared to be lack of understanding or direction among participants regarding the use of projects results and next steps. This may point to a larger issue of project selection and the type of decisions the initiative expected to influence. It may also indicate a need for clarity and additional guidance on how the selected projects should inform decision-making processes.
Although the EW team developed criteria in order to inform the final selection of EW experiments, there were no criteria or definitions regarding the identification of persistent and existing problems that traditional approaches had failed to solve.
Availability of data
The documentation and stakeholder interviews demonstrated that half the departments did not have the necessary data to undertake an experimental project. It is essential that data be available for an experimental project to better enable it to identify and examine existing and persistent problems.
Management support
Support from senior management and executives is critical to the success of experimental projects. The stakeholders mentioned that this is needed in order to
guarantee resources, opportunities and time required by experimental projects. In addition, departments must accept the risk of failure in an experimental project and be willing to report failures in order to take advantage of the lessons learn.
Resource availability
Resource availability for experimental projects is a factor that influences experimentation capacity according to the interviews. Some stakeholders mentioned the need to have additional resources in specific areas such as information technology, communications and graphic design in order to increase the success of experimental projects.
Cohort approach and delivery timelines
The cohort delivery model with their staged phases for delivery, lacked the flexibility needed when participants encountered delays caused by, for example, administrative processes used in grant programs and IT systems.
The EW team could explore other delivery models, such as ongoing project support. This would allow the initiative to better adapt to the variable pace of each project.
Sustainability
The initiative assumes that experimentation would grow within the government from the individual projects that were implemented. In order to do so, project participants would need to continue to undertake experimental projects in order to maintain their skills and knowledge. In the absence of opportunities to engage in experimentation on a sustained basis, the projects are discrete, “one-off” projects which would not necessarily lead to a proliferation of experimentation across government.
It would be useful for the EW team to re-examine the rationale for the initiative and validate whether the program design is likely to result in the growth of experimentation in the GoC.
Appendix A — Review’s Scope, Methodology and Limits
Methodology
As part of this review, the Internal Audit and Evaluation Bureau used three data collection methods to make its findings.
Documentation review
The documentation review provides information regarding the context, process, implementation and administrative data of the initiative. This project examined, without being limited to, the following documents:
- The presentations made by the initiative and offered during events or workshops;
- The memorandums of understanding between TBS and the participants;
- The documents produced by the selection process;
- The articles written and posted on the Medium digital platform;
- The minutes from the Associate Deputy Ministers’ Committee on Experimentation.
Interviews with key stakeholders
The Internal Audit and Evaluation Bureau held 13 group discussions with 24 stakeholders:
- The Experimentation Works Initiative team (4 participants)
- The experts (8 participants)
- The 4 departments participating in the initiative (9 participants)
- The projects that were not completed (2 participants)
These interviews made it possible to collect their feedback on the design, relevance and effectiveness of the initiative as well as to corroborate or invalidate the data collected from the other sources of information.
Case studies
A case study for each project was conducted to better understand the repercussions of the initiative on the experimental projects of the participating departments. The review sought to find out more regarding the following elements:
- The context of the experimental project;
- The initiative’s impact on the strengthened experimentation capacity and its added value;
- A description of the success factors and obstacles.
Limits
The review limited itself to the initiative’s repercussions since its implementation in November 2018 since all of the departments had not implemented or completed their experimental projects during the interviews. Given this limit, the findings are mainly from the interviews held with the participating departments and the review of the initiative’s documentation.
Appendix B — The Initiative’s logic model
The Experimentation Works Initiative developed a logic model to clarify its expected results.
Appendix C — Initiative Timeline
The initiative was designed to take place over a period of one and half years in four separate phases:
Phase I: Setup (October 2017 to March 2018)
This phase places emphasis on the initiative’s design, the presentation to the Associate Deputy Ministers’ Committee on Experimentation, consultations with the departments, the determination of expert support, the preparation of documents and the selection and clarification of the experiments in this initiative.
Phase II: Experiment (April to October 2018)
This phase places emphasis on the strengthening of the design, implementation and completion of experimentation projects by the first cohort with the support of the Treasury Board Secretariat and expert partners assigned to each project.
Phase III: Results (November to December 2018)
This phase places emphasis on the sharing of experiment results through blog postings and the results from the overall process of the Experimentation Works Initiative.
Phase IV: Impact (March 2019)
This phase places emphasis on the retrospective analysis of the experiments conducted as part of this initiative. The results from the lessons learned will be communicated to the public as well as the changes stemming from the results.
Post by TBS’ Internal Audit and Evaluation Branch.
Article également disponible en français ici: https://medium.com/@exp_oeuvre
