Introducing…Experimentation Works
Hi.
We are the Innovation & Experimentation Team at the Government of Canada’s Treasury Board of Canada Secretariat (TBS). Our team’s mandate is to support the Government’s Experimentation Direction, which means we think a lot about how to increase the use of experimental approaches and experimental data when making government decisions. The hope is that being more rigorous in how we collect and use evidence will improve government programs and services.
We’re at the beginning of an exciting new journey: we are about to start an initiative that has taken up much of our little team’s time for the past number of months.
Experimentation Works, or EW for short, is a new Government-of-Canada initiative to build public servants’ capacity in experimentation skills and practice through a unique learning-by- doing model that supports and showcases small-scale experiments. EW seeks to generate practical examples of federal experiments and ensure open access to related learning materials, progress updates and results for broad impact. It will work by supporting 5 experiments running in 4 departments by connecting those projects to experimentation experts already working in the Government of Canada. The experiments will run at the same time and the participating departments will work together, in a cohort-model, to help share lessons-learned and bond over shared experiences.
Why we are doing this? Lots of reasons, but here are our top 5:
1) Guidance has to be guided by real questions, tied to specific context
It’s always better to provide answers that respond to real questions. EW will give us something that is a very important element of learning: context tied to a specific situation and place. We want EW to be responsive, and almost real-time. If, for example, one group in our cohort encounters an issue — say needs help figuring out confirmation bias — we want to document possible answers to tackling that issue through the help of our experts, releasing that as a stand-alone piece as quickly as possible. Chances are that if one group struggles with it, it will be useful to others in similar circumstances. And by detailing the context of the request along with the solution, we can provide rich information that will no doubt be useful to others.
2) You have to practice experimentation, you can’t just write about it in the abstract
This is something that we kept hearing over and over in our early days on the file: that our team shouldn’t write an “authoritative guide on how to experiment” for the Government of Canada (not that we would have, mind you), and that if we are not ourselves immersed in the practice of experiments, our materials would not have the right level of impact.
3) When working to design any intervention, the documentation and self-reflection aspects are often first to go
This is one aspect (among others) that we see as truly a ‘value-add’ on our part. We ourselves have been in many projects where the mere need to keep a project afloat means that there is very little time to properly document what exactly happened, so that others can benefit from this information the next time a similar project happens. Running around and juggling many projects also means that it is difficult for those involved in the project to step back and take in what exactly is happening. We hope to be that fly on the wall, or ethnographer in the room, that can see a bit more of the larger picture and ask probing questions that help explain concepts that may be obvious to those immersed in the project but which require clarification for outsiders.
4) To move beyond pilots to sustainability, we need to closely watch the system for change
We also really want to ensure that EW does not produce a set of pilots that go nowhere, so to us the last phase, Impact and Failure, will be key. We want to take a step back from the whole process once it’s over and come back to it 6 months later to observe what happened: did the experimental intervention and the new knowledge generated make it back into the particular organization that ran the experiment? Have new variations followed, to generate new tranches of evidence, using slightly different variables? Or did the pilot simply end?
5) Cohorts learn better, as it is all about relationships
Finally, our last point is that we thought there would be much benefit to setting up EW as a cohort, since people learn better when they are not alone but surrounded by peers. We hope that this will be just one cohort of many and that the individuals going through the process will see EW as an empowering part of their careers. Without sounding too lofty, we do believe that participants would then be able to recognize “what works” (or at least what the necessary conditions are for what works), although we recognize that this does not mean that EW will turn participants into experts in experimental methodology. However, participants will gain the relationships, ability to know what they don’t know (and who does), and knowledge of what it takes to go from the beginning to the end of an experiment.
Continue to follow our journey in this space — we promise to tell it in the open.
Post by Sarah Chan, Dan Monafu, Sean Turnbull, EW Team, TBS
Article également disponible en français ici: https://medium.com/@exp_oeuvre
