Lessons Learned After Three Years of Experimenting in the Canada Revenue Agency
The CRA has made experimentation a priority in recent years, and has become a leader in the Government of Canada in advocating for the use of experiments in evidence-based policymaking. Whether we’re looking for ways to increase benefits uptake, reduce the number of tax returns that come in late, or increase the use of digital services, experimentation has increasingly become the way in which new ideas and programs are tested at the CRA.
That being said, the path towards experimentation hasn’t always been easy, and we’ve learned some lessons over the past few years. We were happy to share these lessons with the Experimentation Works cohort at the last learning event, and since they might be useful for a team running an experiment for the first time, we also wanted to provide them here.
Understand the Issue
Why do you want to experiment? Is there a known problem with a policy or program? Have you identified a process that might be working decently, but could work better? Fully understanding the issue that you’re tackling, as well as reasons for why this issue might be present, is an important starting point, if not a prerequisite, for a successful experiment.
What if you don’t have a good understanding of the issue? In that case, you may want to consider doing some exploratory work first — such as data analysis or stakeholder interviews. If there isn’t an issue with the program, policy, or process that you’re looking into, even a well-designed experimental intervention may not have much of an effect.
Have a clear and measurable outcome metric
Do you have a clear outcome that you’re wanting to measure? This is essential for an experiment. If you cannot measure the outcome, or if the outcome that you measure is only indirectly related to an intervention that you’re wanting to test, it’ll be tough (at best) for you to see your intervention having an effect. If you have a clear and measurable outcome metric, you also may have an easier time designing your intervention, because you’ll know exactly the outcome that you’re looking to influence.
Start Small
For an initial experiment, there’s nothing wrong with implementing a simple experimental design. Generally, an experiment with one control group and one treatment group is the most straightforward way to design an experiment (and given a set sample size, you’ll actually be more likely to determine whether your intervention had a statistically significant effect on the outcome). Also, if you’re unfamiliar with randomization, or don’t know much about your population, simple random sampling (instead of more complex randomization options) may be sufficient for your needs.
Start Early
This sounds obvious, but it’s important. If you have additional time to identify the problem, think about interventions, and design the experiment, you’ll see positive returns in the quality of your experiment, and you’ll be less likely to make mistakes.
Don’t sacrifice experimental rigour
When running an experiment for the first time, there likely will be many unfamiliar steps. How do I randomize? How large a sample size do I need? Are the differences between my control group and treatment group statistically significant? These are important questions, and when designing and analyzing an experiment, it’s essential to take these questions (and others) into account right from the start. There are enthusiastic experts throughout the Government of Canada who can help you with these question — in addition to a very wide range of online resources.
Get feedback
It never hurts to talk about your ideas for an experiment with others, especially from a wide range of backgrounds and perspectives. The insights that you gain from talking to others will help to refine your thinking, and may illuminate new ways to tackle a particular issue or problem.
Talk about your Results
It’s hard for your experiment to have an impact if no one knows about it. Speaking about your experiment — either within your department, across the Government of Canada, or to external audiences — is the best way to gain support for your work and for your findings to make a difference.
And trust us, we recognize that it’s always more satisfying to talk about experiments with interventions that had an effect — those that “worked”. However, it’s important to still to talk about experiments in which the interventions didn’t have an effect on your outcome. After all, knowing what doesn’t work is valuable evidence for teams conducting experiments in related areas.
David is a Senior Quantitative Analyst with Canada Revenue Agency
