Learning from the Review of EW
We are pleased to have had EW reviewed by TBS’s Internal Audit and Evaluation Branch (IAEB). Knowing that EW was too small of an initiative to be formally evaluated, we still wanted to know if EW is worth doing again, so we asked our colleagues in IAEB to do an independent Review. After all, EW is experimental, and that means measuring what works! (nb: we cannot call this type of assessment an evaluation since, in the Government of Canada, evaluations must abide by the TBS Policy on Results).
We thought we’d spend a little time in this blog post discussing some of the things our team thought about after reading through the Review (you can read the full Review here). We see this as an opportunity to give you our initial thoughts on what we’ve learned and what we plan to do going forward, though of course our thoughts will continue to evolve as we go.
This is also not meant to be an exhaustive commentary on all aspects of the Review — we are just highlighting some of the elements that caught our eye and where we think there are some learning worth reflecting on.
We’ll start with what the Review found worked well:
- Situating EW within TBS. We are happy to hear this. When we started EW, we were told by various folks within TBS that this is not a typical way a central agency attempts to grow capacity or stimulate desired actions. The more traditional model was one where a directive or policy is released, departments have to comply and are then assessed on how well they did so. In our case, we did indeed release a directive, and we are attempting to incrementally raise the bar on what success looks like in this space. However, we have also heard that departments were struggling with the directive in terms of a starting point, capacity and expertise, etc. EW was one of our (many!) responses to this, and a new way to engage, coming alongside departments in a learning-by-doing model that emphasized horizontal collaboration. We knew that experts need to be identified, tools and supporting documents need to be developed, and that a community of practice needs to be built, so we thought why not put it all together under one initiative?
- Pairing of projects with experts. We hoped this element would be key, and when we started, we knew that this expertise existed widely within the Government of Canada (particularly in science-based departments). The expert pairing helped support groups who might not otherwise have access to that level of expertise, and also helped engage experts who otherwise might not work in a policy or innovation-writ-large context.
Below you’ll find our thoughts on some of the elements which the Review found could be improved:
- Confusion about the definition of experimentation versus innovation. EW altered the definition of experimentation provided in the 2016 Directive. As the review points out, using a modified definition contributed to additional confusion in the system. We altered the definition because we felt that the one provided in the 2016 Directive provides broad enough direction to show the linkages between experimentation and various complementary fields such as innovation, but that ultimately, it was helpful to focus attention specifically on the experimentation part of the practice by using a more concrete definition. It’s important to recognize changing the definition also has downsides, though, and that maybe we should do more to communicate that revised definition. Our learning from this is that we should be mindful of the impact of definitional changes, and weigh it carefully if we ever choose to tweak definitions again.
- Building in flexibility / coordinating timelines across initiatives. As the Review found, the original timelines did not work very well for all teams involved — there was not enough flexibility in the model to allow teams to change course or take the time they fully needed with their project. We realized this as it was happening, but in a sense it was too late — the timelines in terms of commitments of experts, teams, etc. were already locked in. The ‘fallow period’ after the experiment phase was to end and prior to announcing results was intended to provide additional time for all projects to catch up, should they need that extra time, but we’re not sure that was communicated well, or that that alone would have solved the problem. Our learning from this is that for our next iteration of EW, we will look into having multiple streams based on capacity and general readiness to experiment — we will also take into account the time the first cohort took relative to the difficulty of the experiment (complexity, design challenges, data gaps, etc.) and budget time accordingly.Embedding results back into departments at experiment end. This is something we had not considered, and that we are very much taking to heart. When designing EW, we did not focus on ensuring project teams embedded results back into departments since we weren’t sure what individual departmental contexts would be like. However, this meant that teams did not always have a plan in place to embed their findings into departments. Our learning from this is that future cohorts should include a clear expectation for a knowledge dissemination / translation strategy so that the feedback loop (between finding what works and empowering decision makers to make decisions about what to do next) goes full circle.
The Review has also surfaced a few other interesting questions for us:
- On what to do about delays and having the ability to speed things up. In a sense, EW was an impetus for action that might not necessarily have happened otherwise. When trying to implement one of the EW experiments, delays were often caused by a lack of clear relationship between functions. When there was lack of progress, participation in EW was something departments could point to as a way to help stick to deadlines. We don’t have a sense of how long projects would have taken if project teams would have attempted to do the same interventions outside of EW, but our assumption is that it would have taken them longer, or that the interventions would not have even happened.
- That said, building a long-term experimentation practice is clearly broader than participating in EW. In our view, EW should be seen as a kickstart for a culture of experimentation in a particular setting. The individuals involved, hopefully bolstered by the experience, could then grow in confidence to replicate it again. Whether one project’s participation in an initiative like EW is sufficient to kickstart a culture of experimentation department-wide or settle for a much smaller scale (e.g. just within the confines of a small team directly involved) remains to be seen. We suspect the ripple effects in some departments might be larger than in others, since it’s also about readiness and enabling conditions.
- On growing experimental expertise across a team versus having it as a specialized function. We’ve been wrestling with this point ever since starting to work on this file. We don’t think that participating in a year-long process like EW will make a policy generalist an expert on experimentation. Experimentation is a rigorous field that takes time to learn. Given that, we will never advocate against having experimental design expertise embedded in as many teams as needed. That said, we do think there is tremendous value in exposing teams (and we include here management) to experiencing experimentation on the ground. All generalists need to be able to understand the role of experimentation, be able to ask the right questions, including identifying when they need further expertise. When we started, the baseline level of basic knowledge was very low, so EW was an attempt to expose more individuals outside of the ‘converted few’ to the practice of experimentation, complementing work going on across the public service to support the policy community.
We appreciate the evaluators’ time and support in reviewing the first cohort of EW. EW is the first time that TBS has done this style of cohort, learning-by-doing, open-by-default project, and so we’re very keen to make sure we’ve learned from the experience so that we can do better next time.
Post by TBS EW team: Dan Monafu, Terhas Ghebretecle, Pierre-Olivier Bedard
Article également disponible en français ici: https://medium.com/@exp_oeuvre
