Experimentation Works for Energy Efficiency in the Home
Team Office of Energy Efficiency (OEE) is excited to represent Natural Resources Canada (NRCan) and deliver two experiments through the Experimentation Works (EW) initiative! Both relate to energy efficiency in homes and EnerGuide, an energy rating service for Canadian homes. EnerGuide is the official mark of the Government of Canada for its energy performance rating and labeling program for consumer items — homes, light-duty vehicles, and certain energy-using products.
Experiment #1: EnerGuide Label for Homes
Research question: Does the EnerGuide label effectively convey energy efficiency information to homeowners?
Context: The EnerGuide label aims to help homeowners become aware of their home’s energy performance (via a rating) and encourage energy efficiency actions, like retrofitting your home. Our preliminary user research has determined that there are opportunities to design the label to be more consumer-friendly. A literature review of residential labeling identified some opportunities to improve the label so that it resonates more with Canadians. The OEE is also exploring and transitioning to digital options to inform Canadians on-line in more customized ways. The results of this experiment will inform the continuous improvement of the label and our digital transition.
Experiment #2: Nudging Homeowners Towards a Home Energy Evaluation
Research question: How do interventions such as message framing and rewards influence homeowners to take action towards getting a home energy evaluation?
Context: The OEE is working with Carrot Insights to engage Canadians on energy efficiency, OEE’s services and tools (like ENERGY STAR and EnerGuide), and ways to be energy efficient via the Carrot Rewards app, where users can earn loyalty points for demonstrated learning and actions. For example, following a series of curated rewards offers related to energy efficiency in the home, more than 18,000 Canadians searched for a service organization that provides a home energy advisor in their area, the first step in getting a home energy evaluation and obtaining an EnerGuide label to inform action. This experiment will build on our previous work and test interventions, like message framing delivered through the app, to encourage homeowners towards getting a home evaluation.
Here are a few insights from our EW experience thus far:
Experimentation ≠ Doing something new and measuring what happens
As stated in the GC Experimentation Direction and by our TBS colleagues when introducing EW, our aim here is to apply experimentation in practice in our policy and service context. That means being rigorous in how we collect and use evidence to understand what works and what doesn’t through:
· deliberate, thoughtful, and ethical experimental design;
· comparisons between interventions and base cases to capture evidence (e.g., randomized controlled trials, A/B testing, counterfactual experiments, baseline performance data, pre- and post-tests);
· randomized assignment to test and control groups, whenever possible;
· rigorous impact measurement and causality assessment; and
· transparent publication of positive, negative and neutral results.
While it’s not as easy as flicking a switch to start experimenting, it also doesn’t have to be an intimidating experience. We can start from where we’re at, find an area that would benefit from a small-scale experiment to understand what works, and begin honing and toning our experimentation mindsets and muscles. OEE’s EW team has benefitted from exploring and generating hypotheses related to its policy and service contexts by applying innovation methods, like service design, lean and foresight prior to EW. Which brings us to our next point…
Our experimentation journey began before EW
OEE is working with partners in and out of government to advance policy innovation and experimentation for energy efficiency impacts. We’re putting into practice more open, social and digital ways of shaping and delivering policies, services, and tools by co-creating value in three interrelated ways:
- Building relationships and capacity for policy and service innovation via collaborative and participatory methods and approaches that directly engage citizens and stakeholders;
- Co-creating and testing insights and interventions by generating new knowledge and evidence through action research and experimentation; and,
- Implementing and scaling what works through program redesigns and new policy directions, where applicable.
For example, EnerGuide is an established service and set of tools to inform homeowners and actions to improve home energy efficiency. OEE has been collaborating with the Canadian Digital Service, and other partners and stakeholders to understand user experiences and where we can innovate to improve EnerGuide’s reach, uptake and impact. OEE also has a working relationship with Carrot Insights in place, which opens a digital channel where we can directly engage Canadians, measure and learn from what works. When we heard about EW ramping up, we were well positioned to say, “sign us up” to take our experimentation practice to the next level.
We need three things to experiment: touchpoints, data and interventions to test. And it helps to work with experts
Our EW experience has already exposed us to colleagues in other participating departments and experts, like David Holysh and Philippe Bélanger from the Canadian Revenue Agency’s Accelerated Business Solutions Lab and (prior and up to the EW launch) Supriya Syal, who have provided advice as we proceed with designing our experiments. We’re also pleased to be working with Michael Kalin, who recently joined NRCan with expertise in behavioural insights, experimentation and data analytics. They’re pushing us to be clear about our research questions, ensure have access to the data we need to measure, and design an experiment with rigour to systematically learn what works or not to inform subsequent action. With their guidance and support, we’re improving our practice to create and demonstrate value for the OEE and the users of our services and tools through experimentation.
Senior leaders and managers play a critical role in protecting the time, space and resources for experimentation
As public servants, we need the capabilities, opportunities and motivations to learn and apply experimentation on-the-job and in relation to what we’re trying to accomplish via our policies, programs, and services. Having senior leaders and managers who understand the value of experimentation and willing to support our learning and practice is why we’re positioned and participating in EW. We wouldn’t be here without them and their commitment to continuous improvement and innovation.
Let’s not let the capacity-building initiative itself become our primary focus of attention
EW is a new initiative. Inevitably, there’s a ton of work required to get it off the ground, like information packages, proposal templates, senior management meetings, briefing packages, weekly calls and check-ins, collaboration tools, and learning events. We’re grateful to the TBS team for all the work they’re putting into it and the opportunity to showcase our work. Sometimes in government though, we have a tendency to focus a lot of our attention on the initiative itself and less on the people, contexts and needs that we’re attempting to reach and support. To get the balance right, we all have a role to play. We’ll have to protect the time and space during our journey to amplify our experimentation experiences and insights to not only fulfill our EW requirements (e.g. deliver one or two experiments), but to actually build and advance a culture of experimentation in government.
Thanks to the participating departments, experts, and the TBS EW team. We’re looking forward to the journey!
Post by NRCan’s OEE EW Team: Zoë Langevin, Mariel Angus, Diana Roca Baltzis, Barbara Omolewa, Jelena Golic, Marty Gaudet, Joël Bourgeoys, John Kenney
Article également disponible en français ici: https://medium.com/@exp_oeuvre
