Mitigating Bias: Integrating GBA+ in the Research Process
When should a project team reflect on the implications of different types of bias in a research project? As Experimentation Works 2 (EW2) experts, we asked our project teams this question in a recent learning session that discussed how to mitigate bias in research.
Bias can affect any phase of the research process; for example, failing to consider the unique experiences and inequities faced by sub-groups in the population or insufficient sampling of marginalized groups, despite being members of the population of interest. Historically, bias has contributed to an underrepresentation of marginalized groups in research and “solutions” that differentially benefit (or disadvantage) varying sub-groups of the population.
There is no perfect solution for solving bias. But there are many ways an individual or project team can reflect, challenge assumptions, and design their data collection and analysis practices to avoid perpetuating bias in their research. We developed a learning event for the EW2 cohort to outline how Gender-Based Analysis Plus (GBA+) can be integrated throughout the research process and to create space for project teams to reflect on how GBA+ can be incorporated in their projects.
Our learning event focused on applying GBA+ to quantitative research because of the experimentation focus of EW2. The presentation portion of the learning event walked participants through ways to apply GBA+ across the stages of the research process, including questions for researchers to consider at these different stages. In what follows, is a summary of these approaches and questions.
Identifying the problem
Before commencing a research project, those involved in the planning should consider the composition of the research team or advisory board. Given the influence of bias in research, collaboration and consultation with those who can bring intersectional perspectives to the research, because of their lived experience and/or expertise in the area, can be critical to producing meaningful research that is beneficial to the communities included in or affected by the research.
When defining the problem at the outset of a research project, Cole (2009) suggests integrating intersectionality by asking three questions:
1. Who is included within this category? Reflecting on who is included within a social category can help to prevent researchers from focusing on only certain, often privileged, subgroups and ensure representation that reflects the diversity of the social category.
2. What role does inequality play? Sub-groups of people within a social category will have different historical and current experiences of inequality. Considering this, and the social processes that contribute to this inequality, can help to shape the types of problems under examination in the research.
3. Where are there similarities? Looking at commonalities across different sub-groups can help researchers identify similarities in experiences that are influenced by the broader culture, social structures, and human behaviour.
Additional GBA+ questions to consider during problem definition:
• Who is the research for and does it advance the needs of those under study? Will the research have unintended impacts/consequences for those not specifically targeted?
• Which groups should be consulted (e.g., internal and external stakeholders, subject matter experts, those with lived experience) to inform problem definition and other aspects of the research process?
Literature Review
GBA+ questions to consider when conducting a literature review:
• Does the literature review reveal gaps in the representation of the experience of different groups of women, men, and gender-diverse people?
• Are there biases in the literature that perpetuate stereotypes?
• Might the findings obtained in the literature yield different results if (other) marginalized groups were considered?
A traditional literature review that incorporates keywords relevant to the subject matter may be insufficient for addressing the above GBA+ questions and for understanding the unique experiences and challenges of different sub-groups. Literature search strategies, such as including intersectional terms through Boolean operators (e.g., subject matter AND race AND gender), and writing literature reviews that incorporate intersectional research results and/or identifies gaps in the consideration of intersectionality in the literature, help to integrate GBA+ in literature reviews.
Sampling & Measurement
Sampling randomly from a population often does not produce a large enough sample size of low-frequency groups to perform quantitative analyses. For this reason Else-Quest & Hyde (2016) recommend stratified random sampling or purposive sampling.
GBA+ questions related to sampling:
• Is the sample representative of the experiences of diverse groups of people for whom the issue under study is relevant?
• Does the sampling approach reinforce traditional notions of representation that focus on most commonly shared experiences?
• Are there barriers to participation for under-represented groups (e.g. access to internet, shift work)?
Conceptual equivalence is an important consideration when it comes to measurement. It refers to whether the measure being used taps into the same construct across groups (Else-Quest & Hyde, 2016). For example, a translated instrument, such as a scale appearing in a survey, may contain terms that are semantically similar across languages. However, subtle differences in a term’s connotations can systematically influence response patterns across linguistic groups. If a tool being applied in a research study is not equivalent for all sub-groups within the sample, it is not possible to decipher whether the results obtained are due to conceptual inequivalence of the tool or more substantive differences between groups. One way to ensure that a tool being applied is equivalent across different groups is by assessing measurement invariance.
GBA+ questions about measurement:
• Does the data collection strategy provide opportunity for expression of diverse experiences and perspectives?
• Does the methodology address the gaps identified in the literature review?
Analyzing & Interpretation
There are several quantitative analytical approaches that may be suited to exploring intersectional differences. Depending on the research question and available data (e.g., sample size, type of data), some options include factorial design, multiple regression with moderators, multi-level modeling, and moderators in meta-analysis. Regardless of the analytical approach applied, the intersectional analysis undertaken should be grounded in an explicit rationale that considers historical and current inequality experienced by the groups being compared. Without this linkage, the results lack context and meaning.
GBA+ questions relevant to analysis and interpretation of results:
• Based on the results, are there implications for diverse groups of men and women and the relationships among them?
• What limitations are there for the interpretation of results if there was unequal sampling or limited representativeness of diverse groups?
• Do power and inequality contribute to the findings? If so, how?
• Are gender roles or other identities of subpopulations presented in absolute terms?
This brings us back to the question, when should a project team reflect on the implications of different types of bias in a research project? We hope, after reflecting on the above, that it is no longer a matter of when but a matter of how and where. To help get you started, the activity sheets from the learning event are available here. Use the questions provided for projects in the exploratory and/or experimental phase to consider how and where an intersectional lens can be applied throughout your research. We hope these questions and materials will help you explore tangible ways you can deliver research while understanding the impacts bias can play throughout your work.
***
Post by Sara Rubenfeld, PhD, Defence Scientist, Department of National Defence and Greg White, Project Manager with Canada’s Free Agents, currently posted at Employment and Social Development Canada
Article également disponible en français ici : Atténuation des biais : Intégrer l’ACS+ au processus de recherche | by L’expérimentation à l’œuvre | Avril, 2021 | Medium
References & Resources:
Better Evaluation (n.d.). Sample. https://www.betterevaluation.org/en/rainbow_framework/describe/sample
Bowleg, L. (2008). When Black + Lesbian + Woman ≠ Black Lesbian Woman: The Methodological Challenges of Qualitative and Quantitative Intersectionality Research. Sex Roles, 59, 312–325.
Bowleg, L., & Bauer, G. (2016). Invited reflection: Quantifying intersectionality. Psychology of Women Quarterly, 40(3), 337–341.
Canada School of Public Service (2020). Removing Bias and Building Trust in Your Data. Retrieved from https://www.youtube.com/watch?v=8bOFIkX213s.
Cole, E. R. (2009). Intersectionality and Research in Psychology, American Psychologist, 64(3), 170–180.
Else-Quest, N. M., & Hyde, J. S. (2016). Intersectionality in quantitative psychological research: II. Methods and techniques. Psychology of Women Quarterly, 40(3), 319–336.
Hachey, K., Bryson, R., & Davis, K. (2016). Defence science research and Gender-Based Analysis+: Shaping the research landscape. Director General Military Personnel Research and Analysis Scientific Poster DRDC 2016. Defence Research and Development Canada.
Hankivsky, O. (2014). Intersectionality 101. The Institute for Intersectionality Research & Policy. https://www.researchgate.net/profile/Olena-Hankivsky/publication/279293665_Intersectionality_101/links/56c35bda08ae602342508c7f/Intersectionality-101.pdf
Hankivsky, O., & Mussell, L. (2018). Gender-based analysis plus in Canada: Problems and possibilities of integrating intersectionality. Canadian Public Policy, 44(4), 303–316.
Harachi, T.W., Choi, Y., Abbott, R.D., Catalano, R.F., & Bliesner, S.L. (2006). Examining equivalence of concepts and measures in diverse samples. Prevention Science, 7(4), 359–368. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3293252/#
Kopper, S., & Sautmann, A. (n.d.). Randomization. Abdul Latif Jameel Poverty Action Lab (J-PAL). https://www.povertyactionlab.org/resource/randomization
Lee, S.T.H. (2018, September 18). Testing for measurement invariance: Does your measure mean the same thing for different participants? Association for Psychological Science. https://www.psychologicalscience.org/observer/testing-for-measurement-invariance
Malone, H., Nicholl, H., & Tracey, C. (2014). Awareness and minimisation of systematic bias in research. British Journal of Nursing, 23(5), 279–282.
Potochnik, A. (2020, August 9). Awareness of our biases is essential to good science. Scientific American. https://www.scientificamerican.com/article/awareness-of-our-biases-is-essential-to-good-science/
Women and Gender Equality Canada (2017). Gender-based analysis plus course. Retrieved from https://cfc-swc.gc.ca/gba-acs/course-cours-en.html
Women and Gender Equality Canada (2017). GBA+ Research Guide. Retrieved from https://cfc-swc.gc.ca/gba-acs/guide-en.
