Pam Sooriyan, University College Dublin, Ireland

Consistency in coding is central to the integrity of qualitative analysis

Bias denotes a displacement of one’s responses, often without retrospective awareness, along a continuum of possible unconscious judgements.¹ Bias is an integral part of research in that it provides a unique and guiding lens throughout various stages of the research. Bias is essential for research in that it is the preconceived views that lead to research. While explicit bias can be consciously registered and reported, implicit bias arises outside of one’s conscious awareness and is not always congruent with an individual’s consciously held beliefs, attitudes and stances.¹

As the researcher’s biases influence all aspects of the research process, it is important for the researcher to be aware of their own biases, assumptions, and inclinations toward the issue under research. Reflexivity and transparency are central to rigour in research.² Formulating an a priori framework enables the researcher to spell out their preconceptions and analytic focus, gained from their knowledge of literature, discussions with experts in the field and their past experiences. Articulating these views as concretely as possible allows the researcher to think outside of them while collecting and analysing the data.

The primary difference between explicit bias and implicit bias revolves around the degree of consciousness.¹ Both mental constructs influence our decisions and understanding. While explicit bias can be self reported by dialogue and introspection, this self reporting is often not sufficient to capture all dimensions of one’s prejudices³ and hence, implicit bias remains unaddressed. Early life experiences and culture lay the foundation for our implicit biases⁴ which tend to be influenced more by affective experiences than by critical frameworks that emerge out of self-reports. Additionally, self-reports themselves are subject to biases caused by impression management and social desirability concerns.

The process of qualitatively analysing interview data includes transcription, indexing and sorting the transcripts for analysis, and interpreting the data. The influence of the biases of a researcher that could not be articulated in the a priori framework can be mitigated by other researchers that are involved in the interpretation of the data. Involving other researchers in the interpretation stage of the analysis and establishing a reliability index can minimise the effects of projection⁷ from the implicit biases of the researcher onto the data. The other researchers being of different backgrounds allows for various perspectives to be included in the analysis.

Inter rater reliability (IRR) is a technique of reaching consensus among the researchers regarding the interpretation of the raw data, and providing a measure of that consensus, a reliability index expressed as a percentage. A reliability index of 75% has been suggested to be the minimum to demonstrate adequate levels of consensus.⁵ 

The process and purpose of establishing inter rater reliability can vary depending on the nature of the research and resources available. In a sample of given number of lines, each researcher classifies the data into categories and constructs as per the a priori framework. Data that does not fit into the a priori categories and constructs result in the emergence of new categories and constructs. The researchers’ codes are cross-tabulated, resulting in a contingency table as shown below.

Inter rater contingency table.

This contingency table can be extended for any numbers of researchers involved in the procedure. Lack of consensus often leads to an iterative process with the main researcher providing more context to the other researchers and/or sorting the data again, and repeating the interpretation of the data. Any disagreements that remain show a part of the spectrum of interpretations. A priori constructs and categories that have not been used are discarded from the framework, new ones that emerged from the data are incorporated in to form the emergent framework.

When more than two researchers are involved, instead of all the researchers going through the same data, samples can be randomly assigned to pairs/groups of researchers.⁶ After establishing the initial consensus, the subset of data that did not gain agreement can be dealt with by involving researchers who were not originally assigned that particular sample of data and repeating the process. This requires the main researcher to keep track of the assignments, and can be more time efficient than all the researchers coding the same data.

The procedure of inter rater reliability can be used to establish a reliability index for the analysis with all of the researchers coding all of the data. This procedure can also be used to establish the process of coding or mapping the data. Once a sufficient level of consensus is reached, the researcher can continue the process by themselves with only occassionally checking with the others. In this case, the percentage of the data used for establishing the reliability index is also reported along with the results of the analysis.

Inter rater reliability can also be used to refine the data collection process. Data collected from the pilot interviews can be analysed using this procedure, and the percentage of consensus reached or failed to reach can be used to modify the interview protocol.

Using inter rater reliability to qualitatively analyse interview data is relatively efficient and easier to implement, and has been widely used in medicine⁷ and in education research⁸.

References

1. www.kirwaninstitute.osu.edu State of the Science: Implicit Bias 2018-2020 [accessed 3 June, 2025].
2. Rosalind Edwards. (2023). Why qualitative research isn’t biased. National Centre for Research Methods online learning resource. Available at https://www.ncrm.ac.uk/resources/online/all/?id=20810 [accessed: 3 June 2025]
3. Justine E. Tinkler, Controversies in Implicit Race Bias Research, Sociology Compass, 6 (12), 2012.
4. Laurie A. Rudman, Sources of Implicit Attitudes. Current Directions in Psychological Science, 13 (2), 2004.
5. Saldana J, The coding manual for qualitative researchers. Thousand Oaks, CA, Sage Publications Ltd, 2009.
6. William D. Perreault, Jr. and Laurence E. Leigh, Reliability of Nominal Data based on Qualitative Judgments. Journal of Marketing Research, 36 (2), 1989.
7. Kate Roberts, Anthony Dowell and Jing-Bao Nie, Attempting rigour and replicability in thematic analysis of qualitative research data; a case study of codebook development, BMC Medical Research Methodology, 19:66, 2019.
8. Jessica T. DeCuir-Gunby and Allison W. McCulloch, Developing and using a codebook for the analysis of interview data: An example from a professional development research project, Field Methods, 23 (2), 2011.

Figure courtesy of Pixabay.com