This week in class, we examined both the field of IDEV as well as the role of citizens and different participatory media groups regarding their contributions to the field of International Development. Integral to this discussion about the field are the various professional standards and “good research” methodologies that provide the framework through which individuals contribute scholarly work.
In this post, I have chosen to highlight a few of the points and key themes that Kentaro Toyama and Jenna Burrell have written about in their fall 2009 article, What Constitutes Good Research?, from the Information Technologies and International Development Journal.
They begin by stating the fact that there seems to be much confusion and unnecessary judgment involving researchers using qualitative vs. quantitative methods. Toyama and Burrell posit that there needs to be more recognition that these are mere differences in research approaches and forms of evidence rather than measures of research quality. In particular, they use the multiple studies of mobile phones and their potential effect on macroeconomic growth to highlight examples of misjudgment and confusion.
Toyama and Burrell broadly identify four distinctive aspects that result in the unique qualities of ICTD research. The first is that ICTD involves a consideration of human and societal relations with the tech. world and specifically considers the potential for positive socioeconomic change. The second aspect is the inherent outsider vs. insider dilemma that allows for interesting perspectives through which researchers must operate. These circumstances greatly influence, according to the authors, the different past, present, and future experiences of ICTD researchers. The third aspect that they pinpoint revolves around the dynamic sensitivities of method due to human interaction and intervention techniques. Finally, they highlight the potential for methodological concerns due to the different backgrounds of ICTD researchers and their inherently diverse background knowledge that they bring to the table.
Toyama and Burrell next cover the issue of functional equivalence, or the overarching set of concerns with research quality and how it can be divided into issues of confidence and relevance. Confidence refers to issues of legitimacy and judgments of data and analysis, which they break down into accuracy, transparency and soundness of method, and empiricism. Relevance refers to the question of why the research would be of interest to others, either those practicing in the field or the general public, which Toyama and Burrell speak about in terms of novelty, disciplinary relevance, and generalizability.
Accuracy is the “extent to which a description of how research was conducted or what was learned from that research matches objective reality”, and posits that it is fairly easy to understand, mostly in terms of confidence intervals. They address the different theoretical schools of thought, such as interpretism and realism, and how they classify accuracy.
Transparency and soundness of method, sometimes referred to as rigor and replicability, are identified as the two most helpful ways of instilling confidence within research. As research inherently involves the presentation of material to outside individuals, Toyama and Burrell find that making the research rely on as many firsthand accounts as possible, in addition to the ease of replication by knowledgeable individuals will result in more sound research.
Empiricism refers to the necessary empirical grounding of ICTD research and reality based claims within research that reflect thoughts “of the world” rather than primarily flowing from theory or imagination. Toyama and Burrell write that research must have claims that are well and clearly supported with good evidence, either quantitative or qualitative, primary or secondary, with the ability for “confidence in the evidence itself”. In addition, they add that notions of “proper ICTD-related fieldwork” must be progressed as expanded, in order to allow for research that might rely upon “under-supported” institutional sites.
Toyama and Burrell find that it is “crucial” for a part of the research to attempt to cover ground that has not been inspected well before, in order to allow the research to contribute to the overall field. The issue of Novelty is an important one, and the authors note that it is important not to “simply repeat” work and consensus of the past.
The authors spend a large part of their article focusing in on the point of Generalizability, and the often over-generalizing that occurs within the ICTD community. They define generalizabilty much like that of statisticians, finding that “whatever patterns are found in the sample are statistically likely to be true of the population as a whole”, adding the varieties of case studies that can inform research. They note however that no single case study is adequate to form facts about a larger area, “though it can contribute toward models or hypotheses that offer illuminating” postulates or insights. Toyama and Burrell pinpoint the research surrounding mobile phones and their causal relationship with macro-economic growth in a region of India. Although the author of the study they mention warned against applying his research to other areas, they note the wide consensus among ICTD professionals towards the worldwide applicability of mobile phones for economic gains. The authors note that this area is most likely to receive tension and consternation, and conclude that great efforts to achieve both novelty and flawless execution must be maintained.
Toyama and Burrell conclude their article with the observation that more often than not, ICTD scholarly research seeks to be more “familiar”, rather than innovative paradigm shifting work. As no two researchers or professionals can dramatically alter the field by themselves, the authors lay out the framework for constructive next steps to be taken in order to consolidate and refine the methodologies that arrive from the various interdisciplinary fields that contribute to International Development. The authors see the need for more discussion between schools of thought and the variety of research fields is a necessary pre-requisite in the establishment of clearer standards and consensus.