Continuity and Communication Across an Epistemologically Heterogenous Research Community: A Response to Kellam and Jennings (2021)

Reactions to the Kellam and Jennings Article

The authors open with a description of NSF panel reviews to motivate the importance of differences in the epistemologies of key engineering education community stakeholders (p. 80). While I resonate with the frustration over how some panelists view qualitative and critical work skeptically, and while I see these discussions as epistemological, I see this domain as more pragmatic than paradigmatic. I approach an NSF panel quite differently than a journal argument. A journal argument may have some underlying need for epistemological consistency, or at least an explanation or underlying reason for heterogeneity. I am usually quite insistent on the form of a journal argument to make sure it does not morph into something I do not recognize, philosophically. An NSF panel, however, seems motivated by pragmatism and thus a time to use all available arguments. When writing to these unknown members of a community, for the very capitalistic (and not critical theoretical) act of deciding how to allocate money, I tend to make use of all available arguments, positivist or critical or otherwise. I know that Kellam and Jennings are not specifically referencing my last successful or unsuccessful NSF proposals with their review, but I know for a fact that they had evidence of mixed critical/postmodernist and post-positivist logics. I do, however, resonate with these authors’ idea that there is some discrepancy between how valued critical qualitative work seems to be when measured with, say, best paper awards, compared to (many) NSF panel review experiences.

Next, the authors characterize the typical ways an epistemology and theoretical perspective can shape the research we do. They consider a hypothetical example (p. 81) where a post-positivist focused on LGBTQ issues could perform random sampling of 100 participants, conduct structured interviews, and present the results in a table; and a critical interpretivist could conduct unstructured interviews with three participants, analyze via a framework of counternarrative, and present results in papers and podcasts. I know this example is probably meant to be simple and illustrative for those new to research epistemology, but it still gives me pause. If the post-positivist in question did disseminate by a podcast, would they cease to be a post-positivist or would it be evidence of welcome epistemological pluralism? If the critical interpretivist decides to incorporate random sampling, is it encroaching positivism or is it just one of many possible conceivable choices? I admit my questions could sound facetious regarding an illustrative hypothetical example and may not be all that different than the spirit of argument the authors intended. Yet I sense Kellam and Jennings are setting up a priority on epistemological coherence across research project aspects. In that respect, I differ with Kellam and Jennings on viewing epistemological coherence as a priority; I see epistemological coherence as more a description of a tendency towards alignment when genuinely engaging epistemologically. I will elaborate my thoughts on an alternative to this priority in the subsequent sections.

Next the authors provide some background on researcher methodologies broadly and in engineering education specifically (pp. 81–83). I note a distinction between my view and the authors’ that probably underlies the higher stakes the authors attribute to epistemological coherence for critical and postmodern studies. That is, the history presented highlights the damage caused by positivism’s eugenicist origins (p. 81) and that alternatively “critical qualitative studies tend to be more liberatory” (p. 82). First, while I don’t disagree about the documented damages of eugenicist positivism, I wonder how to decide whether cases of using positivistic-looking tools that are not in service of eugenics are helpful or not. Are things that look like positivism always damaging because they were once used for eugenics, or are things that look like positivism sometimes useful because they can be repurposed for good? I do not know, but I lean more in the latter pragmatist direction and leave open the possibility for true critical quantitative work to repurpose the old tools. Second, I wonder about the pragmatic claims regarding critical qualitative studies—is the claim that critical qualitative studies tend to liberate more than studies that look like positivism? I might not disagree, but how have we assessed this claim? Is it liberation to simply subscribe to critical qualitative epistemologies? I think here I would measure the importance of critical qualitative work regarding its actual impact and would not presume a liberatory impact just because critical qualitative researchers are good at invoking the concept of liberation. So I understand Kellam and Jennings wanting to have greater (anti-positivist) epistemological coherence if it will help keep us away from the worst histories of scholarly abuse, but I think things could sound or look one way in theory and work differently in practice.

The authors go on to review the specifics of many qualitative articles from a single year (pp. 84–93). As a qualitative and critical researcher, I see myself (metaphorically, I didn’t publish a paper in the 2019 corpus) in many examples highlighted through the paper:

• Highlighting the “large numbers” in qualitative research. Personally, I use numbers when proposing a grant using ethnographic observation or some other critical discourse analysis: for example, “A combined 200 hours of observation” or “60 hours of video data.” Even while I present these numbers, I know deep down that the real power of the qualitative research can’t be quantified, and the vast amounts of data may be set aside for that small chunk that really matters. But quantifying things helps reviewers who may not understand what I’m doing and may not share the epistemology to at least appreciate that there is a lot of work involved. As some of the articles quoted in this piece mention (p. 92, via Koro-Ljungberg & Douglas, 2008, p. 172), there are in fact some easy ways to undercut the entire point of qualitative research as “just talking to two students” or “just observing” and it makes sense to substantiate one’s point at least in showcasing how extensive the work is. I don’t tend to do this as often in a journal paper, however, where I prioritize intellectual coherence and have more rounds of revision and more (epistemologically coherent, yes) tools at my disposal to convince reviewers and audiences.

• Limitations sections. Personally, I do not tend to write limitations sections that apologize for having non-generalizable research and the like. If I do comment on limitations, I usually do so in response to a reviewer and I usually phrase it similar to the Pawley example (p. 87, from Pawley, 2019, p. 19): like “Some might say X, but in reality Y is how we should think of it.” In general, I think if I write precisely enough, I will be limited and precise in my claims such that no reader would reasonably conclude I was trying to, say, generalize to an entire population that I am not. Here I am in firm agreement with the authors, it never seems helpful to write something that undercuts everything I have just carefully claimed. If I ever end up with a limitations section that reads like an apology for doing qualitative research, you can presume it was added under duress.

• Disembodied writing style. I usually prefer very specific writing and language rather than disembodied writing, so there again I align with the authors. I like to write specifically, with positionality, about the specific authors and actors in the situation.

Although I align with some of these trends and not others, I do not necessarily suggest these as any better or different from the papers in the corpus or the author’s interpretation. I only offer my perspective that might help elaborate the “voices” of the researchers in the field with some of the other intentionality behind the choices, beyond singular epistemological incoherence.

I do have a suggestion, however, for perhaps a different way of framing the interpretation/discussion and implications of the paper. The discussion (pp. 90–91) highlights “couched studies” (i.e., the idea of couching an analysis or discussion of a small portion of a dataset in a description of a large sample or number of participants) as evidence of epistemological unconsciousness (i.e., the pervading positivistic assumptions that can assert themselves within a researcher or across members of a researcher community). Regarding this critique, I again wonder about which elements we are identifying as problematic. In my work, I have often written papers on only one participant or a few participants, in line with the critical, narrative, postmodern, and interpretivist frameworks I invoked. But I rarely only recruit or interview one participant, and I frequently advise my mentees pursuing similar work against overly narrow recruitment. There are often pragmatic reasons for doing a wide breadth of data collection and then a small, focused analysis. If I have talked to many participants (for pragmatic reasons, empirical reasons, funding obligation reasons, etc.), and then I only analyze a few, it seems to me the most honest way of describing this in methods is to highlight and explain the winnowing process. It seems the authors view couched studies as a shift towards positivism, but I wonder if some of the authors reviewed feel, like me, that there are important pragmatic reasons or constraints for pursuing and describing such couched studies, that there is no particular downside to doing so, and that doing so may not indicate a problematic epistemological shift for the individual or community.

The authors reference the quality/rigor distinction to highlight rigor’s exclusive connotations (Riley, 2017) and to position quality as the more epistemologically inclusive approach. However, I would argue that quality and rigor are actually not very different words outside of the engineering education research methods discourse, quality meaning “the standard of something as measured against other things of a similar kind; the degree of excellence of something” (Oxford/Google) and rigor meaning “the quality of being extremely thorough, exhaustive, or accurate” (Oxford/Google). Both connote evaluation, both can become exclusive or inclusive depending on how they are used. And, when I think of developing quality criteria, even ones more epistemologically coherent with qualitative research (i.e., the Big 8), I think such an effort represents a blending of/adapting of positivistic ways of assessing and appearing to quantify something important but ultimately unquantifiable. The quality concepts are all vaguely positivistic attributions themselves—the sort of neutral-but-official sounding concepts we qualitative researchers create to help justify ourselves in a positivistic/quantifiable/generalizable world.¹ Something like “trustworthiness” sounds like it could contain measurable quantities that are verifiable by an outside reader, when in actuality it does not. For the record, I am not against such epistemological blends or quasi-quantifiable criteria—as a current journal assistant editor myself, I do have to have a guiding framework for evaluating the quality of journal papers, and I do want those criteria to be fair and coherent and agreed upon by the community. But I would argue that critical and postmodern frameworks would likely actually use an entirely different value system to evaluate effectiveness of the argument— perhaps the extent to which the work troubled existing hierarchies and preconceived categories, convinced others, built up a faction for the cause, and so on. The more general underlying quality judgment I make as a reader is: have I been convinced by the sum total of the argument. I also think of my own work as a pragmatic epistemological blend as well—creating a six-part framework with guiding questions for positionality (Secules et al., 2021) is perhaps not the only way to present this complex and contextual and personal subject - perhaps it risks looking overly simple and easy; but a framework based on a collaborative inquiry is a pragmatic and useful representation that the engineering education community has already understood in discussing methodology (Walther et al., 2017), so I (with my coauthors) utilized it. In short, I think many of us are more pragmatists than ideologues.

In the paper’s final recommendations for researchers and reviewers, the problem of epistemological coherence is placed front and center. The authors call for more consideration of epistemological intentionality and coherence from authors. In their reflective questions they explore context for the incoherence: “…is there a mismatch between your epistemological perspective and that of your audience?” (p. 94). To this question, I would say: there often is, right? Is conceiving of the epistemological mismatch between myself and my audience meant to deepen my resolve to have epistemological coherence? What if, for pragmatic reasons, that positivist-leaning audience (of reviewers, of the average engineering professor, of administrative leaders) is the audience I most want to convince? What if I think the critical qualitative postmodern researchers of the community already get my point, but the positivists among us need more convincing, a different format, a new kind of argument? The authors further motivate these reflections “so that you can design qualitative research projects that have the potential to have the greatest impact within engineering education and that are consistent with your epistemological preferences” (p. 94). Here is perhaps where I differ with the authors the most—I simply do not think that having epistemological alignment in research is necessary or sufficient for having the greatest impact.

For the contentious and biased review process alluded to in the opening, the authors present epistemological pluralism as one possible solution—respecting others’ epistemologies, recognizing what is valued by different communities of researchers (p. 94). While respect and plurality are always positive, I wonder how much purchase there is in the strategy of simply telling reviewers to value a different epistemology. I wonder if, among those who know of and can recognize positivism or postmodernism or critical theory in scholarly work, we are good at identifying the theoretical perspectives but in doing so, we also tend to create a hero out of our favored viewpoints and a villain out of the disfavored viewpoints. Perhaps I think simultaneously calling for epistemological coherence and epistemological pluralism is particularly difficult given this definition of epistemology would presume a genuine worldview on the way knowledge functions (and whether universal truth exists, etc.). It can prove genuinely difficult to hold a view of knowledge and simultaneously accept a different one. In light of the systematic devaluing of certain epistemological commitments that’s implied, what is the actual change process we will undertake to create this more inclusive epistemological landscape? In general, I see the authors arguing for epistemological coherence in terms of individual papers, while asking reviewers and the broader community to practice epistemological plurality and inclusivity. While their implications suggest a plausible resolution, I am not sure it is the only or most feasible option.

To be clear, I am not arguing for an inattention to epistemology or for a pure pragmatism that is atheoretical. I do think the nuances of philosophical difference pointed out by Kellam and Jennings above are important and do point to the epistemological pluralism of the field. But as an alternative to epistemological coherence, I would advocate for a pragmatic approach to researcher epistemology, where impact is prioritized over coherence. To help model this, I next overview to two related bodies of research—epistemological resources and methodological pragmatism.

Conceiving of a Researcher’s Epistemological Context and Resources

To motivate an alternative approach to epistemology, I want to highlight the literature on personal epistemology in terms of education (i.e., how a learner knows what they know and what their framework for knowledge is, Elby, 2009), which is distinct from, but with a similar scholarly lineage to, scholarship on research epistemology (how a researcher knows what they know and what the nature of knowledge is, Crotty, 1998). Among scholars of personal epistemology, there have been some key divergences. Some have seen epistemology as progressions like developmental progressions—we move from a simpler epistemology of trusting authority figures to a complex epistemology of reasoning for ourselves (e.g., Baxter Magolda, 1992) or as a multi-dimensional progression specific to domains like physics (Chi et al., 1981) or math (Muis, 2004). Others show epistemologies as sort of camps or tribes (e.g., Becher, 2001). Scholars have differentiated between expert and novice epistemologies or ontologies within a discipline such as physics. Within these approaches, a student’s epistemology is presumed to be stable and unitary at a given time, such that classification as novice is not overly contingent on context but indicates a level of epistemological or cognitive development (Chi et al., 1981). We could compare the naïve belief that the earth is flat and the sun rotates around it to the expert belief that the earth is round and rotates around the sun.

Scholarship on research epistemology takes a somewhat similar approach to categorizing, where categories such as positivist and postmodern are seen as stable and totalizing. They are also somewhat hierarchized like expert/novice distinctions, where for some (perhaps those of us who talk the most about epistemology) positivism, post-positivism, and objectivism are probably seen as less theoretically sophisticated, whereas for others there is likely still an implicit positivistic bias. That is, we may suggest epistemological pluralism, but individual camps likely have a hierarchy of preference for the arguments from specific epistemological standpoints, us critical/qualitative/postmodern researchers included.

An alternative approach to classification within the personal epistemology literature is an approach called “epistemology in pieces” or epistemological resources (Elby & Hammer, 2010; Smith et al., 1993). These scholars start from the premise and find supportive evidence for a more messy, temporary, and contextual coherence of epistemology, rather than as a unitary or totalizing framework to be categorized. Epistemological resources are the working pieces of knowledge that we build up to more complex situations. The idea is that each person reasons through smaller stable cognitive elements that are applied and reapplied in different contexts. All people work with similar baseline foundations of epistemological resources, but they are activated, utilized, and synthesized in different ways to produce different divergent frameworks of knowledge for specific people and contexts. For example, some children may come to a well-reasoned belief that the earth is flat based on the sound observation that the earth seems flat from our vantage point close up and that the sun appears to move; in contrast, others may accept the knowledge passed along by an authority figure that the earth is round. One of the benefits of considering epistemological resources is that it does not require us categorize or totalize people into one way of thinking or one camp. Instead, it allows us to look at idiosyncrasies and confluences between a myriad of resources that create certain epistemological outcomes or ways of knowing. Through thinking of epistemological resources, we can think about the continuity between novice and expert reasoning and the way context shifts the kind of response a student gives. The child with a flat earth belief and the child who accepts the canonical knowledge of the expert both have consistent worldviews. With more support and accommodation that recognizes the child’s productive resources, the child with the flat earth belief can be helped to develop more sophisticated physical reasoning that is more in alignment with expert views.

I think this epistemological resources approach resonates with some aspects of what Kellam and Jennings have done in their paper. They conduct their discourse analysis of papers to point towards the vocabulary or research design that implies one epistemology or another, and they suggest reasons for one constraint or another (p. 84). They are open to considering multiple and conflicting epistemologies represented in the same paper, within an individual, or across the community.

Still, I think there are further lessons from the personal epistemology literature to an investigation and consideration of research epistemology. Kellam and Jennings seem to characterize epistemology as a category or camp, and even as a hierarchy. Within that qualitative camp (indicating postmodern or critical epistemologies), the authors indicate a preference towards epistemological alignment, or unitary coherence. It seems there’s an underlying assumption that different epistemologies are incommensurate—that talking about generalizability (something very important to positivists) will never be commensurate with qualitative, critical, or postmodern research. It seems to imply a breakdown in communication and values across research paradigms that would seem ominous for collaborating across a research community and creating impact with the opposing camp. (NB: Jawitz and Case (2009) highlight the importance of clarifying one’s theoretical perspective to overcome this communicative challenge.) On the contrary, epistemological resources scholarship would suggest that any genuine epistemology may be too complex to expect to find as unitary and coherent, sitting neatly in a camp or category. We could pay attention to epistemology as underlying all research without presuming or reifying camps. We could notice the ways different contexts (e.g., audiences, venues, formats, goals) create different epistemological representations that in the abstract could appear to be inconsistent (or perhaps, a “novice postmodernist”), but in actuality may be reasonable, coherent, and even effective for the context at hand.

Epistemological resources scholarship could also trouble the causal model implied for researcher epistemology: that epistemological categories are driving or should drive researchers to certain methodological choices. When noting something that may be positivistic, like an apologetic tone, the authors note the possibility that reviewers are encouraging or even requiring that apologetic tone (i.e. the epistemological unconsciousness). If we admit the possibility that the performance of particular epistemology may be contextual, then perhaps there is no need to prioritize epistemology as the a priori cause of methodology. Perhaps methodology likewise does not cause epistemology, perhaps they often sit near each other in tension and can be shifted by and understood relative to each other in context.

What would we gain by considering researcher epistemology “in pieces” or in defining a researcher’s epistemological resources rather than highlighting categories? Rather than problematizing the inconsistent epistemologies and steering researchers’ epistemologies back into their camps, perhaps Kellam and Jennings’s findings beautifully showcase the pragmatic code switching of a heterogenous research community in conversation with one another?

Methodological Pragmatism

One prominent and parallel philosophical tradition that could offer guidance is pragmatism. Pragmatism could sometimes indicate a specific group of educational philosophers (Dewey, 1938), or, oppositely, could indicate an atheoretical avoidance of philosophy (Fairlamb, 1986). Instead I am looking at methodological pragmatism as a philosophical orientation similar to Colin Foster’s recent paper “Methodological Pragmatism in Educational Research: From Qualitative-Quantitative to Exploratory-Confirmatory Distinctions” (2023). Foster argues for an approach that does not require or presume that philosophical coherence of a specific camp (e.g., qualitative and quantitative) is always necessary or useful; instead, he highlights other ways of philosophically grounding the work without camps. Foster argues that distinctions between quantitative and qualitative work are overstated, and these scholars follow different rules and traditions but actually do similar and contiguous sorts of intellectual activities.

As Foster (2023) notes, we might presume that qualitative scholars use words and interpretation while quantitative scholars use numbers and statistics. But qualitative scholars use arguments about frequency and quantity—for example, qualitative authors are not afraid of writing that “most” (i.e., >50%) of their participants responded a certain way or that they identified four emergent themes across five participants. And quantitative authors must interpret the data, including via their own intuition, experience, and positionality—the numbers do not speak for themselves.² Similarly, we might presume qualitative researchers focus on trustworthiness and credibility while quantitative researchers think about generalizability and validity. In fact, similar to rigor and quality I think these terms are much more interrelated than distinct—as a qualitative researcher I don’t typically use the term generalizability or validity because I don’t want to mislead anyone on the paper’s “camp.” But I assure you I think about whether my n = 1 study is only ever going to be applicable to this one participant or if it has some broader significance and resonance (if I don’t think it generalizes at all, I pick a different n = 1 study), and I think about whether my work stands up to scrutiny as a valid, logical process and argument. I presume people in the quantitative research camps think about whether they trust their data, whether people should trust them as representing it correctly, and whether their findings are credible. They probably also don’t use those terms because they don’t want to signal the wrong “camp,” but I suggest the camps are more epistemologically contiguous than we let on.

Mixed methods traditions typically form an additional third camp and highlight specific ways to incorporate qualitative and quantitative data (exploratory, explanatory, sequential, etc.); the presence of mixed methods further show these are not incommensurate camps. Yet Foster (2023) suggests that more than mixed methods, an underlying appreciation for the continuities across all researchers in education would help the research community build more understanding and collaboration towards more useful outcomes. Foster highlights an alternative dimension of research, exploratory/confirmatory which could more usefully help guide scholars on how to pursue and position their work to make a contribution to the field. Exploratory research explores new possibilities; grounded theory, ethnography, descriptive statistics are all primarily exploratory. Confirmatory research tests a hypothesis or theory; hypothesis testing and thematic analysis or discourse analysis that is testing a strong a priori theory are primarily confirmatory. There are “mixed” middle grounds on this dimension as well, as some theoretically driven research is particularly exploratory towards expanding a phenomenon in a new way.

Considering the continuity afforded by methodological pragmatism, we could focus on translating our key activities into terms the other side can understand. I would like to “yes and” to Kellam and Jennings’s advice to reviewers not only to keep an open mind to the other camp (allow epistemological pluralism), but to proactively seek epistemological continuity with the other side and to presume it is possible. A person doesn’t (or shouldn’t) need a well-developed postmodern epistemology to appreciate the consequences of a postmodern manuscript that has presented its argument well enough. It should reveal new things about the world that upend another way of viewing the world. One shouldn’t need to hold a critical epistemology to be convinced by critical research; ideally, the research should convince open minded readers of their critique of power from more baseline principles and shared epistemological resources.

Ultimately, I think this means I would put more of the onus for the issues Kellam and Jennings identify back on the author and researcher who are embracing critical and postmodern epistemology, or any particular methodology or theoretical framework. Readers are not necessarily all in the same theoretical tradition or camp; they do not necessarily have the same heroes, key terms, or fundamental assumptions. How will you communicate and translate your findings so they matter, not only to adherents of your theory or epistemology, but to a wide swath of engineering education stakeholders? If we can hold our own feet to the fire and really work on communication and impact, the strength of the work that comes about will be undeniable.

Competing Interests

The author has no competing interests to declare.