Introduction
Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do. Once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, “and what is the use of a book,” thought Alice, “without pictures or conversations?”
Readers may recognize that the above excerpt is from the beginning of Alice’s Adventures in the Wonderland (Carroll, 1962). Nearly one century and a half later, Alice would be thrilled to see more and more writers include pictures in their texts which may refer to actual or fictional referents. The gradual proliferation of images that marks the “pictorial turn” manifests itself in part through the abundance of novels and short stories that rely extensively on the interaction and interdependence of verbal and visual codes for the production of meaning (Petit, 2004, pp. 9–10). Different terms are used to refer to this particular genre: for example, multimodal novels, multimodal printed literature (Gibbons, 2012), and explicitly multimodal novels (Nørgaard, 2019). Multimodality is conceptualized as “the coexistence of more than one semiotic mode within a given context” (Gibbons, 2012, p. 8); as the combination of diverse modes of expression (Bateman, 2014, p. 6); and where “different kinds of meaning-making are combined into an integrated, multimodal whole” (Jewitt et al., 2016, p. 2). Several formal features of multimodal novels have been put forward (Gibbons, 2012): the use of images, unusual textual layouts and page design, varied typography, use of color in both type and imagistic content, and meaning construction through the combined images, words and sequence, and interactivity, where readers get to choose how the plot develops (Green & Jenkins, 2014) (emphasis mine). In multimodal novels, distinct semiotic modes constantly interact in the production of textual meaning. Therefore, not one specific mode is privileged but rather narrative content, images, fonts, and page layout all have a role to play in meaning constructions (Gibbons, 2010). As a consequence, it is persuasively argued that multimodal printed literature is a highly sophisticated art form, both regarding its challenge for the author to be creative and also for readers to understand and to yield interpretation (Gibbons, 2010). Shklovsky’s (1965) concept of foregrounding is closely linked to the purpose of all art form, which is to make people perceive the world from new and different perspectives. Foregrounding refers to “a form of textual patterning which is motivated specifically for literary-aesthetic purposes, and typically involves either stylistic deviations or parallelism” (Simpson, 2004, p. 50). While parallelism refers to linguistic repetition, deviation refers to something that is different from what is expected. Deviation may come in different forms (Gregoriou, 2011): grammatical deviation, semantic deviation, lexical deviation, and phonological deviation. Given a surge in literature that experiments with those aforementioned formal features of multimodal novels, research on non-linguistic deviations in multimodal literature is timely.
Assumptions of theoretical work and analytic work show that the value of a combination of different modes of meaning is more than the information that we get from the modes when used alone or added together (Lemke, 1998). In other words, combinations are considered in terms of meaning multiplication, which gives us more than solely meaning addition. It is also suggested that multimodal novels accentuate the physical aspects of narrative experiences; hence they blur the ontological boundary of readers in the discourse-world and characters within the text-world, and provide readers the experience of “bi-stable oscillation,” that is, the process of switching between corporeal and conceptual layers (Gibbons, 2012, p. 208). Therefore, such “doubly deictic alignment” of the reader with one or more characters arguably evokes subjective resonances between readers and characters and leads to more active involvement of the reader in the fictional world (Gibbons, 2012, pp. 208–210). In addition, multimodal strategies used in multimodal novels are believed to have the effect of positioning the reader with the character, taking his/her perspective, and seeing what he/she sees. For instance, pictorial components used in Mark Haddon’s The Curious Incident of the Dog in the Night-Time are believed to assist in the process of “putting readers in the shoes of the character,” which is likely to foster readers’ narrative empathy with the protagonist of the novel (Tian, 2022).
Yet, this unique genre of multimodal literature has received only limited empirical attention. Only a few attempts (Caracciolo, 2014; Gibbons, 2012) have been made to investigate how the perceived multimodal feature is experienced by readers. It is argued that multimodality might have negative impacts upon “the nature of readers’ literary visualization and how they experience imagined worlds” (Gibbons, 2012, p. 195), since literary reading-induced imagery might be hindered in reading multimodal novels, as parts of the “blanks” in a literary text left by the author either consciously or subconsciously (Iser, 1974) have been filled by the images used in multimodal novels. In an exploratory study on Amazon.com customer reviews of two novels featuring narrators with developmental disorders (Caracciolo, 2014), it is noticed that the explicitly foregrounded multimodal feature of the novel seems to be linked to an unusual narrative perspective from narrators displaying unusual mind styles. In other words, the theme of the novel (unusual mind styles) seems to impel authors to try out innovative multimodal strategies. Some participants in Carrocciolo’s (2014) study reported that notwithstanding their familiarity with autistic individuals, they still felt the gap in their communication with their autistic children or siblings. Reading Mark Haddon’s The Curious Incident and relating to Christopher helped to narrow the gap and allowed readers to comprehend the unique mind styles of their children or siblings.
For better understanding the effects of multimodal novels and foregrounding theory, this paper takes multimodal novels containing two primary semiotic modes (i.e., texts and images) as a departure, and aims to extend the research on multimodal literature by empirically investigating the effects of multimodal foregrounding on readers’ while-reading experiences (e.g., reading-induced mental imagery, empathy, sympathy, resonance) and after-reading experience (e.g., deepened concepts of self and other perception) (Fialho, 2019; Fialho & Hoeken, in preparation).
Literary reading imagery is the first component that we will discuss. Drawing on Sadoski and Paivio’s (2001) dual coding theory, we propose that the ingenuity of multimodal printed literature is reflected in the dual coding of verbal and graphic stimuli to construct multiplied meaning, and in the powerful influence of imagery as a part of the comprehension process both in terms of the bottom-up and top-down processes. In the reading process, verbal and nonverbal stimuli prompt the reader to construct a particular mental world, entailing the dynamics between three distinct types of images: (a) the intra-pictorial relations among graphic images; (b) the interaction between verbal images (i.e., the indiscernible art form of written texts) and graphic images (i.e., the discernible art form of pictures or paintings); and (c) their further interactions with literary reading-induced imagery formed in readers’ mind as the reading process unfolds (Figure 1).
Figure 1
Dynamics Between Images in Reading Multimodal Literature.
Affective response to characters is the second component that we will discuss. Affective response encompasses emotions such as sympathy, empathy, and resonance with the characters in the story. When reading a story, we almost automatically take on the perspective of the main character, and switch to their time and place in the world of the story. Sympathy and empathy are closely related. It is generally acknowledged that sympathy refers to a feeling for the character and empathy refers to a feeling with the character (Keen, 2007). The difference between them can also be described in cognitive-stylistic terms. While sympathy involves a reader in the real world simply observing a character in a narrative world, empathy relies on a deeper “trans-world mapping” between the real-world reader and the narrative character (Stockwell, 2002, p. 93). It has been suggested that “the choice of perspective by the author from which the reader is to apprehend plot and characters guides the reader’s potential empathy and sympathy” (Sorlin, 2020, p. 8). Resonance is defined as recognizing aspects of themselves in real and/or fictional others and using personal reminding to resonate with the story (Cohen, 2001).
Self-modifying emotion is the third element that we will address. The tenet for how literary reading might change the reader is the focus of the Program of Transformative Reading (TR) (Fialho, 2018, 2019, in press). Four essential components of TR have been articulated: enactment imagery, sympathy, resonance, and self-other perceptual depth; and it has been proposed that readers’ while-reading experiences (i.e., enactment imagery, sympathy, resonance) may yield new and profound insights into after-reading effects. It has been indicated that reading deepens concepts of self and both fictional and real others through this transformative experience (Fialho & Hoeken, in preparation).
The reader’s character trait is also taken into consideration in the current study. When describing readers’ interpretations, Bateman et al. (2017) has argued that “details about a recipient’s contextual and discourse knowledge” should be taken into consideration in order to reach more sound inferences and meaning-making processes (p. 301). In the present study, readers’ trait empathy, fiction exposure, familiarity with autism, knowledge and perception regarding autism, familiarity with the novels, social desirability, and demographic information are taken as covariates and measured with corresponding scales.
Therefore, based on assumptions of the theoretical and analytical work of multimodal novels, and findings from limited and small-scale empirical work, this study hypothesizes the following:
H1: There would be an effect of the medium type. Participants reading multimodal novels would score significantly lower on scales measuring reading-induced imagery and higher on emotional engagements, such as sympathy, empathy, and resonance and self-other perceptual depths.
H2: There would be an effect of the theme of the novel. Participants reading novels centering on characters with an unusual mind style would score significantly higher on scales measuring emotional engagements, such as sympathy, empathy and resonance and self-other perceptual depths.
H3: There would be an interaction of medium type and the theme of the novel. Participants reading multimodal versions of novels featuring characters with unusual mind styles would score significantly different on scales measuring reading-induced imagery and emotional engagements, such as sympathy, empathy and resonance and self-other perceptual depths.
Methods
To test the above-mentioned hypotheses regarding the effect of multimodal features of multimodal novels and narrative perspectives, and their interactive effects on reading-induced imagery and emotional reactions in the process of reading, the current study applied a mixed-methods design, combining a between-subjects design (multimodal and monomodal conditions) with a within-subjects design (ratings after reading an excerpt featuring unusual mind style vs. ratings after reading an excerpt featuring usual mind style).
Participants
For the current study, participants were selected on the basis that they were adult readers (ages ranged from 18 to 60) with UK nationality and English as their first language. Initially, 90 participants were recruited through the online platform Prolific (https://prolific.ac/). Five participants failed to finish the experiment within the specified time (90 minutes) and were therefore excluded. Seven participants chose to withdraw from the experiment due to unknown reasons and were thus excluded. Four participants failed to answer four or more questions from the six comprehension check questions correctly and were excluded. Lastly, five participants completed the experiment exceptionally fast (one in three minutes, one in eleven minutes, and the other one in fifteen minutes). Therefore, these three participants were also excluded. This left 69 participants (Female = 48) for the analysis. Participants’ mean age was 31.33 years (SD = 10.41). Detailed descriptive statistics regarding the distribution of age and gender can be found in Table 1.
Table 1
Mean Age (SD), Age Range and Number of Participants (Females) per Condition.
| CONDITION | READING MATERIAL | MEAN AGE (SD) | RANGE | NUMBER OF PARTICIPANTS (FEMALES) |
|---|---|---|---|---|
| Experimental (Multimodal) | CI + TF | 28.9 (10.2) | 18–48 | 17 (11) |
| TF + CI | 36.4 (13.3) | 18–60 | 18 (12) | |
| Control (Monomodal) | CI+ TF | 29.5 (8.2) | 18–44 | 17 (13) |
| TF + CI | 30.5 (9.2) | 18–51 | 17 (12) |
[i] Notes. CI = An excerpt from The Curious Incident of the Dog in the Night-Time; TF = An excerpt from Tulip Fever.
All participants were asked to fill in the participant consent form to indicate their willingness to take part in the experiment and were debriefed about the purposes of the research when they had finished all the questionnaires. They received five pounds per hour for their participation.
Materials
The present study has selected two prototypical examples of multimodal novels: Deborah Moggach’s (1999) Tulip Fever and Mark Haddon’s (2004) The Curious Incident of the Dog in the Night-Time (hereafter The Curious Incident). The justification in support of the chosen novels is as follows. Firstly, these two novels see the employment of two primary semiotic modalities in the genre of multimodal printed literature (verbal and visual modalities), and they are typical examples that fall on each side of the spectrum of multimodal printed literature proposed by Gibbons (2012, p. 3). As illustrated in Figure 2 below, at one end of the spectrum, the pictorial content has an illustrative role to play in the narrative since it is “clearly and consistently partitioned off from the text” (Gibbons, 2012, p. 222). Moggach’s (1999) Tulip Fever is an example of this category—whereas at the other end of the spectrum, imagistic content is in its most central manifestation, in which texts are subordinate to images. The verbal and visual modalities on the page are not displayed in a separate way but interact and combine to form a “more general syntagm” (Barthes, 1977, p. 41). Haddon’s (2004) The Curious Incident is an example of this latter category.
Figure 2
The Spectrum of Multimodal Printed Literature.
Secondly, these two novels are of distinct literary themes. Tulip Fever (1999), set in the seventeenth-century Netherlands, is a love story and gets its name from “Tulip Mania,” a real event in Dutch history that seized the entire city of Amsterdam in the 1630s. In comparison, The Curious Incident (2004) features a first-person narrator with an unusual autistic mind style, who discovers the body of his neighbor’s poodle on the front lawn one evening and sets out to uncover the murderer. As Burke (2010) suggests, the literary theme has a significant role to play in prompting readers’ emotional engagements. Therefore, it is beneficial to select multimodal novels with distinct themes to explore the interaction between multimodality and literary themes. Thirdly, both novels share similar cultural and chronological backgrounds, as they are written by British authors and were published around the millennium turn. As suggested by Gibbons (2012: 3), the twenty-first century is the era of multimodality. Thus it is advantageous to select multimodal novels from the period.
Concerning the corresponding monomodal versions of these two novels, the Dial Press Trade Paperback edition of Tulip Fever (Moggach, 2005) was used. Since there was no printed monomodal edition of The Curious Incident available for the current study, transcription of the audible version of The Curious Incident (Haddon, 2003) published by Recorded Books and narrated by Jeff Woodman was selected. In terms of the endeavor involved in adaptations from the multimodal version to the transcripts used in the audible version, this study distinguished four strategies of tackling the nondiscursive elements, namely omission, verbalization, modification, and substitution. Omission refers to the case that all the pictograph elements and the accompanying verbal messages in the original version are entirely left out. Verbalization happens when the visual parts are mainly composed of texts, and as a result, are represented by verbal texts in the audible version. The narrator for the audible version simply reads out the textual part without further illustration of the context of its usage. Modification refers to the case that only pictograph elements are dropped out, whereas the co-texts (either before or after the pictograph part) underwent modifications to some extent, such as a replacement of the deixis with words of no indexical meaning. When the pictograph elements are of essential roles to the emplotment, they tend to be substituted with verbal expressions in the audible version, whereas devices indicating the subordinating image-text relations in the multimodal version are entirely left out. Therefore, substitution involves the most endeavors of inter-semiotic translation. This study took the first two ways of handling nonverbal messages (omission and verbalization) as a starting point by selecting excerpts in which only these two ways were present.
Excerpts selected were of nearly equivalent length, counting around 2500 words each (Table 2), which was amenable to the present experiment. Each excerpt took participants about 20 minutes to read, which left them with sufficient time to finish the questionnaires in around 60 minutes.
Measuring Instruments
After reading the stories, participants filled out a set of questionnaires that were taken into analysis as being the dependent measure, comprehension check, or covariate. They are described below.
Reading Measures
Reading-induced imagery was measured with the Mental Imagery subscale of the Story World Absorption Scale (SWAS; Kuijpers et al., 2014), and the Enactment-Imagery subscale of the Transformative Reading Scale (Fialho, 2018; Fialho & Hoeken, in preparation), respectively. These two subscales were used because they assess two different types of imagery. When speaking about imagery, we tend to think about visual imagery, but mental imagery can occur in all sensory domains. We can experience imagery of the setting of a story, but also of the characters. Kuzmičová (2014) distinguishes four types of imagery during reading (enactment, description, rehearsal, and speech imagery) and places them on a continuum. Among them, enactment imagery, a more elaborate measure of imagery, focusing on a “vivid and vicarious experience of the text”, in which readers may report feeling the physical experiences of the characters in their bodies, has been empirically proved to be present in the reading process (Fialho & Hoeken, in preparation). The Mental Imagery subscale (Kuijpers et al., 2014) consisted of three items (e.g., I have an image of the main character in mind. I can see the situations happening in the story being played out before their eyes. I can imagine what the world in which the story takes place look like.), and the Enactment Imagery subscale (Fialho, 2018; Fialho & Hoeken, in preparation) consisted of nine items (e.g., I felt my body responding to the text. I could hear the dialogues and/or voices as though I were listening to an actual conversation. I could almost feel the physical experiences of the character(s) in my body.).
Narrative empathy was measured with the Empathy subscale of the Literary Response Questionnaire (LRQ; Miall & Kuiken, 1995), consisting of 7 items (e.g., Sometimes I feel like I’ve almost become a character I’ve read about in fiction. I sometimes have imaginary dialogues with people in fiction. When I read fiction, I often think about myself as one of the people in the story.).
Narrative sympathy was measured with the Sympathy subscale of the Transformative Reading Scale (Fialho, 2018; Fialho & Hoeken, in preparation), consisting of 4 items (e.g., I felt sorry for the character(s). I felt compassion for the character(s). The character’s situation was of concern to me.).
Resonance was measured with the Resonance subscales of the Transformative Reading Scale (Fialho, 2018; Fialho & Hoeken, in preparation). The subscale consisted of 7 items (e.g., The character’s situation reminded me of situations I have lived in the past. The story mood brought me past memories. While reading the story, I often compared aspects of my life with aspects of the life of the characters.).
With regard to the after-reading experience, the Self-Other Perceptual Depth subscale of the Transformative Reading Scale (Fialho, 2018; Fialho & Hoeken, in preparation) was used. The Self-Other Perceptual Depth subscale consisted of 14 items (e.g., After reading this text, I felt sensitive to aspects of the life of others that I usually ignored. This text makes me (re-)consider how I will treat people that differ from myself in the future. I felt empowered, inspired, enlightened by this text.), measuring the extent to which someone experiences an increased sensitivity to matters that go beyond the text and that are usually ignored, a change in perception of self and others.
Participants rated each statement on a 7-point Likert scale (1 = strongly disagree, 7 = strongly agree.)
The comprehension check measure consisted of three multiple-choice questions per story (4 possible answers per question) that should have been possible to answer correctly for people who had read the stories with normal attention. Participants who answered two or more questions of the comprehension check incorrectly for one or both stories were excluded from the analysis.
Individual Differences Measures
Finally, participants were asked to report their reading habits, their familiarity with autism, their knowledge and perception regarding autism, their familiarity with the novels used in the current study, and their disposition to respond in a socially desirable manner.
Trait empathy was measured by The Interpersonal Reactivity Index (IRI) (Davis,1980), which includes altogether 28 items divided into four subscales: perspective-taking (e.g., Before criticizing somebody, I try to imagine how I would feel if I were in their place.), empathic concern (e.g., When I see someone being taken advantage of, I feel kind of protective toward them.), personal distress (e.g., When I see someone who badly needs help in an emergency, I go to pieces.), fantasy(e.g., When I am reading an interesting story or novel, I imagine how I would feel if the events in the story were happening to me.). IRI is considered to be the best measure of trait empathy developed to date (Baron-Cohen & Wheelwright, 2004: 166) because individual differences in empathy were found to correlate with IRI.
Since all participants for the current study are adult native English speakers with UK nationality, reading habits were assessed by familiarity with fiction measure: the UK version of the Author Recognition Test (UKART, Masterson & Hayes, 2007). The test contained a list of 80 authors, 40 of which were fake names. Participants were asked to mark only those authors they believe to be real without guessing. For each name the participants recognized, they earned a point, whereas, for each fake name, they got a penalty point.
Familiarity with autism was measured with the question ‘Have you had much experience with autism?’, and participants were instructed to choose only one from the following statements, adapted from Dillenburger et al. (2013: 1561) (e.g., Yes, I am autistic. Yes, my child(ren) has autism. Yes, I have siblings/relatives with an Autistic Spectrum Disorder. Yes, I have friend(s) with an Autistic Spectrum Disorder. Yes, I’ve worked with autistic people. Yes, I’ve educated autistic people. I Have read some books but not much real-life experience. Not at all.)
In addition, participants’ knowledge and perception regarding autism were tested with questions adapted from an online ‘World Autism Month Quiz’ (https://www.autismspeaks.org/wam/quiz). The quiz consisted of five multiple-choice questions (four possible answers per question) (e.g., If a person with autism doesn’t make eye contact when speaking with you, it means… a) They are ignoring you; b) They don’t want to talk to you; c) It might be uncomfortable for them; d) I don’t know.) The fourth option ‘I don’t know’ was added for all questions so as to control guessing.
To measure participants’ familiarity with the novels used in the current study, they were instructed to choose only one from the following statements: I have only read the novel before; I have only heard the audiobook before; I have both read and heard the novel before; I have never read the novel nor heard the audiobook before.
The Social Desirability Scale (Crowne & Marlowe, 1960) assessed the disposition of individuals to respond in a socially desirable manner. It was expected that the participants would score low on the SDS, therefore assuming that responses were as unbiased as possible and are not given to please researchers. Participants were also asked to provide demographic information pertaining to their age, gender, and education level.
Procedures
Under the multimodal and monomodal conditions (see Table 3), participants were instructed to read two excerpts from two novels with different themes consecutively. After each reading task, participants were presented with a set of measuring instruments (please see Section 2.3), followed by an open-ended question asking them to describe briefly in a few words or sentences about any of their thoughts, feelings, impressions or memories that were part of their reading experiences. Subsequently, participants completed trait measures, such as trait empathy and social desirability. In addition, they completed a UK version of the Author Recognition Test (henceforth UKART, Masterson & Hayes, 2007, Cf. Fialho & Hoeken, in preparation) and provided demographic information about themselves (i.e., their age, gender, and education level).
Table 3
Experiment Design.
| CONDITION | PROCEDURE | |||||
|---|---|---|---|---|---|---|
| Experimental (Multimodal) | Condition 1 | CI | RM1 | TF | RM2 | TM/Dem |
| Condition 2 | TF | RM1 | CI | RM2 | TM/Dem | |
| Control (Monomodal) | Condition 3 | CI | RM1 | TF | RM2 | TM/Dem |
| Condition 4 | TF | RM1 | CI | RM2 | TM/Dem | |
[i] Notes. CI = an excerpt from The Curious Incident of the Dog in the Night-Time;
TF = an excerpt from Tulip Fever;
RM1 = Reading measures after the first excerpt;
RM2 = Reading measures after the second excerpt;
TM = Trait measures (trait empathy, fiction exposure, familiarity with autism, familiarity with the reading materials);
Dem = Demographics (age, gender, education level).
Design
The first factor tested was the foregrounded multimodal feature of the novel, and it was manipulated at two levels: control (monomodal) condition and experimental (multimodal) condition. The experimental group was instructed to read two excerpts selected from two multimodal novels consecutively. After each reading task, participants were presented with a set of measuring instruments (please see Section 2.3) that were taken into analyses as being either the comprehension check or the dependent measure. The control group was instructed to read the corresponding excerpts selected from the pure verbal versions of these two novels used in the experimental condition.
The second factor tested was the foregrounded narrative feature of the novel, and it was manipulated at two levels: unusual mind style vs. usual mind style. In order to control for order effects, a counterbalanced design was adopted. Participants in the experimental (multimodal) group and the control (monomodal) group were randomly assigned to two groups, one group read the excerpt featuring unusual mind style (selected from The Curious Incident) first, and the other group read the excerpt with normal mind style (selected from Tulip Fever) first. The detailed experiment design is displayed in Table 3.
Results
The Effect of the Medium Type
To test the first hypothesis that the foregrounded multimodal feature would alter reading experiences, the General Linear Model (GLM) Multivariate Analysis of Variance (MANOVA) was conducted. Prior to running the MANOVA, potential outliers were checked, and the boxplot of variables showed that outliers identified were all legitimate outliers. The assumption of independence of observations was also checked and met. But the assumptions of multivariate normality and homogeneity of variance/covariances were violated. Therefore, the dependent variables were transformed by cubing the original values. After the cubic transformation, both the assumptions of multivariate normality and homogeneity of variance/covariances were met. Table 4 shows the means and standard deviations of the questionnaire scores per subscale and per group.
Table 4
Mean (SD) Scores on Mental Imagery, Narrative Empathy, and TR Scale per Condition.
| SUBSCALES | THE CURIOUS INCIDENT | TULIP FEVER | ||
|---|---|---|---|---|
| MULTIMODAL VERSION | MONOMODAL VERSION | MULTIMODAL VERSION | MONOMODAL VERSION | |
| SWAS_Mental_Imagery | 5.55 (1.17) | 5.39 (1.40) | 5.73 (1.09) | 5.25 (1.51) |
| LRQ_Narrative_Empathy | 4.07 (1.33) | 4.04 (1.43) | 4.07 (1.30) | 3.74 (1.30) |
| TR_Enactment_Imagery | 4.62 (1.08) | 4.69 (1.39) | 4.45 (1.24) | 4.37 (1.29) |
| TR_Sympathy | 5.75 (1.06) | 5.84 (0.87) | 4.59 (1.28) | 4.53 (1.24) |
| TR_Resonance | 4.14 (1.72) | 3.61 (1.64) | 3.66 (1.63) | 3.68 (1.68) |
| TR_Self_Other_Perception | 3.81 (1.38) | 3.75 (1.47) | 2.82 (1.31) | 2.69 (1.37) |
The analysis revealed no significant difference between scores for the experimental (multimodal) group and control (monomodal) group, and this held for both reading excerpts. Based on these results, the first hypothesis that the multimodal version of novels would alter while-reading experiences and promote transformative reading can be rejected.
The Effect of the Theme of the Novel
Regarding the second hypothesis, it was expected that there would be an effect of the theme of the novel; that is, within both multimodal and monomodal groups, participants would score reading measures significantly different for novels featuring characters with varying styles of mind. However, the results (showed in detail below) suggest that the second hypothesis can only be partially confirmed, regarding the reading measure of sympathy and self-other perception.
A mixed MANOVA was conducted within the multimodal group and monomodal group, respectively. Table 5 shows the mean scores and standard deviations of dependent variables for readers in the multimodal group and monomodal group per condition, respectively.
Table 5
Mean (SD) Scores on Mental Imagery, Narrative Empathy and TR Scale per Condition in the Multimodal and the Monomodal Conditions.
| MULTIMODAL CONDITION | ||||||
|---|---|---|---|---|---|---|
| UNUSUAL MIND STYLE (THE CURIOUS INCIDENT) | USUAL MIND STYLE (TULIP FEVER) | |||||
| TOTAL | FIRST | SECOND | TOTAL | FIRST | SECOND | |
| SWAS_Mental_Imagery | 5.55 (1.17) | 5.43 (1.41) | 5.67 (0.91) | 5.73 (1.09) | 5.98 (0.67) | 5.47 (1.38) |
| LRQ_Narrative_Empathy | 4.07 (1.33) | 4.27 (1.26) | 3.87 (1.41) | 4.07 (1.30) | 3.95 (1.37) | 4.19 (1.26) |
| TR_Enactment_Imagery | 4.62 (1.08) | 4.41 (1.05) | 4.82 (1.11) | 4.45 (1.24) | 4.48 (1.29) | 4.42 (1.23) |
| TR_Sympathy | 5.75 (1.06) | 5.84 (1.24) | 5.67 (0.89) | 4.59 (1.28) | 4.82 (1.08) | 4.34 (1.44) |
| TR_Resonance | 4.14 (1.72) | 4.10 (1.49) | 4.17 (1.96) | 3.66 (1.63) | 3.80 (1.81) | 3.51 (1.45) |
| TR_Self_Other_Perception | 3.81 (1.38) | 4.10 (1.41) | 3.54 (1.33) | 2.82 (1.31) | 2.55 (1.24) | 3.11 (1.36) |
| MONOMODAL CONDITION | ||||||
| UNUSUAL MIND STYLE (THE CURIOUS INCIDENT) | USUAL MIND STYLE (TULIP FEVER) | |||||
| TOTAL | FIRST | SECOND | TOTAL | FIRST | SECOND | |
| SWAS_Mental_Imagery | 5.39 (1.40) | 5.71(1.12) | 5.08 (1.60) | 5.25 (1.51) | 5.63 (1.20) | 4.88 (1.72) |
| LRQ_Narrative_Empathy | 4.04 (1.43) | 4.72 (1.04) | 3.36 (1.47) | 3.74 (1.30) | 3.71(1.11) | 3.78 (1.51) |
| TR_Enactment_Imagery | 4.69 (1.39) | 4.87 (1.43) | 4.50 (1.37) | 4.37 (1.29) | 4.61 (1.25) | 4.13 (1.31) |
| TR_Sympathy | 5.84 (0.87) | 6.00 (0.67) | 5.68 (1.03) | 4.53 (1.24) | 4.99 (1.26) | 4.07 (1.07) |
| TR_Resonance | 3.61 (1.64) | 3.95 (1.38) | 3.28 (1.85) | 3.68 (1.68) | 4.39 (1.63) | 2.98 (1.47) |
| TR_Self_Other_Perception | 3.75 (1.47) | 4.00 (1.06) | 3.50 (1.79) | 2.69 (1.37) | 2.85 (1.52) | 2.54 (1.23) |
[i] Note. Scores with significant differences appear in bold.
Within the multimodal group, the assumptions of linearity, multivariate normality and homogeneity of variance-covariance matrices were considered to be met for the current study because the sample sizes were nearly equal (N = 17, N = 18) (Leech, Barrett, & Morgan, 2015, p. 234). Significant multivariate effects were found for the main effect of foregrounded narrative feature (i.e., unusual vs. usual mind styles), F (6, 28) = 9.88, p < .001, multivariate η2 = .68, but not for the order effect (F (6, 28) = 1.42, p = .24) nor the interaction between narrative feature and order (F (6, 28) = 1. 62, p = .18). Therefore, there was not an overall order effect. Follow-up ANOVAs revealed that the difference was significant for the Sympathy and the Self-Other-Perceptual Depth subscales of the TR scale, with F (1, 33) = 23.12, p < .001 and F (1, 33) = 19.62, p < .001, respectively. Results of the pairwise comparison (See Table 5) suggested that participants scored significantly higher in Sympathy and Self-Other-Perceptual Depth subscales for the excerpt featuring an unusual mind style (i.e., The Curious Incident). Therefore, the excerpt centering on characters with unusual mind styles elicited more sympathy and deeper self-other Perceptual in comparison with the one with usual mind styles, regardless of the order in which they were presented to the participants.
With regard to the monomodal condition, the sample sizes were equal across the groups (N = 17, N = 17). MANOVA is robust against assumptions of linearity, multivariate normality, and homogeneity of variance-covariance matrices when the sample sizes are equal (Leech et al., 2015, p. 233). Significant multivariate effects were found for the main effect of foregrounded narrative feature (i.e., unusual vs. usual mind styles), F (6, 27) = 9.23, p < .001, multivariate η2 = .67, and also for the interaction between narrative feature and order (F (6, 27) = 3.34, p = .014, multivariate η2 = .43), suggesting an overall order effect. Follow-up ANOVAs revealed that the difference was significant for the main effect of narrative feature regarding the Self-Other-Perceptual Depth subscale of the TR scale, F (1, 32) = 22.85, p < .001, but not for the effect of order (F (1, 32) = .047, p = .83) nor the interaction between narrative feature and order (F (1, 32) = 3.36, p = .08). As shown in the means (See Table 5), the results of the pairwise comparison suggested that within the monomodal group, participants scored significantly higher for the excerpt featuring an unusual mind style, and statistics revealed that the order effect was not significant. As for the measure of Sympathy, the results revealed that the difference was significant for the main effect of narrative feature, F (1, 32) = 28.88, p < .001, multivariate η2 = .47, and for the interaction between narrative feature and order as well, F (1, 32) = 6.43, p = .02, multivariate η2 =. 12). As shown in the means (See Table 5) and the interaction plot (See Figure 3), readers in the monomodal group scored higher after reading the excerpt from The Curious Incident that features a protagonist with an unusual mind style. Besides, the effect was bigger for the group who read The Curious Incident first, as indicated by the steeper slope of the line representing that group of participants in Figure 3.
Figure 3
Interaction Graph of Theme and Group for Sympathy.
Regarding the results for the rest of the dependent variables, it was revealed that the difference was only significant for the interaction between narrative feature and order, which indicated a significant order effect for the measure of Mental Imagery (F (1, 32) = 5.71, p = .02), Enactment-Imagery (F (1, 32) = 11.47, p = .002), Empathy (F (1, 32) = 11.47, p = .002) and Resonance (F (1, 32) = 12.11, p = .001). The means and standard deviations shown in Table 5 and the interaction graphs shown in Figures 4, 5, 6, 7 illustrate the meaning of this result. Participants rated higher for the excerpt that was presented to them first regardless of the theme of that excerpt, which indicated that there was a significant order effect for the measures of Mental Imagery, Enactment-Imagery, Empathy, and Resonance.
Figure 4
Interaction Graph of Narrative Feature and Order for Mental Imagery Subscale.
Figure 5
Interaction Graph of Narrative Feature and Order for Enactment-Imagery Subscale.
Figure 6
Interaction Graph of Narrative Feature and Order for Empathy Subscale.
Figure 7
Interaction Graph of Narrative Feature and Order for Resonance Subscale.
The Interactive Effect of Medium Type and the Theme of the Novel
To assess whether there was an interactive effect between the foregrounded multimodal and narrative features on readers’ experience of mental imagery, enactment-imagery, sympathy, empathy, resonance, and self-other perceptual depth, a mixed MANOVA was conducted. The analysis revealed no significant effect of the interaction between foregrounded multimodal feature and narrative feature, Pillais’ Trace = .089, F (6, 62) = 1.00, p =.431, multivariate η2 = .089. The results disconfirmed that differences between unusual and usual mind styles were traceable to multimodal vs. monomodal versions of these selected narratives.
Discussion
In this study, we investigated how the multimodal features of multimodal novels influence reading-induced imagery and shape our emotional reactions in the process of reading, and whether differences between multimodal and monomodal versions of selected narratives were related to unusual narrative perspectives from narrators with unusual mind styles.
This study draws critical attention to the study of multimodal printed literature as a field of academic research in its own right, which is a more or less hitherto neglected genre. In comparison with studies focusing on other multimodal genres, such as the persuading discourse like advertisements and the multimodal informing discourse like school textbooks, multimodal narrating discourse has been placed out of the limelight (Gibbons, 2008, p. 107). As Gibbons (2010) argues, multimodal printed literature is a highly sophisticated art form, regarding both the challenge for the author to be creative and for readers to understand and interpret.
Quantitative results from the statistical analysis of the current study did not confirm the effect of multimodality on our reading-induced imagery and emotional reactions in the process of reading multimodal novels. The results also disconfirmed that differences between unusual and usual mind styles were traceable to multimodal vs. monomodal versions of these selected narratives. Prior empirical studies (e.g., Mak et al., 2020) have shown positive effects of pre-reading imagery instructions on higher imagery that could subsequently lead to higher absorption. It raises the intriguing possibility that specific pre-reading instruction could be included in future research to test if reading-induced imagery could be manipulated and induce participants to pay sufficient attention to the pictorial elements in the multimodal version.
In terms of the covariates that were included (i.e., readers’ trait empathy, reading habits, familiarity with autism and the novel, social desirability), none were found to moderate effects on dependent variables in the current study. But participants’ comments suggested that their own personal experiences had a role to play in the reading process; thus they could relate the character or the setting of the novel. Prior studies have reported the effect of gender on reader evaluation of excerpts from novels (e.g., Bortolussi et al., 2010), the mediating influence of participants’ fiction exposure on the while-reading experience (Dixon et al., 1993; Kuijpers & Hakemulder, 2018), and the Fantasy subscale from the Interpersonal Reactivity Index (IRI) on imaginal vividness (Mak et al., 2020), more research is needed to investigate why those effects mentioned above are not always present.
The most notable statistically significant finding in the current study was that Sympathy and Self-other Perceptual Depth of the Transformative Reading Scale appeared to be positively associated with the foregrounded unusual narrative perspective. It was found that the excerpt centering on characters with unusual mind styles elicited more sympathy and deeper self-other perception in comparison with the one with usual mind styles. According to Sklovsky’s systematization of foregrounding, “Deviation may come in different forms, for instance in flouting norms and rules of language use (such as Dylan Thomas’s A Grief Ago), but also in an unusual narrative perspective (as in Tolstoy’s Kholstomer, a story told by a horse), or in turning social conventions upside down (illustrated well by Thomas More’s Utopia)” (Van Peer et al., 2021, p. 151). This article confirms Sklovsky’s systematization of foregrounding and shows that an unusual narrative perspective contributes to transformative reading. This result also confirms one type of transformative reading experience through sympathy, called “protagonist-centered self-transformation” (Fialho, in press). Sympathy involves “recognitions” of the suffering of a person’s situation and “judgement” of the nature and extent of that suffering (Sklar, 2013, pp. 55–56). Sympathetic feelings possess ethical implications beyond the experience of reading. These feelings can be transferred from the fictional world into readers’ lives and help readers to (re-)consider how they will treat people that differ from themselves in the future (Fialho, in press). Furthermore, this transfer depends upon a great variety of factors (Sklar, 2013). The unusual narrative perspective from narrators with unusual mind styles is arguably one of those textual features contributing to the “overall persuasive effects of works of fiction that aim to evoke readers’ sympathy” (Sklar, 2013, p. 48). The findings also corroborate Margolin’s (2003) argument that the fictional presentation of cognitive mechanisms’ breakdown or failure (i.e., unusual narrative perspective in this study) is itself a powerful cognitive tool to promote readers’ self-reflections on their own mental functioning (p. 278). Take Mark Haddon’s The Curious Incident as an example, the novel is well received by specialists in cognitive disorders who hold that “The book brilliantly portrays the mind and behavior of a person with autism without being sentimental or condescending” (Semino, 2014). Christian Keysers (2011), a world-leading neuroscientist on the study of empathy, refers to The Curious Incident in his seminal work The Empathic Brain as “a literary introduction to autism” (p. 161). And more importantly, the novel is praised in similar terms for providing the general public a realistic and moving representation of the workings of an autistic mind, as shown in the following comment posted on an online Question and Answer session with the author on the website of the UK Guardian newspaper:
Mark, I read your book last week and was so moved I can barely find the words. My 19-year old son has Asperger’s Syndrome, although he is more socially functional than Christopher. I believe you have done more to advance understanding of this form of autism than all the textbooks and professional journals are ever written. I have bought a second copy of the book to lend to anyone who asks, “What exactly is wrong with him?” That’s the first answer they need: There’s nothing “wrong” – he just sees the world differently, and you’ve made that very clear. (as cited in Semino, 2014, p. 280)
When Mark Haddon was interviewed about the main theme of his novel, he replied, “[I]t’s a novel about difference, about being an outsider, about seeing the world in a surprising and revealing way. It’s as much a novel about us as it is about Christopher.”
Overall, the results provided a negative answer to the question, “Is a picture worth a thousand words?” Nevertheless, the results demonstrated a contrasting effect of stories that have unusual rather than usual mind styles. Unusual narrative perspective is found to be a stronger driver of reading experiences than multimodal features. It is the content rather than the form that is an optimal device to promote readers’ while-reading and after-reading experiences.
Data Availability Statement
The complete dataset is available in the Zenodo repository: https://doi.org/10.5281/zenodo.8266385.
Acknowledgements
We would like to thank Dr. Patricia Canning and Dr. Nina Nørgaard for their very constructive feedback at the critical stage of preparation for the earlier versions of the paper. Any faults, however, remain our own responsibility.
Funding Information
The second author’s research for this article was supported by the project ‘Uses of Literary Narrative Fiction in Social Contexts’, funded by the Dutch Organization for Scientific Research (NWO; grant number 360-30-240).
Competing Interests
The authors have no competing interests to declare.