Contributor: Astrid Giugni & Jessica Hines
Affiliation: Duke University & Birmingham-Southern College
Email: email@example.com, jnhines at bsc.edu
Released: 4 February 2020
Published: Spring 2020 (Issue 24.2)
The increasingly visual and multimodal structure of communication invites new modes of engagement and knowledge formation from students, asking them to master what Gunther Kress (2005) identifies as complex and novel forms of reading that move with dexterity between text and image. While Deana McDonagh, Nan Goggion, and Joseph Squier (2005) have offered a useful corrective to his account, suggesting that images have more in common with text than Kress allows, Kress’s point is still an important articulation of the problem. As he argues, “The new constellation of image and screen—where screen, the contemporary canvas, is dominated by the logic of image—means that the practices of reading becoming dominant are the practices derived from the engagement with image and/or depiction in which the reader designs the meaning from materials made available on the screen” (18). Such evolutions in visual reading practices demand, as Frank Serafini (2010; 2014) contends, an increase in instructional practices that engage students in analyzing visual images and their multimodal contexts.
For us, Kress’s assessment and Serafini’s challenge provoke new questions about teaching data science and multimodal, visual literacy interdisciplinarily. As Michael Buckland (2017) argues, the most immediate challenge facing the information sciences concerns the rhetorical complexity of information. As our society grows more multimodal, so does our data, and data scientists thus must learn how to integrate quantitative interpretive methods with qualitative rhetorical analysis in order to understand complex information. Kress and Serafini thus prompt us to consider the ways that instruction in multimodal, visually rich forms of reading can be deployed to train students in reading complex data sets.
This webtext will describe a project we developed to bridge humanities- and data science-based analytical approaches through instruction in multimodal, visual reading practices. This project engaged students in reading photojournalism of the Syrian Refugee Crisis in order to analyze what Julianne H. Newton (2009) has called the “intention” behind photojournalistic publications (234)—that is, to read the political narratives within photojournalism through interpretation of the images and their paratexts. This process requires building student’s multimodal visual literacy so that they can read the symbolic and political register embedded in photographs in order to see that an often-reproduced image of a Syrian child is not just an image of a child, but that the choices of photographer, author, and editor have intentions to stir particular political and affective responses.
Building students' multimodal literacy allows them not only to be better readers of photographic intention, it also enables them to better understand their role in the process of forming that intention. As Sarah Warren-Riley and Elise Verzosa Hurley (2017) argue, students in their everyday lives engage in acts of digital media advocacy—understood as an activity that is intrinsic to online cultures of “sharing.” They often, however, lack the critical tools to conceptualize their composition, re-composition, and sharing of mundane texts as a form of advocacy. Our project began by considering the moment in which the picture of the body of the three-year old Syrian refugee Aylan Kurdi was quickly and widely shared across media in September 2015 and, in the process, reshaped Western response to the Crisis, including the language and images used to represent refugees (Vis. F., & Goriunova 2015, especially p. 11; UNHCR 2017). We began this way in order to encourage our students to critically interrogate— and then reshape—their own participation in digital advocacy through their daily encounters with popular images. By learning to read these images and their contexts, students learned how photographs are ideologically laden, what we have been calling their intention, and how the daily act of sharing and engaging with such images participates in ideology formation, what we have been labelling advocacy.
Informed by this theoretical and practical framework, students were tasked with creating their own multimodal visualization showcasing their findings. With this assignment, we created a structured space for students to engage with humanistic and computational approaches to research. At the same time, by asking them to produce a visualization of their results for their peers, we encouraged them to evaluate critically the effectiveness of multimodal digital platforms in communicating their research and findings—to think as both a designer (Arola, Sheppard, and Ball, 2014; Arola, 2018) and an advocate (Warren-Riley and Verzosa Hurley, 2017; Agboka and Matveeva, 2018). Rather than, as J. Elizabeth Clark (2010) describes, using “technology as convenient serendipity” (28), our project centered around collaborative digital and multimodal authorship as much as collaborative coding and statistical analysis.
This compositional and authorial process gave students hands-on experience in how design affects perception, and in turn, interpretation, of images and text (Arola et al., 2014; Serafini, 2010). In doing so, our aim was to train students to navigate the increasingly complex world of multimodal data sets by asking them to test how a qualitative question—what images evoke compassion—can be approached through quantitative methods.1
The question for our students: how can we track and analyze the political and affective impact of photojournalism on the Syrian Refugee Crisis?
We developed this project in 2017 for Data+, a ten-week intensive summer research program designed to engage undergraduate students from a range of fields in interdisciplinary data and information sciences. Unlike a traditional classroom environment, students participating in Data+ join a small team of three students and tackle a specific project by working within a cluster of teams that rely on similar computing and statistical skills and methodologies. While the students are closely mentored by a graduate student and a faculty sponsor and are required to attend weekly workshops, they are expected to work collaboratively with their peers within and across teams in shared spaces. Because of this unique format, Data+ asks students to already have a background in coding and statistics or to be willing to acquire these skills independently during the summer. The peer-collaborative aspect of the program means that participating students learn to break down larger—and often not fully defined—tasks set by the faculty into smaller, daily “assignments” on their own.
To guide this process, at the beginning of the summer, we provided the students with a preliminary set of goals. For the first two weeks, we asked them to develop a first procedure of how to efficiently target their search of the database. During the same period, we guided them through a first round of theoretical readings on the history of compassion and visual representations of suffering (see the “Pedagogical Goals” section below). For weeks three and four, we asked the students to determine what quantitative data to track (possible examples included: distribution of representations of women, of children, and of different ethnic groups), and to begin focusing on possible forms of visualizations (see the “Selecting Tools for Visualization” section below). Weeks five through eight were set aside for computational analysis, resulting in the creation of a visualization and presentation of their work during the last two weeks of the program.
Our group of students was comprised of rising sophomores majoring in STEM, social science, and humanities fields. At the heart of our project was an analysis of the relationship among affect, politics, and public debate in photojournalism of the Syrian Refugee Crisis. We asked our students to consider how news outlets use particular images of the Crisis to generate compassionate responses from their reader for political purposes. We presented this question to the students as a problem of computational data and image analysis, but our undergirding principle was to introduce the students to the idea that, in their quantitative analysis, they were working to understand the complex interplay of cognitive analysis and affective response in shaping political responses to the Crisis.
As Newton (2009) cautions, the advent of digital photography has accelerated the spread of the view that photojournalism is a mechanized, objective representation of news events rather than a carefully constructed and edited narrative: “The view that anyone can be a journalist, or a photojournalist, speaks to the underlying assumptions of objectivity, neutral science and a mechanistic world view: that there is a fact-based truth to be discovered and told or shown, and that using a tool, such as a camera, or a method, such as journalism or social science, can achieve such discovery and ultimately capture fact” (237). Our goal was to alert students to the acts of composition and editorial choice, which extend from the moment the photographer trains their lens on a subject to the final arrangement on the page, implicit in any photojournalistic product. That is, to make evident the human and institutional intentions and frameworks underpinning photojournalistic compositions. As Michael Griffin has documented in his analysis of photographic coverage of the “War on Terrorism” in American news magazines, these images “prompt and reinforce those versions of events that have already been established in public discourse and entrenched in media institutions by powerful social interests” (2004, p. 399). By asking students to track, analyze, and in turn narrate how images are selected for publication—and in which types of publications they appear—this project makes evident the process of ideology construction. This construction is visible only in the interplay of how each published image participates in the aggregate discourse formed by the totality of photojournalism of the Crisis.
Students were tasked with creating their dataset from AP Images. This is the world’s largest collection of historical and contemporary photography, cataloguing images from The Associated Press and a selection of global content partners. Each image is accompanied by searchable metadata such as photographer information, date, location, and description of photographic subject. Using this database, students were asked to: 1) develop a methodology for digitally tracking image use and distribution; 2) analyze their collected data looking for visual and linguistic trends and patterns in representations of refugees; and 3) create a digital visualization representing their findings.
The first two weeks of the program were dedicated to orienting the students with the content, layout, and functions of the AP database. Since our students had little prior experience with visual studies, we also introduced them to some of the central rhetorical and theoretical concerns of analyzing visually dense data. Our particular group of students had experience working with numerical data sets, but not with a database that contained complex images. To help them learn how to handle this type of material, we discussed with them what proved to be an ongoing concern: how do you combine data analytic methods with qualitative analysis?
This question first presented itself in the concrete problem of establishing the parameters of their searches of the database. There were over 5,000 images of the Syrian Refugee Crisis from which the students could gather data, but what data did they want to track and what data would give them the information they needed about the interplay of compassion, politics, and photojournalism?
During the remaining time in the program, the students had three major deadlines:
- In the third week, give a preliminary presentation to an audience of eighty peers, detailing their goals for the project and their proposed approach.
- In the ninth week, give a final presentation to us and their peers explaining their results and describing avenues for future research.
- In the final week, create and present their results at an open poster session attended by students and faculty from the university at large.
In determining their dataset, the students were asked to consider the politics of publication of refugee images. Do outlets with strong political commitments discernibly publish images that reflect their political affiliation? This question encouraged students to consider a variety of methodologies for categorizing the political leanings of news outlets. They decided to use a study from Amy Mitchell, Jeffrey Gottfried, Jocelyn Kiley, & Katerina Eva Matsa with the Pew Research Center, “Political Polarization and Media Habits” (2014), to narrow the potential news outlets and to adopt a consistent classification parameter.
In turn, this categorization process led to a larger conversation and critical insights into how visual markers in the images themselves, such as gender, age, clothing, etc., had potential political ramifications. While the selection of a particular feature of an image for publication—indoor or outdoor; group or individual—may at first have appeared to students as “merely” aesthetic, it soon became clear to them that these choices narrowed or expanded the view of the Crisis for the consumer of news media. In turn, their own acts of selection in visualizing the data they analyzed took on larger stakes, making them aware that their own visualization and editorial practices were forms of advocacy.
Student reflection was a critical component of our project. By design students were largely self-directed as they refined the questions and the boundaries of the project. This made it essential that they critically consider why they made certain rhetorical and design decisions. To that end, throughout the 10-weeks, we engaged students in short reflection assignments. During the first two weeks, we met extensively with the students both as a group and in one-on-one consultations asking them to articulate their challenges. Ahead of their third week presentation, they were then tasked with completing a short writing assignment in which they answered the following: 1.) What are the challenges of this project?; 2.) What are its limitations of this project; 3.) How do those limitations shape your understanding of the goals and desired outcomes of the project? At critical junctures in the following weeks, most importantly, the moment in which they selected tools for visualizing their findings, we asked students to reflect on their compositional choices. They were tasked with submitting proposals for visualizing their findings, and the proposals needed to include a short reflection in which they articulated why they chose a particular digital visualization tool and how they intended it to shape the final product. At the end of the summer, students completed a short reflection assignment in which they were asked to consider and then articulate their processes of decision making throughout the project, as well as how they thought those decisions affected the final results. Our goal in engaging students in active reflection throughout was for them to see how the choices that they made shaped the project’s outcome—a process that in turn allowed them to better reflect on how the rhetorical choices news outlets shaped the social and political narrative of the syrian refugee crisis.
Figure 1: Screenshot from the students’ executive summary where they used two visualization methods, interactive maps and a timeline diagram, to show geospacial and chronological distribution of the photographic images of the Crisis. These visualizations reflected their findings that Western European countries often depicted political protests and refugees in large groups, while Middle Eastern countries focused on images of individuals and refugee camps. See the section “Selecting Tools for Visualization” below for further details on the interactive maps.
The students’ research produced three major findings. First, students investigated the relationship between group and individual portraiture. Their exploration of whether or not right-leaning outlets are more likely to publish photographs of large groups of refugees led to one of the biggest discoveries of the project—that American and European news outlets (regardless of political leanings) are more likely to focus on groups of refugees than those based in the Middle East. Our students drew on their analysis of captions, sentiment analysis, and visual composition to preliminarily argue that Middle Eastern outlets worked to build the humanity of refugees via individual portraiture, whereas non-Middle Eastern outlets obscured humanity via group photography.
Second, in order to better understand the link between images and descriptions in media, the group created word clouds for each news outlet selected in the Pew Research Center’s study. The word clouds allowed them to notice patterns of linguistic frequency around descriptive language, patterns related to the political leanings of the outlets. In particular, our students were interested in different emphases on words that had legal ramifications for the immigration systems—such as “refugee” vs. “migrant” (UNHCR 2017). They noted that news outlets that were identified as left leaning by the Pew Research Study were more likely to describe Syrian persons as refugees rather than migrants—thus marking Syrian refugees as having important legal rights and protections. As illustrated in the image below, the group turned the word clouds into “word Europes” so as to clearly show to their peers in the program how each news agency’s language shaped the representation of the Crisis to their audiences.
Figures 2 and 3: Screenshot from the students’ final presentation visualizing the language used to report on the Crisis by the Guardian Newspaper (left) and Fox News Website (right). The students used the word clouds to make evident to their audience how different outlets described the refugees.
Finally, students also observed that the number of refugees in a particular country was not proportional to the number of images published from that country. As they show in their executive summary (screenshot below), while Turkey has the largest refugee population, it ranks third in terms of countries with most published images. Greece has the highest number of published photos despite housing only a fraction of the total refugee population. This raised key questions for our students about the correlation between the photographic country of origin and publishing desirability. While noting that limited access to war zones plays an important role in photojournalism of the crisis, their findings also led them to ask: what might be the political effects of sourcing images from European countries rather than those explicitly in the Middle East? Might this in fact contribute to the tendency to favor images of groups of refugees or inflect the language used to describe the refugees?
Figure 4: A slide from their executive summary in which students detail what is found in their data visualizations, list the number of AP images used by their studied news outlets, and include their final observation that the number of refugees is not proportional to the published image distribution. The image explains that the project culminated in a temporary website entitled visualizingsuffering.com.
The goal of the visualization was mirrored in the kind of qualitative analysis we asked the students to undertake. Many of the students in our group had been trained in statistical methods. This allowed them to analyze the quantifiable elements in the project with ease. The sticking point for them was how to handle the most crucial information in the database—the affective impact of the images themselves.
We thus began the discussion of the refugee crisis by asking them to consider the 2015 change in refugee policy initiated by German Chancellor Angela Merkel. We introduced Merkel’s decision as both a passionate response to the crisis and a display of an appropriate affective stance. Merkel’s response thus might have been an affective response of compassion, but it was also a political maneuver that positioned her and Germany as socially and politically progressive—just in time for German elections.2 The goal of this example was to help students develop a more sophisticated and supple understanding of the role that emotions play in politics and visual rhetoric. Quantitative methods helped us to approach the problem of analyzing the function of affect in political deliberation: What images played a crucial role in responses to the crisis? How do we interpret and describe the use of such images by journalists? And, once we are alert to how our engagement with images can be a form of advocacy, how can images participate in the formation of ideology?
The readings we selected (listed below) address visual literacy by considering these questions. Our task was to move our students from their initial question—how do we objectively analyze the emotional response to an image?—to an understanding of how photojournalism works within and supports complex political and social narratives.
To that end, students read reports from the United Nations High Commissioner for Refugees (2017) and collections of articles on photojournalism and Syria, the most helpful of which being the report by Vis and Goriunova (2015) examining social media response to the death of Alan Kurdi. In addition students read a small selection of theoretical texts designed to increase their visual literacy including:
- John Berger, “Appearances"
- Susan Sontag, Regarding the Pain of Others
- James Elkins, Pictures and Tears: A History of People Who Have Cried in Front of Paintings
- Martha Nussbaum, “Compassion: The Basic Social Emotion”
Engaging this range of readings accomplished two goals. First, it allowed students to see their research as part of a larger scholarly conversation: just as Susan Sontag examines representations of pain in war photography, so too were they analyzing representations of suffering in refugee photography. Second, it helped students to reflect on and refine their quantitative analysis. For every piece of assigned reading, we met with the full group of students and engaged in an intensive discussion about the theoretical framework of each piece, as well as the potential practical implications for how they approached ourwork. With this space for collaborative group reflection and discussion, students quickly recognized that individual responses to photography are subjective and often unpredictable. That didn’t mean, however, that they had no lens through which they could understand subjective response. For example, as one student pointed out, in studying the relationship between photograph and caption, they could learn how editorial practices hoped to guide perceptions of an image.
Alongside these conceptual goals, our project has been part of an ongoing effort within Data+ to better prepare STEM students to reflect critically upon and, in turn, communicate the limitations and advantages of computational methods. In particular, by asking students to reflect on how emotions are continually imbricated in decision-making processes, we aimed to equip them with a better understanding of the subjectivity of all data—even that which seems objective. This in turn prompted a conversation about how students can responsibly present their own research, acknowledging and thinking with the subjectivity and biases inherent in preparing, analyzing, and visualizing large data sets.
Figure 6: Detail from students’ final presentation. The students compared the distribution of refugees with the origin country of the most published images. Alongside our readings, this geographical line of inquiry guided the students' choice of visualizations. The results from the students’ analysis can be found at https://bigdata.duke.edu/projects/visualizing-suffering-tracking-photojournalism-and-syrian-refugee-crisis.
As data analysis becomes more computationally sophisticated and more ambitious in its scope, the need to integrate humanistic modes of analysis into the information sciences has become essential. The ability to work skillfully with complex and heterogeneous data is critically in demand from our students as a professional skill and as a way to navigate an information rich society—an ever pressing concern for both students and institutions as Alexander Kafka’s (2018) article on UC-Berkeley’s newly created data-science division makes evident. Moreover, the problem of correctly evaluating large data sets drawn from vast numbers of multimodal documents is politically and socially urgent. It is now evident that, despite ever-growing computational capacity, even the apparently simple task of automatically retrieving relevant information from a catalogue can still prove daunting from both a theoretical as well as practical standpoint (Buckland, 2017, especially chapter 8). The fundamental problem is not just technological but conceptual; large scale analysis of multimodal data is traditionally at odds with nuanced interpretive practices of reading the unique relationships between texts, paratexts, and images.3 This can be seen, for instance, in Facebook’s failed 2016 attempt to curate news-posts via algorithm after accusations of bias. The move to automation mired Facebook in controversy as the algorithm favored demonstrably false news.4 Factually incorrect news items, however, are only part of the problem. In order to prevent disinformation, algorithms would need to identify correctly everything from parodies to misleading and manipulated content—to, in essence, become multimodal readers. As our students’ analysis began to explore, even outlets with robust editorial standards can demonstrate surprising politically charged selection outcomes, such as preferentially selecting group photos of migrants over individual portraits.
Work with photographic databases is particularly well suited to help students learn how to address, or at least better understand, these challenges. To return to Griffin’s analysis of war photojournalism in news magazines, “photographs do not simply reflect events occurring before the camera but are inextricably implicated in the constructive process of discourse formation and maintenance” (399). By combining big-data analysis with attentive interpretations of selected, representative photographs, our students began to uncover the workings of this “constructive process.” They unearthed the political discourses surrounding the coverage of the Syrian Refugee Crisis exactly where their analysis straddled both the qualitative and quantitative. In doing so they avoided the “mechanizational” fallacy that leads students—and the general public—to misinterpret photojournalism. Extending Newton’s point about the mechanization of photography, with our students we noted that statistical analysis of data is often perceived as automatically produced by an “objective” technology and, as such, as independent of the pressures of ideology. As our students' analyses show, however, this is simply not the case. Qualitative methods of reading and interpreting quantitative data can help to illuminate the nuanced subjectivity of rhetorically complex, multimodal datasets.
We believe that the first step in meeting this challenge at the pedagogical level is to train students to move confidently between quantitative interpretive methods and qualitative rhetorical analysis, giving them the chance to acquire the necessary knowledge to interpret complex information. This can best be done by encouraging students to work collaboratively across the disciplines. Because STEM students often learn complex data science methodologies, but do not always understand how these methodologies can be used to address social and political issues, this process encourages them to think about how qualitative research shapes quantitative analysis. Simultaneously, it allows humanities-based undergraduates to produce innovative, original research using qualitative analytical skills, while also developing quantitative and technical skills. Building on Buckland’s insistence that the greatest challenge facing the information sciences is the rhetorical complexity of data and the work of Kress and Serafini on the necessity of developing multimodal, visual literacy, we advocate for a pedagogical approach that asks students to engage in humanistic and computational practices as complementary and as equally integral to their research project. As illustrated by our students’ Data+ project, our methodology explicitly and insistently argues that large-data statistical analysis needs to be paired with humanistic analysis to produce meaning from multimodal, narratively complex datasets. Only by being introduced to the framing logic of both methodologies, can students discover and begin to solve the theoretical and practical challenges of interdisciplinary work in multimodal data interpretation.
Agboka, Godwin Y., & Matveeva, Natalia. (Eds.). (2018). Citizenship and advocacy in technical communication: Scholarly and pedagogical perspectives. New York: Routledge.
Arola, Kristin L. (2018). A land-based digital design rhetoric. In Alexander, J. & Rhodes, J. (Eds.), The routledge handbook of digital writing and rhetoric (pp.199-213). London: Routledge.
Arola, Kristin L., Sheppard, Jennifer, & Ball, Cheryl E. (2014). Writer/Designer: A guide to making multimodal projects. Boston, MA: Bedford/St. Martin’s.
Berger, John, & Mohr, Jean. (1982). Appearances. In Another way of telling (pp.81-130). New York, NY: Vintage Books.
Buckland, Michael. (2017). Information and society. Cambridge, Mass.: MIT Press.
Clark, J. Elizabeth. (2010). The digital imperative: Making the case for a 21-century pedagogy. Computer and Composition, 27, 27-35.
Elkins, James. (2004). Pictures and tears: a history of people who have cried in front of paintings. New York, NY: Routledge.
Griffin, Michael. (2004). Picturing America’s ‘War on Terrorism’ in Afghanistan and Iraq: Photographic motifs as news frames. Journalism, 5(4), 381-402.
Gries, Laurie. (2015). Still Life with Rhetoric: A New Materialst Approach for Visual Rhetorics. Boulder, Colorado: University Press of Colorado.
Kafka, Alexander C. (2018, November 1). With student interest soaring, Berkeley creates new data-sciences division. The Chronicle of Higher Education. Retrieved December 1, 2018 from https://www.chronicle.com/article/With-Student-Interest-Soaring/244986
Kress, Gunter. (2005). Gains and losses: New forms of texts, knowledge, and learning. Computers and Composition, 22, 5-22.
McDonagh, Deana, Goggin, Nan, & Squier, Joseph. (2005). Signs, symbols, and subjectivity: An alternative view of the visual. Computers and Composition, 22(1), 79-86.
Mitchell, Amy, Gottfried, Jeffrey, Kiley, Jocelyn, & Matsa, Katerina Eva. (2014). Political polarization & media habits. Retrieved May 15, 2017 from http://www.journalism.org/2014/10/21/political-polarization-media-habits/
Newton, Julianne H. (2009). Photojournalism. Journalism Practice, 3(2), 233-243.
Nussbaum, Martha. (1996). Compassion: The basic social emotion. Social Philosophy and Policy, 13(1), 27-58.
Serafini, Frank. (2010). Reading multimodal texts: perceptual, structural, and ideological perspectives. Children’s Literature in Education, 22(2), 92-93.
Serafini, Frank. (2014). Reading the visual: An introduction to teaching multimodal literacy. New York, NY: Teachers College Press.
Sontag, Susan. (2004). Regarding the pain of others. New York, NY: Picardo.
UNHCR: The UN Refugee Agency. (2017, February). Desperate journeys: Refugees and migrants entering and crossing Europe via the Mediterranean and Western Balkans routes. Retrieved May 15, 2017 from https://www.unhcr.org/news/updates/2017/2/58b449f54/desperate-journeys-refugees-migrants-entering-crossing-europe-via-mediterranean.html
Vis. F., & Goriunova, O. (Eds.). (2015). The iconic image on social media: A rapid research response to the death of Aylan Kurdi. Picturing the Social: Transforming our Understanding of Images in Social Media and Big Data Research. Retrieved December 18, 2018 from https://research.gold.ac.uk/14624/1/KURDI REPORT.pdf
Wardle, Claire. (2017, February 16) Fake news. It’s complicated. First Draft. Retrieved December 18, 2018 from https://firstdraftnews.org/fake-news-complicated/
Warren-Riley, Sarah, & Verzosa Hurley, Elise. (2017). Multimodal Pedagogical Approaches to Public Writing: Digital Media Advocacy and Mundane Texts. Composition Forum, 36, Retrieved April 12, 2019 from http://compositionforum.com/issue/36/multimodal.php
Woolf, Nicky. (2016, November 16). How to solve Facebook's fake news problem: experts pitch their ideas. The Guardian. Retrieved December 18, 2018 from https://www.theguardian.com/technology/2016/nov/29/facebook-fake-news-problem-experts-pitch-ideas-algorithms