Teaching Students How to Write Teaching Evaluations

Contributor: Nick Carbone
Affiliation: The Bedford St. Martin's Imprint of Macmillan Education
Email: nick.carbone at gmail.com
Published: 15 January 2015

Why Teaching How to Evaluate Matters

In "Genre as Social Action," Carolyn Miller (1984) argues "that a rhetorically sound definition of genre must be centered not on the substance or the form of discourse but on the action it is used to accomplish" (p. 151). By that "used to accomplish" reckoning, student evaluations of teachers (SETS) are a fraught genre. Many teachers do not find them particularly useful, yet many administrators rely on them to assess teaching. Faculty perceive that better evaluations come from giving better grades (Sojka, Gupta, & Deeter-Schmelz, 2002), a perception supported by research into the correlation of student evaluation to course grades (Carrell & West, 2010). For female professors, research validates their experiences that students rate male professors higher and exhibit sexist behavior in written comments (MacNell, Driscoll, & Hunt, 2014; Martin, 2013). For adjuncts especially, SETs can be fraught because poor evaluations, or too many negative comments in otherwise neutral or even good evaluations, can get you fired (Lewontin, 2014). For faculty of color, especially if they are also women, the written comments from white student evaluations reveal racism and sexism, presented as post-racial egalitarianism that charges the professors as racist and sexist (Evans-Winters & Hoff, 2011).

Meanwhile, advice on guidance to faculty on how to interpret and use teaching evaluations ranges from the frank and often anonymous advice to ignore them (Spiros, 2009); the tongue-in-cheek advice to game students to get better evaluations (Mentor, 2003); the balanced advice to ignore the clearly vengeful, but heed the well-intended (Perlmutter, 2011), and for specific campuses, detailed advice from centers for teaching and learning on how to interpret and respond in earnest to the data and reports SETs create (i.e., Center for Teaching and Learning, Stanford University).

Yet as imperfect a system as SET is—see Schuman (2014) for further elaboration—for those many professors who do care about teaching, student evaluations do sometimes offer constructive feedback. Further, research shows that a lead motivation for students who do SETs is to help professors improve their teaching (Sojka, Gupta, & Deeter-Schmelz, 2002; Chen & Hoshower, 2003). Students want to be helpful.

And student feedback can be more constructive and helpful if students are taught how to write constructive evaluative comments on those portions of SETs that call for written comments. In believe this is a skill worth teaching, that the SET evaluative comments are a genre worth engaging. The intellectual work of understanding teaching and learning is bound in these reviews. To write a good review, students need to know a bit more about the course, which means instructors need to make the learning outcomes for the course clear and need to explain how assignments help students reach those outcomes. Students, in turn, need to think more fully about their responsibilities in courses. And so for students, the review of the course is not just a judgment of the instructor, but also a chance to reflect on their learning and on themselves as learners. Thought of that way, SET writing is not apart from the course but is instead a part of the course. What follows are notes toward making SET reviews a richer learning experience for both professors and students.

SETs and College Writing Courses

Despite being designed for social action—providing student feedback so that instructors can improve their teaching, so that writing program administrators can mentor faculty (or remove faculty if teaching doesn't improve), and so that administrators can evaluate promotion and tenure—most student evaluations of teaching are not taught as a genre, at least in my experience. Many times, SETs are not discussed in the course at all. The trend seems to be for faculty to have students complete the evaluations required, when required, by their department. A minority of faculty use individualized student evaluations of teaching mechanisms of their own design, often at mid-term, with the express goal of using that feedback to guide changes while the course is still in session. But even then, there's often little discussion of how to write a good evaluation. But since the course is a writing course, why not take a little time, given that the writing of SETs is required, to teach students how to write them well?

After all, student evaluations of teaching meet Miller's (1984) five criteria for what constitutes a genre. A genre, says Miller, "refers to a conventional category of discourse," "is interpretable by rules," "is distinct from form," and "serves as the substance of forms" (p. 163). (See her essay for the full articulation of her criteria, the first four of which I quote too briefly to do justice.) I want to draw special attention to her fifth criterion: "A genre is a rhetorical means for mediating private intentions and social exigence; it motivates by connecting the private with the public, the singular with the recurrent." (p. 163). The research about SETs cited above explores this fifth element—the intentions of SETs, the private and different motivations of students and teachers when engaging evaluations of teaching, the way those motivations become public, and how the evaluations recur, accumulate, and build an argument about a teacher. SET writing is an academic genre. It makes the intimate—the usually closed sphere of classroom teaching and learning—public. It does important work, but is can also be hard to do well because it triggers emotions that can swamp a writer, making it harder for him or her to write in a way that conveys what they seek to achieve. It is a very tricky genre. 

Because SET writing is a genre that carries so much potential for good but often causes harm, it's a genre worth addressing. While any instructor in any course can adopt the suggestions below, I focus on writing courses given the ways that writing instruction often includes supporting (often emotion-based) claims with reason and evidence. And so, if a writing teacher can make time for teaching SET writing, it will help not only her evaluations, but also faculty across the curriculum. Writing courses have several advantages in taking this up:

• Many courses use rubrics where quick scores are mixed with written feedback, either for instructor feedback or as part of peer review. Online peer review systems make excellent pedagogical use of the kind of mix of rating and Likert scales combined with written responses that are standard in most SET formats, print or online. See this screen shot of Eli Review, a writing workshop platform, for an example: 

• Writing courses can be dialogic spaces that teach how to give and use constructive feedback. So time spent teaching the transfer of that skill to SET writing reinforces course ethos and outcomes. This is true even when peer review is used minimally and/or doesn't use the kind of format seen in the Eli example above. Furthermore, as courses that include discussions fostering insight by asking students to draw on excerpts and passages from assigned readings to inform their comments, critiques, and assertions, writing courses practice the art of analysis through evidence and encourage writers to make claims that can be supported, two qualities good SET practice should follow.
   
• Asking students to do SETs carefully and thoughtfully in the course, as a direct appeal to students for their help in improving one's teaching, lets professors model the kind of behavior they ask students to follow when asking for feedback on their writing and learning. The SET writing activity then shows students that all learners and practitioners of a craft, be it writing or teaching, can improve by asking for constructive feedback.
   
• In taking the time to bring SET writing into the course, including having students share and revise their SET pieces after feedback and review, faculty can help students understand SET as a genre that works best if it's not confused simply with the kind of quick-draw response given in pique or praise on consumer Web sites or at RateYourProfessor.com. Addressing SETs lets faculty address related issues—student responsibility for learning, the purpose and goals of a course (for these are also course evaluations), practice in observing and assessing teaching and learning more reflectively, and the educational mission of the college.
   
• Research on SET completion (Stark & Freishtat, 2014) shows that "Some students do not fill out SET surveys. The response rate will be less than 100%. The lower the response rate, the less representative the responses might be: there’s no reason nonresponders should be like responders—and good reasons they might not be. For instance, anger motivates people to action more than satisfaction does" (p. 4). By taking SETs seriously enough to teach, and by giving students practice in doing SETs well, instructors increase response rates and the quality of the feedback.

I don't want to suggest that writing courses should all be solely responsible for teaching SET writing. This is a WAC need. But writing courses can offer a great start. Here are some brief notes for helping writing professors, and by extension, other professors, do more with SET as a genre worth teaching.

Four Things I Convey to Students

I don't have a static assignment sheet for teaching SETs. How and when I approach it varies based on each course's class chemistry, workload, and other factors. But when I address it, I always touch on these four strategies.

1. Show Don't Tell 

Early in the semester, sometimes on the first day of class, I write this sentence on the board: My wife has a pretty cat. Then I sit down, tell students to take out a sheet of paper, put their name and the date on it, and declare a test: they have 10 minutes to describe accurately my wife's cat.

Students cannot of course say anything accurate about the cat; "pretty" tells them nothing they can use to describe it. It's an opinion, unsupported by a photo, prior knowledge, context, or even a description of the cat. Much of the writing in SETs is of the "pretty cat" type: students assert a behavior or attribute to the course or professor without context or enough details to make the feedback useful.

So the first thing I tell students is that they need to describe something that leads to the judgment they are making; they need to ground their reviews. Instead of "My teacher is lazy," for example, they might say:

In the syllabus, it said papers would be turned back within three days with comments. However, this semester, we only got our papers back in three days once. The average was five days, and for two assignments took even longer. Three of those times, we were not given more time to revise—our next deadline held even though we lacked the time promised to revise the work. The worst was the last assignment, when papers were returned five days late and we only got two days to revise a ten-page paper that had extensive comments. This made it hard to plan work, and I had to cut into time I'd set aside for other courses. If it's going to take five days instead of three, change the syllabus, then for every day late back, we should get a day extension on the next due date. But it would be better to just stay on schedule.

I require, in essays and other class writing, that students provide the kind of detail this example exhibits. Why should open-ended SET responses be any different? They shouldn't, but without some examples of good descriptive writing that draws on evidence and observations to support conclusions and suggestions, it is hard for novice reviewers of faculty and courses to give better feedback.

2. Transfer Critical Reading and Writing to SETs

The detail in the imaginary critique above describes a teacher not following the syllabus and leads to a suggestion that the teacher either do what the syllabus says they will do or to change the syllabus and course schedule to do what the teacher ends up doing.

Though I teach how to write like this, I've found that SET forms and their open-ended responses do not trigger the skill to write with this kind of detail. I think this is because SET questions that invite writing often come in the context of a survey, where Likert Scales and ratings questions are designed for data gathering sets so students write short, "pretty cat" responses to open-ended questions because the unconscious skill or habit students evoke is the unsupported claim—liked it, hated it, it was ok—models in consumer surveys, even if the SET questions are well-crafted and ask for details.

When I talk to students about SET writing, I have to know how the SET system in use where I teach frames and poses open-ended questions. With that knowledge, I can focus a bit on addressing the desire for students to transfer some of the analysis skills we used reading texts in the course to the course. Instead of analyzing assigned readings, their research, or classmates' works, in SETs, the course and my teaching become the text they are asked to analyze, assess, describe and review.

Something Any Teacher Can Try
To prepare for a semester-ending SET, I'll often ask students to write anonymous mid-term reviews based on the kinds of questions I know the SET will ask. After students post anonymously, I'll log in and respond, focusing both on the structure of the critique—whether it had enough detail for me to understand the basis of the review—and if the structure was good, a response to the review. If the comments lacked the kind of detail needed to be useful or the kind of structure I asked for, I'd explain why I couldn't use the feedback and would invite a revision.

For the mid-term self-generated SET, I mix this up and sometimes include peer review, letting students swap critiques so they can give one another feedback and advice for revision. But also, like with any reading they do, they can share a discussion about the shared reading (me as teacher and the course) and topics they're choosing to write on.

This call to revise and to peer review is how I treat the SET writing like any other writing in the course. Though the final SET circumstance will often not allow for revision and review, getting practice answering a SET question and treating it as writing worth revising helps prepare for official SETs by giving them practice that associates SET writing with the same kind of critical writing and revising they do for other genres.

After receiving these mid-term evaluations, in addition to written responses, I'll take some time to summarize for the class the gist of the responses, to discuss the course, and together, if I'm going to make adjustments to the syllabus and course, we'll talk about what those might be. I want students to see changes in my teaching that come from their reviews. I want them to see that constructive feedback can make a difference and that for me these reviews matter. That helps them write better reviews at the end of the term, thereby reducing some of the cynicism.

3. Do Not Critique Dress, Hair, Make Up, Weight, Gender, Orientation, or Politics

Here's what I tell students: you may not like nor understand as a person every teacher you have. However, these reviews are a place to address teaching. Chances are if you're motivated to say anything about clothing, body, gender, orientation or other issues, you either did not like the course's topic, approach to the topics taught, teaching strategies used, assignments, books, homework, and so on.

A lot of times students fall into the post hoc ergo propter hoc fallacy—a woman professor who assigns readings that criticize male behavior is a feminist who hates men. Because of this, they come to see a direct critique of a professor's gender, orientation, or politics as a critique of a course's content or politics, especially if the content, readings, and discussions make the student uncomfortable. When I teach SET writing, we spend a bit of time discussing this fallacy among others. Instead of addressing the teacher, students can be taught to address the course: What questions did the course ask you to ask? Why did you not like those questions? If the professor critiqued your writing or comments you made in class discussions, was that done respectfully?

But here's the thing. I'm a 55-year-old white male. I'm past the age of caring about remarks on my clothing, haircut, or other personal appearance issues (though when I was just starting as a teacher, everything made me nervous). I don't teach from or with radical pedagogies, though I am culturally liberal. So when I do teach around race, gender, and class issues, even though I'm often drawing the discussion to focus on systemic, unconscious racism built into our culture and ourselves, I don't run into the kind of gender and body comments a younger professor, or female professor of any age, might get. I certainly don't get accused of being racist the way Evans-Winters (2011), a black feminist professor, was in this comment:

Dr. W was a very poor instructor. I was looking forward to learning about teaching students of a diverse background and instead only learned about teaching African American students. I feel there are many other cultures and races to learn about as well. I also feel that Dr. W was very biased on African American education and was racist on Caucasian individuals. (p. 469)

I know that asking students not to address these issues can be more charged for professors of color, for women, for those who are gay, lesbian, transgendered, and for those who are younger and still establishing authority. However, if your pedagogical work requires taking students into discussions and ideas that discomfit them or challenge directly their beliefs in an ideology or their perceptions of themselves, then left unaddressed, the likelihood of the kind of assaultive comments Evans-Winters and Hoff (2011) share increases. In those courses, addressing SET writing might be important especially because it might be more difficult. The request each instructor makes in this kind of requirement—the do nots—speaks directly to fears and resentments professors in these situations sense exist. And so though difficult, this suggestion, combined with others, might help instructors address these kinds of issues in their SETs.

This message can be conveyed to students as a message of empathy and audience awareness, again tying directly to the work done in writing courses. Instructors can tell students that writing effective SETs, where their judgments will be considered, includes describing, critiquing, and praising the teaching, course content, assignments, grading policies, and teacher behavior as a teacher (handing back work on time, treating students fairly and with respect, etc.). Telling students that objecting to a course theme, selection of readings, questions the course asked them to consider, describing why they had objections based on course details, and conveyed in an honest but respectful tone, their views will be heard and considered more fully. 

To prepare students for SET response writing, any faculty member, no matter their race, sexual orientation, age, gender, may want to include excerpts (the References to this article includes numerous examples published in the SET literature) of comments that are hurtful and unfair, as well as faculty responses to those comments, so that students can see more viscerally what is at stake. This kind of direct address of the issue will not end the presence of these kinds of comments; in fact, a truly angry or upset student might even mimic them after such discussion. But I nonetheless have faith that raising the issue in a thoughtful and considered way can help make evaluations better.

4. Describe Strengths and Things that Worked 

A lot can be learned from knowing what works, especially if a learner can tell a teacher why something worked. I ask students to think through these questions: What in an assignment helped you learn? What projects worked well for you and why? What about the grading policy made sense and seemed fair? What motivated you to do better work?

When I address the need to describe what works well and why with students, I remind them that teaching is an experiment. No teacher can control for all variables—the collection of students assembled, the class chemistry that will emerge, how an assignment will work. We talk about how, in a class, an assignment will work for some students and not others. They almost all can recall a time when they've gotten something a classmate hasn't or vice-versa. Plus asking the questions also gives them insight into how they learn and what kinds of learners they are.

I tell students how sometimes as a teacher, an assignment that goes great in one class will bomb in the next when one is teaching two or three sections of the same course. Teaching is not a certain thing, no more than learning is. A syllabus is a learning hypothesis, not a guarantee or a fact. Teachers learn as much from knowing why something was productive and how it worked well for a learner as they do from when it didn't.

I remind students, too, that if they can describe something that worked well—only doing so if something did, of course—then descriptions of what didn't work or suggestions for what might change will be more persuasively heard, assessed, and considered by instructors and administrators.

Notes for Teachers

Assign Practice SETs

As noted above, I'll ask students to practice evaluations, as transferring critical reading, analysis, review and recommendation skills takes practice. When students do a mid-term review of my teaching, I often set up anonymous mini-evaluations using something like Survey Monkey or Google Forms. I'll include some Likert and rating scale questions to mimic the kind of context they'll see in a course-ending SET. Or, if instructors want to dry run the evaluation their departments or programs will use in an unofficial way, they can easily recreate it online.

I don't mind taking time on this practice because the ability to read closely, observe, record detail, draw inferences, and make recommendations are all skills central to my writing course. In treating courses students take (mine included) like a text, I'm asking students to read courses and their own learning critically. In asking them to look at the teaching and learning, I'm asking them to be reflective students, which goes to them being reflective writers, thinkers, and learners overall. And I don't mind either unofficially using or mimicking for practice the actual SET form and circumstances (i.e., online, within a class period if doing Scantrons) students will encounter during official SET writing. In sports, athletes learn by playing their sport in practice, with teaching from coaches and encouragement and feedback from teammates based on conditions that simulate an actual game or performance. The same thing happens in writing courses, and SET writing practice has direct correlation to official SET writing students will have to do. Course instructors are assessed on their teaching in part (sometimes wholly) by SET results. SETs have consequences, and so practicing doing them well makes sense.

I sometimes ask students to do practice SET writing by doing a SET for a course other than the one they are taking with me. Students cannot say the name of the course or the professor, and they know that what we practice stays in class. This is not an approach I'd do at a small college, where I know who most of the faculty are and what courses they teach. But at larger colleges or universities where faculty are diffuse and dispersed, it can work. I often add the caveat to not write about a course offered in the department I'm teaching in, to cut down on the chance of accidental revelation.

Not every course lends itself to asking students to do this kind of writing about what's going on in other courses. If you are not teaching a course where the assignment makes sense, then I still recommend doing a mid-term SET of your course, where, on the writing portion, you ask students for detailed feedback. But teach what you mean by detailed feedback, perhaps by drafting examples (take the one above under "Show Don't Tell" if you'd like) of what you'd like to see.

A Closing Anecdote

How Being Transparent About Teaching Helps Students Write Better Evaluations and Makes Teaching More Fun

A friend described a set of evaluations she once got, where the students complained that she didn't do enough work and that they did more. Students went into her course assuming she would lecture, give tests, and grade their essays and that their job was to show up, listen and take notes, go home, read, draft and revise, and answer questions when called upon. So the reviews were harsh when the course didn't do what they had come to expect.

The next time she taught, she spent more time explaining to students why they were doing more workshops, why they were in small group discussions taking turns as discussion leaders, why they had drafts were the only feedback they got was from fellow writers. All those choices had good pedagogical reasons that once students understood them, they accepted more, which lead to better evaluations and reviews, including ideas for making the workshops even more effective. Students, instead of pushing against, were pulling with the teacher, and the feedback became more constructive and the reviews more positive.

Getting students to understand the purpose of the work in the course—the readings, discussions, workshops, peer review, instructor response time, assignment scaffolding, optional work, assessment—goes a long way to receiving constructive teaching evaluations. Too often students enter a course and are asked to do work without understanding how that work moves them to a teacher's goal. A classroom is a rhetorical space after all—a professor or program authors the course for a learning purpose that the audience of students are asked to engage.

What this anecdote reveals about SET evaluation writing is the importance of knowing enough to write a good review. Without enough knowledge, reviews of any kind are often reduced to unsubstantiated opinions. So to teach SET writing, writing professors need to teach a little about course design, assignment choice, and learning, an act that requires them to think about those things and share with students their teaching philosophy, pedagogical beliefs, and reasons they are asking students to do the work. Making explicit what is often implicit or even invisible to students lays the groundwork for students to write better evaluations. By better, I mean not only better quality and more constructive, but also better as in more favorable. Taking the time to articulate reasons and goals for assignments, how the work connects to course goals and student needs, requires teachers to think through those very items, which improves teaching. Teaching SET writing is really about thinking and teaching about teaching. Conversations about teaching with students, especially in writing courses where students as readers, writers, and thinkers teach one another and themselves, support serendipitously the work of the course and the ethos of the community formed when teachers and students meet.

But as importantly, it's fun. Teaching and learning should be fun and enjoyable. The work is hard and important; it involves frustration and struggle, sometimes failure and discord. But for teaching and learning to work, joy is required. Joy in learning begins at understanding purposes and goals, in trust that work and time, that feedback given and received, and that revisions (and revisions and revisions) drafted and crafted matter. That understanding makes it more possible for writers to discover intrinsic motivations and to have better conversations so that when their private views become public through the genre of the evaluation, they will be better able to express what they learned, and to do so more as partners in learning and teaching rather than consumers and recipients of knowledge. Their feedback will simply be more meaningful, more accurate (even when critical), more useful, and maybe even more fun.

References

Carrell, Scott E. & West, James E. (2010). Does professor quality matter? Evidence from random assignment of students to professors. Journal of Political Economy, 118(3), 409-432. Retrieved from http://www.econ.ucdavis.edu/faculty/scarrell/profqual2.pdf

The Center for Teaching and Learning, Stanford University. (Undated). Interpreting and working with your course evaluations. Retrieved from http://web.stanford.edu/dept/CTL/cgi-bin/docs/faculty/interpret.pdf

Chen, Yining & Hoshower, Leon B. (2003). Student evaluation of teaching effectiveness: An assessment of student perception and motivation. Assessment & Evaluation in Higher Education, 28(1), 71-88.

Evans-Winters, Venus E. & Hoff, Pamela Twyman. (2011). The aesthetics of white racism in a pre-service teacher education: A critical race theory perspective. Race Ethnicity and Education, 14(4), 461-479. Retrieved from http://dx.doi.org/10.1080/13613324.2010.548376

Eli Review. (2014, December). Eli review user support: Review tasks. Image retrieved from http://elireview.com/support/guides/student/review-tasks/

Lewontin, Max. (2014, October 6). For adjuncts, a lot is riding on student evaluations. Chronicle Vitae. Retrieved from https://chroniclevitae.com/news/741-for-adjuncts-a-lot-is-riding-on-student-evaluations

Martin, Lisa. (2013, October 2). Student evaluations of teaching are probably biased. Does it matter? Guest Post. The Washington Post Monkey Cage Blog, ed. by Erik Voetten. Retrieved from http://www.washingtonpost.com/blogs/monkey-cage/wp/2013/10/02/student-evaluations-of-teaching-are-probably-biased-does-it-matter/

MacNell, Lillian, Driscoll, Adam & Hunt, Andrea N. (2014, December 5). What’s in a name: Exposing gender bias in student ratings of teaching. Innovative Higher Education, (Dec. 2014), 1-13. Retrieved from http://link.springer.com/article/10.1007/s10755-014-9313-4

Miller, Carolyn R. (1984). Genre as social action. Quarterly Journal of Speech, 70, 151-167. Retrieved from http://www4.ncsu.edu/~crmiller/Publications/MillerQJS84.pdf

Ms. Mentor. (2003, March 24). The torment of teaching evaluations. The Chronicle of Higher Education. Retrieved  from http://chronicle.com/article/The-Torment-of-Teaching-Eva/45129/

Perlmutter, David D. (2011, October 30). How to read a student evaluation. The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/How-to-Read-a-Student/129553/

Schuman, Rebecca. (2014, April 24). Needs improvement: Student evaluations of professors aren’t just biased and absurd—they don’t even work. Slate. Retrieved from http://www.slate.com/articles/life/education/2014/04/student_evaluations_of_college_professors_are_biased_and_worthless.html

Sojka, Jane, Gupta, Ashok K., & Deeter-schmelz, Dawn R. (2002). Student and faculty perceptions of student evaluations of teaching: A study of similarities and differences. College Teaching, 50(2), 44-49. Published Online 26 March 2010. Retrieved from http://www.tandfonline.com/doi/abs/10.1080/87567550209595873

Spiros. (2009, May 5). Teaching evaluations: Ignore them. Philosopher's Anonymous Blog. Retrieved from http://philosophersanon.blogspot.com/2009/05/teaching-evaluations-ignore-them.html

Stark, Philip B. & Freishtat, Richard. (2014). An evaluation of course evaluations. Science Open. Retrieved from https://www.scienceopen.com/document/vid/42e6aae5-246b-4900-8015-dc99b467b6e4?2