In a recent post, I summarized and explicated on research conducted by C. Kirabo Jackson, a professor at Northwestern University, who published a paper that examined the use of test scores to measure the effectiveness of a teacher using the “value added model,” which is better known as VAM.
In this post, I’m going to continue where I left off last time, as well as present Jackson’s findings using visual representations.
To begin, Jackson shows that by analyzing data from the National Educational Longitudinal Study from 1988, that there are two types of outcomes from teaching: cognitive and non-cognitive. These are summarized in the table below:
A comparison of cognitive vs. non-cognitive ability.
Jackson cites extensive evidence showing that teachers have differing effects on both cognitive and non-cognitive abilities of their students, and that together these two effects these can be represented as a “vector” – that is, each one acts on the long term outcome on an individual student in combination.
Incidentally, “grades” are different from “test scores” in that when a teacher issues a grade to a student , it usually comprises different aspects of the student’s work, which would include things like completing assignments, taking part in activities, and showing up to class. As such, it is much more representative of a student’s cognitive and non-cognitive abilities. What is interesting is that test scores are only moderately related to grade point average (GPA). According to Jackson, there is only a 36% correlation between test scores in math and English and a student’s GPA. Even more interesting is the fact that test scores are completely unrelated to socio-behavioral outcomes. Okay, let’s just repeat that in bold: test scores are unrelated to socio-behavioral outcomes.
Knowing this, we can understand that teachers vary in their abilities to increase their student’s cognitive and non-cognitive ability; similarly, students also have their own vector, which is represented by their cognitive and non-cognitive ability. Therefore, the effect of student “a” being in a class taught by teacher “b” is the sum of their two vectors. Pretty simple, yes?
Some examples of “teacher ability vectors” (my terminology) appear below:
The left graph shows a teacher who improves both a student’s cognitive and non-cognitive abilities equally; the middle graph show a teacher who is very effective at raising test scores, while being ineffectual at raising non-cognitive behaviors. The right graph shows the performance of a teacher who has a great effect on non-cognitive ability, while not improving test scores a great deal. Which leads us to the question: which teacher is the “most effective?”
Which leads us to the following thought experiment: if you had to choose between the teacher in the middle graph and the one on the right (assuming the one on the left is quite rare), which one is more highly correlated with things like fewer arrests, more college-going and improved wages? If you’ve been following me, you know that the data show that improvement in these important outcomes are more highly correlated with teachers who improve non-cognitive ability. According to Jackson, if we went by test scores alone to judge teacher effectiveness, more than half of the teachers who improve these long term outcomes may not be identified as being “effective.” This is because the teacher effects on college going and wage improvements is as much as three times larger than can be identified by using test scores alone.
If you are a educator, parent or policymaker, you should be outraged at this conclusion, which can be represented using this graph:
All of which begs the questions: if you are a teacher whose job performance is being judged using a “value added model,” you may choose to shift your resources to keep your job:
Indeed, this seems to be the focus of all “school reform” since “President” George H.W. Bush slammed us with “No Child Left Behind,” and President Obama followed up with the equally egregious “Race to the Top.” Folks like Arne Duncan like to rattle away at something called “accountability,” and in our mindless pursuit of higher test scores, something will be sacrificed. As Jackson’s research suggests, focusing on test scores could likely lead to more students dropping out of high school, fewer going to college and worst of all, lower wages as adults. Wait, is this what educational reformers really wanted?
