A different way to think about ... student assessment

A different way to think about ... student assessment

Lee S. Shulman

Source:


This is surely the age of accountability, both in higher education and in K-12. As the stakes associated with test performance rise, both for the students and for their institutions, the dilemmas of coaching for test performance have risen as well. Many of the newer forms of assessment make the distinctions between teaching and assessment less and less clear.

When the test takes the form of a dissertation, a portfolio or an architectural design, can we tell how much was done by the coach and how much by the candidate? What if one's peers have collaborated in producing the artifact, whether as co-designers or as editors or critics? And if we insist on tests that reduce dramatically the role of mentors or coaches, might we also be using tests that are more highly artificial and unrepresentative?

Here are some ruminations about the dilemmas of coaching in the contexts of high-stakes testing. They are drawn from an essay titled "Send me in, Coach!" that I've written for the September 2008 issue of The New Educator, a journal published by the City University of New York.

Lee S. Shulman
May 2008


--------------------------------------------------------------------------------


When Coaching and Testing Collide


It's a scene we have watched dozens of times in the movies. A young man or woman of modest talent tries out for the baseball or football or basketball team under the tutelage of a gruff, demanding coach who expresses initial doubts about the likelihood that the kid will prove himself or herself worthy of a spot on the team. The coach is tough and persistent, setting high standards and then mercilessly driving all his charges to meet them. In the climactic scene at the season's end, the good guys or gals are losing by several baskets, or runs, or a touchdown—depending on the sport. "Send me in, coach," pleads our young hero/ine, which coach reluctantly does. The kid scores the winning points, and the team wins. The coach turns out to have a heart of gold, and the reasons for his seeming cruelty become apparent.

What exactly is it that the coach provides the aspirant? Let me propose five processes associated with both the coach and mentor roles: 1) technique, learned through endless drill; 2) strategy, that allows the person who is coached to become capable of a conception of the work that will turn out to be pivotal in their eventual victory; 3) motivation, which produces a "Rocky-like" level of commitment that will help them exceed their own and others' expectations; 4) vision, where players come together in a new vision of the process and their capabilities for success; and 5) identity, whereby the protagonist not only wins, but is transformed, with an internalized new sense of self.

In sports there is always a clear line between the coaching situation and the performance context. When the final jump shot is made from the three-point line by the basketball player, the coach can't jump onto the court and give the ball the extra momentum or spin it might need. I prefer to call such typical relationships between a coach/mentor and player/protégé examples of unmediated mentoring. No separate product comes in the middle between the coaching and the performing that renders the relative contributions of the coach and the coached inherently ambiguous because the entire performance is visible and is itself the basis for evaluating success or failure.



There is, however, an entire genre of mediated mentoring. The performance is not directly observed and has yielded a product which is the focal point of competition and evaluation. Thus in the case of mediated performances, the respective roles of coach and performer are inherently invisible. Although the five processes are in place and just as transformative, there is inherently no way to discern how much of the work was done independently by the candidate, by peers or by advisors.



Whenever mentoring is mediated by a product whose actual authoring processes are not directly observable, as is the case with literature, objects of architectural or mechanical design, scholarly publications, doctoral dissertations, and even paintings, assessment of individual competence is problematic. But are these problems of educational measurement or a new set of realities regarding the conditions of expert performance? Stanford education professor Sam Wineburg and others point out that the crux of the problem may not be measurement error but rather the inherently social and interactive character of the performances whose competence is assessed. Writing is and should be critiqued and edited, as should painting, the designs for buildings and the research performed in scientific laboratories. To avoid mentoring merely to ensure the legitimacy of individual test scores might even be judged a form of malpractice! So we are faced with an essential tension between the inherently social character of most forms of complex human performance and the psychometric imperative to estimate a "true score" for ability or any other personal trait using the individual as the unit of analysis.

In an education setting, the distinction between the scores that a student earns on any test-like event—multiple choice test, essay exam, portfolio or senior sermon in a seminary—and their underlying "true" capability is a reflection of the distinction, borrowed perhaps from the field of linguistics, between competence and performance. Psychometrics rests on the claim that the observed performance is a valid indicator if it tracks the underlying competence faithfully. But what if mentored or coached performances actually track underlying competence more validly than measurement of students working alone? What if the composition written by a student in the presence of his editing team is a better indicator of his future writing competence than having him write alone?



That is what sits at the heart of the puzzle.



My proposal for "getting over" this essential tension is three-fold: making changes in the processes of assessment, making explicit the parameters of mentoring, and developing a clear code of ethical principles for both assessment and mentoring. At the heart of these proposals is the principle of transparency. Everything possible must be done to ensure that the roles of mentors, peers and students be transparently clear in any mediated mentoring activity. There should be ways of reporting on the character of coaching for test performance that make the efforts of the coach entirely transparent to assessment.



I have often written that collaboration is a marriage of insufficiencies; that students can work together in ways that scaffold and support each others' learning, and in ways that support each others' knowledge. Now I call for a marriage of sufficiencies to overcome the essential tensions between individual work and collaborative performance, coaching support and independent assessment, the mentor as an agent of zealous advocacy and the mentor as a steward of the commons.



As Dewey observed, we will not solve this problem, we will get over it. It is built in to the psychometric paradox: Our measurement models are psychometric but our assessment needs are often sociometric, requiring the measurement of socially scaffolded and joint productions.