Late Penalties

I was recently asked by a colleague what my policy is on late penalties. This is, in part what I replied: What I tell my students is, if you submit it before I've gotten around to marking the rest of the class, you're in the clear. That way I don't hold them to a higher standard than I am able to meet myself. It drives students crazy when they have to get an assignment in by such and such a date, and then watch the pile of unmarked papers sit on the professor's desk for four weeks. What was the point of imposing that deadline?

(Instructors often seem quite willing to make excuses for themselves—life happens—while denying the same right to their students, who may have the same or greater out-of-class responsibilities, and considerably less resources to bring to bear to resolve issues then their instructors.)
Photo reflects very BAD assessment practices.

From an assessment point of view, there is no logic to time limits on learning, save in those few subjects where automaticity is in fact a criterion. If it takes one student two days longer to figure out a concept, does that make their understanding of the concept less valid? The argument that the extra time gives the late comer an advantage over those who made the deadline is only valid when normed referenced grading is used, which I would argue is largely inappropriate anyway. Criterion referencing is more valid in most situations, so unless timeliness is a justifiable criterion, I can't see why we have any deadlines, except the logistically necessary ones of "end of the course".

I tell my students that due dates are guidelines for managing their workloads, and that if they miss the deadline I can't guarantee they will received the assignment back before next one is due. So it's in their interest to meet the deadline, and 99% do. The 1% who are late usually have a good reason (family priorities, for example) and it is oppressive of us to arbitrarily and inaccurately certify that they have learned less and performed less well then they really have, just because they missed some due date. For example, many of my students are single moms—why should I deduct 10% from their grade because their child got sick the week the assignment was due? Similarly, many of my students are First Nations, and have responsibilities to an extended family. I would argue that penalizing such students in this way for prioritizing family over school is sexist, racist, ageist and etc., etc. because the young white males living in their mom's basement and having their meals and laundry done for them have much less difficulty meeting deadlines. Deadlines and penalties may have been partially justified when students were full-time students only, but the reality is that our students have jobs, families, responsibilities and lives outside the classroom, and insisting that our course be the most important thing in their life is just egocentric nonsense.

The Multiplayer Classroom: Designing Coursework as a Game

Interesting book on designing courses like designing video games here: http://www.facebook.com/MultiplayerClassroom The site / text was recommended to me by a colleague as coming from an author with impeccable credentials, so I pass it along here

Wrong answers....

My daughter was laughing her way through Richard Benson's "F in Exams: The Very Best Totally Wrong Test Answers", which is a pretty funny read, and asking me what I thought. Well, mostly I thought the answers were amusing, and the questions that illicited them poorly written. If you ask students, "Where was George Washington born" and they answer, "in a bed", you cannot mark it wrong. It correctly answers the question asked. If you want the name of the city, you have to ask "In what city was Washington born?" And so on. But half the answers in Benson's book are in fact wrong answers, but funny nevertheless, either because they are cheeky, or because they simply reveal an understandable misunderstanding. As it happens, I came across just such an example as I was marking my own exams at that moment. A student who meant to refer to the educational limitations of "rote memorization" wrote instead "rogue memorization" -- an unfortunate slip of the pen, or perhaps a genuine mishearing of what the student thought his/her profs have been saying. But I kind of like the phrase! Yeah, you really don't want your students dwelling on rogue memorization. Given the randomness of some of the answers to the questions posed on the exam, I greatly fear some of my class have in fact experienced rogue memorization when studying for the exam....

Questions Answered.

Q: How many alternatives should a multiple-choice question have?

Unless you have a specific subject that demands more than five possible alternatives (say, doing a unit on the solar system, having all 8 planets as alternatives might make sense -- though a matching question would probably work better!) don't do it! The more alternatives, the worse your question is likely to be. Four or five alternatives are standard on professionally designed test (since statistically, this reduces the score students could get by pure chance to 25 or 20% respectively). Some people want to pile on alternatives to make the questions 'tougher' and to 'eliminate chance'. But here's the thing -- having 7 alternatives does reduce the chance of them getting the question right by blind guessing, but why bother? Once they've gotten less than 20%, how much more failed do they need to be? And while theoretically reducing the impact of chance on getting the right answer, the more answers the student has to read through, the more it becomes a readiing test rather than a test of that subject matter. So do we really want to eliminate chance factors by penalizing poor readers, ESL students, and so on. Most test designers agree the trade off isn't worth it!

And in the real world, coming up with 7 or 8 credible alternatives becomes REALLY hard. Again, unless there are an obvious 8 possible choices like the 8 planets of the solar system, you will drive yourself crazy trying to come up with credible but clearly wrong answers six, seven and eight. Why do this to yourself?

Or, people will do terrible things like having alternative 5 as "A and B, but not C". No, no no! ? Never do this! It becomes a test of reading and logic rather than subject knowledge. Students will hate you with justification since the test will not be an accurate reflection of what they know -- indeed, some research suggests that this INCREASES the importance of luck...

But here's the killer -- research in the early 1990s suggested that the overall quality of tests DECLINCED with the increase in number of alternatives per question. Everyone who has ever designed an mc test knows that coming up with the answer is easy, the first two wrong answers pretty easy, it's alternative 4 that's tough, and #5 is almost impossible -- the higher you go, the more desperate one becomes to fill the last spot. So, a test with seven alternatives will reduce the writer to grasping at straws, and they will end up accepting ridiculous alternatives that even those completely ignorant about the subject will have no trouble eliminating -- a complete waste of space and student reading. And then -- this is where human nature gets interesting -- since I've given up and accepted a stupid alternative for this question in desperation, my standards for writing the next question go down, because even though I know this is a terrible alternative, it is not as bad as the last one. Or, having accepted three weak ones, what's one more? Pretty soon, the test is garbage.

In contrast, tests with 3 alternatives (the right answer and two wrong alternatives) turn out to be easier to write, and therefore are written to a much higher standard. Students perceive them to be much tougher tests! And, research says, they really are more valid and reliable! So, I tell my students to write questions with three (good!) alternatives rather than going for four or five. Professional test designers can go for four or five because we have the time to come up with high quality 'd's and 'e's, but the realities for classroom instructors is that that is not going to happen.

It's true that with only three alternatives, students can get 33% just by blind luck, but um, so what? I don't know any course where 33% is a pass. Failed is failed. And the results of this test will more accurately reflect what students actually know than one with 7 or 8 alternatives.

Questions Answered.

From time to time strangers email to ask a question on test construction, which I do my best to answer. If they are the sort of questions that I get a lot, I add them to the "Frequently Asked Questions" file on the test construction site; but I think I'll highlight some of them here in the blog as well.

Q: Where is the best place to put the correct answer? For example, if I provide seven choices, does it make a difference if the correct answer is choice 'b' instead of choice 'e'?

A: Professional test designers place the answers randomly -- I mean that in the literal statistical sense of the word, not 'wherever'. They use tables of random numbers, or complicated computer programs that assign the answer randomly, to decide which spot will hold the answer to each question.

What they do NOT do is place it themselves. Research shows that left to our own devices, most people will attempt to 'hide' the correct answer somewhere in the middle of the list. (Nobody wants to put the right answer in A, because then the students won't even read the other alternatives you worked so hard on; and putting it in 'e', it just sort of seems to hang out there over the edge. Sticking it in the middle feels right! Even though, that's wrong.) Even experienced test construction professionals will unconsciously choose 'c' (or for some individuals, it turns out to be 'b') 3/4 of the time. That's why the rule for taking an mc test is "when in doubt, choose 'c'" -- because unless one takes care to distribute correct answers to get an equal distribution of A, B, C,D, etc, there will be way more 'b's and especially 'c's than other answers, so testwise students can do quite well for themselves simply by answering 'C' to every question. That's why professionals force themselves to do it randomly by using computers or tables of random numbers. And then they'll double check at the end of the test to make sure they have roughly equal number of a, b, c, d, etc.

For classroom instructors etc, I wouldn't bother with tables of random numbers (which are kind of a pain to work with) and let the answers fall where they may by pyramiding questions. To stop students from trying to figure out which answer will come next ("there have been three 'd's in a row so next one must be something else") you let the internal logic of the question dictate placement. Numerical answers are listed in ascending or descending order; dates in chronological order, single word answers are listed in alphabetical order; sentences either on the basis of some internal logic or more usually shortest to longest or longest to shortest. (Incidentally, this also makes the test look really pretty! People who don't pyramid their tests have really ragged looking questions). So if the correct answer turns out to be the longest, it places itself in the 'e' slot, not letting the designer 'hide' it in the middle 'c' spot. After the initial draft of the test is done, one quickly looks through to ensure one has equal numbers of a, b, c,s etc. Where there are too many of one, say 'A's, you go through and change some of the ascending questions to a descending to move the 'A' to a "D" or whatever. It works pretty well!

Peer Evaluation vs "Passing Back"

Caught a student teacher having her students pass their assignments back for the student behind to mark (last person in row brings it forward to person in front desk). This is a fairly common practice among classroom teachers, but it is a very, very bad idea.

It is bad evaluation practice because students routinely 'fix' mistakes for their friends, misinterpret directions or answers, are too narrowly literal or too generously accepting of variations in acceptable answers. The assessment data produced thus collected is therefore unreliable-- and if the data one is collecting is unreliable, why bother?

it is unprofessional because grading is the teacher's responsibility, and should not be delegated to student slave labour -- time spent by students marking other student work for the teacher may well save the teacher's time and energy, but diverts class time from learning. The teacher is supposed to be there for the students' needs, not the other way around.

But mostly my objection is that it is an unethical practice because it violates the students' right to confidentiality. Teachers may regard "passing back" of spelling or math tests a trivial matter, and the waste of instructional time or the danger of unreliable data of little importance. But the student (and their parents) may take a different view.

My daughter, for example, has dysgraphia (a relative of dysexia) that makes it difficult for her to spell. For years, teachers using 'passing back' marking caused my daughter considerable embarrassment as the students marking her work invariably told others in the class laughable examples of my daughter's poor spelling. As a consequence, all her peers believed her to be a weak student (they of course phrased it in less complimentary ways) and their taunting caused her no end of difficulties. All because (a few of) her teachers shirked their responsibilities for marking. And she is just one example -- by definition, every classroom has students who perform badly for one reason or another (including lack of ability) and they all have the unconditional right not to be humiliated. If students choose to share their scores with each other that is one thing; but it is unethical and unprofessional for a teacher to disclose a student's grade to the class or to allow students to view other students grades--the practice is simply unacceptable.

So it is a topic that I hammer on pretty heavily in my evaluation classes. So when I found a student teacher doing "passing back" marking, I of course questioned her. To my utter astonishment, instead of saying she had to because her Teacher Associate insisted, or admitting she'd slept through that part of the evaluation course, she said that she thought we had told her to include peer assessment in her evaluation strategy! As if 'passing back' marking constituted peer assessment! After I picked my jaw up off the floor, I tried to explain the difference between having kids scoring a quiz for the teacher, and students providing constructive feedback to each other through portfolio conferencing, workshops, rubric design, etc., etc., etc. Not sure how successful I was. Generations of teachers have modeled bad evaluation practice; it is an uphill battle to move the profession towards more professional practice. If all this student teacher has ever seen is 'pass it back', how could she know to, or how to, organize her students for peer evaluation?

Textbook Proposal

Pitched assessment textbook to a publisher this week, had it tentatively accepted. Title, The Cheap, Simple, Straight-Forward Guide to Student Evaluation, says it all.

But honestly, it drives me crazy when text salesmen come around and want to sell me a text that sells for $140+ (and god only knows how much markup the u bookstore will add to that!), of which I can only use three chapters in my course, and which the students will never open again once the course is over. Most evaluation texts devote hundreds of pages to theory which no classroom teacher needs, but which ed psych profs insist on teaching anyway. My attitude is that most of that stuff can wait till grad school. What pre-service teachers need is just enough practical hands-on skills (how to write a multiple choice question, how to set an essay assignment, how to handle oral questioning) to survive practicum and the first two years of teaching. The sort of info I offer on the website.

I figure we can sell the text for under $25 and it will be something students would actually keep around and consult for at least their first couple of years of teaching. I want to see my text on teacher's bookshelves, preferably dog-eared from their constantly consulting it, not in student's trash at end of term. I'm guessing the e-version of the text could be even cheaper -- maybe as little $5. I would have no hesitation asking students to spend $5 on a text; but I cannot bring myself to ask them to spend hundreds on the uniformly pompous, tedious, bloated texts that are currently out there. Up to now I have simply refused to buy texts and just used a custom learning resource (i.e., course reader printed by bookstore) but the price on that has kept creeping up, and as our university is one of those refusing to sign off on ACCESSs new copyright agreement, books of readings have become almost impossible to produce. So time for me to produce my own cost-effective text.

I know it's what I want in for my class, and my publisher is betting that once its out there, other profs will use it too. Or if not, their students will certainly buy it, whatever the official text for the course.

Because in addition to a focus on what preservice teachers really need to know, I intend for the book to be readable -- plain, readable English with touches of humor. I can't stand how pompously serious all the other textbooks are. The last time I took a text to a major academic press, they made me edit out nearly all the humour. This makes no sense to me, but they said profs on other campuses reviewing the manuscript all complained that they felt they were teaching a serious course and needed the text to be serious. Because I had a coauthor, I had to go along with deleting huge chunks of what I considered to be the really interesting bits. That's so wrong! Students want entertaining writing. "Serious" so often translates as "pompous" and "tedious", that it kills students' love of the subject. They want to see the same passion in the text as they get from good profs. (I sometimes wonder if those profs who rejected the funny bits in that manuscript felt threatened by the text being more entertaining then they were....) The one article I went to the wall for was a provocative essay -- not funny as such, but totally in your face outrageous attack on standard interpretation of things. They let me keep that chapter in. So the text itself went by the boards as soon as my colleague and I stopped teaching that particular course (I changed campuses) but my provocative essay, I am pleased to say, is still reprinted in course readers across Canada, and there isn't a conference I go to that someone doesn't come up and tell me how much they love that article and how refreshing it was to read something not dull.

So, all those factors coming together -- realizing I've made far and away more money from the reprint of that one provocative article over time than from all my other textbooks and chapters put together; annoyance over the weak selection of outrageously overpriced options available to me from mainstream publishers; and the collapse of ACCESS agreements -- have made mesee it's time for me to write my own evaluation text.

If it goes over well, the Cheap, Straight-forward, Provocative Introduction to the Sociology of Education could be next.