Memes: Rote Memorization

This answer provides diagnostic information: this student is relying on memorization instead of understanding, to pass tests; this requires you to change instruction to deemphasize rote memorization and testing and choose assessment strategies that encourage learning

Pinterest "Assessment" Board

I have a Pinterest board on "Assessment" that I update frequently...usually taking some Pinterest/Tumblr meme and discussing the assessment principle involved. I find I do this more often than blogging these days, but I will continue to post longer arguments here. But check out my Pinterest posts too.

Late Penalties

I was recently asked by a colleague what my policy is on late penalties. This is, in part what I replied: What I tell my students is, if you submit it before I've gotten around to marking the rest of the class, you're in the clear. That way I don't hold them to a higher standard than I am able to meet myself. It drives students crazy when they have to get an assignment in by such and such a date, and then watch the pile of unmarked papers sit on the professor's desk for four weeks. What was the point of imposing that deadline?

(Instructors often seem quite willing to make excuses for themselves—life happens—while denying the same right to their students, who may have the same or greater out-of-class responsibilities, and considerably less resources to bring to bear to resolve issues then their instructors.)
Photo reflects very BAD assessment practices.

From an assessment point of view, there is no logic to time limits on learning, save in those few subjects where automaticity is in fact a criterion. If it takes one student two days longer to figure out a concept, does that make their understanding of the concept less valid? The argument that the extra time gives the late comer an advantage over those who made the deadline is only valid when normed referenced grading is used, which I would argue is largely inappropriate anyway. Criterion referencing is more valid in most situations, so unless timeliness is a justifiable criterion, I can't see why we have any deadlines, except the logistically necessary ones of "end of the course".

I tell my students that due dates are guidelines for managing their workloads, and that if they miss the deadline I can't guarantee they will received the assignment back before next one is due. So it's in their interest to meet the deadline, and 99% do. The 1% who are late usually have a good reason (family priorities, for example) and it is oppressive of us to arbitrarily and inaccurately certify that they have learned less and performed less well then they really have, just because they missed some due date. For example, many of my students are single moms—why should I deduct 10% from their grade because their child got sick the week the assignment was due? Similarly, many of my students are First Nations, and have responsibilities to an extended family. I would argue that penalizing such students in this way for prioritizing family over school is sexist, racist, ageist and etc., etc. because the young white males living in their mom's basement and having their meals and laundry done for them have much less difficulty meeting deadlines. Deadlines and penalties may have been partially justified when students were full-time students only, but the reality is that our students have jobs, families, responsibilities and lives outside the classroom, and insisting that our course be the most important thing in their life is just egocentric nonsense.

The Multiplayer Classroom: Designing Coursework as a Game

Interesting book on designing courses like designing video games here: http://www.facebook.com/MultiplayerClassroom The site / text was recommended to me by a colleague as coming from an author with impeccable credentials, so I pass it along here

Wrong answers....

My daughter was laughing her way through Richard Benson's "F in Exams: The Very Best Totally Wrong Test Answers", which is a pretty funny read, and asking me what I thought. Well, mostly I thought the answers were amusing, and the questions that illicited them poorly written. If you ask students, "Where was George Washington born" and they answer, "in a bed", you cannot mark it wrong. It correctly answers the question asked. If you want the name of the city, you have to ask "In what city was Washington born?" And so on. But half the answers in Benson's book are in fact wrong answers, but funny nevertheless, either because they are cheeky, or because they simply reveal an understandable misunderstanding. As it happens, I came across just such an example as I was marking my own exams at that moment. A student who meant to refer to the educational limitations of "rote memorization" wrote instead "rogue memorization" -- an unfortunate slip of the pen, or perhaps a genuine mishearing of what the student thought his/her profs have been saying. But I kind of like the phrase! Yeah, you really don't want your students dwelling on rogue memorization. Given the randomness of some of the answers to the questions posed on the exam, I greatly fear some of my class have in fact experienced rogue memorization when studying for the exam....

Questions Answered.

Q: How many alternatives should a multiple-choice question have?

Unless you have a specific subject that demands more than five possible alternatives (say, doing a unit on the solar system, having all 8 planets as alternatives might make sense -- though a matching question would probably work better!) don't do it! The more alternatives, the worse your question is likely to be. Four or five alternatives are standard on professionally designed test (since statistically, this reduces the score students could get by pure chance to 25 or 20% respectively). Some people want to pile on alternatives to make the questions 'tougher' and to 'eliminate chance'. But here's the thing -- having 7 alternatives does reduce the chance of them getting the question right by blind guessing, but why bother? Once they've gotten less than 20%, how much more failed do they need to be? And while theoretically reducing the impact of chance on getting the right answer, the more answers the student has to read through, the more it becomes a readiing test rather than a test of that subject matter. So do we really want to eliminate chance factors by penalizing poor readers, ESL students, and so on. Most test designers agree the trade off isn't worth it!

And in the real world, coming up with 7 or 8 credible alternatives becomes REALLY hard. Again, unless there are an obvious 8 possible choices like the 8 planets of the solar system, you will drive yourself crazy trying to come up with credible but clearly wrong answers six, seven and eight. Why do this to yourself?

Or, people will do terrible things like having alternative 5 as "A and B, but not C". No, no no! ? Never do this! It becomes a test of reading and logic rather than subject knowledge. Students will hate you with justification since the test will not be an accurate reflection of what they know -- indeed, some research suggests that this INCREASES the importance of luck...

But here's the killer -- research in the early 1990s suggested that the overall quality of tests DECLINCED with the increase in number of alternatives per question. Everyone who has ever designed an mc test knows that coming up with the answer is easy, the first two wrong answers pretty easy, it's alternative 4 that's tough, and #5 is almost impossible -- the higher you go, the more desperate one becomes to fill the last spot. So, a test with seven alternatives will reduce the writer to grasping at straws, and they will end up accepting ridiculous alternatives that even those completely ignorant about the subject will have no trouble eliminating -- a complete waste of space and student reading. And then -- this is where human nature gets interesting -- since I've given up and accepted a stupid alternative for this question in desperation, my standards for writing the next question go down, because even though I know this is a terrible alternative, it is not as bad as the last one. Or, having accepted three weak ones, what's one more? Pretty soon, the test is garbage.

In contrast, tests with 3 alternatives (the right answer and two wrong alternatives) turn out to be easier to write, and therefore are written to a much higher standard. Students perceive them to be much tougher tests! And, research says, they really are more valid and reliable! So, I tell my students to write questions with three (good!) alternatives rather than going for four or five. Professional test designers can go for four or five because we have the time to come up with high quality 'd's and 'e's, but the realities for classroom instructors is that that is not going to happen.

It's true that with only three alternatives, students can get 33% just by blind luck, but um, so what? I don't know any course where 33% is a pass. Failed is failed. And the results of this test will more accurately reflect what students actually know than one with 7 or 8 alternatives.

Questions Answered.

From time to time strangers email to ask a question on test construction, which I do my best to answer. If they are the sort of questions that I get a lot, I add them to the "Frequently Asked Questions" file on the test construction site; but I think I'll highlight some of them here in the blog as well.

Q: Where is the best place to put the correct answer? For example, if I provide seven choices, does it make a difference if the correct answer is choice 'b' instead of choice 'e'?

A: Professional test designers place the answers randomly -- I mean that in the literal statistical sense of the word, not 'wherever'. They use tables of random numbers, or complicated computer programs that assign the answer randomly, to decide which spot will hold the answer to each question.

What they do NOT do is place it themselves. Research shows that left to our own devices, most people will attempt to 'hide' the correct answer somewhere in the middle of the list. (Nobody wants to put the right answer in A, because then the students won't even read the other alternatives you worked so hard on; and putting it in 'e', it just sort of seems to hang out there over the edge. Sticking it in the middle feels right! Even though, that's wrong.) Even experienced test construction professionals will unconsciously choose 'c' (or for some individuals, it turns out to be 'b') 3/4 of the time. That's why the rule for taking an mc test is "when in doubt, choose 'c'" -- because unless one takes care to distribute correct answers to get an equal distribution of A, B, C,D, etc, there will be way more 'b's and especially 'c's than other answers, so testwise students can do quite well for themselves simply by answering 'C' to every question. That's why professionals force themselves to do it randomly by using computers or tables of random numbers. And then they'll double check at the end of the test to make sure they have roughly equal number of a, b, c, d, etc.

For classroom instructors etc, I wouldn't bother with tables of random numbers (which are kind of a pain to work with) and let the answers fall where they may by pyramiding questions. To stop students from trying to figure out which answer will come next ("there have been three 'd's in a row so next one must be something else") you let the internal logic of the question dictate placement. Numerical answers are listed in ascending or descending order; dates in chronological order, single word answers are listed in alphabetical order; sentences either on the basis of some internal logic or more usually shortest to longest or longest to shortest. (Incidentally, this also makes the test look really pretty! People who don't pyramid their tests have really ragged looking questions). So if the correct answer turns out to be the longest, it places itself in the 'e' slot, not letting the designer 'hide' it in the middle 'c' spot. After the initial draft of the test is done, one quickly looks through to ensure one has equal numbers of a, b, c,s etc. Where there are too many of one, say 'A's, you go through and change some of the ascending questions to a descending to move the 'A' to a "D" or whatever. It works pretty well!