Tricky Directions in Test Questions

This is another meme doing the rounds on the web:

I think we can take it as read that "Draw a dinosaur wearing spectacles and holding a pen" does not relate directly to course objectives. I am a somewhat open to the idea of demonstrating to students the need to (a) read what the question is asking carefully and (b) the importance of showing one's work in such examinations, but um...hoping this was for an ungraded assignment (in class or homework) not a question on an actual exam. As a practice exercise, one of these a semester is fine—a little levity goes a long way, and there is a point to be made. [Doing this more than once would be bad: once is about student learning; twice is about the instructor projecting their self-image as a cool teacher. I think it safe to assume the current example is a one-off by a superior instructor, not role modelling a routine to be adopted by other, less grounded instructors.] So, no harm done if used sparingly and only in a non-graded context.

As an actual exam question, humour is a no-no. Because the examination is supposed to be targetting legitimate curricular objectives: unless your science curriculum includes "demonstrates a sense of humour" or "fine motor skills in the production of art" questions such as this example are out of line. Further, although the instructor may believe the injection of humour is a way to reduce stress, the fact is that at least some students will be panicked by it. Knowing in their hearts that an examination is no place for levity, their reaction to encountering it is likely to be disbelief and the assumption that they have misunderstood the question. In this instance, they will waste precious seconds obsessing about the meaning of the dinosaur. How does the height of the dinosaur change the question? Is the question now 32 meter's height, plus the height of the dinosaur? What is the significance of the pen? the Spectacles? Such superfluous information can drive a student crazy. And that's for normal kids. Throw in say, OCD or second language issues, and one is screwing up the accuracy of the assessment, not only for that question, but all the other objectives being examined that had that fraction of time less to think about.

So, occasional funny questions on practice exercises, maybe. Understand that some students will share your sense of humour and respond well, but others will not. But that can be addressed in class if the class atmosphere is a positive and safe one. On an actual for grades test or assignment, never.

Questions Answered.

From time to time strangers email to ask a question on test construction, which I do my best to answer. If they are the sort of questions that I get a lot, I add them to the "Frequently Asked Questions" file on the test construction site; but I think I'll highlight some of them here in the blog as well.

Q: Where is the best place to put the correct answer? For example, if I provide seven choices, does it make a difference if the correct answer is choice 'b' instead of choice 'e'?

A: Professional test designers place the answers randomly -- I mean that in the literal statistical sense of the word, not 'wherever'. They use tables of random numbers, or complicated computer programs that assign the answer randomly, to decide which spot will hold the answer to each question.

What they do NOT do is place it themselves. Research shows that left to our own devices, most people will attempt to 'hide' the correct answer somewhere in the middle of the list. (Nobody wants to put the right answer in A, because then the students won't even read the other alternatives you worked so hard on; and putting it in 'e', it just sort of seems to hang out there over the edge. Sticking it in the middle feels right! Even though, that's wrong.) Even experienced test construction professionals will unconsciously choose 'c' (or for some individuals, it turns out to be 'b') 3/4 of the time. That's why the rule for taking an mc test is "when in doubt, choose 'c'" -- because unless one takes care to distribute correct answers to get an equal distribution of A, B, C,D, etc, there will be way more 'b's and especially 'c's than other answers, so testwise students can do quite well for themselves simply by answering 'C' to every question. That's why professionals force themselves to do it randomly by using computers or tables of random numbers. And then they'll double check at the end of the test to make sure they have roughly equal number of a, b, c, d, etc.

For classroom instructors etc, I wouldn't bother with tables of random numbers (which are kind of a pain to work with) and let the answers fall where they may by pyramiding questions. To stop students from trying to figure out which answer will come next ("there have been three 'd's in a row so next one must be something else") you let the internal logic of the question dictate placement. Numerical answers are listed in ascending or descending order; dates in chronological order, single word answers are listed in alphabetical order; sentences either on the basis of some internal logic or more usually shortest to longest or longest to shortest. (Incidentally, this also makes the test look really pretty! People who don't pyramid their tests have really ragged looking questions). So if the correct answer turns out to be the longest, it places itself in the 'e' slot, not letting the designer 'hide' it in the middle 'c' spot. After the initial draft of the test is done, one quickly looks through to ensure one has equal numbers of a, b, c,s etc. Where there are too many of one, say 'A's, you go through and change some of the ascending questions to a descending to move the 'A' to a "D" or whatever. It works pretty well!