Question writing

I appreciate when instructors attempt to relate their abstract course content to real-world situations. Math teachers who just come in and cover the board with numbers—without any explanation of when one might need to deploy that equation—do a disservice to math instruction. You have to teach Z-scores in the context of when you'd need to use Z-scores, or whatever.

But by the same token, you need to have some passing familiarity with your real-world example. "If a train leaves Pittsburgh at 4PM going 100mph..." was fine when most people traveled by train; now, not so relatable. And if you going to use, "If a plane leaves Pittsburgh" you'd better include, "not counting the time spent clearing security or the delay for the flight crew showing up" as part of the problem.

More States Opting for Robo Grading of Student Essays.

This article from NPR on the use of software to grade student papers refers to a trend that is so appalling, I almost don't know where to begin. I was for a decade the person responsible for one of the standardized test programs in Alberta, and we were very proud of our involvement of classroom teachers at every stage of the development and marking process so that the exams would reflect actual classroom practice and the results would be normed accurately across Alberta. We trained teachers to use rubrics to mark accurately and consistently so that it wouldn't matter which marker the student got or what time of day the marking happened, or other chance factors. We did not train them to be robots, but designed rubrics that allowed them to award excellence, even when the student's answer did not easily fit within the rubric. If more than a few answers didn't fit the rubric, we changed the rubric. Routinely.

In social studies, we eliminated bias by making it explicit: when a marker found himself grinding his teeth over a student's stated opinions, instead of trying to grade it himself, he would hold the paper up above his head and shout "right-wing nut-job" or "left-wing snowflake" (as the case applied) and a suitably left or right oriented marker would happily swap it for its opposite, so that every paper was marked by a sympathetic marker. When a paper managed to offend everyone, it was taken off the marking floor and sent to a special committee of veteran markers who would grade it as a team. We went out of our way to not be robots.

This is NOT the case with many standardized tests, particularly in the US where it is private publishers rather than the ministry of education designing the exams. There are so many things wrong with most standardized testing programs, I will limit myself in this post to the observation that the purpose of most standardized testing programs is not to educate, nor to reward diligent learning, but to reproduce the class structure. Most tests reward the culture of the white upper-middle-class, and screw everybody else. They are there to explain inequality by saying, "Well you had your chance, but you only got a 58% on your test, so you deserve to spend the rest of your life in the underclass, unlike Frank here, who got 98%"(because we asked Frank questions in a way that makes the most sense to most white males, about stuff that matters to the professional/management class that Frank grew up in and had picked up on from his professional parents by the time he was 8, and because Frank had money for tutors, books, computers, if he happened to turn out a bit 'slow').

So...replacing teacher-markers with robots is perfect. Relying on software APPEARS to increase objectivity by removing the last vestiges of human intervention (what will be identified as 'bias' as these programs are being implemented) and keeps conscientious teachers from giving an "A" to a paper for its ideas when the student has written "ain't never" which is grammatically incorrect--i.e., has written her answer in the perfectly clear dialect of a to-be-suppressed population. A good teacher knows quality writing when they see it; a software program has the algorithms to suppress the underclass.

Not why I became an educator.

Bringing Notes to the Examination

A meme downloaded from "S**t Academics Say" Oct17, 2017:

There are several points to be made here.

First, good on the examiner for recognizing that was on them, not the student. I detest when an instructor makes a mistake and then tells a student, "well, that's not what I meant!" Unless mind reading is criteria in your psychic examination, it is inappropriate to hold students accountable for the instructor's intentions. If they got the literal answer to one's poorly worded question or followed one's poorly worded instructions literally, they get the mark or behaviour.

Second, you have to love this student! That is out of the box thinking! Exactly what every discipline needs more of.

Third, open book examinations are actually harder. Okay, if it's closed book for everyone else and open book for you, that does give one a slight advantage, but not as much as most people assume. There is relatively little knowledge that we need to memorize, as opposed to understanding and having available to us on our computers or google or etc. In most disciplines, one ends up memorize the important facts because that's easier than looking them up every time. If the fact is one you have to look up, it is by definition one that did not come up enough to be worth memorizing. Exams should be testing understanding, not just rote memorization. If the exam is testing higher orders of cognition (analysis, synthesis, evaluation and so on) then making information available to students makes for a better test of skills than the usual rote memorization that many exams seem to be geared towards.

Do Testwise Strategies Really Work (Testwiseness Part Three)

Hell, yeah.

Here is a personal anecdote from back when I worked as a full-time Test Development Specialist.

I had been working for a few years as a Test Specialist when the senior manager suddenly realized that there was only one person on staff who knew how the software for grading our exams worked. As it occurred to him that the whole operation could grind to a halt were something to happen to that individual, he transferred me into the computer programming section as the backup programmer. (This made sense to him because I was the only person back in those days—30 years ago!—who owned his own personal computer, so I looked like a computer guy to him.) Since I had no programming training, and since all our computer programs were written in FORTRAN, my boss signed me up for a university-level correspondence course on FORTRAN programming. By the time the course materials had arrived in the mail two weeks later (this before online courses!), however, my boss had come to his senses, hired a team of FORTRAN programmers, and switched me back to writing tests. I therefore had no further need to learn FORTRAN, and I pushed the unopened box of course materials under my bed and forgot about it.

Six months later I received a notice in the mail saying the examination for the FORTRAN course would be at such and such location in a week's time. And I thought, "Well, the course has already been paid for, why not take the test?" Not, you understand, that I had ever opened the box or looked at any of the course materials. I just wondered if I could pass the test on my knowledge of test design.

The examination, if I recall, was 70% multiple-choice and 30% written response. I had no hope of answering the written response part, since I had absolutely no idea how to write a FORTRAN program. I did repeat back the question (because some markers wrongly give at least one mark for a student filling in the space with something) and attempted a line or two of code based on examples from the multiple-choice part of the test, but since I had no idea what it was I was copying, I highly doubt it made any sense at all.

The multiple-choice questions, however, were a different matter. As a test specialist, I was able to examine each question for the tiny flaws that gave the answer away. Examining the alternatives for the odd one out, or for one answer that was significantly longer than the others, and so on, (see previous post for these techniques) I gave it my best college try, even though I often had no idea what the question was asking, let alone what the correct answer might be.

Three weeks later I received a letter saying I had obtained 70% on the test; which considering I had left the written response essentially blank, means I must have aced the multiple-choice. Not bad for a topic I literally knew nothing about!

I subsequently wrote the institution in question pointing out this weakness in their tests, and sometime later they hired me to give a one-day workshop on the do's and don'ts of multiple-choice item writing. There is, therefore, no use asking me how to register for that course; these techniques won't work to pass that test again!

This was probably an extreme case; and, to be fair, such tests are not intended for people who make their living designing multiple-choice tests. Most people would not have found the flaws quite so obvious.

In sharp contrast, I sat down with a copy of Alberta's Grade 12 Mathematics Diploma Examination to see what I could get using these test-wise tricks. In this case, I only managed 23%, slightly less than one would expect by pure chance. This test had been written by test experts and it was impossible for me to get through it without actually knowing the course content (calculus) which I did not.

Most classroom teachers' tests are somewhere in between these two extremes. A student will not likely be able to pass a teacher-constructed exam solely on knowing testwise strategies, but such strategies are likely to improve the testwise student's grade a by significant percentage if the teacher does not know and implement correct test construction technique.

---

In that context, it is worth noting that teachers should be cautious about using test banks that come with textbooks. Some of the larger textbook publishers do employ professional test developers as editors for their test banks, but many do not. Test banks are sometimes written by the author of the textbook (who likely will not have any expertise in test-construction) but are more often written by graduate students from that discipline desperate for cash (and who definitely do not have test-construction skills) and dashed off over a weekend before the publisher's deadline. In other words, don't assume the test bank is worth anything. At a minimum, any instructor using a test bank written by anyone else should edit the questions before putting them on their test to ensure the questions are less flawed and more relevant to their own classroom. (On the other hand, starting from a flawed test bank is sometimes easier than writing one's own items starting from a blank page. Using a test bank as initial rough draft can be okay, provided one guards against it being all rote memorization. You will probably need to write your own higher level thinking questions, because publishers can't pay grad students enough to come up with those!)

Why Teach Students to be Testwise (Testwiseness Part Two)

Some instructors may be horrified at the suggestion that we teach students test-taking strategies, let alone testwise strategies. Allow me to suggest why teaching students some testwise strategies is a good thing.

1. It levels the playing field

   Some students already know slogans like "when in doubt, choose c" even if they don't know the rationale behind it. If some students are testwise and others are not, it is not a level playing field. The assessment becomes inaccurate because the test is now measuring testwiseness rather than the skills and knowledge intended, since students who are testwise can significantly increase their scores (on poorly constructed tests). Instructors need to ensure their assessment is measuring learning outcomes not cultural capital. Teaching everyone the same basic testwise skills levels the playing field by ensuring everyone has the same knowledge.

2. It reduces test anxiety

   Students often do poorly in testing situations because of test anxiety. Students with test anxiety often feel they have no control over the many variables that make the outcome of the test uncertain: which subtopics will questions be drawn from, how hard each question will be, how long it will take them to remember and answer or solve for a question, the reading level of the question, the ambiguity of the question stem or of the answers offered, and so on. Having some strategies available for topics about which they know or recall little significantly reduces overall anxiety because it gives the student a sense of control. "Well, at least I know what to do if I have no clue what the answer is!" As part of a lesson on how to study for and take multiple-choice tests (two previous posts on this blog), giving students some testwise strategies goes a long way towards lowering (though never eliminating) test anxiety.

3. It forces teachers to write better multiple-choice tests

   Testwise strategies only work on poorly constructed tests, so if students know testwise strategies, teachers are forced to develop the test-construction skills, and to take the time necessary, to write the better tests necessary to defeat testwiseness.

Students must always be cautioned, however, that the only way to do well on tests is to learn the course material, and that they should not rely on testwise strategies. It is natural for some students (especially in middle school) to embrace testwise strategies as a substitute for studying, so it is important for instructors to note that these strategies will not work on one's own tests, because one's own tests are properly constructed.

Sadly, I have also encountered some instructors who take teaching test-taking strategies to the point of emphasizing these strategies over actual curriculum (i.e., teaching to the test). Such teachers are thinking like a middle school adolescent and should be relieved of their teaching certification.