Reinvent standardized tests for openness
Monday May 7, 2012
Identical tests kept secret until administration should be replaced by large open banks of test questions an appropriate sampling of which are faced by individual test takers. The only secrecy that should remain is in the particular combination of questions that the individual test taker faces when taking the exam.
It is always possible for test content secrecy to be compromised. There seem to be such problems now in California. It is claimed that breaches in security "could lead to invalidating test scores for entire schools or prevent the state from using certain tests." A system in which this can happen is not a robust system, and it is always possible for this to happen in a system that relies on test content secrecy. Perfect test content secrecy is not possible.
Limiting access to test materials before administering exams, especially when this means limiting it to companies motivated to produce tests at the lowest possible expense, does not effectively prevent problems with the exams. And since there is only one version of the exams, problems affect every test taker. This has been the case with English and math tests recently in New York.
A far better solution to testing is to curate large banks of questions accessible to the public at large. This would allow many eyes to identify problems with questions, and give every student a fair chance to prepare for the kinds of questions that will appear on their exam.
The number of questions should be large enough that the probability of a test taker encountering a previously viewed question is fairly small. A test taker, especially one who prepares voraciously, may recognize a question during an examination, but this can also happen with secret content tests, and in any event it is the result of the test taker's learning.
If the banks of questions are sufficiently large and rich, this method may ameliorate the problems of "teaching to the test". It should become clearly more efficient to learn general principles and problem-solving skills, rather than simply reciting every existing question. Of course some practice with questions from the banks for practice is not necessarily bad, but by itself such an approach is unlikely to yield optimal results.
Since taking the test does not yield any new information about the test anyone else will face, it becomes possible for test takers to take and retake exams at any time with no new costs in exam development. This eliminates problems of exam-day sickness and the like.
Appropriate scoring of exams may be more difficult since test takers do not face the same questions. An intelligent and open system of evaluating question difficulty should be developed and marking should be aligned with the purpose of the exam. It may not be that the goal of the exam is to perfectly rank every test taker, and it may be easier to determine if a test taker has demonstrated competency and is ready to move on.
The modern world is advancing where transparency increases, and so should it be with testing.
This post was originally hosted elsewhere.