Equity & Fairness of Multiple Provincial Exam Forms
On March 13th to 21st 2006 the individual student provincial exam results will be released. This year when students in the Nanaimo school district walked out of their provincial exam and compared notes on how they did, some discovered that they wrote several different exam versions for a given subject. In the 2005-2006 hand book of procedures, under the Security section, the following is listed:
Beginning in 2004-2005 with Grade 10 Graduation Program examinations, the Ministry began using enhanced statistical methods to equate tests and improve fairness and equity for students from one examination session to another. In 2005-2006 these statistical methods will be applied to exams in courses that meet the 2004 Graduation Program social studies requirement and to selected Grade 12 examinations. The goal of test equity can only be met if selected test questions, called “anchor items” are re-used.
The reason the Ministry chose to do this are listed:
• ensure equivalency of scores from administration to administration and across years
• permit more frequent administration while limiting development cost
• permit rapid response to emergent situations
• provide a basis for multiple forms of examinations to support electronic-examination delivery.
A mark is generally understood as a measure of knowledge gained. It indicates that if a student receives 60% then they should have an amount of knowledge equivalent to 60% of the learning outcomes. Let’s look at the example of two students with identical capabilities. One student writes version A—a version measuring the appropriate level of knowledge—and the other student writes version B—a version measuring different learning outcomes than expected. The marks for the student who wrote version B are then statistically post processed to compensate for the discrepancy in learning outcomes being measured. For the student writing exam version B, is this an accurate assessment of the knowledge they have acquired? Does statistical exam mark post-processing decrease the importance placed on exam design quality and exam validation? Is it fair to students when they don’t know whether they wrote exam A or exam B, whether their mark will be post-processed or not, how it will be processed, and by how much?
The Ministry of Education has indicated that exams can be equated to one another, within and across exam sessions, through the linking of a relatively small proportion of common questions and through looking at the relative difficulty of those questions, based on students' responses. This does not account for the situation where “the small proportion of common questions” may be completely missed by students who decide to skip or speed right through them if they are writing a difficult version of the exam. In this case, their comparative base score on the common questions will be lower. This can be further aggravated by the psychological impact of watching one’s classmates leave early, wonder why they are having more difficulties, and get flustered.
Also there is the issue of exam content bias. Recently an English exam had a section that used specialized terminology. For example, let’s assume a section was about basketball and that it used basketball specific jargon. In this example, one would expect basketball players and fans to do better than others. There is bias in the exam that favours a group of students. This bias cannot be corrected by statistical methods unless one knows which students are familiar with the basketball jargon.
The mark post-processing and the exam multiple versions are intended to ensure fairness in evaluation and treat students the same over time, but let's imagine that the first round of grade 10 exams or this year's grade 11 socials exams were unfair to some students because some teachers were not prepared to teach the provincial prescribed learning outcomes being evaluated by the provincial exams. Will the Ministry retro-actively correct the marks for an unfair exam implementation? I doubt it; this hardly seems fair or equitable.
Labels: Provincial Exams
posted by Simone @ 12:44 a.m.
11 comments
11 Comments:
I agree...I am a gr.12 student. I am wondering how they can accomplish their goal of figuring out how well everyone is doing, if they give out different exams in the same semester. Isnt the goal with the provincial exams to compare a level of knowledge? to test a level of knowledge based on information that everyone knows? personally, I disagree with the differentiating exams of one session. I finished my gr 12 english exam, and found it was very different from the practise exams that i had been given. I had 91% in class before the exam, I got 65% in the exam, so I am now down to a final grade of 81%. I am wondering how this can be a good evaluation of classroom knowledge.
Hey Grade 12er. That's a big gap.
Just so you know, your teacher will not have any idea of what was given on the provincial exam as this year the teachers were not allowed to have a copy of the exam–never mind even looking at them according to the new security rules. District officials have also indicated that teachers are not given any feedback on which multiple choice questions students may be failing, therefore district teachers are unable to determine which topics to improve covering with their students. This said, the teachers in this district who contract themselves out to the Ministry to correct the essay portion of the exams will definitely have an advantage in knowing what is on the essay portion of all versions of the provincial tests (as will their students) over the rest of the district teachers (However, these contracted teachers cannot correct their own students' exams).
If you are university bound to the big universities, their provincial exam performance requirements will be a show-stopper for you. Check out the University Presidents' Council table 2.1 for university GPA quota cut-offs. (http://www.tupc.bc.ca/facts_figures/pdf/tupc2.1.pdf)
Although some of these universities will drop these quota GPA cut-offs progressively to fill their seats based on student demand, this will not happen for programs that are in high demand. This table is updated annually, the GPA cut-offs are based on student demand (demographics) and government-funded supply of university seats. These quota cut-offs are generalized for groups of programs, therefore you should review the specific program requirements at the institution that you intend to apply to.
So what do you do now? You probably already know that you are able to take the provincial exam again (one rewrite) by contacting your school and letting them know when you want to take the exam again. It will be tough on you though if you also have to prepare for your other school and provincial exams at the same time in this semester. There is also a summer exam session but this may cut into your future plans–not to mention your summer career or other personal plans.
Failing to obtain a grade that is somewhat in line with what you accomplished in class, there is a process by which you can request a reread of your exam. However, even if the reread gives you a higher score, it has to exceed the statistical error in measurement to get your mark changed. Don’t get me started on a discussion about the statistical error in measurement, but, in a nutshell, if the statistical error is high for this session of the exam, you are hooped!!
It will also cost you $50, and there are also some pretty tight deadlines for applying for a reread that you should keep in mind before you decide to go that route.
These are all outlined in the Ministry of Education Handbook of procedures for the Graduation program: http://www.bced.gov.bc.ca/exams/handbook/handbook_procedures.pdf
Use the PDF search function to look up all the sections on rewrites, rereads and locate the information you need.
I’m in the opposite situation.
I’m in grade 10. I got 66% in my class and I got 87% on my exam. My final mark was 70%. None of my classmates had the same exam so I couldn’t compare, everybody else said their exams were really hard. Even the top students had a hard time. To me the exam seemed really easy compared to the class tests.
The multiple tests really mucks things up.
Well 10 grader…
The following should give you a place to start to examine possible reasons for the difference. There are three possible explanations that I can think of for the difference you experienced:
1) If test scores in your course work were consistently around 66% then either the exam was way off or your teacher has been hard on you or your class.
2)If your test scores in your course work were not consistently around 66% but higher, then you probably bombed a test that had a high weighting.
3)The last possibility is that you knew the content well of the questions that happened to be on the exam and/or your review of sample exams paid off in your version.
This should help you determine whether you know 66% or 87% of the learning outcomes.
It would be good to ask your classmates in that class about how different their exam scores were from their class scores. Even with different exam versions, if the Ministry “equating” procedure is working as they claim it should work, and if your classmates prepared in the same way you did, then the exam marks of your classmates should have been equated across exam versions and should hopefully be somewhat consistent with their class performance.
It would also not hurt to bring the results of your provincial examination to the attention of your former teacher and ask for possible reasons for the difference. The teacher likely will have other suggestions to add to this list.
Correction to the grade 12 student:
Your teacher will have access to the exam "after" the exam has been administered if the principal at the school has established a system for secure access by teachers who wish to review the tests.
Also please note there is more material on rewrites/rereads as well as the ability for students to review exams at the following website: http://www.bced.gov.bc.ca/exams/rereads_rewrites/
Also the deadlines are still very tight (end of March I believe) for applying for a review and you may not get to review your exam before the next examination session.
Good luck!
For those with an interest in further public debates on the merits and short-comings of provincial exams, the Tyee—an independent alternative BC daily newspaper—has provided a couple of articles with many insightful commentaries on both sides of this issue in 2004 and 2005.
The first article (2004) is by Judith Ince; a Vancouver-based journalist with a special focus on education:
Student Testing at Breaking Point?
thetyee.ca/News/2004/12/06/StudentTesting/
The second article (2005) is by David Russell
Expel the Grade Ten Exam: Reason One: The unreliable way it’s graded defeats its own stated purpose.
thetyee.ca/Views/2005/06/20/ExpelExam/
The following talk may interest the readers of this blog post.
THE DEADLY EFFECTS OF "TOUGHER STANDARDS" IN EDUCATION: Challenging High Stakes Testing and Other Impediments to Learning."
Mr. Kohn will be speaking with parents, teachers, students, and the public at large about the risks of standardized testing and accountability in education.
Date: Thursday, May 4, 2006
Time: 6:00 PM
Location: ICBC Concourse Room at the Wosk Centre for Dialogue,
580 West Hastings, Vancouver.
More information about this event can be found on the event webpage at www.educ.sfu.ca/events-1/alfie-kohn
I’ve noticed testing has gotten much worse over the recent years, but the worst part is that some teachers have made it incredibly difficult to review your test. They make you have to schedule an appointment after school to go over your test. They have their own reasons such as fear of cheating but it getting ridiculous. How are you supposed to learn from your mistakes if you can’t see what you did wrong?
You have asked a very good question, but one for which there is no satisfactory answer. Most students (and parents) would probably appreciate being able to bring the exam home so that they can learn from their mistakes and use the exam for review for their final examination. Apparently, the reality is that teachers have the authority to administer the exams and not allow students to review them at home (just like the Ministry is doing with the provincial exams). In my view, this is a grey area which the Ministry and the BC College of teachers needs to clarify.
The Ministry already has some guidelines on these in some IRPs. For example, the new Math 10-12 draft IRP states:
“Students benefit most when assessment feedback is provided on a regular, ongoing basis. When assessment is seen as an opportunity to promote learning rather than as a final judgment, it shows students their strengths and suggests how they can develop further. Students can use this information to redirect their efforts, make plans, communicate with others (e.g., peers, teachers, parents) about their growth, and set future learning goals.”
I can see that students’ learning and achievement will suffer if the exam is not corrected in class and if students are not given a sufficient enough opportunity to review and learn from their exam with their parent, tutors, etc..., in particular, those students who, in general, experience difficulties with writing exams. I also agree with you that it is ridiculous for all students (or even half) to make appointments outside of class to review every cross grade exam with the teacher individually. Sadly, it is the students that are struggling and where the parents may not be aware (or not know enough to ask) that will suffer the most.
The BC College of teachers—the professional organization that regulates the teaching standards—have some teaching standards. Check out standard number 8 at http://www.bcct.ca/documents/edu_stds.pdf.
The standards are fairly high level and consequently are not clear enough in such situations. However, the standards do note that: “Assessment and evaluation practices have the potential to motivate students to greater success or create barriers to their success”. In my view, not allowing students to keep their copy of the cross grade exam falls into the “barrier” category.
On the bright side, the BC College of teachers is developing some criteria for each of these standards (these are due in April). These criteria may provide something more useful and perhaps educators will have a discussion about the best way of meeting the students’ needs while still providing the teachers with enough exams that they don’t have to worry about cheating and do not have to experience undue increases in their workload.
After all, if the district teachers for a given grade had about 10 different exams that they rotated through, and “if” a student did manage to get a hold of ten different exams and worked on all ten of them in preparation for their final examination, would that be considered cheating or exam preparation?
Testing has become a big business. One of the things that seem most important to the testers is engineering compliance. There is a great book by philosphor Bertel Ollman called how to write an exam and change the world at the same time. It's well worth reading.
Here's a coment that i wrote on my own blog about exams and testing:
Blog entry on testing mania.
i'm getting 55%in my claas will I fail my class Help ME.
Post a Comment
<< Home