Equity & Fairness of Multiple Provincial Exam Forms

On March 13th to 21st 2006 the individual student provincial exam results will be released. This year when students in the Nanaimo school district walked out of their provincial exam and compared notes on how they did, some discovered that they wrote several different exam versions for a given subject. In the 2005-2006 hand book of procedures, under the Security section, the following is listed:

Beginning in 2004-2005 with Grade 10 Graduation Program examinations, the Ministry began using enhanced statistical methods to equate tests and improve fairness and equity for students from one examination session to another. In 2005-2006 these statistical methods will be applied to exams in courses that meet the 2004 Graduation Program social studies requirement and to selected Grade 12 examinations. The goal of test equity can only be met if selected test questions, called “anchor items” are re-used.

The reason the Ministry chose to do this are listed:
• ensure equivalency of scores from administration to administration and across years
• permit more frequent administration while limiting development cost
• permit rapid response to emergent situations
• provide a basis for multiple forms of examinations to support electronic-examination delivery.

A mark is generally understood as a measure of knowledge gained. It indicates that if a student receives 60% then they should have an amount of knowledge equivalent to 60% of the learning outcomes. Let’s look at the example of two students with identical capabilities. One student writes version A—a version measuring the appropriate level of knowledge—and the other student writes version B—a version measuring different learning outcomes than expected. The marks for the student who wrote version B are then statistically post processed to compensate for the discrepancy in learning outcomes being measured. For the student writing exam version B, is this an accurate assessment of the knowledge they have acquired? Does statistical exam mark post-processing decrease the importance placed on exam design quality and exam validation? Is it fair to students when they don’t know whether they wrote exam A or exam B, whether their mark will be post-processed or not, how it will be processed, and by how much?

The Ministry of Education has indicated that exams can be equated to one another, within and across exam sessions, through the linking of a relatively small proportion of common questions and through looking at the relative difficulty of those questions, based on students' responses. This does not account for the situation where “the small proportion of common questions” may be completely missed by students who decide to skip or speed right through them if they are writing a difficult version of the exam. In this case, their comparative base score on the common questions will be lower. This can be further aggravated by the psychological impact of watching one’s classmates leave early, wonder why they are having more difficulties, and get flustered.

Also there is the issue of exam content bias. Recently an English exam had a section that used specialized terminology. For example, let’s assume a section was about basketball and that it used basketball specific jargon. In this example, one would expect basketball players and fans to do better than others. There is bias in the exam that favours a group of students. This bias cannot be corrected by statistical methods unless one knows which students are familiar with the basketball jargon.

The mark post-processing and the exam multiple versions are intended to ensure fairness in evaluation and treat students the same over time, but let's imagine that the first round of grade 10 exams or this year's grade 11 socials exams were unfair to some students because some teachers were not prepared to teach the provincial prescribed learning outcomes being evaluated by the provincial exams. Will the Ministry retro-actively correct the marks for an unfair exam implementation? I doubt it; this hardly seems fair or equitable.

On the outside...looking in

The French Immersion program in the Nanaimo Ladysmith school district 68 has grown by about 24% over the last five years with new French Immersion elementary schools being opened and increased elementary student enrolments. This choice program has been very popular with parents because of its enriched curriculum, the increased value of a second language in the work place, and because, let’s face it, some of us believe in the idea of Canada—all of it—English and French.

Many French immersion parents pursued this ideal with their children successfully over several years, but in the last few years they have encountered some (large) bumps along the way:

- Funding cuts and the removal of class size caps in the last few years have increased class sizes for both English stream and French stream students across the district,

- District staff positions that coordinated the French Immersion program were cut,

-The Ministry of Education added new provincial exams in grade 10 and 11 where French learning resources that matched the provincial exams were not in place,

- The demographics of an ageing population became a reality as experienced teachers started to retire, and

- A nationwide shortage of the supply of French teachers (with specific discipline specializations such as math) started to emerge.

The end results of the increasing bulge in enrolment, teacher shortage and new provincial exams was the prefect storm for the first large group of elementary students that converged into the French immersion high school a few years ago. Long term planning for French Immersion enrolments was not handled well for by the district and resulted in large high school classes of up to 34 students. This affected teacher morale, the quality of student learning, and increased tension between administrators, teachers, and parents. This stress was further magnified by the inability to replace French immersion teachers who had to go on leave this year in several high school grades. French immersion students were taught in English for some of their subjects but then some were expected to write their exam in French.

These events emphasized the need to pro-actively look at long term planning issues for French Immersion. Sadly, parents have been lobbying the district for a couple of years, to pay attention to these issues and have been told that things are being looked after or things are fine. Letters were written, board presentations were made, and many a PAC meeting’s agenda was taken up discussing these issues without being able to move forward. The prevailing environment has been not to include parents as partners in educational planning nor accord them any educational planning credibility.

Farcically, parents are regularly invited to attend Ministry sponsored technically impressive web casts where the honourable Minister of Education speaks about things like how crucial parent involvement is, and “perfect” districts where principals, teachers, and parents are truly engaged in dialogue are showcased. These presentations are usually peppered with PR messages on the wonderful things happening in K-12 education in BC. Although these sessions are supposed to stimulate parent and staff discussions locally, inevitably the Web cast hosts are long on their glowing speeches and short on the scheduled short breaks for local district discussions. The discussion breaks are shortened even further as the hosts go overtime and carry on with their on-camera love-in.

School Planning Councils (SPC) are still largely rubber stamp exercises for parents in this district. Many administrators have been well trained in avoiding meaningful consultation with parents. The debates and decisions occur elsewhere and parents are invited to sign-off without the benefit of being part of the relevant discussions or of having any real input into the decisions. Teachers in this district have not participated in SPC since its inception, and “Parking lot” parent advisory committees (PLACs) are alive and well in Nanaimo as parents discuss what didn’t happen at the PAC. And more recently, the French immersion parent community was once again on the outside looking in as a recent report on creating a French Advisory Committee was written without parents being consulted.