MCQs FACTS-DO YOU KNOW ?

MCQs or multiple choice questions are widely used whether it be in the general education, medical education or Amithabh Bachachan's Kaun Banega Crorepati.

Why should one go in for MCQs ? Because it is convenient and the fact that it overcomes the inherent defects of the conventional system of evaluation and makes it more objective. How frequently are they used? MCQs are now widely used for formative evaluation and for selection purposes.

Does it guarantee reliable evaluation? No, this does not mean that the MCQs guarantee a more valid or a more reliable evaluation. This is because the response is all or none as far as the student is concerned. Also, the tendency to test trivial and frequently less important numerical data decreases the validity of the examination.

Much has been written about the advantages and disadvantages of MCQs. What has not been well known to teachers are the rules for item writing of different types of MCQs, the role of prevalidation and some other issues connected with this form of evaluation. The purpose of this chapter is to focus on these areas,including different formats and their uses.

TYPES OF MULTIPLE CHOICE QUESTIONS

1. The Single or One Best Response Type

e.g. The most effective prophylactic agent for the prevention of rheumatic fever is

A. Acetyl salicylic acid

B. Para aminobenzoic acid

C. Adrenocorticotrophic hormone

D. Cortisone

E. None of the above

This is the most common format. It usually tests only recall of facts. With care, however, items are constructed which require interpretation, analysis or problem solving abilities.

1. Multiple Completion Type

This is another common format used in the VQE and the USMLE examinations and also the UPSC examinations. It is an improvement on the first type mentioned earlier and requires higher levels of cognition than mere recall of facts. The stem is followed by four completions, one or more of which are correct.

e.g. Live virus is used in immunization against

1. influenza

2. the common cold

3. cholera

4. small pox

The code of response is usually standard. To avoid confusion the candidate must choose.;

A. If response 1,2 and 3 correct.

B. If 1 and 3 are correct

C. If 2 and 4 are correct

D. If only 4 correct

E. If all 4 are correct

This type of item is useful to test higher levels of knowledge.

1. Relationship Analysis Type

e.g. Cow’s milk is preferable to breast milk for infants because;

Cow’s milk has a higher content of calcium.

This is another popular type of item. The candidate faced with two statements is asked to respond by choosing.

A. If both statements are true and causally related

B. If both statements are true but not causally related

C. If the first statement is true and the second false

D. If the first statement is false and the second true

E. If both statements are false

This type of item is useful to test higher levels of cognition as the candidate has to decide individually whether each statement is correct and then determine their cause-effect relationship but it is becoming less popular due to problems in scoring. Items with correct response of A or B require the candidate to assess individual veracity of each statement and then to determine their relationship to one another, whereas, in items with C, D or E as correct responses, he does not have to bother about a cause-effect relationship.

1. Multiple True-False Completion Type

This is the most common format of MCQs used by the Royal College of Surgeons and Physicians, UK and also in the PLAB examination. However, this format is not widely used in India.

The model questions in this type could be as follows:

(i) The consequences of breathing 5% CO₂ in area include

a. increase of mixed venous PCO₂

b. increase of pH of arterial blood

c. increase of alveolar ventilation

d. decrease of cerebral blood flow

Each one of these choices can be true or false. Hence, the item can have from nil to five true or false responses.

1. Other Types of MCQs

Rare types of MCQs such as the matching type or the pictorial type are also used, but less often than the above four types. Such rare type of MCQs are often found in UPSC exams.

NUMBER OF ITEMS

For an MCQ examination to have validity as well as to ensure breadth of sampling, it is necessary to include sufficient number of items. 60-100 items is generally considered to be the optimum number for an examination of 60 to 90 minute's duration. If the number is less, the scores obtained may not indicate the depth of a candidate’s knowledge.

TIME

One aspect that should be remembered is that the time allotted should be sufficient for the candidate to attempt all the MCQs as the primary purpose is to test knowledge and not speed of response, which though desirable is less important. Provision of too much time is an incentive to guess and too little time makes it a test of speed of comprehension of language rather than response based cognitive acquisition. Different item formats require different time duration and this must be kept in mind while formulating the question paper. In general, appropriate time required for different items is as follows:

i. One best response type : 40 seconds
ii. Multiple true/false type : 45-50 seconds
iii. Relationship analysis type : 50 seconds
iv. Multiple response type : 50 seconds
v. Case history type : 60-90 seconds

COGNITIVE LEVELS AND FORMATS OF MCQs

Of late there has been some criticism about the educational value of MCQs as it is felt that they may be assessing a specific type of recall. Keeping such criticism in mind, other formats of MCQs have been designed and developed. Use of alternative formats such as the multiple completion type, the relationship analysis type, the case history type, and the multiple true false completion type widely used in the PLAB and the Royal College examinations have enabled testing higher cognitive levels.

Examiners constructing MCQs have necessarily to be aware of the different formats since items which otherwise perform poorly in one format can be made acceptable in another format. This exercise may also increase the cognitive level of performance required by the candidates. For example, unequal length of distracters which is totally unacceptable in the one best response type is not so disadvantageous when used in the multiple completion type (Type G of Hubbard and Clemans).

NEGATIVE MARKING AND GUESSING CORRECTION

An important issue, which generally worries the examiners when using MCQs, is the possibility of candidates scoring merely on the basis of guessing the correct response. By guessing alone, it is impossible for any candidate to secure a pass if the minimum pass level for the MCQ examination has been set as per accepted norms.

However, fears of guess influencing the score have lead to evolution of a formula to detect guessing. The most popular one for this is as follows:

No.of wrong answers

Corrected Score = No. of right answers - -----------------------------

> No. of choices – 1

The psychological and educational value of applying a guessing correction is highly questionable since decision making is not without an element of doubt which is often sought to be overcome by a reasoned guess. If sufficient time is given for the candidate to respond to all items and if the candidates are asked to attempt all items without fail, then guessing corrections lose their meaning. Though the final scores are reduced, the inter-se ranking of students remains unaltered. However, under this system intelligent students are penalized vis-a-vis less industrious but more enterprising students.

The fallacy of applying a guessing correction is shown in the following examples of candidates A, B and C, all of whom attempt a 100 item MCQ examination wherein each item carries 2 marks. The guessing correction per wrong item as per the formula above would be loss of 2 / (n-1) or 0.5 mark per wrong item. Candidate A is bold and willing to hazard a guess, candidate B is timid and not willing to take a risk while candidate C is undecided. All of them know correct answer to 50 items of the 100.

Candidate Known and Partly known Left blank

Correctly and guessed

Answered correctly wrong

A 50 20 20 10

B 50 -- -- 50

C 50 10 30 20

Their scores would be as shown below:

Candidate A: 50 x 2 + 20 x 2 – 20 x 0.5 = 130

Candidate B: 50 = 100

Candidate C: 50 x 2 + 10 x 2 – 30 x .05 = 105

Thus, though all 3 candidates possess the same absolute quantum of knowledge, their ultimate scores are different. These differences are not based on cognitive levels but on differences in personality which facilitate or hinder taking a chance.

In spite of the above, guessing corrections continue to be applied. They may have some role when MCQs are used in entrance examinations in order to discourage and prevent clustering of marks. However, in the pass or fail situation or in the internal assessment examinations, they have absolutely no role. When applied, it is mandatory that the candidate be warned about a correction to be applied for guessing. As a policy, negative marks for wrong answers should never equal positive marks for a correct answer to the same question. If x is the mark for a correct answer to an item, then a wrong item should have x/(n-1) marks deducted where n is the number of choices for that item.

If all candidates are required to attempt all items then a guessing correction is no longer necessary. This is because, guessing correction being dependent on number of wrong answers, the corrected score of all candidates securing the same marks would be equal and therefore, although absolute marks would be reduced, relative ranking between candidates would not be changed. Besides, no moral or educational evils are implied in encouraging intelligent and rational guess. Correction for guessing tends to lower the accuracy of the scores.

There is also a rare problem of a knowledgeable student failing to answer a question correctly because of what has been called "anti-knowledge". This is the result of scientific disagreement between the examiner and the student or ambiguity on the part of the examiner. As random guessing penalizes the ignorant student, "anti-knowledge" penalizes the knowledgeable student. It is therefore necessary to recognize and eliminate items that are ambiguous or debatable.

FIXATION OF PASS LEVEL

Fixation of the minimum pass level in an MCQ examination is more difficult than an essay type of examination (which itself is rather arbitrary at 50%) because of the possibility of a significant chance score. There are several methods of arriving at a pass level for MCQs.

a. Based on objectives being tested; if an MCQ has 100 items for a class of 100 students classified into the "must know" category, the "desirable to know" category which at least 50% of the students should be able to answer correctly and the "nice to know" category which one expects to hope 20% of the class to answer, then this fact can be used to fix the pass level as below:

i. Must know – 70 items; Weightage 100 x 70 = 7000

ii. Desirable to know – 20 items; Weightage 50% of 100, i.e., 50 x 20 = 1000

iii. Nice to know – 10 items; Weightage 20% of 100 = 20 x 10

Total marks securable by 100 students = 200

8200

Therefore mean pass mark should be ------ = 82%

100

a. Based on the mean between maximum chance score and maximum possible score; in the above example of 100 items of one best response type, the maximum chance score would be 20% and the maximum possible score would be 100%. Hence, pass level

100 + 20

would be ---------- = 60%

2

· Mean between maximum chance score and highest actual score. If the highest actual score is 80%, then minimum pass mark

20 + 80

Would be --------- = 50%

2

· Obtain mean of the maximum possible chance score and the lowest score obtained (x). Obtain average of ideal mean score and actual mean score (y). The passing score is then mean of x + y.

· Pass mark can also be fixed to allow roughly the same number of students to clear the examination.

There are several other methods of fixation of minimum pass level in an MCQ, none of which is satisfactory. MCQs function much better for the purpose of ranking a student rather than for deciding a pass or fail situation. In no case should MCQs have the same pass marks as for an essay question.