4.01

The practice of test and measurement techniques

4.02

This fourth chapter reviews a certain number of measuring instruments, their advantages and limitations, and how they should be used. It shows the wide range of such instruments and the importance of choosing them in accordance with the educational objectives to be measured. It is stressed that every teacher should make a certain minimum of statistical calculations when he carries out an item analysis of a test.

Those with a deeper interest in these problems are strongly advised to consult the following publications:

Public Health Papers - WHO

No. 36 A review of the nature and uses of examinations in medical education, 1968.

No. 52 Development of educational programmes for the health professions, 1973.

No. 72 Assessing health workers' performance: A manual for training and supervision, 1980.

After having studied this chapter and the references indicated, you should be able to:

1. Indicate the different elements that should be considered in the evaluation of a teaching programme.

2. Indicate the different elements that should be considered in the evaluation of the educational objectives, learning materials and human resources for a teaching programme.

3. Define the advantages and limitations of a system of evaluation of teaching by the students.¹

4. Construct an observational rating scale and/or a practical test to evaluate the behaviour of a student in the domain of communication and/or practical skills.¹

5. Propose a question for a written (open-book) examination of the “essay” type or a series of six short, open-answer questions and indicate the norms of performance permitting objective marking (marking table).¹

6. Draw up three multiple-choice questions (MCQ) in the domain of intellectual skills - at least two of the objectives must measure an intellectual process superior to level 1 “simple recall” (either level 2 “interpretation of data” or level 3 “problem-solving”).¹

7. Indicate the advantages and limitations of a programmed examination.

8. Define the following terms: prerequisite level test, pre-test, interval test, comprehensive, pre-final; indicate their purpose and the stages at which they are set.

9. Explain the difference between a relative and an absolute criteria test.

10. Calculate the acceptable pass level for an MCQ examination and establish the scoring criteria and norms which permit determination of the passing grade of a mini-test (made up of the questions mentioned in objectives 5 and 6).¹

11. Do an item analysis of a question (calculate the difficulty index and the discrimination index) and draw the relevant conclusions.

¹ Work in small groups is recommended for these objectives. Individual work will usually be appropriate for the others.

The educational planning spiral

4.03

Why evaluate?

4.04

Society, which finally pays the bill for health activities, relies on us to train health personnel to perform a social function corresponding to the health needs and demands of the community they are to serve.

Training centres for health personnel form an integral part of society, and must be prepared to operate within and for that society.

Health activities are of a nature to stimulate social awareness and to provide leverage for social development. The following are some of the questions that must be asked in any attempt to discover whether training centres go to sufficient lengths in preparing the different kinds of health personnel and giving them a training that is “pitched” to meet the health needs of our societies.

Do the graduates think and behave in terms of health rather than of disease? That is to say, do they apply techniques of prevention and health promotion and not only those of cure and rehabilitation?

Do the graduates think and behave in terms of family and community, rather than in terms of the individual sick patient?

Do the graduates think and behave in terms of membership of a health team consisting of doctors, nurses and other health workers as well as social scientists and others?

Do the graduates think and behave in terms of making the best and most effective use of the financial and material resources available?

Do the graduates think and behave in terms of their country's patterns of health and disease, and the relevant priorities?

Consider the institution where you are working - and reflect on the above questions. If you can reply “yes” to all of them, then the essential has been achieved.

If, on the other hand, you cannot unreservedly say “yes” to certain of them, it is urgent for you to reconsider the orientation and the training programme of the school concerned.

To that end, several steps can be suggested:

1. Evaluate the programme as a whole.
2. Evaluate the general and intermediate educational objectives.
3. Have the students evaluate certain aspects of the curriculum.
4. Evaluate the students' level of performance.

Guidelines for evaluating a health personnel training programme - summary description

4.05

The following guidelines are meant to be both general and comprehensive. The evaluator must adopt a realistic approach and decide in each case what is essential in any particular situation.

The guidelines take the form of a series of operations but, obviously, in practice these will often be carried out in a different order. What is essential, in any evaluation study, is for the evaluator to be constantly re-examining the information obtained, reformulating his questions and, even more important, revising his judgements. He is thus often obliged to retrace his steps and, ideally, should review his whole orientation.

The guidelines are presented under four main headings corresponding to the four phases of the evaluation:

I Orientation
II Design of the evaluation
III Gathering information on the programme and its effects
IV Analysis and reporting

Guidelines

Phase I - Orientation

1. Determine the general characteristics of the teaching programme.

2. Ascertain the general characteristics of the administrators, teachers and students.

3. Determine the financial resources available for the programme and its physical facilities.

4. Clarify the aims of the proposed evaluation.

5. Make an inventory of what information is readily available about the teaching programme.

6. Determine the resources available for the evaluation and fix a time limit for its completion.

7. Clarify the evaluator's role and that of the other persons taking part in the evaluation.

8. Make a preliminary appraisal of the nature and the feasibility of the proposed evaluation.

Phase II - Design of the evaluation

9. List the questions and issues to be considered.

10. Determine the appropriate sources of information and the procedures for its collection.

11. Design the evaluation.

12. Draw up an evaluation schedule.

13. Draw up an evaluation budget.

14. Obtain feedback check on the evaluation design, schedule and budget.

Phase III - Gathering information on the programme and its effects

A. Context and objectives

15. Describe briefly the context in which the programme operates.

16. Define the objectives by drawing up a list:

(a) of the future functions, taking account of the main health problems and the tasks and responsibilities deriving from each, and

(b) of the training objectives, taking account of the principal tasks the student should be able to perform satisfactorily at the end of his training.

B. The characteristics of the students on enrolment

17. Ascertain the students' characteristics that are likely to affect their performance as learners and subsequently as health workers.

18. Gather information about the students at the beginning of their training.

19. Make a summary of the characteristics of the student group(s).

C. Resources and training processes

20. Make an inventory of the resources available for the programme and describe how they are allocated.

21. Examine and record the training processes used in the programme.

22. Make a summary of information relating to students' experience, using as themes the principal tasks and role models.

D. Effects and impact of the programme

23. Ascertain the effects of the programme that are to be evaluated.

24. Describe and make a critical examination of the assessment procedures already in use in the programme.

25. If those methods are not adequate for the purposes of the present evaluation, develop and apply others.

26. Work out a set of procedures for observing and describing the long-term impact of the programme, if that is to be evaluated.

Phase IV - Analysis and reporting

27. Prepare a concise description of the programme using the information gathered in the course of Phase III.

28. Analyse the relationship between the various aspects of the programme.

29. Prepare an evaluation report recapitulating the qualities and achievements of the programme, the problems and difficulties encountered in carrying it out, and the available options or courses of action for its improvement.

For further details you are advised to read WHO Offset Publication No. 38, Guidelines for Evaluating A Training Programme For Health Personnel by F.M. Katz (WHO, Geneva, 1978) or Evaluation of Educational Programmes in Nursing by Moira Allen (WHO, Geneva, 1977).

Points to consider in assessing the extent to which programme changes foster closer relationships between schools for health personnel and the wider society

4.07

1. Are changes in the government's priorities concerning health care more clearly understood by the programme planners inside the school?

2. Does the school now respond more swiftly and appropriately to any changes in national planning and priorities?

3. Does the school now work more harmoniously with the “consumers” of the trained personnel, i.e. with the relevant government agencies and the population?

4. Is direct feedback from (recent) graduates' “on-site” performance used as a criterion in assessing the strengths and weaknesses of the (new) curriculum? Is the school responsive to this type of feedback?

5. Are the various schools for health personnel within the whole national system learning from one another and assisting each other to replicate successful changes and progressively eliminate those that do not seem so successful?

6. Is the course content of the (new) programme really relevant to the common health problems of the country? How could it be made more relevant?

7. What observable changes have taken place in

(i) patterns of mortality and morbidity

or

(ii) the responsiveness of people to the prevention of disease and disability?

Could such changes be the result of the programme changes?

8. Are consumers more satisfied with the quality of health care delivered by the health personnel trained according to the (new) programme?

9. What are the principal forces that have accelerated the processes of realistic, valuable programme change and development? What have been the major restraining forces on development? How might accelerating forces be strengthened and restraining forces altered?

Based on a paper by P. Blizard, WHO, New Delhi.

For each main question try to reply in one of the following ways

Yes	-	I have good reason to believe so, and I have even obtained some evidence to that effect.
No	-	I do not think that it is true.
+/-	-	It is very possible but I have no facts to prove it.
?	-	I am unable to reply to the question.

Points to consider in assessing the effectiveness of programme changes

4.08

- in terms of improvement in the administration and functioning of schools for health personnel

1. Has the (new) programme been based on a careful study of the skills needed (list of tasks) by health personnel in the community they are destined to serve?

2. Have student failure rates and drop-out rates been reduced? To what extent?

3. Has the time from initial enrolment to final graduation been reduced? To what extent?

4. Has the volume of content in the (new) programme been reduced in comparison with what it was previously? To what extent?

5. Have there been reductions in the financial and economic cost “per graduate”? To what extent?

6. Are scarce teaching staff and associated facilities more effectively and efficiently used? In which particular respects?

7. Is the content of the various subject areas vertically and horizontally integrated? What are the costs and benefits of that integration?

8. Are collaboration and cooperation between the various departments seen as an easier, simpler process than before the programme changes were made?

9. Have positive (or negative) effects been observed in the allocation of resources between teaching, research and administration?

10. Have changes taken place in the teachers' conditions of work so that they can now spend more time and energy on their teaching and related responsibilities? Have such changes been an improvement? If so, why?

11. Have locally organized teacher education programmes had some effect? If so, which?

12. Has the local education bureau (if one exists) contributed to the changes that have taken place? In what ways could the functioning of the bureau be improved?

13. Have administrative tasks become simpler and easier or have they become more difficult, time-consuming and complex?

14. Are the roles of the chief administrator (Director, Dean, etc.) and other senior administrators now clearer, easier and more precise or have they become less precise and more difficult?

Based on a paper by P. Blizard, WHO, New Delhi.

For each main question try to reply in one of the following ways

Yes	-	I have good reason to believe so, and I have even obtained some evidence to that effect.
No	-	I do not think that it is true.
+/-	-	It is very possible, but I have no facts to prove it.
?	-	I am unable to reply to the question.

- in terms of improvements in teachers' performance

1. Are the tasks involved in teaching the new programme seen by teachers as more interesting, relevant and rewarding?

2. Is the preparation of course content seen as a clearer and easier task, notwithstanding the possibility that it may be more time-consuming?

3. Have teachers improved their capacity to “facilitate” students' learning, i.e. have they developed the skills necessary to help students learn effectively?

4. Have there been demonstrable improvements in the programme for training new teachers to teach more effectively?

5. Do teachers now spend more time, effort and energy (in productive ways) on their tasks as teachers?

6. Do teachers actually use the skills they have acquired during training programmes when they are working with students?

7. Have there been any negative effects on teachers' other roles?

- and in terms of students' performance

1. Are students provided, during their studies, with a greater volume of information and variety of learning activities?

2. Do students retain a greater proportion of the information and skills they have acquired?

3. Do students show an improvement in their ability to use these skills in practical health care situations?

4. Do students now show more initiative and independence in their learning habits, during their time in school, and in the later practice of their profession?

5. Do students show an improvement in their willingness and ability to keep up to date as regards their professional competence (and under adverse conditions)?

6. Are students now more actively involved in the tasks and processes of learning within the new programme?

7. Do students now spend more time, energy and effort in their learning activities, and generally work harder?

8. Does the working (and content) of the new programme provide a more interesting, exciting and rewarding experience for students?

9. Do students play a fuller, more active, more rewarding and more useful role in programme planning, implementation and evaluation?

Based on a paper by P. Blizard, WHO, New Delhi.

Guidelines for evaluating general and intermediate educational objectives

4.10

How can a training school for health personnel make sure that the teaching provided meets the needs and expectations of the population that the future graduates will have to serve? One of the best, but underused, tools to ensure such relevance in teaching is the educational objective.

A WHO Study Group met in 1976 to consider the place of educational objectives in the training of the different categories of health personnel, and in particular to devise guidelines for evaluating such objectives.

The guidelines given below are concerned not only with the way in which the objectives are stated but also with the process by which they were developed. Were the objectives defined so as to reflect, for instance, the needs of health workers and the consumers of their services? Does the statement of each objective contain an action verb clearly indicating what the student must do to show that he has attained the objective? Are the proposed objectives consistent with one another?

The guidelines are arranged under 20 headings. In most cases, the user can simply check one of the answers indicated. A space has been left at the end of each item for the user's remarks.

The report¹ of the Study Group, and especially the proposed guidelines, should be useful to administrators and teachers of schools for health personnel by helping them to define and evaluate educational objectives for their own use and for the benefit of their students.

¹ Criteria for the evaluation of learning objectives in the education of health personnel. Report of a WHO Study Group. WHO Technical Report Series No. 608, 1977, 47 pp. (The Report is also published in French and Spanish.)

A. Persons involved in drawing up the objectives

1. Which of the following persons were involved in either developing or reviewing the objectives?

Teachers (if so please state what disciplines or specialties)
Practitioners (if so, please state what disciplines or specialties)
Education specialists
Students
Administrators (education, health, economy)
Consumers of health care (i.e. “patients”)
Other (please specify)

2. How was the final decision made concerning the adoption of the objectives?

By me alone (if so, please explain why)
By my administrative supervisor (if so, please explain why)
By the chief institutional administrator (if so, please explain why)
By an interdepartmental committee (if so, please explain why)
Jointly (if so, please indicate who was involved)
Other (please specify)

(Circle the answer(s) you wish to give, and add any remarks.)

3. Please describe any steps taken to avoid the objectives being distorted as a result of pressure exerted by certain interested groups (specialties, teaching disciplines, or others to be specified).

Remarks:

B. Basic data and other reference information

4. Given that the objectives should reflect the actual health needs of the community, which of the following needs are reflected?

Present and future health problems
Present and future functions of this category of health worker
Functions of other health workers
Expectations of health care consumers
Expectations of health care workers
Interrelationship between health needs and other needs of society
Official government health policies
Structure of the existing and future health care system
Available data and research on health care and health manpower
Current health manpower planning

5. When the objectives were developed, which of the following factors were considered? (In each case, please explain how.)

Cost to the community (in relation to community resources)
Characteristics of students
Available teaching facilities
Cultural setting of the educational institution
Existing educational system(s) and philosophies
Orientation of political and professional organizations
Other (please specify)

6. Does each objective describe a specific competence that is significantly related to the performance of one or more of the tasks of that health worker?

Yes or No

If “yes”, please give one or more examples.

Examples:

7. Do the objectives represent an adequate sample of the expected professional competences?

Yes or No

If “yes”, please indicate how the sample was derived.

Remarks:

Circle the answer(s) you wish to give, and add any remarks.

C. Characteristics of the objectives

8. Given that the educational objectives are by definition student-oriented:

Do they relate to the actual work the student is going to do after graduation?

Yes or No

Do they describe what the graduate will be able to do (using an action verb)?

Yes or No

9. Given that the objectives should be consistent with one another:

Do some objectives contradict others?

Yes or No

If “yes”, please give an example.

Do some objectives support others?

Yes or No

If “yes”, please give an example.

Remarks:

10. Are the objectives realistic with regard to:

Characteristics of students?

Yes or No

Characteristics of teachers?

Yes or No

Facilities available?

Yes or No

Time available for learning?

Yes or No

11. Given that the objectives should be directed towards significant results of learning in all domains (i.e. intellectual, practical, and communication skills):

Are some of them specifically designed to facilitate personal affective development?

Yes or No

Are some of them specifically designed to facilitate the development of intellectual enquiry?

Yes or No

Are some of them specifically designed to facilitate development of the “total person”?

Yes or No

Are some of them specifically designed to promote the development of competence (and, where appropriate, leadership competence) for team work?

Yes or No

Do some refer to development of research skills?

Yes or No

Does the set of objectives refer to the need for life-long self-learning?

Yes or No

Does the set of objectives refer to the need for life-long (continuous) self-assessment?

Yes or No

Does the set of objectives reflect the scientific base for problem-solving needed by that category of health worker?

Yes or No

Does the set of objectives reflect adequately the range of technical skills required from that category of health worker?

Yes or No

Do some objectives reflect the importance of the health worker's responsibility to society?

Yes or No

Do some objectives recognize the need to prepare the health worker to be responsive to new methods?

Yes or No

If “yes”, please give examples.

D. Intended use of the objectives

12. Has consideration been given to how the objectives will be used in your institution?

Yes or No

If “yes”, please indicate what plans were made for their use.

Remarks:

13. If so, how was the final decision made concerning the use of the objectives?

By me alone (if so, please explain why)
By my administrative supervisor (if so, please explain why)
By the chief institutional administrator (if so, please explain why)
By an interdepartmental committee (if so, please explain why)
Jointly (if so, please indicate who was involved)
Other (please specify)

Circle the answer(s) you wish to give, and add any remarks.

14. Are students informed about the objectives?

Students were given copies of the objectives.

Yes or No

Students have discussed them.

Yes or No

If so, under what circumstances?

Students have not seen them.

Yes or No

If not, why not?

Some of the students have probably seen them.

Yes or No

If so, why only some of the students?

Remarks:

15. Are teachers informed about the objectives?

All teachers were given copies of the objectives.

Yes or No

Teachers have discussed them.

Yes or No

If so, under what circumstances?

Teachers have seen them.

Yes or No

If not, why not?

Some of the teachers have probably seen them.

Yes or No

If so, why only some of the teachers?

E. Usefulness of the objectives

16. As you examine the set of objectives, which among the following results do you believe likely to be produced?

They will be achieved.
They will facilitate student learning.
They will facilitate an integrated approach to learning.
They will facilitate a problem-solving approach to learning.
They will facilitate evaluation.
Other (please specify).
They may not even be used.

Circle the answer(s) you wish to give, and mention any facts in support of your opinion.

17. Has the existence of the objectives had any impact on any of the following aspects of the teaching programme?

Teaching/learning activities (if so, please give examples)
Evaluation methods (if so, please give examples)
Other (please specify)

Circle the answer(s) you wish to give, and add any remarks.

F. Evaluation and revision of objectives

18. Are there provisions for a periodic review of the objectives?

Yes or No

If so, what is the review based on?

Students' performance data.
Operational research data.
Evidence of changing needs.
Other (please specify).

Circle the answer(s) you wish to give, and add any remarks.

19. If so, who is involved in the review?

Teachers
Practitioners
Education specialists
Students
Administrators
Consumers (i.e. “patients”)
Other (please specify)

Circle the answer(s) you wish to give, and add any remarks.

20. If the answer to question 18 is “yes”.

Does such a review facilitate programme changes?

Yes or No

(If so, please specify how.)

The results indicated that the [examination] unduly emphasized activities for which there was little patient demand and failed to distinguish or require some frequently performed procedures. Willis, J.B. et al. A preliminary analysis of accreditation criteria for surgical procedures by orthopedic residents. Journal of medical education, 58 648-654 (1983)

Evaluation of learning materials

4.15

A distinction should be made between technical validation and educational evaluation of materials.

Technical validation:¹ verification that the content is technically correct, up to date, written in the appropriate technical terms and full enough to meet learning objectives but without irrelevant information detrimental to the clarity or usefulness of the materials. This ensures the relevance of the content.

Educational evaluation:¹ verification that the materials are property structured, easy to use, clear, etc., so as to facilitate learning and enable the students to attain the specific objectives for which the materials have been selected or prepared.

¹ Based on Romiszowski (see footnote to page 1.72).

EXERCISE

Key question on self-learning materials (SLM)	Yes	In part	No	Impossible to say from data available
1. Do the SLM set out the professional tasks to be mastered?
2. Do the SLM deal with a problem of high social and economic importance or that is life-threatening?
3. Do the SLM deal with a problem that is not being well hand led by health professionals?
4. Will the SLM force the students to formulate or reformulate their learning objectives?
5. Do the SLM make clear which important principles and concepts students need to understand in order to carry out their professional tasks in the unit?
6. Do the SLM include alt the learning resources that are needed?
7. Are the students involved in the selection of the learning materials they will need?
8. Do the SLM have enough practical exercises and repetition to enable the students to reach the required level (in terms of the criteria set for specific educational objectives)?
9. Do the SLM specify the background work which the students can do on their own?
10. Do the SLM include provision for the students to evaluate their own progress towards the attainment of the learning objectives?
11. Are the students involved in the evaluation of the SLM (including evaluation of the tutor/specialist)?
12. Do the SLM specify how the final evaluation of the students will be done?
13. Does the final evaluation match the professional tasks that have bean specifically enumerated (see question 1)?
14. Are the SLM sufficiently clear as a whole (test: teacher peer review)?

Evaluation of human resources1

4.16

¹ Based on Romiszowski (see footnote to page 1.72)

The evaluation of staff involved in the educational programme can be approached from several perspectives. One is to evaluate instructional performance and take this into account for career development, which will encourage faculty staff to attach greater importance to teaching as part of their career. Specific teaching techniques can also be evaluated, by observation and analysis of the learning activities of students, with a view to improving instructional performance. A third approach is pre-evaluation of the skills, experience or knowledge of the personnel selected for the programme, to determine how well prepared they are for this type of programme. It will thus be possible to choose competent staff or to define the type of training they need before the programme is implemented.

Many techniques have been developed to evaluate the performance of teachers. Here, we are mainly interested in what the staff contribute to the education programme. In higher education, faculty staff are usually recruited on the basis of scientific merit (particularly their publications), while their teaching/instructional abilities are rarely taken into account. However, it is increasingly being realized that it will be very difficult to cope with the world's present and future health problems without qualified personnel. The background and abilities of faculty staff must therefore match their role and functions in the educational programme.

The work of most faculty staff will involve the following four functions:

A. Communicating

B. Planning educational programmes

C. Using active educational methods to give students the benefit of their expertise in their specialty or discipline

D. Using teaching aids and materials.

It is difficult to know what proportion of faculty staff feel they are experts in these four functions. Most of them have not received any training in educational technology, but they must nevertheless be able to perform all the above functions adequately when they “teach”.

Tutors can evaluate their own level of preparation in these four areas of competence with the help of the following questionnaire for self-analysis.

How to use the questionnaire

4.17

1. Choose the sections that correspond to the tasks to be performed or actually being performed by the person to be evaluated.

2. An expert in the functions to be evaluated should read the list of aptitudes in the sections selected and pick out those that relate to the task.

The expert may also identify others that are not on the questionnaire and add them in the spaces that have been left blank.

3. The expert will then evaluate the importance of each of the relevant elements using a six-point scale (see page 4.21) and enter the score in column 1 on the right-hand side of the questionnaire.

4. Proceed in the same way for all relevant elements of all relevant sections.

5. The person to be evaluated should then rate his current level of aptitude in each of the selected elements. It may not seem very reliable to ask teachers to evaluate their own competence in relation to a type of work they are not yet doing, but it has been found that, with a little practice, they are able to pick out their strong and weak points surprisingly well with the help of a questionnaire of this kind. If it is possible to contact a person who knows the teacher professionally (e.g. his supervisor), the reliability of this method can be further consolidated by asking this person to evaluate his subordinate. This evaluation will also be done using a six-point scale (see page 4.21).

6. Proceed in the same way for all relevant elements of all relevant sections.

7. Each element has now been rated in respect of its importance for good performance and the teacher's current level of competence in that element. Both will be scored on a scale ranging from 5 to 0. The difference between the required standard of work and the present level of competence are then obtained by subtraction. When the score in column 1 is subtracted from the score in column 2, a positive result is obtained if competence exceeds what is required and a negative result if it falls short of this level.

8. These results (score in column 2 minus score in column 1) are then entered in column 3 to obtain a profile of competence indicating the extent to which the teacher is ready to do the proposed work. This profile also highlights the teacher's strong points (positive scores), which can be put to good use if tasks are intelligently delegated, and weak points (negative score), which can be improved with suitable training.

Model questionnaire for the evaluation of human resources¹

4.18

¹ Modified after Romiszowski (sea footnote to page 1.72).

One day you may need to carry out one or more of the following tasks. So as not to be caught unprepared, you can estimate your current level of competence now and set to work to improve it if necessary.

Start by evaluating the relevance of each of the skills listed below, adding others if necessary, and then evaluate their importance using the six-point scale given on page 4.21.

Aptitude for communication	1 Importance	2 Your competence	3 Your score
1. Lead discussion in a small group.
2. Act as tutor to a small group working with the problem-solving approach.
3. Use questions to give guidance in group work.
4. Correct students in their learning activities and give them feedback.
5. Communicate orally (ease and fluency).
6. Listen attentively and understandingly to others.
7. Write clearly, accurately and concisely.
8. Draft effective materials for self-learning.
9. Present audiovisual programmes, sound recordings, etc.
Aptitude for designing educational programmes
1. Construct didactic problems for problem-based learning.
2. Direct role-playing games and simulated social situations.
3. Conduct a case study.
4. Organize and conduct workshops and effective brainstorming sessions.
5. Plan and carry out individualized plans of study.
6. Use educational objectives to plan, organize and monitor teaching.
7. Make use of diagrams, figures, etc., to facilitate learning.
The score is positive when competence exceeds the importance of the task and negative when it falls short of this level.
8. Use formative and certifying evaluation to guide the learning process.
9. Organize and facilitate learning in a professional setting.
Aptitude for using teaching aids and material
1. Plan and prepare simple visual aids.
2. Plan a slide sequence and a video film sequence.
3. Plan and manage a system for the cataloguing of visual aids.
4. Plan and manage a system for the maintenance of teaching aids.
5. Use simple film strips, slides, still pictures, etc.
6. Use an overhead projector.
7. Use a blackboard, flip-chart, etc.
8. Use flannelboards, magnetic display boards, etc.
9. Use 8-mm and 16-mm film projectors.
10. Use tape recorders to listen, record, copy and edit materials.
11. Use video equipment to record, play back, copy and edit materials.
12. Use computers (already programmed) for teaching purposes and administration.
13. Use data processing systems to produce educational materials.
14. Make effective use of printed materials for educational purposes.
15. Use photographic equipment to make slides and photographs.
For skills in the planning of training, refer to pages 12 and 13.

Six-point scale to evaluate the importance of a given area of skills to ensure good performance

4.20

5. Essential: it is impossible to carry out some aspects of the work without a high level of mastery of this item.

4. Very important: it is difficult to carry out some aspects of the work without almost perfect mastery of this item.

3. Important: it is desirable to have a reasonably complete mastery of this item to be able to carry out the work satisfactorily.

2. Relatively important: average competence in this item will suffice for the work to be carried out satisfactorily.

1. Not very important: it is desirable to have a general notion of this item, but fairly elementary competence will suffice for the work to be carried out satisfactorily.

0. No importance: this item is irrelevant to the work which the person to be evaluated is doing or will be expected to do. It can be dropped from the list.

Six-point scale to evaluate current competence in an area of skills or a particular skill

5. Complete mastery: the person is an expert or specialist in the subject, is completely up to date and has considerable practical experience.

4. Almost complete mastery: the person is very well informed and skilled with respect to this item, while not reaching the level of an expert or specialist with fully up-to-date knowledge. Has considerable practical experience.

3. Good mastery: the person is reasonably well informed and skilled in this item. Has some practical experience but may have some gaps.

2. Fairly good mastery: the person has some knowledge or competence in this item, but little or no practical experience in the use of this knowledge or skill in a real work situation.

1. Very superficial mastery: the person has some knowledge or skill in this item, but no practical experience, and is therefore hesitant in applying this knowledge. Would probably be able to do so if help, explanations, evaluation and feedback from more experienced colleagues were available.

0. No mastery: the person may have heard or read something about this item, but has never studied it closely or used it in practice.

Monitoring the process of implementation of the programme1

4.21

¹ Based on Romiszowski (see footnote to page 1.72).

In evaluating the learning process, we are concerned to see how the results are achieved. For the tutor in charge of a tutorial group, the main purpose of evaluating the process is to show up the weak points in the students' approach to discussion and to take immediate steps to remedy them. For the designer of the course, the main purpose is to identify those parts of the course that could be improved in the future. For the director of training or course administrator, the purpose could be to evaluate the manner in which the staff are approaching certain instructional tasks in practice. The students should also be involved in the evaluation of the teaching/learning process and should be invited to give their views on the methods that are being used. This feedback could influence the way the course is planned or carried out in the following year.

(a) Technique for the evaluation of teachers

Evaluation will involve direct observation and measurement, by a variety of techniques, of the actual process of teaching.

· Free observation followed by discussion in the form of an oral summary for the purpose of evaluation. These will be facilitated by the use of video recordings for self-evaluation, which should make it easier to accept constructive criticism.

· Observation with the help of instruments based on a questionnaire filled in by the person in charge of evaluation. The questions will generally be based on the desired model of performance, and evaluation may be backed up video recordings, which will make for greater objectivity and credibility.

· Techniques of objective measurement, such as sampling or analysis of interaction. These techniques are meant to obviate the subjective aspect of the previous examples. One way of sampling the work process is to observe selected types of learning activities in a series of visits at random intervals, counting the number of times the teachers are active while the students are passive, all are active, etc. The teacher's style will be evaluated by the frequency of the different types of activity as compared with the indicative norms for teacher-centred approaches. Techniques of interactive analysis differ in that they are based on observation of the totality of teaching/learning activity and directly measure the time spent talking by the teachers and answering by the students, the time spent on group work, individual work, etc. These measurements may be presented in the form of a profile of the learning process showing the percentage of total time spent on different kinds of activities/interactions.

(b) Technique for the evaluation of students

Evaluation of the way the students work should make it possible to identify their difficulties so that steps can be taken to improve the learning process.

A specific outline for diagnosis and feedback can be prepared, and should include the methods of teaching, learning and formative evaluation that are being used. This outline could specify the verbal or other types of behaviour that the students are expected to adopt as a result of the learning activities, together with the features of this behaviour that will be used by the teacher (or any other person involved in the teaching system) to identify the causes of learning difficulties.

An outline can also be devised for the students themselves to identify learning difficulties connected with their methods of work and the tutor's style of teaching. An outline of this kind will also serve as an important instrument of communication between the student and the teacher or tutor.

Personal notes

4.22

Evaluation by students of programmes, teaching techniques and teachers1

¹ Summary of a paper presented by J.-F. d'Ivernois, 21 May, 1975, Séminaire de Pédagogie universitaire, Laval University, Quebec.

4.23

Introduction

An evaluation made by students can provide the teacher with a most useful feedback of information on the quality of his teaching. Anyone who genuinely wishes to teach better should therefore seek his students' opinions. That may be done simply by a friendly talk with a few students, but it is preferable to prepare and distribute a questionnaire. Such an evaluation questionnaire may cover the whole or only part of the teaching, and it is for the teacher together with the students to decide just what should be evaluated by them. It would, for instance, be very valuable for the teacher to know how the students regard any changes in the teaching routine such as a new organization or the introduction of audiovisual material.

It may be noted that anyone who asks students to evaluate his teaching need not doubt the validity of their judgement. For a decade or so many psychometric studies have revealed the validity and the accuracy of student opinions as well as their close correlation with “objective” measurements of the instructor's effectiveness. The many biases which were ascribed to the evaluators, for example, the influence of sex, academic efficiency (poor students/good students), level of studies (beginners, finishers), the status of the course (compulsory/optional) have all proved to be of negligible importance.

Preparation of “simple” evaluation questionnaires

Once the teaching aspects which are to be evaluated have been determined, the evaluation questionnaire is drawn up.

This questionnaire must comply with several essential requirements as concerns the wording of the statements, the scale of the answers and the method of “administration”.

1. Wording of the statements

The statements should be clear, simple and directly related to what it is desired to measure.

Statements that would be accepted straight away by everyone should not be included.

Statements containing double negatives should be rejected.

Each statement should contain a complete thought.

The terms “uniquely”, “solely”, or “most of the time” should be avoided or used in moderation.

A statement should preferably be in the form of an affirmation concerning which the student is asked to express a favourable, neutral or unfavourable opinion.

The number of statements should be limited. Too long a questionnaire wearies the evaluators and this inevitably affects the validity of the answers. It is estimated that a questionnaire should not exceed 60 statements.

2. Examples of statements

“Specific objectives make it possible to identify what is most important in the content to be learned.”

“The time allocated for a learning activity is sufficient for you to achieve all the objectives aimed at by that activity.”

3. Scale of answers

The student should not have to answer merely “yes” or “no” to a statement, for such an answer provides only scanty or dubious information.

That is why preference is given to the use of scales of answers comprising 5 degrees (and sometimes 7 or 9) (see below).

Students should enter against each statement (or on an answer-form or card) the number corresponding to their opinion. It is advisable to provide for an additional answer corresponding to a lack of opinion concerning the statement (coded 0).

The answers scale should be explained to the students at the beginning of the questionnaire, or mentioned against each statement. It is also as well to leave a space below each statement for comments by the student.

4. Method of administration

A person's reaction to a given stimulus is first immediate and then delayed. If the questionnaire is handed out immediately after the event the results observed will be different from those obtained if the questionnaire is administered later. These differences in distribution are attributable to the effect of experience (also called the “practice effect”). The teacher should therefore take this factor into account when he envisages the administration of an evaluation questionnaire.

5. Analysis and interpretation of the answers

Processing (machine or manual) of the answers is simple. The answers to a given statement are then analysed by calculating the number of students (frequency) who have replied by 1, 2, 3, 4, 5 or 0 to that statement. The mean of the answers to the statement can also be calculated. To do this one point is assigned to an answer “1”; two points to an answer “2”; three points to an answer “3”; etc. The total of the number of points obtained for the statement is calculated and then divided by the number of students answering. To make interpretation easier, the trends (favourable, unfavourable) should be grouped together.

A scale ranges from disagreement (1 and 2) through neutral¹ (3) to agreement (4 and 5). Example

¹ For certain questions, it may be considered that a participant in an educational workshop, for instance, should be obliged to adopt a definite position and not take refuge in neutrality: the same applies to the position “no opinion”.

4.24

Example

4.25

Statement: “The time allocated for a learning activity is sufficient for you to achieve all the objectives aimed at by that activity”.

Number of answers: 100

Distribution:	1 : 3	3 : 5	5 : 69
	2 : 2	4 : 21	0 : 0

Trends:	1 - 2 (disagreement)	: 5
	4 - 5 (agreement)	: 90

Mean of answers: 4.51

Construction of complex questionnaires

“Simple” questionnaires enable the students' perception of reality to be measured. It is for the teacher who analyses the answers to deduce from the what them students desire or expect, but this interpretation may leave certain grey zones. If, for example, the statement submitted is: “the course provides an adequate coverage of the subject matter” and 40% of the students disagree with that statement then it is clear that these students feel that the course does not cover sufficient subject matter. On the other hand, what proportion of the 50% of students who agree with the statement think that the subject matter covered is not only adequate but even too much?

“Complex” evaluation questionnaires, such as the one devised by F. Gagné, covering the perception by the students of the teacher/students relationship have the aim of measuring as unambiguously as possible both the students' perception of reality and their level of expectations.

In a questionnaire of the Gagné type, each statement should include:

a title;

a detailed description of the aspect to be measured;

a 5-degree scale with qualitative expression of each degree;

two questions: the first (A) measuring the student's perception of reality and the second (B) the level of his expectations.

Frequency of problem-solving activities: What is required is to measure the frequency of the learning activities requiring students to search for solutions to priority health problems. 1. Very rare examples 2. Rare examples 3. More or less frequent examples 4. Frequent examples 5. Very frequent examples Question A: Where would you place this course on the evaluation scale? Question B: Where should it be in order to satisfy you?

Analysis and interpretation

The means of the answers to questions A (Reality) and B (Expectations) are calculated for each statement. Interpretation of these two means is easy since degree 1 on the scale shown usually corresponds to the minimum frequency or intensity, while degree 5 corresponds to maximum frequency or intensity (see example). The deviation between these two means is then calculated (S score). This deviation shows the dissatisfaction of the students, a dissatisfaction expressed in terms of a lack or an excess. The lower the value of the S score (the nearer it is to zero) the more the particular aspect measured is deemed satisfactory. The S scores of several statements can be compared with one another for one and the same group of students. Furthermore, all the S scores in the questionnaire can be added together so as to give an overall and valid measurement of satisfaction or dissatisfaction. Finally, it should be noted that a questionnaire of the Gagné type can be adapted to different educational methods, e.g. lecture courses, small group activities, teaching by computer, by television, etc.

Example of Interpretation of a Questionnaire of the Gagné type.

Statement	Title of statement	Mean: Reality	Mean: Desires	Deviation (S score)	Interpretation
4	Variation in educational approach	2.39	3.74	1.35	Dissatisfaction
13	Active student participation (+ or -)	4.26	4.65	0.39	Satisfaction
27	Number of references to be consulted	4.51	3.27	1.24	Dissatisfaction (excess)

On this subject, see also Evaluation of teachers and teaching effectiveness by Christine H. McGuire in WHO Public Health Papers No. 61, Geneva, 1974.

EXERCISE

4.27

List the advantages and limitations of this type of evaluation of the educational process by students.

Advantages	Limitations

Compare your list with the list on the next page.

Advantages and limitations of teacher evaluation¹

4.28

Sources	Methods	Advantages	Limitations
(a) Students	Questionnaire, informal contacts, observation of students' attendance. Participation in discussion.	The opinion of the students as “consumers” is very important, since it is given in a very critical spirit.	Likely to be highly variable. Limited value of biased feedback.
(b) Results of evaluation of the students' performance	Formative and certifying tests.	Important impact on both students and teachers,	Influenced by factors other than the role of the teacher.
(c) Colleagues	Observation during teaching sessions.	(Assumed to have) ability to evaluate.	Possibility of professional rivalry.
(d) Self-evaluation	Self-evaluation, sound and video recordings, discussion with others.	Less threatening and helps towards improvement.	Tends not to take account of own shortcomings.

¹ This list was drawn up at a teacher training workshop at Kelaniya University, Sri Lanka, 1986.

Evaluation of students' level of performance

4.29

The following pages contain some examples of tests. In each case you are advised to practise making up a test on a subject with which you are familiar, after having selected a specific educational objective whose achievement can be effectively measured by the test in question (i.e., the test must be valid).

	Pages
Oral test	4.29
Real or simulated practical test	4.30
Execution of a project	4.30
Observational rating scale	4.31
Scale of human values	4.35
Essay	4.36
Short, open answer questions (modified essay question, and restricted response test)	4.36
Multiple choice questions (MCQ)	4.39
Programmed examination	4.47

Excellent examples may be found in WHO Public Health Papers No. 72, Assessing health workers' performance: a manual for training and supervision, Geneva, 1980 and Catalogue of instruments for the performance assessment of health workers, HMD/81.6, WHO, Geneva, 1981.

A few words about the traditional oral examination

Definition: An examination consisting of a dialogue with the examiner who asks questions to which the candidate must reply.

In its standard form, the oral examination is a closed-book test. In that form it can evaluate only level 1 educational objectives (see page 1.51) in the domain of intellectual skills. Like traditional written examinations using short, open-answer questions or MCQ, it provides a check on whether the student can express, more or less clearly, his knowledge of isolated facts or groups of facts that he ought to remember. Most often, it takes the form of a series of not necessarily interrelated questions.

There is a dialogue only if the examiner so wishes.

It should be pointed out that, apart from its advantages and limitations which were described on page 2.30, this type of examination suffers from a scarcity of examiners who are really capable of making the best use of it in practice.

A better way of assessing a student's ability to communicate orally with another person is to use simulation methods, such as role-playing or a telephone conversation, which are much nearer to actual professional tasks.

Assessment of professional skills

4.30

Real or simulated practical tests using check-list	Execution of a project
These tests are based on the direct observation of a professional task (cf. page 2.22).	These tests are based on the indirect observation of a professional task.
A practical test is one that requires the student to perform a professional task in an environment and under conditions the same as or similar to those in which he will have to perform it in his future professional life.	A project execution test is one that requires the student to carry out an activity, in a variable period of time, that results in a product which is to be evaluated by the teacher.
For example: The student is required to weigh a baby in an MCH centre and note the result on the appropriate record card. The student measures the blood pressure of another student. The student is required to prepare a blood slide. All these tasks are entered on a check-list (done well, done badly, not done).	For example: The product might be a concrete piece of work such as a dissection, a dental impression or a histopathological slide; in addition to practical skills, certain intellectual processes can be evaluated by project in a written form, such as a research report or a bibliography.
It is advisable to use this technique when the main component of an educational objective is a practical skill.	It is advisable to use this technique when the main component of an educational objective is a complex practical or intellectual skill, and when the product is more important than the student's manner of working.
The disadvantages of this type of test are: the relatively high cost in teaching personnel in view of the time required to observe each student's work; the risk of the patient being placed in a disagreeable situation; the difficulty in standardizing the test conditions; and the heavy administrative task of coordinating the time schedules of teachers, students and services.	The disadvantages of this type of test are: the relatively high cost in teaching personnel responsible for evaluating the result of the project; and the need to establish a relationship of confidence with the student to avoid cheating.

EXERCISE

Draw up a practical test, either real or simulated, or else a project test, designed to assess a professional task (pp. 1.54 or 1.64 or any task chosen by you), and keeping in view the criteria of validity, objectivity and practicability (re-read p. 2.35). Evaluate your results by constructing a specification table (see p. 3.88).

Assessing attitudes by observational rating scale

4.31

Everybody agrees that the attitudes of the physician and the other members of the health team to the patient and the patient's family are of the greatest importance.

Yet when one observes the way in which students' skills in this matter are assessed, it is seen to be so inadequate that one cannot help being struck by the paradox of the situation.

It must be recognized, in all humility, that this is the most difficult domain in which to make assessments, and the efforts of research workers in this field deserve every support. In a book such as the present one, only a superficial treatment of this subject can be given, and the reader should refer to more specialized publications (see bibliography).

1. Selection of students and assessment of attitudes

The term “attitude” denotes certain constant traits in an individual's ways of feeling and of thinking, and his predispositions towards action with regard to another person such as a patient or collaborator. An attitude is generally considered to be a hypothetical construct which is not directly observable but can be inferred from speech or outward behaviour. The inventory of attitudes constitutes the operational definition of the attitude.

Attitudes are probably not innate. The whole personality structure of an individual, and thus the whole of his behaviour, is constituted by a complex of interlinked attitudes. In its present state, research seems to indicate that it is illusory to expect to be able to change attitudes in the relatively short period of a programme of higher studies. Perhaps then it would be preferable to make sure, by a suitable selection process, that students at the beginning of their studies already possessed the appropriate attitudes for their future profession. It must however be remembered that the definition of those attitudes may easily be contestable; the preparation of such definitions must therefore be a group activity free from arbitrary influences. It would be wise, more over, to conduct carefully controlled experiments before taking any action based on the results of attitude assessment.

The situation being what it is, the drawing of lots may not unreasonably be considered as a method of selection.¹

¹ The selection of students is made by drawing lots in the Netherlands.

2. Observational rating scales during studies

The method of observational rating scales requires the repeated and standardized direct observation of students' activity over a long period (several months) and in natural professional situations such as an outpatients clinic, a consulting room, a laboratory, or a hospital ward. The rating scales are used in evaluating the students' reactions and attitudes. The fact that the scales are easy to use often leads people to forget the many possible sources of error in such evaluations.

If a student is being observed when examining a patient, and if it is wished to make a separate evaluation of how he gains the patient's confidence, the following rating scale may be used.

The student has taken all the necessary precautions, and the patient appears completely relaxed	4
The student has taken the necessary precautions and has reassured the patient several times	3
The student has made an effort, and has followed it up	2
The student has made an effort, without following it up	1
The student seems to be quite unaware of the problem	0

It is advisable to use this technique when the main component of an educational objective is an interpersonal relationship (see p. 1.50).

An example of an attitude rating scale

4.32

Task/Educational Objective: During a telephone conversation, to reassure the mother of a newly hospitalized child

Attitudes	-2	-1	0	+1	+2
1. When giving the mother information on her child's condition	refuses to reply to the mother's questions	gives no information spontaneously	gives inaccurate information	gives accurate information but does not reply to mother's questions	gives accurate information and replies to mother's questions
2. When giving a clear explanation of what has been done for the child	often uses medical terms without ever explaining their meaning	often uses medical terms and seldom explains their meaning	seldom uses medical terms but does not always explain their meaning	seldom uses medical terms and always explains their meaning	uses only terms that the mother can understand
3. When suggesting that the mother should see her child	refuses the mother's request to see her child	does not suggest that the mother should see her child	agrees when the mother makes the request	spontaneously suggests that the mother should see her child	spontaneously makes the suggestion and explains any precautions to be taken
4. When telling the mother how she can obtain information about her child's condition	tells her nothing	when the mother enquires, refers her to the nurse	when the mother enquires, tells her the visiting hours and the persons she should ask	spontaneously informs the mother how to obtain information about the child's condition outside official hours	encourages the mother to come and obtain information about her child's condition outside official hours
5. When suggesting to the mother how a rupture of the mother/child relationship can be avoided	refuses to say how when the mother enquires	makes no suggestion	puts off the question until later	spontaneously suggests that the mother should stay at the hospital with her child	attempts to overcome the difficulties in the way of the mother staying with her child

Criterion: The student should score 7 out of 10 on the above rating scale.

After Professor J.-P. Grangaud (Test and measurement workshop, Timimoun, 1977).

The above example uses one of the most frequently employed rating scales. Others may be found in The assessment of attitudes by Agnes G. Rezler, WHO Public Health Papers No. 52, Geneva, 1973, and in J.P. Guilford, Psychometric methods, McGraw Hill, 1954.

EXERCISE

4.33

Select a professional task (educational objective) whose main component is an attitude and construct a descriptive rating scale for its evaluation.

Task/Objective: The student should be able to:
__________________________________________________
__________________________________________________