Examination Feedback Survey: Summary Report

VCAA Examination Feedback Survey 2007 Summary for Teachers

Background

During November 2007, the VCAA made an online survey available to all teachers of VCE Units 3 and 4 studies with a written examination component. The survey invited teachers to comment on aspects of end-of-year VCE examinations. A total of 764 valid sets of responses was obtained from VCE teachers.

Summaries of the feedback from this survey have subsequently been used to inform the VCE Examinations Unit and setting panels in the development of examinations.

Findings

Overall feedback from teachers was positive about the 2007 examinations. The majority of teachers were satisfied that the questions provided good coverage of the unit(s) of study or the specified VET units of competence, that questions were within the scope of the study design or the specified VET units of competence, and that the examination content was accurate. Most agreed that the instructions for sections were clear and appropriate, that labels on tables and diagrams were correct and consistent with text in the exam questions and that questions were worded clearly.

Where answer books were used, they were rated highly by teachers in terms of planning space, space for full written answers and room for drawing diagrams. The majority of teachers agreed that the exam was accessible to all groups of students irrespective of their religion, gender, ethnicity and age, that gender inclusive language was used throughout the examination, and that mark allocation for questions was appropriate. Slightly fewer teachers rated the overall difficulty of the examination as suitable, and a small percentage indicated that the examination was too long or that some of the questions were too hard.

Methodology

The VCAA engaged Australian Survey Research (ASR) to conduct the online survey.

Questionnaire development and trial

As part of the development process, VCE Coordinators from a randomly selected group of 50 VCE providers were emailed an invitation to participate in a questionnaire trial. A teacher from each VCE provider was asked to answer the questionnaire and provide feedback on a range of issues including clarity and relevance of questions, layout, instructions and navigation, as well as reporting any technical issues. As a result of the trial, and with VCAA consent, some changes were made to the wording of the questions and the introduction to the questionnaire was expanded. Trial participants did not report any technical issues.

Deployment

VCAA sent a memorandum to VCE providers through the VASS system. The memorandum was addressed to Principals, with attention to VCE teachers, VASS administrators and VCE coordinators. In the body of the message to VASS administrators, there was a note to forward the message to all teachers of VCE Units 3 and 4 studies with a written examination component. The invitation also contained the Internet web address of the survey. A link to this address was also displayed on VCAA’s website.

The survey was conducted anonymously, i.e., no unique password was required and respondents were not asked to identify their school or themselves.

Timeline

Questionnaire development and trialling was conducted in September to October 2007.
The questionnaire was made available throughout November.
Data was analysed and reported on in the first half of December 2007.

Respondents

Responses were received on over 70 examinations. However, apart from 11 examinations receiving 20 or more responses, and a further 12 receiving between 19 and 10 responses, the majority received less than 10 responses. English (105) and Biology Exam 2 (79) obtained the most responses.

Of the teachers responding 43% were from Government schools, 19% from Catholic schools and 34% from Independent schools. The remaining 4% were either ‘other’ or did not provide a response. 94% were teaching at Unit 3 or 4 in 2007, and 63% of the teachers had more than 10 years’ teaching experience.

Analysis

For each examination, a pro forma was populated with summary statistics for each rated question and summaries of any free-text responses.

For each rated question, the 25th percentile, 50th percentile and 75% percentile values were calculated and displayed in tables and graphs. In addition, the percentage of survey respondents who chose a response in the Satisfied part of the rating scale (a response of Agree or Strongly Agree) was calculated for each item. Due to the different rating scale for the overall degree of difficulty question, only percentile values were calculated for this question.

There were two free-text questions in the survey: rationale for choosing the degree of difficulty score and constructive suggestions for improving future exams. Where there was a sufficient number of responses for an exam (typically around 20 responses), ASR used a textual analysis tool called Leximancer to identify themes and sub-themes in comments.

Questionnaire

Question	All exams average
The questions provided good coverage of the unit(s) of study or the specified VET units of competence.	3.6
Questions were within the scope of the study design or the specified VET units of competence.	3.9
Examination content was theoretically and technically accurate.	3.9
Where options were offered to students within the examination, the options were of equal difficulty.	3.3
Overall, questions were worded clearly.	3.6
Instructions for sections were clear and appropriate.	4.3
Labels on tables and diagrams were correct and consistent with text in the exam questions.	4.0
There were enough lines for candidates to give a full written answer.	3.8
There was enough space for candidates to draw diagrams, etc, if required.	3.9
There was enough working space for candidates to plan their answers.	3.8
The time allowed for the exam was appropriate.	3.4
Overall, mark allocation for questions was appropriate.	3.7
The examination was accessible to all groups of students irrespective of their religion, gender, ethnicity and age.	3.7
Gender-inclusive language was used throughout the examination.	4.0
Examination advice published on the VCAA website was helpful.	3.7

The table shows the average (mean) score for each question on the 5-point scale (1 = Strongly Disagree to 5 = Strongly Agree, where 3 is a neutral mid-point).

Aggregated feedback for all examinations

When aggregating teacher feedback over all exams, a majority of teachers indicated on all survey questions that they were satisfied and, in particular, with wording, scope, content, layout and length. The satisfaction scores for items ranged from 54% for Where options were offered to students within the examination, the options were of equal difficulty to 88% for Instructions for sections were clear and appropriate.

With regard to the degree of difficulty of exams, using a ten point scale with 10 being ‘too hard’ an average score of 7 was assigned. The majority of teachers (about 75%) gave a score of 6 or higher, suggesting that the exams overall had a slightly higher level of difficulty than the teachers who responded to the questionnaire might have expected. A large number of teachers also provided written comments on the difficulty of an examination, including length of time to complete or question complexity, and others suggested possible improvements.

How VCAA has used the feedback

All of the survey feedback on the 2007 examinations has informed project managers for the task of vetting the 2008 examinations. The individual results for each examination, including the comments from teachers, have been sent to project managers in the VCAA Examinations Unit and to Curriculum Branch curriculum managers. Project managers have been given these results to share confidentially with their examination setting panels. Project managers routinely forward examination feedback to panels: e.g., examination statistics, informal feedback from assessors and assessor statistics. The results of the online survey were added to this stream of feedback.

For some studies, where teachers offered specific suggestions, the exam panels have received this specific advice and it has assisted in the development of the 2008 exams.

Download

Examination Feedback Survey: Summary Report (PDF - 86KB)