During November 2008, the VCAA made an online survey available to all teachers of VCE Units 3 and 4 studies with a written examination component. The survey invited teachers to comment on aspects of end-of-year VCE examinations. A total of 467 valid sets of responses was obtained from VCE teachers.
Summaries of the feedback from this survey have subsequently been used to inform the VCE Examinations Unit and setting panels in the development of examinations.
Overall feedback from teachers was positive about the 2008 examinations. The majority of teachers were satisfied that the questions provided good coverage of the unit(s) of study or the specified VET units of competence, that questions were within the scope of the study design or the specified VET units of competence, and that the examination content was accurate. Most agreed that the instructions for sections were clear and appropriate, that labels on tables and diagrams were correct and consistent with text in the exam questions and that questions were worded clearly.
Where answer books were used, they were rated highly by teachers in terms of planning space, space for full written answers and room for drawing diagrams. The majority of respondents agreed that the exam was accessible to all groups of students irrespective of their religion, gender, ethnicity and age, that gender inclusive language was used throughout the examination, and that mark allocation for questions was appropriate. Many respondents rated the overall difficulty of the examination as suitable, and a small percentage indicated that the examination was too long or that some of the questions were too hard.
The VCAA engaged Australian Survey Research (ASR) to conduct the online survey.
Deployment
VCAA sent a memorandum to VCE providers through the VASS system. The memorandum was addressed to Principals, with attention to VCE teachers, VASS administrators and VCE coordinators. In the body of the message to VASS administrators, there was a note to forward the message to all teachers of VCE Units 3 and 4 studies with a written examination component. The invitation also contained the Internet web address of the survey. A link to this address was also displayed on VCAA’s website.
The survey was conducted anonymously, i.e., no unique password was required and respondents were not asked to identify their school or themselves.
Timeline
Responses were received on over 60 examinations. However, apart from 3 examinations receiving 20 or more responses, and a further 11 receiving between 19 and 10 responses, the majority received less than 10 responses. English (91) and Biology Exam 2 (21) obtained the most responses.
Of the teachers responding 41.5% were from Government schools, 15.2% from Catholic schools and 39.6% from Independent schools. The remainder 7% were either ‘other’ or did not provide a response. 96% were teaching at Unit 3 or 4 in 2007, and 65% of the teachers had more than 10 years’ teaching experience.
For each examination, a pro forma was populated with summary statistics for each rated question and summaries of any free-text responses.
For each rated question, the 25th percentile, 50th percentile and 75% percentile values were calculated and displayed in tables and graphs. In addition, the percentage of survey respondents who chose a response in the Satisfied part of the rating scale (a response of Agree or Strongly Agree) was calculated for each item. Due to the different rating scale for the overall degree of difficulty question, only percentile values were calculated for this question.
There were two free-text questions in the survey: rationale for choosing the degree of difficulty score and constructive suggestions for improving future exams. Where there was a sufficient number of responses for an exam (typically around 20 responses), ASR used a textual analysis tool called Leximancer to identify themes and sub-themes in comments.
|
Question |
All exams average |
|
The questions provided good coverage of the unit(s) of study or the specified VET units of competence. |
3.9 |
|
Questions were within the scope of the study design or the specified VET units of competence. |
4.1 |
|
Examination content was theoretically and technically accurate. |
4.0 |
|
Where options were offered to students within the examination, the options were of equal difficulty. |
3.4 |
|
Overall, questions were worded clearly. |
3.7 |
|
Instructions for sections were clear and appropriate. |
4.2 |
|
Labels on tables and diagrams were correct and consistent with text in the exam questions. |
4.3 |
|
There were enough lines for candidates to give a full written answer. |
3.7 |
|
There was enough space for candidates to draw diagrams, etc, if required. |
3.9 |
|
There was enough working space for candidates to plan their answers. |
3.8 |
|
The time allowed for the exam was appropriate. |
3.7 |
|
Overall, mark allocation for questions was appropriate. |
3.8 |
|
The examination was accessible to all groups of students irrespective of their religion, gender, ethnicity and age. |
3.8 |
|
Gender-inclusive language was used throughout the examination. |
4.1 |
|
Examination advice published on the VCAA website was helpful. |
3.5 |
The table shows the average (mean) score for each question on the 5-point scale (1 = Strongly Disagree to 5 = Strongly Agree, where 3 is a neutral mid-point).
When aggregating teacher feedback over all exams, a majority of teachers indicated that they were satisfied on all survey questions. The satisfaction scores for items ranged from 58% for Where options were offered to students within the examination, the options were of equal difficulty to91% for Instructions for sections were clear and appropriate.
With regard to the degree of difficulty of exams, using a ten point scale with 10 being ‘too hard’ an average score of 7 was assigned. The majority of teachers (about 57%) gave a score of 6 or higher, suggesting that the exams overall had a slightly higher level of difficulty than the teachers who responded to the questionnaire might have expected. The 2008 result is a decrease from last year when 75% of respondents gave a score of 6 or higher, indicating that the 2008 exams were at more appropriate difficulty level. However, the difference in scores is not statistically significant. A large number of teachers also provided written comments on the difficulty of an examination, and others suggested possible improvements.
All of the survey feedback on the 2008 examinations has informed project managers for the task of vetting the 2009 examinations. The individual results for each examination, including the comments from teachers, have been sent to project managers in the VCAA Examinations Unit and to Curriculum Branch curriculum managers. Project managers have been given these results to share confidentially with examination setting panels. Project managers routinely forward examination feedback to panels: e.g., examination statistics, comments from blog sites, informal feedback from assessors and assessor statistics. The results of the online survey were added to this stream of feedback.
For some studies, where teachers offered specific suggestions, the exam panels have received this specific advice and it has assisted in the development of the 2009 exams.
Examination Feedback Survey: Summary Report (PDF - 53KB)