Background and Overview
The Literacy and Numeracy for Adults Assessment Tool is an online adaptive tool primarily designed to provide robust and reliable information on the reading, writing, numeracy and vocabulary skills of adults. This information informs the development of learning interventions that match learners’ needs and strengthen their literacy and numeracy skills. The Assessment Tool also allows learners to track their progress over time and enable educators and organisations to report on the progress made by groups or cohorts of learners.
In summary the Tool provides:
- Reading, writing, numeracy and vocabulary assessments linked to the learning progressions
- Hundreds of assessment questions, using New Zealand adult contexts
- Adaptive (where the computer alters the difficulty of questions in response to the learner’s answer) and Non-adaptive assessments (online and printed)
- Reports for learners, educators, organisations and the TEC
- A reliable infrastructure built and tested to IT industry standards.
The development team
The Assessment Tool was developed for the TEC by the New Zealand Council for Educational Research (NZCER) who led a consortium involving NZCER, the Australian Council of Educational Research (ACER) and Fronde. The development process drew upon the collective curriculum and assessment expertise of NZCER, ACER and TEC, and the ICT expertise of Fronde.
The assessment questions
At the centre of the Assessment Tool is a database in excess of 2000 reading, writing, numeracy and vocabulary related questions. Each assessment question in the database is associated with a detail description (or signature) that includes information about its links to the learning progressions, calibration values and usage history. The ability to add to and maintain this database over time is a central feature of the Assessment Tool.
The questions were developed using procedures employed by ACER’s professional test development teams. A central component of these procedures is “paneling”. Paneling is a team approach to reviewing assessment materials. It is a rigorous, robust quality control mechanism and is based on the importance of exposing material to multiple viewpoints. The paneling process involved three to five expert test developers jointly reviewing all aspects of draft assessment material with a view to accepting, modifying or rejecting the material. The questions asked of the material during the paneling process included:
- Is it relevant to NZ adults, especially those in the workplace? Is it interesting? Is it worthwhile? Is it of some importance?
- Is it coherent? Unambiguous? Clear?
- Is it at the right level in relation to the learning progressions?
- Does it breach any ethical, cultural or other sensitivities?
- How will this material stand up to public scrutiny (including project stakeholders and the wider community)?
- Is it likely to be biased, i.e. is it likely to be easier or harder for certain subgroups for reasons other than differences in the ability being measured?
- Is the reading level at the appropriate level?
An extensive trialing process has been used to ensure the robustness of the assessment questions, that they discriminate appropriately between learners and they function together to define the underlying dimensions (reading, writing or numeracy) that they are intended to assess. In order to do this each of the assessment questions was trialed by about 150 learners. The trial process was also used to provide qualitative information about the relevance of the materials for Educators and learners and the degree to which the materials appropriately engage adults from a range of social and cultural backgrounds working and learning in a range of contexts.
The Assessment Tool scales
The Assessment Tool measures achievement and progress using Rasch Measurement scales that map the progression of competency in reading, writing, numeracy and vocabulary respectively. The use of Rasch Measurement scales means that assessment results for different learners, or for the same learner at different time points, will be able to be compared regardless of which actual questions were used in any of the individual assessments.
As well as locating the achievement level or competency of different learners, Rasch Measurement scales allow us to locate the knowledge and skill levels needed to correctly answer each of the different assessment questions in the Assessment Tool. The location of a question on the scale is sometimes thought of as the difficulty of the question. This means that a learner's location on the scale can be understood in terms of the different types of questions that are located at the same place on the scale.
Rasch Measurement scales are what are called equal-interval scales. Each unit on an equal interval scale represents the same amount of change. This makes them ideal for quantifying change.
Primary purpose of the Assessment Tool
The primary purpose of the Assessment Tool is to support educators and learners in their teaching and learning of reading, writing, numeracy and vocabulary. The following figure illustrates the role that the assessment tool plays in the process of strengthening the reading, writing, numeracy and vocabulary of learners. The Assessment Tool provides information to help “know the learner”. This information can then be used to provide next learning step. This learning step is made in relation to what the learner currently knows and what the learner needs to know. While the Assessment Tool provides a strong starting point for understanding the literacy, numeracy and vocabulary skills of learners it may be necessary to use other assessment tools or techniques to probe further the literacy and numeracy skills of some learners.
The Assessment Tool allows learners to track their progress over time and enables educators and organisations to report on the progress made by groups or cohorts of learners