A research programme was conducted to measure the impact of the professional development intervention. We began this research by asking what happens when the Effective Teaching Profile (ETP) is implemented in mainstream secondary classrooms. Because of the complex nature of this exercise, we used a triangulation mixed methods approach (Creswell, 2005) to gather and analyse qualitative and quantitative data from a range of instruments and measures. As a result we have multiple indicators (Kim & Sunderman, 2005) that form the basis of our investigation.
Numeracy Gains were measured by effect sizes, which is an internationally recognised measure of the strength of the intervention from pre-test to post-test. The results show that the effect size for the experimental group (Māori students of Maths teachers involved in Te Kotahitanga) was larger than typical (d = .76). The effect size for two control groups were: (a) typical for Māori students of Maths teachers not involved in Te Kotahitanga (d = .52) and (b) typical for Māori students nationally (d = .51). This means that Māori students whose teachers are in the project are achieving significantly higher in numeracy than Māori students where teachers are not in the project. This tells us that the context created in Te Kotahitanga teachers’ classrooms is better for improving the achievement of Māori students than numeracy interventions alone.
Literacy gains were measured by an analysis of stanine gains which are normalised standard scores, again internationally recognised among educators as a useful guide to student achievement. The most impressive gains were from the lower third of stanines and this is encouraging because this is where many Māori students perform (according to the international PISA study). The results for the lower three (out of 9) stanines in this study showed an effect size of .80 in the first year and .58 for 2005. In 2004, 46% and, in 2005, 34% of Māori students who achieved stanines between 1 and 3 in the literacy pre-test, achieved stanines between 4 and 6/7 in the literacy post-test. This means that this group of Māori students, as identified in the PISA study, are making significant gains in literacy as a result of their teachers being involved (at least in part) in Te Kotahitanga. Overall both Māori and non-Māori are making similar progress in literacy.
On the basis that Te Kotahitanga is focused on raising the achievement of Māori students through changing teacher practice, we adopted Elmore’s (2002) model for demonstrating improvement by measuring increases in teacher practice and student performance over time. This model demonstrates improvement by measuring the quality of teacher practice and student performance on the vertical axis and time on the horizontal axis. Improvement then is shown by movement in a consistently north-easterly direction.