The issue is that the idea of a given approach or finding being validated by research make intuitive sense and therefore it seems logical if not common sense that such an approach be taken.     As such we fail to consider the full implications of research and in particular the importance of sample size within the research methodology.

We seek to identify approaches which will be transferable and applicable across the whole of education.   We seek to find those magical teaching methods and learning activities that can successfully be used independent of whether we are in a UK state school in a deprived area or a private school in the UAE.     We seek to make general statements in relation to the state of Maths education, or other subjects, in whole countries or even continents.   The sum total of all children currently in education therefore forms our overall target population.    Based on this any study of 10 schools or even 100 schools makes up a tiny, need I say insignificant, proportion of the overall target population.   Taken on face value the sample size of 600,000 students for TIMSS 2015 sounds impressive however as a percentage of all students within the age ranges covered by TIMSS across all countries involved I suspect it will be a small number.

Daniel Kahneman in his book Thinking fast and slow (2014) discusses the issue of “the law of small numbers” in that, where the sample size is small there is a greater tendency for variance to occur.    He specifically mentions education and how research evidence has suggested, and I am careful to say suggested as opposed to proved, that small schools perform better than larger schools.    He then mentions contradictory evidence which suggests small schools perform worse.    The reasoning behind these contradictory findings Kahneman suggests is the fact that the small sample size used in a small school involved in these studies allows for local variance within the sample which is not mirrored across the target population.   So a small number of high achieving students in one year can result in a significantly positive average, whereas the following year a small number of low achieving students in a year can result in a significantly negative average.   Where the sample size is bigger, such as in a bigger school, the impact of a small number of students is lesser as a result of the total number of students.   So there is a greater likelihood for small schools, those with a small sample size, to appear in either the top or bottom as a result of random variation.

Taking the above into account I wonder about TIMSS 2015 and the fact that Singapore and Hong Kong are both at the top.   These each have a total population according to google of 5.4 and 7.2 million people.   How can we compare these with the UK and USA with populations of 64 and 319 million people?    The smaller sample size allows for more random variation.   Now it might be claimed that the fact they have remained at the top across different years shows this isn’t random variation however as Naseem Taleb suggests in The Black Swan, it only takes a single set of data to refute findings which countless previous data might have appeared to confirm.   TIMSS so far has only seen 6 data sets, 1 every 4 years since 1995, so maybe the next TIMSS data will be the one which provides the Black Swan.

Having given this some thought I wonder if the issue is the viewpoint we are taking which is one of education on a macro level.    Maybe the intuitive pursuit of research based practices is as valid and worthwhile as it feels however the problem lies in trying to look holistically.      Looking at practices in our own school or in a small number of local or very similar schools and at things, practices and approaches that work may be more productive.    We could still use a research based approach however it would be at a micro rather than macro level.       I can also see some linkages here to the teachmeet movement as surely it has been about grassroots teachers getting together to discuss their approaches and what works in their classrooms.

Maybe we need to stop looking for “the” answers and start focusing our energy on looking for “our” answers to the question of how we provide the students in our individual schools with the best learning experience and opportunities possible.