Tag: Data

Data, data and more data

This morning it was the turn of the NHS to be the focus of the morning TV discussion about how things aren’t going well.    I suppose I should be partially thankful as this takes the spotlight off education at least for a short while.    That said it also once again shows the superficial use of data.

This mornings TV took some time, along with fancy graphics, to outline how the NHS waiting times had increased.   The specific figure they presented being the percentage of patients at A&E who were seen within 4 hours.   This seems like a reasonable statistic to use from the perspective of a patient as it suggests the likelihood that should I need to turn up at A&E I would be seen in 4 hours of less.   I suspect the fact that it is so potential meaningful for prospective patients, the average TV viewer, is why they picked this statistic over others.

The issue with this is what it doesn’t tell us the additional context which may be important in interpreting the figures.    Over the period under consideration did the number of patients attending A&E remain static or did they in fact increase which may be a contributing factor to increased waiting times?     A briefing report by Carl Baker from November 2016 suggested that in 2016 the number of A&E patients at major A&E departments increased 6.3% over attendance levels in 2015.   Were there any changes in the demographics of patients attending A&E as an increase in elderly people attending may mean that patients are less likely to be able to be quickly seen and discharged, again contributing to increased waiting times.    What about the staffing levels of A&E over the period?   Did this change as a reduction in staffing may account for increased waiting times?   Also the figures look specifically at average data for the whole of England; were there any regional variations?   Personally I live in the South West and feel that it is difficult to access a doctor which may mean that I would attend A&E on occasions where someone with more ready access to a GP would not.    Are there also differences between A&Es serving urban and rural areas?   Are there differences between A&Es serving large versus those serving smaller populations or population densities?

In the current performance indicator and accountability led environment we often focus on specific figures such the percentage of patients seen in 4 hours or the number of pupils achieving A*-C or Progress 8, PISA, EMSA, TIMMS, PIPS or other measures.    Each of these pieces of data is informative and tells us something however equally there are a lot of things that it doesn’t tell us.    We need to ask what doesn’t this data tell us and seek data to add context.

Only with context is data useful.

Accident and Emergency Statistics: Demand, Performance and Pressure, C Baker (2016), House of Commons Briefing Library (6964)

 

Some thoughts on GCSE and A-Level results

Having read various articles following the recent A-Level and GCSE results I cant help but think that schools and more importantly education in general needs to make a decision as to what we are seeking to achieve, and stop acting re-actively to limited data which has been used to draw generalized conclusions.

Take for example the shortage of STEM graduates and students.    This was and still is billed as a big issue which has resulted in a focus on STEM subjects in schools.   More recently there has been a specific focus on computer programming and coding within schools.     In a recent article it was acknowledged that the number of students taking A-Level Computing had “increased by 56% since 2011” (The STEM skills gap on the road to closing, Nichola Ismail, Aug 2016).     This appears to suggest some positive movement however in another article poor A-Level ICT results were cited as a cause for concern for the UK Tech industry (A Level Results raise concern for UK tech industry, Eleanor Burns, Aug 2016).  Now I acknowledge this data is limited as ideally I need to know whether ICT uptake has been increasing and also whether A-Level Computing results declined, however it starts to paint a picture.

Adding to this picture is an article from the guardian discussing entries:

Arts subjects such as drama and music tumbled in terms of entries, and English was down 5%. But it was the steep decline in entries for French, down by 6.5% on the year, as well as German and Spanish, that set off alarm bells over the poor state of language teaching and take-up in Britain’s schools.

( Pupils shun English and physics A-Levels as numbers with highest grades falls, Richard Adams, Aug 2016)

So we want STEM subjects to increase and they seem to be for computing, however we don’t want modern languages entries to fall.   Will this mean that next year there will be a focus on encouraging students to take modern foreign languages?    And if so, and this results in the STEM numbers going down will we then re-focus once more on STEM subjects until another subject shows signs of suffering.

It gets even more complex when a third article raises the issue of Music A level Entries which “dropped by 8.8% in a single year from 2015 and 2016”.  (We stand back and allow the decline of Music and the Arts at our peril. Alun Jones, Aug 2016).    Drama entries are also shown to have seen a decrease this year (Dont tell people with A-Levels and BTecs they have lots of options, Jonathan Simons, Aug 2016).  So where should our focus lie?   Should it be on STEM subject, foreign languages, drama or Music?

I suspect that further research would result in further articles raising concerns about still further subjects, either in the entries or the results.   Can we divide our focus across all areas or is there a particular area, such as STEM subjects, which are more worthy of focus?  Do the areas for focus change from year to year?

As I write this my mind drifts to the book I am currently reading, Naseem Talebs, The Black Swan, and to Talebs snooker analogy as to variability.     We may be able to predict with a reasonable level of accuracy, a single snooker shot however as we try to predict further ahead we need more data.    As we predict five shots ahead the quality of the surface of the table, the balls, the cue, the environmental conditions in the room, etc. all start to matter more and more, and therefore our ability to predict becomes less and less accurate.      Taking this analogy and looking at schools what chance do we have of predicting of the future and what the UK or world will need from our young adults?    How can we predict the future requirements which will be needed from the hundreds of thousands of students across thousands of schools, studying a variety of subjects from a number of different examining bodies, in geographical locations across the UK and beyond.

These generalisations of data are subject to too much variability to be useful.    We should all focus on our own schools as by reducing the scope we reduce the variability and increase the accuracy.   We also allow for the context to be considered as individual school leaders may know the significant events which may impact on the result of their cohort, individual classes or even individual students.  These wide scale general statements as to the issues, as I have mentioned in a number of previous postings, are of little use to anyone.   Well, anyone other than editors wishing to fill a space in a newspaper or news website.

 

 

 

 

 

Some thoughts on Data

A recent article in the Telegraph (read it here) got me thinking once more about data.   This also got me thinking about the book “Thinking, Fast and Slow” by Daniel Kahneman which I have only recently finished reading.  The book highlighted a number of issues which I feel have implications for education and need to be considered by school leaders.

Firstly the small numbers effect:  The Bill and Melinda gates foundation commissioned a study to examine schools in search of the most effective schools.    It found, unsurprisingly that small size, in terms of student numbers, schools achieved the best results, over larger schools.   Contradictory it also found that small schools also achieved the worst results.   The reason for this as explained by Kahneman is that where a data set contains only a small number of items the potential for variability is high.   As such, due to a variety of random variables and possibly a little helping of luck, some small schools do particularly well, out achieving big schools.    Other small schools are not so lucky and the variables don’t fall so well, resulting in the worst results.

To clarify this consider throwing three darts at a dart board aiming for the centre.   This represents the results of a school with a small number of students with higher scores being nearer centre and a lower score being those darts ending further from the centre.   In the case of student results an average result would then be calculated for the school and the same can be done looking at the position of the darts.   Assuming you are not a professional darts player you may do well or you may not do so well due to a variety of random variables.     Given the limited number of darts the potential for variability is high hence a high average or low average is very possible.   Next consider if you were to continue and throw sixty darts at the dart board, taking the average across all the dart throws.    Given the number of darts the average will regress towards your mean darts throwing ability.    The increased number of data items means that variability is reduced as each significant good or poor throw is averaged out among the other throws.

Within schools a great deal of value is being attached to statistical analysis of school data including standardised testing however care must be taken.   As I have suggested above a statistical analysis showing school A is better than school B could easily be the result of random factors such as school size, school resourcing and funding, etc as much as it may be related to better quality teaching and learning, and improved student outcomes.

Another issue if how we respond to the results.  Kahneman suggests that commonly we look for causal factors.   As such we seek to associate the data with a cause which in schools could be a number of different things however our tendency is to focus on that which comes easily to mind.   As such poorer (and better, although not as often,) results are associated most often attributed to teachers and the quality of their teaching as this is what is most frequently on the mind of school leaders.    We arrive at this conclusion often without considering other possible conclusions such as the variable difficulty of the assessments, assessment implementation, the specific cohort concerned, the sample size as discussed earlier and a multitude of other potential factors.   We also, due to arriving so quickly at a causal factor which clearly must be to blame and therefore needs to be rectified, fail to consider the statistical validity of our data.   We fail to consider the margins for error which may exist in our data including what we may consider acceptable margins for error.   We also fail to consider a number of other factors which influence our interpretation of the data including the tendency to focus more on addressing the results which are perceived to be negative.   This constant focus on the negative can result in a blame culture developing which can result in increasing negative results and increasing levels of blame.   Maybe an alternative approach which may work would be to focus more on the marginally positive results and how they were achieved and how they could be built upon.

The key issue in my belief is that we need to take care with data and the conclusions we infer from it.   We cannot abandon the use of data as how else would we measure how we are doing, however equally we cannot take it as fully factual.   The world is a complex place filled with variables, randomness and luck, and we need to examine school data bearing this fact in mind.   We also need to bear in mind that data is a tool to help us deliver the best learning opportunities for students;  data is not an end in itself!

 

Inconsistency in the quality of teaching

I have came across the above statement or similar statements across schools both here and in the middle east.   At first reading I would suggest that everyone, myself included, will take this to be a negative comment.   On reflection I am not so sure it necessarily is negative or in fact that it tells us anything.

Consider the “average” school and lets consider that the measure of quality of teaching is student outcomes.    Now I know this is a very limited model however it will hopefully serve its purpose in terms of proving a point which could equally be proven by using a different measure for the quality of teaching.

Within this “average” school there will be some above average teachers where outcomes come out as very positive.   There will also be those that come out as below average.    Would this be considered as consistent as clearly having different qualities of teaching would suggest inconsistency?

Lets assume what is meant be inconsistency is an inconsistency when compared with the national profile for the quality of teaching within any given school.    In this case our average school now becomes consistent in terms of quality of teaching.    Consistency is therefore referring to the distribution of individuals within the school with regards the quality of teaching, and how this compares to other schools.

Modifying the scenario a little lets say that some of the so called “weaker” teachers performance only gets worse while the stronger teachers only get better.    Our average still remains the same however is the school any more or less consistent given the wider variance between teachers and given the difference between this profile and the profile of the “average” school?

If some of the teachers formerly within the “average” band improve this would shift the average and change the distribution.   Is this inconsistency and if so could it not be viewed as a positive inconsistency?

Now I was considering using some further examples however have decided not to.  Instead I will point out my belief in the fact that teaching is a social activity involving a class full of students and a teacher all interacting.   Given it is a social activity involving 30 or more human beings and therefore influenced and affected by a multitude of different dynamic variables, consistency is highly unlikely.    Teaching is very much like chaos theory in that it is highly sensitive to its conditions, which are frequently changing.    As such how could any school be expected to demonstrate consistency?   Like chaos theory, we can only possibly perceive a pattern by looking at the much wider picture, as under close inspection we see nothing except the variability and the differences.    How might an inspection team or an internal mock-sted see this big picture?   I doubt they would do so how can a judgment indicating an inconsistency be arrived at?

And maybe something different, unique or not fitting in with the usual run of play may be a positive thing.   So maybe consistency isn’t all it’s meant to be!

Silos of Data

Day 11 in the #29daysofwriting house and the housemates are getting a little restless………

Sorry couldn’t resist!  This posting every day is starting to feel a little like the diary room on an episode of Big Brother.   It is also getting steadily more difficult to decide on the topic of the day.

Today I would like to just spend 29mins writing on systems.   In schools we have a large number of different systems.   We have a school (or management) information system, an HR and payroll system, an email and file storage system, a library system, a bus/transport system and a multitude of other systems.

Each system is designed for a specific purpose.   The SIS (or MIS) system has all the personal details of students along with their academic performance data.   The library system has details of students, books and loans.   The HR system has details about all of the staff.

Each system reports its data in a specific way.  The SIS system can produce class registers and parental reports, while the HR system can produce staff lists and the Library system information about student lending habits.

The issue is that even where the systems are supposedly “integrated” in actual fact they are not.    The data exists in Silos, independently in each different system albeit linked by a common identifier such as a student ID number or other ID number.

Having recently read about the impact of Silos and how overcoming them can have a significant effect it makes me wonder about the Silos in school systems.    If we could extract all the data into a single common location where we could apply various business intelligence tools to analyse it we would likely be able to draw new conclusions and through doing so be better informed.   We might be able to identify linkages which previously weren’t apparent.   Maybe students in particular classes or with particular teachers borrow more books and maybe, of these students, a majority perform better.   Obviously I speculate here for illustrative purposes.    The key point being is that we might be able to identify patterns which currently cannot be identified due to the Silo’d nature of data.

1.5Mb! Why would I need more?

Technology continues to have a significant impact on our everyday lives but not only that, as technology advances at an ever quickening pace, so it changes our everyday lives.  During the last week two specific events have caused be to reflect on this.

The first event happened earlier this week when I came across the below tweet:

This got me reflecting on my own journey with computer systems and in particular the storage space and data which I interacted with.    My early life with computer systems involved 5 ¼ inch floppy disks and later the higher capacity 3.5 inch not so floppy variety.   The sum of all the data I could create and consume could be stored on 1.5Mb discs without any problem.    As time went on however it became apparent as my disc collection expanded into the hundreds that I needed a new storage solution to cope with the ever increasing amount of data I was both producing and consuming.   Enter my first proper IBM compatible PC complete with a whopping 500Mb hard drive.   This was the solution to all my problems or so I thought for a while anyway.   It wasn’t long before the 500Mb drive inside my nice new PC had to be provided with a friend to play with, in the shape of a 2nd 500Mb hard drive taking my total storage to the impossible to fill level of 1Gb of storage space.

I won’t bore you with the intervening years so let’s fast forward to the here and now.   My storage requirements these days run into terabytes and let me just say I have a few terabytes worth of data.    Part of this might be that I am a bit of a hoarder including keeping the data, files and other creations which I have developed over the last 10 years of so.   I may even be able to lay my hands on specific emails from 3 jobs and almost 10years ago.    Part of this may be the ever higher quality and complexity of creations and part may also be the increasing speed with which I can generate new digital data or content.     The tools I have at my disposal now allow me to create new content at a far faster rate than I could all those years ago working with 5 ¼ inch floppy discs.

This is the point I want to make here, that our ability to create and consume content is increasing at what must almost be an exponential rate as technology provides us new methods for both producing and creating content, plus to share this content for others to adapt and share again.     This brings us to the second of the events I mentioned at the start of this post.   While reading “The New Digital Network” I came across the below statement:

“Every future generation will be able to produce and consume more information [or content] than the previous one”

Given this it is important that we as teachers are adequately preparing our students to deal with this ever increasing amount of content for consumption plus to manage the increasing levels of content which they find themselves producing.       Students need to understand how search engines work and why certain items float to the top, they need to be able to evaluate information for validity using alternative sources to triangulate and confirm the truth.    They need to understand differing standpoints, religious differences, cultural differences, philosophical differences and ethical and moral differences.      Students then need to be able to present their own beliefs and viewpoints while understanding that others may disagree.   They need to understand how the content they produce will be stored and presented on the internet and how any shared content or information could have a profound impact on their life as it is now or in the future many years from now.

The above represent only a small number of the issues which we should be seeking to help students understand.   I would suggest that the current programmes of digital literacy and e-safety lessons barely begin to scratch the surface of the discussions we really need to be having with our students.

School Data: A puzzle or a mystery?

€Ž

Over the last couple of weeks I have tweeted on a couple of occasions regarding data in schools. Most of the tweets revolved around the fact that the importance of data and in particularly standardised data such as EMSA and PISA is often stated. As such a fascination seems to have developed with number crunching through detailed student performance data followed by the creation of colourful pie and bar charts with the occasional line graph thrown in for good measure. Now my tweets focused on the fact that I feel educators are being sucked into this world of data, and in some cases are pouring over these tables, charts and graphs for hours on end, presenting them to all who will listen. This time spent looking at data to me seems to be a distraction from what is really important which is student learning. I accept that we need some data to know how students are doing and progressing however I think we need to balance this against the more important task which is key to schools; teaching.     If the data takes hours of time to analyse or if it doesn’t result in changes or action within lessons is it worth it?

It was while listening to Malcolm Gladwells what the dog saw, on the usual journey to work, that something struck me. We are treating student performance and the need to improve it as what Gladwell described as a puzzle. A puzzle according to Gladwell is solved through gathering additional information. As such our fascination with data and having more and more data in the hope of more insight and therefore better results seems logical. However, Gladwell also describe Mysteries; these are situations which are not solved through more data or more information but through the insightful use of what we do know. It is at this point that it struck me; student performance is a mystery not a puzzle. We cannot solve it through more data and in fact all this will do will detract from the core task at hand in schools; teaching. We instead need to focus on using what we do know and have readily available to draw insightful conclusions which we can action.

One thing, it strikes me, stands in the way of this and this is the dreaded school management system. It is designed to gather all the data you will ever need into table after table of grades, scores and criteria achievement. Some will even create the pretty charts and graphs for you. The issue at hand is the usability of these systems. The way they present data requires analysis. It is not instantly user friendly for teachers who want to be able to view data and draw conclusions quickly and on the move. It is my belief that school management systems need to be redesigned. Now, to that end I have started to build a concept for a new more intuitive and user friendly school management system focusing first and foremost on the teacher in the class.

Please if you have any suggestions or would like to contribute ideas to what the ideal school information or school management system should look like and do, etc get in touch.

 

Image courtesy of cooldesign from http://www.freedigitalphotos.net