Month: July 2024

AI and Marking

Given the concerns in relation to teacher workload, and you just need to take a quick look at the teacher wellbeing index reports to see this, it is clear that we need to look find solutions to the workload challenge.   Artificial intelligence (AI) is one potential piece of this puzzle although is by no means a silver bullet.    The issue I have come across on a number of occasions is concerns regarding some of the challenges in relation to AI such as inaccuracies.  I avoid talking of hallucinations as it anthropomorphises AI;  The reality is that its probability algorithm just outputted something which was wrong so why cant we simply say AI gets it wrong occasionally.    And we are right to have concerns about where an AI solution might provide inaccurate information, especially where it might relate to the marks given to student work or the feedback provided to parents in relation to a students progress.   But maybe we need to stop for a moment and step back and look at what we do currently.    Are our current human based approaches devoid of errors?

I did a quick look on google scholar and found a piece of AQA research from 2005 looking at Markling reliability and the below is the first line in the conclusion section of the report:

“The literature reviewed has made clear the inherent unreliability associated with assessment in general, and associated with marking in particular”

We are not talking about AI based marking here, we are talking about human based marking of work.     We are by no means the highly accurate marking and assessing machines we convince ourselves we are.     And there are lots of other studies which point to how easily we might be influenced.  I remember one study which focussed on decision making by judges where, when they analysed the timing of different decisions, they found that the proximity to a courtroom lunch break had a statistical impact on judges decisions.   Like marking, we would expect a judges decision to be independent of the time of the decision, and to be consistent, however the evidence suggests this isn’t quite the case.    Other studies have looked at how the sequence which papers are marked in can have an impact on marking, so the marking of a paper following a really good or poor paper, will be impacted by the paper which proceeded it.    Again this points to inconsistency in marking.    Also, that if the same paper is presented to the same marker on different occasions over a period of time, different marks result where if we were so accurate in our marking surely the marks for the same paper should be the same.

It seems clear to me that we are not as accurate in our marking and assessment decisions as we possibly think we are.   I suspect, calling out AIs inaccuracies is also easier than calling out our own human inaccuracy, as AI doesn’t argue back or try to justify its errors, in terms of justifying to us, or even internally justifying how the errors are valid to itself.  And this is where a significant part of the challenge is, in that we justify and convince ourselves of our accuracy and consistency, where any objective study would show we aren’t as good as we think we are.   When presented with such quantifiable evidence, we then proceed to generate narratives and explanations to justify or explain away any errors or inconsistencies, so overall our perception of our own human ability to assess and mark student work is therefore that we are very good and accurate at it.  AI doesn’t engage in such self-delusion.

Conclusion

In seeking to address workload and in considering the use of AI in this process we need to be cautious of wanting to get things 100% right.   Yes, this is our ideal solution but our current process is far from 100% right so surely we need only be able to match our current accuracy levels but with a reduced workload for teachers.    Now it may be that the AQA research may present the answer in that “a pragmatic and effective way of improving marking reliability might be to have each script marked by a human marker and by software”.   Maybe rather than looking for AI to do the marking for us, it is about working with AI to do the marking, using it as an assistant but ensuring human insight and checking is part of the process.

And I also note that the above applies not just to the marking of student work but also to the use of generative AI in the creation of parental reports, another area of significant workload for teachers.   Here also an approach of accepting the frailties of our current approach then seeking to use AI to achieve at least the same level of consistency while reducing workload seems appropriate.

Maybe we need to stop taking about Artificial Intelligence and talk more about using AI to create Intelligent Assistants (IA)?

References:

A Review of the literature on marking and reliability (2005), Meadows. M. and Billington. L., National Assessment Agency, AQA

Efficient mental health?

Am currently reading The Lonely Century by Noreena Hertx and managed to get a good few chapters read while on the train from Bristol up to Leeds for the EduFuturists Uprising event.    It was in reading it, and on thinking of AI in education, the efficiency narrative and concerns in relation to mental health and wellbeing, that I thought I would scribble this blog post together.   And I apologise in advance for what will likely be a series of rambling and loosely connected thoughts, but hopefully you will find some value in them and be able to draw some semblance of sense from them.

Now I have written on the efficiency narrative before, that we are forever looking to be more efficient.   This is about getting more done with less so taking less time, or using less resources, or reducing errors.    The issue though is that this isn’t sustainable as ever time we save time, or save resources, we simply identify further things to do which will make use of the saved time, or saved resources.    Often under the guise of “continual improvement” we end up doing more, but we cannot continue this approach indefinitely.     One look at the teacher wellbeing index and we can see the resulting stress and workload issues brought about by a culture of “efficiency” and “do more”.   So individually we are all being challenged to do more, and our organisational structures, our teams, our leadership structures, etc, are all being challenged also to do more, but also to oversee continual improvement, which in turn means doing yet more things.

I have also written in the past on the subject of social media and the false sense of connectedness that it provides.    I have found myself surrounded by virtual friends and colleagues, sharing thoughts and ideas, but still finding myself isolated and alone.     This presents another challenge as we seek to be connected, to get involved and contribute, but all without the benefits of face to face get togethers and proper, real life socialisation.

Loneliness, or as Hertx describes it, that lack of connection not just from friends, family and colleagues but from our wider communities, our political parties and society as a whole is another concern.   Now Hertx points towards this as being a contributory factor in the fracturing of current society, including the growth in extreme views particularly as related to far right politics.     Now for me this links to social media where such ideals can be easily shared and garner favour or even see their way to going viral.    And for the students currently in our schools this is quite significant given that all they will have lived through is a world of social media, of extremist views, the breakdown of social cohesions, covid and its related lock downs and isolation, and similar other challenges.

Maybe we now live in a world where more than ever we need to reconnect with what it means to be human and in particular being the human, social, animals we are.   Maybe we need to spend time rebuilding our communities and rebuilding the social cohesion which I feel, on reflection, was an important part of my childhood.   A time where we knew who our neighbours were, and we worked, or as kids, collectively looking after each other, our kids and our families.    Technology can do so much for us but I do wonder if too much of our lives are now experienced or moderated through technology, rather than being “properly” experienced.   As I sit typing this on the train, I can see some irony in that.   I note I did enjoy some momentary chit chat with my fellow travellers, given the crammed nature of the train.   I could have been getting on with work and being efficient, but instead I sparked up a conversation with the strangers around me and felt, in myself, a little better for it.

Maybe it is these conversations which we need to have much more, putting down our technology, putting aside our “get more done” culture and just being human social animals a little more!   

Attendance or Family Time?

There is currently much debate in the UK on school attendance and parents who choose to take kids out of school to benefit from cheaper holiday costs during term time.   The debate is often framed as quite basic in that taking kids out of school impacts their learning however I feel the issue is more complex and multifaceted, touching upon academic performance, mental health, family bonding, and socioeconomic factors. As education systems around the world emphasize the importance of regular attendance for student outcomes, families face the challenge of balancing these demands with the need for quality time together. This balance becomes particularly complex when families consider taking children out of school during term time to take advantage of cheaper travel options.

School Attendance

Regular school attendance is widely recognized as a key factor in academic success. Many studies show that students who attend school regularly are more likely to achieve higher grades and perform better on standardised tests.   Additionally, schools provide students with a structured environment where they can develop important social skills, learn to work in teams, and build friendships. These experiences being essential for developing emotional intelligence and resilience, with attendance at school therefore being a prerequisite.

But attendance is not just about being present; it’s about engaging with the curriculum, participating in class discussions, and building relationships with teachers and peers. An unhappy child is unlikely to be “fully” present so this might point to the importance of family time and holidays possibly.

We also need to consider that the attendance discussion is largely being promoted from the point of view of government, from that central viewpoint where, on average, students with poorer attendance achieve less than those with good attendance.   But there is an issue in terms of how “achievement” is measured as it is often measured in terms of exams or standardised tests;   Are these tests, this grading and categorising of students all that really matters?   Or is there more to education than a GCSE or an A-Level?

Family Time

While the importance of school attendance is generally clear, family time is equally crucial for a child’s development and well-being.  Quality time spent with family fosters a sense of security, builds strong relationships, and provides opportunities for parents to impart values and life skills. For many families, vacations and time spent together are important for bonding and creating lasting memories.  In today’s fast-paced world, both parents and children often lead busy lives, and taking a break to relax and enjoy each other’s company can reduce stress and improve overall wellbeing. These moments of respite are essential for maintaining a healthy work-life balance and preventing burnout.

Traveling with family can also be educational. Exposure to new cultures, environments, and experiences can broaden a child’s perspective and enhance their learning in ways that classroom education sometimes cannot, igniting curiosity, encouraging a love for learning, and providing practical lessons in geography, history, and social studies.

But for many families, the cost of travel during peak school holiday times can be prohibitively expensive where traveling during term time can make vacations more affordable.   So, some parents may decide to take kids from school to allow for a family holiday and their children to experience destinations they might otherwise not be able to visit. This economic consideration is particularly relevant for lower-income families who may have limited opportunities for leisure travel outside of term time.

Conclusion

The current attendance or lost learning argument is yet another false dichotomy.   Additionally, we need to be careful in viewing holiday time with parents as frivolous and without any educational value.  This is a very simplistic view of learning, and might result from the narrow association of learning with traditional exams and standardised testing.    Learning is much broader, more nuanced and more complex than this.

I feel there is a balance to be reached as parents taking students out of school, when done proportionately and with appropriate communication with the school, can have educational and more importantly wellbeing benefits.    It is therefore very much about the relationship between school and parent, accepting that both have a responsibility for learning, and that such responsibility is interdependent.

Or maybe schools should look to vary their timetables to challenge the holiday, supply-and-demand model, or the travel agents need to be challenged on their pricing model and prevented from pushing prices so high during the holiday periods?    Sadly I suspect these options are difficult to implement, so it is very much about school and parents communicating and working together.

Is using AI cheating?

Ever since ChatGPT burst onto the scene in November 2022 there have been various people in education citing concerns related to how LLMs such as ChatGPT, Gemini, Claude, etc might be misused by students.    But to misuse AI, it must therefore be possible to use AI where I feel the sense is that students should be prevented from using, and who decides what is an appropriate or inappropriate use?   Those invested in change and evolution, who may understand AI, its benefits and risks, or those invested in retaining the status quo with limited understanding or exercise of using AI, let alone using it in a classroom?

Concerns, concerns and more concerns

Concerns have been raised regarding student plagiarism and cheating where students might use generative AI to complete assignments, tests, or essays, undermining the authenticity of their work and misrepresenting their “true” abilities.   This in itself is interesting in ascertaining our “true” abilities.     My spelling and grammar needs work but through spelling and grammar checkers it appears better than it is, but given such checkers are so common does this matter when writing an assignment, blog post or other piece of content?    And does a piece of written coursework or an exam expose the “true” abilities of students, or is it simply a convenient proxy?    Concerns have also been raised in relation to dependency and over-reliance on AI tools which may hinder the development of critical thinking and problem-solving skills if students use them to bypass challenging tasks.   But in a world of search engines and suggestion algorithms suggesting our TV, shopping and music habits is this dependency or simply about convenience?   Access Disparities and digital divides have also been raised given not all students have equal access to generative AI tools, leading to disparities in academic performance and opportunities.  And I suspect this is the most troublesome of the concerns, where the argument regarding the issues some perceive with generative AI may simply fuel an increasing divide between those who can and do use generative AI and those who cant or won’t.

Solution or not?

In relation to assessment some have therefore suggested that the best solution is for simple pen and paper based assessment to be brought back.   I am not sure how this would work as students could still use generative AI to create their coursework before simply copying it by hand.   It also feels a bit like a “we’ve always done it this way”.

AI detection tools have been suggested however I simply don’t believe these are ever going to be reliable.   The key aim for generative AI is to create content which looks like it was created by a human so this will result in a race between AI vendors and the those creating the AI detectors, with only one likely to win that race.    And it ain’t the vendors providing the AI detectors (or the schools spending on money on said detectors). Oh and lets not forget the poor students who will be accused of cheating just because their writing style is highly typical and therefore falsely flagged by these so called AI detectors.

But maybe we need to take a step back and ask ourselves what is the purpose of education and of assessment?  

What is education about and why assess?

If part of the purpose of education is to provide students with the knowledge, skills and experiences which will allow them to flourish and thrive in the world post compulsory education, then shouldn’t we be looking to provide them with the knowledge, skills and experience in relation to using generative AI?    I can only see the use of generative AI increasing across different job types and careers, as I have seen my own use increase post November 2022.  As such, to me it is clear that we should be engaging and working with students in relation to the proper and effective use of generative AI.

And what is the purpose of assessment?    Is it to test memorisation?   And if so, is this as important in a world of search engines and generative AI?    Or in the case of coursework, is it to test the students ability to apply knowledge or demonstrate skills?    And if this is the case, shouldn’t the students be encouraged to use the tools they have available to them which therefore surely needs to include generative AI?    We now, for example, support the use of calculators in Maths exams and we don’t ban the use of spelling and grammar checkers when creating coursework.     And if a student with a learning difficulty uses technology to level the playing field through allowing them to type or dictate, why should it be different for a second language speaker of English using AI translation tools, or simply any student using generative AI to help them create better work, to get started, to refine or to seek feedback?    Why would we want students to create lesser work than they are capable, when using the tools which are now so widely available to them could allow them to achieve more?   Should we not be empowering students to achieve their very best using the tools readily available to them?

Maybe we need to question our current model for assessment, namely tests and coursework, accepting that in a world of generative AI these are no longer suitable or appropriate.   Focussing on assessing the outcomes, the product such as coursework, is no longer possible as students will all be able to create similar output using generative AI tools, so instead I would suggest we need to look towards exploring and assessing the processes students undertake.  

I also note lots of discussion on teachers using Gen AI to help with the workload challenges, using it to create lesson plans and lesson materials, to help with marking, etc.How is this ok for a teacher but for a student to use the same tools, in largely the same way, it isn’t acceptable?

Time for change, finally?

This does feel like a time where we education, and in particularly assessment, need to change significantly.    Gen AI is here to stay, so how can education, how can we make the most of it, preparing our students and providing them with the skills and experiences need to thrive and flourish?

ANME South West

Last week I ran the ANME’s South West meeting, once again allowing me to take part in some of the excellent discussion with other IT staff in schools across the South West.   It was also great to see fellow ANME ambassador, Andrew White, back at the meeting after his recent health concerns.   As always the event proved useful in allowing IT staff from schools in different contexts and at different stages in their digital journey to get together and share thoughts and ideas.    It fits perfectly with the David Weinberger quote I so often use, “the smartest person in the room, is the room”.    In seeking to manage the increasing pace of technology and change, sharing and seeking collective solutions is likely to be our best chance to be successful and to thrive or flourish.  

One of the sessions, delivered by Michael Bewis focussed on wellbeing among school IT staff.   Now I found this session to be very refreshing as when looking at workload in schools the focus is often on teachers with well known research such as the teacher wellbeing index seeking to assess stress and workload of teaching staff over time.  But what about the IT staff, who are often quietly working behind the scenes to ensure the technology works as it should, often being very busy to make sure everything works, even when it all seems to be working well, never mind how hard they work when things aren’t going so well.   As such it was good to see some feedback from a survey of IT staff, involving SalamanderSoft and the ANME itself in gathering the data.   That ¼ of IT staff feel their workload is unacceptable and 40% feel undervalued is concerning although I do think this goes to a wider issue in education, including with teachers.   Now budgets, and linked to this staffing, were mentioned, however this is outwith our control, however our expectations in terms of what is done, when and to what standard is within our control, plus communication is also within our control.   We can therefore focus on what is within our control to hopefully seek to reduce workload and increase job satisfaction.    In the session I mentioned my current 3 keys words or entropy, prioritisation and reasonableness.    That we need to accept that education as a social endeavour has so many variables that there will also be a bit of the unexpected and a bit of chaos, but in dealing with this we need to prioritise what matters and do what we reasonably can.    I also note in relation to workload I have concerns as to the efficiency narrative, and trying to solve the issue by being more efficient.  This invariably leads to simply doing more but maybe we should be asking what matters most in schools, in teaching and learning and in IT, and then focussing on this rather than simply trying to do more.    I also think a key part of workload isn’t the tasks, but the culture of the team and of the organisation.  Is working long hours, being first in and last out, being knackered seen as the sign of a good employee?   Or is a good employee the person who gets the job done but knows when to say no or “not now/yet”, and sometimes works late, but other times leaves early, who clearly seeks to balance out work with their wider life?     It is the little things which build the culture, so do the little things in your school build a culture of wellbeing or not?    And as to wellbeing groups and initiatives, I am not a fan, as all too often these are just things tagged on in the interests of being seen to do something, rather than the cultural change which is really needed, and for cultural change to work it needs to be at all levels, at teacher and support staff level, middle leaders, senior leaders and even governors and trustees.

The other session delivered by Toby Ratcliffe, another ANME member, discussed building a resilient IT support team.    I liked the acknowledgement that things are never simple and plans seldom progress as you planned.  This aligns nicely with my concept of entropy as mentioned earlier.    Some of the other ideas presented matched very much with mine such as the importance of gathering data in relation to the performance of the IT team as a whole.   I personally make use of data from our help desk ticketing system as well as office 365 usage and storage information plus also data gathered from an annual staff and student perceptions survey.  This data allows me to highlight all the work my team do plus the ongoing increase in work as we have more systems, more users and more data to support, helping others understand the nature of the work we do on a day to day basis, never mind when things go a bit wrong.   The annual perceptions survey, as Toby noted, tends to be very subjective however, this aside, having some data is surely better than having no data, as would be the case if you never run a survey.   The key about satisfaction surveys in that it allows you to make decisions based on data, or data driven decision making as it may be referred to.  

Overall, it was a very useful and interesting day, with lots of sharing and discussion above and beyond the two presentations mentioned above.    Discussions dipped into cyber security and business continuity, esports (and I note this came up randomly and not of my doing…..honestly 😉) and Windows 11 deployment among other areas.

So that’s the last of my ANME meetings for this 2023/24 academic year, but I look forward to 2024/25 and further meetings in the new academic year.  Through sharing and collaboration we can best meet the challenges of the future, especially where technology is moving at such an increasing pace.

Reflecting on 2023/24

And so another academic year draws to a close so I thought I would share some initial reflections:

Artificial Intelligence (AI)

AI continued to be a big topic of discussion throughout 2023/24 and saw me speaking to school leaders, teachers but also to school support staff on its potential as well as the risks and challenges.    I think like a lot of Tech AI has those which are heavily engaged and supportive of its use, then a larger body with are unsure or reluctant, followed by those that are anti and against its use.    As such at the moment the impact of AI when viewed generally, may appear less than its potential or what those positive about AI are proclaiming.   For me it is about getting more staff experimenting and finding out how AI can benefit them in schools, often in little and simply ways rather than the flash examples we often see.   Personally, I am slowly introducing greater use of AI into my various workflows and in doing so seeing benefits in time saved but also, and possibly more importantly, on the quality of outputs.    By using generative AI to assist me, AI and I (??) are coming up with more ideas, using a wider vocabulary range, creating better graphics and reducing errors among other things.    Two heads, even if one is a headless AI, are better than one!

Digital Divides

I have already mentioned AI and generative AI but it represents yet another aspect to the issue of digital divides.    Technology, and generative AI has such potential to enable greater creativity, address imbalances such as those related to SEND or to language proficiency, support collaborative and communication and much more.   But you need to have access to the technology, the software, the hardware, the internet bandwidth and more, with this access often the product of a digital strategy or plan, and the relevant budget and finance.   You also need access to support and help, to a culture which embraces the potential of technology and generative AI, at school but also add home and in your local community, friends and colleagues.    The number possible divides between those that have and those that have not is only increasing, and the magnitude of each divide is only widening with each passing day as those that have, experiment, adapt and innovate, while those that have not are held back, continuing to work in ways that are long established, as the world around them changes.

Digital Future Group (DFG), collaboration and sharing

Generative AI advancement is just one indicator of the increasing pace of technology change, with resulting impact on society more broadly.   But how can we keep up with these dizzying advancements and changes?    Can one person keep up with all the apps, the tools, the different approaches?  For me the key here is in approaching the problem collectively and collaboratively rather than individually.   For example, this year has seen the creation and my involvement in the Digital Futures Group, which is all about sharing and networking.   I am so blessed to be part of a group of professionals who operate across different contexts, in different roles and with different skills and experience, across the UK, and I am better for my involvement.   I have also had the pleasure of being involved with the Association of Network Managers in Education (ANME) and also the Independent Schools Council (ISC) Digital Advisory Group.    Each of these organisations has allowed me to network with some amazing people, gaining from their experiences, their values and views, the knowledge and maybe even their humanity, and in a world of increasing use of AI, maybe our humanity, and sharing our humanity is all the more important.

Wellbeing

I think another reflection on the academic year relates to wellbeing, and it may be that this jumps to mind due to a recent presentation at the ANME South West event in relation to wellbeing and IT support staff.    This academic year has for me been a very difficult one personally with a major personal upheaval hitting me in the first term, something that I am not sure I have fully recovered from as we complete the final term.   Actually, thinking about it I don’t think it is about recovery but more about adapting to the changed circumstances I now find myself in.   This has forced me to stop and reflect and in doing so identify a lot of personal practices and habits I have developed which have led to an unbalanced life.   It took a significant life event to make me stop and reflect such that I am now trying to rebalance and establish new habits.   This has also got me thinking about the “be more efficient” narrative and what it means to be a good employee.   I get the concept of being more efficient and therefore doing things quicker or using less energy, etc, but if all this leads to as being asked to do yet more things, surely this isn’t a sustainable model.    Laura Knight talked about resilience and how this isn’t something we should aim for as its ok to being resilient to short term issues, but having to constantly exist in a state of resilience isn’t really living and in all honesty isn’t something we can do for any significant period of time.    For me “efficiency” may suffer similar problems.    And if being efficient is driven by an organisational need for efficiency, so does this mean that to be a good employee I need to be efficient and get more done than others, and if this is the case does it not possibly drive unsustainable hours, stress and workload issues.   So maybe schools and other organisations need to consider what it is to be a good employee, with leaders modelling this and with the expectations clearly espoused.  Maybe we also need to stop and identify what really matters, rather than constantly adding more tasks, more requirements and more considerations to our everyday roles.

Conclusion

This for me has been a year of difficulties but also of a greater sense of community and collaboration.   AI, Digital Divides, Networking and Wellbeing are definitely the four themes which currently stick out for me from what has been a busy academic year, but then again when are academic years in schools or colleges not busy?    Am hoping that 2024/25 will be another positive year and soon enough it will be upon us.   I am going to post in the coming weeks a month by month review of some of my highlights for the year including some photos but for now let me just wish everyone a good holiday period acknowledging that myself and my team, plus many others, particularly IT teams, will actually be working much of the holiday period on IT upgrades and many other things, ahead of the new academic year.    All the best to all.