Tag: exams

AI: Time to give up pen and paper?

I have been reading the Experience Machine by Andy Clark off and on for quite a few months however the other day, on a trip down to London for an InfoSec event I found myself on a train to London, where once again I had an opportunity to do some reading.    It wasn’t long before I was reading Clark’s thoughts as to the extended mind and it got me thinking of the current discussion in relation to AI use in schools, and in particular by students for “cheating”.

Clark talks about how humans have sought to extend their capabilities through the use of tools, including both basic tools like the pencil as well as technological tools like devices and apps.   He makes the point that rather than just being a tool which is used, that the use of tools results in fundamental changes to our thinking processes, to our minds.   We have developed a species through our ability to use tools and to adjust our thinking processes around these tools, in order to do something more than we could prior to the use of the tools.

Taking this into the world of education, I have repeatedly talked about the JCQ guidance in relation to Non-Examined Assessments (NEAs) where it talks about making sure the work is the students own work.    Well, if we take Clarks comments, then the output produced by a student using the tools of a pen and paper, was shaped not just by the students but by the pen and paper they used.   The pen and paper shaped thinking processes, ordering and more, influencing what the student produced.  Maybe the sheet of paper will influence how much the student produces?   Maybe the difficult in erasing content written in pen will influence the students decision making as to whether to change or remove sections they have written.   So is it still the students own work?

Considering a different tool, this time a laptop in the hands of a student with exam concessions which facilitate their typing rather writing.   Again, I would agree with Clark that the tool, rather than being just a tool, changes the thinking processes.   With a laptop a student can more easily shift and reform their thoughts and ideas, moving paragraphs around and erasing or adding content as needed.  This means processes related to the ordering of content which might be needed when using pen and paper are no longer as important.   A student with a laptop might be more willing to take risks and explore their writing knowing they can easily change, add or edit, whereas a student with pen and paper may be a little more risk adverse, and therefore more creatively limited.

So now let me take a leap, and I suspect some will see it as a leap too far.    What if the tool rather than just pen and paper, is actually a generative AI solution.    The interactions with the AI, assuming the student has been taught to use AI and has developed the appropriate skills, will shape the students thinking processes.   Maybe the broad training data of the AI will result in the student considering aspects of the topic they may not have otherwise explored.   Maybe their language will change, making greater use of more academic language as a result of the academic content which makes up the AIs training data.  Maybe their language will be a bit more flowery and expressive than they might write without an AI tool.   As with the laptop, AI may make the student even more creative and less risk adverse, knowing they can easily edit but that they can easily get feedback and make iterative improvements.    Is this any less the students own work?

I need to be clear here, that I am not suggesting we just jump on the AI bandwagon without thinking.   We definitely need to consider the risks and challenges and to seek to find a path towards the ethical, responsible and safe use of AI schools.    But, we also need to acknowledge we now use many tools which we would not give up.   We would not give up the pen and pencil, the calculator, email and much more, and each of these is more than just a tool for use.    As we have become accustomed to use them they have changed how we as humans think and operate.   These tools have changed how our minds operate.    AI will do the same, and we need to think about it, but if our reason for not using AI is that it will change us, it is cheating or produces things which are not our own work or truly not representative of the real me, then does this mean that we need to give up all other tools including pen and paper and the written word?

Exam Results 2024

It’s the start of the exam results period this week with the release of the Cambridge International results ahead of the A-Level exam results on Thursday and GCSE results the following week.   The pressure on teenagers to perform well in exams such as GCSEs and A-Levels is immense. Schools, parents, and society place great emphasis on achieving high grades, often portraying these results as the ultimate determinant of a young person’s future success. However, while exam results are undeniably important, they are not the be-all and end-all of a teenager’s life. There are numerous other factors that contribute to personal development, mental well-being, and long-term success that deserve equal, if not more, attention.

The Limitations of Exam Results

Firstly, it’s crucial to recognize the limitations of exam results. Exams typically measure a specific type of intelligence which revolves around memorization, understanding of theoretical concepts, and the ability to perform under exam pressure within a limited time frame. However, intelligence is multifaceted. Skills such as creativity, emotional intelligence, problem-solving, leadership, and the ability to work well in teams are not measured by traditional exams but are equally vital in both personal and professional contexts.   I also note that school and college exams, being sat in a hall with hundreds of other students, is far from ideal for many students, plus is not really representative of the kind of experiences students will encounter in life beyond formal education, and there is the research which points to the environmental conditions in exam halls in relation to heat, CO2, etc simply not be conducive of maximum performance.    

Mental Health and Well-Being

This intense focus on exam results can have detrimental effects on a teenager’s mental health. The pressure to achieve high grades can lead to anxiety, depression, and burnout where mental well-being is a critical component of a teenager’s overall development.    Teens need time to relax, explore their interests, and build relationships. These aspects of life help develop a balanced and healthy individual.   I note that I put too much pressure on myself in relation to exams when I was younger, which led me to difficulties with mental health mainly related to results below that which I expected in my Higher grade exams in Scotland.    These results were not the end of me, as I felt they were at the time, but actually spurred me on and taught me a valuable lesson regarding the unpredictable nature of the future, and how so-called life defining moments, such as exam results day, are simply just a stop on a much longer life journey.

The false narrative

The focus on exam results, as will be evident in the press over the coming week or so, perpetuates a false narrative of winners and losers, where students are unfairly categorized based on their results. This binary perspective suggests that those who achieve high grades are destined for success, and are “successful”, while those who do not make these grades are not. Such a narrative overlooks the diverse range of talents, skills, and potentials that exist beyond the confines of standardized testing.  It also overlooks the multitude of factors which might impact a students exam performance.   Success, or failure, in my view, is determined by the longer picture of your life, your contributions, your memories and efforts, and exam results are a small part of that in the longer term.

Conclusion

Life is a journey rich with diverse experiences, and exams are just one of many milestones along the way. Whether a pass or a fail, exams do not define a person’s entire future; they are simply a moment in time, a stepping stone on a much larger path.  For those getting their results this week and next, know that they are a small part of your life.   If you do well, celebrate, and then move onto the next stage in your life.    If you don’t do well as defined by your own expectations rather than the press or others, learn from this.   It may be that exams weren’t for you, that your performance was impacted by external issues, that the courses didn’t fit your interests, that you didn’t put in enough effort or many other factors;  That’s fine, so pick your next steps and move forward.   I know I did and with that I will leave but one last thought:  I learned from my disappointing Scottish S5 Higher results and worked harder to get to University in S6 but it was many years later before I learned about the need for balance in relation to study or work, and our wider lives.    How do you balance out work/study with the wider need for fun, enjoyment and need for experiences and human flourishing?   As your exam results come and go, I would suggest this is something well worth considering.

Digital Exams

I suspect digital exams are the way of the future albeit I see challenges in getting there after around 100 years of doing things on paper.   Change will be a key challenge but is not the only challenge as was evidenced by the digital exam I supported this morning.

Now our support for these exams goes above and beyond what is normally possible however myself and my team do this to try and ensure the students have the best experience of their online exams.   We had around 35 students sitting the 2hr online test and myself and two of my team had made ourselves available.  A ratio of 1 tech (if I count myself as such although this is debatable these days) to 10 students isnt something that could be achieved if exams such as the GCSEs or A-Levels went heavily digital so this is the first point to consider.    If exams went largely digital the tech staff to student ration would be way higher than this and I would suspect run to 1:100, 1:200 or higher.  But for this exam we were heavily staffed and this morning, this was a good thing.

In terms of our infrastructure and Wi-Fi as a school we have invested heavily in infrastructure to support 1:1 devices and the embedded use of technology throughout the school.  As such having the infrastructure to support digital exams isnt an issue although going forward we would need to look at additional Wi-Fi capabilities in the large sports halls where exams might be held, but where we hadn’t previously installed more than a single access point, which may struggle as soon as you have 300 students sitting a digital exam at once.

Client devices is easy as we are 1:1 and for those external students sitting the digital exam with us, as we operate as an exam centre during the process, they are simply told to bring a suitable device and we then provide them access to the Wi-Fi.   So all good so far.

And this is where the difficulties appear;   During the course of the exam a number of students were unable to start their exam.   Their devices were ok, they had access to Wi-Fi, they had the software, but still they couldn’t get in.    The issue was on the exams platform end.  The admin portal we could see showed everything was fine and they were listed and had joined the exam, but on their devices it wouldn’t let them proceed.   The usual steps were taken by myself and the team;  Log out and back in, shut down and restart the app, etc, but to no avail so the next step was a call to the exam board help desk to see if they could resolve the issue.    After around 40 minutes or so the issue was resolved and the students went about their exam but this was maybe 5 students from 35, 14% of students.    How would that work when there are 300 students, that would be 42 students with difficulties?     And what about the students themselves, nervous enough about an exam and then presented with issues while their fellow students get on with their exam, forced to listen as we wade through a help desk automated call options to finally reach a person who could help?    And even when we did get through to someone they then had us work through a resolution check list of steps, where most we had already done and were unlikely to resolve the issues at hand.

This is where we will see the challenges in terms of the infrastructure provided at the exam board end to minimise the risk of things going wrong, and then the support provided to deal with issues where they do go wrong.   If the infrastructure is robust then you hope to have a failure rate of maybe 1 or 2% of students, which would be manageable even at 300 students.   I wonder if the exam board from this morning would ever release data on its failure or issue rate;  I suspect not however from our exam session the rate was around 14%.   And if the support is there then you would quickly be able to get to someone who can help and then resolve the issue, ideally in 5-10 minutes as a maximum, rather than the 30 to 40min we experienced.     For me it is clear that for digital tests the work the exam boards need to invest heavily up front as otherwise students will suffer.

It s a bit like EdTech in general;  If you don’t invest in the fundamentals which includes your infrastructure, including hardware and software, plus your support provision, it is likely your tech will not be reliable enough to provide a good service to users.    From an EdTech point of view this is bad enough but put this in the context of students sitting their terminal exams which might shape the options they have available to them at the next step in their educational journey;   the stakes are too high for things to go wrong.

Exams and AI: A look at the current system

I recently presented at a conference in relation to AI and assessment.   I think this was reasonably good timing given JCQ had just released further guidance in relation to student coursework and AI plus AQA had announced they were going to use online testing as part of their exam suite in the Italian and Polish GCSEs starting from 2016.    I think this is a positive step forward in both cases however I think it is important that we see this journey as more than simply replacing pencil and paper exams with a hall full of students completing the same exams but as an online/digital exam.   There is significant potential here to ask ourselves what are we seeking to assess, why are we seeking to assess it and how are we best to assess?

The SAMR model

The SAMR model is useful when looking at technology change programmes.   The first element of SAMR is that of simple substitution, similar to the example I gave above in the introduction.   The concern for me is that this might be the goal being aimed at where technology and AI present such significant potential beyond mere substitution, and where the world has moved at a fast technologically drive pace, yet our education system has changed little, and our key assessment methodologies, of terminal coursework and exams have barely changed at all.

In looking to progress beyond substitution it might be useful to unpick some of the limitations of the current system.  For this purpose I am going to focus purely on terminal exams given they are such a significant part of the current formal education system in the UK.   So what are the limitations of the currently accepted system?

Logistics

One of the key drawbacks in the current system, as I see it, is the massive logistical challenge it presents.   Students have to be filed into exams halls across the country and the world all at the same time, to complete exam papers which have been securely delivered to exam centres.    Its quite an undertaking and even more so when you consider trying to keep the papers and questions secure.   In a world of technology where content can quickly and easily be shared it doesn’t take much before questions are out in the open ahead of the exam, advantaging those who have seen the information when compared with those who have missed it.    Then you have the issue of gathering all the completed papers up, sharing them with assessors to mark, quality assurance of marking and then eventual release of results to students some months later.    This is a world where technology supports the sharing of information, written, audio, video and more instantly.  Why cant the exams process be quicker and more streamlined, making use of technology to achieve this?

Diversity

Another key drawback has to be that of diversity.  We, more than ever, identify the individual differences which exist in us all.    Discussion of neurodiversity is common at the moment but despite this we still file all students into a hall to complete the same exam paper.     Now there are exam concessions which can be provided to students but this barely scratches the surface in my opinion.    Where is the valuing of diversity in all of this?

Methodology

We also need to acknowledge that the current exams system very much values those students who are able to memorise facts, processes, etc.   Memorisation is so key to exams success however out in the real world we have access to ChatGPT and Google to find the information we need when we need it, with the key then being how we then interpret, validate and apply this information to the challenges or work in front of us.    Shouldn’t the assessment methodology align with the requirements of the world we live in?   Now I will acknowledge the important of key foundational knowledge so I not suggesting we stop teaching any basic knowledge, but knowledge and memorisation should be less of a focus than it is now.

Conclusion

I believe technology could address a lot of the drawbacks listed above.  Now I note the use of technology will present its own challenges but how often do we find the “perfect” solution?    Wouldn’t a solution which is easier for schools to administer, is quicker and more efficient, is more student centred and more in line with the world we now live in be a good thing?

AI and assessment (Part 1)

I recently spoke at an AI event for secondary schools in which one of the topics I spoke on related to AI and its impact on Assessment.   As such I thought I would share some of my thoughts, with this being the first of two blogs on the first of the sessions I delivered..

Exams

Exams, in the form of terminal GCSE and A-Level exams still form a fairly large part of our focus in schools.  We might talk about curriculum content and learning but at the end of the day, for students in Years 10,11, lower 6 and upper 6 the key thing is preparing them for their terminal exams as the results from these exams will determine the options available to students in the next stage of their educational journey.   The issue though is that these terminal exams have changed little.   I provided a photo of an exam being taken by students in 1940 and a similar exam in recent terms and there is little difference, other than one photo being black and white and the other being colour, between the photos.   The intervening period has seen the invention of DNA sequencing, the mobile phone, the internet and social media, and more recently the public access to generative AI but in terms of education and terminal exams little has changed.

One of the big challenges in terms of exams is scalability.  Any new solution needs to be scalable to exams taken in schools across the world.  Paper and pencil exams, sat by students across the world at the same time accommodates for this.  If we found life on Mars and wanted them to do a GCSE, we would simply need to translate the papers into Martian, stick the exams along with paper and pencils on a rocket and fire them to Mars.   But just as it is the way we have done things and the most easily scalable solution doesn’t make paper and pencil exams the best solutions.   But what is the alternative?

I think we need to acknowledge that a technology solution has to be introduced at some point and the key point is the scalability based on schools with differing resources.   As such we need a solution which can be delivered in schools with only 1 or 2 IT labs, rather than enough PCs to accommodate 200 students being examined at once as is the case with paper based exams.  So we need a solution which allows for students to sit the exams in groups, but without compromising the academic integrity of the exams where student share the questions they were presented with.    The solution, in my view is that of adaptive testing as used for ALIS and MIDYIS testing by the CEM.   Here students complete the test online but are presented different questions which adapt to students performance as they progress.   This means the testing experience is adapted to the student, rather than being a one size fits all as with paper exams.    This helps with keeping students motivated and within what CEM describe as the “learning zone”.   It also means as students receive different questions they can sit the exam at different times which solves the logistical issue of access to school devices.   Taken a step further it might allow for students to complete their exams when they are ready rather than on a date and time set for all students irrespective of their readiness.

AI also raises the question of our current limited pathways though education, with students doing GCSES and then A-Levels, BTecs or T-Levels and then onto university.    I believe there are 60 GCSE options available however most schools will offer only a fraction of this.    So what’s the alternative?    Well CalTech may provide a possible solution;  They require students to achieve calculus as an entry requirement yet lots of US schools don’t offer calculus possibly due to lack of staff or other reasons.   CalTechs solution to this has been to allow students to evidence their mastery of calculus through completion of an online Khan Academy programme.   What if we were more accepting of the online platforms as evidence of learning and subject mastery?   There is also the question of the size of the courses;   GCSEs and A-Levels and BTec quals are all 2 years long but why couldn’t we recognise smaller qualifications and thereby support more flexibility and personalisation in learning programmes?   In working life we might complete a short online course to develop a skill or piece of knowledge on a “just-in-time” basis so why couldn’t this work for schools and formal education?  The Open University already does this through micro credentials so there is evidence as to how it might work.   I suspect the main challenges here are logistical in terms of managing a larger number of courses from an exam board level, plus agreeing the equality between courses;   Is introductory calculus the same as digital number systems for example?

Coursework

Coursework is also a staple part of the current education system and summative assessment.    Ever since Generative AI made its bit entrance in terms of public accessibility we have worried about the cheating of students in relation to homework and coursework.    I suspect the challenge runs deeper as a key part of coursework is its originality or the fact that it is the students own work but what does that look like in a world of generative AI.    If a student has special educational needs and struggles to get started so uses ChatGPT to help start, but then adjusts and modifies the work over a period of time based on their own learning and views, is this the students own work?   And what about the student who does the work independently but then before submitting asks ChatGPT for feedback and advice, before adjusting the work and submitting;   Again, is this the students own work?  

There is a significant challenge in relation to originality of work and independent of AI this challenge has been growing.   As the speed of new content generation, in the form of blogs, YouTube videos, TikTok, etc, has increased year on year, plus as world populations continue to increase it become all the more difficult to be individual.  Consider being original in a room of 2 people compared with a room of 1000 people;    The more people and the more content, the more difficult it is to create something original.   So what does it really mean for a piece of work to be truly original or a students own work?

The challenge of originally and students own work relates to our choice of coursework as a proxy for learning;   It isnt necessarily the best method of measuring learning but it is convenient and scalable allowing for easy standardisation and moderation to ensure equality across schools all over the world.   It is easy to look at ten pieces of work and ensure they have been marked fairly and in a similar fashion;  Having been a moderator myself this was part of my job visited schools and carrying out moderation of coursework in relation to IT qualifications.   If however generative AI means that submitted content is no longer suitable to show student learning, maybe we need to look at the process students go through in creating their coursework.    This however has its own challenges in terms of how we would record our assessment of process and also how we would standardise or moderate this across schools.

Questions

I don’t have a solutions to the concerns or challenges I have outlined, however the purpose of my session was to stimulate some though and to pose some questions to consider.    The key questions I posed during the first part of my session were:

  1. Do we need an annual series of terminal exams?
  2. Does there need to be [such] a limited number of routes through formal education?
  3. Why are courses 2+ years long?
  4. Should we assess the process rather than product [in relation to coursework]?
  5. How can we assess the process in an internationally scalable form?

These are all pretty broad questions however as we start to explore the impact of AI in education I think we need to look broadly to the future.    In terms of technology the future has a tendency to come upon us quickly due to quick technology advancement and change, while education tends to be slow to adapt and change.    The sooner we therefore seek to answer the broad questions or at least think about them the better.

2023 Exam Results: A prediction

And so exam results day once again approaches and I would like to share a psychic prediction: That the newspapers will be filled with headlines as to how A-Level results have fallen when compared with last year. 

Ok, so it isnt so much psychic as based on what we know about the UK exams system.    We know that each year the grade boundaries are adjusted and that the trend pre-pandemic was for grades generally to be increasing year on year.    The ever increasing grades werent necessarily the result of improving educational standards or brighter students, although both of these may or may not be the case, they were the result of a decision taken when setting grade boundaries.    With the student exam scores available, the setting of the grade boundaries decided how many students would get an A*, an A, etc and therefore the headline results.    It’s a bit like the old goal seek lessons I used to teach in relation to spreadsheets.   Using Excel I could ask it what input values I would need to provide in order to attain a given result.    So, looking at exam results, what grade boundaries would I need to set in order to maintain the ever increasing grades but while also avoiding it looking like grade inflation or other manipulation of the results.  Now I note that in generally increasing grades across all subjects, some subjects showed more improvement than others, with some subjects showing dips, but summed across all subjects the results tended to show improvement year on year.

And then we hit the pandemic and teacher assessed grades and the outcry about how an algorithm was adjusting teacher awarded grades into the final grades they achieved.    Students and parents were rightly outraged and this system of adjustment was dropped.   But how is this much different from the adjustment of the grade boundaries as mentioned above?     The answer is quite simply that the teachers and often students and parents were aware of the teacher assessed grades and therefore could quantifiably see the adjustment when compared against the awarded grade.   When looking at the pre-pandemic exams teachers, students and parents don’t have visibility as to what the students grade might have been before adjustments were made to the grade boundaries.    They simply see the adjusted score and adjusted final grade.  Now I note that a large part of the outrage was in relation to how the grade adjustment appeared to impact some schools, areas or other demographics of students more than others, however I would suggest this is also the case when the grade boundaries are set/adjusted, albeit the impact is less obvious, transparent or well know.

So, we now head into the exam results following the period of teacher assessed grades with students back doing in-person exams.    Looking at this from an exam board level, and reading the press as it was after the 2022 exam results, we know that a larger than normal increase was reported over the teacher assessed grade years, with this being put down to teacher assessed grades versus the normal terminal exams.   As such I would predict that the exam boundaries will be set in such a way to make the correction.    I predict the exam boundaries will therefore be set to push exam results downwards although it is unclear how much the results will be pushed down.     It may be that the results are reduced slightly to avoid too much negative press or it may be that a more significant correction is enforced based on the fact that this might be easily explained by the previous teacher assessed grades plus also the lack of proper exams experience held by the students who sat their A-Level exams this time;  remember these students missed out on GCSE exams due to the pandemic.

Conclusion

My prediction is that the exam results stats will be lower than last year but not due to students necessarily doing worse, but due to a decision that the results should be worse given last years apparently more generous results plus the fact these particular students have less exam experience than previous years, pre-pandemic.   I suspect my prediction is all but guaranteed but an interesting question from all of this has to be, is this system fair?   I believe the answer is no, although I am not sure I can currently identify a necessarily fairer system.  But I think in seeking a better system, the first step is to identify the current system isnt necessarily fair.

And one more final thought:  To those students getting their results:   All I can simply say is very well done!  This was the culmination of years’ worth of study and effort, and during a period of great upheaval the world over, unlike anything in my or your history to date.   No matter the grades, you did well for getting through it.   The grades, no matter what they are do not define you, but your effort, your resilience and what you decide to do next, your journey is what really matters.    Well done and all the very best for the future!! 

100+ years of exam halls and paper exams

And so, the exams season is in full flow with students across the world once again sitting in rows in exam halls, which are often simply school sports halls, with pen and paper to complete their end of course GCSE and A-Level exams.   If you looked at the halls the setup might be very much similar to exams from 100 years ago or more albeit education is now more accessible to the masses and exam halls now contain posters about “mobile devices” and how these are prohibited.    How is it possible that the exams process has changed so little?

Lets consider the wider world;  I asked ChatGPT for the significant technology advancements from the last 100 years and it came up with the below:

Computing and Information Technology:

The development of electronic computers and the birth of modern computing including the emergence of the internet and the World Wide Web, revolutionizing communication, information sharing, and commerce.

Transportation:

The rise of commercial aviation, making air travel accessible to millions and facilitating global connectivity along with the development of high-speed trains and advanced railway systems, enhancing transportation efficiency and connectivity.   Also, the proliferation of automobiles and the continuous improvement of electric vehicles and autonomous driving technologies.

Medicine and Healthcare:

The discovery and widespread use of antibiotics, dramatically reducing mortality rates from bacterial infections along with the development of vaccines against various diseases, leading to the eradication of smallpox and the control of many others.   Additionally, advancements in medical imaging technologies, such as X-rays, MRI, and CT scans, enabling non-invasive diagnosis and improved treatment planning plus progress in genetic research and biotechnology, including the mapping of the human genome and the development of gene therapies.

Space Exploration:

The first human-made object in space, the launch of Sputnik 1 in 1957, and subsequent manned space missions, culminating in the moon landing in 1969.    The establishment of space agencies like NASA, ESA, and others, leading to significant advancements in space technology, satellite communications, and planetary exploration.   And more recently the development of reusable rockets, such as SpaceX’s Falcon 9, reducing the cost of space travel and opening up opportunities for commercial space exploration.

Energy and Sustainability:

The expansion of renewable energy sources, including solar and wind power, as alternatives to fossil fuels plus improvements in energy storage technologies, such as lithium-ion batteries, facilitating the growth of electric vehicles and renewable energy integration.   This combined with a greater focus on sustainability and environmental awareness, driving innovations in energy-efficient buildings, green technologies, and eco-friendly practices.

Communication and Connectivity:

The evolution of telecommunications, from landline telephones to mobile phones, and the subsequent development of smartphones with advanced features and internet connectivity.   Also, the introduction of social media platforms, changing the way people connect, share information, and communicate globally and the advancement of wireless communication technologies, such as 4G and 5G, enabling faster data transfer, enhanced mobile connectivity, and the Internet of Things (IoT).

Conclusion

A lot has changed over the last 100 years, with a lot of the above occurring maybe in the last 10 to 20 years, yet in education we are still focussed on terminal exams like we were over 100 years ago.   We still take students in batches based on their date of birth and make them sit the same exam at the same time.    These exams are still provided as a paper document with students completing them with pen or pencil while sat in rows and columns in sports halls in near utter silence.  The papers are then gathered up and sent away to be marked with results not available for almost 3 months.

The above might have been ok 100 years ago but with the modern technology available to us now surely we should have made some progress.    I suspect, although there have been those who have suggested change, there hasn’t been a catalyst to drive it forward.   My current hope is that recent advancements in Artificial Intelligence (AI) and recent discussion regarding its use and potential, may be the catalyst we need.   Here’s to not still using the same exam processes 10 years from now, never mind 100!

Originality

Producing original content is a fundamental aspect of creating meaningful and valuable information for audiences across various mediums. In terms of assessment within schools, colleges and universities, students are expected to produce “original” work to evidence their learning.

So, what does it mean to produce original content? At its core, originality means creating something that is entirely your own. This could be a new idea, a fresh perspective on a familiar topic, or a unique approach to storytelling. Whatever the case may be, originality is about bringing something new and valuable to the table that hasn’t been seen before.

But lets flip that premise;  There are a limited number of words available and these words are shared with all writers for all time, so as people continue to write the probability of two people writing the same thing can only increase.   It’s a bit like buying a lottery ticket.   The more tickets you buy and the longer period over which you buy them, the more likely you will hit the winning numbers.   And that analogy may fit in other ways in that the probabilities of a winning lottery ticket and an exact match of wording and phraseology may be similarly unlikely.    And the longer the piece of writing the less likely whereas for shorter pieces of text, the probability is greater.   But either way it isn’t impossible!   

Let’s step back for a moment and look at an academic concern, that of plagiarism. Plagiarism is the act of taking someone else’s work, ideas, or words and presenting them as your own. It’s a form of intellectual theft and can have serious consequences, including invalidating qualifications or exam results for students who are caught.   “Taking someone else’s work” and “presenting as your own”;  But if I read something, agree with it, and then present it as my viewpoint haven’t I just taken someone else’s work and presented as my own?    Does writing it in my own words make it original and my own contribution and at what point?    How many words do you need to change before it becomes my original contribution as opposed to plagiarism?   I note that the plagiarism detection services I have used in the past present a plagiarism score which tries to quantify how similar a piece of work is to other pieces of student work on file.  And if I combine with readings from other sources is this better or just plagiarising from a number of sources?     And what if I get AI to write the first draft of the content, then I refine it?    Is this plagiarising from the multiple sources the AI used as training data or simply plagiarising from the AI, or maybe it isn’t plagiarism at all? Considering art work rather than writing, if I get an AI to produce a self portrait of Van Gogh but painted in the style of Monet, who have I plagiarised?

I don’t believe the concept of originality and of plagiarism, beyond plagiarism of a paragraph of cut and paste text, was ever an easy issue in schools albeit we have treated it as easy in the past.   With AI this issue becomes that bit more complex and difficult to traverse.    We may present our students with the assessment and with a marks scheme, but do we need to start providing more discussion in relation to originality, and what acceptable use of AI platforms might look like?    I suppose the challenge here is do we know what this might look like.    

But a bigger question may be why we ask for these written assessments to be completed in the first place;   Is the written work a proxy for evidence of learning and understanding, where this is easier, and possibly more reliable, than actually having a discussion with each and every student to check their understanding?   And if we can no longer rely to the same extent on the piece of extended written work do we need to move to more student/teacher discussions, but if so, how will we address bias and other factors impacting on individual teacher assessment of students?

Conclusion?

Am not sure the above has presented any answers beyond presenting some of my musings and more questions.    But for now that maybe enough, to try and add to the discussion in relation to education and how it may look in the future given effective AI solutions are already available to our students.

References

Written with the help of ChatGPT (OpenAI)

Technology and Exam boards: Time to modernise?

I recently received a request from a teacher in relation to getting some software installed on their school device to support them in marking for an exam board.    Now I know this isn’t part of their school role however having been a standards moderator in the past, I understand the benefits to schools and colleges of having markers or moderators within teaching departments.   I am therefore eager to try and enable staff by supporting such requests however this request involved a piece of software which requires admin rights to the laptop, both for install and for the operation of the application according to the exam board.   When the concern re: cyber security was raised the exam boards final reply was that the staff member should install the software on a personal rather than school laptop.   This got me thinking about how technology has changed but how exam boards have been slow to change.   This is all the more evident currently.   Just look at the advances in Large Language Models (LLMs) with ChatGPT over the last six months.

Traditionally, examination boards have relied on paper-based tests and manual grading systems. However, these methods have several drawbacks, including the potential for errors and delays in results processing.    One way examination boards could modernize is by moving towards computer-based testing. Computer-based testing allows for faster and more accurate grading, as well as the ability to customize exams to the specific needs of each student.  I very much believe that adaptive testing is the way forward, with this also enabling students to take exams in their own time when they are ready as opposed to at a set time with all other students.   Adaptive testing also supports students taking their tests anywhere, including at home, rather than having to be crammed into a large exam hall where the conditions themselves are not exactly designed for optimum student performance.    Additionally the results would be available much quicker reducing the stress associated with a long waiting period between the exams and the results being released.   There is also the potential benefit in the reduction in the amount of paper used in exams, transporting of these papers, etc, which may help with making the exam process more environmentally friendly.

Another way examination boards can modernize is by utilizing artificial intelligence (AI) in the grading process. This appears all the more relevant at the moment with development in LLMs like Chat-GPT.   AI-powered grading systems can quickly and accurately grade exams, allowing for quicker results processing and reducing the potential for errors. AI can also analyse student performance data to provide insights into areas where students may need additional support and guidance.   Now I note here that the use of AI may introduce new errors to the marking process however I would suggest that the volume or magnitude of these errors when compared with human based marking is likely to be lower.  It isn’t “the solution” to errors but definitely a step in the correct direct.

Related to the above, exam boards need to acknowledge the existence of AI and LLMs and the fact that they will become an increasing part of life and therefore a tool which students will increasingly use in their studies be it for revision, to help in developing critical thought or for creating coursework or other learning content.   So far only IB (International Baccalaureate) have really acknowledged ChatGPT and how they see it impacting on their courses, providing at least some steer for schools on what appropriate or inappropriate usage might look like and proving at least some direction for schools and teachers for managing these new technologies.

Moreover, examination boards can use technology to improve exam security. Online proctoring tools can help ensure that students are taking exams in a secure and controlled environment, preventing cheating and other forms of academic dishonesty.    Related to this, I have seen exam boards continuing to send out resources on CDs or USB drives, or requesting student video or audio work using similar formats.   It is about time that they provided appropriate online portals to allow the quick, efficient and secure transfer of such exam and coursework data.  

Finally, examination boards can use technology to make their exams more accessible to students with disabilities or special needs. For example, screen readers, text-to-speech software, and other assistive technologies can help students with visual or hearing impairments to take exams on an equal footing with their peers.   This is already happening for a subset of students however I suspect eventually will need to acknowledge that all students are individual and having differing learning preferences including their device use and the online tools they use.  In classrooms teachers support students using a range of tools and techniques so it is only correct to seek to support the same in the final exams which are, at least for now, viewed as so critical in a students format education.   As such examination boards will need to adapt to this.

Conclusion

Technology has the potential to revolutionize the examination process, making it more efficient, accurate, and accessible. Examination boards must embrace these technological advancements to ensure that their exams are of the highest quality and that students receive accurate and timely results. By doing so, they can help prepare the next generation of students for success in a rapidly changing digital world.   

And at a time when the pace of technology, particularly in relation to Artificial Intelligence solutions, has never been faster, the exam boards will need to significantly increase their agility and their ability to adapt to and embrace change.

The future of exams

We are now in the exams season with students all over the world sat in exam halls with pen (and pencil) and paper, completing their GCSE and A-Level exams.    5 years ago, it was the same, 10 years ago, 20 years ago, in fact I suspect we could go back over 100 years at we would see a similar scene of rows of students sat taking paper-based examinations.    Isnt it about time we looked at a more modern solution to the need for terminal exams?

Computer Based Testing – Challenges

One of the big challenges in any computer-based examination solution would be the requirement for schools and colleges to have large numbers of computers available for students to use in taking their exams.   If we are simply substituting the paper test for an electronic test, where all students across the country are expected to sit the same exam at the same time, I feel this problem will be difficult for schools and colleges to resolve especially with core subjects like Maths and English.   

We could as an alternative look to allow the taking of tests using students own devices however equally this is problematic as students will not have equal access to equipment and in some cases might not have access to a suitable device, plus there would be concerns in relation to cheating where students are using their own equipment.  We saw some of these issues, particularly in relation to access to technology during the pandemic.

Remote Invigilation or proctoring

There is also a question as to whether we even need to get students into a common location.    Following the pandemic where a lot of teaching went to using online tools and video is it possible to use the same technologies to allow students to take their exams remotely in their own time.    I myself experienced this only a few years ago when doing a Cybersecurity exam which involved remote proctoring and someone monitoring my exam efforts via my web camera.   This might be another option that could be considered however the potential safeguarding implications would need to be considered.

Adaptive Testing

The use of adaptive testing might be another solution here as in this situation the students do not necessarily do the same questions.   The questions are selected from a pool with the adaptive testing solution then selecting subsequent questions based on how the students do in each question.    Using adaptive testing we wouldn’t be as worried about all students sitting the same test at the same time, given the students wouldn’t be receiving the same questions.    As such schools could use their available IT resources over a period of time to allow students to access the relevant tests.   The challenge, I suspect, with adaptive testing will be convincing parents and students that it is fair.  Fairness is easy to point to where all students do the same test at the same time but not so easy where they are doing different questions at different times.

And do we need knowledge based final assessment

We also need to question whether there is still the need for the final assessment of students.   For some students it is an opportunity to show all they have learned, but for others it is a massive stress and a negative impact on their wellbeing.   I have long been a supporter of vocational qualifications based on ongoing assessment throughout the course rather than the heavily weighted final exams of so called “academic” qualifications.   

Additionally in a world where we routinely use technology tools such as google to search for answers and solutions, should we actually be considering how such technology might have a place in future exams, rather than banning such devices from exam halls.

Conclusion

I don’t have an answer for this challenge;  Any change is likely to be difficult especially after over 100 years of terminal exams.    It is however noteworthy that a number of examination bodies are actively looking and trialling alternative digital exams solutions.

Here is another example of where the pandemic has fuelled an exploration of future solutions.    I suspect however it will be some years, maybe 10 or more, before any real change happens, although I hope it happens sooner.