Remote Online Proctoring / Invigilation and the Future of Assessment - A report from a recent Association for Educational Assessment Europe (AEA-Europe) webinar.

This week the Association for Educational Assessment Europe (AEA-Europe) hosted a webinar on the subject of online proctoring (remote invigilation) and the future of assessment. The presenter was Louella Morton, co-founder and Executive Director at TestReach.

Many awarding bodies are moving to e-assessment and are adopting new exam delivery methods such as online proctoring. There are a range of technologies and methods that can be used to provide online proctoring (or remote invigilation) in various different ways, which can lead to confusion about specific technical requirements and how everything works in practice. In this webinar, Louella Morton explained the ways in which online invigilation can be provided, the types of technology involved, the impact of artificial intelligence and the ways in which this type of examination delivery has transformed the assessment processes at a range of awarding bodies. Some more details on the subjects covered is provided in this blog post, and because there was a lot of information covered, there will be a separate blog post on this subject next month.

What is involved in Online Invigilation / Proctoring?

Just to complicate things, remote invigilation has lots of different names. In Europe, it tends to be known as remote invigilation or online invigilation, and in the US it tends to be known as proctoring, online proctoring or remote proctoring. This refers to the process whereby the exam hall experience is recreated online. But they mean the same thing, that instead of coming to a test centre, exam candidates can take their exam on a computer from any location – as long as they have internet connectivity. The exam candidate can log into their exam and sit it online, while being securely invigilated over-the-web by a remote supervisor.

There are two main steps with online proctoring. The first is that the supervisor has to authenticate the candidate, which is to identify that the person sitting the exam is the correct person. When you are authenticating the candidate, typically there is a review done of the individual’s photographic ID. This would usually be photographed so that there is a record of the person who presented to sit the exam. In addition, there may be some biometric scanning to supplement that process, such as key stroke pattern recognition. The second step is to secure the environment, and this might involve panning the camera around the room, and looking at wrists and ear pieces. Once the authentication process and environment check have been completed, the candidate can sit their exam.

Types of Online Proctoring Solutions

The first remote invigilation option is that the supervision is conducted in real time. So you have actual humans carrying out the supervision and it’s all done over the web. Typically the supervisor will monitor the candidate using audio, video and remote screenshare, so they can see the candidate, hear the candidate and they can see their screen and that monitoring is very up close and personal. When a person is sitting an exam their frame of reference tends to be the screen, keyboard and mouse, so they are very still when they are doing an exam. If there is any kind of movement, supervisors will monitor this very closely. In an actual exam hall environment, you can have the opportunistic cheaters, so for example when the supervisor walks away you can see their back is turned, giving the candidate 30 seconds to check something before they turn around. With remote invigilation, you do not have that issue because the candidate cannot see their remote supervisor, although the supervisor can see them. Also the session is always recorded, so if there is any behaviour that is called into question there is a full record and it can easily be analysed.

The second option in online proctoring is “record and review” which is where the session is not monitored in real-time, but the session is recorded and looked at after the exam has finished. Typically a person reviews high-speed footage after the event. Alternatively, instead of reviewing all the footage, random samples may be taken. So, for example, every 5th exam is reviewed, or every 10th exam, and the costs will vary depending on how many hours of video reviewing have to be done. Technology can be used to enhance detection, so similar to live invigilation if noises are heard, then certain things can be flagged. Technology can also be used to review the video footage, but humans are always involved in the process.

How does Live Online Proctoring compare to “Record and Review”?

If you compare the live invigilation option vs. the ‘record and review’ there are a number of things to consider. With live proctoring there can be immediate intervention if people are cheating, so if someone was copying down the questions the exam can be immediately terminated. You have the benefit that people are involved the whole time and they are by far the best decision makers. Also, candidates can be a bit nervous when they come to do their exam, and they really like having a person at the other end of the line who understands the assessment solution and can answer questions if required.

With ‘record and review’ invigilation there is no real time intervention and everything happens after the exam event. That can be a big problem because there are some organisations where item banks (exam question content) are incredibly precious and the last thing you want is someone who transcribes your exam questions and puts them up on the internet. So ‘record and review’ may be suitable for lower-stakes exams or where cost is an issue.

What about Artificial Intelligence and Online Proctoring?

In the area of online proctoring, there is a lot of talk about artificial intelligence, because this is an area where artificial intelligence could add a lot of value. So for example, computers could watch videos and highlight things and actually learn what behaviour looks like cheating. However, we need to be cautious about the immediate impact of artificial intelligence.

If you consider the muffin and the chihuahua challenge as per the above photograph, when humans look at these photos, we can easily identify which is the dog and which is a muffin. However, when Microsoft and Google computers tried to identify a picture of a muffin the answer given was either a brown and white teddy bear, a close up of a stuffed animal or a dog. These are the results that artificial intelligence gives with a static image, and the analysis of video footage is much more complicated.

So at this moment artificial intelligence in remote invigilation is more of a ‘work in progress’ and helps humans to make decisions. We are a long way from a computer being able to monitor a candidate and identify cheating behaviours.

What does the candidate need?

There can be a lot of misinformation about what is required to sit an online proctored exam, with huge broadband speeds sometimes suggested. However, a candidate just needs:

• A standard desktop or laptop (no specialised hardware is required).
• A working webcam and microphone (to make sure invigilator can see and hear the candidate).
• Minimum internet connection speed of 0.5 Mb/S. A 3G network will deliver these types of speeds so it is not excessive, but you do want to have a stable connection, without huge fluctuations.
• A quiet location.

How is Online Proctoring Technology Set Up?

There are three different technologies that are used in remote proctoring which affect the way it can be delivered. The first is where, as with the TestReach assessment software, you have a single system that is an assessment system and a remote invigilation system, and they two are completely intertwined. The second type is where you can have a separate assessment solution and remote invigilation solution, and they are integrated with each other. This is where you might already have an assessment solution, and you find a separate proctoring provider to ‘bolt onto’ your assessment software. In this model, the candidates typically go through the proctoring system before accessing the assessment system. The third option is ‘embedded’ which is where the proctoring solution is embedded into an assessment solution (often called “white labelled”) so you don’t see a separate proctoring system and there are no integration points, as one system is built into the other one.

Facts to consider when reviewing different proctoring technologies

• Is the candidate process simple? Sometimes systems can be clunky and there’s a lot of technologies involved, so it’s important to have a system that is easy to navigate.
• Can the system work behind firewalls? Some remote invigilation systems can be tripped up by firewalls, so if you have exam candidates who want to take exams while based in their office, so this is something that needs to be checked.
• Is the system based on older technologies that require downloads, which may be blocked in certain environments?
• Does the proctoring system look different to the assessment system? If they are two separate systems, is the candidate experience very different in each system?
• How reliable is any integration point? For example, we’ve heard of a situation where 1000 people sat the exam in the assessment system, but only 998 were showing on the proctoring system.

Next month’s blog post will cover some of the other topics from the AEA-Webinar, including the impact of remote invigilation on awarding bodies, training of supervisors, benefits of using this technology, tips for building a business case, and advice if you’re looking at this area.