Today’s Challenges and Considerations for Medical Schools Seeking to Modernise Exams

The Challenge of Delivering Online Medical Assessment

In the world of professional certifications, examining bodies must achieve the highest quality of examination provision, while delivering valid and reliable assessment. Professional exams help individuals to progress in their careers, but the majority of participants find them stressful. Balancing the level of pressure on candidates with the stringent requirements of accrediting bodies is a significant challenge.

There is an argument to be made that medical exams are the most high-stakes examinations of all. From doctors to nurses to pre-hospital emergency care staff, the people we trust with our health must maintain the highest standards at all times. Thus, any changes to medical examination provision must be carefully considered. The ramifications will impact staff and could, potentially, have negative consequences for patients.

When everything is so high-stakes, it can be hard to change how things are done. The move to the use of the online medical exam began gradually. Prior to the pandemic, at TestReach we talked to medical colleges who were keen to modernise their management of question banks and exam papers. Moving online would make it easier for subject matter experts, who tend to be dispersed, to collaborate when writing and reviewing exam questions. Other areas of interest included the use of detailed statistical analysis on the performance of questions, to ensure that questions were asking for the correct information in the right way, accurately assessing the candidate’s understanding of the topic.

For the most part, examinations were run in test centres, and there was a gradual move towards delivering those exams on computers instead of on paper. Working on a computer was easier for medical students who could, for example, zoom in on an x-ray as part of question resources, or for written answers, could copy and paste text, as we are used to doing when writing emails and documents. It is very rare nowadays to spend hours writing information down by hand.

However, once the pandemic hit, the option of asking medical staff to congregate to take examinations together in a test centre was an impossibility. At the same time, the pressures still existed for individuals to complete their qualifications and progress to the next stage of their practice within hospitals. This meant that medical schools and colleges had no choice but to quickly move exam delivery fully online, with remote exam supervision, to ensure that the exam was completed with integrity.

Remote Proctoring for Online Medical Exams

Online exam supervision is often referred to as remote proctoring (also known as remote invigilation) and is where the exam hall experience is recreated in an online environment. At TestReach, remote proctoring is delivered as part of the overall assessment application. This means that a TestReach supervisor (who is an employee of the company and trained on the software) supervises the candidate online for the duration of their exam, through the use of video, audio and remote screenshare connections. The supervisor ensures that the correct person has presented to sit the exam by checking their ID. They also watch the candidate throughout the exam to ensure there is no cheating. In addition, they continually look out for suspicious behaviour, such as the use of mobile phones, external materials, talking to someone else etc. and if an exam infringement is severe, the exam may be closed down or rendered invalid.

This sudden switch to remotely proctored exams was a huge challenge for many medical schools, overturning the status quo in a matter of weeks. At TestReach we were fortunate to be able to help a wide range of medical schools and colleges in a short space of time. We have rolled out online examinations with remote proctoring delivery for clients that include the Royal College of Physicians of Ireland, the Faculty of Public Health, the Faculty of Sexual and Reproductive Health, the Faculty of Pharmaceutical Medicine and Royal College of Pathologists, amongst many others.

The Future of the Online Medical Exam

Now that pandemic lockdowns have eased, medical schools are still using online medical exam delivery where possible, because of the significant benefits to candidates and to the awarding organisation.

Examinations are stressful, and candidates really welcome being able to take their assessment from home without having to travel to a test centre. Being connected to a remote supervisor, who is welcoming and reassuring, is also a great help. Online medical assessment reduces the amount of time medical staff have to take away from their daily work as travel time is eliminated, which is another significant benefit. Before the use of remote proctoring, it wasn’t unusual for a candidate to have to take multiple days off work to fly to a test centre in another country. Not only inconvenient, but also comes with an excessive carbon footprint. With an online medical exam, there is more flexibility as to when and where exams can be run, removing a lot of the test centre restrictions.

The Royal College of Physicians of Ireland was already working with TestReach on the management of exam questions and papers prior to the pandemic. When Covid struck, a quick and very successful move to remote proctoring was made. More details on this customer story can be read more in this eassessment case study.

Key Areas to Consider for Online Medical Assessment Solutions

There are a number of key factors that medical schools may want to take into consideration when moving to remote examinations, which could make a significant difference in transitioning to online assessment and the experience of candidates sitting exams. Here are the top five considerations when analysing a possible solution for your medical organisation:

1. Quality of Remote Proctoring Service provided

Although there is a lot of noise about Artificial Intelligence (AI) in remote proctoring, the reality is that humans are much better than computers at processing video streams and detecting subtle attempts to cheat, and this is unlikely to change in the short to medium-term future. Given the high-stakes nature of medical assessment, based on our experience it is definitely the case that medical colleges prefer live proctoring to any kind of automated supervision.

With live supervision, so much of the quality of the service is determined by the invigilators. Always check, who are the supervisors? Who trains them? How is the quality of the service maintained? The standard approach with many assessment software providers is for the proctoring to be provided by a separate company to the assessment solution. At TestReach, the supervisors are TestReach employees, trained and overseen by us, who understand the underlying assessment application. This all-in-one approach leads to a professional, knowledgeable and supportive service to candidates. It is seamless for the candidate to access their exam and move through the proctoring stages.

2. Level of Helpdesk Support

Check that there are adequate numbers of support staff available to provide help and guidance to candidates if needed. This might be during the exam booking process or sorting out a problem with a candidate’s computer where they are failing a system check. What are typical response and resolution times? Can you talk to other customers who can verify the quality and professionalism of the support provided? Also make sure that there are comprehensive critical incident management processes in place, as it is essential to be fully prepared and to have mechanisms to communicate with candidates in the event of something unexpected arising, for example an internet outage.

3. Scalability and Security of Medical Assessment Solution

Some online systems don’t save content on an continual basis, so if the internet fails, some of the candidate’s answers may be lost. It is important to fully investigate this when considering any potential assessment solution. It is also essential to understand what exactly happens if a candidate’s internet connection goes down – can candidates easily get back into their exam and is exam content saved? Internet speeds fluctuate, even with the most reliable connections, so it is really important that systems have built in contingency measures to deal with this that do not unnecessarily disrupt the candidate’s exam. At TestReach, the underlying architecture in the system ensures content is saved and that candidates can easily be resumed into exams in the event of internet issues.

4. Does the software supplier understand online medical assessment?

The question types for medical exams are often very complex and require multi-media options, so it is important to ensure that your question types and formatting options are fully supported. Useful features in online solutions like TestReach include the option of reviewing exam questions in the format that they would be presented to candidates. Also consider exam paper assembly options available. For example, are blueprints in the system to help design medical exams? Is there an option to build multi-part papers? Is there a sophisticated marking platform available for marking constructed-response question types to streamline the marking process?

5. Statistics and Data Analysis

Always consider the quality of the moderation tools and exam analytics accessible in any online solution, and whether the statistical tools will meet the requirements of the examining team and psychometricians. This is an area where TestReach has a lot of experience, such as the use of test equated moderation and standard error normalization.

There are a large number of ways in which online examination software can streamline the administrative burden for medical schools, as well as improve the exam candidate experience. To read more about how TestReach can help with online medical assessment, you can read more here: TestReach for Online Medical Exams.

If you are considering moving some element of exam creation or exam delivery online, please contact us or book an online demonstration.