The Royal College of Physicians of Ireland

The Royal College of Physicians of Ireland (RCPI) is the largest postgraduate training body in Ireland and is dedicated to improving the practice of general medicine and related medical specialities. RCPI was established in 1654, and has a global network of 10,000+ Fellows, Members and Trainees who work across 29 medical specialities in 90 countries. RCPI tests over 4,000 candidates a year in both written, clinical and OSCE formats both nationally and internationally.

In 2019, RCPI began sourcing a new online assessment solution. Initially they looked for software that would provide all the tools required to create and store complex medical examinations - question authoring, item banking and exam paper assembly. The RCPI also wished to find a solution that had the flexibility to support different models of computer-based exam delivery into the future.

TestReach was the supplier of choice for collaborative question authoring, banking and exam paper assembly because the platform met RCPI’s requirements with minimal customisation, while also providing different models of online exam delivery.

Having successfully used the TestReach’s authoring capability, in 2020, the RCPI explored using TestReach for computer-based exam delivery throughout a network of worldwide test centres. This approach minimised risk, by keeping question banking and test centre delivery with the same supplier. TestReach’s system is designed to be intuitive to use for exam coordinators and has many features to support a seamless roll-out in multiple test centres.

The COVID-19 pandemic disrupted the RCPI’s plans when the use of test centres became impossible due to restrictions on people congregating indoors. It would require a quick change of approach to meet the challenges the pandemic presented. At this point, it proved very beneficial that the RCPI chose a supplier with both test centre exam delivery and integral remote proctoring delivery options, within one application.

The immediate impact of COVID-19 was that RCPI examinations were rescheduled. But they could not be deferred indefinitely, as these qualifications are essential for doctors to proceed with their specialist training and any significant delays would have serious impact on the healthcare system. The RCPI, in consultation with the exam boards and candidates, made the decision to deliver exams using TestReach remote proctoring, starting in March 2020. With remote proctoring, the test centre experience is recreated online. Candidates are monitored during their exam by a supervisor who connects to them via video, audio, screenshare and instant messaging. This approach meant that doctors could take their exam, securely and with integrity, from any location, without disrupting the qualification process. TestReach scaled up their service to allow increasingly large cohorts of doctors sit their exams simultaneously in 2021.

The RCPI decision to work with TestReach has been transformative. When the COVID-19 pandemic hit, TestReach’s solution offered the flexibility required to quickly switch exam delivery methods without any delay. This ensured that doctors could continue with their career progression at a critical time, despite restrictions on the use of test centres. Over 2700 exams were delivered in the first phase and this will expand further in 2021.

Around the world, a diaspora of medical doctors can take exams without having to travel long distances. This significantly reduces the times away from work, which is a major benefit given that these candidates are frontline healthcare professionals. It also enables the RCPI to accept applications from over 38 countries, and demand continues to grow.

TestReach includes remote proctoring as an integral part of a complete assessment software application. This creates a secure and robust solution for high-stakes medical exams. It avoids problems with third-party proctoring systems, where exams take place in browser windows, which can be accidentally closed, and where the supervisors may have no knowledge of the underlying assessment software.

TestReach employs and trains its own exam supervisors and this team has grown considerably to meet the requirements of the RCPI, allowing exams to be delivered to larger cohorts of doctors.

Alan O’Mahony, Project Manager at RCPI, said: “TestReach remote proctoring has transformed how we conduct our exams not only from an operational perspective but from the candidate’s viewpoint too. It has improved the professional and personal lives of our candidates who no longer have to travel to exam centres, which eliminates travel expenses and the need to take significant annual leave. We regularly run feedback surveys after exams to gauge the reaction to this approach, and I’m happy to say that that satisfaction is consistently over 90%.”

One RCPI candidate provided this feedback: “I found the remote invigilation really good. It was much easier taking the exam at home as opposed to having to travel the day before after finishing work. I also found I wasn't as nervous sitting the exam at home as I would have been driving to an exam venue and sitting the exam with other people. I would prefer to sit all exams by remote invigilation in the future and I found the whole experience very easy. The online invigilator was very helpful and nice and the trial run a few days prior to the exam was very helpful.”

As well as online exam delivery, the RCPI utilises a wide range of other features, such as:

• Comprehensive question banking tools, to streamline the management of a large bank of complex exam questions for 6 medical specialities. With over 100 authors creating questions, having a central online repository in which questions can be created, reviewed, trialled and added to the live item bank is essential, and all this work can be done remotely.
• Exam assembly tools, including blueprints which enable the automatic creation of exam papers to test the right disciplines at the right standard.
• A pass mark moderation tool, to enable the comparison of the relative difficulty of tests between cohorts. TestReach moderation tools assist administrators in ensuring that students are being graded fairly and that the test difficulty is consistent.
• Question statistics to analyse question difficulty and performance which then feeds into future exam paper creation.

“If an organisation is looking to move from traditional question authoring and moderation, we would highly recommend TestReach, as it really streamlines how your questions are authored, approved and moderated. Also, from a staffing perspective, I would emphasise the strong working relationship we have with the project managers and support team at TestReach, who are great to work with and very responsive,” said O’Mahony.

The RCPI and TestReach are working together as an on-going partnership, with the College giving feedback on product design and testing new prototypes. This helps ensure that the TestReach solution continues to provide innovative and best-in-class tools to run online medical examinations, now and into the future.

View downloadable copy of this case study.