8 Questions to Ask Before Choosing Online Assessment Software

What are Online Assessment Tools for?

You might think that online assessment tools for examinations would be primarily used by educational institutes such as schools, colleges and universities. Although it is the case that higher education is engaging more with using software to assist in running faculty exams, there are a wide range of other organisations utilising software in this area.

Online assessment tools are employed by a wide range of organisations to run examinations via computer based delivery, and this is done for varying reasons, and there are many benefits to this approach. Many corporations and public bodies will use online assessment tools to run recruitment screening tests as part of the pre-employment process. They will also use them for internal promotion assessment and for tracking staff knowledge and retention.

Other organisations, such as professional membership associations and accredited training companies, use online assessment tools as a way to deliver examinations to students who are seeking a key qualification in their respective field. Because these qualifications are often for professional students who are already in full-time employment, it is important to offer flexible online options for candidates to sit their exams.

Choosing Online Assessment Software

It is very easy to generate a list of desired features for running examinations, and you can use them as a checklist to ‘tick off’ which supplier has the features you need. But if most suppliers can meet the required features, what then do you do in order to more fully understand what company is going to be the best fit for your organisation’s needs? Read on to hear our take on the top 8 questions you should ask yourself or your supplier when trying to compare your online assessment options:

1. What does my organisation want to see from a new supplier?

Even if you have the responsibility to choose the supplier that will be used to run your online assessments, you need to involve your technical team and senior management from the beginning of the process. Seek out as much internal and external expertise as possible. Take into account views from a range of colleagues. All this can then feed into the generation of a key set of objectives from the beginning. This will in turn help you to narrow down the number of suitable vendors and make a choice that fits the requirements and objectives of all stakeholders at your organisation.

2. What are my ‘must haves’ and what are my ‘nice to haves’?

Remember that in making any comparative judgement of software, it is crucial to have a list of the ‘must haves’ to help you start narrowing down your selection. Then when you’ve narrowed down the list to your ‘must haves’ you can start to see how many of your ‘nice to have’ features are included. While this is a useful exercise, it is also important to retain some flexibility and be open to hearing about features you may not have considered. We would recommend that in meetings with suppliers, ask them are there any features that have not been addressed already, that they think might add value or be useful for you in the future.

3. Is there a good cultural fit with our organisation?

As human beings we all want to enjoy our daily work. If your work involves running assessments, this means whichever assessment software supplier you pick is going to have a large impact on your day-to-day work. Also, once you decide on a supplier, it can be hard to change. There is the implication that this is a long-term relationship. If the various contacts you make at the supplier are pleasant to work with and you can discuss things easily with them, it will make all the difference both to the enjoyment of your work and the success of the projects involved. We would suggest that when you’re making a decision about a supplier, do not just have meetings with your commercial contact on the sales side. Ask the question, if you decide to use the software, who is going to be your daily contact on the project management and technical support side of things? Find out their names and have a conversation with them if possible. If you feel they are friendly, supportive and knowledgeable – that’s a very good sign that the company staff are customer-focused. Check what the company culture and key aims are – is customer focus and customer happiness their main goal? If it is, that can be very reassuring. You can also ask for permission to contact other clients working with the supplier. If this is possible, ask them questions about the culture of the organisation, which can give you insight into how much of a fit they will be for you.

4. What types of exam delivery do we need now and in the future?

It can be easy to get caught up in the day-to-day requirements that are facing you right now. Even more so during the current Covid-19 pandemic, when just getting through the next day can feel like a challenge. However, when making a choice of what software to use to run assessments, try to envisage both the current and the future requirements of your organisation. For example, maybe computer-based exam delivery in a local test centre might suit you now, but perhaps in the future having the option to run remotely proctored exams would be useful? If that was the case, choosing a supplier that had all exam delivery options in one solution would be a good idea to give you some flexibility to change direction if required. Another area to consider may be question types and the question authoring process – perhaps today you only use simple pick-one multiple choice questions. But in the future, your organisation might be interested in presenting case studies on screen and using constructed response answers e.g. spreadsheets, essays and graphs. Or using multi-media options like playing video or audio and asking questions about what the candidate has seen and heard. If you think in the future other question types might be useful to create very immersive assessments, do consider the full range a supplier offers when you are at the comparison stage.

5. Is the software easy to use for both administrators and candidates?

I don’t think there is any SaaS supplier that will say that their software is difficult to use! Think of the Dilbert cartoon where Dilbert mentions to a colleague “Your user requirements include four hundred features. Do you realise that no human would be able to use a product with that level of complexity?” to which the colleague replies, “Good point, I’d better add ‘easy-to-use’ to the list.”

What do you do if all your prospective suppliers say they are “easy-to-use”? We would advise that you make sure you can take a good look at the system (via a demonstration) and see it for yourself – is the interface clear and easy-to-navigate? Does the layout make sense with options clearly presented? If you have narrowed down the number of suppliers you are considering to a small number, you might also consider arranging a trial of the software. To do this you can choose a few colleagues who would be using the system and arrange access for them to see what it’s like to use in practice. Also, if you are thinking of running exams with remote proctoring, why not be a trial candidate? Ask for some colleagues to be enrolled in the exam and ask them for feedback on the experience.

6. What level of customer and candidate support is provided?

No matter how easy-to-use a system is, if you are running online assessments for many candidates around the world, you are going to need a good support desk at the supplier’s company. You will want quick and easy access to support agents who can help with both the administration of the system, and to support candidates when they are sitting exams. Ask what hours can support be provided? How can they be contacted? How quickly can you expect a response – as in, what is the service-level agreement (SLA)? This will all give reassurance that you have the technical back-up you need when you’re rolling out assessments.

7. How easy is it to access key reports and statistics on examinations?

With any online assessment software, it is really important to be able to access the data you want on the exams you are running. This can include detailed question reports, that show you how effective your questions are in practice. By looking at time spent on a question, or the facility or discriminatory index (which are measures of how difficult a question is), you can tweak your questions over time, thus improving the reliability and validity of the question bank. You will also want to access reports on results, with easy options to moderate those results. If you’re assessing a supplier in this area, ask for a demonstration of all the reports that are available, so you can truly understand all the data and how it can be viewed. See for yourself if the data is easy to access and presented in a clear way.

8. How do you want to mark your examinations and what marking tools do you need?

Depending on the question types you use, you may need to facilitate a team of examiners who will manually mark all the candidate answers. If your question types are all auto-scored, then this is not so relevant to you – although you may want to keep this in mind if you plan to extend your question types in the future. If manual marking is required, try and look for a solution that has a range of marking features that help to simplify the management of a team of markers and enable certain standardisation features such as double-blind marking and rubrics. You can read more about marking online at our blog post on the subject here.

The TestReach team hope that this summary of questions to ask will be a good reference for anyone who is looking for an online assessment software solution and trying to compare options available. If you have enjoyed our blog, please sign up at the link below to receive our monthly blog update. Don’t worry – we won’t spam you – it will just be one email, once a month, so you can receive the latest information on online assessment straight to your inbox.