Common Online Assessment Mistakes and How to Avoid Them
It is a big step for many examining organisations to move their examinations online. There are so many areas to consider such as question creation, exam paper set up, marking, moderation and results reporting, as well as moving the delivery of the exam itself onto a computer-based platform and deciding what form that will take.
There can be challenges on the path to moving online. There are many stages to this process which must be approached in a phased manner to ensure success. There will also be a number of stakeholders who need to be kept informed and engaged to ensure a roll-out proceeds smoothly. This affects not only professional accrediting bodies but also corporations who need to run online assessments for the hiring process and to ensure employees are aware of key compliance or safety information.
With a wide variety of challenges to overcome, moving tests online can become a tumultuous task. So we have asked a number of eAssessment experts to give us their views on the common online assessment mistakes they witness. You can learn from their experience and can be aware of what to watch out for!
And if you enjoy this blog post and find it useful, don’t forget to subscribe at the end of this post to receive a once-a-month update, containing useful assessment news and information from our team.
Seven Mistakes in Online Assessment Procurement and how to Avoid Them, by Graham Hudson
Graham Hudson is an International Examinations Delivery Practitioner at the GA Partnership Ltd. Graham is a long-standing member of the educational assessment community, having worked in the sector for 35 years, and has contributed to our “Examining the Experts” blog series in the past. Here are his thoughts on how to ensure the procurement process for an online assessment solution is conducted in a successful way, and he is happy to talk to anyone starting into this process who would like advice.
This last year has been one of rapid procurement for many organisations. Feeling stranded as the tide of past security receded, many wondered what to do. Time will tell whether or not the choices made in very pressurised circumstances will remain. Some lessons learned in my many years of being involved in procurement at all levels, may be of help to others who are starting this journey. Here goes!
1. Not being clear about what you want
Please don’t fall into that trap of thinking that you have to produce multiple pages of detailed requirements before you start a procurement. You will get it wrong. I’m not saying that in a nasty way – but it is inevitable.
Instead, be crystal clear on what you want – what the landscape will look like when the change has been made. How much of your offering has to change and in what way? What are the outcomes that you want to see? How far into your existing systems will change be required? What do your stakeholders want?
Get that clear and many of the future issues will evaporate. Suppliers will help you with the ‘how’ if you describe the ‘what’.
2. Not treating suppliers as equals
Get early engagement with suppliers. Explain what you want to achieve. Get into dialogue soon. Invite them to question you until your brain hurts! Do not go down the arms-length path of giving perfunctory responses to genuine questions. Do not undermine trust by missing deadlines, changing your mind or even worse cancelling a tender process. You need them more than they need you!
3. Not having a clear process
You need a clear process, shared with suppliers at the start, one that includes a realistic timeline that will only be varied by exceptional circumstances. Get that sorted out before you take anything to the market.
4. Not discussing budget (money) early on
I know that the discussion of money gets some organisations all worried. Will the suppliers ‘price build’ to my budget if I tell them what it is? Is there some arcane governance procedure that prevents this? Has anyone in the organisation actually discussed it before going to tender?
There is no point in wasting your time or suppliers’ time by not doing this early on. It is part of treating suppliers equally. But, more to the point, it gives everyone in the process the right starting point from which to work.
How many times have you asked for a quotation (in business or in personal life) only to find it was way more than you expected? (Or, more worryingly, even way less!) Then entering into uncomfortable horse-trading. Get it on the table right away!
5. Not having dedicated staff for the project
I have worked in project and programme management for over 20 years. I have an axiomatic saying; ‘Busines-as-Usual Always Wins’. It does – full stop! Make sure you ring-fence staff to do this work or big frustrations, poor work and missed deadlines will result. More to the point, suppliers will lose confidence in you and may even pull out of the process.
6. Not including ‘Culture’ as a criterion
Sometimes we forget this. We can think of all the other ‘hard’ and ‘soft’ criteria, but not include this one. You have to get on with suppliers. There’s no point trying to work with people you don’t get on with, no matter how technically competent they are.
7. Not having explicit backing from the Chief Executive or Board.
I ask all my clients to put in a paragraph written by the CEO of the organisation committing to seeing the procurement process through. It signals intent from the whole organisation and builds trust with suppliers right from the start.
Key Areas to Consider when Moving Online, with Matthew White
Our second contributor to this blog is assessment consultant Matthew White, who is an expert in building assessments and managing the quality of assessments on a national scale. Here are Matthew’s thoughts on some of the mistakes he has witnessed in the move to online assessement:
Three key areas to consider in moving online:
1. Logistical Oversights
Well, where to start. There are the obvious logistical oversights of course. I worked in one organisation whose e-Assessment pilot fell over because there were not enough power sockets for the laptops their students were going to take the e-Assessments on!
2. Systems Issues
At the other extreme are the systems issues. Most e-Assessment platforms are primed for easy integration. However, the organisation implementing them often have legacy systems that were not designed or built to integrate and this leaves a massive void that can often be filled by manual manipulation of data and that old workaround of spreadsheets.
3. Managing Expectations and Resource Investment
Perhaps where most organisations go wrong is in having unrealistic expectations from the start. The switch to e-Assessment is often sold internally as a cost reduction exercise (out of necessity) and not from the plethora of benefits (increased security, improved customer service etc.). This means that insufficient resources are assigned to the project and ongoing business-as-usual improvements.
So the key take out here may be that if you are responsible for “selling” internally the idea of moving online, you need to focus on the general benefits of online assessment rather than the cost reduction, to ensure that a sufficient amount of resources and focus is given to the process of moving online. A clear strategic plan, incorporating the benefits as the reasoning for decisions being taken, will help everyone focus towards a common goal.
6 Online Assessment Mistakes, with Geoff Chapman
Geoff Chapman is an Assessment Consultant, who has contributed to our popular Examining the Experts blog post series in the past. Geoff has a lot of experience in moving assessment online, and he lists below the common mistakes that can be made when making the switch. Read on to learn from his insights.
1. Paper on Glass
Replicating paper and analogue processes on-screen is a weak, limiting reason for switching. You don’t physically turn over your Kindle when reading a book! A swipe is much more elegant!
2. Going Global
What is deemed acceptable in one culture or country, can be frowned upon or even considered taboo in another. Mistaken beliefs, such as ‘everyone is online’ or ‘laptops are cheap’, set up many exam owners for a fall when they try to export their qualifications and e-Assessment prowess. Make sure that you understand the infrastructure and cultural requirements for each country you are running online exams in.
3. Psychometric Savvy
Wading through all the data gathered from e-assessment can be intimidating. Understanding the language of assessment and psychometrics is not for the faint hearted. But reaching a better understanding of how learners and the exam itself are performing are critical factors which benefit from making the switch to online.
If you invest in new analysis skills, this will mean better oversight of learning and test centres, cohort performance, and forward fraud indicators. An ability to really understand assessment data reaps huge benefits – but not enough people do it!
4. Re-skilling
Administration tasks involving paper storage, transport and printing are greatly reduced, and usually eliminated, when examinations are moved online. Exam owners often tend to forget the human impact of that change. Experienced administrators need support and re-skilling to support better assessment in an online format. Item banking, data analysis, IT security and test centre support are just some of the ways to re-deploy colleagues displaced by switching to e-Assessment.
5. Learner Impact
More enlightened exam owners have learner forums, whereby the actual people taking the assessments can have their say. Systematically harvesting feedback from end users in a structured, coherent and actionable method is a way to evidence continuous improvement and mitigate negative user impact.
6. Change in Risk Profile
Myopic, ill-advised commentators moan about accessibility and security issues with e-Assessment. But they never compare it to the rotten issues that plague and hamper paper-based testing! No assessment regime can be risk-free. However, just like a credit card payment has benefits over paying with cash, there are also risks. The change in risk profile, when trying to achieve the same result, is often overlooked. No longer do centres have to worry about having enough secure space to store paper. However, they do have to consider equitable access to equipment to access the assessment. It just means there may be different risks which require different approaches.
Understanding Potential Online Assessment Mistakes
We hope you find the points given above of help when planning your own organisation’s move to online assessment. There are obviously pitfalls to watch out for, but the overall benefits of moving examinations online are so wide-ranging that it is an essential step to take.
If you would like to learn more about TestReach, and how we can help you to transition your assessment processes online, please click here to request a demonstration, and we will be in touch.
