Examining the Experts in Assessment, with Geoff Chapman

What piece of advice would you give to anyone planning a career in assessment?

Start with a large international company, learn good practices, soak up wisdom from cross-cultural team-working and be exposed to well-known exam owners with big challenges. Get a maths and/ or statistics qualification if you’re interested in test development or analysis. Assessment and Psychometrics degrees are rare in the UK, but do exist in the USA.

If the delivery of tests stimulates you, get a qualification in facilities management, geography or logistics. If you want to manage an assessment company, get an engineering degree and an accountancy qualification. Every engineer I’ve met who works in assessment totally gets it: the process, procedure and system inter-linkages, as well as the overarching big risks.

Question assessment leaders who have never taken a high-stakes exam after compulsory education. Do they think exams are for ‘other people’ or do they not believe in assessment as a career?

Thinking back over your career, what would you rank as the most important innovation in management of certification programmes?

Social media changed everything. Finding study groups, tips on exam preparation, malpractice and actually locating those exam owners – social media ripped up the marketing book. The black box of advertising in magazines and 96-sheet billboards still exists, but we have the new black box of algorithms and a fluid ‘Wild West’ of global compliance and legality to work out.

In many countries, the tri-net (Google-Facebook-Amazon) has direct influence over 70% of internet traffic. In Portugal, ISPs offer cheaper internet access based on ring-fenced tri-net access, not the full internet. So for some key countries, exam owners are compelled to align with tri-net providers as the only realistic way to be visible to learners, candidates and members.

What would you say was the biggest challenge facing awarding bodies today?

I’ve codified 12 common challenges, including malpractice and integration. Exam owners can meet these challenges by understanding which to address first, what’s ‘Business As Usual’ and what grows their organisation. Most only handle three challenges at a time, for example, internationalisation.

One exam owner I work with believed their home market was saturated. Setting up new routes to new markets, tailoring products, setting up new risk management processes can be incredibly disruptive. Ironically, admitting that the skills and experience to make that happen is not “in the room” is often the biggest challenge.

Exam owners and suppliers are also often held back by legacy ways of working and business culture that have not moved with contemporary social norms and practices.

Do you think that the use of paper in examinations will be phased out or will it always have a role?

Paper has a role in three approaches:

• 1. To build a new qualification and use on-screen assessment as a sandbox to get stakeholder buy-in.
• 2. Have parallel delivery of on-and-off screen tests to maintain equivalence.
• 3. Phase in a completion conversion over a number of years.

Also, many countries will always lack traditional internet infrastructure, so it’s mobile-first: PCs and laptops are not ubiquitous. However, exam owners often frown upon summative exams performed on small screens, so that’s where paper will also persist.

It's bizarre that some experts claim paper is still needed to accommodate candidates with vision or motor-skills challenges, or have IT unfamiliarity. Paper delivery actually locks out special-needs candidates. Candidates with special needs are better catered to with screen-based delivery, which can be adapted to their requirements.

Familiarisation and opportunities to practice are omnipresent: test prep is a massive industry, especially in countries such as China and India.

What do you think have been the most useful technical advancements in assessment software?

Technical advances such as plagiarism detection software, identity/credential checking and on-screen marking of digitised scripts have gained traction. I believe the most useful technical advancements to getting the job done are outside of the assessment world.

Collaboration tools have the biggest business and useful personal impact because they enable colleagues and exam owners to quickly corral knowledge, understanding and experience.

Online messaging tools (Slack, Hangouts), workflow software (Trello, Asana) conferencing (Zoom, Skype) and sharing features within item authoring software empower exam owners and suppliers to get things done and offer continuously improving service. Why wait weeks for an answer when it can be crowd-sourced, or you can reach out easily to an expert?

What are the most dramatic changes you have seen in the running of exams over the course of your career?

Competition and Markets Authority investigations notwithstanding, this sector is not known for drama. As an old boss said to me, “If you want drama, go and work on your Hamlet!”, but some causes célèbres in the last two decades come to mind:

2018: Indian Railways: 9.2 million high-stakes exams delivered in three days

2008: Microsoft introducing simulation into its professional exams

1998: VicRoads in Australia introducing a hazard perception video into its driving theory test, three years before the Brits got there.

The speed of introduction, often using agile techniques, has also catalysed change. Some of the changes involve facets we take for granted in Western society. For example, exam owners are trying country-specific and culturally-acceptable payment methods such as M-Pesa in Nigeria, AliPay in China, and Paytm in India where candidates do not have a bank account.

This is quite a change from the early ‘00s when cheques, postal orders and cash-at-the-centre were common payment methods for UK-based exams. The currency plans being formulated by the social media giants will also create drama in this context.

Should we be running assessments differently? What would you change if you could?

Did you know that most exams are still created by a single person? I read a paper on England’s GCSE Physics exams and how those exams are still created by a single person, rather than compiled from an item bank. The paper claimed that there is no strong evidence to say one method produces ‘better’ papers than the other. I would make item banking the default method.

There are multiple upsides to item banking, so I’m puzzled why it’s not the default. Multiple exam series, ‘test-when-ready’, adaptive test delivery, LOFT, as well as giving protection and confidence to the qualification. It seems strange not to change.

Changing awareness is also important. For example, in areas such as the professionalisation of the invigilator/proctor. Remote invigilation has been the big driver to start that change – because it is far better to have an invigilator who performs the task everyday, than a warm body handing out papers for minimum wage and their petrol money for six weeks of the year.

What’s the most unusual assessment story you have encountered or experienced?

It’s difficult to stop becoming blasé to cheating stories, but tales about beating the cheater still make me chuckle. One exam owner told me they caught a candidate who had an implanted a subcutaneous radio behind their ear and was reading the exam questions to a remote scribe. Apparently, the implant is a technique used for duckling and seal tracking, although I’ve read they can also induce the receiver to empty their bladder!

Have you had a professional mentor who was especially influential in your career? If yes, what lessons or advice have proven to be most impactful for you?

Professor Cliff Beevers at Heriot-Watt University was always accessible, easy to relate to and personable to me. I’ve found that keeping his maxims to hand such as ‘good practice, not best practice’ and ‘nobody knows everything, and if they did, nobody likes a smart alec!’ helps to prevent hubris and conceit.

What social media platforms do you use regularly, and, of these, which one do you find to be the most useful in your professional life?

I tweet as @geoffchapman_ most days to share good practice, and LinkedIn rarely: nothing else.

The level of aggression and bullying on Twitter is frightening. I’m amazed that teachers stick with it for CPD. Maybe I should follow new people! LinkedIn seems back-slappy, insubstantial and incredibly clunky for retrieving useful information. If assessment leaders are on social media in a business capacity, they must show leadership – not just retweeting their boss, company press release, or some social media ‘guru’.

It’s a daily occupational hazard to avoid social media noise, ads, kudos-giving and FOMO. Filtering and analysis tools are wonderful, but being judicious with your time is key.

What business book or blog do have you found most interesting or useful in the past year?

I read many business books during my post-graduate studies. Some books have potent and seductive hypotheses, such as Matthew Syned’s 10,000 hours theory, which have now been debunked.

They dodge muck-and-nettles issues such as performance plateaus and continuous improvement, but that book will just gather dust at WH Smiths or Easons at the airport!

I like assessment books such as ‘Making Good Progress’ by Daisy Christodoulou, and ‘Embedded Formative Assessment’ by Dylan Wiliam. They are leading practitioners who give clear, pragmatic paths to fixing problems. However I can’t imagine either will have Danny Boyle asking for a Hollywood screenplay soon!

Biography

Geoff is an assessment consultant with two decades of international education experience. Geoff shares his insights and approach with presentations, blogs and other free resources on his website www.geoffchapman.com and via Twitter @geoffchapman_.

Geoff's consultancy bridges and solves shared issues across exam owners, regulators and technology companies.

He supports customers who need to change their existing assessment and learning processes, procedures, platforms and strategy.

Geoff helps exam owners with their assessment and learning technology choices, and technology companies with their strategic issues.

A published author on E-Assessment, Geoff is also co-owner of trade publication, World Exam Tech. He holds contemporary qualifications in assessment, education, IT service, project and bid management.

He is a member of international trade bodies including Association for Education Assessment in Europe (AEA Europe), Association of Test Publishers (ATP), British Assisted Technology Association (BATA), E-Assessment Association (EAA) and Education Policy Institute (EPI).