Examining the Experts in Computer-Based Assessment, with Teresa Jacobs

Teresa Jacobs is a quality assurance assessment professional, with considerable experience of transforming examinations using computer-based assessment.

Read on to hear Teresa’s unexpected and incredibly funny stories about working in assessment. You wouldn’t normally link examinations with entertainment, but this is as entertaining as exams can get!

What piece of advice would you give to anyone planning a career in assessment?

That’s an interesting question as I have never been asked how to get into assessment. Just about everyone I have met has sort of ended up in assessment as a way out of classroom teaching or as progression in a career in administration or IT development. There are lots of different jobs in assessment but generally it helps to have a way with words, an eye for detail, to be very well organised and good at working under pressure. An interest in interpreting stats can help as well. Good customer skills are an asset, as assessment is a bit unusual in that we take money from our customers and sometimes we don’t give them the certificate they want and they are not happy.

Thinking back over your career, what would you rank as the most important innovation in the management of certification programmes?

The most important innovation has to be the computer and the whole shift from paper-based to computer-based testing (CBT) or computer-based assessment. CBT has enabled us to automatically mark assessments, deliver assessments outside the exam room, gather powerful performance stats about our tests, provide assessment on demand and deliver results in an instant. It has also saved literally tons of paper and thousands of air miles. It has also eliminated long meetings and boring sandwich lunches because we can standardise and mark remotely.

What would you say was the biggest challenge facing awarding bodies today? And how can they best meet this challenge?

The competitive environment and the simple fact that awarding bodies often offer assessments in very important subject areas, but those assessments are not compulsory. Basically, nobody really likes assessments so they have to add real value in they eyes of the beholder – the employer in most cases. So, it’s a challenge to get people to want to pay money to do tests that are not compulsory.

There is also a lot of competition in the UK. Lots of awarding bodies are looking to export their qualifications but of course, while English might be a world language not everyone is comfortable about being assessed in English. I mean, how many native English teachers would choose to be assessed on their flower arranging skills or drumming techniques, in say French or Swahili.

Do you think that the use of paper in examinations will be phased out or will it always have a role?

I would like to see paper completely disappear from assessment and move entirely to computer-based assessment. Having seen a storage space the size of an aeroplane hanger crammed full of exam scripts, its hard to justify that level of wastage.

I once received feedback from a candidate who described the experience of handwriting an exam script for 3 hours as torturous. I do not want to be in the business of selling torture. Paper equates to hand writing and who handwrites these days? Most of us take our tablets to meetings or at most scrawl the occasional note.

Then there are the security issues. Such as the call from Catford bus station telling me that a bag full of exam scripts had been found on a 136 bus. Or the time a box of scripts was helpfully left in the recycling bin because the exam marker was not at home from Monday to Thursday and the recycling bins are collected on Tuesday. How would you feel if the script you struggled and scribbled to write at a wobbly desk in a freezing cold converted church cum exam centre, ended up as recycled toilet paper before it was even marked!

On the matter of security, once upon a time, multiple choice exams were marked by an optical mark reader (OMR). This was a fantastic invention that allowed a candidate to shade tiny little boxes in HB pencil on a paper mark sheet. It absolutely had to be an HB pencil and any rebellious architect or artistic type who dared to use a 2H or a 6B was destined to have their paper summarily spat out unmarked by that OMR machine. It had to be a sharp pencil too otherwise the poor candidate would miss the box or shade two boxes. So the exam was conducted in silence interrupted by frantic grinding caused by pencil sharpening. Have you ever listened to 50 people in an exam room, sharpening their pencils? The invigilator then gathered up all those OMR sheets, shook off the pencil shavings, stuffed them in an envelope, stuck a stamp on and hoped it would arrive at the awarding body’s office. The OMR then gobbled up the sheets and spat out the results in seconds, unless of course some rebel had used a 6B pencil.

Or perhaps not. Let’s say this was a high stakes test. The invigilator gathered up all those OMRs, put them in the unscrupulous boss’s office. Now the pencil has two enemies, the sharpener, as detailed above, and the rubber/eraser. Magically overnight all the wrong answers were changed to the right answers. The OMRs were given a shake to remove the bits of rubber, stuffed into an envelope and magically everyone passed. So, there are very good reasons to ditch the paper and especially the pencil. And you can read more about the benefits of computer based assessment here.

What are the most dramatic changes you have seen in the running of exams over the course of your career?

Well apart from the technological changes, I think the most dramatic changes have been in terms of the test takers rights. Back in the old days you took your exams, suffered writer’s cramp, then waited ages for the results and when that fail slip finally arrived you ranted and raged that it’s not fair, maybe enjoyed an underage alcoholic beverage and resigned yourself to a resit. Did you ask why you failed? Did you get any feedback? Did you appeal or complain about the wobbly desk or the freezing exam room? Did you even know what a Reasonable Adjustment or Special Consideration was? No. You did the resit 6 months later and the exam room was even colder. Actually, I am not sure that many people know what a Reasonable Adjustment or a Special Consideration is anyway. Sadly, asking for 25% extra marks because your hamster got toothache on the exam day is not considered reasonable or indeed special. Seriously, assessment is much more transparent and regulated now and that’s great. I am proud to work on making assessments fair.

The other major change I think is that we are moving more and more into the assessment of practical competence and that’s the right direction. Back in the dark ages I did an O Level (that dates me) in Domestic Science. Now I was and still am the most undomesticated scientist. In plain English this means I can’t cook. However, I came top of the class. That was simply because of the weighting of the assessments. Most of the marks were for remembering the recipes, knowing a carbohydrate from a hydrocarbon, the calorie value of a banana and dollop of treacle and what kind of spoon to use. I didn’t really have to cook anything so I did well because I knew how to cook in theory. It’s easier now because the ready meals I live on, despite my A grade in domestic science, have the calorie counts on the packets. Of course, now the assessment is practical and quite rightly I would fail.

Should we be running assessments differently? What would you change if you could?

Nice question. I have held a couple of roles where there was consensus that we should be running assessments differently and it has been my task to make the changes. So, my answer is that I was able to make changes and most of them were positive. Notably using computer-based assessment in multiple choice examinations, taking exams out of the exam hall and introducing remote invigilation. These days I don’t think I could work for an awarding organisations that was still firmly committed to paper-based tests.

Another big change is that assessment is now much more of a profession. I have enjoyed working as the expert in mark schemes or MCQs, standardisation processes and mark adjustments. These are transferrable skills that have enabled me to learn about all sorts of subjects ranging from Islamic finance, music production, wine and food pairing and safety in the construction industry. Not surprisingly, you can learn a lot by going straight to the mark scheme for the exam.

What’s the most unusual assessment story you have encountered or experienced?

I like this one and I touched on it earlier. Here’s the full story. The test provider calls me up to ask if we will accept a candidate’s attempt at an exam. The situation is that the candidate booked a remotely invigilated exam, which he intended to take from the comfort of his arm chair, wearing his favourite furry slippers etc. On exam day he pops out in his car to buy some chocolate to chomp on while doing his exam - but loses track of time in the sweets isle, realises he is running late and won’t make it back to his armchair and slippers in time - so he pulls into a Tesco car park. Gets onto 4G, does his exam with only a few distractions from shoppers loading their car boots with the weekly shopping. He drives off. He scores a distinction. Did I make this up? No, though I cannot say categorically that it was a Tesco carpark as the remote invigilation recording focusses on close up rather than the whole forecourt.

Then there is the story about the exam candidate who got so nervous and queasy about his exam that despite his best efforts to keep calm and carry on he vomited onto his script 2 hours and 45 minutes into his exam. Yes, vomited onto the script containing all his hard work. Now I’m in QA and spend a lot of time spelling out what to do if this, that or the other happens and recording it in the risk assessment. What to do if a candidate is sick on a nearly completed script? Well we don’t actually interpret that in the exam regulations.

Well, long story short, the offending script arrived at Head Office, in 7 layers of plastic and helpfully marked ‘script that was vomited on’. Deliberations ensued on what to do next. Open it? Send it back and ask the candidate to open it, copy it and send back the copy? Offer a free resit? From the back of the room Andrew declares, “I‘ve opened it a bit and its all crisp and dry!” Great – now we can’t really send it back.

Long story now coming to a short end. I finds myself at the photocopier, with a pair of surgical gloves on, wearing a face mask and spraying anti-bacterial surface cleaner left, right and centre. No, I did not make this up.

Then there was the launch for an awarding body consortium-based qualification in door supervision. We were offered a beautiful venue with plush carpets, velvet curtains, mahogany furniture and a very tasteful glitter ball. It was free. Yes, please we say and thanks very much. A few days before the launch a call comes through from one of the partner awarding bodies. “Do you realise that Spearmint Rhino is not a sanctuary for an endangered species? Spearmint Rhino is actually a gentleman’s club and I think they do that dancing that you never see on Strictly…. Lap dancing or whatever it’s called. We can’t launch a qualification there!”

Well, we explain, door supervisors do work both inside and outside of gentlemen’s clubs and they are some of the best trained door supervisors in the business. I hope the lovely lady at the awarding body is still telling the story of when she went to a lap dancing club as part of her assessment role.

I started work at a new awarding body and when I arrived on my first day, the office was just about deserted - everyone was at Abbey Road recording studio. How cool is that! It beats the disapproving looks you get from the public when banging on the doors of closed pubs at 8.30am, which are doubling up as training centres. So, awarding can also be rewarding. By the way, if you visit a pub or a brewery in the line of training duty your chances of a cup of tea are not great but samples of the core products abound. Also, did you know that you can spit wine and still taste it, but you have to swallow beer. That’s another story.

There are days when I wish I could make my dream April Fool’s trick a reality. Why don’t we ask candidates what grade they want on their certificate instead of going to all this effort and then not giving them the distinction they wanted. It would leave me with fewer stories to tell so maybe not!

Have you had a professional mentor who was especially influential in your career? If yes, what lessons or advice have proven to be most impactful for you?

Dr Brian was my inspiration. We worked together on early CBT RASCH analysed English language tests. I realised that there is a real skill in writing questions. I then became very interested in the common complaint of ‘its not fair’. I realised that quite often tests were not fair. I developed a passion for validity and reliability and bored people with explanations of the difference. These days I hope I am a mentor to others and I still recall Dr Brian’s advice on summative and formative assessment.

What social media platforms do you use regularly, and, of these, which one do you find to be the most useful in your professional life?

I’m an exams person. I’m not sociable! No, seriously, I use LinkedIn on a daily basis. I rarely use Facebook. My go-to equipment is my iPhone and I'm so glued to the screen, it’s a wonder I have not gone completely cross eyed. Then there’s my Kindle, loaded with 348 books, which pretty much goes everywhere with me.

What business book or blog do have you found most interesting or useful in the past year?

Pass – I read to escape.