Best Practice in Procurement of an eAssessment Solution

Athens - location of the eATP conference in 2018 where the procurement session was hosted

At the e-ATP (European Association of Test Publishers) conference that took place in Greece last month, TestReach co-hosted a “peas-in-a-pod” discussion session with its client, The Chartered Institute of Procurement & Supply (CIPS). This blog post distils some of the main themes that emerged from the discussion, and will be useful reading for any organisation involved in the transition to e-assessment.

CIPS is the world’s largest procurement and supply professional organisation, and is the worldwide centre of excellence on procurement and supply management issues. In 2017 the organisation ran an extensive tendering process for an e-assessment software provider. In this session, Rachel Wilson, Head of Assessment Standards and Operations at CIPS, shared how she and her team approached this eassessment procurement process and the key learnings they made. Louella Morton, Executive Director, TestReach, added a further dimension to this session by giving an overview of the software tendering process from the vendor point-of-view.

The discussion was divided into three sections, the preparation for a tendering process, running the tendering process and evaluating bids / selecting a preferred vendor. There were some valuable techniques that emerged that can be used by organisations at these different procurement stages.

1. Preparing for an e-Assessment Procurement Process

Rachel Wilson pointed out that the first thing she did when starting into this process was to seek out expertise within the organisation and from external consultants, who could advise on assessment RFP and RFI processes. It was essential to CIPS to seek expert advice, determine exactly what was required from the solution and identify the critical success factors. CIPS was receptive to information coming from vendors, and it was valuable to have some early exploratory discussions with suppliers, who provided suggestions in areas that CIPS might not have considered. In the early stages of research, the CIPS evaluators attended a lot of assessment conferences to understand the various solutions in the marketplace.

The tendering process run by CIPS was very formal, so it was important to have clear timelines for the research and preparatory aspects. Another key success factor was to include the technical team and senior leadership at an early stage.

2. Running the Tendering Process

CIPS were very clear from the offset that this procurement process was going to require an extensive amount of time and resources from the vendors taking part, and this was made clear in all documentation issued to suppliers. CIPS initially issued an RFI (request for information) with a clear list of the “absolute must-haves”, which meant that vendors who could not meet the requirements could rule themselves out before committing more time to the process. It was very important to allow vendors to qualify in or out early in the process and save time for everyone involved in the long run. As Wilson pointed out, “the RFI was an essential checklist of technical aspects that we needed. We sent this RFI out to 16 suppliers, and 10 came back as a fit with the points listed.”

CIPS had both technical and commercial panels who reviewed these 10 vendors’ submissions, and this was supplemented by webinar interviews with vendors to check any assumptions and clarify questions the panels had. This process narrowed the list of 10 vendors down to 5, who then moved forward to the RFP (request for proposal) stage. The assistance of AlphaPlus was very helpful to CIPS, as they had significant expertise in documenting requirements for the RFP.

CIPS looked for vendors to supply proposals in a standard format so that comparisons could easily be made. However, they had, observed Rachel, “quite a variety in responses” - the advice to vendors was to stick with the supplied format, and be concise. Essentially the organisation was trying to see were vendors able to deliver against a very clear set of requirements, and the panels used a scoring system to assess this. There were some items that could not be scored, and were just a straightforward pass or fail. This process narrowed the vendors down further. The short-listed vendors were then invited on-site to present which gave further opportunity to discuss any questions that had arisen from the RFP. Face-to-face meetings also helped in assessing the cultural fit between the organisations.

Louella Morton commented on the process to say that what was really important was that CIPS were flexible throughout, and were not rigid about what was presented at face-to-face meetings. This was very useful as it provided flexibility to the vendor to show innovative areas and approaches, which perhaps CIPS had not considered. “When face-to-face meetings are very rigid and prescriptive, it limits the knowledge sharing that takes place. CIPS were open and flexible to learning from vendors, and because the goals were set out clearly from the beginning, this flowed through the process. We could see the vision and they had the openness to allow us to share our knowledge and insights,” said Morton.

3. Evaluating Bids and Selecting a Preferred Vendor

A formal scoring system was used, and this was supplemented with a ‘softer’ evaluation once the list was narrowed down. CIPS wanted to get a sense of how well the culture of the organisation was a fit, so they considered:

Is the vendor’s experience relevant to CIPS?
Does the vendor have a similar organisational ethos?
Will this vendor develop alongside CIPS into the future?

One of the ways of assessing this cultural fit was by engaging with the vendors’ existing clients, and understanding those clients’ experiences in some detail. Wilson explained that it is good to interact with a range of organisations who already use the respective vendors’ solutions, and understand the scope of how they work together. These clients explained to CIPS how the system works for them, and were honest about their experience. Morton commented that “…I can totally understand how you will learn a lot more about a vendor by engaging with that vendor’s customers in a more in-depth way, and really understanding how they use the system and the benefits it brings.”

The other important activity, once the list of vendors was narrowed down, was to trial the system. The CIPS team tried many aspects of working with the solutions, such as question authoring, test assembly, delivery, marking and moderation. They also had people act as students and give their feedback on the experience from a test-taker perspective. This comprehensive review helped CIPS to determine the extent to which the solution would fit with their processes.

“We gained a massive insight from doing a trial of the system. When you’re down to the last two vendors, having a full trial makes all the difference. We wanted to get to a position where we could compare like-for-like, and it is much easier to compare when you actually use the system. It really adds another dimension to the selection process” said Wilson. As Morton pointed out: “When you do an RFP, and you ask a question like ‘Is the system intuitive and easy-to-use?’ then every vendor will answer ‘yes’! It is only by trying it out that you can really see how the vendors compare.”

As well as reviewing scores given to vendors for different aspects of their solution, Wilson worked with the CIPS Procurement team to review the operational and commercial risks of each solution, and a risk heat-map was jointly developed. The risk assessment was very useful as it helped to differentiate the vendors, and talking to existing clients also provided a means to audit these risks.

From the vendor point of view, Louella added that it was good to be part of a very structured evaluation process with clear criteria set out from the beginning. This meant that everyone was aligned and knew all the important areas. It was positive that CIPS maintained a very objective process, because “relying on gut feel can be way off the mark and introduce bias” as Morton explained.

Attendees at the session also gave input, with one participant pointing out that a face-to-face Q&A session with potential vendors is a really good way of getting clarity on the process in the initial stages. With this approach, you can also see the calibre of the other vendors and assess whether it is something in which the company want to get involved. Wilson agreed with this, saying that “it sounds like a more efficient way to do the Q&A, instead of having rounds of anonymous questions answered online.”

John Winkley, from AlphaPlusConsultancy, commented that RFIs can sometimes come with long “tick lists” of technical functionality, but the reality is that every quality supplier can tick every box, which makes identifying true and important functional differences very difficult. Winkley emphasised that the most important thing to do was to “get the team who actually do the work using the system to try it out.”

A question was also raised about whether penalty clauses should be introduced within contracts to cover cases where suppliers default or do not have the functionality promised. John Winkley pointed out that this needs a mature approach. If things go wrong, reimbursing the client with some money does not resolve the difficulties that the client faces in delivering their service – in most cases, it’s not nearly a sufficient remedy and can negatively affect working relationships. As he pointed out, “once you make the decision it’s a long marriage, you can’t quit in 2 years, you’ve got to make it work!”

Summary of Key Take-Outs from the e-Assessment Procurement Session

Ensure your objectives are established from the outset, and that you issue a clear outline of essential requirements in the initial RFI, which will narrow down the suitable vendors.
Involve your technical team and senior management from the beginning of the process.
Seek expertise internally and externally if necessary.
Involve vendors early and be open to learning from their experience and insights.
Do not be too rigid about how vendor presentations should be done, as you will restrict insight into new ways or processes that might meet your needs.
Set-up a trial of the system.
Use panels of experts on commercial and technical aspects to score and assess vendors.
Assess commercial and technical risks of vendor solutions and develop heat maps to compare these.
Engage with the vendor's existing clients to find out more and assess softer aspects like cultural fit.
Remember it’s a long marriage, put the effort in to make sure you choose the right partner and make it work, as opposed to focussing on penalty clauses.