Evaluation: the problem with ROI

It seems that a day does not go by without an article appearing extolling the need to prove L&D’s worth by measuring its ROI. Yet there is very little, if any, mention of the fact that it is nigh well impossible to measure the ROI of L&D! In part one of two articles, Judith Christian-Carter takes a detailed look at the latest conundrum facing L&D professionals.

The Learning and Development (L&D) function of today is constantly under the hammer, with all manner of advice and entreaties to prove its worth. How often have you been told any or some of the following? “If you can’t prove that L&D is value for money then there’s no business case for it”; “L&D needs to be able to justify its budget; it’s got to be accountable to the business”; “If all L&D does is to take orders for courses then we need to see the proof of those orders”; “Marketing can justify its budget through increased sales, so why can’t L&D?”

As if this was not bad enough, there has also been a veritable deluge of articles, blog posts, etc, over the last few years — most of which have been written by L&D professionals — saying that measuring the return on investment (ROI) from L&D provision is an absolute necessity. Time and time again, this message is repeated almost ad nauseam, with the obvious aim of numbing the poor reader into the unconditional acceptance that measuring the ROI of L&D is an absolute given. However, the problem is that none of these writers provides any valid and reliable way of actually measuring the ROI of L&D Indeed, it would seem that all these people do, by going on and on about ROI, is to show that they do not actually understand what is really involved.

Measuring ROI

ROI is a very easy concept to understand. In economic terms, which usually are how it’s applied, it is one way of assessing profits in relation to capital invested. It is calculated by dividing the return (net profit) by the cost of the resources that were committed (investment). The resulting figure is then usually multiplied by 100, to show the resulting figure as a percentage.

For example, the marketing department has been given a budget of £100,000 (the investment) to spend on an advertising campaign for a new product. As a direct result of the advertising campaign £200,000 worth of the new product is sold (net profit), realising a 200% ROI. Simple, or is it? The key words here are “as a direct result of”, in other words, it can be proved beyond any reasonable doubt that it was the advertising campaign and only the advertising campaign which was responsible for the net profit, and that no other factors were involved.

Measuring the ROI of L&D

Now, this is where it gets extremely tricky, if nigh on impossible. Take this situation as a typical example: The L&D department spends £50,000 of its budget on providing a blended learning course to help sales staff sell a range of existing products. These products are not selling as well as the company had hoped for and it seems that many sales staff are struggling to understand them and to promote them to customers.

Following all sales staff completing the new course, the number of products sold over the ensuing six months is calculated. The value, in monetary terms, of the products sold is converted into the profit realised for the company, which comes to a total of £300,000. The L&D department is delighted when a ROI of 600% is calculated, that is, until someone points out several major problems with this calculation!

Wait a minute!

The ROI for the sales blended learning course was based on a spend (investment) of £50,000 but is this the true figure? The £50,000 was spent on producing two hours of eLearning, instructor-led training (ILT) materials in the form of handouts and slides, and setting up an enterprise social learning network service on the company’s learning management system (LMS). However, other costs were involved which have not been factored into the overall spend.

In order to work out a true “investment” figure, the following costs would need to be calculated and added to the £50,000.

  • The total cost of the time sales staff’s spent away from the workplace selling products — how long did they take to complete the eLearning, how long was the classroom-based element of the blended learning course, and how long did they spend using the social learning network service asking questions and in discussion with peers and tutors?

  • The total cost of the trainers’ time (including preparation time) in delivering the classroom-based element of the course and when acting as tutors to those sales staff using the social learning network service.

  • The total cost of travel and subsistence for all attendees and trainers, and the cost of hiring the venue for the classroom-based element of the course.

Of course, it could be argued that some of the above costs, such as the cost of trainers’ time are hidden costs, in that the salaries of these people are already paid for by the company. However, it is very important that the total costs, including so-called hidden costs, represent those that are incurred by the company/organisation in the provision of any L&D programme/course; otherwise, the resulting ROI is completely false. While all these costs can be calculated, doing so is no small task and will involve a certain amount of someone’s time. In many cases, this requirement alone is sufficient to make people think twice about even trying to calculate a true ROI; that is even before they are presented with the real “show-stopper”!

Proving cause and effect

In order to be able to state categorically that a L&D programme/course resulted in a ROI of xxx%, it has to proved that the net profits achieved were “as a direct result of” the programme/course. This is what is meant by proving cause and effect, and this requires all the variables that could have produced the effect (profit) to be controlled or isolated, just as you would when conducting a scientific experiment.

Taking the above example of the blended learning course for sales staff, many other variables could have been involved in bringing about the increase in product sales and, hence, the profit. For example, the company actively advertising the products and thereby heightening customers’ awareness, the motivation and moral of sales staff due to new financial incentives or due to a decision to reduce their number, the culture of the sales staff changing by becoming more competitive and so on, none of which has anything whatsoever to do with the L&D course.

There is a way of controlling these variables and that is to provide the course to an experimental group and then match the number of products sold by this group with those sold by a control group — where the latter must not only mirror in its entirety the experimental group but also where this group is denied the training course. Not only does setting this up take some doing but, even more importantly, it also means that some much needed training is not provided to everyone when the business needs it to be!

ROI? Forget it!

When it comes to L&D provision, proving a ROI is nigh well impossible and the sooner people realise this, the better. However, this does not mean that L&D is not accountable to the rest of the business for its spend and effectiveness. What it does mean is that, L&D has to find other ways in which to demonstrate its worth; which will be addressed in the next article.

Judith Christian-Carter is a Director of Effective Learning Solutions Ltd. — www.effectivelearningsolutions.co.uk. She can be contacted on 01926 614229 or via e–mail: judith@effectivelearningsolutions.co.uk.