Good custom-developed software is not routine work that every programmer can produce. It is a high-quality knowledge product and therefore demands a substantial investment from the client. In this branch, like many others, the least expensive solution can often turn out to cost the most. In the EQUITY project, academic researchers from the VU University in Amsterdam collaborate with IT specialists from ABN AMRO to quantify the yield of IT investments.
This article was originally published in Dutch in I/O Informatica Onderzoek, October 2007
The dream of the Dutch government was called P-Direkt. It was an IT dream that was smashed to pieces in 2005. P-Direkt was a plan to have all government ministries making use of a single central personnel and salary administration system via a shared service centre. More quality for less money – that was the idea. But instead, both the government and IT contractor IBM (and subcontractor Logica CMG) concluded two years ago that the project no longer had a future. Buying the contract out cost the government more than twenty million euros.
This is just one of the recent examples of how large IT contracts can go dreadfully wrong. And it is one of the reasons that it is necessary to quantify what a particular IT investment will yield. What does a company receive in return for every euro it invests in IT? A good deal that turns out to be a costly mistake... or an expensive but ultimately profitable investment? These are the questions driving the JACQUARD project EQUITY: Exploring Quantifiable Information Technology Yields.
In EQUITY, academic researchers from the VU University in Amsterdam collaborate with ABN AMRO employees. VU's Professor of Information Management, Chris Verhoef is responsible for the academic aspects of the project. ‘The IT investments of a large bank run into the billions of euros per year. Thus it’s no wonder that there is a desire to determine what such a bank receives in return.’
EQUITY is a four-year project that began in 2005. In addition to Verhoef, four PhD students from the VU are also involved. Jean Kleijnen, Senior Vice-President of the IT department, is closely involved with EQUITY as a representative of ABN AMRO: ‘We invest manpower in the project– as much as is required – and by providing all the necessary data for confidential use. The PhD students can work in our offices as often as they like, and they each have a personal workspace. Every three months we meet to discuss how the project is proceeding.’
Golden nuggets in the sand
Verhoef’s contacts in ABN AMRO go back to the beginning of the 1990's. ‘From the bit-level to the management-level,’ he explains. He worked as an adviser for various financial institutions. It was not until 2001, however, that he also gained permission from ABN AMRO, ING, Deutsche Bank and others to publish his scientific research. After all, he required hard business data for the research, and companies were initially understandably reluctant to have that data published – even anonymously – in scientific journals. But ultimately these companies realized that they too could benefit from more scientific research into the quantification of IT yields.
Verhoef: ‘Company problems form an important source for my scientific research. I know what practical needs exist, and on the other hand I see the academic “golden nuggets” in the sand. As the costs for IT expenditure rise, it becomes ever more important to obtain good quantitative insight into the yield of those software investments.’
At the beginning of 2000, ABN AMRO began its own improvement programme to investigate how the price-quality ratio of an IT investment could be optimized. Kleijnen: ‘We quickly recovered the cost of that internal improvement programme. Once we began asking ourselves how good this improvement programme actually was, we asked Chris Verhoef to begin a more extensive study. That is how the collaboration began. My dream was to find a parameter that I could – so to speak – present at a boardroom meeting. That parameter would then tell us: if we invest this much, we will get this much in return.’
In the same manner that the international stock markets have their own index to compare financial funds – via the AEX in Amsterdam or the Dow Jones in the U.S. for instance – the researchers in the EQUITY programme use a “function point” as the unit of measurement. A function point is a measure of the functionality of a system. The greater the scope of the system, the more function points it contains. IT investments are then evaluated in terms of cost per function point. Future costs for maintenance are also included in the calculation. You may be able to purchase an inexpensive system, but if the quality is poorer than expected you may later be confronted with very high maintenance costs. In that regard, the purchase of an IT product is just like the purchase of a car or a house. Verhoef: ‘In construction you calculate in cubic metres. The contractor can build a block of flats with so many cubic metres for such and such a price. Likewise, you can see function points as the cubic metres of the IT branch.’
Millions of lines of code
A quantitative analysis serves many objectives simultaneously. First and foremost is the determining the yield of an IT investment. ‘The IT portfolio of a large company – whether a bank, an insurance company or a government agency – contains a million lines of programming code,’ explains Verhoef. ‘Often we are looking at custom work, not commercial off-the-shelf solutions. The scientific challenge is to extract from those million lines of code the quality characteristics the company is interested in. We are not examining the errors in the code but rather the possibility to extract management information from large quantities of code.’
Besides determining what an IT investment yields, there is also a second reason to quantify IT performance. Kleijnen: ‘The EQUITY project has helped us enormously in handling contract negotiations with the software developers. We can now substantiate why we think a function point should have a certain price and not cost more.’ By presenting hard numbers, we create a healthy balance of power in the price negotiations.’
A third reason can be found at the management level. Managers had to make decisions about IT investments on the basis of a sort of IT weather prediction: How much maintenance do you expect in the future? How much additional expense results from changes to the product requirements during development? Verhoef: ‘Here, too, it is just like the situation with a house. Once the first blueprint is finished and has been sent to the contractor, the wishes often change. You want an addition here, an extra room there and so forth. Suppose you request changes representing 2% of the total project every month – the question is what that will cost and whether it all fits together as was originally intended. It is fairly common that extra requirements arise during implementation. But you want to know the extent to which it is still prudent to add extra requirements and when you would have been better off to start with a new design. With P-Direkt you saw that the implementers ultimately could not deliver the promised product at the agreed price, probably because the product requirements were continuously changed.’ This is the kind of situation ABN Amro wants to prevent, and that is the motivation to investigate what actually qualifies as healthy growth.
Stimulating collaboration
Kleijnen is very pleased about the collaboration with the VU researchers. ‘Bringing together academic researchers and business people should produce a win-win situation and EQUITY certainly makes good on that promise. It is positive that such collaborative associations are actively stimulated by the government. That is also a form of initiative. But it does require willingness from both sides.’
Jean Kleijnen senior vice-president of the ABN Amro IT department, is involved in EQUITY as a representative of the bank.
Chris Verhoef VU University professor of information management, is responsible for the academic aspects of the EQUITY project.
In March 2007, the international IEEE conference on international EQUITY-related projects took place in Amsterdam: http://www.cs.vu.nl/equity2007/equity-report.php
