How the Apotti project will fail, and how to save it!
Finnish Medical Association has calculated that poor information systems take the annual man hours of 600 medical doctors.
Over the years, I have seen software development done in multiple different ways. This picture gives some idea of my experiences in software development and the development of medical devices.
Apotti is the name for the future patient information systems for the cities Helsinki, Vantaa, Kauniainen, Kirkkonummi and the Hospital District of Helsinki and Uusimaa in Finland. Helsingin Sanomat wrote that it is estimated to cost more than the new childrens’ hospital.
Most of us rely on the above type of news to understand what Apotti is about, but there is also a lot of more detailed documentation available from the internet.
Here is the Apotti project plan (PDF):
It is a much better plan than some talk may have lead you to believe. For instance, the strategy includes open interfaces and the idea of being able to use multiple suppliers for the system. We customers, eventual patients, are promised access to our own information so that we can take a more active role in our own care. The architecture is intended to be modular so that individual parts can be changed to something more modern or a different structure in the future. This is wisely intended to make the system also more error tolerant. However, the plan has one flaw which will sink the project.
Apotti has been presented to us as a 10 year development project. This does not mean what you might think. The implementation phase is planned to be 4 years. The calculations are done based on a 10 year period and take into account the maintenance costs for that time. The 430 or so million euros is the budget for the first 10 years include initial development, infrastructure and maintenance for 6 years after the development. This is close to what the current system would cost in just maintenance in the same amount of time. Sounds reasonable then.
There are clear identified benefits during the 10 years. The estimate is that Apotti will save us 330 million euros in that time. When the initial investment for the new system is about 250 million, the payback time is around 7 years.
This could be the reality without that one major flaw in the plan. Unless Apotti project is run by a genie from a bottle, the one mistake that sinks nearly all failed software projects, will also sink Apotti.
Many of us have seen similar projects and we can already, from the past experience, guess how this one will probably go.
This image from the Apotti web site shows the most current schedule I could find. It already shows the flaw that will sink the project. It makes some of us slowly shake our heads. Don’t do that! Again!
Another document worth reading is the cost benefit analysis (PDF): http://www.hel.fi/static/taske/apotti/Kustannushy%C3%B6tyanalyysi.pdf
From these sources we get the noteworthy planned milestones for the project.
|2014||Requirement elicitation begins|
|Mid 2015||Implementation begins|
|End of 2015||Requirements elicitation ends|
|Mid 2016||According to the project plan the first intermediate pilot version of Apotti is taken into use. This is 1.5 years after the development began.|
|2017 forward||Intermediate versions of Apotti are taken into use in production. This will result in savings already during the project.|
|End of 2018||Apotti is fully implemented|
The 2013 cost benefit analysis assumes that during the 3rd year of the project we already get 25% of the benefits which are worth 16.5 million euros. I cannot easily map which year this would be in the above plan but I can assume it to mean 2017.
One Flaw to Rule them All
What then is the one flaw that will rule them all, the one flaw that will cause them all and in the darkness bind them?
It is this: for a complex system it is not only difficult, it is impossible to gather the requirements before the development. If there is no trial and error, constant feedback from the users, the project will result in something that does not fit.
This is especially true in the case of Apotti. If you look at the cost benefit analysis, nearly all of the expected benefit comes from better time usage of the staff. In other words, Apotti is expected to make the everyday work of its 45 000 users so much smoother that they get their work done 10% faster. This is the critical success factor for the whole project.
In Apotti we expect developers who do not directly talk to the users or frequently gather user feedback come up with such a great system that it makes hospital work 10% more efficient. This will not happen unless the users get their hands on the intermediate releases of the software. In the project plan, even the first pilot is to be seen only after 1.5 years of development.
The lack of early feedback, which leads to constantly changing requirements is the main reason for software project failures.
The lack of early feedback, which leads to constantly changing requirements is the main reason for software project failures. The problem is not that the requirements change. They always, always do. If saving 10% of all medical staff’s time is the problem we want to solve with the Appoti software, the solution is so big that it becomes impossible to know what exactly the right solution is or even to fully understand the problem without continuous feedback. When the knowledge increases during the project, the original plans need more and more additions. The money and time needed to solve the problems grow to a size many times larger than what was expected. If, on top of this big uncertainty about the solution, the system is built in layers so that it cannot be used before it is fully developed, in the end you have nothing except the bill to pay.
Here’s what I predict will happen based on my experience is the following.
Requirements gathering continues. It is in the benefit of the supplier that this phase ends on schedule. The more the supplier can use the change request process and bill for each addition or change from the original requirements, the more money they get. The project management on the other hand wants to show that they are on track and want to put an end to the documentation phase. Hospital staff which should take time to participate will not do so nearly as much as they should have. Their daily work will take their energy and not enough is left for reading Apotti requirements documents and commenting them in detail.
At the end of 2015
There will be people in the project who know that there are serious omissions in the requirements. The amount of documentation is already big, internally contradictory, unclear and difficult to follow at this stage. After all, the system is very large and there will not be enough people around who know how to clearly document what the users of the system would want from it. Communicating requirements clearly is a rare skill.
Very few, if anyone, have read all the continuously changing requirements. No one understands the full size and complexity of the system. Some try to say this but they are seen as negative messengers of doom who are never happy with anything. The benefits await. The project must move forward.
The contract for the order is signed a few months late. No problem. There needs to be some tough negotiations from both sides and for a project this size and it is important to show that both sides, the supplier and the customer, are tough negotiators. In fact, they rarely are. They may play tough and make a few demands to give the appearance. If they are lazy, they miss important details or even big whoopsies. Very rarely in this situation both sides try together to find a mutually beneficial agreement.
First half of 2016
Development of the system starts. Obviously most of the development work is done outside Finland or perhaps even in different time-zones. Therefore communication relies on documentation, bad English, email, and some online meetings. Already there starts to be questions about the architecture and what the system is really supposed to do. Some claim they cannot move forward before the questions are answered but many just assume and develop functionality based on what they think is right.
The development appears to be only at the beginning if you look at the progress reports, but in reality, no one knows. It is months before the first system version is seen at all. Mostly, management just looks at reports and mostly believes them.
Second half of 2016
During the first months individual teams have created first versions of their own parts of the system. Now that they are put together, and inevitably new and unexpected problems arise.
Getting the pilot together is a real struggle. Suddenly the project is more late than expected. There are change requests when the customer gradually learns what was missing from the requirements. Implementation has problems too. The buggy pilot is presented no earlier than at the end of 2016, probably later. Immediately when the users see it, they are hopeful of the benefits and see many improvement areas. Much of the functionality is missing or just wrong. Obviously changes are needed.
Sometime between mid 2017 and early 2018
The change request process takes months and the first version that is forced into production is very problematic.
It is late 2017 or early 2018. The project is at least one year late though some still don’t admit to this. There are benefits from the new system but a lot more problems. There are no time savings. In fact, the hospital staff loses time because of the problems, big and small.
At first glance the system seems to work quite well but if one tries to do any serious work, there are all kinds of trouble.
I also have an alternative prediction: the Apotti project customer side may hire some independent testers, good ones, who tear the pilot apart. There are so many bugs and wrong user flows that the first version of the system goes to production only in the late 2018, the year the project was supposed to end. Some still stubbornly claim the project will be only slightly late but many know better.
When the users have used this first version of Apotti for a while, they notice many problems and omissions because the original requirements were “wrong”. The change request pile is sky high. It is however an expensive pile and most of the change requests will not be done.
If we are lucky, by the end of 2019 we have a system that implements something in most of the areas originally expected. Most of the savings are supposed to come from the hospital staff using their time more efficiently from this point onwards. The original goal was reached. There is a common patient database and staff needs to enter patient data only once. However, there are other problems. The system is lacking some vital functions which require workarounds and manual work which eat the time savings. Fixing these problems will take most of 2020. The supplier says these changes were not in the original plan and they cost extra. We’ve seen this before…
Apotti is declared to be late but still a success. Now it actually starts to save time compared to the old system. Ease of use in many parts is not there. Also the processes of the hospital would need changing but they cannot because Apotti does not allow it.
This is the optimistic version of the prediction. Apotti will be only 2 years late and it is actually taken into use. Development schedule will be 170% of the original plan. By this time it has cost between 300 and 400 million euros, at least 100 million more than expected. This is still acceptable since we are aiming for over 300 million euro savings during the first 10 year period. Unfortunately the new system started saving time and money only 2021, 3 to 4 years late from the original plan and therefore the savings are far smaller than originally envisioned.
If the management of the project is hopelessly unrealistic, bad at clarifying confusion and communication and well skilled in yelling their demands, the project may fail completely. The architecture parts may never work together well. The system has so many even patient safety related problems that it cannot be taken into use. It may take years before someone has the guts to admit that the project is a failure and pulls the plug. In the end there is a huge bill to pay and no benefits. We, the tax payers will have to continue paying for an outdated system which exercises the patience of its users.
All this is predictable and most of it is the cause of one wrong assumption: some people stubbornly believe that it is possible to describe a poorly understood complex problem and its solution so that the solution can be developed with minimal changes, without feedback from the users. This is mathematically, scientifically, utterly wrong!
Projects like this more often fail than succeed even partially.
Here is data from Steve Mc Connell’s book “Software Estimation”. A fairly small software project with 100 000 code lines historically has 20% probability of failure or 80% probability for success. Even when successful it will be late 20% of the time. The bigger the project, the more likely it is to fail. A project of about 1 million lines of code has less than a 50% chance of success.
Apotti is bigger than that.
Projects with the expected size of Apotti have a very very small chance to be successful. Clearly less than 40%. Would you bet 430 million euros if you were given 35% probability of doubling your money in 30 years and 65% probability of losing it all?
One example of a typical failure is the originally 25 million euro information system, Vitja, for the Finnish police. See this article from January 2014.
The story says that, because of the delay in the project, it will cost at least half a million euros more and cannot be taken into use before it is complete. Only half a year later, in June, an article says that the Police is cancelling the contract with Tieto (the supplier) completely.
What a dramatic change in the situational awareness! First the project management says they are 2% over budget, far above the average success rate for similar projects, and a few months later they say they have failed completely. What on earth happened?
The most common flaw is this: we expect that we can describe what the system should do, very early in the project. This does not happen. No one has succeeded to do this well. Ever! Just stop believing in it already.
The following picture shows some of such failed famous software projects and their final cost compared to Apotti. When we say “failed” it means that there were no useful results from these projects. The money, time, effort and skills of the developers were wasted. It would have been better for their souls to do nothing. If we are lucky they learned from the mistakes.
The failed projects are not only government projects. The reason for the failures is clearly not that the purchaser didn’t know enough about software projects.
The big story, which many miss, when we talk about Apotti is this: there are well-recognized and well known wastes, money pits and inefficiencies in the health care which costs the tax payer 41.8 million every year. The figure is not 41.9 million euros or 41.7 million euros or even somewhere in the vicinity of 40 to 60 million euros. It is known to be exactly 41.8 million. That means around 100€ smaller tax return in the capital area to all tax payers before Christmas and someone in our health care system knows exactly where this money goes.
There is a clear opportunity for us to improve our health care and save costs.
Here’s my proposal: at the heart of Apotti is the patient information database. This is not small stuff. A small agile team could do it, but to add all the user right management, information privacy, etc. on top, would take a lot of time. There must be something like this available off the shelf.
Whatever the database solution, it can cost some serious money. However, if we make the architecture modular, use as much open standards as possible, declare open interfaces from the core to the outside world, we should be able to find software system suppliers and integrators that have experience in this area. With this approach they don’t need to be such big suppliers.
Using flexible interfaces allows us to develop around this core useful services that we cannot even know to ask yet.
Each part of the system core has a clear and easily understood role.
Each part of the system core can have a different supplier.
The core can be built incrementally. First, we need the interfaces and the first version of the modular architecture. Yes, we expect the first version of the architecture to need changing as well. The modules themselves can be very simple in the beginning and grow to handle more data and more users as the load on the system increases. This allows developing the actual applications very quickly.
Outside the core is the work flow management and application development. We must design the work flows such that they save time and staff nerves. This is the number one acceptance criteria. We don’t necessarily need to be stuck with the current process. The overall system is built incrementally around the common core not as one large project but rather as multiple smaller projects. The first ones should be ready in a few weeks, not years. They are focused on the question how the common patient information system could save time from the staff.
The requirements for the supplier are not in the form of who needs to enter what value to what field. No such details. The requirements describe what the hospital staff want to do and why. For instance: As an anesthesiologist I want to see the patient’s history, allergies and plans for the surgery in the operating room in order to save time and ensure safe surgery for the patient. Then the developers spend a week or two making the first version, go see a few anesthesiologists and ask them if what they have done meet the needs. They listen to the feedback and adjust accordingly.
When the changes are done like this, continuously, very little work is done to the wrong direction and there is very little wasted effort.
The acceptance criteria for the supplier is not if the system was delivered or not but did the system deliver what the hospital needed. In other words, acceptance criteria needs to be that the new system saves 10% of the work time previously spent with the old system.
Waiting more than a year for the first pilot and feedback from the users whose time the system should save is silly. It is like a tailor made you a suit based on one fuzzy picture without fitting it during the process. The first pilot must be ready in less than a month. There is no other way to learn from the users whether the direction is correct or not. They need to try it on. It is completely unrealistic to expect that the future users can picture the final system from the paper documentation. They need to try it out. Everyone who has used an unintuitive old smart phone and something new that just works can witness the difference. The first is a constant pain and frustration, the other a pleasure.
I want our physicians, nurses and administrators to have a tool that was tailor made to fit them. It must be as flexible and light as possible. It should not be a medieval Abbot to whose whim the community must bend. It should be a wise and efficient servant like the staff in our health care.