From: Boston SPIN Newsletter February 1997
JANUARY 1997 MEETING REPORT by Ed Maher (Digital Equipment Corporation)
SUMMARY
November's speaker was Michael Mah, Director and Principal of QSM Associates; the topic was "Controlling Software Development".
I found the material in this talk to be interesting. He presented techniques for planning software projects, tracking progress against plans, managing risks, and projecting a completion date based on progress to date.
DETAILS
Michael started out by stating his belief that you can not effectively deal with risks without addressing measurement. Quoting Tom DiMarco: "Risk Management is project management for adults". He pointed out that most people have no problem using measures to gain understanding of things in their personal life; citing the APGAR score assigned to newborns, the Dow-Jones Index, and Cholesterol readings as examples.
On the other hand, people still are uncomfortable using numbers to describe the status of their own work activities, or the status of a project on which they are working. In the software world, people often will measure schedule or effort; but not cost, product size or defects.
He said that many companies do not know how likely they are to complete a given project on schedule. What is more important is that they don't know if the committed schedule is even possible.
He showed us a concept called "The impossible zone". This is based on the QSM database of historical projects. It shows graphically theduration that all of the projects in that database took to complete. The graph had product size (in lines of code) on the X-axis and duration on the Y-axis. It was fairly obvious that (for a given product size) there is a minimum project duration under which it is practically impossible to achieve. Unfortunately, many project managers are pressured into committing to schedules that fall into "the impossible zone", and they either don't recognize it or they are unable to clearly demonstrate it.
He illustrated the concept using some real data from their clients. He showed us graphical representations of the planned size, effort, duration, and Productivity Index (PI) for a couple of project plans {More about PI later on}. One project was planned to complete two standard deviations faster than the industry average for their project type; this looks to be a plan that is highly likely to fail. Another project had a plan that was clearly right in line with industry averages, and in line with with their own historical data; this plan is demonstrably likely to succeed.
Being able to plan in this way requires that you can estimate with some amount of accuracy; and better estimation requires the effective use of metrics. He quoted Ed Yourdon;:
"If you underestimate the size of your next project, common sense says that it doesn't matter which methodology you use, what tools you buy, or even what programmers you assign to the job."
Managers need to use the SEI Minimum Data Set (size, effort, cost, duration, & defects), and "get out of the two dimensional flatlands of only looking at cost and schedule." They need to monitor size metrics and defect metrics.
On the other hand, organizations need to be very careful about attaching judgments to metrics, or politicizing them -- stressing that you should never use these measures to evaluate people. Michael described how trying to capture them can be made more difficult than it needs to be because of people's fear of how the data may be used.
When used correctly, these measurements can allow everyone to see that any blame usually doesn't lie with those doing the engineering or planning. It is almost always a management problem; with people being asked to execute a plan that can't possibly succeed.
The CMM Level 2 and 3 key process areas have associated practices and guidelines that can help to improve an organization's estimatingcapabilities. [Note: The practices in the CMM are more focused on the "what" of estimating and not the "how". The guidelines he is referring to are contained within an SEI document titled: "Software Cost and Schedule Estimating: A Process Improvement Initiative". I did not find this document very helpful. It was more focused on describing how to roll-out a new estimating process -- not on what a good estimating process is.]
He spent some time explaining the concept of the Productivity Index (PI). Simply put, it can't be simply explained. It is a number derived based on size, effort, and duration. A project plan has anassociated planned PI, and a completed project has an associated actualPI. A higher PI indicates an ability to produce product with less effort or time. One easy rule-of-thumb: one PI higher translates to roughly 10% less schedule and about 25% less effort. The PI is explained in detail in "Measures for Excellence, Reliable Software On Time, Within Budget"; by Lawrence H. Putnam and Ware Myers; Yourdon Press. He called this his only shameless plug -- since this was co-authored by QSM's founder.
An organization's knowledge of their PI allows them to better plan and to assess the likelihood of success of the plans with which they commit. In the first example above, they had a plan that called for a PI of 21.2, when the industry average is 9.9. They really need to understand the extreme levels of risk that this implies. He believes that many project failures are due to plans that have an unrealistic implied PI.
He presented some PI data from the QSM database;
| Category | PI | Standard Deviation |
| Business | 16.9 | 4.9 |
| Scientific | 12.1 | 3.9 |
| Process Control | 12.1 | 3.3 |
| System Software | 11.9 | 4.1 |
| Telecom | 11.0 | 3.5 |
| Command & Control | 9.9 | 3.9 |
| Real-time | 7.5 | 3.6 |
| Avionic | 7.0 | 3.2 |
| Microcode | 6.6 | 2.6 |
Someone asked why "Business" software projects are so different from "Avionics"? He pointed out that the PI shouldn't be looked at as a measure of goodness. It can be affected by things like complexity, constraints, and reliability. Shuttle software has to be of significantly higher quality than your average business application.
What about a company that knows if they bid for a project showing a certain schedule, then they won't get the contract? (but their data shows that's what it would take). The solution to this common dilemma is to show the data to the customer to make them more confident that delivery will occur when planned.
He cited Tim Lister as saying that two identical projects can complete at the same time; the team that comes in ahead of plan are heroes, the team that comes in behind plan are goats. The only difference between the two were the deadlines that they were aiming for. Negotiation needn't be just about who comes to the table with the best offer; it can also be about who comes with the most credible bid.
Moving on to tracking; he briefly touched on the concept of using red, yellow, and green traffic lights (red means that an indicator is significantly off track, yellow that it is a little off track, and green means that it is as expected.) The key is to take complicated progress data and transform it into easy to read visual images. He showed them being applied to code size, but they could be used for anything that can indicate progress (i.e., size, effort, cost, defects, and milestone attainment).
You can also use data about progress-to-date to predict a "trajectory" of where a project is going. This can be a valuable technique for determining if the current plan is still reasonable. These techniques can be done manually or with creative use of a spreadsheet package.
It is critical that you look at as many indicators as possible to insure that you are getting a complete picture. He illustrated this by asking if a pilot would be happy flying with only a clock and a fuel gauge to guide them. He believes that this is analogous to a software project being tracked by only watching cost and milestones.
Michael presented a case study from the Netherlands Telecom; they are using the red/yellow/green traffic lights. There is a "Quality Management & Control" function reporting directly to the Chief Executive. This function is responsible for collecting the data and showing management a monthly status of software projects. The reviews are to focus management actions and not to punish the guilty. Project data is submitted every month, and a red/yellow/green assessment is determined. If a project doesn't have data, then they are tagged as being "Red". All "Red" projects must have action plans to get back on track. He pointed out that this level of tracking isn't appropriate for every project. Netherlands Telecom applies it on about 80% of their projects.
He presented some things to avoid and some things to strive for;
Demotivational Factors (Things to avoid):
Things to strive for:
In different places of this talk, he addressed the problems associated with introducing change into an organization. In his opinion, attention to real process improvement is often jeopardized when an organization is largely in chaos because of projects in crisis. Process improvement can easily take a back seat to the fire drills and deadlines. This is a classic Catch- 22; the org is in chaos -- causing them to be unable to get the needed focus or cycles onto the improvements that could get them out of the chaos.
In closing: If organizations can become successful at estimating and tracking, then they can find the rewards of projects being under control from the start (at time of commitment). They can then effectively manage projects mid-stream -- doing real risk management. Senior management can use techniques like red/yellow/green to identify risks to their projects early, and channel energy to help where needed. They can have the numbers to manage their software projects, much like a Wall St. trader or even a layperson investor using charts and visual indicators to manage a financial portfolio.
Questions and Answers (paraphrased):
Q: How does all of this apply if you are moving toward system integration (away from pure code production)?
A: The "size" shrinks so the other factors also go down. The benefits are then seen in geometric fashion. The tricky part is that building code in isolation can be easier to test. Testing large integrated systems can be very difficult. For example, you could have 10,000 lines of new code but have to regression test a "system" that represents 1,000,000 lines of code.
Q: Can you normalize the Productivity Index so that different types of projects can be compared?
A: This isn't a good idea. Normalizing tends to wash over differences that shouldn't be ignored.
Q: What types of companies are looking for this kind of help?
A: Right now, QSM is seeing many requests coming from the Telecom community. In addition, the Wall Street community has recently started becoming involved.
[Note: Just about all of the processes or techniques that he presented can be automated by the QSM tools SLIM and SLIM Control; however, I don't recall Michael ever mentioning that fact. This is good and bad; on the one hand it is nice that he didn't overtly use the SPIN for marketing purposes; on the other hand, it made some of his talk seem more complex and theoretical than it may have needed to be.]
Last updated: April 11, 1997
David.Constant@ProcessInc.com