The recent post about the 5 Ways To Rethink Software Projects has elicited a response from the author stating I was being unfair to his thesis that in the end - Number 5 - we didn't need to make estimates. Of course without any domain or business processes governing the expenditure of other peoples money, it's hard to tell where the proposed ideas would fit it.
Going back to basics with the clip below, I'm reminded we seem to have lost all visibility to why estimating is needed in the first place. The clip below is from a 1970 Naval Post Graduate School (NPS) thesis about the troubles with estimating. Many of the problems are still in place today. So you may say we haven't move beyond the orginal problem. But that would be too simple. The papers below, including this 1970 thesis shows how far we have come.
I'd conjecture to move beyond twitter exchanges and interviews of personal anecdotes we need to test the notion of how to improve estimates in the market place. Anecdotes are fine, if derived from actual hands-on experience in a domain or similar domain. The bottom's up discussion many times ignores governance and simple business processes and is stuck on the you're not listening to my great idea that XP was in early on. As Carl Sagan suggests
"Extraordinary claims require extraordinary evidence." Carl Sagan
The systems referenced below are not in the domain of the CIO magazine article. But the principles for forecasting the cost of a project before - or near the start - work is still an important part of Project Governance. The notion of project governance seems to have been skipped over when we look at the problems of estimating. From the management of the ACA web site, to large ERP systems, to major weapons, to major science projects - managing the project with the paradigm of governance is actually harder than it looks. It is hard because the value at risk is more important than the interests of the individual. The subordination of self interest some times is more difficult than seeing the bigger picture of managing other peoples money.
So we can choose to explore how to improve our estimating capabilities or we can seek out ways to avoid estimating all the together. But in the end, those with the money likely want us to learn to be better estimators. We should ask them, before assuming they want us to stop estimating the cost of work - #5 in the list of ways.
That's the message here. If you have a customer who will pr ovide you with funding in the absence of knowing - in some way, with some confidence - how much it will cost in the end, I'd suggest you proceed and hope for the best.
There is a common idea found in some software places, especially one that start with the notion that we don't want to do estimating, or estimating is not needed when spending other peoples money, that estimating is hard, is the same a guessing, and produces results not valuable or not needed by the customers.
Let's start with a paradigm. If you choose to not estimate because you don't want to learn, well not much to do about that. But let's read what Barry has to say.
There is no good way to perform a software cost-benefit analysis, break-even analysis, or make-or-buy analysis without some reasonably accurate method of estimating software costs and their sensitivity to various product, project, and environmental factors.
-Barry Boehm
Professional organizations for cost estimating
- International Cost Estimating and Analysis Association
- Association for the Advancement of Cost Engineering
- NASA Cost Analysis Division and an example of their work in producivity studies. Tom Coonce is my current supervisor on contract to IDA.
- DOE Cost Estimating Guide
- Cost Assesment and Program Evaluation
Ignoring for the moment the problems with the logic of ignoring the needs of those providing us money, here's some places to start in the domain where estimating is the basis of good project management
- Software Cost Estimating site as University of Southern California (one of my alma mater's). Start with this site and then visit the Center for Systems and Software.
- GAO Cost Estimating and Assessment Guide
- GAO Schedule Assessment Guide
- NDIA Planning and Scheduling Excellence Guide (PASEG)
- 2008 NASA Cost Estimating Handbook. There is a 2013 coming soon?
- Application of Selected Software Cost Estimating Models to a Tactical Communications Switching System: Tentative Analysis of Model Applicability to an Ongoing Development Program
- Pragmatic Quality Metrics for Evolutionary Software Development Models, Walker Royce
- Review and Analysis of Statistical Cost Estimating Relationships, Marshall Carter, Naval Post Graduate School.
- JPL's Cost estimating processes for software intensive systems, with emerging requirements, iterative development, and fixed delivery dates and fixed budgets. You can fly to Mars in a 3 week window every 3 years.
- JPL's Cost Estimating Handbook
- Software Cost Estimating, Software Engineering 7th Edition,
- Software Cost Estimating, DAU BCF 106 course
- Software cost estimating problems at JPL
- Validation of Spacecraft Cpst estimation Models, since domain is king, speaking about estimating in the absence of a domain is pretty much useless, here's a domain specific study of the effecrtiveness of cost estimating.
- Software Cost Estimating resouce page
- Software Development Cost Estimating Handbook, Naval Center for Cost Analysis and Air Force Cost Analysis Center
- RIT's Cost Estimating 1 and Cost Estimating 2 class
- Software Cost Estimating and Sizing Methods, RAND
- Emperical Validation of Software Cost Estimating Models
- Cost Estimation Methods for Software Engineering
- UKHEC Report on Software Estimation
- Improved Size and Effort Estimates for Software Maintenance
- Cognitive Limits of Software Cost Estimation
- Software Cost Estimation Review
- Technical Reports Site for the Center Systems and Software Engineering
- Accurate Estimates without Local Data
-
Software Intensive Systems Data Quality and Estimation
Research in Support of Future Defense Cost Analysis, A013 - Annual and Final Scientific Technical Report SERC 2012-TR-032 - Software Cost Estimation, Leung and Fan
- Software Cost Estimation Methods: A Review
- Software Cost Estimating and Control
- Software Cost Estimating Handbook, Systems and Software Conference
- Software Development Cost Estimating Approaches, A Survey
- Realistic Software Cost Estimates for Fractionated Space Systems
- Reducing Estimation with Continuous Assessment
- A Model for Estimating Agile Project Schedule Acceleration
- An another whose title that might be useful for the discussion of probabilistic estimating
- Here’s the rest of the cost estimating literature at JPL. Each NASA center has similar content.
- And another client, again software intensive systems.
- Software Cost Estimating Relationships
- Applied Software Project Management, chapter on Estimating
- Estimating Software Costs, a brochure, but with good discussion
- Jeff Sutherland's Story Points and Why They Are Better. There is conjecture that Stories are better, but only and this is a huge only when they are statistically the same size.
There are many more approaches. Google will get you them all. But in the end you actually have to want to learn how to do software estimating. If you start with the notion that estimates are either not needed, serve no purpose, or you simply have heard this from a source that has no connection to your domain, nor wants to - then you'll probably want to ask a few more questions:
- When spending other peoples money, do they have an expectation that you can tell them something about the cost and schedule for the project they are funding?
- When you hear there are alternatives to estimating, does the speaker have an alternative ready to go? Or does the speak leave it up to you to figure out what those alternatives are?
- Is your project the size that estimating is simple and straight forward - start work, we'll be done in a few weeks, maybe six. We have confidence in what we're doing, so an estimate is redundant.
So when you hear someone suggest we need question everything then question the questioner if he has any experience at all spending other peoples money on a project where the value at risk is substantial to the point if the project fails he'll ge fired.
No? not working in that domain - manifestly important and nearly impossible - according to Edwin Land, then that advice about estimating may be a bit too narrowly focused, limited in domain for your need. You get to decide.
But if you do maybe these will help as well
- How to Estimate Almost Anything if you Want To
- Reference Class Forecasting for Software Projects
- Software Estimating Taxonomy
- And a summary of other software cost estimating resources
