There's a never-ending opportunity to learn how to estimate in the presence of uncertainty. Here's some resources for informing that learning process.
Don't need estimates? - the project is de minimis
And the Principles, Practices, and Processes to Increase Probability of Success
There's a never-ending opportunity to learn how to estimate in the presence of uncertainty. Here's some resources for informing that learning process.
Don't need estimates? - the project is de minimis
The presentation "Quantifying the Impact of Agile Practices," Larry MacCherone at the RallyOn 2013 Conference, presents some results on estimating impacts. The chart below shows 4 estimating types, including No Estimates, the sample sizes for each type and the components that make up the estimating types.
The Software Development Performance Index (SDPI) scale on the left ranges - by eyeball measurement - from 46 to 55.
The Higher the number the better the performance of the process. The presentation speaks to the components of the index further.
But first another piece of information ...
Teams doing Full Scrum have 250% better Quality than teams doing No Estimating
But are these differences meaningful statistically?
Let's start with several reading assignments, before answering
Let's start with the numbers from the chart
Since the raw underlying data is not available, we can't do any p-Factor assessment from the population samples, but there is a simple question that can be asked.
Are there any statistical differences between the 4 SDPI's? If you look below at the quick and dirty assessment of the only data available, it looks like all 4 approaches are within a single digit variances of each other. Not that useful actually.
So the critical question still remains
How can you make a decision in the presence of uncertainty without estimating the impact of that decision?
Thanks to Sean Craig's Live Sketch Note for capturing concepts directly from Woody Zuill's talk. This is a good starting point for answering the mail on the notion that decisions can be made in the presence of uncertainty without estimating the impact of those decisions.
Let's look at these Notes and plausible responses to the conjectures. But first, let's look at the motivation for a blog post like this one from Bono's admonition ...
There’s nothing more dangerous than an idea – or a person – that’s almost right. – Bono
I attended our Agile Meetup last night, where the speaker walked through the three current Agile at Scale development methods, all based on Scrum - SAFe, LeSS, and Nexus. He had an interesting statement.
Each of the authors have a different, sometimes slight, sometimes major, approach to producing software using their methods. But each author (or currator of the mehod) has extensive experience testing that method in the field against a broad range of domain and environments. 1,000 to 10's of 1,000's. There is no sure basis of credibilityy for the No Estimates conjecture that decision can be made in the presence of uncertainty without first estimating the impact of the decision.
When we hear a suggestion about a process that inverts the logic of normal business activities based on a governance framework - say Microeconomics of Software Development, we need to ask who benefits? How would that suggestion be tangibly beneficial to the recipient that is now inverted?
Estimates are for the business, why would the business no longer want an estimate of cost, schedule, or technical performance for the provided capabilities they are paying to have developed?
Turns out of course, it's not the business that has lost the need to estimate, it's those spending the money provided by the business that are suggesting estimates are not needed. This conjecture started long ago when an original post, from his chief coder position at the lawn sprinkler company. (I have those sprinkler heads on my lawn, and they're pretty nice). But his conjecture starts with estimates are a waste, not saying for whom they are a waste for.
Anyone in the finance department, paying his salary, paying the developers has a fiduciary need to know what something will cost when it is done. And since ALL and I Mean ALL software development operates in the presence of uncertainty, the only when to have knowledge of that cost, schedule, and technical content is to make estimates. The further conjecture that estimates are hard, can't be made, and are always wrong, simply affirms that those making those claims don't know how to estimate. That is clear, but that has nothing to do with the principles of estimating in the presence of uncertainty.
I'm no longer a developer, having left development in the early 80's after a career of writing FORTRAN 77 code for missile defense systems that is still in place today. But that has nothing to do with the principles, practices, and processes of writing code using current languages and tools. If you find it hard, but go find someone who doesn't find it hard.
So let's look at what's being said here about managing software development in the presence of uncertainty using other people's money
Transformation comes from pursuing profound questions than seeking practical answers
It's not your money
Here's an extract from a Deloitte handbook about business decision making
Effective governance and decision making establishes a framework for transformation and can improve the odds of solidifying change and realizing the benefits of transformation.
Control and Certainty about What?
If you're not managing risk, if you're not making estimates about these risk, you're not managing the project as an adult
Control our Urge to Control Things
If the types of decisions we make requires estimates, then are there better types of decisions?
If you accept that project have uncertainty, then making decisions in the presence of this uncertainty requires you estimate the impact of that decisions. #NoEstimates has yet to state how, in the presence of uncertainty, that a decision can be made with NO Estimates. Only that they are exploring for ways to to this. Been exploring for going on 4 years now, seems to have found nothing. Which would support the notion that the microeconomics of decision making is still in place and any decisions in the presence of uncertainty requires making an estimate.
The Fear of Losing Control is a Big Barrier to Change. Control is Illusion. Fear is Real
Just another platitude without any basis in principles of closed loop business and technical management
The cycle of non-improvement
There is no evidence that No estimates improves anything for those paying for the software development. This is a consistent issue with the conjecture that No Estimates fixes problems, without idnetifying the root cause of the problem, testing the hypothesis that No Estimates fixies the problem, and that this fix can be syndicated beyound the personal ancedote of the orginal conjetcure
What is an Estimate?
This question has a simple answer. It is not a Guess. It is not a promise. This question and the answers used by No Estimates advocates is used to manipulate the situation. Bad Managers use estimates as promises. Those with no experience, skill, or knowledge of estimating consider estimates as Guess's. Both are wrong, both are behaving as if they are in a Dilbert carton, following the script of a dysfunctional set of characters.
There endless numbers of books, papers, tools, classes on how to estimate software projects. These have all been provided in this blog before and can be found again. But here's a starting place - Estimating Software Intensive Systems. Read this and you'll have the start of all you need to break the cycle of uniformed, unsubstantiated states about what is an estimate, how are they made, and how can they be used to make decisions in the presence of uncertainty
Most of the things that matter can't be measured
Let's establish a baseline here. We're writing software for money, usually other people's money. In this context - a business context, Lord Kelvin has good advice for us. Writing software for money is not about the self-actualization of the individuals on the team. That may an outcome, but those paying ae not paying to improve the staff's psychological wellbeing.
When you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind. …It may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the stage of science. - Lord Kelvin (1824—1907)
So what is the quest of the advocates of No Estimates? It's not actually clear.
I can only conjecture that the purpose of Not Estimating is to return control of the work processes to those doing the work by removing the decision making processes from those owning the money. It's the anarchistic approach to software development.
I, the one providing the labor in exchange for payment, am the one who will be telling you, the paying, when I'll be done, how much it will cost when I'm done, and what you'll be getting for that time and money. I am the one in charge of the effort to spend your money.
Much of the discussion around project management processes, especially around agile, and most especially around the misconceptions of Estimating as espoused by the #NoEstimates advocates, starts with the misuse of reductive reasons based on single factor analysis.
Here's how it goes.
When we see these two concepts used together we get things like as the cartoon of the Reductionist view of a single concept - If you did it this way, you'd be from 3X to 10X faster.
So here's the problem and the solution. Complex systems are part of the solution to all complex problems. Anyone claim complex problems can be solved with simple systems, needs to have a testable working system, in that complex problem space. I work in a complex problem space - literally space flight, aircraft flight, the ground systems that enable those systems to Fly. As well as biopharma, electric utilities (nuclear and conventional fired), complex enterprise IT systems (dozens to many dozens of interacting systems).
When you hear a simple and many time simple-minded solution to a complex problem - Applying No Estimates will remove the dysfunction on software projects (this is the ontological inverse of the statement estimates are the smell of dysfunction). We can be reminded by H L Menken's quote:
For every complex problem there is an answer that is clear, simple, and wrong.
I came across a couple of interesting tweets over the weekend
(Scrum is) more defined rather than empirical process. Agile in name only. scrum, not agile. not as it was prior to scrum. lost art.
Scrum is prescriptive and defined rather than empirical. it seeks to be guidance for work it's too far removed from to understand.
This took me back since Scrum is derived from an empirical closed loop control system developed in the USAF by Col. John Boyd. There's a recent paper "What Lessons Can the Agile Community Learn from A Maverick Fighter Pilot?" by Steve Adolph. I was puzzled by the tweet, asked why that was the opinion, but didn't get a response. As a start, here's a resource for Col. Boyds papers.
No need to explain why OODA is the basis of Scrum, here's a much better post - OODA: The Mindset of Scrum.
The OODA concept is very simple:
Boyd's presentation "A Discourse on Winning and Losing Introducing core ideas & themes Of Boyd’s ‘Theory of intellectual evolution and growth,"was presented at “Patterns of Conflict” at the U.S. Marine Amphibious Warfare School in June 1980. Here's some more Boyd from the Boyd Library. A good book on Boyd is Boyd: The Fighter Pilot Who Changed the Art of War
In our Software Intensive System of Systems domain, this is the definition of Agile we use from the now-Secretary of Defense Dr. Carter
So when we hear anything about the current state of Agile or even that Scrum is NOT empirical and needs to be replaced, ask what new process is being suggested that will meet Dr. Carter's definition?
…the OODA loop sketch and related insights represent an evolving, open-ended, far from equilibrium process of self-organization, emergence and natural selection. This is a good starting point for not only Scrum and other agile methods, that can be used to test the validity of any method suggested to replace or augment the current agile software development processes.
Assessing the outcomes of the feedforward and feedback loops is the basis of all Closed Loop control system. (See link below on Closed Loop Control). Making decisions in the presence of this emerging uncertainty requires making estimates of the impacts or outcomes of any decision. There's simply no way out of this principle when making decisions in an emerging domain with uncertainty (reducible or irreducible). Anyone suggesting otherwise either hasn't the knowledge of closed loop control systems or is try to sell you something that violates the principles of closed loop control, microeconomics of decision making, or managerial finance. Don't buy it. Learn to manage in the presence of uncertainty using principles not persoanl ancedotes.
All project work is uncertain. This uncertainty comes in two forms - Aleatory and Epistemic. Aleatory Uncertainty is irreducible, Epistemic uncertainty is reducible. More can be found on this in Both Aleatory and Epistemic Uncertainty Create Risk.
But now what to do about it. Here's a collection of papers that has served me well in defining the processes needed to make decisions in the presence of uncertainty when managing projects using other people's money:
So when you hear about making decisions in the presence of uncertainty without estimating the outcomes of those decisions, you'll have a basis of knowledge to realize that is completely bogus, a violation of the principles of microeconomics, and a violation of the principles of managerial finance.
A nice conversation on twitter about estimates on software brought up the topic of estimates as commitments. The #NoEstimates advocates see estimates as making commitments. Yes, commitments are made when we estimate. But that commitment is not a promise. A Promise is a guarantee, with 100% confidence it will be net. Our wedding vows are a promise to each other. A commitment must have a probabilistic confidence. Just for the reference,
A commitment must have a probabilistic confidence. Just for the reference, a commitment is the state or quality of being dedicated to a cause, activity, etc.
"the company's commitment to quality or an engagement or obligation that restricts freedom of action. Take for example Boulder's commitment for 100% recycling of consumer products inside the city limits. Is that commitment are guarantee every single consumer waste product collected at our curb will be recycled? That is a tall order and not likely to ever be the case.
I have an 80% confidence (an estimate) I can deliver what you need on or before September 15 (an estimate), at or below $15,000.oo (an estimate) with a 15% error band (an estimate).
The misuse of that commitment is a real problem in all domains. But that's got NOTHING to do with the need for the estimate and EVERYTHING to do with bad management. No Estimating isn't going to make the need for estimates go away or fix bad management, no matter how many Dilbert cartoons are used to illustrate that.
In any non-trivial domain, there are larger business processes in play that is looking for value in exchange for money. When business people think about that exchange, they think about some agreement between those paying for the value and those providing the value . This agreement can be formal or it can be informal. But that agreement is always in place.
In our domain of software intensive system of systems, we have formal agreements at the top of the engagement and less formal agreements lower down based on agile software development processes. Here's how it is done where we work.
So when you hear Estimates can't be done, estimates are a waste, estimates are the smell of dysfunction, estimates are evil, this may actually be true for those working on de minimis projects. Projects where those paying for the work have NO concern about how much it will cost to produce the value, NO concern when that value will be available for use, NO concern if the capabilities that produce that value actually work and actually do produce the value for the needed cost on the needed date.
Here's an article, recently referenced by a #NoEstimates twitter post. The headline is deceiving, the article DOES NOT suggest we don't need deadlines, but that deadlines without credible assessment of their credibility are the source of many problems on large programs...
The Core Problem with Project Success
There are many core Root Causes of program problems. Here's 4 from research at PARCA
Before diving into the details of these, let me address another issue that has come up around project success and estimates. There is a common chart used to show poor performance of projects that compares Ideal project performance with the Actual project performance. Here's the notional replica of that chart.
This chart shows several things
So let's look at the estimating process and the actual project performance
But here's three problems. First, there is no cause stated for that variance. Second, the ideal line can never be ideal. The straight line is for the estimate of the cost (and schedule) and that estimate is probabilistic. So the line HAS to have a probability distribution around it. The confidence interval on the range of the estimate. The resulting actual cost or schedule may well be within acceptable range of the estimate. Third is are the estimates being updated, when work is performed or new work is discovered and are those updates the result of changing scope? You can't state we did make our estimate if the scope is changing. This is core Performance Measurement Baseline struff we use every week where we work.
As well since ideal line has no probabilistic attributes in the original paper(s), no shown above - Here's how we think about cost, schedule, and technical performance modeling in the presence of the probabilistic and statistical processes of all project work. †
So let's be clear. NO point estimates can be credible. The Ideal line is a point estimate. It's bogus on day and continues to mislead as more data is captured from projects claimed to not match the original estimate. Without the underlying uncertanties (aleatory and epistemic) in the estimating model the ideal estimates are worthless. So when the actual numbers come in and don't match the ideal estimate there is NO way to know why.
Was the estimate wrong (and all point estimates are wrong) or was one or all of Mr. Bliss's root causes the cause of the actual variance
So another issue with the Ideal Line is there is no confidence intervals around the line. What if the actual cost came inside the acceptable range of the ideal cost? Then would the project be considered on cost and on schedule? Add to that to coupling between cost, schedule, and the technical performance as shown above.
The use of the Ideal is Notional. That's fine if your project is Notional.
What's the reason a project or a collection of projects don't match the baselined estimate. That estimate MUST have an accuracy and precision number before being useful to anyone.
- Essentially that straight line is likely an unquantified point estimate. And ALL point estimates are WRONG, BOGUS, WORTHLESS. (Yes I am shouting on the internet).
- Don't ever make decisions in the presence of uncertanty with point estimates.
- Don't ever do analysis of cost and schedule variances without first understanding the accuracy and precision of the original estimate.
- Don't ever make suggestions to make changes to the processes without first finding the root cause of why the actual performance has a variance with the planned performance.
So what's the summary so far:
So in the end, any estimate we make in the beginning of the project, MUST be updated at the project proceeds. With this past performance data we can make improved estimates of the future performance as shown below. By the way, when the #NoEstimates advocates suggest using past data (empirical data) and don't apply the statistical assessment of that data to produce a confidence interval for the future estimate (a forecast is an estimate of a future outcome) they have only done half the work needed to inform those paying what is the likelihood of the future cost, schedule, or technical performance.
So Now To The Corrective Actions of The Causes of Project Variance
If we take the 4 root causes in the first chart - courtesy of Mr. Gary Bliss, Director Performance Assessment and Root Cause Analysis (PARCA), let's see what the first approach is to fix these
Unrealistic performance expectations missing Measures of Effectiveness and Measures of Performance
Unrealistic Cost and Schedule estimates based on inadequate risk adjusted growth models
Inadequate assessment of risk and unmitigated exposure to these risks without proper handling plans
Unanticipated Technical issues with no alternative plans and solutions to maintain effectiveness
When we hear we can't estimate, planning is hard or maybe not even needed, we can't forecast the future, let's ask some serious questions.
The answers should be YES to these Five Immutable Principles of Project Success
If not, you're late, over budget, and have a low probability of success on Day One.
†NRO Cost Group Risk Process, Aerospace Corporation, 2003
There is a popular Agile and No Estimates phrase...
It is by doing the work we discover the work we must do.
This of course ignores the notion of engineering or designing a solution to the needed Capabilities of the customer, BEFORE starting coding. It is certainly the case that some aspects of the software solution can only be confirmed when the working software is available for use. But like all good platitudes in the agile community, there is no domain or context as to where this phrase is applicable. Where can coding start before there is some sort of framework for how the needed capabilities will be delivered?
This sounds not only näive, but sounds like we're wandering around looking for a solution without any definition of what the problem is. When that is the approach, when the solution appears it may not be recognized as the solution. Agile is certainly the basis of dealing with emerging requirements. But all good agile processes have some sense of what the customer is looking for.
This understanding of what capabilities the customer needs starts with a Product Roadmap. The Product Roadmap is a plan that matches short-term and long-term business goals with specific technology solutions to help meet those goals.
A Plan is a Strategy for success. All strategies have a hypothesis. A Hypothesis need to be tested. This is what working software does. It tests the hypothesis of the Strategy described in the Product Roadmap.
So if you have to do the work to discover what work must be done, you've got an Open Loop control system. To close the loop this emergent work needs to have a target to steer toward. With this target the working software can be compared to the desired working software. The variance between the two used to take corrective actions to steer toward the desired goals.
And of course, since the steering target (goal) and the path to this goal are both random variables - estimates will be needed to close the loop of the control processes used to reach the desired outcomes that meet the Capabilities requested by the customer.
Where You Stand Depends On Where You Sit
This notion of the basis of all good discussions. It is also the basis of discussions that get us in trouble. For example, I sit in a FAR 34.2/DFARS 234.2 Federal Procurement paradigm and a similar paradigm for Enterprise IT based in ISO 12207, ITIL V3.1, CMMI, or similar governance processes.
Both these domains are guided by a Governance framework for spending other people's money, planning for that spend, performing the work with that money, reporting the progress to plan for that spend to produce the needed value, and taking corrective actions when the outcomes don't match the plan to increase the probability that the needed value (Capabilities) will be delivered for the planned cost to keep the Return On Investment on track.
This paradigm is independent of the software development method - traditional or agile.
If you work where the customer has a low need to know the cost, schedule, or what will be produced for the money, then you likely sit somewhere else.
Sitting in our seats at last night's Rockie v. Phillies game and dawned on me the analogy between Moneyball strategy and good management of software development. In Moneyball, Billy Beane was faced with a limited budget for players. He hired a statistics guy and they figured out the getting on base was just as important as hitting a homerun and a hell of alot cheaper.
Last night there were a few home runs. But most of the action were singles and a few doubles. If you do the simple minded math, when the rotation all get singles with batting averages of 0.303, then there'll be runs scored every inning. 4 singles equals a home run. Getting on base is the key to winning games.
Getting incrementally more software out the door - assuming it's the right software needed by the customer and that customer can put the software to work - then progress toward the win is being made.
So the Strawman of Waterfall and Big Bang is just that a Strawman. The Straw Man of No Projects is also nonsense, along with No estimates. The manager of the Rockies has a plan in the presence of uncertanty and emerging situations. That's why he's called the MANAGER because he manages in the presence of uncertainty. And in doing that job he makes estimates on the probability of success of the emerging play options.
We can learn a lot from Baseball about managing projects. First get on base. You can't score unless you're on base. First, Second, Third, then Home. You can't count on hitting Home Runs to win the game. It doesn't work that way. Offense of good, but so is defense. Managing the risks is defense. Defense in baseball is more than just putting players on the field. It's how those players react when the ball is hit. Go for the out at first? Try for a double play? Hold the ball after a single bounce in the outfield?
While baseball is not a contact sport, it still requires teamwork. I played competitive Volleyball in College - the ultimate Team Sport, since you're only as strong as the weakest player. Much like the software development team. But all teams have a strategy, a game play that changes as the game emerges and most of all - as Peter Kretzman has lambasted some NE advocates who have not likely ever played baseball - all the players are making estimates all the time in order to catch the ball, keep control of the ball and the emerging situation of the game.
It's popular in the #NoEstimates community to claim Forecasting is not Estimating. Using English Dictionaries, they build a case using logical like this. It's been repeating nearly continuously since the start of the discussion about how to make decisions in the presence of uncertanty without estimating.
Salviati: If you have a number of uniform sized tasks & know velocity you can predict ttc (Time to Complete) Simplicio responds: Indeed...gives us much stronger forecasting ability than estimation
Let's use the Oxford Dictionary of Mathematics, not the High School English Dictionary.
It ain't so much the things we don't know that gets us in trouble. It's the things we know that ain't so - Artemus Ward
Estimating is about the past, present, and future outcomes of a process, a model, or some external observation. We can estimate the number leaves that have fallen and need to be raked come fall, to size how many bags have to be bought at Home Depot (past). We need to estimate the number of people on the Pearl Street Mall right now to assess how long the walk will be to Starbucks from our parking spot (present). We can estimate the number of minutes we'll need to reach our favorite parking spot at Denver International Airport from the office parking lot (future).
Forecasting is Estimating about the future. The weather forecast for tomorrow is 70 degrees and a 30% chance of rain in northern Boulder County (where we live). With this forecast, we can estimate what time we need to tee off to have a high probability of getting all 18 holes of golf on our course in before the rain starts.
Forecasting is estimating about the future
There's another rash of Twitter posters supporting the conjecture that decisions can be made about how to spend other people's money without estimating the impact and outcome of that decision. This is a core premise of #NoEstimates from the Original Poster
Here's a starting point to learn that is simply not true...
There are 100's more books, papers, conference proceedings on the topic of decision making in the presence of uncertanty.
It comes down to a simple concept
All project work is uncertain. Reducible uncertainties (epistemic) and irreducible uncertainties (aleatory) are present on all projects. When money is being provided to develop software, those providing the money have a reasonable expectation to know something about when you will be providing the value in exchange for that money, how much money will be required to provide that value, and what capabilities will be provided in exchange for that money. The field of study where these questions and answers live is microeconomics of decision making. Software development decision making is a well studied subset of that field.
When it is conjectured decisions can be made in the presence of uncertanty without estimates - like the Original Poster did, and there is no supporting theory, principle, practices, or evidence of how that can be done - run away - it's a bogus claim.
When riding a single track like this one behind our neighborhood, I came to an understand of the tradeoffs between stability and control. In our conference sessions, we speak about how the Wright Brothers learned this concept as well.
|Here's another trail being ridden by our son at Nationals in 2013. who's massively better than I will ever be. But same tradeoff between stability and control is needed to be ranked 33rd Cross Country and 23rd Short Track at DII in the nation, with team taking 2nd overall.|
In both cases, nationally ranked and rank amaetur, control of bike is the key. Stability is a relative term, depending on speed, terrain, skill, risk taking tolerance, experience, technical equipment, and other intangible factors.
We speak at conference where program planning and controls is the topic. Starting 2 years ago, our foundation for speaking about managing in the presence of uncertanty is the Wright Brothers.
Most earlier experimenters in powered flight focused only on one or two of the primary problems - (a) A set of lifting surfaces, or wings, (b) A method of balancing and controlling the aircraft, and (c) A means of propulsion - and did not consider the final design from the outset. The Wrights recognized that each of these areas had to be successfully addressed to build a working airplane. They believed that the aerodynamic and propulsion problems would be comparatively easier to solve, so they first concentrated on how to maintain balance and control.
Many believed that air currents were too swift and unpredictable for human reflexes. Therefore, an aircraft had to be inherently stable for the pilot to be able to maintain control. Because of the Wrights’ extensive experience with the bicycle—a highly unstable but controllable machine (just like the mountain bike)—they saw no reason why an airplane could not be unstable yet controllable as well.
The notion of Control is missed used in software development projects. Misused and misunderstood. Control inside the upper and lower control bounds is a critical success factor. What are those upper and lower control bounds? Good question. They have to be estimated in the start. They become cleared as the project progresses. They have to be adjusted as new information emerges.
In projects, the question for Stability is a false quest. All project work is uncertain. Stability is short lived. New inputs arrive every day. Just like new inputs arrive while riding down the flat trail for cruising around the open space in the neighborhood or more so on the bumpy high speed trail of collegiate racing.
Even if the trail is defined in from of you, the small disruptions and many times big distributions input your control of the bike. The key is Control over Stability. Staying in control will get you across the finish line, down the trail, and home again to ride another day.
Skills of Maintaining Control on the Mountain Bike and the Project
Starting from our house there is a nice loop Left Hand Trail, that is easy, has a few climbs, and a few descents, uncrowded, and provides a nice view of the small hills before the real mountains start . Combine that with the Eagle Trail, Sage Trail, and North Rim Trail and it's a nice 7 mile loop
There's lots of misinformation going around again in the #NoEstimates community
Return on Investment is ROI = (Value - Cost)/Cost. Both Value and Cost are needed to make any decision when spending other people's money
Here's the starting point to clarify and debunk those concepts
Here's some more details, once it's been confirmed you have domain experience
In the end it's simple. Anyone speaking about the difficulties or the waste of estimating is likely on the spend side of the software development equation. Those on the pay side know (or should know) they needed estimates. They can get the estimates from those most qualified to provide them - the developes. Or they can just make them up. Probably better to have a bottom up estimate.
Here's some thoughts on the economics of software development using other people's money, after 3 weeks of working a proposal for a major software intensive system of systems using Agile.
So to come full circle
Why Do We Need Estimates?
It can't be any clearer than this - if you don't know what something costs or is going to cost in the future, you can't make a business decision about it's value
When you read estimates are a waste, estimates are non-transferable, estimates are wrong, estimates are temporary think again. Go ask those paying if they need estimates to make decisions for the business. If they don't, then continue to spend your customers money. If they do, they may consider looking for someone who knows the difference between willful ignorance of how to estimate software development and someone who can provide that information needed to stay in business. On our proposal team, this would get you a 2nd place prize in the estimating capabilities contest - which is a one way trip home with weekends off.
We all know estimates are hard. But there are lots of hard things in the development of enterprise software. We wouldn't be whining about how hard it is to construct a good First Normal Form database schema, or bullet proof our cyber security front end from attack by the Chinese would we.
So why is estimating a topic that seems to be the whipping boy for software developers these days?
My first inclination is that estimating is not taught very well in the software arts. In engineering schools it is. Estimating is part of all engineering disciplines. One undergraduate and one graduate degree is in physics. Estimating is at the very heart of that discipline. A second graduate degree is in Systems Management - which is a combination of Systems Engineering and Managerial Finance - how to manage the technical processes of engineering programs with the principles of managerial finance, contract law, and probabilistic decision making.
This book comes with a spreadsheet for making the needed estimates to increase the probability of project success. It opens with an important quote that should be a poster on the wall of any shop spending other people's money
For which of you, intending to build a tower, sitteth not down first, and counteth the cost, whether he have sufficient to finish it? Lest haply, after he hath laid the foundation, and is not able to finish it, all the behold it begin to mock him, saying, This man began to build, and was not able to finish - Luke 14:28-30
A primary learning process is research. This process is part of all science, engineering, management processes. Here's a starting point for learning how to estimate in the presence of uncertainty on agile projects.
This is the start of a literature search. Anyone making suggestions about making decisions in the presence uncertainty on agile project can start here for establishing the basis of any arguments of how and why that suggestion (conjecture possibly) is based on some set of principles. Not just anecdotal opinion.