Observational studies are inherently challenging because of the potential for self-selection bias and confounding factors. Always be wary of studies the use the data to discover the theory. - Standardd Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Stastistics, Gary Smith.
When we hear or read about any conjectured statement without a supporting theory and principles. When you read about some study that supports your beliefs, there is a natural inclination to conclude your beliefs are confirmed. It is much better to closely look and think about the confounding factors.
We see a pattern and make up a theory that fits that pattern. This is an example of the Law of Small Numbers. Kahneman and Tversky observed the belief in the law of small numbers. The mistaken law causes two related errors.
The first is the gambler's fallacy. The second error occurs when we don't know how many samples are needed to draw a conclusion.
These classic errors can be found in the #Noestimates conjecture. These advocates provide an example of a project that is executed without estimates and make the claim that Not Estimating is a proper process because they have observed projects with did not estimate and somehow showed up on time, on budget, with needed capabilities.
But these are anecdotal observation, with no statistical basis for the claim.
Anecdotes are not evidence, they're just small samples, biaed by the person making the observations.
