There are several studies on how software development can be improved by applying various process. Some studies have very large data sets, with analysis of each process and the differences between the processes. Other studies have small data sets, no error bands on the data, and no real statistical analysis of this raw data.
Here's a checklist for the steps needed to produce a credible answer to - Is there any difference between one process or another process for improving the outcomes of a process
- Was the dependent variable precisely, completely, and unambiguously defined?
- Were there examples and examples of the process outcome provided, if doing so will enhance the clarity of the study?
- Were the most relevant, measurable dimensions of the target process specified?
- Were the important concomitant processes also measured?
- Were the observation and recording procedures appropriate for the process?
- Did the measurement provide valid (meaningful) data for the problem or research question?
- Was the measurement scale broad and sensitive enough to capture the significant differences between the processes?
- What procedures were used to assess and assure the accuracy of the measurement?
- Were any contingencies in place in the study that may have influenced the observations?
- Was there any expectation or indication that the dependent variable may have been reactive to the measurement system?
- Were appropriate accuracy and/or reliability assessments of the data reported?
This is a small sample of the questions that needed to be answered when there is a data set used to conjecture there is an improvement impact to some process that involved humans applying tools, processes and the resulting outcomes.
Without this foudmation, making recommendations or suggestions that one way is better than another is suspect at best.
