What are the most common types of threats to validity in empirical studies in software engineering you have witnessed? What are the most challenging to overcome?
Posted by elbaum | Posted on 07-05-2011
0
I think that one of the most critical and also one of the hardest threats to address in practice is the internal threat due to researcher bias. In software engineering it is common for researchers to perform experiments and studies of their own approaches (e.g., algorithms, techniques, processes.) By definition, this is biased. We all have a desire for our approach to succeed and even when we try very hard our preferences may lead to decisions that affect the outcome of a study.
Ideally we would like for the approaches we create to be evaluated by others that have no stake in the outcome. In practice, however, the time and cost of conducting studies, and the speed by which technical progress happens, makes self-evaluation the standard practice.
One mitigating practice in our community is to compare the proposed technique or approach against others, and perhaps use standard methods and artifacts to avoid selecting the “right context for my tool”. But again, that fails to address a lot of things happenning under the hood as the study is conducted and reported.
Note that systematic reviews address this issue only tangentially as they work with existing repoted data that may already be biased.
