What are the guiding principles you use when reporting the results of empirical studies in software engineering?
Posted by elbaum | Posted on 07-05-2011
0
A key guiding principle for reporting is transparency, and for me that means making the raw data available, following a standard reporting format, and describing the context of the study as much as possible.
I am pushing so that every paper we publish that contains any empirical component makes all the raw data available in the web in a format that is easily readable/processable by others. I find that, even when papers include what I would deem as complete sets of tables and figures, as a reseacher I often have questions or doubts that require to dig a bit deeper on an aspect of the work and that often requires access to the raw data. Besides, that is completely necessary for reproducing some studies.
Over the years of performing empirical studies on testing and analysis techniques I have developed and convereged towards a standard reporting structure. I believe the consistency of a standard report makes papers easier to follow since readers know what to expect and where to expect it, and if it is not there then it will raise some questions. It is kind of neat to see that that type of format has pretty much propagated throughout the testing and analysis community in the last 10 years (even in the late 90s it was ok for a T&A paper with an empical component not to state its limitation or threats).
