Empirical Software EngineeringQuestions in Need of Answers

There are at least three key elements that make a study relevant.

First, a relevant study pursues a research question that matters to practitioners or that might affect groups of researchers or both. Very often I see very well conducted studies, with careful designs and implementations, but the question being asked is either uninteresting or inconsequential making the study just a good exercise in empirical software engineering. With some many good questions left to answer this is a really a shame.

Second, a relevant study can teach us something. A study that provides a new data point can contribute to the body of knowledge, a study that confirms a hypothesis in a new context can extend what we know, a study that challenges an existing assumption can change the way we think or work. Clearly, there is a spectrum or relevance depending on the level of contribution of each piece of work. Not all studies will challenge the status-quo but those that do so soundly are often very relevant.

Third, for a study to be relevant it must be disseminated it in the right venues, to the right audience. Two natural tendencies make this harder than it sounds. First, we often send our work to venues that we are familiar with even when the best targets of our studies might be domain-specific venues where we can reach more relevant folks.   Second, we are used to a presentation style that may not be appropriate for all venues yet reusing that style is often “cheaper” (in the short term at least). These myopic tendencies can render the best studies “invisibles” to the relevant audience.