The topic of our last lecture was about groupware evaluation. Altough evaluation is critical for designers and practitioners, it also seems to play an important role in scientific research. In 2000, Pinelle and Gutwin [1] provided a review of evaluation methods used in papers from ACM CSCW conference proceedings 1990-1998. They selected all papers that introduced or evaluated a groupware system. They found that 17 of all 45 papers did not contain any formal evaluation. Further, 13 papers included controlled laboratory experiments and only 1 paper described a field experiment. The remaining papers relied on less rigorous field tests or explorative evaluations.
In 2007, Wainer and Barsottini [2] conducted a similar analysis of ACM CSCW papers published between 1998-2004. Unlike Pinelle and Gutwin, they also included papers with no groupware systems or evaluations. Two major differences between the results were discovered: significant increases of using field experiments and quantitative data. In general, experimental research with rigorous hypothesis testing seemed to be an increasing trend.
Personally I hope this trend keeps increasing in the future. In the end, scientific research is not about creating fancy systems and testing if they work. It's about creating knowledge, generalizable to a wide range of applications. As summarized by Wainer and Barsottini:
"Creativity, in natural sciences, comes in the proposal of new theories and hypothesis, and progress comes from systematically testing the theories and hypothesis with well planned, precise, and rigorous experiments."
[1] D. Pinelle and C. Gutwin. A Review of groupware evaluations. In Proceedings of WETICE2000,
pp. 86–91. IEEE Computer Society, 2000.
[2] J. Wainer, C. Barsottini. Empirical research in CSCW – a review of the ACM/CSCW conferences from 1998 to 2004. Journal of the Brazilian Computer Society 13 (2007) 27–36.
No comments:
Post a Comment