Analyzing Qualitative Data
An IEEE VIS 2017 Tutorial

Introduction


Evaluation is increasingly recognized as an essential component of visualization research. However, evaluation itself is a changing research area. In particular, the many variations of qualitative research are emerging as important empirical methods. This half-day tutorial is designed for beginning to intermediate audiences. We will focus on the basic methods for analyzing qualitative data using a mixture of talks and hands-on activities. In particular we will consider closed and open coding as well as clustering and categorizing coded data. After completing this tutorial, attendees will have a richer understanding of the benefits and challenges of qualitative empirical research and, more specifically, how to analyze qualitative data.

Schedule


Time Description
8:30 Introductions of the organizers and brief round of participant introductions. [Slides]
8:45 Talk 1: Benefits and Challenges of Qualitative Data Analysis. [Slides]
9:00 Talk 2: Choosing a Focus. [Slides]
9:15 Activity 1: Choosing a focus.
9:30 Talk 3: Closed Coding. [Slides]
9:45 Talk 4: Open Coding. [Slides]
10:10 BREAK
10:30 Activity 2: Open Coding.
11:05 Talk 5: Clustering and Categorizing. [Slides]
11:20 Activity 3: Clustering and categorizing.
11:40 Talk 6: Consensus and Agreement. [Slides]
11:45 Closing Panel for questions and discussion.
12:10 END

Important Readings


  • Beyer, H. and Holtzblatt, K., 1998. Contextual design: defining customer-centered systems. Morgan Kaufmann Pub.
  • Boyatzis, R., 1998. Transforming Qualitative Information: Thematic Analysis and Code Development. Sage Publications.
  • Bryman, A., 2001. Social Research Methods, Oxford: Oxford University Press
  • Carpendale, S., Evaluating Information Visualizations, in A. Kerren, J.T. Stasko, JD. Fekete, and C. North (Eds.). Information Visualization – Human-Centered Issues and Perspectives, Volume 4950 of LNCS State-of-the-Art Survey, Springer, pp. 19-45, 2008.
  • Corbin, J. and Strauss, A., 2015. Basics of Qualitative Research 4e. Sage Publications.
  • Crabtree, B. F., and Miller W.L., eds. 1999. Doing qualitative research. Sage Publications.
  • Creswell, J. W., 2007. Qualitative Inquiry & Research Design. Sage-Pub Ltd.
  • Grammel, L., Tory, M., and Storey, M.-A., 2010. How information visualization novices construct visualizations, IEEE Transactions on Visualization and Computer Graphics.
  • Heath, C., Hindmarsh, J., and Luff, P., 2010. Video in Qualitative Research. Sage Publishing.
  • Hogan, T., Hinrichs, U. and Hornecker, E., The Elicitation Interview Technique: Capturing People’s Experiences of Data Representations. In IEEE Transactions on Visualization and Computer Graphics, 2016.
  • Kaplan, A., 1964. The conduct of Inquiry. San Francisco: Chandler.
  • Knudsen, S., Carpendale, S., 2016. View Relations: An Exploratory Study on Between-View Meta-Visualizations. In Proceedings of the NordiCHI. ACM. New York, NY, USA.
  • Knudsen, S., Jakobsen, M. R., Hornbæk, K., 2012. An exploratory study of how abundant display space may support data analysis. In Proceedings of the NordiCHI. ACM. New York, NY, USA.
  • Lam, H., Bertini, E., Isenberg, P., Plaisant, C. and Carpendale, S., Empirical studies in information visualization: Seven scenarios. In IEEE Transactions on Visualization and Computer Graphics, 18(9), pp.1520-1536, 2012
  • Maxwell, J. A., 2005. Qualitative Research Design: An Interactive Approach. 2nd Edition. Sage Publications.
  • McGrath, E., Methodology matters: Doing research in the behavioral and social sciences. In Readings in Human-Computer Interaction: Toward the Year 2000 (2nd ed.), 1995.
  • Mendez, G. G., Hinrichs, U. and Nacenta, M., 2017. Bottom-Up vs. Top-Down: Trade-Offs in Efficiency, Understanding, Freedom and Creativity with InfoVis Tools. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems.
  • Munzner, T., A nested model for visualization design and validation. In IEEE Transactions on Visualization and Computer Graphics, 15(6), pp. 921-928, 2009.
  • Saldaña, J., 2015. The coding manual for qualitative researchers. Sage Publications.
  • Thudt, A., Lee, B., Choe, E. K., and Carpendale, S., 2017. Expanding Research Methods for a Realistic Understanding of Personal Visualization. IEEE Computer Graphics and Applications, 37(2), 12-18.
  • Tory, M., 2014. User Studies in Visualization: A Reflection on Methods. In Handbook of Human Centric Visualization (pp. 411-426). Springer New York.

Organizers


Sheelagh Carpendale is a Professor in Computer Science at the University of Calgary where she holds a Canada Research Chair in Information Visualization and NSERC/AITF/SMART Technologies Industrial Research Chair in Interactive Technologies. She has many received awards including the E.W.R. NSERC STEACIE Memorial Fellowship; a BAFTA (British Academy of Film & Television Arts Interactive Awards); an ASTech Innovations in Technology award; and the CHCCS Achievement Award, which is presented periodically to a Canadian researcher who has made a substantial contribution to the fields of computer graphics, visualization, or human-computer interaction. She leads the Innovations in Visualization (InnoVis) research group and initiated the interdisciplinary graduate program, Computational Media Design. Her research focuses on information visualization and large interactive displays. She both conducts and publishes about evaluation in information visualization with a particular focus on qualitative evaluation.

Uta Hinrichs is a Lecturer at the University of St Andrews, Scotland, UK in the SACHI research group. Her research is at the intersection of visualization, HCI, design, the humanities, and art. Her work focuses on designing and studying the use and experience of interactive systems that facilitate the exploration and analysis of (cultural) data collections from academic, leisurely, and artistic perspectives. Studying the use of technology in-situ through qualitative research methods such as field observations, interviewing and video analysis is core to her research. Uta holds a PhD in Computational Media Design from the University of Calgary.

Søren Knudsen is a Postdoctoral Fellow in the InnoVis group at the Interactions Lab at the University of Calgary. He holds a PhD in Computer Science from University of Copenhagen. His research focuses on information visualization, HCI, and large interactive displays. He is interested in studying technologies in-situ and in bringing parts of reality into lab contexts. He uses a mix of qualitative and quantitative methodology in his approach, and study visualization problems as they occur within and across a range of application domains.

Alice Thudt is a PhD student in Computational Media Design at the University of Calgary. She is interested in how visualization of personal data can support self-reflection and expression. Her research aims to understand how people construct meaning with personal digital data collections and how both digital and physical visualization can be used for personal storytelling and reminiscing. She has used different qualitative research and analysis methods in her research ranging from observations and interviews to variations of a technology probe method. She also published an article on the benefits of qualitative methods for gaining a more realistic understanding of personal visualizations.

Melanie Tory is a senior research scientist at Tableau. Her research focuses on interactive visual data analysis. This includes intuitive interactions with visualizations and the design and evaluation of tools that support the holistic data analysis process, including sensemaking, analytical guidance, and collaboration. Before joining Tableau, Melanie was an Associate Professor in visualization at the University of Victoria. She is Associate Editor of IEEE Computer Graphics and Applications and has served as Papers Co-chair for the IEEE InfoVis and ACM Interactive Surfaces and Spaces conferences. Melanie has conducted a large number of evaluation studies and contributed a chapter on empirical methods to the Handbook of Human Centric Visualization.