Considering Qualitative Evaluation
An IEEE VIS 2016 Tutorial
An IEEE VIS 2016 Tutorial
Evaluation is increasingly recognized as an essential component of visualization research. However, evaluation itself is a changing area of research. New methods to extend and validate our research continue to emerge. This half-day tutorial is designed for beginning to intermediate audiences. We will focus on methods for collecting qualitative data using a mixture of talks and hands- on activities. After completing this tutorial, people will have a richer understanding of the benefits and challenges of qualitative empirical research.
| Time | Description |
|---|---|
| 2.00 | Brief introduction of the organizers and qualitative evaluation approaches. [Slides] |
| 2.15 | Talk 1: Setting the stage: generalizability, precision, and realism. [Slides] |
| 2.45 | Talk 2: Thinking of observation as a skill. [Slides] |
| 3.00 | Activity 1: Practicing observation. [Slides] |
| 3.40 | Break. |
| 4.00 | Activity 2: Practicing interviewing I. [Slides] |
| 4.20 | Activity 3: Practicing interviewing II. [Slides] |
| 4.40 | Talk 3: Thinking of interviewing as a skill. [Slides] |
| 4.55 | Activity 4: Practicing interviewing III. [Slides] |
| 5.15 | Panel: Open discussion. |
| 5.35 | Closing: Next steps and discussion. [Slides] |
| 5.55 | End. |
Sheelagh Carpendale is a Professor in Computer Science at the University of Calgary where she holds a Canada Research Chair in Information Visualization and NSERC/AITF/SMART Technologies Industrial Research Chair in Interactive Technologies. She has many received awards including the E.W.R. NSERC STEACIE Memorial Fellowship; a BAFTA (British Academy of Film & Television Arts Interactive Awards); an ASTech Innovations in Technology award; and the CHCCS Achievement Award, which is presented periodically to a Canadian researcher who has made a substantial contribution to the fields of computer graphics, visualization, or human-computer interaction. She leads the Innovations in Visualization (InnoVis) research group and initiated the interdisciplinary graduate program, Computational Media Design. Her research focuses on information visualization and large interactive displays. She both conducts and publishes about evaluation in information visualization with a particular focus on qualitative evaluation.
Uta Hinrichs is a Lecturer at the University of St Andrews, Scotland, UK in the SACHI research group. Her research is at the intersection of visualization, HCI, design, the humanities, and art. Her work focuses on designing and studying the use and experience of interactive systems that facilitate the exploration and analysis of (cultural) data collections from academic, leisurely, and artistic perspectives. Uta holds a PhD in Computational Media Design from the University of Calgary.
Trevor Hogan is a Lecturer at Cork Institute of Technology, Ireland and an external PhD candidate at the Bauhaus-University at Weimar, Germany. The aim of his research is to describe and better understand how embodiment influences and augments an audiences experience of data representations. His work is strongly interdisciplinary and may be situated in the field of interactive design, at the intersection of tangible computing, human-computer interaction, information science and psychology.
Alice Thudt is a PhD student in Computational Media Design at the University of Calgary. She is interested in how visualization can transform personal data collections into evocative mementos. Her research aims to understand how people construct meaning with personal digital data collections and how a symbiosis of digital and physical visualization can be used to support emotional resonance with digital artefacts and archives. She has used variations in cultural probe techniques in her evaluations.
Melanie Tory is a senior research scientist at Tableau. Her research focuses on interactive visual data analysis, exploring techniques and tools to help people analyze data more effectively. This includes intuitive interactions with visualizations and the design and evaluation of tools that support the holistic data analysis process, including sensemaking, analytical guidance, and collaboration. Before joining Tableau, Melanie was an Associate Professor in visualization at the University of Victoria. She is Associate Editor of IEEE Computer Graphics and Applications and has served as Papers Co-chair for the IEEE Information Visualization and ACM Interactive Tabletops and Surfaces conferences. Melanie has conducted and published a large number of evaluation studies using both qualitative and quantitative techniques and contributed a chapter on empirical methods to the Handbook of Human Centric Visualization.
Jo Vermeulen is a Postdoctoral Fellow in the InnoVis group at the Interactions Lab at the University of Calgary. His research interests lie at the intersection of Human-Computer Interaction, Information Visualization and Ubiquitous Computing. He is interested in addressing interaction challenges with non-traditional interactive technologies and observing how people interact with them. His work has focused on techniques to improve discoverability, feedback and feedforward.
Jagoda Walny is a PhD Candidate in the InnoVis group at the Interactions Lab at the University of Calgary. She is interested in how everyday visual thinking methods such as sketching can inform information visualization interfaces — both in terms of representation and interaction — to support people in better engaging with and understanding information. To gain a deeper understanding of these thinking practices, she has conducted extensive qualitative evaluations using information visualization as a lens to study visual constructs left behind from visual thinking.
Abstract: Evaluation is increasingly recognized as an essential component of visualization research. However, evaluation itself is a changing research area. In particular, the many variations of qualitative research are emerging as important empirical methods. This half-day tutorial is designed for beginning to intermediate audiences. We will focus on the basic methods for analyzing qualitative data using a mixture of talks and hands-on activities. In particular we will consider closed and open coding as well as clustering and categorizing coded data. After completing this tutorial, attendees will have a richer understanding of the benefits and challenges of qualitative empirical research and, more specifically, how to analyze qualitative data.
Abstract: Evaluation is increasingly recognized as an essential component of HCI research. However, evaluation itself is a changing research area. In particular, the many variations of qualitative research are emerging as important empirical methods. This half-day tutorial is designed for beginning to intermediate audiences. We will focus on the basic methods for analyzing qualitative data using a mixture of talks and hands-on activities. In particular we will consider closed and open coding as well as clustering and categorizing coded data. After completing this tutorial, attendees will have a richer understanding of the benefits and challenges of qualitative empirical research and, more specifically, how to analyze qualitative data.
