Table of Contents

User Experience Evaluation tools

Evaluation should be integral to any design activity. Evaluation in innovative product development practices however is highly complicated. It often needs to be applied to immature prototypes, while at the same time users’ responses may greatly vary across different individuals and situations. This creates highly complex data which traditional statistical practices often fail to analyze. This section describes a number of tools for the elicitation and analysis of rich subjective data about product experience.

XGms - Interactive Statistical Analysis

XGms is a Multi-dimensional Scaling and Regression tool that enables its user to explore a set of data in an interactive way by forming and testing alternative models. The tool has been deployed by Jean-Bernard Martens and extensively used in the field of psychophysics and image quality assessment. A number of new procedures have been tested on XGms for the analysis of users’ subjective judgments of interactive products, where quality judgments have been found to be substantially different from attribute judgments that are typically analyzed in the field of psychophysics (e.g. principle of homogeneity of perception).

XGms Interactive Statistical Analysis tool

Selected publications:

iScale - Cost-effective elicitation of longitudinal data

iScale is a survey tool aimed at eliciting longitudinal data about users’ experiences in a cost-effective manner. iScale employs sketching in imposing a process in the reconstruction of one’s experiences. Two versions of iScale, the Constructive and the Value-Account iScale, were motivated by two distinct theories on how people reconstruct emotional experiences from memory. The constructive iScale tool imposes a chronological order in the reconstruction of one’s experiences. It assumes that chronological reconstruction results in recalling more contextual details surrounding the experienced events and that the felt emotion is constructed on the basis of the recalled contextual details. The Value-Account iScale tool explicitly distinguishes the elicitation of the two kinds of information: value-charged (e.g. emotional) and contextual details. It assumes that value-charged information can be recalled without recalling concrete contextual details of an experienced event due to the existence of a specific memory structure that stores the frequency and intensity of one’s responses to stimuli.

In recent experiments it was shown that iScale resulted in a) an increase in the number of experience reports that individuals provided, b) an increase in the amount of contextual information for the reported experiences, and c) an increase in individuals’ accuracy in recalling concrete details of the experienced events over free recall.

 iScale

Selected publications:

Latent Conceptual Analysis

Latent Conceptual Analysis is a semi-automated analysis technique for the content analysis of qualitative data. It combines traditional qualitative coding procedures (open and axial coding) with computational approaches (vector-space model) for assessing the similarity between documents. It employs visualization in enabling the interactive exploration of qualitative data. Current applications concern the analysis of user feedback data (e.g. interview data, self-reported experience narratives) as well as the analysis of free-text data relating to field complaints in call centers.

 Latent Conceptual Analysis

Selected publications: