A method for evaluating long-term user experience
Kujala, S., Roto, V., Väänänen-Vainio-Mattila, K., Karapanos, E. & Sinnelä, A. (2011).
UX Curve: A method for evaluating long-term user experience. Interacting with Computers, 23(5), 473–483.
David Coimbra
More
The same
Less
And why? Hold on to your answer. We'll come back to it.
Most users in the paper liked their phone less over time. Why? That's what the method is built to find out.
of UX evaluation methods focus on long-term experience.
Vermeeren et al. (2010), cited in Section 1
What we already knew
Longitudinal studies: expensive, slow, and participants drop out.
5 templates per participant
| Method | When data is collected | Who draws / how | Effort for the participant | Focus |
|---|---|---|---|---|
| ESM | In real time, several moments a day | Questionnaire that interrupts the activity | High: hard to sustain for months | Feelings in the moment, no memory bias |
| DRM | End of day, chronological reconstruction | Participant reconstructs the day's episodes | Medium, but still needs a longitudinal study | Daily episodes (e.g. iPhone over 5 weeks) |
| CORPUS | Retrospective, over 1 year | Interview; the researcher draws; from today backwards | Low | UX dimensions on a 10-point scale |
| iScale | Retrospective | Participant sketches the curve in an online survey | Low, and it scales to many people | Usefulness, ease of use, innovativeness; reliability of recall |
| UX Curve | Retrospective, 3–12 months | Participant draws by hand; the researcher asks why | Low (1.5 h session) | Attractiveness, ease of use, utility, degree of usage; link to satisfaction and loyalty |
28 researchers, 13 products of their choice (Facebook, phones, laptops, a work system…)
Everyone could draw the curves and found it “fun”.
…but writing reasons for 5 curves was exhausting → reasons are now spoken aloud.
10 curve-shape categories were too many → down to 3: improving, deteriorating, stable.
Finnish users
months of use (mean 8.8)
curves
Aged 21–53 · Nokia and LG (13/20 smartphones) · recruited via a newspaper ad and the Nokia customer database · 2 movie tickets
Session up to 1.5 h: initial questionnaire → general curve → 4 curves (counterbalanced) → final questionnaire (satisfaction, expectations, repurchase, recommendation)
Analysis: open coding of reasons → Hassenzahl's model (pragmatic / hedonic) · Mann-Whitney U test
482 reasons (287 positive, 195 negative)
28.7 experiences per person vs 3.4 with CORPUS
Attractiveness curve: mean answers (1–7 scale). ● improving ● deteriorating
No significant relationship: repurchase intention (brand loyalty) and degree of usage.
Not explained by length of use: 8.5 vs 8.2 months.
20 Finns, phones only, 2 brands. Part were recruited from the Nokia customer database, and one of the authors was at Nokia Research Center.
The method requires that the person didn't abandon the product after the first bad experiences. Those who quit never show up.
Only the start is compared with the end (< 1 mm = stable). A curve that dives in the middle still counts as “improving”.
“I did not remember when something happened.” Annotations are inconsistent, and comments like “the camera quality was surprisingly low” are too vague to design from.
Users who are already satisfied may remember the curve going up (peak-end). The curve may reflect the evaluation rather than predict it.
Small n, several tests, and only 1 of 5 curves significant. The study itself calls itself exploratory.
Pros
Cons
Where to use it
Not for a prototype in development: that's what usability tests are for.
Think of an app you got bored of. What would make its attractiveness go back up?
Users whose attractiveness curve went up were also the most satisfied. What comes first?
Vote on your phone
Netflix ratings, app reviews, NPS surveys: they all ask people to remember. When is memory exactly what a company should measure?
What do you think?
New things to discover: updates, new features, a fresh look.
What it says about me: personalisation, things worth showing others.
Attractiveness is hedonic: usability fixes alone won't bring it back.
Both were collected in the same session, from the same memory. It's a correlation, not a direction.
A longitudinal or cross-sequential study (Kujala et al., 2019).
People act on what they remember (Norman, 2009). For fixing an interface, you need what happened.
By the paper's rule (start vs end): improving, despite the dip.
Who said less at the start? That's the attractiveness curve: 13 of 20 users went down.
Thank you.