Emotion as a cognitive artifact

Emotion as a Cognitive Artifact and the Design Implications for Products That are Perceived As Pleasurable
Frank Spillers, Experience Dynamics – USA


Product design that provides aesthetic appeal, pleasure and satisfaction can greatly influence the success of a
product. Traditional cognitive approaches to product usability have tended to underestimate or fragment
emotion from an understanding of the user experience. Affect, which is inexplicable linked to attitudes,
expectations and motivations, plays a significant role in the cognition of product interaction, and therefore can
be usefully treated as a design aid. Emotion influences and mediates specific aspects of interaction before,
during and after the use of a product. These affective states regularly impact how a user manipulates and
explores a user interface in order to support a desired cognitive state.
To better understand the specific qualities of user experience impacting desirability and pleasureability, it is
necessary to understand how artifacts trigger and mediate affect and how these processes aid user cognition
during interaction. The implications for design are that emotion acts as a critical component of artifact sensemaking
and determines how artifacts are interpreted (Rafaeli and Vilnai-Yavetz, 2003). Designers that
understand how cognitive artifacts interchange with affective artifacts will be better able to support actual
product use and perceived pleasure.
Keywords: Emotion and cognition, affect, pleasure, interaction design, Kansei Engineering, artifacts, cognitive
artifacts, affective artifacts


The field of usability has traditionally focused on ease of use and functionality based on
measurable, observable cognitive activity. Only recently, (Norman, 2003) has the usability
and design community begun to pay closer attention to the aesthetic, or affective aspects of
interaction design in the usability evaluation process. New avenues in emotion design
research have been opened up by the work of Jordan & Macdonald (1998), Jordan (2000) and
Desmet (2002) who in their work have advocated for a broader focus on pleasure and
emotion in the usability and design of a product’s user experience.
Emotion and ‘pleasure engineering’ is beginning to occupy a critical role in product design as
usability becomes more of a competitive differentiator in new device design such as mobile
handsets and communication devices (Lindholm; Keinonen and Kiljander, 2003).
Furthermore, pleasurable products are being seen as a key contributor to the competitive
advantage of a firm (Oh and Khong, 2003). Attractive interfaces with high aesthetic qualities
arouses attention (Kallio, 2003) are easier to learn, produce more harmonious results and
work better (Norman, 2003).

To understand how emotion can be captured and used as a design tool, it is necessary to
understand the role of cognitive artifacts and how emotions play the role of “affective
artifacts” in the interaction design process.

Artifacts and emotional state changes

Artifacts are the devices, both physical and mental, that reveal the problem solving and
problem structuring strategies of users during task completion (Spillers, 2003; Goel and
Pirolli (1992) cited in Pearce 1994). Artifacts are instrumental in problem-solving, decisionmaking
and sense-making. Norman (1991) extended artifacts to include cognitive
phenomenon, which he termed “cognitive artifacts”. Cognitive artifacts are created or elicited
in order to aid successful task achievement. They may be used as triggers to preserve
workflow integrity, as “task-switching” or “role-switching” aids to manage disturbances, or
as mediators of social activity or rhythms (Spillers and Loewus-Deitch, 2003).
Artifacts carry emotional clues for designers. Identifying the role that artifacts play during
product interaction can lead to an understanding of the emotional requirements necessary for
a design. For example, Wensveen, Overbeeke and Djajadiningrat (2002) designed an alarm
clock that predicted mood and acted accordingly based on input from the user. Their work
illustrates the importance of a tight coupling between the emotional level of interaction, the
appearance and the actual use (interaction design).

Hutchins (1995) defined cognitive artifacts as physical objects made by humans for the
purpose of aiding, enhancing, or improving cognition. Likewise, affect serves a crucial
function in interpretation, exploration and appraisal of a user interface. The more confusion a
user feels with a product, the more likely they are to engage in problem solving behaviors in
an attempt to reach a state of understanding. As users explore their concerns by appraising a
product, they become either more successful or less successful with a user interface. When
examining a new icon on a screen, a user may adopt a state of curiosity or annoyance in order
to bridge expected notions of what the icon symbolizes and what it is really supposed to
represent. According to Spillers (2004), the curiosity or annoyance provides an emotional
state change that can either propel the user toward a feeling of satisfaction (success) or
disappointment (failure).

Changes in emotional state may serve any of the following functions:
• Explore, manipulate or investigate the interface
• Produce a shift in concentration or attention
• Free up cognitive resources to focus on the task
• Alter the social arrangement or group dynamics where the product is being used

Just as a cognitive artifact is used as a vehicle to perform a task (Hutchins, 1999), so to is
emotion used as a variable in task completion. For the designer, emotions in this view are
viewed as co-active aspects of the design, and not merely by-products of the design or
interaction. In short, the significance of the emotion in the user interaction becomes of
primary importance due to its sense-making properties.

Affective artifacts as cognitive aids

The primary role of an artifact is to aid and extend cognitive abilities. Cognitive artifacts
mediate emotional state changes, and help manage workload, error minimization and task
accomplishment (Hutchins, 1999; Norman, 1991; Spillers, 2003). “Affective artifacts”
represent or elicit emotions and assist product interaction and user cognition during the
product appraisal process (See figure 1).


Figure 1, Artifacts that are created or accessed during product interaction take on affective
properties as they interchange with emotions in order to aid cognition and task performance.
(Note: Cognition is separated in Figure 1 merely for illustrative purposes and is not intended
to imply that emotion is a “separate” activity of cognitive processing).

Desmet (2002) emphasized the role that concerns play in how people relate to and appraise
products. Concerns may also serve more specific task functions, such as acting as triggers to
problem solving or to restarting interrupted tasks (Dix and Wilkinson, 2003). Concerns that
arise during product interaction, may serve the user in practical ways. For example, users
who responded to a new PDA operating system interface (the Sharp Zaurus 5600 Qtopia
Desktop), raised concerns of complexity with the PDA, with regard to it’s use of top tabs as a
global navigation metaphor (Spillers 2004). See Figure 2.

Most users in the Zaurus study sought the familiarity of the Palm desktop and used the
concern of complexity to try and “mentally map” device navigation during interaction. Users
brought preconceived attitudes to the Linux based PDA (i.e. that it should be as familiar as
the Palm Operating System). Attitudes such as these, according to Keinonen (1998), are
formed because they serve a number of psychological and social functions. Users do not want
to have to relearn the system. They also do not want to feel inferior to a high tech device
(PDA) that they feel they are already experts at on the “standard” dominant platform (Palm


Figure 2, The Sharp Zaurus 5600: Top tabs confused users and added a sense of overwhelm
to interactions with the device. Users were unfamiliar with the design metaphor of the Qtopia
Operating System (Qtopia is an Open Source, free OS).
(Image credit: Amazon.com and Sharp Corporation).

Jaasko and Mattelmaki (2003) found that product availability and novelty is regarded as a
remarkable aspect when choosing a product. In the case of the Zaurus 5600, the novelty
assessed during usability testing (Spillers 2004) was not related to the interface, but to the
external attributes of the device. The bulkiness, heaviness and ‘clunkiness’ were criticized as
being unattractive and ranked in the top three reasons users claimed they would not purchase
the system.

Emotional state changes

Task environments are the backdrop where artifacts are created, shared and manipulated.
According to Kirsh (2000), users alter their physical environments to gain leverage over
problem solving and to aid task completion. Emotions appear to provide a similar purpose in
appraisal and performance. Hence, changes in emotional response before, during, and after
product interaction are important to note, when identifying concern in the design of products.
Fluctuations in affect may serve a similar function as environmental change, in the sense that
they can help buffer the user from error or failure. Spillers (2004) found that when interacting
with the Sharp Zaurus PDA, users generated emotional states as a way to explore, manipulate
or investigate the user interface. For example, in order to try and understand whether the
interface contained a specific feature, one user articulated confusion and annoyance. By
generating confusion, the user was able to continue persevering until she understood the
mechanics of interface functionality.

Kansei Engineering: Precursor to emotional design

Emotion sensitivity in design has its industrial origins in the early “Kansei Engineering”
approach of Mitsuo Nagamachi which was established over thirty years ago. Kansei is a
design approach aimed at capturing the consumer’s expected feeling (‘kansei’) when they
perceive images and objects toward a new product and embedding the emotion into the
product. In Japan, Kansei Engineering has been applied widely and successfully from
automobile manufacturing to community development (Nagamachi, 2002). Kansei, mixes
sensitivity, sense, sensibility, feeling, aesthetics, emotion, affection and intuition (Lee;
Harada and Stappers, 2000).

Historically, Kansei Engineering filled a gap in the product design world that traditionally
connected designer and consumer. Mass production of products resulted in a “disconnect”
between consumer and designer (Lee and Stappers, 2001). Kansei Engineering re-unites the
pleasure based qualities of a design with the individual it is designed for. Kansei is also part
of a wider pattern of consumer preference for experiential based interactions that offer a
“high touch” feeling over products that primarily deliver high-tech interactions (Naisbitt,
Naisbitt and Philips, 2001).

If Kansei Engineering’s purpose is to organize design requirements around the emotions that
embody user expectations and interaction, then emotion can meaningfully be treated as a
design tool. The “kansei” can be considered the “emotional signature” of a product. For
interaction designers, instead of looking at purely functional behavioral criteria, the focus
ought to be on identifying artifacts that trigger and mediate emotional response.

Sense-Making properties of artifacts

Emotion is a critical element of artifact sense-making according to Rafaeli and Vilnai-Yavetz
(2003). Emotion, they argue, is central to how artifacts are interpreted. Shifts in emotion
assist sense-making. Reliance on physical artifacts may also trigger and elicit cognitive
artifacts (emotion) to extend sense-making abilities. For example, when planning an event
without a calendar, a user may verbally re-cite the days of the week based on a mental
reference of the current date. While this recall is occurring, the user may simultaneously
recall events from the previous week, year or decade (triggered by a special date or time of
year). The recall may elicit an emotion such as urgency, disappointment or excitement. The
benefit of this affective state might be to add cognitive resources (artifacts) to the current
situation in order to learn more from past events. Or it may assist in applying perspective to
an anticipated situation or problem.

According to Rafaeli and Vilnai-Yavetz, sense-making of the artifact involves emotion in
three ways:
1. Instrumentality: Tasks the artifact helps accomplish.
2. Aesthetics: Sensory reaction to the artifact.
3. Symbolism: Association the artifact elicits.

Artifacts appear to both trigger and elicit emotional states. Wertenbroch and Carmon (1997)
found that “Consumers enable themselves to maintain the quality of their experiences over
time by affecting the internal or external resources and constraints under which they make
their choices”. They refer to this as engaging in ‘dynamic preference maintenance’. Emotion
in product interaction seems to play a similar role. For example, users may delay gratification
(or evaluation) with a product feature in order to feel fully satisfied that the overall product
meets expectations and desires.

Perception of pleasure

Emotions govern the quality of interaction with a product in the user’s environment and
relate directly to appraisal of the user experience. Jaasko and Mattelmaki (2003) presented a
framework for user experience where pleasure must satisfy two levels. The first level
involves appearance (aesthetics) and user interface (usability).The second level extends to
user personality (socio-cultural context), product meaning (time/historic context),
environment (physical context), interaction (use context) and product novelty (market
Figure 3 below, provides a narrative of how a product can violate a user’s perception of
pleasure with a product.


Figure 3, Analysis of a British design magazine writer’s appraisal (Exon, 2000) of a WAP
(Wireless Application Protocol) enabled mobile phone, provides insight into how concern,
appraisal and emotion influence perception of pleasure throughout the interaction lifecycle.
According to Keinonen (1998), emotions that accompany product usability inevitably lead to
generalizations made about the product with regard to its perceived usefulness. Keinonen also
found that expectations users have toward the expected usability of a product also differ
greatly to actual measured usability.

Perception of pleasure encapsulates the usability experienced, the attitudes formed, and the
emotions felt during product appraisal. In the WAP phone example, Exon’s concerns before,
during and after product use, illustrate this negative attitude formation. In short, lack of
satisfaction at any stage of the lifecycle can jeopardize the user experience.
A closer analysis of the pleasure that Exon failed to perceive with the WAP phone, provides
valuable design clues. Note the affective artifacts that are generated as a result of the
appraisal (See Table 1 below).

Concern 1: “WAP is for early adaptors” (Friends may think he is excessive with technology). Affective Artifact: Social identit(Phone should to be acceptable to peers).
Concern 2: “WAP phone is humiliating to use” (Device creates shame) Affective Artifact: Competitive pride (Performing regular social tasks should be elegant).
Concern 3: “WAP phone is all hype” (Disappointment in marketing promises). Affective Artifact: Enthusiasm for new device (Promoted features should meet expectations).

Table 1, Concerns are accompanied by the implied needs (affective artifacts) that the device
fails to deliver.

The user’s evaluation about whether to keep a product or return it to the store; recommend
the product to a friend; or generate an emotion of ownership, loyalty and commitment to the
product are outcomes of perception of pleasure. The more closely a product can invite and
deliver on user expectations, while intensifying emotional response sets that form favorable
attitudes, the more pleasurable the product will be perceived by the user (Jordan 2002).


Emotions govern the quality of interaction with a product in the user’s environment and
relate directly to appraisal of the user experience. Users generate emotion as a way to
minimize errors, interpret functionality, or obtain relief from the complexity of a task. As a
user appraises a product, they may develop new concerns that cause them to alter their task
exploration, seek or solicit help, or begin another task in order to gain a feeling of confidence
before completing the more difficult task.

Emotion acts as a cognitive artifact in task achievement and is central to how other artifacts
are interpreted and how pleasure is perceived. Emotion also plays a valuable role in sensemaking
(Rafaeli and Vilnai-Yavetz, 2003) and impacts how users interpret, explore and
appraise a user interface. Artifacts that embody affective properties can be viewed as
affective artifacts and therefore captured as valuable design criteria.

Emotion plays a significant role in the actual and perceived experience with products
(Norman, 2003; Jordan and MacDonald, 1998; Jordan, 2002; Desmet, 2002). Cognitive
artifacts mediate and arbitrate the performance and capabilities involved in how users
perform their tasks (Spillers and Loewus-Deitch, 2003). Affective artifacts are artifacts that
are transformed by the process of emotional state changes during product interaction.
Measurable emotional responses with products are apparent where attitudes, values, goals
and expectations are coupled with usability and pleasureability (Jaasko and Mattelmaki,
2003). In this view, emotion is seen as an integral component of the design and an important
driver of cognitive processing and task performance. User expectations are coupled with the
emotional state that accompanies or codifies interaction expectations and the emotional
signature or “kansei” is reflected in how users perceive pleasure with the product.


I would like to acknowledge my father, Roger Spillers, for his assistance in locating some of
the more obscure references used in this work. Also special thanks to Marco Van Hout who
provided valuable critiquing, support and access to Desmet’s papers and findings.


Carmon, Ziv. & Klaus Wertenbroch. 1997. Introduction to the Special Issue on the Dynamics
of Consumer Preferences. Marketing Letters. (8), 55-56. Available from URL:
0the%20Dynamics.pdf>. [Accessed 2004 March 12th].

Davidson, Richard. 2003. Seven sins in the study of emotion: Correctives from affective
neuroscience. Brain and Cognition. 52(1), 129-132.

Desmet, Pieter.M.A. 2002. Designing Emotions. Doctoral Dissertation. TU Delft,

Desmet, Pieter.M.A. 2003. A multilayered model of product emotions. Delft: Netherlands.
Available from URL:
<http://static.studiolab.io.tudelft.nl/gems/desmet/papermultilayered.pdf >. [Accessed 2004
May 24th].

Desmet, Pieter.M.A., Hekkert, P. and Hillen, M.G. In press. “Values and emotions; an
empirical investigation in the relationship between emotional responses to products and
human values”. Barcelona: Proceedings of the fifth European academy of design conference.

Dix, A. and D. Ramduny-Ellis, J. Wilkinson. 2003.”Trigger Analysis - Understanding Broken
Tasks”. The Handbook of Task Analysis for Human-Computer Interaction, D. Diaper & N.
Stanton (eds.), New Jersey: Lawrence Erlbaum Associates. Available from URL:
pdf>. [Accessed 2004 March 12th].

Ekman, Paul. 1999. Basic emotions. In T. Dalgleish and T. Power (Eds.) The Handbook of
Cognition and Emotion. Pp. 45-60. Sussex, U.K.: John Wiley & Sons, Ltd.

Exon, Mike. 2000. “WAP's the point? Do people buy WAP mobile phones for a terrible link
to the Internet, or because the phones have a cool auto flip cover on the mouthpiece?” Design
Week, 15(33), 48.

Gray, Jeremy.; Braver, Todd. and Raichle, Marcus. 2002. “Integration of emotion and
cognition in the lateral prefrontal cortex”. PNAS, 99(6), 4115-4120. Available from URL:
<http://www.pnas.org/cgi/reprint/99/6/4115.pdf>. [Accessed 2004 April 4th].

Hekkert, Paul., Mostert, Marc and Guido Stompff. “Dancing with a Machine: A Case of
Experience-Driven Design”. Pittsburgh: Proceedings of the DPPI ’03.

Hutchins, Edwin. 1995. Cognition in the Wild. Cambridge: MIT Press.

Hutchins, Edwin. 1999. “Cognitive Artifacts”. In Robert Wilson and Frank Keil (Eds.). The
MIT Encyclopedia of the Cognitive Sciences. Cambridge: MIT Press, 127.

Jaasko, Vesa. and Tulli Mattelmaki. 2003. “Observing and Probing”. Pittsburgh: Proceedings
of the DPPI ’03.

Jordan, Patrick W. 2000. Designing Pleasurable Products. New York: Taylor and Francis.

Jordan, Patrick W. 2002. How to make brilliant stuff that people love and make big money
out of it. Chichester: John Wiley and Sons.

Jordan, Patrick.W. and Alastair Macdonald., 1998. ‘Pleasure and product semantics’, in M.A.
Hanson (ed.) Contemporary Ergonomics. London: Taylor & Francis, 264-8.

Kallio, Titti. 2003. “Why we choose the more attractive looking objects- Somatic Markers
and Somaesthetics in User Experience”. Pittsburgh: Proceedings of the DPPI ’03.

Keinonen, Turkka. 1998. One-dimensional usability - Influence of usability on consumers'
product preference. University of Art and Design Helsinki, UIAH: A21.

Kirsh, David. 2000. “A Few Thoughts on Cognitive Overload”. Intellectica . [Online]. San
Diego, USA. Available from URL: <http://iclserver.
ucsd.edu/~kirsh/Articles/Overload/published.html>. [Accessed 2003 October 14th].

LeDoux, Joseph. 1996. The emotional brain: The mysterious underpinnings of emotional life.
New York: Simon & Schuster.

Lee, SeungHee., and Pieter J. Stappers., 2001. Kansei evaluation of matching 2D to 3D
images extracted by observation or 3D objects. Available from URL:
[Accessed 2004 April 4th].

Lindholm, Christian; Keinonen, Turkka and Harri Kiljander. 2003. Mobile Usability. New
York: McGraw Hill.

Nagamachi, Mitsuo. 2002. Kansei engineering as a powerful consumer-oriented technology
for product development. Applied Ergonomics. 33(3): 289-294.

Naisbitt, John, Naisbitt, Nana and Phillips, D. 2001. High-tech, High-touch. New York:
Broadway Books.

Norman, Donald. 1991. “Cognitive artifacts”. In John M. Carroll, (Ed.) Designing
interaction: Psychology at the human-computer interface. Cambridge: Cambridge University
Press, 17-38.

Norman, Donald. 2002. “Emotion and design: Attractive things work better”. Interactions
Magazine, ix (4), 36-42.

Norman, Donald. 2003. Emotional Design: Why We Love (Or Hate) Everyday Things. New
York: Basic Books.

Oh, Wilson and Poh Wah Khong. 2003.“Competitive advantage through pleasurable
products”. Pittsburgh: Proceedings of the DPPI ’03

Overbeeke, Kees. and Wensveen, Stephan. 2003. “From Perception to Experience, from
Affordances to Irresistables”. Pittsburgh: Proceedings of the DPPI ’03.

Pearce, Annie. 1994. Design of Artifacts from a Cognitive Engineering
Perspective. [Online] Atlanta: USA. Available from URL:
<http://maven.gtri.gatech.edu/sfi/resources/pdf/TR/TR004.PDF>. [Accessed 2004 April 4th].

Rafaeli, Anat and Iris Vilnai-Yavetz. 2003. “Emotion as a Connection of Physical Artifacts
and Organizations”. [Online] Tel Aviv: Israel. Available from URL:
<http://www.si.umich.edu/ICOS/Rafaeli-Vilnai-Dec-3-2003.pdf >. [Accessed 2004 May

Ratner ,Carl. 2000. A Cultural-Psychological Analysis of Emotions. Culture and Psychology.
6(5-39). Available from URL: <http://www.humboldt1.com/~cr2/emotion.htm>. [Accessed
2004 April 12th].

SeungHee, L; Harada, Akira; Stappers, Pieter. 2000. Pleasure with Products: Design based on
Kansei. [Online]. Delft: Netherlands. Available from URL: <http://www.io.tudelft.nl/idstudiolab/
research/pdfPool/2000/00Lee_PWPPlea.pdf>. [Accessed 2004 April 12th].

Shih-tseng, T.H.; Chin-chueh, C; Mu-feng, C. 1995.“The Role of an Object's Affective and
Cognitive Properties in Attitude Formation and Change”. Journal of National Chung Cheng
University. Sec. II: Social Sciences. 6(1:1), 127-147. Available from URL:
<http://www.lib.ccu.edu.tw/indoor/journal/jnccu/v6s2_6.htm>. [Accessed 2004 April 4th].

Spillers, Frank. 2003. “Task Analysis Through Cognitive Archeology”. The Handbook of
Task Analysis for HCI, Eds. Laurence Erlbaum Associates, D. Diaper and N. Stanton (Eds.).
[Online]. New Jersey. Available from URL:
[Accessed 2004 March 10th].

Spillers, Frank., and Daniel Loewus-Deitch. 2003. “Temporal attributes of shared artifacts in
collaborative task environments” in Proceedings of HCI2003: Workshop on the Temporal
Aspects of Tasks. [Online]. Bath: United Kingdom. Available from URL:
[Accessed 2004 March 9th].

Spillers, Frank. 2004. Product Review: Sharp Zaurus SL-5600. [Online]. Los Angeles: USA.
Available from URL: <http://www.newsfactor.com/story.xhtml?story_id=24062>. [Accessed
2004 May 22nd].

Van Hout, Marco. 2004. Interactive Products and User Emotions: How designers of
interactive products can get insight in evoking targeted emotions. [Online]. Twente:
Netherlands. Available from URL:
<http://members.home.nl/l.vanhout/archive/FINAL-THESIS300104.pdf>. [Accessed 2004
March 25th].

Vygotsky, Lev. 1962. Thought and Language. Ed. and Trans. E. Hanfmann and G. Vakar.
Cambridge, MA: MIT Press.

Wensveen, Stephan., Overbeeke, Kees. and Djajadiningrat, Tom. 2002. Push Me, Shove Me
and I Show You How You Feel: Recognizing mood from emotionally rich interaction.
[Online]. Delft: Netherlands. Available from URL:
WensDISPush.pdf>. [Accessed 2004 March 25th].

Wertenbroch, Klaus.; Carmon, Ziv.. 1997. “Dynamic Preference Maintenance”. Marketing
Letters, 8 (1): 145-152.

Frank Spillers is a usability consultant and user centered design specialist with Experience Dynamics, a
consultancy based in Portland, Oregon USA. Frank holds a Masters in Cognitive Science from Birmingham
University UK, with a focus on the usability of Collaborative Virtual Environments. He has worked with
companies such as Sony, HP, Intel, Key Bank, Bank One, Land Rover and others. Frank has been recognized
for his contribution to the field of Usability Engineering, contributing as a subject matter expert in Human
Factors Engineering for the National Skills Standards Board (U.S. Dept. of Labor) and as one of 30 current
leading experts in the field of task analysis (user research). His innovative technique for capturing user behavior
was recently published in the Handbook of Task Analysis for Human Computer Interaction.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License