HCI Engineering: Possible Blog Posts – ex

The Blog addresses…..

The Blog addresses any issues, which relate to the different ways of doing or researching HCI and in particular to the approaches and frameworks, proposed here. It will also be used to introduce relevant feedback to the site and to extend both the approaches and the frameworks, as and if required.

1. 29 February 2016 – User Requirements and Design Problems – Same or Different?

I am often asked ‘What is the difference between ‘user requirements’ and ‘design problems’? I need to pay my dues on this one. The Craft design research exemplar, presented here, includes only user requirements. The Engineering exemplar includes both. No-where is the difference explained.

User Requirements and Design problems – Same or Different?

2. 4 March 2016 – HCI Design Research and HCI Design Practice – What are Their Relations?

I have always argued for the strongest possible relations between HCI research and HCI practice. Here, I make some suggestions as to what those relations might be.

HCI Design Research and HCI Design Practice – What are Their Relations?

3. 7 March 2016 – Old Papers Never Die, They Only Fade Away……?

I often wonder, if old research papers have anything to say to us now, given the radical changes in information technology? Here are some comments on a joint paper I wrote some 35 years ago with John Morton, Phil Barnard and Nick Hammond entitled: Interacting with the Computer: a Framework

Old Papers Never Die, They Only Fade Away…..

4. 12 March 2016 – HCI as Art?

HCI as Art has the problem of conceptualising the work of art itself (the application) and the experience of the person, engaging with the work of art (the user). There is no general consensus about how to conceptualise either. However, I was lucky enough to come across an interesting example of the latter.

HCI as Art?

5. 3 April 2016 – From MSc Student to UCLIC Director – Some Reflections by Yvonne Rogers

Yvonne Rogers was a member of the MSc class of 1982/83. She was appointed Director of UCLIC in 2011. As a result, her MSc Reflections take on a wider and more general significance. For that reason, they are included here. The reflections are prompted by a set of standard headings. For other MSc student reflections – see elsewhere on this site.

From MSc Student to UCLIC Director – Some Reflections by Yvonne Rogers

6. 1 May 2016 – Is It Important to Distinguish ‘Hard’ from ‘Soft’ Design Problems.

‘Hard’ and ‘soft’ are used with respect to problems in general. The terms have also been used in HCI research to describe design problems, notably by Dowell and Long (1989). The comprehensiveness of their address of the differences warrants inclusion here, as the basis for arguing for their importance. Additional comments clarify issues, which arise in the application of the concepts to approaches and frameworks for HCI research.

Is It Important to Distinguish ‘Hard’ from ‘Soft’ Design Problems?

7. 25 May 2016 – HCI design frameworks are intended to support research, Can they also be used to support design practice? Peter Timmer suggests how an HCI Engineering framework can integrate HCI design practice with business services.

Introduction to Peter Timmer

Peter Timmer was an MSc student of the class of 1989/90.

He worked at the EU from…..to…..as…..doing…..

After leaving the EU, he worked as…..doing…..

In the following paper (Timmer, 2010), Peter suggests how HCI design practice might be integrated with business services, like sales and marketing, better to relate to their associated practices of end-to-end and incremental sales and profit measurement for digital channels. To this end, he applies the Dowell and Long conception (1989 and 1998). Additional comments are intended to clarify issues of particular concern to HCI design research. Read more…..

A Sketch of the ‘Conversion Funnel’. Can Cognitive Engineering Assist in its Design?

Peter Timmer

Bisant Ltd, 49 Clonmell Road, London, N17 6JY, England

John Long Comment 1

Cognitive Engineering, as used in the paper, can be considered co-extensive with HCI, as used here in the engineering approaches and frameworks.

End Comment 1

1. Abstract

The ‘white heat’ of commercial web design is increasingly around a business’ ‘conversion funnel’. Conversion funnels are the means by which the business services its customers, and such funnels are critical to the performance of the digital channel. In this paper, a sketch of commercial conversion funnel design practice is offered. A complex relationship is described, where web analytics increasingly helps the business measure performance. Dowell’s and Long’s conception (1998) [1], of the cognitive engineering design problem, is then used to suggest how the conversion funnel design process can be better structured, in a manner suitable for ‘design for performance’, and to address a business context of engineering that is at present ignored in commercial web site design.

Comment 2

The Dowell and Long conception (1998) appears here as the HCI research engineering framework.

End Comment 2

2. Introduction

Business services are delivered by ‘channels’, and channels are maintained by businesses for the purpose of contact and communication with consumers. Door‐to‐door sales representatives, direct mail catalogues, a branch network of shops, call centres, and digital (web sites) are all channels for managed contact with consumers. Channels all come with associated financial expenses to the business, at minimum, for maintenance of the channel.

Sales and Marketing functions within businesses drive product sales to consumers across all relevant channels, via techniques such as advertising campaigns. While it has always proved hard to measure the impact of a budgeted television commercial on sales within a particular channel, in the case of digital marketing this has proved to be easy. While a television advert rarely mentions a particular store, to the detriment of the product or brand being advertised, in the case of the digital channel, a banner advert can drive the prospective customer directly to a retail ecommerce site, and onwards into a purchasing journey.

One consequence of the directness of this relationship, between digital adverting and digital purchasing, is that all budgets can be measured within the business, and the expense of a banner advertising campaign, combined with the expenses of designing, building and maintaining a retail web site for the digital channel, can be weighed against the direct contribution to sales that the digital channel makes. Business expenses can be measured end‐to‐end, and incremental sales from the digital channel, and therefore profits, are known. The digital channel has a bright future because of this accountability between the marketing expense of the channel, and the channel’s marginal contribution to product sales.

Given that the design of ‘effective’ digital web sites has a bright future; this paper looks at how design for the digital channel might be structured within this context. In the first part of this paper, commercial practice is examined, in terms of the analysis tools available to the information architect who will design the core goal‐oriented journeys (education, consideration & fulfilment) and interactive experiences, such as navigation, that the web site needs to support. The concept of the ‘Conversion Funnel’ will be outlined in sketch form due to the size of the actual area of interest. In the second part of the paper, an attempt will be made to understand the sketched space in terms of a design problem of ‘cognitive engineering’ where ‘performance’ is the beginning and end of a design cycle; and the hypothesis testing of website analytics should be replaced with efforts to optimise web site performance through a more rigorous approach to the prescription of informed design specifications. The information architect will thereby become a cognitive engineer.

3. A Sketch of the conversion funnel

The concept of the conversion funnel has arisen from web site analytics, and speaks of a user’s journey across web pages that lead to a business service goal being achieved, such as a sale, appointment being booked, or completion of an application form. In these instances, such as clicking on a ‘Checkout’ button (in a retail context); the user is normally guided through a step‐wise process, to buy the product. This normally involves steps such as ‘Personal details’ (needed for shipping and logistics), Financial details (needed for billing and finance), Accept Terms and Conditions (needed by legal), ‘Opt‐in’ to future product marketing updates and offers, and so forth. Each of these sub‐sets of customer data is mandated by the business, for service delivery in the digital channel.

A statistical phenomena of such step‐wise journeys, is that you can’t have more users at step 3 than were at step 1. Through the conversion funnel you can only lose users, and subsequent service opportunities. Web site analytics enable each page on the user’s journey to receive a unique ‘tag’ that enables user clicks to be tracked. All buttons and controls on all pages can be tagged, so where there is more than one way to the next step, individual buttons on the page can be assessed for success, at guiding the user’s journey. Web analytics’ claim is that this tagging and analysis process informs web site ‘optimisation’, by generating statistical aggregations of user behaviour on a set of tagged pages.

If we take an example, Figure 1 shows illustrative conversion funnel data for a ‘quote and buy’ service, offered by a business in the digital channel. A five page journey is of interest, and while all pages are tagged, a conversion funnel has been set‐up for a user journey from offering a quotation, to purchase confirmation – the last four pages of the journey.

Figure 1 tells the following story.

On a business web site, a Pet Insurance product is described, and offered for purchase in the digital channel. Product information is displayed to the user on a single page, the ‘Pet Insurance Product Page’, shown in Figure 1 top left. Analytics data reveals that on a single day, this product page had 832 unique visitors. The page holds an important primary Call to Action, ’Get a quote’.

When the user clicks the ‘Get a quote’ button, on the product page, they are taken to a ‘Quote’ page. Accurate quotations require a number of pieces of information to be elicited from the prospective customer, so on the Quote page radio buttons and drop‐down choice controls enable the pet (to be insured) to be described. A dynamic quote is thereby generated within the page, based on the user’s actions. Here, a quote of £9.00 results, and once a well‐formed quotation is generated and displayed, a ‘Buy now’ button is offered to the user. Analytics show 261 users, of the 832 users who visited the product page, progressed to see the Quote page. That is 31.3% retention of interest in the product. This group, of 261 users who click on the ‘Get a Quote’ button, may be considered by the business to be a target group of ‘warm prospects’, users judged as having a genuine interested in the product, and are ready to act within the digital channel. Of the 571 visitors to the product page that didn’t seek a quote, some may have been conducting research, and then progressed an order via a Call Centre channel, or approached a shop branch or broker channel. The digital channel may still be serving important user needs even when action within the channel is not evident.

If the prospective customer then clicks the ‘Buy now’ button on the Quote page, having considered the quote, they are taken to a ‘Personal details’ form, where contact and financial details are captured. A ‘Next’ button then progresses the user onto a ‘Review Order’ page, and a ‘Confirm’ button will generate an Order Confirmation page. In theory, everybody who sees the Order Confirmation page is a customer of the business, as they have bought the product via the digital channel. The Order Confirmation page is therefore the end of a transaction process, and cannot be found and visited via the navigation, or passed as a URL within an email.

Analytics data show, across these pages, a gradual drop‐off in visitor numbers as prospective customers ‘bounce’ out of this journey to purchase the product, maybe via bookmarks or by closing the window. It can be seen that only 10 users saw the Confirmation page on the day in question, 10 prospective customers became actual customers of the business. Tabulated below the analytics data for page visits are percentage statistics that reflect the volume of visitors retained by the digital channel, from preceding steps. Only 14.6% percent of people who sought a quote, elected to then ‘Buy now’, and of this elite group, two thirds were then lost during form filling.

Finally, a conversion funnel has been set up from the Quote page through to the Confirmation page. If the 261 visitors to the ‘Quote’ page are considered a target group for purchasing (the warm prospects), and only 10 of that group of prospective customers actually became customers (the ones that saw the Confirmation page), the businesses digital channel has a 3.8% conversion rate for this product.

4. Information architecture and the conversion funnel sketch

Now, information architects design the pages that are reflected in such statistical analyses of the web site, and web site analytics reflect a limited ‘glimpse’ of a web site design’s ‘performance’. Where a business has sales targets for the digital channel, it is possible that the current volume level of 10 product sales per day is insufficient to justify the expense of the channel, and uplift is needed to 20 sales per day, for the achievement of business benefit in offering the channel to prospective customers. In this case, of the 261 visitors to the Quote page, changes need to be made to the digital channel (web site) and/or sales process, so that a 7.6% conversion rate is attained (20 confirmation pages are seen from the 261 who saw the Quote page). This may be possible without changing the website significantly. A ‘5% off’ offer could be given to the user. Alternatively, theories may be created about how the Quote page could be improved, to get more people through to click ‘Buy now’ and start filling the Personal Details Form. Instructional text may be insufficient to reassure the user how easy the process will be, or ‘Buy now’ buttons may be out of sight and require scrolling. The business, in conjunction with the information architect, may thereby seek to redesign parts of the site to achieve improved conversion rates and sales volume uplift. Alternatively, the business, in conjunction with digital marketers, may place an increasing number of banner adverts on partner sites, to drive twice the volume of users to the product page, and thereby mine the existing statistical patterns through the conversion funnel to reach the same end result, 20 target sales for the channel. This paper is concerned with the former.

Drawing on conversion funnel data, the web site is thereby changed, maybe with clearer instructional text and two ‘Buy now’ buttons, one at the top and one at the bottom of the page. The changed site is then measured, in a similar manner to the first design, and over time a comparative conversion funnel is generated. While a method of applied science would appear to be being employed, the difference between two (measured) digital experiences may be a great number of site alterations, each alteration reflecting a hypothesis, with many parties within the business generating hypotheses and requesting changes. If uplift is attained, it may be hard to know which alteration was most effective; but then all the business wants is uplift, not the practice of a purely scientific approach.

Comment 3

Both Science and Applied approaches to and frameworks for HCO research are proposed here.

End Comment 3

To compliment multiple design changes to pages, it is possible to conduct ‘split’ testing, or AB testing, on page elements such as button designs, labelling, font size, imagery and so forth. In this case, two buttons may be designed, one saying ‘Buy now’, and another saying ‘Join us’, and placed in the page in equal measure until a statistical difference in usage is detected. Then, one design may prevail as being more effective, and so testing single hypotheses can also be undertaken. An image of a sick animal may be more effective at driving people to ‘buy’ pet insurance than an image of a happy customer. The page ‘design specifications’ that support the conversion funnel can in this way be tweaked and thereby optimised to meet business goals. The cost of changing the digital channel’s user experience needs to be weighed against incremental gains, through better conversion rates.

Earlier, when the conversion funnel was sketched, the claim that web analytics can ‘optimise’ a web site was touched upon. Web site analytics does appear to support web site optimisation, but optimisation is about more than measuring aggregated and individual click paths. Optimisation has a second component that has been mentioned here, design specifications. Behind the aggregated statistics are designed pages that form the user’s experience. To optimise the pages, in line with business targets, design specifications are required, and specifically ‘designs for performance’. Designed changes need to attain targets, and address unnecessary user problems getting through the funnel. The user can at any point leave the funnel simply by clicking away on a bookmark. If the prospective customer is to be converted into a business customer in the digital channel, but are confused by what is being viewed during the purchasing journey, such confusions need to be removed from the design. A specification for a design solution is needed, to compliment the design problems uncovered by the web site analytics. Together, analytics and design specifications support web site optimisation.

Comment 5

The possible relationships between user requirements and design problems are addressed in the Blog 1 of 29 February 2016.

End Comment 5

5. Cognitive engineering and design problems

The above picture of commercial design practice takes in many individuals, disciplines and concerns; and so the design process needs guidance, in how to structure the way in which the root design problem of digital channel ‘performance’ is conceived, and thereby approached in the design process. In this regard, the paper asserts that the sketch provided, of the conversion funnel, can be best understood in terms of a design problem of cognitive engineering. A conception of cognitive engineering as ‘design for performance’ (Dowell & Long, 1998 [1]) will be used to re‐express the problem of conversion funnel design, in a manner suitable to the engineering context of ‘optimisation’ that exists around it, in all but name.

The digital channel to prospective customers is one channel of many, but all channels can be conceived of in terms of Ergonomics, in that all channels will involve to a greater or lesser extent, humans interacting with devices to perform effective work (Dowell & Long, 1989 [2]; Long, 1987 [3]). Where businesses offer common services across multiple channels, the humans in question (prospective customers) interact with functionally equivalent devices to perform the same ‘work’, such as product purchase. Web sites offer ‘electronic form’ devices to the user. Direct mail channels offer paper‐based form devices, pre‐paid envelope devices, and sometimes even pens. Via different channels a common range of products can be purchased. To the prospective customer, the work is the same – generating an order; but with different devices, levels of performance may widely differ.

Comment 6

The same ‘work’ would be expressed by the Engineering Framework, proposed here, as transforming a product from to unpurchased to purchased. Different channels offer different means by which the transformation can be brought about.

End Comment 6

Dowell and Long conceive of performance as having two inseparable components, ‘Task Quality’ and ‘User Costs’. Quality here refers to how well formed the order is, by interacting with the devices that the business offers (via the channel).

Comment 7

‘Task Quality’ would be expressed by the Engineering Framework, proposed here, as how well (that is, as desired) the unpurchased product is transformed to purchased – see also Comment 6.

End Comment 7.

The static and standardised images of a paper‐based catalogue may generate more returned products to a business than a web site that offers zooming, panning, and a library of product photographs from different angles. When the order is fulfilled, users of the digital channel may be more likely to be delighted with their purchase, than a user who was unable to determine aspects of the product until closer inspection was possible. The quality of the user’s work varies across channels. Associated with work of a given quality are costs to the user. These may be most simply conceived of as time and effort, or may be more refined and make reference to costs of mental and physical human behaviours (e.g. training), plus device costs such as load on a server of posting back information to that server, after every click. In our retail example, the time cost of waiting for the postal delivery of a paper‐based order form (back to the business in the direct mail channel) may be removed by using the digital channel’s electronic form. Going to a branch to buy the product may involve the prospective customer incurring greater costs (time, effort, financial), but generate the highest quality of work. There again, a lack of stock at a local branch may lead to the work being left undone when this channel is chosen, with other consequent costs to the user, of performing work of zero quality, such as frustration. When looking at digital channel performance, it is therefore helpful to look at performance in these terms, quality of work done at some cost.

In this retail example, the similar work undertaken across different channels may be abstractly conceived of as one whereby ‘ownership’ of an object or objects, is transferred from the seller to the buyer, along with the physical location or evidence of ownership.

Comment 8

This characterisation of work is consistent with that appearing in Comments 5 and 6.

End Comment 8

A lamp appears in your living room, or a policy number is sent to you via post. In each case, the prospective customer interacts with devices, provided by the seller, to ensure this work is performed effectively. Dowell & Long call the human interacting with the channel’s devices the ‘worksystem’. The worksystem incurs costs as its contribution to the expression of work performance.

Comment 9

Both humans and channel devices incur costs. See also earlier in the paper.

End Comment 9

Worksystem design for the digital channel needs to be similarly conceived. The product browsing experience needs to be simple and informative, the route to the checkout needs to be self evident, and product selection supported by devices such as ‘baskets’, with clear means of action (‘Add to Basket’ buttons) and action reversal (‘Remove’ buttons). A shopping checkout experience that looses 96% of customers that approach it, with something in their basket, thereby becomes a very interesting and commercially valuable design concern. Channel performance needs to be improved, but can that improvement be brought about by increasing the likelihood of higher quality work being done, at a lower cost of time and effort to the worksystem? It is suggested here that this is a helpful way to think about structuring the root design problem of the digital channel, performance. Generating uplift in the digital channel through redesign of web pages is a problem of cognitive engineering.

Dowell and Long’s conception mirrors the behaviours of worksystem, with an abstract place, entitled the ‘domain’ which is comprised of domain ‘objects’.

Comment 10

Domain objects are modelled in Dowell and Long and in the Engineering Framework, proposed here, as objects having both physical and abstract attributes and states.

End Comment 10

In retail domains, the objects may be modelled in a variety of ways; what is important to the conception is that each behaviour of the worksystem is understood in terms of progressing some desired transformation of a domain object on its way to a final state whereby the work is accomplished. In a retail domain, ownership of a domain object is transferred. One simple model of such a domain may be in terms of transforming ‘order’ objects from a state of ‘empty and unfulfillable’ (no object in a shopping basket) through to ‘valid and unfulfillable’ (something in the shopping basket that can be purchased), and then on through the conversion funnel to a domain object (target) state of ‘valid and fulfillable’ (all personal, financial, and logistical details known to pay for and deliver the object to the new owner in return for the cash value). Until ‘Terms and Conditions’ are accepted for example, a valid order is unfulfillable, as the prospective customer has not yet accepted the ‘conditions of business’, for scenarios such as theft in transit. Such business logic pervades the digital channel, and yet is not a necessary part of the ‘work’ carried out by worksystems in the branch (shop) channel, where responsibility for safe transport is undertaken by the consumer once they have left the shop, with a well manufactured bag supplied by the business. The conception of a cognitive engineering design problem therefore ties worksystem behaviour into a view of how that behaviour is transforming the domain object into its target state – accomplishing effective work. Task quality is used to measure the work done in the domain, and so the measurement of worksystem performance has a second element, inseparable from worksystem costs. Costs cannot be truly understood unless we know what work they were incurred ‘doing’.

Comment 10

And indeed, how well that work is done – whether as desired or not. See the frameworks proposed here.

End Comment 10

Engineering ‘design problems’ then arise when desired levels of worksystem performance are poorly aligned (do not match) with levels of cost and quality desired by the business (digital channel provider). At such a point, after desired performance is known and expressed, and actual performance known and expressed, and poor alignment is thereby expressible, cognitive engineering design processes can be undertaken, as they can be grounded in worksystem performance measurement.

The cognitive engineering design processes that are broadly employed when solving performance‐ oriented design problems are ‘diagnosis’ and ‘prescription’. The business, at minimum, will have expectations that a digital channel will generate, across all its ‘valid and fulfillable’ orders, a level of profit. If this level of profit is being attained, but the business has unusually high levels of returned goods, and subsequent re‐funds, a place to start diagnosis might be on the ‘browse’ and ‘add to basket’ experience. Are product sizes well displayed? Do prospective customers know what they are adding to the basket? Alternatively, profit may be visible in the value of total order objects that progress from ‘empty and unfulfillable’ to ‘valid and unfulfillable’, but enormous numbers of prospective orders may be abandoned in the Conversion Funnel, maybe at the Terms and Conditions step. Diagnosis in this case may start by looking at the Worksystem at this point in the retail experience.

The cognitive engineering process of diagnosis will draw on as much Business Intelligence data as is available, about worksystem behaviour, to establish a plausible theory about why the performance data indicates a design problem exists. Prescription, then involves specifying a design solution, one that will alter the performance data, bring about ‘uplift’, and align the businesses desired level of performance with actual levels. Cognitive engineering, by separating worksystem from domain, modelling worksystem behaviours as separate from domain object transformations, and measuring worksystem costs alongside work quality, offers digital channel designers a valuable means of structuring how the root design problem of performance is conceived (Dowell & Long, 1998) [1].

6. Performance measurement

Dowell & Long’s conception of the cognitive engineering design problem is not alone, in trying to outline basic fundamental components that make up an engineering discipline of ‘cognitive design’. It has however been chosen, because of its emphasis, less on ‘cognitive behaviour’ leading the design process, as much as deficiency in ‘performance’, and then re‐specifying the worksystem, and most importantly the human ‘cognitive’ component (that leads to the creation of business ‘customers’), as a means to address a design solution to the problem. The user’s cognition drives their action, which thereby progresses them through the conversion funnel. A button may be rendered larger, and brighter, to capture a customer’s mental process of ‘attention’ (Long & Baddeley, 1984 [4]), a clear table may support ‘reasoning’ about a choice (Buckley & Long, 1990 [5]), and thereby encourage the desired user ‘click’ (behaviour), that will improve performance data towards the businesses desired performance levels, and turn problem into solution (alignment).

By attributing a measurement of cost and quality to an expression of performance, and the concepts of ‘desired’ and ‘actual’ levels of performance to assist in the framing of a ‘design problem’, cognitive engineers have a number of places to start the diagnosis process.

Firstly, the business needs to be able to express desired performance. This will likely be in aggregate terms, and largely reference task quality, especially the number of occasions the worksystem generated ‘valid and fulfillable’ orders, and the total value of those orders, against some measurement of cost (expense), in supporting the digital channel to the consumer. Performance is also likely to make reference to the missed opportunity in terms of low task quality and high user costs. Orders abandoned in the conversion funnel (‘work’ left undone in the domain), time to progress through the conversion funnel at each step, complaints and customer satisfaction ratings (worksystem costs). It is in measuring actual performance via such criteria that web site analytics, and wider Business Intelligence, provides the cognitive engineer with a valuable toolset for performance measurement and thereby design problem diagnosis.

While the engineering approach relies on the business to express ‘desired performance’, once this is expressed, means are required for measuring the actual performance, and so by: tagging pages across the site; modelling user journeys and paths across the site; measuring the time spent on each page; and comparing this to the value of the basket at each step; measurements of actual performance (of individual prospective customers) can be aggregated into summary statistics. These summary statistics are models of performance that support diagnosis, diagnosis of points in the journey where performance starts to deteriorate, against the performance which is desired. Diagnosis may start from pages where performance deteriorated beyond repair, instances where prospective customers were lost from the funnel (and wider site) completely – they ‘bounced’ away, or pages where time spent before advancement appeared unusually long, as hesitancy and uncertainty sets in. Web site analytics can provide granular and aggregated data to support the cognitive engineer in diagnosis, and reasoning about why the performance data are as measured, and indicative of the origins of a plausible design problem; the design problem under consideration.

In contrast to scientific hypotheses, that are part of a scientific method to generate scientific knowledge that supports better ‘explanation and prediction’; cognitive engineering needs to develop a set of diagnoses that when addressed during the cognitive engineering process of ‘prescription’ (specification), will improve and align the Worksystem’s performance data so that it better matches the businesses desired level of performance. If uplift is not possible, the digital channel may disappear as a candidate user experience choice for prospective customers, when interacting with the business.

Diagnosis therefore needs to go beyond an approach of applied science, and instead to look at the user’s journey to a point where performance deteriorates, as well as the page where deterioration is most marked. Theories need to be generated by the cognitive engineer that draw upon all the disciplines that support cognitive design (attention, reasoning, vision), to explain the performance (design) problem. The abandonment of a basket at the page before confirmation may have little to do with a problem with the order summary page itself. Quite the opposite is possible when the user’s journey is examined. The order summary page may reveal the true transportation expense of the ‘valid and fulfillable’ order, expenses that if known earlier would have resulted in fewer prospective customers abandoning their order at such a late stage. Diagnoses need to consider the customer’s journey, and the mental events that the pages support, such as ‘true order cost realisation’. Prescriptions will then follow, in the form of design specifications that address the diagnosis and thereby solve the design problem.

7. The Conversion Funnel design revisited

Business Intelligence for the digital channel is taking many forms, and introducing many new issues around not only consumer privacy, but also around the limits of what can be inferred about consumers from their digital footprints (Baker, 2008 [6]). As many interests compete for influence over the user experience through the conversion funnel, a framework is required to structure the design process so that interests are weighted appropriately. The business owns the channel to its prospective customers, and once it has decided on the key objectives for that channel, it will inevitably set performance targets for the channel. Technologies are evolving to subsequently service the businesses interest in digital channel performance measurement, and it is important to understand the strengths and limitations of the ‘glimpse’ of customer cognition that statistical models provide, something Guy Debord would call ‘the spectacle’, and how we design based on these impoverished representations of the consumer’s reality (Debord, 1967 [7]). In this paper, Dowell & Long’s (1998) conception of the cognitive engineering design problem has been used to bring some order to these competing interests; and better understand what contribution different parties are making to the primary business objective of digital channel performance. Click paths and conversion rates represent aggregations of humans interacting with computers to perform work, and their greatest contribution to the primary business objective is to measure performance, and provide the cognitive engineer with diagnostic inputs, and evidence a re‐designed conversion funnel really is the design solution the business is looking for. To compliment measures of quality and cost, the cognitive engineer needs to tie a digital experience in the digital channel (prospective customers interacting with digital devices), to a domain where work is done, and business targets are attained. This design process will then draw upon models of cognition. Cognitive ‘engineering’ of a design solution is thereby grounded in performance, at the start (diagnosis) and end (prescription) of each design cycle. It is because this is increasingly how the digital channel is being designed, that Dowell & Long’s conception has been chosen. Cognitive engineering in practice, is about designing devices that support cognition, which supports action, which transforms domain objects to accomplish work, of a quality and at a cost. The interests of the cognitive engineer and the business are thereby aligned, by a common interest in channel performance.

8. References

[1] Dowell, J., Long, J. B., 1998. Conception of the cognitive engineering design problem. Ergonomics 41 (2), 126‐139.
[2] Dowell, J., Long, J. B., 1989. Towards a conception of an engineering discipline of human factors. Ergonomics, 32 (11), 1513‐1535.
[3] Long, J.B., 1987. Cognitive Ergonomics and Human Computer Interaction. In: Warr, P., (Ed.) Psychology at Work. Penguin, Harmondsworth.
[4] Long, J. B., Baddeley, A., (Eds.). 1984. Attention and Performance IX (International Symposium on Attention and Performance). Lawrence Erlbaum.
[5] Buckley, P., Long, J. B., 1990. Using videotext for shopping – a qualitative analysis. Behaviour & Information Technology, 9 ( 9), 47‐61.
[6] Baker, S., 2008. The Numerati. Jonathan Cape, London. [7] Debord, G., 2004. Society of the Spectacle. Rebel Press.
[7] Debord, G., 2004. Society of the Spectacle. Rebel Press

8. 10 June 2016 – Smell-enhanced human-computer interaction ? Surely not! Marianna Obrist, however, does not agree.

Introduction to Marianna Obrist

I first met Marianna Obrist, following a seminar, which she presented at UCLIC in May, 2014. The seminar was entitled: Multi-Sensory Experiences: How we Experience the World and How we design Technology. I much enjoyed Marianna’s seminar, which together with a chat afterwards revealed common interests in understanding human experience/behaviour and its relationship to design. My PhD thesis involved multi-dimensional vision and audition. However, research on the multi-sensory experiences of touch, taste and smell was new to me.

Marianna and I subsequently exchanged e-mails about the research and in particular concerning the way forward and the issues raised. I offered to review two papers reporting this research for the HCI Engineering website – an offer, which Marianna accepted, as follows.

‘I am particularly interested in the impact of the research on technology design, as it was not only a question, raised following my UCLIC seminar; but also on other occasions. However, I still believe that we need to establish the foundation and vocabulary for the senses of touch, taste and smell. I would like, then, for the review to include both the understanding of the experience of these senses and the application of that knowledge to the design of technology involving HCI. Both the future direction of the research should be considered, including the issues raised.’

In the following paper, Obrist et al (2014) argue that technologies for capturing and generating smell are emerging and our ability to engineer such technologies and use them in HCI is rapidly developing. They investigated the experience of smell by means of an on-line questionnaire, which produced 10 categories of smell experience. The categories, in turn, were explored for their design implications. The research is considered to contribute to the development of smell-enhanced HCI. Comments are based on the original review of the paper and are intended to clarify issues of particular concern to HCI design research. Read more…..

See also Comments 7, 17 and 34.

End Comment 7

We collected 439 smell stories, that is, descriptions of personal memorable experiences involving smell.We distributed a questionnaire through crowdsourcing, ensuring a large-scale coverage and variety of smell stories. We analyzed the stories and identified 10 main categories and 36 sub-categories. Each category was described with respect to its experiential and emotional characteristics and specific smell qualities. Besides smell stories associated with the past (e.g., memory of loved people, places, life events) we identify stories where smell played an important role in stimulating action, creating expectations, and supporting change (e.g., of behavior, attitude, mood). Smell can sometimes also be invasive and overwhelming, and can affect people’s interaction and communication. Within the categories, we identify common smell qualities and emotions, which support the exploration of opportunities for design. In particular, we discuss the implications for technology based on feedback from participants and on a brainstorming session with HCI researchers working on smell technologies. The main contributions of this paper are (1) an experiencefocused understanding of smell experiences grounded in a large sample of personal smell stories, which allowed us

Comment 8

Issues, concerning the meaning of ‘understanding’ are raised in Comment 3.

End Comment 8

(2) to establish a systematic categorization and description scheme for smell experiences, leading to

Comment 9

What motivates the particular categorisation and description scheme chosen? It might, for example, be ‘simply descriptive’, that is intended to characterise the phenomena described. However, the interest is, here, in and to what extent it was driven by design concerns or indeed is appropriate to address the needs of the latter – see (3). In turn, this will impact the relationship with the particular design knowledge acquired – see also Comments 7 and 17.

End Comment9

(3) the identification of technology implications by participants, and

Comment 10

See Comments 7 and 9, as concerns the research motivation with respect design and design knowledge.

End Comment 10

(4) the exploration of design potentialities by HCI

researchers.

Comment 11

See Comments 7, 9, and 10 concerning design, design knowledge and so ‘design potentialities’.

End Comment 11

THE HUMAN SENSE OF SMELL

The sense of smell is the most complex and challenging human sense.

Comment 12

This is a very strong and general claim. It could do with more justification, otherwise it seems like ‘special pleading’. The reasons, which follow are not convincing. There is not much difference between the senses, when it comes to complexity. They are certainly all complicated enough, when it comes to including them in design.

End Comment 12

Hundreds of receptors for smell exist and the mixing of the sensations, in particular with our sense of taste, is immense [2]. The sense of smell is further influenced by other senses such as vision, hearing, and touch; plays a significant role for memory and emotion; and shows strong subjective preferences. Willander and Larsson [33] showed that autobiographical memories triggered by smell were older (mostly from the first decade of life) than memories associated with verbal and visual cues (mostly from early adulthood). Moreover, smell-evoked memories are associated with stronger feelings of being brought back in time, are more emotionally loaded, and are experienced more vividly than memories elicited through other modalities [15,33]. No other sensory system makes the direct and intense contact with the neural substrates of emotion and memory, which may explain why smell-evoked memories are usually emotionally potent [15]. The emotion-eliciting effect of smell is not restricted to the context of autobiographical memories. Smell is particularly useful in inducing mood changes because they are almost always experienced clearly as either pleasant or unpleasant [8]. For instance, Alaoui-Ismaïli et al. [1] used ‘vanilla’ and ‘menthol’ smells to trigger positive emotions in their

participants (mainly happiness and surprise) and ‘methyl methacrylate’ and ‘propionic acid’ to trigger negative emotions (mainly disgust and anger). Interestingly, Herz and Engen [15] pointed out that almost all responses to smell are based on associative learning principles. They argued that only smells learned to be positive or negative can elicit the corresponding hedonic response and that people, therefore, should not have any hedonic preference for novel smells. The only exceptions are smells of irritating quality that strongly stimulate intranasal trigeminal structures. Such smells often indicate toxicity.

While neuroscientists and psychologists have established a detailed understanding of the human sense of smell, insight into the subjective characteristics of smell and related experiences is lacking.

Comment 13

See earlier Comments 3 and 8, concerning the differences between everyday day and scientific meanings of understanding.

End Comment 13

The exploration of this subjective layer of smell is often understood as going beyond the interest of these disciplines, but is highly relevant for HCI and user experience research.

Comment 14

If so, there is a need to define the supposed differences of what is lacking in scientific research, that is, ‘subjective characteristics of smell’ and ‘related experiences’. Also, is ‘going beyond’ a claim about the scope or about the level(s) of knowledge (or indeed both). Frameworks differ as to these aspects. See also Comments 1, 2, 5 and 6.

End Comment 14

SMELL IN HUMAN-COMPUTER INTERACTION Ten years ago, Kaye [17] encouraged the HCI community to think about particular topics that need to be studied and understood about smell. While some attempts have been made to explore smell during recent years, the potential of smell in HCI remains under-explored.

Comment 15

With regards to what has the potential of smell technology remained under-explored in HCI? The current state of smell technology seems at best modest. Its potential with respect to HCI design may be likewise. At this stage, an open mind would seem to be indicated. See also Comment 12 on complexity.

End Comment 15

Most work on smell in HCI focuses on developing and evaluating smell-enhanced technologies.

Comment 16

The use of ‘enhanced’, here, is interesting, because it implies the notion of a design problem and design solution. The difference between the two is presumably ‘better’ (that is, enhanced) performance of some sort. This is, of course, important for evaluation of the enhancement, more generally in the forms embodied in different frameworks for HCI.

End Comment 16

Brewster et al. [5] used smell to elicit memories, and developed a smell-based photo-tagging tool (Olfoto). Bodnar et al. [4] showed smell to be less disruptive as a notification mechanism than visual and auditory modalities. Emsenhuber et al. [9] discussed scent marketing, highlighting the technological challenges for HCI and pervasive technologies. Ranasinghe et al. [24] further investigated the use of smell for digital communication, enabling the sharing of smell over the Internet. More examples of smell-enhanced technologies can be found in multimedia applications [13], games [16], online search interfaces [19], health and wellbeing tools (e.g., http://www.myode.org/), and ambient displays [22].

The exploration of smell-enhanced technologies is mostly limited to development efforts and the evaluation of users’ smell perception of single smell stimuli. The smells used are often arbitrary and not related to experiences. This is because of the lack of knowledge pertaining to the description and classification of smells required for HCI [17]. Kaye points out that “There are specific ones [classification and description schemes] for the perfume, wine and beer industries, for example, but these do not apply to the wide range of smells that we might want to use in a user interface” (p. 653). Thus, previous work has a general and quite simple usage of smell.

Comment 17

It would be interesting to know if, and how, this work has been or might be built on by other researchers. Was it of influence in the present case? See also Comments 7, 9, 10, and 11, concerning design.

End Comment 17

THE POTENTIAL OF STUDYING SMELL EXPERIENCES

In contrast to the work reported above, the present paper focuses instead on experiences with smell and links them to potential technologies. We do so through stories of experiences with smell. Stories are increasingly used within user experience research to explore personal memories of past experiences, but also to facilitate communication in a design process [3,34]. Stories are concrete accounts of particular people and events, in specific situations [10],

Comment 18

Stories may be ‘concrete’; but they are also abstract, in the sense that they very often (maybe always) have meaning. Without the latter at all, they are hardly ‘stories’ (although they may be sensory experiences). The truth of the stories is, of course, here unvalidated, as are the memories at the time of perception and the reports of the memories, at the time of recall. This uncertainty needs to be carried forward into the research in some way and especially as concerns the conclusions. Otherwise, the reader may be misled as to the actual state of affairs described and the description itself. When participants make claims about the relationship between a smell and an emotion, the claim is a (subjective) description (and no more or less). There is no corroborative evidence of the claim. This is critical for the type of framework, appropriate for this case. See also Concluding Comment.

End Comment 18

and are more likely to stimulate empathy and inspire design thinking than, for example, scenarios.

Comment 19

This is a strong claim and needs more justification, than that given here. Also, scenarios and histories both have their own strengths and weaknesses. Were scenarios seriously considered, as a means for conducting the data gathering?

End Comment 19

STUDY METHOD

We asked a large sample of participants to report smell experiences that were personal and meaningful.

Comment 20

See Comment 18, concerning the concrete and abstract aspects of stories with respect to their meaningfulness. Also the Concluding Comment.

End Comment 20

We refer to the description of these experiences as smell stories. These stories were captured through a questionnaire described below, which included inspirational examples of smell-enhanced technologies at its end. Based on the examples we asked participants to reflect on their experience and future technologies. The rationale of this approach was to begin from smell experiences that matter to participants, instead of starting from an application or a particular technology.

Comment 21

The rationale is good, as far as it goes; but does not address the issues raised in Comment 18. The latter must, at least be taken as qualifications of some sort, especially as concerns the conclusions. See also Concluding Comment.

End Comment 21

Questionnaire

We created a web-based questionnaire consisting of six parts. We started with an open question to stimulate the report of a personal memorable smell experience. This was followed by closed questions aiming to elucidate the relevant emotional and experiential characteristics, as well as the smell qualities. Participants could freely choose the story to report. The questionnaire was administered through a crowdsourcing platform to obtain a large sample of smell stories. Crowdsourcing provides valid and reliable data [20] and has been used for capturing user experiences [31].

Comment 22

This is a very general rationale and is acceptable as such; but it also needs particular tailoring to the present case, in order to show how the information obtained is of the kind sought and why we might have confidence in the latter relationship. See also Comments 18, 21 and the Concluding Comment.

End Comment 22

Part 1: Smell Story

The smell stories were elicited through an initial exercise, where participants were asked to think about situations and experiences where smell played an important role. The aim was to get participants into the right frame of mind and sensitize them to smell. Next, participants were asked to describe one memorable smell experience in as much detail as possible, inspired by the questioning approach used in explicitation interviews [23]. This questioning technique is used to reconstruct a particular moment and aims to place a person back in a situation to relive and recount it. Part 1 of the survey was introduced as follows: Bring to your mind one particular memorable moment of a personal smell experience. The experience can be negative or positive. Please try to describe this particular smell experience in as much detail as possible. You can use as many sentences as you like, so we can easily understand why this moment is a memorable experience involving smell for you.

Participants were asked to give a title to their story (reflecting its meaning) and indicate if the experience was positive, negative, or ambivalent (i.e., equally positive and negative). They were also asked to indicate how personally relevant the experience was (from ‘not personally relevant at all’ to ‘very personally relevant’).

Comment 23

Again, it would useful to have the rationale, which relates these questions to the knowledge that the research intended to acquire. See alo Comments 7, 9, 10, 11 and 16.

End Comment 23

Part 2: Context

Part 2 asked participants to give further details of their reported experience via open and closed follow-up questions. There were four questions on the context of the described experience, including the social context (who else was present), the place (based on the categories used by [26]), the location (as an open field), and the time when the reported experience took place (days, weeks, months, or years ago).

Comment 24

As noted earlier (Comments 18 and 21 ), the correctness/truth of the responses, here, cannot be checked or indeed corroborated or in any other way cross-referenced. Any conclusions need to reflect these concerns, for example, by way of caution. See also Concluding Comment.

End Comment 24

Part 3: The smell

Specific questions on the characteristics and qualities of the smell were asked in Part 3. Participants characterized the smell itself using a list of 72 adjectives (i.e., affective and qualitative terms) derived from the ‘Geneva Emotion and Odor Scale’ (GEOS) [7]. Participants could also add descriptions to characterize the smell in an open feedback box. In addition, they rated the smell with respect to its perceived pleasantness, intensity, and familiarity.

Part 4: Experienced emotions

In Part 4 participants had to describe how they felt about the experience as a whole, using a list of affective terms (101 in total). They could go through the list and tick the words that best described their emotions during the experience. The words were derived from Scherer [27]. Participants could also add their own words in a free-text field.

Part 5: Smell technologies

After the participants had selected, thought about, and described a particular smell episode, Part 5 linked their personal experience to technology. The participants were engaged in a envisioning exercise inspired by work on mental time travel [30]. They were shown six inspirational examples of smell technologies, namely: Olfoto: searching and tagging pictures (CHI, [5]); Smelling screen: ambient displays (IEEE, [22]); Digital smell: Sharing smell over the Web (ICST, [24]); Scent dress: interactive fabric with smell stimulation (http://www.smartsecondskin.com/); Mobile smell App: iPhone To Detect Bad Breath and Other Smells (BusinessInsider 01/2013), and Smell-enhanced cinema: Iron Man 3 Smell-Enhanced Screening (Wired 04/2013). These six technologies cover areas of relevance for HCI (mobile, ambient, wearable, personal, and entertainment computing), give realistic examples of smell technologies from research, and include recent, commercial examples.

We asked the participants to imagine any desirable change that future smell technology might make (or not) with respect to their personal smell experience.

Comment 25

Change here presumably implies enhancement and so better human (smell)-computer interactions, for example, performance of some sort. See also Comment 16. Performance, in one form or another, figures in all the frameworks, proposed here.

End Comment 25

We asked them the following questions:

(1) How could your experience be enhanced?

(2) What technology are you thinking about?

(3) Why would such a combination of your experience and the technology be desirable, or why would it not?

Comment 26

Again, ‘desirable’ invokes the idea of enhancement and performance and can be linked to the notion of design solution (and so design problem). See also Comments 7, 9, 10, 16 and 25. The issue is important, as it is central to evaluation of smell technology designs and so to all types of framework.

End Comment 26

Finally, the participants could express any other ideas for smell technology in a free-text field.

Part 6: Personal background

At the end of the questionnaire, participants answered questions on their socio-demographic and cultural background. The goal was to try to identify any geographical and cultural influences on smell attitudes (as found by Seo et al. [29]). The participants were also asked to assess their own smell sensitivity. All the questions, except for those on demographics, were mandatory. On average, the survey took 16 minutes to complete (SD = 7.57 minutes). Participants received US$ 1.50 for completing the questionnaire, corresponding to an hourly salary of 5.63 dollars.

Comment 27

It may surprise some, how short a time participants took to complete the questionnaire. Were they researchers too? If so, any particular reason or additional comment?

End Comment 27

Collected data and participants

A total of 554 participants began the questionnaire. Of these, 480 completed the questionnaire and answered three verification questions at its end. These questions required participants to describe the purpose of the study without being able to go back and look at the earlier questions or guidelines. After data cleaning, 41 stories were excluded. Fake entries (n = 11) were identified immediately, while repeated entries (n = 10), incomplete stories (unfinished sentences; n = 6), and incomprehensible stories (which did not make sense on their own; n = 14), were excluded iteratively throughout the coding process. This left us with 439 smell stories.

Comment 28

Is this par for the course? Is it about what was expected? Are there any implications for the conclusions?

End Comment 28

At the time of the study, all 439 participants (52.8% female) lived in the US; most had grown up in the US (95%). The participants’ age ranged from 18 to 67 years (M = 31.5, SD = 10.0). A majority of participants (84%) indicated being sensitive to smell (rating 4 or higher on a scale from 1 to 5).

Data analysis

The analysis process followed an open and exploratory coding approach [25]. Two researchers conducted the qualitative coding process. After coding an initial 25% of the stories, two more coding rounds (to reach 33% and then 50% of the data), led to the establishment of an agreed coding scheme. The coding scheme contained 10 main categories and 36 sub-categories, and a category entitled ‘not meaningful’ for cases where smell did not seem to have any relevance in the described experience. Based on this coding scheme, one researcher coded the remaining 50% of the data, and the second researcher coded a subsample of 25% of that data, resulting in a good inter-coder agreement (Cohen’s kappa of κ = .68) [12].

Comment 28

But as expected? Good enough (for what)? An additional comment would be informative here.

End Comment 28

Follow up design brainstorming

In addition to the feedback from our participants,

Comment 29

‘Feedback’ seems an odd choice of term to use here. Do you have any particular meaning in mind? Otherwise, ‘responses’ or ‘information’ might be better, since more neutral. It would stop the reader from unnecessarily looking for inappropriate, additional meanings, if none are intended.

End Comment 29

we also explored the design value of the smell stories with experts in the field. We organized a two-hour design brainstorming session with three HCI researchers, two working on smell technologies and one working on advanced interface and hardware design. None of them were from the same organization as the authors and none were familiar with the details of the study before the session.

The brainstorming session aimed to share and interpret the smell stories and followed four stages [11]: (1) prompting, (2) sharing, (3) selecting, and (4) committing. We selected 36 stories (one representative story for each sub category) as brainstorming prompts. All 36 stories were printed on A6 sheets (including the story title, the smell story, context information, and personal background). Each researcher was asked to read through the stories individually before discussing them together. They were asked the same questions as our participants (e.g., how they might imagine a connection between the experience and technology). Each researcher chose the most interesting/inspiring stories to share with the group, then they generated ideas as a group, and selected three to four ideas to be developed in more detail. The outcome of the brainstorming session is presented in the implication section, after the description of the findings from the smell stories.

Comment 30

This is an interesting method structure; but it would be helpful to have more rationale for the decisions taken. It is also essential to provide examples of the data for the reader to appreciate the smell stories and the brainstorming data, as you do later in the paper. After trying to answer the questions myself given to both groups, I feel more confident, as to the form of the actual data collected, after reading the examples.

End Comment 30

FINDINGS ON EXPERIENCES WITH SMELL

In the following sections we present our findings according to the 10 identified categories. The 439 smell stories were organized via their primary category, as agreed by the coders. This categorization does not define a strict line between the categories, as they are not wholly independent, but it does enable us to organize the material and generate a useful dataset for design.

Comment 31

Add criteria, here, for this claim. See also Comments 7, 9, 10, and 23.

End Comment 31

Below we provide for each category a rich description of the particularities of the stories, excerpts from example stories, and their associated smell qualities and emotions. Each category also contains information about the participants’ own rating of the stories as positive, negative, or ambivalent.

Category 1: Associating the past with a smell

This category is the largest and contains 157 stories. In these stories, the participants described a past experience in which a smell was encountered during a special event in life (e.g., ‘Wedding Day’, ‘New House’), at a special location (e.g., ‘The Smells of Paris’, ‘Grandma’s House’), or as part of a tradition (e.g., ‘The Smell of Thanksgiving’ or ‘Christmas Eve’). In these stories the smell was described as having a strong association to those particular moments in the past, with no actual smell stimulus in the present. A particularity of this category is the distinction between stories describing personal memorable events versus personal life events (e.g., ‘Disneyland’ versus ‘When my mother died’). Smells were also associated with personal achievement/success (e.g., ‘Scent of Published Book’, ‘New Car Smell’) and other important episodes of change, such as “‘Fresh Start’: I was taking a job in a new city. …. I took a plane trip across the country and the moment I took a step off the plane and took a deep breath will always stick with me. It felt so clean and the air actually smelled fresh and new” [#488]. Within this story, the qualities of the smell were for instance described as fresh, energetic, and invigorating. Some of the emotions experienced at this moment were courageousness and excitement. Although this category is dominated by positive experiences (n = 127), negative experiences were also reported (n = 27), such as ‘Car Crash’.

Category 2: Remembering through a smell

The 40 stories in this category described a recent experience of a smell, which reminded participants about past events, people, locations, or specific times in their life. In contrast to the previous category (where stories describe a direct link from the recollected past smell to the present; e.g., the smell of ‘Grandma’s House’), this category contains stories that describe an indirect link from the present experienced smell stimulus

Comment 32

‘Smell stimulus’ meaning is very clear here. See also Comments 1, 2, and 5 .

End Comment 32

to the past event, person or place (e.g., the smell of chocolate cookies as sudden reminder about grandma). Most stories in this category contain reminders of childhood described as ‘sweet’, ‘reassuring’ and ‘nostalgic’ with respect to the qualities of the smell. A sub-set of stories in this category (n = 10) also highlight the particular power of smells to take a person back in time. The description of such a flashback caused by a sudden smell stimulus was described as: “‘My first love’: It was the next day, when I was walking through the local Macy’s that I smelled something that threw me back into that situation, I could feel and see everything that had happened the day before when I smelled a perfume in the store” [#630]. Some of the qualities used to describe the smell were attractive, erotic, and fresh. The experienced emotions were described as amorous, aroused, excited, hopeful, and interested. The stories in this category were mainly positive (n = 37), except for three.

Comment 33

It would be useful somewhere to provide the criteria, by which a description was characterised as an ’emotion’. The latter is notoriously difficult to pin down. See also Comments 18 and 21.

End Comment 33

Category 3: Smell perceived as stimulating

The 62 stories in this category described experiences with a unique, mostly unknown smell (all stories, except one, were positive). The smells arose from different sources, such as perfume, food, and nature. A particularity of this category is the quality of ‘first time’ encounters with a smell across all origins. One participant described the first time he was at a beach: “The smell was very different from anything I had ever experienced before. At first I was kind of grossed out by the smell, but I grew to love it” [#921]. Another participant described the smell of a tornado experienced for the first time: “It was similar to the smell before rain but had a certain sharpness to it, as if to warn of the incoming danger. I felt like I knew this smell but at the same time, it felt foreign to me. It wasn’t a bad smell, it was just slightly unfamiliar” [#713]. The smell qualities and experienced emotions were often described with mixed attributes (e.g., heavy, imitating, and stimulating; attentive, serious, and calm), but still rated as positive experiences by participants. Most of the other stories in this category reported on the first experiences with food (e.g., ‘Slice of Heaven’) and nature (e.g., ‘Grass’), and were described as desirable, fresh, or pure, and provoked feelings of happiness at the moment they occurred. Although specific memories were established, including unique new associations (e.g., ‘Tornado smell’), the stories in this category did not evoke the kind of strong connections to the past as described in Category 1 and 2.

Category 4: Smell creating desire for more

This category contains 48 stories (45 positive). Key to these stories is that the smell grabbed the persons’ attention unexpectedly. The smell was either associated with food (triggering appetite), nature (triggering curiosity), or the scent of other people (triggering attractiveness), which motivated one to do or get something. In some stories smell was described in relation to the sensation of newness (e.g.,‘The sweet smell of CPU’: …There was the smell of the cardboard boxes it all arrived in, the smell of new metal– perhaps it was a combination of these and other things, but when the building was complete there was just a singular smell that was unique to a new computer built by my own hands” [#685]). The qualities of the smell in this story included beneficial, heavy, sophisticated, energetic, and pleasantly surprising. The experienced excitement was expressed through words such as confident, delighted, enthusiastic, impressed, or triumphant. This category also contained one story where the smell at a funeral stimulated reflection in the moment (e.g., ‘The scent of moving on’). The story was rated as a positive experience and at the same time the smell was described as clean, penetrating, and persistent, and the participant indicated that she was afraid, anxious, discontented, sad, tired, and uncomfortable. Despite the negative situation described in this story, the smell gave hope and a desire to live and move on, looking into the future in contrast to the stories in Category 1 and 2.

Category 5: Smell allowing identification and detection

This category captures the enabling role of smell in certain situations, such as allowing one to identify or detect a smell (e.g., “‘Gas leak’: I was cooking something on a gas stove and went out for a few minutes. When I came back, the fire was extinguished but the gas was still on. My roommate was sat at the table doing schoolwork, completely oblivious to the poisonous gas that was filling the room. I told him to get the hell up and open the windows and doors” [#951]). The qualities used to describe the smell were distinguished, penetrating, dirty, and light. The emotions related to this situation were described as anxious, conscientious, confident, and serious. Although the category is rather small (n = 11), the lesson to be learnt from the shared stories was the immediacy of the smell, allowing the participant to act or prevent something.

Category 6: Overwhelming power of smell

This category includes 37 stories where the smell overwhelmed the person in a positive way (n = 5; e.g., ‘The Chocolate Factory’) but predominately in a negative way (n = 30; e.g., ‘The Smell of Death’). In the latter case, people described the smell as something disturbing, as something that hit them suddenly on their way or during an activity. A subset of the stories was recounted as traumatizing, so that the person could still vividly remember the particular moment in the past although years have passed and no recent similar smell stimulation had occurred unlike in Category 2 (e.g., “‘Visit to a local county jail’: My guide warned me ahead of the time that it was going to be a little foul in there, but nothing could have prepared me for the obscenely acrid stench of hundreds of men crammed into every available space of the jail, right down to windowless storage rooms converted into more cells. … For days afterwards, I couldn’t shake the smell…. There weren’t enough showers to take it away. It’s been several years since then, and my memory of that smell is just a strong as ever” [#604]). In this category, the qualities of the smell were described as heavy, penetrating, dirty, or sickening.

Amongst others, the experienced emotions were described as alarmed, anxious, distressed, frustrated, or uncomfortable. In contrast to Category 1 and 2 (where the smell was associated with an event from the past or triggered a specific memory), Category 6 is about the smell as such during the experience and not about the memory associated with this smell. As opposed to the first two categories, in most stories forgetting – not remembering – the smell was the key element.

Category 7: Smell invading private and public spaces

All the stories in this category (n = 32) described an experience where one could not get rid of the smell. The smell invaded private and public spaces. In contrast to the previous category, the smell entered the person’s personal space (the person did not enter the space where the smell already existed) and took over the space. The loss of control over the smell was linked to the lingering quality of the smell (e.g., “‘Don’t want to smell that twice!’: I woke up one morning suddenly confused and was hit with an odor so horrible I couldn’t figure out what it was. … It was not like the smell you get a whiff of when a skunk stinks up the outdoors” [#530]). In the story the power of the smell, causing them to leave the house for several hours, was described with qualities such as foul, nauseous, penetrating, and persistent. One of the experienced emotions was surprisingly ‘amused’, however it was overruled by other emotions including annoyed, anxious, disgusted, taken aback, and uncomfortable. Despite the glimpse of humor in some stories, this category mainly contains negative experiences and underlines the power of the smell with its sudden and lingering qualities.

Category 8: Social interaction is affected by the smell

Within this category, smell was related to a person’s own smell or to the smell of others. Smell negatively affected the interaction among people and their togetherness (e.g., “‘Dragon breath teacher’: Once a teacher yelled at me during class. She got so close up into my face that I could smell her bad breath. This made the experience much worse because I wanted to get up and walk away but she was grabbing me to keep me focused on her while she was talking” [#744]). The smell qualities were described as nauseous, penetrating, and sickening, and caused negative emotions experienced as bitter, distressed, or insulted.

Despite frequent interactions among people, this category only contains 11 stories. This set of stories (overall negative experiences, apart from two) contains interesting elements with respect to a person’s own awareness of body smell andthe overbearing effect of other peoples’ smell on one’s comfort.

Category 9: Smell changes mood, attitude and behavior

This category contains 23 stories, which underlined the power of smells to change a person’s mood, attitude, or behavior. Stories reported the active regulating effect of smells with respect to mood, but mostly (n = 14) the change of behavior due to smells (e.g., ‘Accidental vegetarian’ or ‘Saved by the Smell!’). One story showed the active usage of smells to change one’s mood. A participant had recently been divorced and reported on the day her husband had moved out: “‘White Lilac Sheets’: “I made the bed with my lilac sheets and the atmosphere changed. I still remember that scent and how I felt on that day. I was going to be okay. The room didn’t look or feel or smell lonely anymore. It looked and smelled fresh and clean and lovely and a bit romantic and it was mine” [#526]. The qualities of the smell were described as fresh, reassuring, and spring-like, while the experienced emotions were determined, hopeful, longing, tense, but also worried. Overall, the stories in this category were reported as mainly positive (n = 12) experiences, but also as negative (n = 7) and four stories were rated ambivalent, neither positive nor negative.

Category 10: Smell builds up and changes expectations

This category shows the potential of smell to build up expectations and to surprise. In the former case (11 stories) the smell was building up expectations until the actual contact with the trigger, such as food or a perfume (e.g., “‘The Smell of Hungry Anticipation’: “I was trying a new soup for the first time. When it was brought to the table, the soft smell of rosemary immediately hit my nostrils. …It complimented the taste of the soup and built anticipation” [#585]). The smell was described as mouthwatering, healthy, and pleasantly surprising, and was further related to emotions such as conscientious, expectant, and relaxed.

In other stories (n = 7), expectations were exceeded to the extent that they surprised and diverted anticipations (e.g., ‘PomVinegar Surprise’: “I could smell the pomegranate and vinegar from about 10 steps away, and it was a very pungent (thought not unpleasant) odor. I almost felt my nose becoming runny and took out a tissue. When I tasted the dish, however, the taste wasn’t nearly as sour as I expected it to be from the smell” [#542]). The smell in this story was described as distinguished and penetrating, and was associated with emotions such as attentive and excited. Key quantitative facts behind the smell stories While the above-described categories can be used as an inspiration and as a starting point for exploring design opportunities for smell in HCI,

Comment 34

This claim should be made earlier in the paper, as well as here, so as to avoid the reader considering other design support options to be the paper’s aim. See also Comments 7, 9, and 10.

In addition, how do the researchers consider that designers might carry forward their work in practice for the purposes proposed? Designers tend to start with problems or solutions (theirs or others’). Here, however, the starting point for ‘inspiration’ or ‘exploration’ is an extensive and complex set of data, for which corroborative evidence has yet to be established. The point here is not that, it cannot be done; but rather how it might be done. See also Concluding Comment.

End Comment 34

our quantitative data provides additional background information. Below, the key quantitative information across all the collected smell stories is summarized. The majority of the 439 collected stories were positively valenced (n = 296), 112 were negative, and 31 were ambivalent. On average, negative stories tended to be slightly longer (M = 90 words) than positive stories (M = 79 words), but the difference is statistically not significant (U = 14600, p = .063, r = .09).

Contextual information

Most stories occurred in a context where one or more familiar persons were present (64.2%) or where participants were alone (21.6%). The presence of one or more strangers was reported less frequently (8.7%). With regard to location, most of the experiences happened at the participant’s or a friend’s home (38.1%) or in a public building (20.7%). Quite a few participants reported that their experience took place in the streets or another public space (14.4%), in a natural setting (7.3%), or at work (6.4%). The remaining participants (13.2%) indicated other places (e.g., stranger’s home). On average the reported experiences occurred 8.7 years ago (SD = 10.3), rangingfrom 1 day to 58 years ago.

The qualities of smell

The most frequent smell qualities reported in positive stories were pleasant (60%), fresh (42%), sweet (38%), clean (31%), and mouthwatering (30%). Smells in negative stories were described as unpleasant (62%), penetrating (55%), heavy (54%), foul (53%), and nauseous (51%). In ambivalent stories the smell was perceived as fresh (39%), pleasant (32%), mouthwatering (32%), attractive (26%), and penetrating (23%).

Experienced emotions

When asked to describe how they felt during their experience, participants’ used the affective terms happy (63%), pleased (53%), joyous (42%), delighted (41%), and excited (39%) in positive stories and uncomfortable (55%), disgusted (51%), distressed (43%), miserable (42%), and taken aback (29%) in negative stories. Ambivalent experiences were most frequently described as happy (42%), excited (39%), enthusiastic (35%), joyous (32%), and serene (29%).

An overview of all 10 categories and 36 sub-categories including qualitative and quantitative information (including a full example for each sub-category, used in the design brainstorming session) is provided as supplementary material. All smell stories and related qualities of smell, experienced emotions, and context information, are also available at www.multisensory.info for further exploration.

IMPLICATIONS FOR TECHNOLOGY

This study focused first on experiences and second on the implications for technology. This section turns to technology. Below we summarize the feedback from the participants on how technology would fit with their experience, and describe input from a brainstorming session with HCI researchers working on smell technologies and advanced interface and hardware design, based on a sub-set of the smell stories (one from each sub-category).

Ideas for technology from participants

Below we summarize the six areas and ideas for desirable future smell technologies mentioned by participants in Part 5 of the questionnaire:

(1) To share smells with family/friends: allow one to participate in a family event through remote smelling; share smells of special moments such as the smell of a newborn baby with distant relatives; share smells with people who they know would appreciate it (such as through social media); allow capturing and sharing of smells to create a common understanding where you can’t explain it. Participants also desired to be able to design and share new smells from a personal database and create a personal smell box/bottle.

(2) To support decision making: use smells for a quick judgment in online shopping (like/dislike can be determined easily); create smell profiles about holiday places and travel destinations; smell match maker in dating websites for allowing a better decision making about going on a date or not with a person (smell enhanced profiles).

(3) To regulate mood actively/passively: smell to relive good experiences whenever you want to get in a better mood; to calm yourself down in stressful moments such as in traffic jam or at work; as a reminder of past memories you would have forgotten otherwise but that can cheer you up when you feel depressed and life seems too difficult.

(4) To combine with other technologies and activities: integrate smell into radio; combine smell with music such as with ‘soundhound’ or ‘shazam’ apps; smell-enhanced advertisement on TV (for food channels); enhance visits of concerts, theater and performances with smell; allow underwater smelling when diving.

(5) To combine with everyday objects: enhance wristband with smell for keeping a preferred perfume lingering; have special glasses to see and smell the beach; smell-enhanced jewelry and clothes. One participant imagined her wedding ring enhanced with the smell of that day.

(6) To make oneself and others aware about body smell: to avoid embarrassing moments; provide invisible cues to a person about her/his smell level; quick smell check after sporting activities.

Comment 35

All 6 ideas are interesting. Of course, what would be welcome next is a set of design case-studies, instantiating the ideas, which diagnose design problems and prescribe design solutions, involving smell technology. See also Concluding Comment.

End Comment 35

The first idea matches the experiences in Categories 1 and 2, where particular events/moments in life are associated with a smell. The desire for capturing and sharing these experiences enhanced with smell becomes prevalent and suggests design implications for real-time smell-enhanced technologies (e.g., mobile phone, photo or video camera).

The second idea can be linked to Category 5, allowing people to identify and detect a smell. Moreover, smells are seen as very powerful for supporting quick decision-making (e.g., smell-enhanced website navigation and searching).

The third idea shows a direct link to Category 9 and the potential of smell to change mood. Interestingly participants whose story was in Category 1 or 2 were wishing for the possibility to capture pleasant smells, for instance from their childhood, and released to them in the present. This desire for smell-enhanced technologies or products is also apparent in the fourth and fifth ideas, where technology, objects, or even activities can be enhanced through smells from the past, or actual smells sourced through nature (e.g., diving in the ocean).

Finally, the sixth idea is linked to Category 8, aiming to avoid embarrassing moments in social interactions.

Participants also expressed concerns about future smell technologies. They were concerned about the possible misuse of smell when sharing it through the Internet or mobile phones (e.g., teasing people with smells, how to trust a smell message), and about the potential manipulation through smell (e.g., TV ads, online shopping). Some participants were also afraid to get sick, catch an allergy, or become addicted if they are exposed to chemical stimulations from technology. Finally, one participant raised the question of copyright and ownership of smells (e.g., ‘can I share others’ smells?’).

Comment 36

What is the thrust exactly of matching ideas to categories? If it is in support of design inspiration and exploration, then this should be made explicit (and an example should be provided to illustrate the relationship). See also Comment 35 and Concluding Comment.

End Comment 36

Ideas for technology from HCI researchers

Below we outline the ideas that emerged from the two-hour brainstorming session prompted by 36 smell stories (one from each sub-category). Four groups of design ideas emerged from this session and are described below:

(1) Smell-enhanced performance regulator: a technology stimulating smell in order to structure the day, taking activities and moods into account, and combining different smells to avoid habituation (training and evaluation phase needed). Smell as a reminder to take a break or as motivation to keep going a bit longer to meet the deadline [inspired by #526 ‘White Lilac Sheets’, Category 9].

(2) Autonomous smell agent: a technology spreading ambient cues (e.g., a robot) to guide someone to a certain place, to build up expectations, and motivate action. Smell trails in the environment can also make hidden information accessible, for instance, before entering a room (e.g., smell warning: tense working atmosphere) [inspired by #801 ‘Don’t forget to check your gas stove before you leave the house’, Category 5].

(3) Reminder alert with smell: a technology to remind us about important events, birthdays, and appointments. Although we have reminders on mobile phones and computers, they are often ignored, snoozed or in the worst case forgotten about. A smell can provide a pleasant reminder to say ‘it is time to call your mom’ by presenting the smell of your favorite dish your mom makes for you. On the other hand, if more critical, bad smells can be very powerful as a reminder and are not easily ignored [inspired by #530 ‘Don’t want to smell that twice’, Category 7].

(4) Smell-enhanced storytelling: a technology that stimulates storytelling around a digitized version of an incense stick. A stick was imagined with different layers, representing smells related to a loved person who passed away. When friends or family members come together, for instance at an anniversary year, they can add new smells to be shared in the group and thus trigger new stories about the dead person. It is as if looking through a photo album, telling the stories from the past, and using the smells as anchor points for keeping the memory alive [inspired by #672 ‘The Scent of Moving On’, Category 4]. We saw that the smell stories, even if they only provided limited information (story, story title, context, gender, and age), triggered vivid discussions, created empathy, and stimulated the sharing of personal smell experiences.

The four ideas described above provide a starting point for exploring smell in HCI. The categorization along with additional background information on smell qualities and experienced emotions (see supplementary material) can inspire further explorations of smell technologies.

Comment 37

Maybe here would be a good place to underline the status of the data with respect to its intended use – see also Comments 18, 21, and 22.

End Comment 37

DISCUSSION

Our findings about experiences with smell in combination

with the ideas for technologies just presented show several

design opportunities for smell. Below we do not provide

solutions for smell-enhanced technology designs, rather we

illustrate where our findings might be relevant to stimulate

novel designs in existing areas of interest within HCI. We

see three anchor points for smell-enhanced technology.

First, the smell stories in Categories 1 and 2 suggest design

opportunities for remembering and recalling the past. Our

findings might enrich ongoing research on the design for

personal memories. Apart from enhancing research

supporting the capturing and sharing of personal

experiences (e.g., in family relationships [18]) through

smell, our findings support research to support people who

are living with memory loss (e.g., patients with dementia

[32]), where smell can play an important role in

remembering the past. An increasing body of research also

explores the potential of digital technologies to support our

memory in everyday tasks (e.g., reminder systems), to

recall past events and experiences (e.g., life-logging tools),

to design end-of-life technologies allowing reminiscence of

passed away people [21], and to record and reproduce

smells [35]. All this research shows the potential of smell to

enrich experiences, for instance by enhancing personal

memories such as photos or videos with smell. Based on

their study of a smell-based photo-tagging system, Brewster

et al. [5] stated that participants asked for personal smells to

be added. The information on how to classify such smells

was still missing; the present analysis allows us to relate

smell qualities to particular types of experiences.

When designing with smell, as for any memory-based

technology, access to such memories has to be considered

carefully to preserve their uniqueness. One participant

wrote: “I could see it being desirable in that it would allow

me to experience the scent whenever I want, but it’s kind of

a two edged sword in that experiencing that scent time and

time again will make it common place” [#513]. The power

of smell might not persist if always available, thus

participants suggested to either restrict the access and

retrieval of smells to special times (e.g., at ‘grandmas

birthday’) or to link them to a certain social setting (e.g.,

smelling only in company with ‘your sister’). This way the

uniqueness of the smell can be preserved.

Second, the stories in Categories 3 to 8 as well as 10 draw

the attention away from past memories and suggest design

opportunities for the present moment. Designing for in-situ

stimulation, the ability to capture and share smells in the

moment, and the capability to mask and neutralize bad

smells creates a vast space for smell interaction design. One

suggestion made by participants was the combination of

smell and social media, such as “An app that would allow

me to store smells, send smells, or attach smells to a picture

that I could post on social media or Instagram or

something”. This supports existing research on the delivery

of smells through the Internet [24]. We draw attention to

three additional design directions concerned with (1) first

time experiences with smell, (2) the power of smell to build

up expectations, and (3) the potential of designing for bad

smells. User experience designers put a lot of effort into

designing ‘out-of-the-box’ and first time experiences to

create positive experiences [13]. Our categorization not

only provides designers with rich descriptions of such first

time experiences, but also describes the related qualities of

smells in combination with descriptors of the experienced

emotions. This can be used to stimulate positive smellenhanced

experiences with technology, build up

expectations, and create anticipation as studied within

experience research [33]. Typically this anticipation stage is

influenced by a variety of aspects (e.g., advertisements,

product descriptions, accounts from existing customers).

Smell stories in Category 10 provide evidence for the

power of smell to build up expectations, create surprise by

exceeding anticipated experiences, and enhance momentary

experiences through capturing and sharing pleasant smells.

Categories 6 to 8 contain stories about bad smells, which

are wished to be neutralized or masked to change the

experience from something negative to something positive.

While the idea of outbalancing smells seems to be

desirable, the design brainstorming session stimulated a

discussion on the usage of bad smell in design, particularly

as part of the design idea (3) Reminder alert with smell.

Designing for bad smells might not seem appropriate at

first, but through intensity manipulation it can open up an

interesting space for design. Similar to a snooze function,

which slowly increases volume, smell stimulations could be

added to certain events (e.g., reminder for mother’s

birthday). Starting with a pleasant smell, it could turn

slowly into something unpleasant if you did not act.

Category 8 also contained stories recounting social

experiences with smell, where the smell of a person or of

other people caused embarrassment or discomfort. Despite

the importance and frequency of social contact in everyday

life, few such stories were shared. They might not seem

meaningful enough to be memorable or to be shared. Yet,

this set of stories holds potential for personal and social

smell-enhanced awareness systems, as well as for wearable

technologies, and smart fabrics. Technology could, as stated

by a participant, “…make the people in those settings feel

more comfortable if I interact with them… My holding my

nose could be insulting and impede communication.”

Third, the smell stories in Category 9 suggest design

opportunities reaching out to the future through positive

stimulation, with potential relevance for wellbeing and

behavior change research in HCI. The stories shared in this

category were about the power of smell to regulate mood,

change attitudes, and behavior. Designing for smell could

be combined with behavior change research in HCI (e.g.,

tools to support healthy nutrition and diet), and thought of

in relation to positive psychology and research on

wellbeing. Seligman and Csikszentmihalyi [28] suggested

that happiness can be learned and cultivated and that

positive psychology can help change how a person feels.

They point to the power of positive emotions for our health,

happiness and wholeness. We would suggest that our

findings add an understanding of the positive emotional

impact of smells that might be a valuable research strategy

in wellbeing research (e.g., for regulating mood).

Comment 38

For this sort of application, one would need to have more confidence in the truth/correctness of the claims of the participants, collected as data – see also Comments 18, 21, 22, and Concluding Comment.

Smell can have a regulating impact on a person’s mood and can, as in

one case explicitly reported, be used to regulate mood

(‘White Lilac Sheets’). The participant wrote, “I guess the

experience could have been enhanced by some kind of

mood moderator. Something that would have sensed my

sadness and filled the room or house with comforting

scents” [#526]. The participant pointed out that technology

would not change the situation to something more positive,

as it just was not a happy time at all, but that it could

support the sad moments in this transitional period of life.

Limitations

We would like to acknowledge three limitations of this

work. First, by using Amazon Mechanical Turk for

recruiting and asking participants to describe personal

relevant experiences, we were limited to the US and do not

know to what extent the smell stories are representative of

more general experiences with smell. We are aware about

cultural and geographical differences (as described by Seo

et al. [29]), which require further studies with a more

diverse group of participants. Second, collecting narratives

by means of an online questionnaire has an influence on

how people narrate their experience and deprives us of the

advantages of an interview situation where we can engage

in a dialogue with the participant to explore the meaning

behind the shared experience in more depth as described by

Bruner [6]. We tried to collect information beyond the

initial trigger of the shared smell stories in order to allow

the establishment of meaningful categorizations and the

creation of a basic understanding of experiences with smell

in HCI.

Comment 39

For issues concerning ‘understanding’ – see Comments 3, 8 and 13 .

Third, our approach provides an overview on the

emerging field of smell-enhanced technologies. Future

studies will, we hope, lead to in-depth research into

experiences with smell inspired by our identified categories.

Comment 40

Such research would also do well to specify one or more design problems and explore the design solutions that smell might inform. In this way might the usefulness of these smell data be used beyond ‘inspiration’ and ‘starting point’ for HCI exploration. See also Comments 7, 9, 10, 34 and 35.

CONCLUSIONS

Despite interactive technologies increasingly disappearing

into our environment (in ubiquitous and mobile computing)

and becoming essential in everyday life, the senses used to

interact with technology are still limited.

Comment 41

Presumably in number and not in terms of the percentage of any given modality needed for interaction across the whole of HCI?

We have discussed the opportunities for smell in HCI based on an

analysis of 439 smell stories. We identified 10 primary

categories for stories about experiences with smell, which

help discuss the potential implications for technology.

Implications were drawn from feedback from our

participants envisaging desired connections between their

own personal experience and future smell technology. The

implications for designing for smell were further enriched

through ideas from an initial brainstorming session with

HCI researchers. Our findings provide guidance for smell

enhanced technology design, not only giving a

categorization of the role of smell in personal experiences,

but also extracting the qualities of smell across the smell

stories and the experienced emotions. We argue that this

research enriches existing technology driven research on

smell in HCI and provides a fruitful starting point when

designing for experiences with smell.

Concluding Comment

1. The paper is interesting and certainly opens up a novel modality for interaction as part of HCI. To what extent it actually provides inspiration and a starting point for HCI exploration of the potential for smell remains to be seen.

2. It would be interesting to know how this research could be carried forward, either by the authors or others, to evaluate the support provided by the data, the categories and the ideas for the design activities identified. There follow some suggestions in this respect.

3. Stories are created, as well as told. One way forward for the research, then, is to provide conditions, under which participants create their own stories. Smells from the framework could be generated to test that the experience, which results, corroborates (or not) the findings, incorporated into the framework. This would be an initial move towards validating the framework.

4. Stories can also be acted out, as well as told and created. Another way forward, then, is for researchers to generate smells, which are actually used by participants in some form of interactive behaviour. Again, this would constitute a further move in validating the framework.

5. Finally, stories could be told, created and acted out in the form of interactive behaviour, involved in the diagnosis of design problems and design solutions. For example, we know that smell can serve as an alarm (smoke for fire; bad odour for food, which is ‘off’). Elsewhere, other modalities are used in HCI, for example, vision for misspellings on a computer and audition for newly arrived e-mails. The challenge is to use smell, as a modality, to solve the generic HCI ‘alarm design problem’. In this way, might stories be developed and built on to progress the inclusion of smell in HCI.

ACKNOWLEDGMENTS

This work is supported by the Marie Curie IEF Action of

the European Union (FP7-PEOPLE-2010-IEF) and the

Swiss National Science Foundation (PBBSP1 144196). We

thank our participants and especially Annika Haas for her

valuable support in designing the supplementary material.

REFERENCES

1. Alaoui-Ismaïli, O., Robin, O., Rada, H., Dittmar, A.,

Vernet-Maury, E. Basic Emotions Evoked by Odorants:

Comparison Between Autonomic Responses and Self-

Evaluation. Physiology & Beh. 62, 4 (1997), 713–720.

2. Bakalar, N. Sensory science: Partners in flavour. Nature

486, 7403 (2012), S4–S5.

3. Baumeister, R.F., Newman, L.S. How Stories Make

Sense of Personal Experiences: Motives that Shape

Autobiographical Narratives. Personality and Social

Psychology Bulletin 20, 6 (1994), 676–690.

4. Bodnar, A., Corbett, R., Nekrasovski, D. AROMA:

ambient awareness through olfaction in a messaging

application. Proc. ICMI, (2004), 183–190.

5. Brewster, S., McGookin, D., Miller, C. Olfoto:

designing a smell-based interaction. Proc. CHI (2006),

653–662.

6. Bruner, J. Life as narrative. Social Research: An

International Quarterly 71, 3 (2004), 691–710.

7. Chrea, C., Grandjean, D., Delplanque, S., Cayeux, I., Le

Calvé, B., Aymard, L., Velazco, MI., Sander, D.,

Scherer, KR. Mapping the Semantic Space for the

Subjective Experience of Emotional Responses to

Odors. Chem. Senses 34, 1 (2009), 49–62.

8. Ehrlichman, H., Bastone, L. The use of odour in the

study of emotion. In S. Van and G.H. Dodd, (Eds.),

Fragrance: The psychology and biology of perfume.

Elsevier, (1992), 143–159.

9. Emsenhuber, B. Scent Marketing: Making Olfactory

Advertising Pervasive. In J. Müller, F. Alt and D.

Michelis (Eds.), Pervasive Advertising. Springer,

(2011), 343–360.

10. Erickson, T. Design as storytelling. Interactions 3, 4

(1996), 30–35.

11. Faste, H., Rachmel, N., Essary, R., Sheehan, E.

Brainstorm, Chainstorm, Cheatstorm, Tweetstorm: new

ideation strategies for distributed HCI design. Proc. CHI

(2013), 1343–1352.

12. Fleiss, J.L., Levin, B., Paik, M.C. Statistical Methods

for Rates and Proportions. John Wiley & Sons, (2013).

13. Ghinea, G., Ademoye, O. The sweet smell of success:

Enhancing multimedia applications with olfaction.

TOMCCAP 8, 1 (2012), 2.

14. Hassenzahl, M. Experience Design: Technology for All

the Right Reasons. Syn. Lect. on HCI 3, 1 (2010), 1–95.

15. Herz, R.S., Engen, T. Odor memory: review and

analysis. Psy. Bulletin & Review. 3, 3 (1996), 300–313.

16. Jonsson, F., Verhagen, H. Senses working overtime: on

sensuous experiences and public computer game play.

Proc. ACE (2011), 56:1–56:8.

17. Kaye, J. “Jofish.” Making Scents: aromatic output for

HCI. Interactions 11, 1 (2004), 48–61.

18. Kazakos, K., Howard, S., Vetere, F. Revisiting the

relationship between reunion and technology-mediated

separation in periodically transitioning families. Proc.

CSCW (2013), 1157–1168.

19. Loumakis, F., Stumpf, S., Grayson, D. This image

smells good: effects of image information scent in

search engine results pages. CIKM (2011), 475–484.

20. Mason, W., Suri, S. Conducting behavioral research on

Amazon’s Mechanical Turk. Behavior Research

Methods 44, 1 (2012), 1–23.

21. Massimi, M., Moncur, W., Odom, W., Banks, R., Kirk,

D. Memento mori: technology design for the end of life.

Proc. CHI EA (2012), 2759–2762.

22. Matsukura, H., Yoneda, T., Ishida, H. Smelling Screen:

Development and Evaluation of an Olfactory Display

System for Presenting a Virtual Odor Source. IEEE

TVCG 19, 4 (2013), 606–615.

23. Petitmengin, C. Describing one’s subjective experience

in the second person: An interview method for the

science of consciousness. Phenomenology and the

Cognitive Sciences 5, 3-4 (2006), 229–269.

24. Ranasinghe, N., Karunanayaka, K., Cheok, A.D.,

Fernando, O.N.N., Nii, H., Gopalakrishnakone, P.

Digital taste and smell communication. Proc. BodyNets,

(2011), 78–84.

25. Saldana, J. The Coding Manual for Qualitative

Researchers. SAGE (2012).

26. Scherer, K.R. Appraisal considered as a process of

multilevel sequential checking. Appraisal processes in

emotion: Theory, methods, research 92, (2001), 120.

27. Scherer, K.R. What are emotions? And how can they be

measured? Social Science Inf. 44, 4 (2005), 695–729.

28. Seligman, M.E.P., Csikszentmihalyi, M. Positive

psychology: An intro. Am. Psy. 55, 1(2000), 5–14.

29. Seo, H.-S., Guarneros, M., Hudson, R., et al. Attitudes

toward Olfaction: A Cross-regional Study. Chem.

Senses 36, 2 (2011), 177–187.

30. Suddendorf, T., Corballis, M.C. The evolution of

foresight: What is mental time travel, and is it unique to

humans? Beh. & Brain Sciences 30, 3 (2007), 299–313.

31. Tuch, A.N., Trusell, R., Hornbæk, K. Analyzing users’

narratives to understand experience with interactive

products. Proc. CHI (2013), 2079–2088.

32. Wallace, J., Wright, P.C., McCarthy, J., Green, D.P.,

Thomas, J., Olivier, P. A design-led inquiry into

personhood in dementia. Proc. CHI (2013), 2617–2626.

33. Willander, J., Larsson, M. Smell your way back to

childhood: Autobiographical odor memory. Psy.

Bulletin & Review 13, 2 (2006), 240–244.

34. Wright, P., McCarthy, J. Experience-Centered Design:

Designers, Users, and Communities in Dialogue. Syn.

Lect. on HCI 3, 1 (2010), 1–123.

35. Wyszynski, B., Yamanaka, T., Nakamoto, T. Recording

and reproducing citrus flavors using odor recorder.

Sensors&Actuators B: Chemical 106, 1(2005), 388–393.

[/expand]

Temporal, Affective, and Embodied Characteristics of

Taste Experiences: A Framework for Design

Marianna Obrist1,2, Rob Comber1, Sriram Subramanian3, Betina Piqueras-Fiszman4, Carlos Velasco4, Charles Spence4

m.obrist@sussex.ac.uk | rob.comber@ncl.ac.uk | sriram@cs.bris.ac.uk |

betina.piqueras-fiszman@psy.ox.ac.uk | carlos.velasco@psy.ox.ac.uk | charles.spence@psy.ox.ac.uk

1Culture Lab, School of Computing Science

Newcastle University, UK

2School of Engineering and Informatics

University of Sussex, UK

3Deptartment of Computer Science,

University of Bristol, UK

4Department of Experimental Psychology,

University of Oxford, UK

ABSTRACT

We present rich descriptions of taste experience through an

analysis of the diachronic and synchronic experiences of

each of the five basic taste qualities: sweet, sour, salt, bitter,

and umami. Our findings, based on a combination of user

experience evaluation techniques highlight three main

themes: temporality, affective reactions, and embodiment.

We present the taste characteristics as a framework for

design and discuss each taste in order to elucidate the

design qualities of individual taste experiences. These

findings add a semantic understanding of taste experiences,

their temporality enhanced through descriptions of the

affective reactions and embodiment that the five basic tastes

elicit. These findings are discussed on the basis of

established psychological and behavioral phenomena,

highlighting the potential for taste-enhanced design.

Author Keywords

Taste; user experience; taste experiences; sensory research;

explicitation interview technique; sensual evaluation tool.

ACM Classification Keywords

H.5.2 Information interfaces and presentation (e.g., HCI):

Miscellaneous.

INTRODUCTION

Experts in taste perception agree on at least five basic tastes

[40]. Beyond this, however, we lack insights into the rich

experience of these tastes. This lack of experiential

understanding

Comment 1

Understanding ‘experience’, as you pointed out in your seminar, is considered generally to be central to the concept of HCI as ‘User Experience’ (see Rogers, 2013). The strength of the concept lies in its inclusivity. Nothing concerning the user is excluded, unlike the more limited concept of ‘usability’. However, experience is a very general term and so needs better definition for it to be operationalised and tested, both of which are preliminary to generalisation – the ultimate aim of HCI research. It might be an idea, then, in future work to consider more exactly what you mean here by the experience, ‘of which there is a lack of understanding’.

‘Understanding (taste) experience’ in everyday language, that is most generally, means to identify with or to recognise someone else’s experience, as in agreeing with a friend’s assertion, that ‘bitter beer is too bitter, and so nasty’. This provides us all with ‘insights’ into our and others’ experiences. Is this the sort of understanding and insights you are seeking? If so, an analysis of the every day descriptors of taste experience, in terms of what can and cannot, be said about taste, might prove helpful (also to designers). If not, then further definition of the understanding and the insights, that you have in mind might prove useful. See also Comment 6 later.

Finally, ‘rich’ occurs in the paper in different contexts, for example, ‘rich experience’ and ‘rich description’. I am not sure that it adds much to your arguments; but if it is intended to, then you need to inform the reader a little more, as to the meaning and value, that you ascribe to it.

extends beyond HCI, as sensory researchers

have also acknowledged that: What is not well researched

is the link between the food that goes into our mouth and

what we think of it [12]. There is a growing interest in taste

within the HCI community [e.g., 16,17,18,22,27,28],

particularly relating to technical challenges in designing for

taste stimulation and one-off designs to enhance user

experiences through the manipulation of taste.

There is a need for a more systematic study of people’s

taste experiences and their specific characteristics in order

to make a fuller use of this sense in future taste-enhanced

technologies. This paper stands as a first step in addressing

this need. Drawing on neuroscience and sensory research in

combination with user experience evaluation techniques,

we investigated how all five basic tastes are experienced at

a given time (synchronic) and how they evolve over time

(diachronic). We used pure tastants (i.e., that have no smell

or visual qualities) with an explicitation interview technique

[41] designed to encourage the participants to verbalize

their experiences. Additionally, we used physical objects

from the Sensual Evaluation Instrument [13] to elicit

affective responses, and create a flexible, non-verbal

channel of communication between the user and designers.

This paper makes a number of contributions: First, we

provide a rich description of subjective taste experiences

along both the diachronic and synchronic characteristics of

the five basic tastes.

Comment 2

I think that you are correct to distinguish subjective taste experience from the verbal and non-verbal description of them. This distinction is important, when you come to constructing the framework for design – see Comments 13 and 14 and elsewhere.

This point is particularly relevant to the type of understanding and insights that you are seeking. See also Comments 1 and 6.

Second, these taste characteristics

establish a framework for taste experiences and elucidate

the potential design qualities of individual tastes.

Comment 3

The scope of the framework is, indeed, taste experiences. However, the framework is presumably one of descriptors – see also Comment 2. The distinction is important. Both taste experience and taste descriptors can give rise to (user) behaviours, which can be recruited to interacting with computers. The descriptor framework, however, can be used by designers – see also Comments 4 and 10.

Are ‘design qualities’ the same as descriptors or are they a subset of descriptors or indeed something else? The importance arises in the distinction between understanding (see Comment 1) and designing (see Comment 4).

We demonstrate how each quality can be described along three

main themes: temporality, affective reactions, and

embodiment. Third, our findings extend human-computer

interaction research on taste through a user experience

perspective. Overall, our findings provide interaction

designers and user experience researchers with a richer

understanding of taste experiences and their specific power

to influence human behavior and decision-making. The

framework presented here enables the HCI community to

think and talk about taste in the design of interactive

systems in a fine-grained manner.

Comment 4

This claim is an important one and appears to be well-justified by the research. However, it important to consider if and how the framework also constitutes a contribution to design knowledge and so contributes to the HCI discipline. Design knowledge here is understood to support the diagnosis of design problems and the prescription of design solutions. The transition from exploration/reflection to a notion of understanding and design, that can be built on by others in the search for generalisation, is at the heart of HCI discipline progress. There is a clear way forward, here, for the research.

RELATED WORK

This section provides an overview of the human sense of

taste and its relevance for HCI based on ongoing research.

The sense of taste

Sensory researchers and neuroscientists agree on five basic

tastes (sweet, sour, salty, bitter and umami), and a

‘gustotopic map’ linking these classes of receptors with

particular brain areas is currently being developed [40].

However, despite breakthroughs in understanding the sense

————————————————————————————————–

Permission to make digital or hard copies of all or part of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed

for profit or commercial advantage and that copies bear this notice and the full

citation on the first page. Copyrights for components of this work owned by others

than ACM must be honored. Abstracting with credit is permitted. To copy otherwise,

or republish, to post on servers or to redistribute to lists, requires prior specific

permission and/or a fee. Request permissions from Permissions@acm.org.

CHI 2014, April 26 – May 01 2014, Toronto, ON, Canada

http://dx.doi.org/10.1145/2556288.2557007

——————————————————————————————————-

of taste, scientists have still not approached the

phenomenology of taste nor developed a semantic

understanding of how taste is experienced [30].

Comment 5

Does ‘phenomenology’ here refer to the science of taste or the everyday description of taste or both? The same question also arises concerning ‘semantic understanding’. Is semantic here to be distinguished from syntactic and maybe even lexical (you seem to prefer the term ‘vocabulary’)? If so, we need to be told how these different levels (?) relate. The issue is linked to the different possible types of understanding, identified in Comment 1. Clarification, here, would be helpful to any researcher trying to replicate and extend the research.

Although a wide body of sensory research has studied the temporal

evolution of taste perception using labeled intensity scales

[e.g., 1,8] and more specific time-intensity sensory

evaluation scales [26], insights are limited to the

quantification of temporal responses to perceived taste

intensities. Such scale-based evaluations leave us

uninformed as to the subjective qualities that lie behind the

ratings of the perceived taste experience over time.

Recently, neuroscientists have studied taste-specific

temporal profiles by comparing sensory evaluation scales

with functional MRI (fMRI) data [19]. Their results

suggested that salty tastes change more rapidly than sweet

tastes in the cerebral cortex, and confirm the same patterns

that have been observed using time–intensity sensory

evaluation [19]. While such results are intriguing, they

cannot explain the differences in experienced tastes.

Comment 6

The aim of science is to understand natural phenomena. Understanding involves the explanation of existing phenomena in terms of theory and the use of theory to predict new phenomena. To be validated, theory needs to undergo the processes of: conceptualisation; operationalisation; test and generalisation. Maybe the results, to which you refer, indeed, cannot in this case explain the differences in experienced tastes. However, it must remain their goal to do so. Scientific understanding (here presumably the Psychology of taste or somesuch) is to be contrasted with the everyday meaning of understanding – see Comment 1.

The different meanings of understanding are critical for your research, as is made clear in your review request. First, because you want to contribute to our understanding. Second, because you want to use this understanding to support HCI design. To assess your own progress with respect to these aims and to allow others to build on and further your work, you need to make explicit the kind of understanding you have in mind – see also Comment 1.

To account for subjective differences, the ‘taster status’

measure has been introduced [2,6]. By means of such tests,

it is possible to identify participants’ subjective sensitivity

to bitter tastes and to distinguish between supertasters (25%

of population), medium tasters (50%), and non-tasters

(25%) [3]. Taster status has been considered to partially

explain why some consumers like certain foods more than

others and how they describe the way they experience them.

Comment 7

Concerning ‘explanation’ – see Comments 6 and 11.

Concerning descriptors and experience – see Comments 2 and 13.

Food-interaction design

The last few years have seen increasing interest in

designing human-food interaction in HCI [e.g., 4,9,11,33].

Such research looks to position human-food interaction

within the wider spectrum of social, environmental, and

physiological influences on our food practices. In this area,

there is a growing realization of the potential for new

technologies to support pleasurable experiences around

food [20,35], and the potential for designers to draw on the

extensive research on multisensory experiences (i.e.,

auditory, tactile, visual, olfactory, and gustatory). Despite

this increased interest in food experience, we know little

about the richness of people’s taste experiences. The

majority of the studies on food experience combine taste

with other modalities, where taste is but one component

[e.g., 18,27,28].

For instance, Schifferstein et al. [31] elicited emotional

experiences across the different stages of food product

usage, from choosing a product in the supermarket through

to cooking and eating [31]. Taste experience is interwoven

with vision, touch, and olfaction, which, in combination

create multisensory food experiences. Desmet and

Schifferstein [5] also explored the emotions elicited through

eating and tasting food. They describe variables related to

food-evoked emotions, such as sensory features, product

type, food-related activities, context, and the agent (who

consumes, prepares, or produces). Due to the wide range of

influencing variables, it is not clear how well these findings

translate beyond the specific context of their studies.

Taste-enhanced technology

Technological advances in creating taste stimulations

[27,28] and one-off applications exploiting taste in games

[16] and other scenarios [18,22] demonstrate a growing

interest in the use of taste in interactive applications. For

instance, Ranasinghe et al. [27,28] developed a tongue

interface that creates taste through the combination of

electrical and thermal stimulation. They use electrical

pulses applied to the tongue. Verbal descriptors provided by

participants were, for instance, a ‘refreshing taste’ or ‘minty

taste’ in relation to the change in temperature. The authors

call for future work to understand the particularities of such

taste (flavor) experiences.

Comment 8

Concerning ‘understanding – see Comments 1 and 11.

They focused on the introduction

of taste in digital communication to enhance long-distance

family relations and create remote co-presence and coliving

experiences (e.g., remote dining) [28].

Comment 9

By way of illustration, un-enhanced or ineffective long-distance family relations would be the design problem here and enhanced or effective relations, by means of taste, a design solution. Taste design knowledge (for example, your framework) would be the means of moving from the problem to the solution. See Comment 4 for how design might be conceived. There are, of course, many more ways; but you need to envisage at least one.

Murer et al. [16] designed a gustatory game device,

LOLLio, which consists of an interactive lollipop that

serves as a haptic input device that dynamically changes its

taste between sweet and sour. Remote triggering of taste

while motion sensing with accelerometers allows LOLLio

to be used as an input modality. The authors identify

various ways in which taste could be used in an interaction,

such as to provide reward or punishment or else to provide

hidden information through taste stimuli. LOLLio was

evaluated in a game context with children [17]. Sweetness

was constantly used in the game session and sour stimuli

were used in combination with game mechanics to provide

‘negative reinforcement’. Their findings suggest an

enhanced playing experience through taste stimulation

motivating further explorations of such taste-enhanced

interaction experiences.

Comment 10

For the purpose of design, interaction is best conceived as the interaction of human behaviours with computer behaviours (in both cases both physical and abstract (mediated by the physical)). Here, the human behaviours include taste perception and the consequent reactions and LOLLio’s behaviours, which change the taste from sweet to sour. An ‘enhanced playing experience’ arises by means of the human-technology interaction of physical behaviours, which may entail abstract behaviours. It is these behaviours, which can be designed and which affect the user’s experience. Your framework for design will need to make some assumptions, concerning the conceptualisation of interactions, whether these or others. As it stands, it is unclear how experience(s) are involved in interactions. See also Comments 4 and 9.

STUDY METHOD AND PROCEDURE

Sensory research provides important information regarding

the objective measures of taste perception, temporality, and

subjective sensitivity levels. Yet, an understanding of the

subjective understanding of taste experiences is missing.

Comment 11

The two types of ‘understanding’ here and their relationship with ‘experience’ need to be made explicit as well as their relationship to design (and experience). In brief, there is understanding as in science; understanding, as in everyday language; and design of interactive human and computer behaviours. The issues are addressed in more deatil in Comments 1, 2, 4, 9 and 10.

This study explores the diachronic and synchronic structure

(explained below) of each of the five basic tastes.

Methodology

For our study, we combine two verbal and non-verbal user

experience and elicitation methods, the explicitation

interview technique (verbal method) and the ‘Sensual

Evaluation Instrument’ (non-verbal method).

The explicitation interview technique [41] is used to elicit

verbalizations of subjective experiences. This technique

helps to explore the unfolding of an experience over time,

the ‘diachronic’ dimension, and examines the specific

facets of the experience at a particular moment, the

‘synchronic’ structure (see also [24,39]). The value of this

interview technique lies in helping participants to express

their experiences at a specific moment. Participants are

encouraged to talk about the experiential (cognitive,

perceptive, sensory, and affective) aspects of the moment

without building on rational comments and explanations

[24].

Comment 12

This particular set of experiential aspects of the moment need to be rationalised or grounded in some way – for example, so that they can be checked for completeness, coherence etc. Alternative aspects might be derived from the the three ontological primitives of ‘being’ (which might, for the purposes in hand, be equated with experience), which are: physical; psychological; and social. The associated aspects of: cognition (knowing); conation (trying); and affection (feeling) can then be related to each primitive. The key point here is that the aspects are grounded, so that other researchers are persuaded to use the same set. Such considerations might be useful in the development of your design support knowledge – Comments 20 and 21.

Questions related to the diachronic structure help to

understand how the description of an experience unfolds

over time (e.g. “What happened after you opened the

door?” and “What did you perceive next?”). With respect to

the synchronic structure of an experience, the participant is

questioned about a particular moment (e.g. “At the moment

when you pushed the handle down, how did it feel?” or

“What else came in your mind?”). In comparison to open

questioning approaches, this technique is non-inducive but

directive [24] in the sense that it keeps the participant

talking about the experience without inducing any content;

it focuses on the structure of the experience, and directive,

as it keeps the participant focused on the singular

experience being explored. Although it is typically used

retrospectively to support the reconstruction of an

experience, it has also been used in-situ (e.g., [15,23]).

The Sensual Evaluation Instrument (SEI) is a non-verbal

tool that can be used to elicit users’ affective reactions [13].

SEI is composed of sculpted objects that can be held in the

hand, used by a person to indicate how they are feeling as

they interact with a system. The SEI includes eight objects

with different shapes, which represent various levels of

arousal and valence (positive and negative). Isbister et al.

[13] describe SEI objects as evoking and expressing a range

of emotions; they do not claim a direct mapping between

the objects and the mentioned emotions,

Comment 13

The issue of the directness of mapping must also be raised with respect to the relationship between descriptors and experience (see Comment 2). It is, of course, a fundamental and difficult issue; but needs to be addressed here at least as concerns (for the purposes in hand) of supporting design – see also Comments 12, and 20.

but emphasize the

benefit of the objects for stimulating expressiveness. The

value of the SEI is to elicit real-time, affective responses,

and to create a flexible, non-verbal channel of communication

between user and designers. The latter defines a key

advantage compared to other methods that are often limited

to verbalizations or visualizations that lack physicality.

Taste stimuli

The stimuli used for each taste are specified in Table 1.

Each stimulus was prepared as an odorless and colorless

water solution using a stock solution as specified in ISO

3972. We prepared the solutions according to the

specifications detailed by Hoehl et al. [10] and used

deionized water for the tastants. These compounds

standardised stimulus features and controlled for sensory

differences, such as texture, vision, etc. All of the solutions

were prepared the day before each study day. The

participants received 20 ml of each stimulus in a disposable

40 ml cup. A Latin square design was used to avoid order

bias [42].

Table 1. Stimuli used for the five main tastes, including the

stock solution (indicating the threshold specified in ISO 3972).

Participants

The study was conducted with 20 participants (nine female)

aged between 21-38 years (M=29.4, SD=5). Participants

were recruited based on the following criteria: not having

any food allergies, being non-smokers, not being pregnant,

and not having any sensory dysfunction (e.g., dysguesia, a

taste disorder), by self-report. The participants were

recruited through the staff list within the lead university. 16

participants were native English speakers, and the

remaining four were fluent in English. All participants gave

informed consent prior to the study.

Study set up and procedure

The participants were instructed and reminded 2 days prior

to the study not to eat spicy food 24 hours before the study

and not to drink or eat 1 hour before attending the study.

The study had 2 parts (see Figure 1): In the first part, we

applied the explicitation interview technique for all five

tastes; in the second part we introduced the SEI objects to

enhance the verbalizations for each taste.

In the first part, participants were given 5 minutes per

stimulus. They could take as many sips as they wanted of

the stimulus and were prompted with specific questions

about their taste experience (e.g., Could you describe what

you perceive? How does it feel in your mouth?). The aim

was to receive insights regarding the diachronic and

synchronic structure of the taste experience. We used this

technique in-situ in order to account for the rapidly

decaying sensory memory trace related to the human sense

of taste [21]. Before continuing with the next stimulus, the

participants were asked to have a sip of the deionized water

in order to cleanse their mouth. The same procedure was

repeated for all stimuli.

In the second part of the study, the participants were

instructed to match each taste experience to one or more of

the eight shapes inside the box. The participants could only

feel, and not see, the objects, to exclude any visual

influences and to focus on the mapping between ‘taste and

shape’ via the sense of touch. The participants were

instructed to select one or more or none of the shapes (they

could also reuse shapes for different tastes). Before going

through each taste stimulus again, the participants were

given the chance to put their hands into the box and

familiarize themselves with the 8 shapes.

Next they were asked to take a sip of water and start with

the first taste stimulus. They were asked to express the

thoughts they had in mind and to describe their choices or

lack thereof (if none of the shapes was selected). Finally,

the participants were asked to rate the pleasantness/

unpleasantness of the shapes on a four-point Likert scale

from ‘very pleasant’ to ‘very unpleasant’. They were also

asked about their personal favorites amongst the 5 taste

stimuli and their personal food preferences to support the

interpretation of the data.

In a final step, we tested the participants for their taster

status, which classified participants into supertaster, normal

tasters, and non-tasters. Overall, the study lasted one hour

and was audio/video recorded with the consent of the

participants. No incentives were paid to the participants.

Data analysis

All 20 tasting sessions were transcribed and a qualitative

analysis based on the transcripts was conducted. Two

researchers independently performed an open thematic

coding based on 5 cases (25%). The resulting themes were

discussed and an initial coding scheme was established.

Two more cases (10%) were coded independently leading

to a final coding scheme consisting of three main themes

(described in the next section), which were then applied to

the remaining 13 cases by both researchers. We also

performed a qualitative analysis of the mapping between

the SEI objects (see Figure 4) and the taste experiences,

captured through the transcripts and the visual material

from the recorded hand movements in the second part of the

study. Based on participants’ ratings of the shapes (their

physical pleasantness/unpleasantness) we could confirm

previous ratings of Isbister et al. [13] – the more spiky

shapes were rated as ‘unpleasant to slightly unpleasant’

(shapes 8,7,2), the more rounded shapes were rated ‘very

pleasant to pleasant’ (shapes 3,4,5,6), and only one shape

was perceived as ‘neutral’ (shape 1). Finally, the supertaster

test provided us with insights on the different taste

sensibility of participants and ensures a good distribution of

taster statuses in our study. Overall, we identified 5 nontasters,

11 normal taster (4 tending towards the upper edge

of bitterness sensitivity), and 4 supertasters. These results

are consistent with the known distribution amongst the

general population [3].

STUDY FINDINGS

The description of taste experiences is based on both parts

of the study. We describe the characteristics of taste

experiences across all five tastes along three identified

themes: (1) temporality, (2) affective reactions, and (3)

embodiment (see overview in the supplementary material).

We also discuss the particularities of each individual taste

in order to elucidate the potential design qualities of single

tastes. Each identified theme is represented in a pictorial

visualization of its key characteristics based on the

identified patterns across participants’ verbalizations.

Temporality

While taste experiences have expected elements of

changing intensity (e.g., strong taste, weak taste), the tastes

were also perceived as being mobile (e.g., moving within

the mouth, moving intensities), and occasionally exerted a

physical presence (e.g., building up, eroding, lingering).

These temporal characteristics are intertwined in the

unfolding of the experiences from its initial stimulation

(diachronic structure) and set the stage for the different

taste journeys (synchronic structure). Below, we describe

the different time-intensity profiles of taste experiences.

Taste intensities are generally experienced as being

dynamic and participants’ verbalizations offer a lexicon of

growth and decline. The diachronic nature of taste

experience is also revealed in the immediacy or longevity

of dynamic intensities. For instance, all participants agree

on the immediacy of the sour taste. Such immediacy is

expounded in similes such as ‘a firework in the mouth’, ‘a

punch’, and ‘a flash that hits you’. Yet, despite the

immediacy of this experience, it is short-lasting and decays

rapidly. “When you drink it, you get that bit of a rush. Yes,

it’s basically gone now [P15, sour]. In contrast, other tastes

were described as slowly building up or maintaining

consistent intensities (e.g., high for umami, and low for

salty). Such intensities could be seen to be ‘lingering’,

rather than ‘explosive’, as one participant described it:

“You’ve got this “Whoa” sensation, feels quite strong to

start with. Then it has gone super quick” [P19, sour].

While the dynamics of intensity imply variation (intensity

increasing and decreasing), the vocabulary of movement

animates these changes. Describing the bitter taste, one

participant stated: “I guess it’s not sticky like the first one

[umami]. It’s a bit lively… I feel like it’s moving around”

[P15, bitter]. While certain movements can be attributed to

mouth-feel (e.g., moving left to right across the tongue),

others were externalized (e.g., “I feel it almost into my

sinuses and into the rest of my face” [P14, bitter]). These

expressions were not confined to the temporal

characteristics of taste experiences, but already shed light

on the bodily reactions that can be elicited by tastes.

Movement was also invoked to describe stasis (e.g., ‘stays’)

and repetitive movement (e.g., ‘waves’). “So it is kind of

strong and it also stays. It doesn’t have a peak; it doesn’t

go up and down; it just stays” [P2, umami]. Other tastes

fluctuate rapidly: “Yes, ups and downs, but quite quick.

They’re quite sudden crests and falls…” [P3, sour].

Participants often appealed to similes of physicality in order

to explain their taste experiences (e.g., ‘round’, ‘soft’,

‘heavy’). Such physical experiences are tied to a synchronic

perception of taste. In contrast, the diachronic physicality of

taste experiences is given in the implied and experienced

characteristics of taste as a residual presence (e.g.,

‘lingering’, ‘stays there’): “It just stays in your mouth, so it

kind of keeps developing” [P10, umami] or “it just leaves

its mark in your mouth and doesn’t go” [P7, umami]. Such

experiences are, much like the increasing intensities, those

that ‘build up’, or ‘get a bit stronger”. Such presence is

understood to ‘erode’. Moreover, the implied residual

physicality is associated with experiences of absence. When

tasting sourness, many participants described the

immediate, almost physically imposing intensity followed

by a marked absence. This absence is seen to draw the

taster back into the taste, leaving them wanting more: “it

creates an expectation of sweet flavour, like if you were

biting into a slice of orange or something. … It’s gone now

and actually I’d quite happily have another sip, to be

honest” [P18, sour]. This residual physicality can also be

seen to afford agency to taste experiences, where tastes

‘grab you’, and ‘hit you in the face’. As such, taste

experiences can become reified in exerting influence over

the taster. This can be achieved in the residual physicality

or in absence, for instance, where the marked absence in

sourness is seen as “a forward feeling… It has the feeling of

tartness, your mouth moves forwards” [P14, sour].

Sweetness in contrast is associated with the feeling of

filling the mouth, and when the taste is gone it leaves one

with a kind of stickiness on the teeth.

Figure 2 shows a pictorial representation of the different

types of temporality identified based on the above

descriptions across all five tastes. The intensity is

represented through the thickness of the lines in the bars,

while movement is captured through the frequency of the

lines. Finally, residual physicality as temporal characteristic

is shown through the length of the whole bar. Overall, sour

is the taste delivering the highest intensity, followed by

umami and bitter. Umami presents a high intensity, and is

also characterized by lingering without losing much of its

intensity. Such an extensive residual presence can also be

seen for bitter, however with a lower intensity. Sweet and

salty are also of low intensity and can be characterized by

particular movements. While sweet starts slowly, builds up

and then dies out, salty does not peak at all and is constant

in its perception and moderate in unfolding over time. Sour,

by contrast, is short-lived with a rapid end. Specific to sour

is the sharp beginning followed by the absence of a taste

and the return of it through a forward pulling feeling, which

disappears quickly.

Affective reactions

Affective reactions refer to both the sense of pleasure or

displeasure gained from the taste experience, but also

feelings most often regarding familiarity, such as comfort,

or, by contrast, unfamiliarity, such as surprise and

suspicion. These affective characteristics, to be captured as

pleasant-unpleasant and familiar-unfamiliar, operate not

only as a static attitudinal response to taste experiences

(synchronic structure), but also as evolving characteristics

of the taste experience (diachronic structure).

When sampling the taste stimuli many participants related

their own uncertainty (e.g., I don’t know what to expect).

After one sample, this uncertainty is replaced for familiar

tastes. For unfamiliar tastes, particularly bitter and umami,

the sense of unease pervades and persists. Thus familiarity

produces responses at singular points (e.g., I am/am not

familiar with this), while also producing responses across

time (e.g., I know/do not know what to expect). A recurring

phrase throughout the taste study was “I know what it is, but

I don’t”. While we can at times attribute this to the nature

of the stimuli as water solutions (i.e., those not regularly

experienced by participants), the sentiment expressed also

refers to the lived and felt experiences of the tastes. That is,

while participants on the one hand had the taste ‘on the tip

of their tongue’, those tastes also brought to mind a variety

of known experiences, or, in the absence of known

experiences, feelings of uncertainty or unease. Such

feelings must presumably be associated with evolutionary

causes (considering many bitter foods are poisonous) or in

form of personal memories (e.g., salt, salty water, and the

seaside) and cross-modal experiences (e.g., with color, or

sounds). “If I drink or eat something that leaves that kind of

trace, I always imagine a colour. Glowing…. It’s weird. I

have no idea what this is, but there’s a bitterness that

stays” [P2, bitter]. Participants identified as supertasters

expressed their affective reaction more clearly: “Definitely

bitterness… I don’t like it” [P8, bitter], or “It’s immediately

bitter.… It’s like swallowing medicine” [P18, bitter].

There were few predictable or consistent affective reactions

among participants, and those experienced as pleasurable

by some, were experienced as disgusting or unsettling by

others. The affective response of participants could often be

tied to the participant’s familiarity with the taste. This was

particularly noticeable with umami. Participants who were

familiar with this taste indicated familiarity with savory

Asian cuisine, and could therefore interpret the perceived

taste and experienced it as pleasant. Those who did not eat

Asian cuisine were less familiar with the taste, particularly

in this intensity, and described unease and uncertainty when

tasting it. Such responses also evolved over time, notably

with sweet and sour tastes. While, as mentioned, sour

produced an immediately unpleasant experience, followed

by a refreshingly pleasant experience (e.g., “yes it probably

gets more pleasant as the intensity of the taste dissipates”

[P17, sour]), the taste of sweet was often initially pleasant,

followed by a distinct unpleasantness. This unpleasantness

could be so strongly felt as to produce nausea for some

participants (e.g., “although it’s dying off over time. It’s

quite sickly actually” [P20, sweet]). Such experiences were

tied to the physicality of the taste residing in the mouth, and

were perceived in two extremes for umami, influenced

through the participants’ familiarity/unfamiliarity with this

taste. Participants familiar with this taste perceived the

mouth filling and lingering experiences as comforting

(satisfaction after a full meal), while other participants who

were unfamiliar with it perceived it as disgusting, obtrusive,

and annoying referring to the fact that the taste takes over

control, without the chance to get rid of it quickly.

As with temporality, we created a representation of the

different types affective reactions on the five tastes (see

Figure 3). The pleasant-unpleasant characteristics of the

taste experience are represented through the ‘green’ and

‘red’ colors and in cases of a neutral experience colored as

‘orange’, and finally ‘white’ in case of absence of the taste.

The familiar-unfamiliar characteristics only find an explicit

representation for the umami. The familiarity of the taste

lead to its pleasant perception (upper bar for umami), while

unfamiliarity with the taste was expressed through

unpleasantness (lower bar for umami). Overall, some tastes

are characterized by the change from unpleasant to pleasant

(sour) or the other way around from pleasant to unpleasant

(sweet), while the bitter taste was clearly unpleasant and

salty was described as neutral. For umami, we identified

two separate experiences (participants either love or hate it)

grounded in the familiarity and unfamiliarity of the taste.

Embodiment

Although we would expect food experiences to involve

embodied, textural, responses (such as ‘crunchy’, ‘slimy’),

here each taste stimuli is experienced in the same form (i.e.,

as a colorless and odorless solution), and yet produce varied

embodied responses. Embodiment in relation to the

described diachronic and synchronic taste experiences

refers to the mouth-feel of tastes (how something is felt in

your mouth). Some participants additionally describe whole

body reactions (reactions described beyond the mouth) and

others refer to imagined and disembodied responses

(resulting from the taste stimulation and its associations).

Mouth-feel, referring to the experienced chemical and

physical sensations in the mouth, is frequently used to

describe different characteristics of foods, including coffee,

wine, and textured foods. Such descriptions are offered by

our participants for qualities of texture and viscosity. “It’s

just like a softness, but I guess a little bit more viscosity

even though I’m quite sure it doesn’t have any viscosity. It’s

just sort of the feeling of viscosity, the sweetness and this

cloud is just a bit more mouth feel” [P14, sweet]. The

mouth-feel also relates to a sense of movement, where

tastes evolve in space. Most often these are lateral

movements within the mouth, or commonly tastes are felt to

move backwards. Such experiences can be a feature of the

physical movement of the taste stimuli during the swallow

reflex and also associated with the location of taste

receptors on the tongue. However, in other cases, taste

experiences defied the location of taste receptors and tastes

could be experienced on the teeth, gums, and lips. One

participant goes as far as to describe the absence of mouthfeel:

“I don’t know really. It leaves this numbness in my

mouth like the lemon, but without the initial burst” [P9,

sour]. In addition to the sensations described in mouth,

some participants described bodily reactions that were

opposed to the mouth-feel or isolated taste experiences. “I

think the first part of it, the sour part, is a bit of a shock to

the system. I don’t think you’re expecting it to be like that”

[P16, sour]. Another participant said “I kind of see it from

the moment it enters my mouth and goes down all the way

to my stomach. It’s like I can see where it’s going” [P2,

bitter]. In this sense, participants described tastes as

producing expansive responses, including pleasure, nausea,

and, others including reactions associated with allergy such

as increased body heat (e.g., “If you eat it, it’s like your

body – the heat just changes” [P2, umami]). Feelings of

pleasure were often described as filling, particularly filling

the face or the whole body. A participant describes it as

such: “I feel that my whole face feels pleased with it” [P14,

umami]. Such feelings were not always positive and for

some participants, overwhelming feelings of nausea

accompanied tastes of salt, umami, and sweet. Nausea

could also be experienced in undulating taste experiences –

those taste which were experienced as prone to fluctuations

in intensity, almost mimicking travel or sea sickness.

Participants also described disembodied reactions, which

refer to something experienced that lingers between the

mouth and the body. Rather than experiencing direct bodily

reactions, participants describe an imagined reaction. “It’s

like it’s there but it’s not there” [P2, salty]. Disembodied

reactions could also be seen in expected or caricatured

responses, such as the imagined ‘pucker face’ of the sour

taste. Although few participants actually exhibit such a

reaction, it is an ingrained image of biting into a lemon. “It

feels a little bit, not uncomfortable, but it feels like it makes

you kind of screw you face up a bit” [P11, sour].

Shapes assigned to the overall taste experiences

The usage of the SEI objects (see Figure 4) as a physical

engagement with shapes enriched the description of the

taste experiences. The shapes particularly contributed

descriptors related to the combined temporal, affective, and

embodied experiences of each individual taste. Below, we

summarize the key characteristics and the mapping of the

eight objects for each taste across all participants.

The sweet taste, generally experienced as ‘smooth’ and

‘rounded’, was most reflected in shapes that present

elements of change such as “phases” (shapes 4 and 5) or

have protruding elements (like 7 and 1, or even the halfspiky

shape 2). While typically a pleasant taste, there is a

dynamic modulation of intensity and pleasure in the shapes.

The sour taste produces a ‘sharp’ response and for many is

best characterized by shapes such as 8 or 2. There are,

however, also elements of temporality, a shifting/phasing

associated with shape 4, starting with the big part as an

explosion and then rapidly decaying. The salty taste has a

broad aspect (mapped towards 3 and 6) and a finer

granulated and dynamic experience expressed through the

shape 1. Similar to sweetness and sourness there is a

repeating wave assigned to this taste experience, verbalized

around shapes 4 and 5 though this time associated with an

unpleasant feeling/sickliness as aftertaste. More than other

tastes, salty was associated with a subtlety of the temporal

characteristics, an experience of something moving, not

doing much, but still being there. This made participants

want a shape that they could manipulate (“These [objects]

are kind of too permanent; you’re not able to manipulate

them” [P6]) or something more neutral, such as a flat shape,

or a shape, which can be changed. Despite the fact that the

bitter taste was experienced as unpleasant, the mapping to

the shapes created two distinct experiences. For some

participants, bitter is a spiky but lingering experience

associated with a dull unpleasantness (1, 2, and 7 shapes

selected). For others it is a rounded and smooth taste (these

participants chose shapes 5 and 6), associating it with

medicine (form of pills), which dissolves in the mouth, and

you cannot get rid of it. Similarly to bitter, the mapping for

umami resulted in two distinct experiences. If umami was

experienced as unpleasant, participants tended to describe

the taste as disgusting and chose the shape 8 or 2. In those

cases where umami was perceived as pleasant, participants

described it as a more rounded taste with depth and chose

combinations of the rounded shapes (such as 3 and 1 shape

were used most, and combined with either the 5, 4, or/and 7

shape). This mapping confirms the descriptions of umami

as a full, mouth-filling experience with lots of things to it.

Overall, sweet and sour seem to be the two tastes where

participants show high agreement with respect to mapping

the shapes to taste experiences. Bitter and umami seem to

share some associations and create two different mappings

between shapes and taste experiences, while salty shows a

tendency towards smooth and round shapes, but with the

lack of the ability to change and manipulate the shapes.

Combined representation of the taste experiences

Figure 5 shows the final pictorial representation of all three

characteristics combined for each of the five tastes. The

length of the forms represents the temporal aspects, while

the width captures the mouth-feel. Whole body and

imagined embodiment could not be captured as such, but

are described in detail above. The expression ‘lingering’

was used particularly for sweet, bitter, and umami. When

used for sweet and umami ‘lingering’ is experienced in

combination with a ‘mouth filling’ element (it is filling the

whole mouth), while for bitter there is no filling experience

but it is described as a thin (straight through your mouth to

the back) experience, next to being unpleasant. In the bitter

case, ‘lingering’ thus refers to the residual physicality of

this taste (in the back of your mouth). Sour has an initial

unpleasant taste, dies down quickly, but comes back after a

short absence and leaves one with the feeling of wanting

more. Salty at last is similar to bitter, however with a

shorter life and perceived as less unpleasant. Salty is

perceived as a neutral taste with little consequence.

DISCUSSION AND FUTURE RESEARCH

While sensory researchers and neuroscientists study the

perception of taste and its temporality, their focus is on

quantifying the intensity and perceived changes of intensity

via a wide range of evaluation scales [26] or, in some recent

attempts, by means of time-intensity profiles of fMRI data

Comment 14

The aim of sensory and neurological Psychology is to understand the associated natural phenomena, including taste, in terms of explanation and prediction (see Comment 6). The resulting knowledge could be assimilated to both everyday language (see Comment 1) and to design (see Comment 10). However, in being so assimilated, its scientific status may be lost or at least transmuted in unknown ways. As a result, the knowledge needs to be (re-)validated in its new domain of application – by use, in the case of everyday language and by diagnosing design problems and specifying design solutions, in the case of design. Failing re-validation, some account must be provided, if the scientific status of the knowledge is considered somehow to be special, for example offering a better guarantee or some such.

[19]. Our findings add a semantic level of understanding

underlying the taste experiences, their temporality enhanced

through descriptions of the affective reactions and

embodiment that the five basic tastes provoke.

Comment 15

For issues concerning the added ‘semantic level’ and its relation to other possible levels – see Comments 5 and 14.

This understanding may be useful when designing for taste

experiences as it provides designers and developers a

vocabulary to talk about taste and the design potentials

related to the different characteristics.

Comment 16

Of course, designers and developers already have a vocabulary (indeed a language) for talking about taste, as demonstrated by the participants in the reported experiments, who express their taste experiences in a very articulate manner, with no training at all. Your claim here should surely be that temporal, affective and embodied characteristics of taste experience now have some empirical support, provided by your experimental studies. The latter can now be replicated and developed for the purpose of either better understanding (of whatever sort – see Comment 1) or better support for design (see also Comments 9 and 10). In fact you claim, in addition, that the characteristics constitute a framework for design. This claim will be addressed later – see Comments 20 and 21.

First, we discuss the particularities of each taste quality, and then discuss them

with respect to established psychological and behavioral

phenomenon highlighting their design potential for HCI.

design. This claim will be addressed later.

Comment 17

I think that it is a good idea to suggest how taste qualities might be used in design, as you do here. I also think that it is just about right to categorise these thoughts as having ‘design potential’. Stronger claims with respect to design need to address some of the design requirements, identified by Comments 9 and 10. Concerning the framework – see Comments 20 and 21.

How is taste experienced?

Here we discuss the specific experiences each of the five

basic taste qualities create and can inspire design in HCI.

Comment 18

‘Inspire’ here presumably means more like ‘inform’. Stronger interpretation might be hard to justify at this stage. If you mean ‘inspire’ in the sense of ‘invent’, you should say so, as it would impact the sort of discipline you assume HCI to embody.

Sweet: Pleasant but with a bittersweet ending

The sweet taste was consistently described as pleasant,

which turned into something unpleasant. Participants

struggled between the instinctive taste likeability and the

learned taste values and rules (sweet is bad for the teeth),

which can be seen in light of learned associations, discussed

by Schifferstein and Hekkert [32] with respect to taste

experiences of products. Of particular interest with respect

to our findings on crossmodal interactions for sweet

stimulations are the embodied reactions (e.g., “It’s just sort

of the feeling of viscosity, the sweetness and this cloud is

just a bit more mouth feel” [P14]). Such reactions can be

explained through learned associations with sweetened food

and beverages. It is a combination of learned as well as

innate, genetic, and cognitive factors [32]. Sweet sensations

can be used to stimulate and enhance positive experiences,

however, on a limited timescale, as the sweetness is quickly

disappearing leaving one unsatisfied. It’s a pleasant taste

but one that is tinged with a bittersweet ending.

Sour: Unpleasant at first, but with the need for more

In contrast to the sweet taste, the sour taste is described as

short-lived and it often comes as a surprise due to its

explosive and punchy character. This taste overwhelms one

with its rapid appearance and quick decay. It leaves one

with the feeling that there is something missing. Based on

childhood memories, such as for instance of sweet-sour

drops, participants were expecting sweetness, but were left

disappointed, leaving them with the feeling of wanting

more. This phenomena was also observed in the evaluation

of a gustatory gaming interface with children, where sour

was used for negative reinforcement linked to the game

dynamics [17]. Children intentionally failed in the game in

order to get another sour stimulation.

Salty: Not doing much

The salty taste experience was not linked to an extreme

reaction unlike sour, bitter, and umami. This taste is often

described as ‘bland’, ‘discrete’, and ‘just being there and

not doing much’. It is minutely moving around, giving the

feeling of cleansing the mouth, but not being mouth filling

as sweet or umami, and certainty not as unpleasant as bitter,

however lingering almost as long as the bitter taste. The

modesty of saltiness in contrast to all of the other tastes

opens up some interesting questions when looking at the

neuroscience findings. Nakamura’s [19] findings based on

time-intensity fMRI profiles suggest that salty tastes change

more rapidly than do sweet tastes. This is not quite

consistent with how our participants described their

experiences and needs further studies.

Bitter: Unpleasant, not to be experienced again

The perceived intensity of the bitter taste was not the same

for everyone, as confirmed by the supertaster test. While

supertasters felt the experience with more immediacy,

others had to allow the taste to travel to the back of their

mouth to recognize it. After this initial difference, the bitter

experience becomes consistent with respect to its

‘lingering’ features, of ‘staying’ either on the tongue or at

the back of the mouth. Bitter was also described as ‘thin’.

The character of bitter was further revealed through learned

associations referring to ‘biting into a flower’, or

‘medicine’, things you had to take as a child, but after

which you would rather avoid this experience of bitterness.

Bitterness can indicate the presence of toxins [32] and is

found in evolutionary development of humans (e.g., feeling

of suspicion regarding bitter food as poisonous) [7]. It may

be useful for design to make people avoid certain behaviors.

Umami: Like/dislike, but still confusing as a taste

The familiar-unfamiliar characteristics of umami caused

much confusion in our study and participants could not rely

on their intuition. While the ‘like’ or ‘dislike’ of the taste

was decided instantly, the unpicking of the still ‘confusing’

elements of the umami taste was more challenging.

Different word pairs depending on the like/dislike of the

taste were expressed: ‘pleasant–unpleasant’, ‘comforting–

uncomforting’, and ‘liking–disgusting’. We could also see

participants using additional bodily descriptors, in

particular when describing umami as a pleasing experience

(‘face feels pleased’ or ‘body heat changes’). In cases of

dislike, the focus of attention in the verbalizations was the

lingering characteristic of the taste founded in the inability

to get rid of it. In these cases, the residual physicality can be

seen to afford agency. The taste experience becomes reified

in the influence it exerts over the taster. Depending on

personal familiarity/unfamiliarity (which may be defined by

cultural factors) and personal preferences, this taste

experience is quite interesting for design. Umami grabs

one’s attention and initiates a conscious process of

reflection. While judgment on the taste is defined quickly,

the reflective thinking brings to the fore the richness and

variety of the taste. Even when perceived as unpleasant, the

richness is recognized, and linked to the motivation to

remove the taste from the mouth.

How can we design with taste experiences?

Taste experiences can be discussed with respect to their

relevance for design, building on existing psychological and

behavioral phenomenon: rational and intuitive thinking,

anchoring effects, and behavior change.

Comment 19

The rationale is unclear here in two ways. First, how does building on ‘existing psychological and behavioural phenomena’ affect the relevance of the reported taste experiences to design? Secondly, on what basis were these phenomena selected, and not others? In both cases, we also need to know, whether the relevance is only to taste experience or to experience more generally. See also Comments 1 and those concerning scientific understanding.

The dual process

theory [14,37], for instance, accounts for two styles of

processing: the intuition based System 1 with associative

reasoning that is fast and automatic with strong emotional

bonds, and reasoning based on System 2 which is slower

and more volatile, being influenced by conscious judgments

and attitudes. Based on our findings, we can see that sweet

is intuitively perceived as pleasant, and bitter as unpleasant,

while sour, salty, and umami cause a reflective process,

confused, for instance, by the surprise appearance and rapid

disappearance of the sour taste. Our findings also give

insights into how to time the presentation of the taste

qualities so that the user can transition from System 1

thinking to System 2 thinking. Figures 2, 3, and 5 can be

used to create the appropriate transitions and time them. For

example, the rapidity of the sour taste experience does not

leave enough time for System 1 to engage with it and

triggers System 2 to reflect on what just happened. Such

reactions when carefully timed can prime users to be more

reason based in their thinking during a productivity task

(e.g., to awaken someone who may be stuck in a loop).

Moreover, an appropriately presented taste can create a

synchronic experience that can lead to stronger cognitive

ease (to make intuitive decisions) or reduce the cognitive

ease to encourage rational thinking. For example, a pleasant

taste can be used to provide achievements across the

workflow, however with the slight hint that there are still

more tasks to do before you are finished (e.g., the slight

unpleasant aftertaste of sweetness). Below, we outline

potential design directions for using taste experiences in

work-related activities and for personal behavior

management. Doing so, we draw on the potential of

different taste qualities and their power to stimulate

intuitive and rational thinking described above.

Managing anchoring effects through taste

A common aspect of everyday activity is interruption. We

are often interrupted by emails, telephone calls, or other

unanticipated events. These interruptions can either be short

(e.g., a quick glance at an email pop-up) or slightly longer

requiring us to change our activity (e.g., a line-manager

walking into your office to ask for something). All these

activities have anchoring effects. In other words, the initial

activity affects our judgments and decision making in the

latter activities. It has also been shown that users often find

it hard to avoid these biases in their judgments [38].

Our study of taste experiences suggests that taste interfaces

can be carefully designed to manage interruptions in such a

way that anchoring effects can be either minimized or

maintained. For example, we know that the salty taste has a

long temporal component with a feeling of “not doing much

but being there”. This taste could be very useful in those

situations where the interruption is small and the user is

expected to return to the initial activity soon. As an

example, when the user notices a pop-up in the bottom left

corner of their desktop (for email or other social media

interruptions) a small salty taste in their mouth which starts

just before the user switches their activity can be useful.

This will prolong their initial experience and remind them

of the initial activity when still checking the social media

page. This could enable smoother transitions back to the

initial activity. Alternatively, however, if the interruption is

a longer activity then it is useful for the user to drop any

priming effect that might transfer to the new activity. In this

case, a sour taste in the mouth would leave the user a quick

sharp taste engaging their rational System 2 but rapidly

decaying helping the user return to a more neutral state by

the time they switch to the new activity. Such management

of anchoring effects is not only useful for productivity

activities but also in other activities, such as gaming. For

example, LOLLio – the taste-based game device described

above [16], currently uses sweet and sour for positive and

negative stimulation during the game play. We suggest that

such a game could be improved based on our framework by

providing fine-grained insights regarding the specific

characteristics of taste experiences, which can be integrated

into the game play. When a person moves between related

levels of a game a continuing taste like bitter or salty is

useful. Whereas when a user is moving to distinct levels or

is performing a side challenge an explosive taste like sour,

sweet, or umami might be useful. The choice of specific

tastes in each category can be tuned by the designer to

create different affective reactions and a sense of agency.

Priming positive behavior through taste

Taste and taste preferences play an important role in our

food choices [24] and food plays a significant role in our

health and wellbeing. The stimulation and manipulation of

taste experiences therefore offers potential to improve a

variety of food behaviors. Using taste stimulation technology

to alter the taste of unpleasant but healthy food is one

obvious route. Expanding the design space for healthy taste

technology, our framework suggests alternative routes.

Taste experiences might be heightened through appeal to

related experiences and sensations. Morphing physical

objects, such as recently suggested shape-changing devices

[29], might also be used to replicate the embodied

expansiveness of the umami taste to stimulate an increased

taste experience for patients receiving chemotherapy who

may suffer from hypogeusia, a decrease in taste sensitivity.

Taste stimulation might also facilitate sustainable food

practices, for instance, linking food waste to taste

experiences. Taste stimuli might thus supplement other

post-actional cues in the effective disturbance of food waste

habits and promote critical reflection. When disposing overripe

bananas, a user might get a sour stimulation for the

waste of food but the immediate reward for waste

separation. Taste stimulation might also reflect various

characteristics of food waste, such as its lengthy impact on

environmental sustainability through the bitter taste. In this

way, the framework for design points to the potential for

taste experiences to be incorporated into timely and

rewarding persuasive messages for positive food behaviour.

Comment 20

Just to be clear, I assume the taste experience framework consists, then, of the 5 basic taste qualities : sweet, sour, salt, bitter, and umami and 3 themes/dimensions/ qualifiers: temporal, affective, and embodied. In all, there are 15 qualified descriptors, constituting a vocabulary, which along with the other additional information can be used to inform design. Each lexical descriptor, however, also implies a concept. The scope of the framework is user taste experience. This clarification is important for the development of additional guidance to designers – see Comment 22 later.

CONCLUSIONS

In this paper we presented the results of a user study

exploring the experiential characteristics for each of the five

basic taste qualities. Our analysis of participants’

verbalizations, collected by means of verbal and non-verbal

methods, resulted in three key themes. We provide rich

descriptions on the temporality, affective reactions, and

embodiment of taste experiences. We discuss these themes

for each individual taste elucidating the design potentials

with respect to the specific structure and qualities of sweet,

sour, salty, bitter, and umami tastes.

Comment 21

This is presumably the ‘understanding’, which is referenced early in the paper (see Comment 1) and its potential support for design.

Our findings help to establish a framework for the design of taste experiences in

HCI, enhancing existing technology driven research around

taste, and food interaction design research. Although we do

not provide guidance for the design of a specific interactive

system in this paper, we are convinced that our framework

provides a starting point for designers and developers to

think about design/development potentials for taste in HCI.

Concluding Comment

1. This is an interesting paper, which reports an initial exploration of taste as a potential interactive modality for HCI.

2. The paper reports a taste framework, intended to support HCI design. The question arises, however, of how this framework might be further developed to provide such support. To that end, knowledge can generally be thought to include knowing what and knowing how (declarative and procedural knowledge respectively). Understanding, as used in this paper (see Comment 1), might be similarly expressed. The framework is already declarative (descriptive) knowledge and conceptualises taste experience (see Comment 20). It could be further developed into a (design) model, in which the relationships between the concepts are made explicit.

3. Subsequent development would include the model’s operationalisation, test and generalisation towards the aim of validation (see Comments 6 and 10).

4. To be used by designers, the model would also need a method of application – procedural design knowledge. The model and method would be validated together for their ability to support the diagnosis of design problems and the prescription of design solutions (see Comments 6 and 10).

5.There are, of course, many other possible ways forward. The way forward, suggested here, could be followed; but should otherwise be considered as encouragement for finding a way.

ACKNOWLEDGMENTS

This work is supported by the EU Marie Curie Action (FP7-

PEOPLE-2010-IEF) and RCUK SiDE (EP/G066019/1). We

wish to thank our study participants and Annika Haas for

the audio-visual support, as well as Katerhine Isbister and

Kristina Höök for providing us with a set of the SEI objects.

REFERENCES

1. Bartoshuk, L.M., Duffy, V.B., Fast, K., Green, B.G,

Prutkin, J., Snyder, D.J. Labeled scales (e.g., category,

Likert, VAS) and invalid across-group comparisons:

What we have learned from genetic variation in taste,

Food Qual Pref, 14(2), (2003), 125-138.

2. Bartoshuk, L.M. Comparing sensory experiences across

individuals: Recent psychophysical advances illuminate

genetic variation in taste perception. Chem. Senses,

25(4), (2000) 447-460.

3. Chen, J., Engelen, L. (Eds.). Food oral processing:

Fundamentals of eating and sensory perception. Wiley-

Blackwell: Oxford, UK (2012).

4. Comber, R., Ganglbauer, E., Choi, et al. Food and

interaction design: designing for food in everyday life.

CHI EA (2012), 2767-2770.

5. Desmet, P.M.A., Schifferstein, H.N.J. Sources of

positive and negative emotions in food experience.

Appetite, 50(2-3), (2008), 290-301.

6. Drewnowski, A., Kristal, A., Cohen, J. Genetic taste

responses to 6-n-propylthiouracil among adults: A

screening tool for epidemiological studies. Chem.

Senses 26(5), (2001), 483-489.

7. Glendinning J. Is the bitter rejection response always

adaptive? Physiol. Beh., 56, (1994), 1217-1227.

8. Green, B.G., et al. Evaluating the ‘Labeled Magnitude

Scale’ for measuring sensations of taste and smell.

Chem. Senses, 21(3), (1996), 323-334.

9. Grimes, A., Harper, R. Celebratory technology: new

directions for food research in HCI. CHI (2008), 467-

476.

10. Hoehl, K., Schoenberger, G.U., Busch-Stockfisch, M.

Water quality and taste sensitivity for basic tastes and

metallic sensation. Food Qual Pref 21(2), (2010), 243-

249.

11. Hupfeld, A., Rodden, T. Laying the table for HCI:

uncovering ecologies of domestic food consumption.

CHI (2012), 119-128.

12. Humphries, C. Delicious science. Chefs are teaming up

with researchers to create avantgarde dishes. Is

‘molecular gastronomy’ more than a fad? Nature, 486

(2012), 10-11.

13. Isbister, K., Höök, K., Sharp, M., Laaksolahti, J. The

sensual evaluation instrument: Developing an affective

evaluation tool. CHI (2006), 1163-1172.

14. Kahneman, D. A perspective on judgement and choice.

American Psychologist, 58(9), (2003), 697-720.

15. Light, A. Adding method to meaning: A Technique for

exploring peoples’ experiences with technology. Beh.

and Information Technology, 25(6) (2006), 175-187.

16. Murer, M., Aslan, I., Tscheligi, M. LOLLio: Exploring

taste as playful modality. In Proc. TEI 2013, 299-302.

17. Moser, C., Tscheligi, M. Playful taste interaction. IDC

(2013), 340-343.

18. Narumi, T., Nishizaka, S., Kajinami, T., Tanikawa, T.,

Hirose, M. Augmented reality flavors: gustatory display

based on edible marker and cross-modal interaction.

CHI (2011), 93-102.

19. Nakamura, Y., Goto, T.K., Tokumori, K., et al. The

temporal change in the cortical activations due to salty

and sweet tastes in humans: fMRI and time-intensity

sensory evaluation. Neurorep, 23(6), (2012), 400-404.

20. Ngo, M.K., Velasco, C., Salgado, A., et al. Assessing

crossmodal corresponddences in exotic fruit juices: The

case of shape and sound symbolism. Food Qual Pref 28,

(2013), 361-369.

21. Núnez-Jaramillo, L., Ramirez-Lugo, L., Herrera-

Morales, W., Miranda, M.I. Taste memory formation:

Latest advances and challenges. Behav Brain Res,

207(2), (2010), 232-248.

22. Maynes-Aminzade, D. Edible bits: Seamless interfaces

between people, data and food. CHI EA (2005), 2207-

2210.

23. Obrist, M., Seah, S. A., Subramanian, S. Talking about

tactile experiences. CHI (2013), 1659-1668.

24. Palmer, S. E., Schloss, K. B. An ecological valence

theory of human color preference. Proc. of the National

Academy of Sciences, 107(19) (2010), 8877-8882.

25. Petitmengin, C. Describing one’s subjective experience

in the second person. An interview method for the

science of consciousness. Phen. Cog. Sci., 5(3-4),

(2006), 229-269.

26. Pineau, N., Schlich, P., Cordelle, S., et al. Temporal

dominance of sensations: Construction of the TDS

curves and comparison with time-intensity. Food Qual

Pref, 20(6), (2009), 450-455.

27. Ranasinghe, N., Cheok, A.D., Nakatsu, R. Taste/IP: The

sensation of taste for digital communication. ICMI

(2012), 409-416.

28. Ranasinghe, N., Karunanayaka, K., Cheok, A.D.,

Fernando, O.N.N., Nii, H., Gopalakrishnakone, P.

Digital taste and smell communication. BodyNets

(2011), 78-84.

29. Roudaut, A., Karnik, A., Löchtefeld, M. et al.

Morphees: toward high “shape resolution” in selfactuated

flexible mobile devices. CHI (2013), 593-602.

30. Savage, N. Technology: The taste of things to come.

Nature 486(21), (2012), 18–19.

31. Schifferstein, H.N.J., Fenko, A., Desmet, et al. Influence

of package design on the dynamics of multisensory and

emotional food exp. Food Qual Pref, 27 (2013), 18-25.

32. Schifferstein, H.N.J. Hekkert, P. Product experience.

USA: Elsevier (2007).

33. Smets, G.J.F., Overbeeke, C.J. Expressing tastes in

packages. Design Studies, 16(3), (1995), 349-365.

34. Spence, C. Crossmodal correspondences: A tutorial

review. Atten Percept Psychophys 73, (2011), 971-995.

35. Spence, C., Piqueras-Fiszman, B. Technology at the

dining table. Flavour, (2013), 2-16.

36. Spence, C., Smith, B., Auvray, M. Confusing tastes and

flavours. In M. Matthen and D. Stokes (Eds.). The

senses. Oxford: University Press. (in press, 2014).

37. Stanovich, K.E., West, R.F. Individual difference in

reasoning: implications for the rationality debate?.

Behav. Brain Sciences 23(5), (2000), 645–665.

38. Strack, F., Mussweiler, T. Explaining the enigmatic

anchoring effect: Mechanisms of selective accessibility.

Pers. and Social Psy. 73(3), (1997), 437-446.

39. Tosey, P., Mathison, J. Exploring inner landscapes

through psychophenomenology. Qual. Research in

Organiz. And Management: Inter. J., 5(1) (2010), 63-82.

40. Trivedi, B.P. Hardwired for taste. Research into human

taste receptors extends beyond the tongue to some

unexpected places. Nature, 486, (2012), 7-9.

41. Vermersch P. L’entretien d’explicitation [translated as

Explicitation interview], ESF (1994).

42. Wakeling, I.N., MacFie, H.J. Designing consumer trials

balanced for first and higher orders of carry-over effect

when only a subset of k samples from t may be tested.

Food Qual Pref, 6(4), (1995), 299-308.

[/expand]

Blog Categories

Innovation and Art

Craft and Engineering

Applied and Science

Discipline and Design problem

Research

Knowledge And Practices

HCI Engineering TEST

HCI Engineering: Possible Blog Posts – ex

1. 29 February 2016 – User Requirements and Design Problems – Same or Different?

Leave a Reply Cancel reply