Download Direct Observation: A Method for Studying User Interactions with Products and more Slides Communication in PDF only on Docsity!
observation is said to have high face validity, also referred to as external validity or ecological validity. However, one must always be aware of the possibility of the so called “Hawthorne Effect”; the fact that people usually perform better under observation because of the attention paid to them.
During direct observation it is common for an observer to be present who sits passively and records as accurately as possible what is going on. Usually it is the behaviour of one or more persons that is recorded, and an advantage of the technique is that a number of people interacting with each other and the same piece of equipment can be observed. A variation on this technique is to have a video camera mounted at the point of usage, which records interactions which can later be watched and analysed by an observer.
The observation can be totally “free” or more structured i.e. where obser- vers record events as belonging to one of a number of discrete categories identified. The number of categories adopted largely depends on what the observers intend the data will be used for, and very broad categories may be used for some studies, whilst detailed categories will be used for others. In some investigations a more free approach may be used where the observer records all of their impressions during observation rather than trying to group them in some way. However this introduces a high degree of subjectivity into the evaluation process, and in practice it is usually better to try and define the categories of behaviour that will be observed. One way of achieving this is to perform a pilot study where free recording
Tools & Techniques
Direct Observation
Direct observation has high external validity.
The Hawthorne Effect: People tend to perform better under observation because of the attention they get.
SPECIAL CONSIDERATIONS:
Mental impairment
Communication impairment
Blind/visually impaired
The elderly user
Direct
observation
What does this technique do?
The term “non–intrusive” is often used to characterise this technique: Users do what they normally do without being disturbed by the observers. One of the advantages of this method is that users can be observed in the environment where the system is normally used. This is why direct
Poulson, D., Ashby, M., Richardson, S. (eds.): USERfit. A Practical Handbook on User-Centred Design for Assistive Technology. TIDE EC-DG XIII, ECSC-EC- EAEC, Brussels-Luxembourg, 1996.
takes place, and then to use the results of this to identify relevant catego- ries for use in a wider study, and to define clearly the criteria to be applied by observers in putting observed behaviour into particular categories e.g. types of errors made. The degree of structure is related to the “objectivity” of the method, as less structure may result in observations that are more the result of the observers point of view than of the users behaviour, and in addition can make it difficult to make comparisons when more than one observer is used. Where more than one observer is used it is particularly important to ensure that all observers are in agreement as to what they are recording and the criteria they are using.
The data captured during direct observation can include objective as well as subjective information, as it is possible for observers to accurately record the amount of time taken to perform particular activities and the errors that they make in use. However more subjective information can also be valuable, e.g. any anxiety or frustration observed, and the observers impressions of the state of mind of the user.
Direct observation has the highest degree of ‘ecological’ validity in that direct observation attempts to monitor usage of a product in settings which are close to actual usage. However there are effects of having observers present as we have already indicated, and for this reason it is recommended that any direct observation study should allow time for those being observed to become less aware of the observers presence and view them more as being a “fly on the wall”, rather than another person present. This can be promoted to some extent by making the observers role clear to those being observed, and the observer not allowing themselves to be drawn into social interactions with those being observed. Observation is often needed over an extended period as it is important to try and ensure that the periods being observed cover the range of usage that the product might face in actual use. Thus it can be important to ensure that a person is observed whenever they might normally use the product, and not just at set times which may in fact be atypical. One common approach is to try and observe “a day in the life”, when a products usage is observed throughout the day, from getting up, going to bed, and where appropriate getting up in the night.
Direct observation does not allow observers to interfere in the users normal interaction with the products, which is of advantage for ensuring that realistic usage is observed, but is also of disadvantage in that the observer has to interpret what they observe without the active clarification of the person being observed, and that in addition they cannot control the experiences the person faces. This lack of control means that the observer may not see the users responses to rarely occurring events which may be of interest, and for this reason direct observation often needs to be used in conjunction with other techniques e.g. user trials.
Direct Observation
not uncommon. It is in general very difficult to estimate the costs before you have a detailed evaluation plan, and it is recommended that a pilot investigation be carried out in order to determine the likely resources needed for data capture and subsequent analysis.
Who can use it There are different demands on the observers and on those that plan and prepare the study. If you use a rather unstructured method, it is very important that the observer has good knowledge of the usage domain. With well structured observational categories, observers can easily be trained to look for duration and occurrences of specified types of behaviour.
One should consider using observers that personally know the parti- cipant in these studies. However there are advantages and disadvantages to this. One advantage is that this can make it easier for those being observed, as they are likely to feel uncomfortable in the company of strangers. On the other hand some participants may be happier inter- acting with a neutral outsider, where their professional role can ensure anonymity of information. An observer who has no personal involve- ment with a user may also be able to gain valuable insights, which a person more familiar with a situation may miss. There are no hard and fast rules as to which alternative is the most appropriate. Factors to take into account include: how the user (and others who may be present) feel about being observed by a stranger or someone they know, the degree to which the observer is likely to be impartial, and the resources available for the investigation.
To plan an observational study demands that one have a clear idea of what the goals for the study are. For simple studies it should be sufficient to follow the procedures described below, however, more ambitious projects should consider the possibilities of using professional psychologists or ergonomists.
Who are the informants
Users End users of the system under consideration are the typical participants in direct observational studies. It is important that the users previous experience with the system is recorded and taken account of in the planning of the study. Users may be recruited through relevant organisations or by contacting local schools or institutions.
Direct Observation
Selection of users
Although the ideal is to let a “representative sample” of the user popu- lation try out equipment, this is often difficult and expensive to achieve. After having defined the user population the testers should decide whether they would try to approach the extremes of the distribution (best case — worst case) or the mode (the typical case). See the Procedure section below on advice for number of users to be involved.
Special considerations
General
To perform a direct observational study usually requires getting access to the user at home or at work. In both cases one should take careful con- siderations that the necessary permission is obtained, and not only from the user. This may be helpers or other persons present in the same location, or it may involve parents or guardians having a responsibility for supporting the users decision in these kind of situations. More than ever the principle of “informed consent” applies when using direct observational methods, as it is unethical to make observations without such consent. It may be tempting to use a “candid camera” technique where interactions are recorded without the awareness of participants. It is not advised to do this under any circumstances as there are serious ethical problems with such hidden monitoring.
One should also be aware that many disabled people are embarrassed when asked to expose their disability to strangers. Special attention should therefore be paid to the motivation of users to participate as it may not be obvious why users would want to participate voluntarily in these kinds of studies. As with other forms of investigation financial incentives can be considered, but the indications are that this is unlikely to be a major incentive for most people prepared to take part in investigations. Careful explanation as to the value of the work can often be more effective, along with appropriate feedback as to the results of the study and its practical consequences.
Many elderly and disabled people are lonely, and might see this as an opportunity to engage in social interaction. This can be an incentive for the user to participate and can also provide the user with some benefit from the investigation. The practical implication of this is that the investigator needs to allow time for such interactions to take place, and build this into the evaluation process. It is a good idea to establish ground rules for when observations should be allowed to take place uninterrupted, and when social interactions are to encouraged. Without a careful discussion of these issues, the investigation can suffer from unwanted interruptions which limit the value of the data obtained.
Direct Observation
Using a candid camera raises serious ethical and legal questions.
Direct Observation
constant positions. The visitor should be particularly aware of actions on their part which may prove hazardous to the visually impaired e.g. leaving cupboards and doors in an open position.
The Elderly User
Direct observation of elderly people is in principle not very different from observing members of other user groups. However, if the observation takes place in the users home, one should be aware that many elderly have lived in the same place house or flat for a number of years and may have established routines or habits that help them cope with everyday activities. For this and other reasons some elderly people may not appreciate any significant disruption to their home e.g. the rearrangement the furniture in order to facilitate the observations.
Procedure
Planning
Nature of observations The nature of the problem will decide what kind of observation are required. If the project is in an exploratory phase, the observations might be unstructured with an aim to discover “what is important”. With a more precise set of problems, one should prepare an observational form, allowing the observer to record the occurrence of different behavioural categories. One would normally construct this after a period of unstructured observation.
Example: A product developer may suspect that there is a need for better devices for keeping track of the time of day among people with mental impairment. She decide to visit a group home where four mentally impaired adults live with the support from a team of helpers, one always present. After having made the necessary appointments she is invited to visit the home. The first visit must be used to be introduced to the users and the helper on duty that day.
She may explain that her company makes watches and that she is interested in knowing what kind of watches are needed. She will probably have many reactions to that, which is strictly speaking not raw data from direct observations, but rather should be treated as opinions. However, such opinions may be valuable in their own right. When the residents are used to the observers presence (which may take a day), they will be in the position of being able to observe without causing disruption to the activities of interest.
more…
Direct Observation
The result of this initial unstructured study, may be an impression that the importance of time of day varies for the different participants, e.g. the helpers may actively want the users to be aware of the time, while the users themselves may have very different concerns about this. Some may ignore the problem, while others may be nervous not to keep appointments etc. The observer will also have an impression of how time is communicated in the group, e.g. referring to the hours of the day or to events, like going to school, time for dinner etc.
Based on these first impressions the observer is able to set up the observational categories for a more formal investigation. They can be classified as being either events or states. An event is coded when it starts and ends, a state is coded as being on/off at a certain time.
Some of the event categories could have been:
- Questions about time of day
- Answers to such questions
- Utterances about time of day
- Responses to utterances about time (also lack of response)
- Events triggered by the time of day
- Responses to such events (also lack of response)
- Appointments etc. missed because of an inadequate understanding of time
- Worries about time
With such broad categories, it is not sufficient just to indicate the category, the persons involved and the actual behaviour must be recorded, for example "Peter asks what time it is".
Some of the state categories may be:
- Person x is in a time-critical phase (for example about to take the bus, get dinner ready)
- Person x is waiting for something (description of the situation.....)
- No time critical activities in the group home
Data analysis There are several ways to treat raw data from direct observational studies. The simplest way is to count frequencies and duration’s of different cate- gories of activity. These categories are often the same as those used to focus
Direct Observation
Event observation form
Observation period Date : April 14. From 0815 to 0915
Event Starts Ends
Peter asks Paul about the time 08:
Paul does not answer 08:
Mary tells Peter that it is 0830 08:
Peter says that he must go to school 08:
Mary tells Sara that she must go to the bus 08:
The taxi arrives to take Peter to school, the taxi must wait 08:40 08:
Sara leaves to catch the bus, 08: (bus arrives in 30 mins, Sara needs five minutes to get to the bus stop)
Mary tells Paul that it is 9 o'clock. 09:00 09: She shows him the clock on the wall and explains how the hands point to the time
Example The observations described in the group home example might be sampled several times during the day. Since the activities are obviously not very frequent, one must be assure that observation periods are long enough to cover relevant activities. It could also be expected that the frequency of relevant activities would vary periodically, therefore it was decided to sample activities at irregular times of the day. It was also decided that observation should go on for two days, and that a total of 5 one hours sessions should be sufficient and that they should be distributed differently the two days. Then 10 different starting times were generated (drawn randomly) and distributed so that the first, third etc. was on the first day, and the second, fourth etc. on the second day. This resulted in the raw data forms seen below.
the data capture, but may also be inferred from any wider descriptions recorded. In the previous example “Utterances about time referring to an event rather than to the clock”, might be such an inferred category.
In summarising complex data there is always the danger of missing val- uable information, and that often more full accounts of interactions can provide insights that are missing from summaries. For this reason many researchers prefer to capture more information than they believe they will need. Thus if data analysis suggests some interesting aspects of the inter- actions that are worthy of further investigation, it is possible to go back to the data and re analyse it with such changes in perspective in mind.
Direct Observation
Example From the above example a tabulation of frequencies of some selected categories occurring in the specified periods during one week, might look like this:
From this tabulation it may easily be inferred that Peter and Paul have little understanding of time, and is constantly reminded by the helpers. Sara worries a lot, while Ben likes to tell others what time it is. The helpers very often tells the residents what time it is.
It might have been concluded that Peter, Paul and Sara will benefit from some automatic device that on pre-programmed hours reminds them of the time. Setting up a table of how the time is communicated might have shown that time should be stated in relation to an event, not as an hour of the day. If such a device would decrease Sara's anxiety is still to be seen. However, even in a case like this, with a rather clear conclusion, the total size of the potential user group remains to be assessed.
Asks Is told Tells what Worries about Person about time the time time it is the time
Peter 0 10 0 0
Paul 0 8 0 0
Sara 15 15 0 11
Ben 0 13 0
Helper 1 0 30 5
Sum 16 33 43 16
