P004

P004

Preliminary Questions

Before we look at the device, I'd like to ask you just a few quick questions. First, what's your occupation?

• Grad student

Roughly what do you know about your electricity use?

• Don't know much, just came to the states

Do you know your bill, kWh?

• $14
• Don't know kWh

How do you feel about learning about your electricity use?

• Great, you can learn how to cut costs

Part 2: Looking at the Device and Unaided Exploration (i.e., see what the user can do by themselves)

First, I'm just going to ask you to look at this device and tell me what you think it is, what strikes you about it, and what you think you would do first.

For now, don't actually do anything. Just tell me what you would do.

And again, as much as possible, it will help me if you can try to think out loud so we know what you're thinking about.

Let them interact and think out loud

--When done: If you had to take a guess, what do you think it is about?

Ok. Now, if you were at home, what would you do first?

...continue

Notes (just let them explore, but take notes on these items):

1. How much can the person do with no instruction; just “use the device”?

• Didn't seem to know what the first page was talking about
• Seemed to have a grasp on what the other slides were talking about

1. Without any instruction, what do they think the purpose of the device is, the features are, the buttons do?

• Had a basic understanding of the remote, however, he mentioned he would like to see a manual for the remote

1. Can the person tell the state of the device just by looking at it?
2. Can the person tell the action alternatives just by looking at the device?

• Yes

1. What is the mental model of the participant? Does this match the designer's model and the system image?

• Yes, but not the goal-setting page
• We might want to think about making goal-setting appear after usage information so that they have a reference point to see what a 10% reduction goal would look like

1. What are the errors people make? How do we fix them? Are they detectable, minimal consequences, and reversible?
2. What are the mappings between controls and functions?
3. What is the gulf of execution? What is the difference between intentions and allowable actions?
4. What is the gulf of evaluation? What is the amount of effort required to interpret the physical state of the system and determine whether intentions have been met?
5. Can the person determine the functions of the device unaided?

• Yes

1. Can the person tell what actions are possible unaided?

• Needed to clarify that they can navigate from the calendar to the bar graph

1. Can the person determine the mapping between intentions and physical movement unaided?
2. Can the person perform intended actions unaided?

• Yes

1. Can the person tell if the system is in the desired state unaided?
2. Can the person determine mapping from system state to interpretation unaided?
3. Can the person tell what state the system is in unaided?
4. Can the person determine the relationships between actions and results?
5. Can the person determine the relationship between the controls and their effects?
6. Can the person determine the relationship between the system state and what is visible?
7. Can we adjust the arrangement so as to reduce the number of labels while improving c comprehension (i.e., can we find natural mappings)?
8. Can people muddle through without reading the instructions or help from the experimenter?

Part 3: Goal-directed tests

1. Can you think of something you'd like to learn from the device? If you wanted to learn that, what would you do?

• Weekly consumption

1. Can you access the calendar?

• Yes, from the bar graph

1. Can you show a bar graph of monthly consumption?

• Yes

1. What is the monthly bill?

• $90

1. How much electricity is being used right now?

• Are you talking about aggregate?
• Can tell you each day's usage

1. What is the current price of electricity?

$.13

1. Can you switch from calendar to bar graphs and back?

• Can't switch from calendar to graph
• Don't know what the options on the button really do
• Not sure if the arrows navigate across the days on the calendar or switch to the graphs

1. How many kWh were used yesterday?

• 22.72kWh

1. How many kWh are projected to be used tomorrow, next week, month...

$140~

1. Can you switch from bar graphs to pictures?
2. Can you switch from pictures to bar graphs?
3. Can you switch from calendar to pictures?
4. Can you switch from pictures to calendar?
5. Can you switch from bar graphs to calendar?
6. Can you switch from calendar to bar graphs

Part 4: Trunk test

Have participant close eyes and take them to randomly selected pages (actually use a random number generator for this). Can they answer the following questions?

1. What is this?
2. What page am I on? (page name)
3. What are the major sections of this site? (sections)
4. What are my options at this level? (local navigation)
5. Where am I in the scheme of things? (“you are here indicator”)
6. How can I get back to pictures and the default page?

Part 5: In the dark test

Turn off the lights. Secretly give them the control upside-down. Can they still use the device?

Part 6: Dual task

Can participants dual task (e.g., n-back) and also use the IHD?

Part 7: Destructive Testing

Can they break it? Not physically.

Part 8: Open-ended suggestion period

• Even with comparison graphs, you still don't know what the contributions to consumption are
• Expect weekly or even monthly comparisons to be the most useful
  • Should be set at default rather than daily
  • Better control of usage knowledge because daily doesn't give you a big enough picture
• Maybe you can make the comparison mode the default display
• Do you want one?
  • Maybe depending on the cost
  • WTP $50, but depends on the functionality
  • Would not be willing to pay a subscription

Navigation

• Go back to 2012/07/06
• Notes for Technology Usability Testing (IHD_field_trial)
• Go back to Technology Usability Testing
• Go back to IHD_field_trial