Overview
A popular technique from the field of user interface testing is to record test subjects "thinking aloud":
- Think aloud protocols involve participants thinking aloud as they are performing a set of specified tasks. Users are asked to say whatever they are looking at, thinking, doing, and feeling, as they go about their task. This enables observers to see first-hand the process of task completion (rather than only its final product). Observers at such a test are asked to objectively take notes of everything that users say, without attempting to interpret their actions and words. Test sessions are often audio and video taped so that developers can go back and refer to what participants did, and how they reacted.
[http://en.wikipedia.org/wiki/Think_aloud_protocol]
A "brainlog" is the written equivalent: instead of speaking, the test subject types in a log of their thoughts as they perform a task.
A typed brainlog is naturally not as rich an information source as a videotaped think-aloud session. It is presumably also a less accurate narration of the subject's thought process, since typing is more laborious and intrusive than speaking. However, a brainlog can be recorded nearly anywhere and at any time, it can be distributed and archived via mailing lists, and it is easy to consume quickly.
Using brainlogs to test API design, documentation, and code clarity
Apache in general and Lucy in particular place a high value on API simplicity and code clarity. We use brainlogs to judge how transparent our codebase is and to help us improve.
Typically, a user or developer will record a brainlog while exploring an API or reviewing a section of code for the first time. The authors of the materials being explored then examine the contents of the brainlog and evaluate how effectively their materials guided the test subject.
The exercise is directly analogous to the reviewing the interface of a web page to see whether the design is guiding visitors along the path that the designers intended.
The "curse of knowledge"
Authors are uniquely unqualified to gauge how consumable their work is. Of course it makes sense to them – they wrote it!
In contrast, newbies make good test subjects, as they are not yet afflicted by the "curse of knowledge":
- And that brings us to the villain of our book: The Curse of Knowledge. Lots of research in economics and psychology shows that when we know something, it becomes hard for us to imagine not knowing it. As a result, we become lousy communicators. Think of a lawyer who can't give you a straight, comprehensible answer to a legal question. His vast knowledge and experience renders him unable to fathom how little you know. So when he talks to you, he talks in abstractions that you can't follow. And we're all like the lawyer in our own domain of expertise.
- Here's the great cruelty of the Curse of Knowledge: The better we get at generating great ideas – new insights and novel solutions – in our field of expertise, the more unnatural it becomes for us to communicate those ideas clearly. That's why knowledge is a curse. But notice we said "unnatural," not "impossible."
- Chip Heath and Dan Heath, "Made to Stick: Why Some Ideas Survive and Others Die"
Innocence is precious: once you have become familiar with a source, any brainlog you might contribute no longer reflects the experience of those who are coming to the material for the first time. Therefore, if you are going to record a brainlog, you should do so right away.
Editing brainlogs
It you make a "mistake" during testing, it may be tempting to edit the brainlog after the fact to conceal or minimize it. Please don't!
If multiple test subjects make the same "mistake", that indicates that there is a flaw in the design that needs to be corrected. In fact, that sort of pattern is exactly what UI testing is designed to reveal.
On the other hand, it's probably not a good idea to publish a brainlog that contains egregiously inflammatory material, even if it's an accurate record of your thoughts. Before you hit "send" – especially for the first brainlog you write – step away for a few hours or a day, and consider whether you might want to swap out certain passages for placeholders like "[intemperate rant about XXXXXX here]".
Evaluating brainlogs
When evaluating a brainlog, there are two things to bear in mind.
First, it's important to avoid blaming the user for "mistakes" documented by the brainlog. The brainlogger is performing a valuable service precisely by revealing where they went wrong or right, and they are doing a job that you cannot do by yourself. Instead of criticizing the path they took, consider how you might modify your source material so that the next user doesn't make the same "mistake" – even if you think it was a "dumb" mistake.
Second, brainlogs are raw materials by nature, rather than carefully prepared constructive criticism. A critique is a contribution, even if it is impolitic. If you feel miffed after reading a brainlog, consider it a challenge to rise above and extract every last drop of value from it.