MYP Design Collecting and Interpreting Test Data Notes

Record As You Go, Not From Memory

1. Test data is what you write down while the test is happening, and memory is a poor substitute an hour later.
2. Draw up the recording sheet before the first run, so you are filling gaps rather than inventing columns halfway through.
3. Write the raw value you actually observed, such as 3.4 seconds, rather than a tidied version like "about 3".
4. Note the odd things too: the battery was low, the third tester had already seen the app, the floor was carpet instead of vinyl.
5. Date every sheet, because criterion D evidence is stronger when a moderator can see the testing came before the changes.

Tables Turn Scribbles Into Evidence

1. Use one table per test, with the thing you changed down the left and each repeat in its own column.
2. Put the unit in the column heading so the cells hold bare numbers and stay readable.
3. Keep every value to the same number of decimal places, since 3.4 and 3.40 in one column look like two different precisions.
4. Leave a notes column, because "lid popped but the box stayed closed" explains a result that a number cannot.
5. Qualitative results need tables as well: user, task, finished yes or no, time taken, comment.
6. Never delete a result you dislike, and mark anything you exclude as an anomaly with the reason written next to it.

Repeats: One Run Is an Anecdote

1. A single run can be luck, so repeat each test at least three times under the same conditions.
2. Repeats show the range of results, and a range of 2.1 to 9.8 seconds is a finding by itself.
3. If repeats disagree wildly, something is uncontrolled, so check the tester, the surface and the starting position.
4. For user testing, a repeat means another user, because the same person's second attempt is never a first impression.
5. Five users is a sensible minimum for a school project, and you say so honestly instead of claiming the result speaks for everyone.

Averages Give You the Middle, Not the Story

1. The mean adds the results and divides by how many there were, which suits times, masses and distances.
2. The mode is the most common answer, and it fits rating scales and yes or no counts better than a mean does.
3. One extreme value drags a mean badly, so 2, 2, 3 and 40 seconds averages to 11.75 seconds even though most users took about 3.
4. Quote the range next to the mean so a reader sees the spread as well as the middle.
5. Round sensibly, since a mean of 3.466666 seconds from a stopwatch you read to 0.1 s should be written as 3.5 s.
6. Percentages need their denominator, so write "4 out of 5 users (80%)" rather than 80% on its own.

Charts Earn Their Space Only If They Show Something

1. A bar chart compares separate things, like the time each of five users took to finish the sign up task.
2. A line graph only makes sense when the horizontal axis is a continuous scale, such as load in grams against bend in millimetres.
3. Label both axes with the quantity and the unit, or the chart says nothing the table did not already say.
4. Start the scale at zero, because a chopped axis makes a 4% difference look like a landslide.
5. Pie charts suit one set of parts that add up to a whole, such as which of four features users picked as most useful.
6. If the chart just repeats the table without making a pattern easier to see, keep the table and drop the chart.

Reading the Data: Hold Each Result Against the Threshold

1. Interpretation starts by putting each result beside the success threshold you set when you designed the test.
2. Say what the number means in plain words, so 8 of 10 drops held becomes "the lid meets the point in most falls but not all".
3. Look for patterns across tests, such as every failure happening at the same corner joint.
4. Keep what the data shows separate from what you think caused it, and label the cause as your explanation.
5. Watch for results that answer a different question, like a fast load time measured on a page the browser had already cached.
6. State how confident you are, since five users and three repeats supports a modest claim rather than a certainty.

A Failed Test Is Still a Result Worth Having

1. Data showing the solution missed the specification is evidence, and writing it earns credit.
2. Record how far short it fell, since 9.2 seconds against a 6 second target is a different problem from 45 seconds.
3. Ask whether the fault sits in the product or in the test, because a snapped clamp may have been loaded with twice the mass the client would ever use.
4. Failures point straight at your improvements, so a clear failure is often the most useful data you collect.