📊 Program Dashboard Research
Idemia
The question was not whether a spreadsheet is the right tool. It was where the people filling it were losing their time, and which of the things everyone assumed about them were actually true. I ran the study with another UX designer from the team.
February to June 2021
UX research, in a pair, from protocol to recommendations
67 survey answers and 5 interviews, across 5 regions
Find where to make the programme review and the reporting faster: save programme managers' time, cut the time it takes to reach a piece of information, and automate what can be automated.
15 assumptions, written down before the research
The business unit already had a view of the problem, and so did I. So before collecting anything we wrote the assumptions down, fifteen of them, in plain sentences: that filling the finance file automatically would save time, that scrolling in the spreadsheet slows the review, that managers want to be told when a file is updated, that leaders do not use the slide deck the file generates, that the document is updated at the last moment, that managers do not fix errors because they have no time.
At the end of the research each one came back into one of three piles: eight validated, two invalidated, five needing more insight. Two of those five were explicitly marked as needing user tests before anybody could claim them.
That sorting is the deliverable I would keep if I could only keep one. It is what stops a study from being a list of things we already believed, written up with percentages attached.
Quantitative first, then the five conversations
The survey targeted at least 60 answers out of a population of around 90 users, spread across countries. It came back with 67, about three quarters of everyone who uses the tool.
Then five interviews, chosen to spread across regions and seniority rather than to confirm the survey: Africa, Europe, two in Asia-Pacific, Latin America. Quality and programme management roles, with different levels of experience.
The survey says how many. The interviews say why, and they are what turned percentages into something a build team can act on. Four out of five interviewees wanted information that already exists in other tools to arrive in the dashboard on its own, rather than through copy and paste.
We tagged every finding with the ergonomic criterion it belonged to, guidance, explicit control, cognitive load, and with a criticality level, so the recommendations could be sorted by impact on the user's goal rather than by how loudly anyone argued for them.
What the numbers said that nobody wanted to hear
Access was never the problem. Seventy-seven per cent of users had no trouble reaching the file at all.
The problem was what the file had become, and the three figures that mattered sit next to each other.
41% do not correct the errors they find in it.
45% update their dashboard the day before the review.
Read together, those three describe the actual working situation: a document filled the night before a review, carrying figures some people know to be wrong, that nobody has the time to fix, and on which a report to top management is then built.
And 75% never use the slide deck the file generates automatically, because it is either useless to them or too complex to use. A feature built to save the reporting effort, producing nothing anybody takes to the meeting.
Sixty-seven per cent wanted to be told when a dashboard is updated. Thirty-one per cent said it should simply be organised differently.
The sections they fill and nobody reads
The survey asked two separate questions about every section of the dashboard: how important is it, and is it useless to you. Putting the two answers side by side is what made the result impossible to argue with.
Meanwhile, the sections people wanted automated first were Financials, fifty out of sixty-seven, then Risks, Invoicing and Internal Data. And the hardest to fill were Internal Data, Financials and Timeline.
So the effort was going into the parts nobody uses, while the part everybody depends on was the one being typed by hand the night before.
18 recommendations, and the KPIs to check them against
The recommendations came back in five families, grouped under two ideas: put the tool at the service of the person, and put it at the service of the job. Each one carries the criticality the deck gave it, high, medium or low.
At the service of the person. Lighten the cognitive load.
Reduce the risk of error.
At the service of the job. Fit the job.
Guide the person.
Make it adaptable.
Two things I would defend in that list. The recommendations are labelled as intermediate working trails, meant to feed design workshops, not as final answers, because the research did not go far enough to claim them as such. And the list did not stop at advice: it came with nine UX KPIs to judge it on afterwards, in two families. Efficiency: the time to fill the dashboard, the share of users keeping a personal file to track finances on the side, the share making mistakes, the share needing a colleague's help to fill it. Satisfaction: the share who find the tool complex, and the share who believe it holds values that are wrong.
Key Results
The finding that reframed the project: the tool was not hard to reach, it was hard to trust. Filled the day before the review by 45% of its users, carrying values 36% know to be wrong, with 41% not correcting them.
Outcomes
This is the project where I saw both sides of the same job. Here I produced the research and handed the recommendations to the build. At Pernod Ricard, a year later, I was the one building from somebody else's research, which is why I now write a handover as though I will never be in the room to explain it.
It is also the one that taught me to ask about importance and uselessness separately. Asking only what matters gets you a list where everything matters.
Constraints
The users were spread over four continents and several time zones, so the interviews were run remotely, with each person sharing their own file rather than a prototype.
And the study stopped where its mandate stopped. The recommendations are working trails for workshops, two of the fifteen assumptions still needed user tests, and the choice of what the dashboard should actually run on was left open rather than decided on the strength of a survey.