← All work

📊 Program Dashboard Research

Idemia

UX ResearchDiscoveryTestingB2BSoftware
📌
A programme, at Idemia, is the execution of a contract: an order, deliverables, a budget, and a review where a programme manager reports to top management. The tool that carried all of it, across four continents, was a spreadsheet.

The question was not whether a spreadsheet is the right tool. It was where the people filling it were losing their time, and which of the things everyone assumed about them were actually true. I ran the study with another UX designer from the team.

67/90 users answered the survey, across four continents
45% filled the dashboard the day before the review
18 prioritised recommendations, with 9 KPIs to judge them on
⏱️
When
February to June 2021
🧑‍💻
Role
UX research, in a pair, from protocol to recommendations
🌍
Reach
67 survey answers and 5 interviews, across 5 regions
⛳
Objective
Find where to make the programme review and the reporting faster: save programme managers' time, cut the time it takes to reach a piece of information, and automate what can be automated.
15 assumptions, written down before the research

The business unit already had a view of the problem, and so did I. So before collecting anything we wrote the assumptions down, fifteen of them, in plain sentences: that filling the finance file automatically would save time, that scrolling in the spreadsheet slows the review, that managers want to be told when a file is updated, that leaders do not use the slide deck the file generates, that the document is updated at the last moment, that managers do not fix errors because they have no time.

At the end of the research each one came back into one of three piles: eight validated, two invalidated, five needing more insight. Two of those five were explicitly marked as needing user tests before anybody could claim them.

Written with the business unit, before the research15assumptionsValidated8filled the daybefore the reviewInvalidated2the tab is filledwithout mistakesNeeds more5a website wouldmake it easierThe two that turned out to be false were both about error: that the tab is filled without mistakes,and that the errors left in it are left there for lack of time. Neither held.A sorted assumption outlives the recommendation it produced. A recommendation on its own does not.
Fifteen assumptions written down before I looked at anything, sorted once at the end. Nothing was added to the list along the way.

That sorting is the deliverable I would keep if I could only keep one. It is what stops a study from being a list of things we already believed, written up with percentages attached.

Quantitative first, then the five conversations

The survey targeted at least 60 answers out of a population of around 90 users, spread across countries. It came back with 67, about three quarters of everyone who uses the tool.

Then five interviews, chosen to spread across regions and seniority rather than to confirm the survey: Africa, Europe, two in Asia-Pacific, Latin America. Quality and programme management roles, with different levels of experience.

The survey says how many. The interviews say why, and they are what turned percentages into something a build team can act on. Four out of five interviewees wanted information that already exists in other tools to arrive in the dashboard on its own, rather than through copy and paste.

We tagged every finding with the ergonomic criterion it belonged to, guidance, explicit control, cognitive load, and with a criticality level, so the recommendations could be sorted by impact on the user's goal rather than by how loudly anyone argued for them.

What the numbers said that nobody wanted to hear

Access was never the problem. Seventy-seven per cent of users had no trouble reaching the file at all.

The problem was what the file had become, and the three figures that mattered sit next to each other.

⚠️
36% said the dashboard contains inconsistencies, such as computed values that are wrong.
41% do not correct the errors they find in it.
45% update their dashboard the day before the review.

Read together, those three describe the actual working situation: a document filled the night before a review, carrying figures some people know to be wrong, that nobody has the time to fix, and on which a report to top management is then built.

And 75% never use the slide deck the file generates automatically, because it is either useless to them or too complex to use. A feature built to save the reporting effort, producing nothing anybody takes to the meeting.

Sixty-seven per cent wanted to be told when a dashboard is updated. Thirty-one per cent said it should simply be organised differently.

The sections they fill and nobody reads

The survey asked two separate questions about every section of the dashboard: how important is it, and is it useless to you. Putting the two answers side by side is what made the result impossible to argue with.

Called importantCalled useless529Streams2024Timelinealso one of the three hardest to fill2021Opportunities1716Change Request1413Banner67 of about 90 users answered. The recommendation was to remove or hide all five.
The five sections the fewest people call important are the same five the most people call useless. Nobody had put the two questions side by side before.

Meanwhile, the sections people wanted automated first were Financials, fifty out of sixty-seven, then Risks, Invoicing and Internal Data. And the hardest to fill were Internal Data, Financials and Timeline.

So the effort was going into the parts nobody uses, while the part everybody depends on was the one being typed by hand the night before.

18 recommendations, and the KPIs to check them against

The recommendations came back in five families, grouped under two ideas: put the tool at the service of the person, and put it at the service of the job. Each one carries the criticality the deck gave it, high, medium or low.

At the service of the person. Lighten the cognitive load.

  • Co-create the alert system with the program managers, rather than deciding for them which alerts they want. High
  • Make the dashboard as simple as possible to fill, because it is updated in a hurry. Medium
  • Delete the work that can be automated away, to kill the copy and paste errors. Medium
  • Centralise and link the files and tools that belong to the same programme. Medium
  • Merge the tools a program manager opens every day. Medium
  • Remove or hide the sections the survey had just shown to be dead weight: Timeline, Opportunities, Streams, Banner, Change Request, the automatically generated deck. Low
  • Reduce the risk of error.

  • Let a programme be tracked in its own local currency, so the finances stop drifting. High
  • Choose an underlying tool less prone to error than a spreadsheet, and guide people as they fill the fields. High
  • Automate everything that can be automated. Medium
  • At the service of the job. Fit the job.

  • Allow foreign currencies, with a conversion that does not touch the main figures. High
  • Add a monthly view of finances to make the follow-up possible. High
  • Open a way for similar programmes around the world to stop solving the same costly problem separately, to be confirmed by a later survey. High
  • Guide the person.

  • Give hints on how to fill each field. High
  • A simplified dashboard for newcomers, with training or mentoring to go with it. Medium
  • Reorder the sections to cut the scrolling in both directions, the order itself to be settled in a workshop. Medium
  • Bring the most consulted sections to the front: highlights and lowlights, financials, invoicing, contractual milestones. Low
  • Make it adaptable.

  • A separate view when a sub-project appears inside a programme, next to the global one. Medium
  • Move off the spreadsheet to something more visual, modular and personal. Low
  • 18 recommendations, rated on the deckHighMediumLowAt the service of the personAt the service of the jobLighten the cognitive load6the biggest family, mostly mediumReduce the risk of error3Fit the manager's job3all three rated highGuide the person4Make it adaptable2Seven high, eight medium, three low.Each level is the one the deck carries next to the recommendation, not one I added afterwards.
    The three recommendations about fitting the manager's real job are the only family where every item came out high. The longest family, the one about cognitive load, is mostly medium.

    Two things I would defend in that list. The recommendations are labelled as intermediate working trails, meant to feed design workshops, not as final answers, because the research did not go far enough to claim them as such. And the list did not stop at advice: it came with nine UX KPIs to judge it on afterwards, in two families. Efficiency: the time to fill the dashboard, the share of users keeping a personal file to track finances on the side, the share making mistakes, the share needing a colleague's help to fill it. Satisfaction: the share who find the tool complex, and the share who believe it holds values that are wrong.

    Key Results


    🚀
    67 answers out of roughly 90 users, five interviews across five regions, fifteen assumptions sorted into eight validated, two invalidated and five unproven, and 18 prioritised recommendations handed to the build.

    The finding that reframed the project: the tool was not hard to reach, it was hard to trust. Filled the day before the review by 45% of its users, carrying values 36% know to be wrong, with 41% not correcting them.

    Outcomes
    This is the project where I saw both sides of the same job. Here I produced the research and handed the recommendations to the build. At Pernod Ricard, a year later, I was the one building from somebody else's research, which is why I now write a handover as though I will never be in the room to explain it.

    It is also the one that taught me to ask about importance and uselessness separately. Asking only what matters gets you a list where everything matters.

    Constraints

    The users were spread over four continents and several time zones, so the interviews were run remotely, with each person sharing their own file rather than a prototype.

    And the study stopped where its mandate stopped. The recommendations are working trails for workshops, two of the fifteen assumptions still needed user tests, and the choice of what the dashboard should actually run on was left open rather than decided on the strength of a survey.