← All work

๐Ÿชช Verinium Access Terminal

Idemia

UX ResearchTestingField ResearchHardwareSoftwareApplication
๐Ÿ“Œ
Verinium is a secure biometric authentication product: you enrol your face in a phone app, then authenticate at a terminal to get through your company's door, holding your phone to an NFC reader. It exists to prove that the biometric check actually happened.

The whole product rests on one sequence, done once, by somebody who has never seen it. On the last step of that sequence, one tester out of six got through on their own. The technology worked. The sequence did not.

1/6 testers who completed the final step unaided, before the fixes
16 testers across two guerrilla waves, on the real technology
26 recommendations across 9 screens, one aimed at the technology itself
โฑ๏ธ
When
Two test waves, July and September 2021
๐Ÿง‘โ€๐Ÿ’ป
Role
UX Designer in R&D, from protocol to handover
๐Ÿงช
Testers
16 across the two waves, recruited guerrilla style
โ›ณ
Objectives
Check that the instructions are understood, that the app reassures people about what happens to their data, and that somebody meeting the product for the first time can enrol and authenticate without help.
The product, and the one moment it can fail

An access terminal is not an app you learn. You meet it in a corridor, with people behind you, on a morning when you are thinking about something else. The enrolment happens once. If it fails, you go and find somebody with a badge, and the product has lost its argument.

The sequence runs: a tutorial, the terms and conditions, permission requests, a selfie capture, a confirmation screen, a screen telling you your identity is valid, switching NFC on, then at the terminal a face capture and the phone presented to a reader.

Nine moments where a person who has never seen any of it can stop.

Testing on the real technology, not on a mock-up

I put the prototypes on the working technology rather than on a screen mock-up, because face capture fails in ways no prototype reproduces. A prototype always recognises you. The real camera does not.

Two guerrilla waves, people recruited where they were rather than through a panel, six testers then ten. I wrote the protocol, ran the sessions, kept an observation grid per tester, and recorded the audio with their consent so the recommendations could quote what was actually said.

One bias I wrote into the report rather than around: the prototype under test was the English version, and the tests ran in France. Two testers did not read English, and that shaped what they could and could not do.

What the testers showed
The nine steps, in the order the testers lived themTesters concernedTutorialtried to capture their face during the tutorialTermsgave their consent, none opened the termsPermissionsrefused access to see if the app still workedof 4Selfie capturecapture failed on a black face behind a mask, gave upConfirmrestarted the capture, nothing said it had workedIdentity validread the word awesome as terribleNFC activationdid not know what NFC on a phone isFace detectionthe terminal fired before they were in positionNFC authenticationcompleted this step unaidedThe engineers had built the technology and it worked. What failed was the sequence.
One tester in six reached the end on their own. Every other number above is a step they were not guided through.

People understood the promise. Five out of six understood they would have to open the app and show their face. Five out of five said they trusted the technology with their face, and the average score they gave the experience was 4.5 out of 6. The product was not rejected. It was just not usable yet.

Nobody reads what you write. Six out of six accepted the terms and conditions without opening them, while understanding perfectly well that there were terms. One tester's suggestion was better than ours: summarise the few points about data protection on one screen, and let people click through if they want more.

The capture gives no feedback. Two testers out of six restarted the capture because nothing told them the photo had worked. Two complained that the terminal took the picture before they had time to stand properly. One read the word "awesome" on the success screen as a bad sign.

NFC means paying. Three testers out of six did not know what NFC was on a phone, and read it as something to do with contactless payment. Three clicked "turn on" and then stopped on the system screen that followed without pressing the switch.

And the last step. Three out of six hunted for the NFC reader on the terminal and struggled to find it. One got through unaided.

The failure that was not an interface problem

One tester could not enrol at all. The capture failed, again and again, on a black face behind a mask. He gave up, because nothing on the screen told him why it was not working, and he ended up tapping the phone screen to try to take the photo himself.

Four out of six kept their mask on for the capture, so this was not a corner case of somebody doing it wrong. It was the product working for some faces and not others, in a building where everybody was masked.

That one went into the report as a recommendation to the technology, not to the interface: make the capture work on a black face behind a mask, or ask every user to remove the mask on the capture screen so nobody is singled out by a failure they cannot explain.

It is the finding I am most glad we ran these tests on the real camera. On a mock-up, every face works.

The A/B test that did not say what I wanted

The NFC step was the worst one, so we tested two wordings for the prompt, five testers each.

๐Ÿ“ˆ
Four out of five got it on the first attempt with a plain "Yes / No", against two out of five with "Close / Settings". But the three who failed on the second version all succeeded on the next attempt, and nobody in that group gave up, while one tester did in the first.
The same pop-up, two wordings, five testers eachYes / No4 of 5through on the first try4 of 5through in the endClose / Settings2 of 5through on the first try5 of 5through in the endfirst tryafter a failuregave upThe two wordings were shown to two separate panels. Not a controlled comparison.
First tries went up. Completion did not. I wrote at the time that no notable difference in behaviour could be observed between the two wordings.

So the wording moved the first attempt and not the outcome. My own line in the file that day: no notable difference in behaviour between the two wordings.

I kept it in the report as it was, because the half of the result that looked good would have sold a fix that does not fix anything. What actually blocked people was underneath the wording: they did not know what NFC was, or why a product that had just recognised their face still needed it. No label solves that. An explanation and an error screen with a way out do.

26 recommendations, and a documented handover

The report came back with twenty-six changes, each tied to the observation that produced it. They are not ranked. They follow the journey, screen by screen, because that is the order in which the failures happened.

Tutorial

  • Make it obvious that the tutorial has several parts and that you swipe through it: a next button, or a glimpse of the illustration to come.
  • Add a first step showing what enrolling a face actually involves.
  • Add a step at the very start to choose the language.
  • Terms

  • Rewrite the terms so the underlined parts carry the points that matter, with a way to click through for the rest: that the app only reaches the camera and the photo library, that file access is needed to retrieve the photo, that location is used only when the phone is within about ten metres of the reader.
  • Make people tick a box before moving on, so they know they have agreed to something.
  • Add a GDPR certification from an accredited body, if one can be had.
  • Say plainly that the data does not leave the company.
  • Explain why face and NFC together are what protect it.
  • File, photo and camera permissions

  • Remove the "don't ask again" option, which can lock somebody out of the app for good if they pair it with Deny.
  • If it cannot be removed, fix the bug that then stops Allow from being clicked at all.
  • Offer, to those who want it, access to files, location and capture only while the app is open.
  • Location permission

  • Put the screen into production to test it, if tracking location turns out to be needed.
  • Selfie capture

  • Fix the technology so it recognises a Black face behind a mask, or ask everybody to take the mask off on the capture screen.
  • Add a countdown, so the person knows the capture is under way.
  • Confirm

  • Add a green tick saying the capture was taken and is good enough.
  • Add a line under the photo confirming the photo is correct.
  • Rework the buttons so continue and recapture read as two different levels.
  • NFC activation

  • In the pop-up: explain what NFC is, and swap "turn on" and "close" for "yes" and "no".
  • Add an error screen saying NFC is required, with a button that takes you straight to it.
  • In the settings screen: guide the person to the switch they have to press.
  • Face detection

  • Add a three second delay with a visible countdown on the terminal, so people register that they have been recognised.
  • NFC authentication

  • Put an NFC logo on the physical reader, and on the illustration of it.
  • Move the physical reader under the terminal, in the centre, exactly where the testers had tried to put their phones.
  • Say on the terminal that the app must be open and NFC on for the reader to work.
  • Say that the phone goes screen side against the reader.
  • Outside the flow

  • Offer another way in for somebody whose phone is flat, left at home, or not compatible.
  • 26 changes, in the order of the journeyLed the reportTutorial3Terms5where nobody failedFile and camera permissions3Location permission1Selfie capture2the countdownConfirm3the green tickNFC activation3Face detection1NFC authentication4the reader moved under the terminalOutside the flow1Nine screens, and one recommendation that belongs to no screen at all: a way in for a phone left at home.
    The screen where nobody failed, the terms, is the one that produced the most changes. Six testers out of six gave their consent there, and not one of them opened the terms.

    The deck ends on a question rather than an answer: whether a specific document has to be signed if the technology is installed at Idemia itself. I left it in rather than resolve it, because it was not mine to resolve.

    I closed the project with a documented handover rather than a slide deck, because the team that inherited it had never sat in a session. A deck says what we found. A handover says what to do with it when we are gone.

    Key Results


    ๐Ÿš€
    One tester in six completed the final step unaided, and the study came back with twenty-six changes spread across the nine screens of the sequence, one of them aimed at the technology rather than the interface.

    Sixteen testers across two guerrilla waves, an A/B test reported with its negative result intact, and a documented handover to the team that took the product on.

    Outcomes
    The value of this one is not the interface changes, it is where they came from. Every recommendation is attached to a number of testers and a moment in the session, so the engineers could argue with the finding rather than with me.

    And the product was never the problem people assumed. The face recognition worked. What failed was a phone, a reader, and a step that asked people to know a piece of vocabulary nobody outside the building uses.

    Constraints

    The prototype was in English and the testers were in France, which cost us two of them on wording questions. I reported it as a bias rather than smoothing it over.

    In R&D the UX team billed its time to the project teams, so the study ran in weeks. There was no third wave to confirm the fixes, which is why this one hands over its observation grids rather than its conclusions.