OECD Says It’s User-Driven. But Where Are the Users?

The OECD Digital Government Index measures how governments are progressing in their digital transformation. One of its six dimensions is called User-Driven.

That sounds reassuring. Until you ask a rather simple question:

Where are the users?

The question emerged during a DYRÍGO ThinkLab investigation into the OECD Digital Government Index. We were not trying to prove that the index was wrong. We wanted to understand what it actually measures and whether the language used to describe the results matches the underlying evidence. That distinction turned out to matter.

A high score does not necessarily mean a good digital experience

Portugal scores approximately 0.94 on the OECD’s User-Driven dimension. Estonia scores approximately 0.72. Spain approximately 0.73. If those numbers are read as a measure of how user-driven digital government actually is, Portugal appears to outperform Estonia substantially.

But when we compared those capability scores with independent evidence about experienced administrative services, a very different picture appeared. Portugal reported around 52% satisfaction among recent users of administrative services. Estonia reported above 80% satisfaction.

That does not prove that the OECD numbers are wrong. It does, however, raise a much more important methodological question:

What exactly is being measured?

The answer appears to be primarily the existence of capabilities, policies, standards, mechanisms and institutional arrangements associated with user-driven government. Those things matter. But having the machinery for user-driven government is not the same as citizens actually experiencing government as user-driven.

CAPABILITY ≠ EXPERIENCE

And that distinction becomes important when the resulting scores are presented as country performance.

The missing part of the journey

A citizen does not experience a policy framework. A citizen experiences a journey. They need something. They try to understand what is required. They find a website. They identify the correct procedure. They collect documents. They authenticate themselves. They make an appointment. They submit something. They wait. Something may go wrong. They try to recover. Eventually, hopefully, they achieve the outcome they came for.

That is the end-to-end customer journey. And this is where our investigation found an interesting gap.

OECD’s own evidence shows that only 10 of 36 OECD countries report standardised mechanisms or guidance for measuring the transaction costs imposed on users, including time, steps, documents, handovers and repeat contact when something goes wrong.

That does not make the framework useless. It means the framework is looking at only part of what determines whether a digital public service actually works for the person using it.

Websites are part of the service too

This becomes particularly visible when you stop looking at the architecture of government and start trying to use it. A website can technically contain the right information and still be extraordinarily difficult to use. A process can be digitally available while sending the user through several organisations. An error message can arrive only after the user has already completed the wrong step. A system can send someone into a loop from which there is no obvious recovery. And an appointment system, cita previa, can technically exist while access to an appointment remains the actual obstacle.

From the organisation’s perspective, all the individual components may be functioning. From the citizen’s perspective, the service may still not work. That difference matters.

Because if we measure only whether governments have created the mechanisms for user-driven services, we can end up measuring the design of the system while missing the experience produced by that system.

There is another problem with rankings

Rankings do more than describe reality. They can also legitimise it.

A country that receives a high User-Driven score can point to that ranking as evidence that its digital government is performing well for users. That creates a methodological responsibility.

If the underlying indicator primarily measures institutional capability rather than realised user experience, the ranking can unintentionally communicate more than the evidence supports.

The problem is therefore not that OECD should stop measuring digital-government capability. Quite the opposite. Those capabilities are important and worth measuring. The problem is the label and the interpretation.

A measure of user-driven capability should not quietly become a measure of user experience.

And what happens when the user asks a question?

During our investigation, another uncomfortable parallel emerged. We had questions about what the User-Driven ranking actually demonstrated. So we shared our analysis and the underlying report with the OECD and explicitly invited them to challenge our findings, correct us, or show us evidence we may have missed.

That is not just about right of reply. It goes directly to the subject we were investigating.

The user asks a question. The system provides information. But the question itself is not answered.

It is a familiar problem in end-to-end customer journeys. An organisation can publish information, provide procedures, build dashboards and formally offer all the necessary channels. But if the user is left with a specific question and no usable answer, the journey has not been completed for that user.

There is an additional responsibility here. The OECD is funded by its member countries, ultimately through public money. The people using public services are therefore not simply an external audience looking in. They are also, indirectly, helping to fund the institutions that assess how those services perform.

That does not mean every question must receive the answer the user wants. But shouldn’t a user at least be able to expect a substantive answer to a substantive question?

Our question to the OECD was simple: Show us where our analysis is wrong.

We received no substantive answer to that question.

That does not automatically make our analysis correct. Nor does it mean the OECD is obliged to agree with our conclusion. But it makes the central question of this investigation unusually tangible: When can an organisation call something user-driven? When the mechanisms are in place, or when the user is actually helped forward?

User-driven ultimately begins with the user.

So what should happen?

Our conclusion is unusually simple.

Take the current standalone User-Driven country ranking offline.

Not because the underlying work has no value. But because the performance-like ranking risks implying something the current evidence does not sufficiently establish.

Keep measuring the capabilities. Make clear what those measurements do and do not demonstrate. Then add the missing layer.

Investigate what happens when real people actually try to complete real public-service journeys, from the moment a need arises until the human outcome is achieved. Look for the journeys that work exceptionally well. And look just as seriously for the ones that do not.

Measure the friction. Measure recovery. Measure the hand-offs between organisations. Measure whether digital availability becomes actual usability. And, above all, ask the people who have to use the system.

We have shared our findings and questions with the OECD and invited them to challenge the analysis. That matters. Research becomes stronger when the organisation being studied gets the opportunity to show what we may have missed.

The full DYRÍGO ThinkLab investigation sets out the evidence, methodology, limitations and unresolved questions behind this analysis. But the next stage is different.

Now we go looking for the users.