SUNANDO ROY – On Banking, Finance and Society

Supervisory authorities are investing steadily in Sup Tech — for data collection, risk assessment, document analysis, early-warning systems, market-conduct surveillance and supervisory workflow. The harder question is what those investments are worth.

The most instructive available evidence comes from the Financial Stability Institute (FSI). In a 2024 survey of 32 supervisory authorities, close to three quarters reported having successfully deployed SupTech tools, but only about half reported having tools that had become critical to supervision — that is, indispensable to carrying out a supervisory process (Prenio, 2024). The gap between those two numbers is the measurement problem in a single statistic. Deployment is not value.

The FSI’s earlier work on insurance supervision found the same thing from the other direction: supervisors did not yet have clear key performance indicators or methodologies for assessing the effectiveness and efficiency of their tools, and typically fell back on user feedback, with some reaching for a more objective measure such as time saved (Garcia Ocampo, Lehtmets, Pandey and Prenio, 2022). Counting applications built, dashboards launched or users registered tells a supervisor very little.

The value proposition is not technology deployed. It is supervisory capability improved.

From usage to supervisory outcomes

A tool can be heavily used without improving supervision. Conversely, a specialised tool used by one inspection team can be worth a great deal if it surfaces a material risk months earlier than conventional review would have.

That suggests a simple chain:

Data → Analysis → Insight → Supervisory action → Risk mitigation

Measurement should follow the same chain. Most SupTech measurement stops at the first two links.

The IMF identifies five broad areas of SupTech application — data collection, processing, storage, analytics and data products — and notes that faster, more flexible data capture improves off-site monitoring and supports earlier detection of potential risks (Bains and Wu, 2023). For market-conduct supervision, the World Bank identifies four use cases: automated data collection; advanced data validation, analysis and visualisation; platform and database integration; and data management and storage (Randall and Lopes, 2018). The FSI’s own survey work finds authorities concentrating on four areas in practice: data visualisation, regulatory reporting, financial risk assessment and supervisory automation (Prenio, Pustelnikov and Yeo, 2024).

These are capability categories. They become value only when they change what supervisors do.

1. Efficiency

The easiest dimension to measure, and the one authorities already use.

If an inspection team previously spent 400 hours reviewing financial statements, board papers and risk reports, and a document-analysis tool reduces that to 150:

Hours saved = baseline processing time − SupTech processing time

Efficiency value = hours saved × fully loaded supervisory cost per hour

Two cautions. First, the saving is real only if the hours are redeployed; otherwise it is slack, not value. The stronger case for SupTech is reallocation — moving supervisors from data preparation towards analysis, challenge and judgment — rather than headcount reduction. Second, efficiency is the dimension most likely to be measured because it is the easiest, which is precisely why it should not be the only one.

2. Analytical quality

A tool that processes quickly but adds little insight has limited supervisory value. Relevant indicators include anomalies correctly identified, material issues that conventional review missed, incremental coverage of entities or transactions, and the share of alerts subsequently judged to be genuine supervisory concerns.

A workable summary measure is:

Alert precision = material alerts / total alerts generated

Its complement is the false discovery rate — the share of alerts that turn out to be noise. (This is not the same as a false-positive rate in the classification sense, which is measured against the population of non-events, not against alerts raised. Authorities should be explicit about which they are reporting.)

A system generating 10,000 alerts is not superior to one generating 200 relevant ones. This matters most for AI, machine-learning and NLP tools, where calibration is the central difficulty. The FSI’s stocktake of 71 prudential SupTech tools across 20 jurisdictions found that more than half analysed mainly qualitative information, and identified data quality, parameter calibration and supervisors’ data science skills as the principal constraints on wider adoption (Beerman, Prenio and Zamil, 2021).

3. Early warning

The largest potential benefit of SupTech is time. Conventional supervision often identifies deterioration once it is already visible in ratios, financial statements or examinations; analytics may detect unusual patterns earlier.

Lead-time gain = date risk would conventionally have been identified − date identified by SupTech

This is the most valuable metric and the least tractable one. The counterfactual detection date is unobservable in advance and can only be estimated retrospectively, on cases where the risk actually crystallised — which biases the sample towards successes and says nothing about risks flagged early and correctly that never materialised. It is worth tracking, but as a case-by-case supervisory narrative supported by dates, not as a portfolio average presented to a board.

Lead time also only converts into value where earlier detection permits earlier intervention.

4. From signal to action — and coverage

Detection is not the objective. An algorithm producing interesting findings that supervisors ignore creates no institutional value.

Action conversion = findings resulting in supervisory action / material findings generated

Supervisory action here means information requests, risk-rating changes, inspection activity, remediation requirements or escalation. This ratio is the single most direct link between a technology investment and the supervisory process, and it is the one most authorities do not compute.

The FSI’s findings on why tools fail to become critical point in the same direction: tools that add a step to a supervisory process, or create a parallel process, tend to be poorly received, while tools that become indispensable fit an existing process and address a specific pain point (Prenio, 2024). Low action conversion is usually a design problem, not a supervisor problem.

Coverage is the related dimension. SupTech can widen the supervisory perimeter without a proportionate increase in resources: an authority that reviews 20 institutions deeply each quarter may be able to screen 100 meaningfully while directing scarce attention to the highest-risk cases. This matters most for supervisors overseeing large populations of smaller institutions, payment providers and other intermediaries, and it is the explicit rationale in the World Bank’s market-conduct work (World Bank, 2021).

5. Decision quality

The hardest dimension, and arguably the most important: does the tool improve supervisory judgment?

This cannot be settled by comparing machine output with supervisor output. The more useful questions are whether technology-supported assessments are more consistent across teams, whether material risks are escalated more reliably, and whether supervisory conclusions prove directionally correct over time. A tool that reduces unexplained dispersion between comparable assessments can create substantial institutional value without saving much time at all.

The governing principle, emphasised throughout the FSI’s work, is that tool outputs should support supervisory judgment rather than replace it (Beerman, Prenio and Zamil, 2021).

6. Risk-adjusted value ( Similar to RAROC)

Technology introduces its own risks. A sophisticated tool resting on poor data, opaque models or fragile infrastructure can generate apparent productivity while weakening supervisory reliability.

Risk-adjusted value = supervisory benefits − operating costs − amortised development cost − model and technology risk costs

The risk component should cover model risk, data governance, explainability, cyber and operational resilience, and the availability of human override. Not all of it can be monetised precisely; identifying it still improves investment decisions. Recent cross-country evidence suggests that the institutional prerequisites matter more than the tools: across 112 authorities in 97 countries, those with integrated strategies for digital transformation, data governance and SupTech deployed substantially more applications and encountered fewer implementation problems, and a dedicated SupTech unit was the strongest single driver of moving from experimentation to deployment (Gambacorta, Lauridsen, Kiuhan-Vásquez and Prenio, 2025).

A SupTech value scorecard

The FSI’s recent work has shifted the question from whether authorities can build SupTech tools to whether deployed tools actually become integral to supervision. That is the right test, and it applies at the level of the individual application.

The best SupTech application is not the one processing the most data, using the most sophisticated models, or attracting the most users. It is the one that helps supervisors identify material risks earlier, understand them better, and act on them more effectively — while using scarce supervisory capacity more intelligently. This has remained a constraint of off the shelf products. They are general , oblivious of what supervisors actually need ( lacking domain awareness ) and hence constrained in value creation.  Domain Knowledge experts, data science expertise and software specialists  must work together to create enduring value in Sup Tech.

References

Bains, P. and Wu, C. (2023). Institutional Arrangements for Fintech Regulation: Supervisory Monitoring. IMF Fintech Note 2023/004. International Monetary Fund. https://www.imf.org/en/Publications/fintech-notes/Issues/2023/06/23/Institutional-Arrangements-for-Fintech-Regulation-Supervisory-Monitoring-534291

Beerman, K., Prenio, J. and Zamil, R. (2021). Suptech tools for prudential supervision and their use during the pandemic. FSI Insights No 37. Bank for International Settlements. https://www.bis.org/fsi/publ/insights37.htm

Gambacorta, L., Lauridsen, N., Kiuhan-Vásquez, S. and Prenio, J. (2025). Making suptech work: evidence on the key drivers of adoption. BIS Working Papers No 1309. Bank for International Settlements. https://www.bis.org/publ/work1309.htm

Garcia Ocampo, D., Lehtmets, A., Pandey, M. and Prenio, J. (2022). Suptech in insurance supervision. FSI Insights No 47. Bank for International Settlements, Access to Insurance Initiative and International Association of Insurance Supervisors. https://www.bis.org/fsi/publ/insights47.htm

Prenio, J. (2024). Peering through the hype — assessing suptech tools’ transition from experimentation to supervision. FSI Insights No 58. Bank for International Settlements. https://www.bis.org/fsi/publ/insights58.htm

Prenio, J., Pustelnikov, A. and Yeo, J. (2024). Building a more diverse suptech ecosystem: findings from surveys of financial authorities and suptech vendors. FSI Briefs No 23. Bank for International Settlements. https://www.bis.org/fsi/fsibriefs23.htm

Randall, D. and Lopes, L. (2018). From spreadsheets to suptech for financial sector market conduct supervision. World Bank Blogs, 2 August. https://blogs.worldbank.org/en/psd/spreadsheets-suptech-financial-sector-market-conduct-supervision

World Bank (2021). The Next Wave of Suptech Innovation: Suptech Solutions for Market Conduct Supervision. Technical Note, March. https://openknowledge.worldbank.org/handle/10986/35322