sdmx-readiness

Check how ready an SDMx web service is for a direct move from SDMx 2.1 to SDMx 3.1:

which structural artefacts (codelists, concept schemes, DSDs, dataflows, constraints, categorisations, ...) can move to semantic versioning;
what changes or breaks in 3.1, including DSD structure such as the position of the time dimension;
whether the service already speaks the SDMx 3.1 formats.

It started as the OECD script loop_artefacts_and_versions.py (first git commit). See docs/METHODOLOGY.md §12 and the side-by-side run in reports/comparison.

What "readiness" means and how it is measured: docs/METHODOLOGY.md
Every rule, with its source: docs/RULES.md (generated)
Rules awaiting validation: docs/RULES_REVIEW.md + docs/rules_review.xlsx
What .Stat Suite requires beyond SDMx (FREQ, TIME_PERIOD, lengths, versions, ...): docs/DOTSTAT_CONSTRAINTS.md
How to add a rule or an endpoint: CONTRIBUTING.md
SPC assessment: reports/spc/SPC_SDMX_3.1_readiness.md

Quick start

Requires Python ≥ 3.11 and uv.

uv sync                                   # install
uv run sdmx-readiness endpoints           # list configured endpoints
uv run sdmx-readiness scan -e spc         # scan SPC's public .Stat (< 1 min, ≈ 12 MB)

The report is written to reports/<endpoint>/<date>/. Open report.html in a browser first: it has filters, drill-down per artefact and CSV export, and works offline.

Scan your own endpoint

Either add a block to endpoints.toml:

[my-org]
label   = "My organisation - dissemination"
v1_base = "https://example.org/rest"      # SDMx REST 1.x -> SDMX-ML 2.1 (what you publish today); .Stat NSI: ".../rest"
v2_base = "https://example.org/rest/v2"   # SDMx REST 2.x -> SDMX-ML 3.1 (target), if the server offers it
auth    = "none"

and run uv run sdmx-readiness scan -e my-org, or scan ad hoc:

uv run sdmx-readiness scan --name my-org --v1-base https://example.org/rest --v2-base https://example.org/rest/v2

v1_base is what the rules run on. v2_base is used for the SDMx 3.1 platform probes, and for the 2.1-vs-3.1 comparison once the server serves SDMX-ML 3.1.

Authentication

`auth`	Use
`none`	public endpoints (default)
`bearer`	token in `$SDMX_BEARER_TOKEN`
`dotstat-adfs`	interactive ADFS login, as in the original OECD script; needs `uv sync --extra dotstat` and `CLIENT_ID`, `RESOURCE_ID`, `AUTH_URL` in the environment or in `.env`

Useful options

Option	Effect
`--from-cache reports/spc/2026-10-23`	rebuild a report from a previous scan's `raw/`, without network access
`--formats 2.1`	skip the SDMX-ML 3.1 structure queries (default `2.1,3.1`)
`--resources codelist,datastructure`	fetch only some REST resources
`--rules VER-003,DSD-001`	run only some rules
`--full-codelists`	fetch codelists in full instead of `detail=allstubs`
`--no-probes`	skip the platform capability probes
`--out DIR`, `--timeout S`	output folder; read timeout per query
`--concurrency N`	structure queries sent to the endpoint at once (default 4; `1` = one at a time, like the original script). Results do not depend on it
`--max-mb N`	largest response accepted, after decompression (default 1000 MB; `0` = no limit)

Output

File	Content
`report.html`	explorable report, self-contained (no network needed). Tabs: overview, findings, rules, migration plan, inventory, platform, queries. Filters, sorting, search, CSV export of any filtered view, click an artefact for all its findings and plan row. Filters are kept in the URL (`#tab=findings&rule_id=DSD-001`), so a view can be shared as a link
`summary.md`	the same headline figures as plain text (diff-friendly, readable on GitLab/GitHub)
`report.xlsx`	every table below as a sheet, plus the rule catalogue
`findings.csv`	one row per rule violation (artefact, reference, message)
`migration_plan.csv`	one row per artefact: proposed semantic version, action, bottom-up tier, blockers
`inventory.csv`	one row per artefact and format
`capabilities.csv`	platform probes (media type, HTTP status, result)
`fetch_log.csv`	one row per query (URL, status, size, time)
`meta.json`	scan metadata and headline metrics (input for `report.html`)
`raw/`	raw responses + `manifest.json` (not committed; see `.gitignore`)

sdmx-readiness html <scan folder> rebuilds report.html from these files without network access, e.g. after updating the tool.

Browsable site

uv run sdmx-readiness site --out site      # docs + all reports as HTML
python -m http.server -d site 8000         # then open http://localhost:8000

The site renders every Markdown file (README, docs, reports, write-ups) to HTML with working links, and puts every scan's report.html under one reports index. It is static, so it can also be published as-is (e.g. GitLab Pages). Diagrams load mermaid.js from a CDN; offline, they show as text. site/ is a build output and is not committed.

Web app

sdmx-readiness serve runs the same scan behind a web page, so organisations can scan their endpoint without installing anything:

the endpoint list combines endpoints.toml with the public services in the SDMx Global Discovery Service (refreshed daily, cached in <data>/gds.json); endpoints that need authentication are not offered. endpoints.toml also carries the SDMx Gateway MCP endpoints that GDS lacks (Fiji, Samoa, ABS, UNICEF) or lists without a REST v2 base (OECD); an endpoints.toml entry replaces a GDS entry with the same base URL;
visitors can also type their own REST v1/v2 base URLs;
a scan runs in the background with a live progress page and ends on the same report.html, report.xlsx and CSVs as the CLI; every scan of an endpoint is kept (newest 10) so progress can be followed over time;
a JSON API (/api/docs) lets organisations script it, e.g. curl -X POST https://<host>/api/scans -H 'Content-Type: application/json' -d '{"endpoint": "spc"}' then poll the returned /api/scans/<key>/<stamp> until state is done;
the methodology, rules and reference reports are served at /guide/.

uv sync --extra web
uv run sdmx-readiness site --out site          # optional: enables /guide/
uv run sdmx-readiness serve                    # http://127.0.0.1:8000

Hosting

The app is one process with no database: all state is in the data folder (<data>/scans/<endpoint>/<UTC stamp>/ with the report files and status.json). A container image builds the docs site in:

docker compose up -d --build                   # or: docker build -t sdmx-readiness . && docker run -p 8000:8000 -v sdmx-data:/data sdmx-readiness

The public instance runs on a dedicated Infomaniak Public Cloud VM: https://179-237-95-52.sslip.io/. deploy/README.md explains how it was created (OpenTofu), how to update it (deploy/deploy.sh) and how to operate it. The same deploy.sh works on any Ubuntu VM with Docker: Caddy (deploy/compose.vm.yaml) handles HTTPS, and the app port is not exposed. 2 vCPU and 4 GB are enough for the default of two scans at a time.

Settings (environment variables):

Variable	Default	Effect
`SDMX_READINESS_DATA`	`data` (`/data` in the image)	scans and the GDS cache
`SDMX_READINESS_ENDPOINTS`	`endpoints.toml`	configured endpoints (only `auth = "none"` ones are offered)
`SDMX_READINESS_SITE`	`site`	built docs site served at `/guide/` (skipped if missing)
`SDMX_READINESS_GDS`	`1`	list endpoints from gds.sdmx.io
`SDMX_READINESS_CUSTOM`	`1`	let visitors scan URLs they type
`SDMX_READINESS_PARALLEL_SCANS`	`2`	scans running at once; others queue
`SDMX_READINESS_CONCURRENCY`	`4`	queries at once within a scan
`SDMX_READINESS_TIMEOUT`	`120`	read timeout per query, seconds
`SDMX_READINESS_MAX_MB`	`200`	largest response accepted, MB (decoded)
`SDMX_READINESS_REUSE_MINUTES`	`60`	a scan this recent is returned instead of starting a new one
`SDMX_READINESS_KEEP_SCANS`	`10`	scans kept per endpoint
`SDMX_READINESS_SCANS_PER_HOUR`	`20`	new scans per client address and hour
`SDMX_READINESS_ALLOW_PRIVATE`	`0`	allow non-public addresses; only for intranet deployments

Safety, because the server fetches URLs chosen by visitors: only http(s) URLs whose host resolves to public addresses are fetched, checked again on every redirect (netguard.py; no localhost, private ranges or cloud metadata addresses); responses are capped in size; scans of the same host never overlap; raw responses are not kept. DNS can still change between the check and the connection, so run the container where it cannot reach anything sensitive (no cloud instance credentials, own network). Every scan, including custom ones, is listed on the home page; scanned endpoints are public anyway.

How to read a report

In report.html:

Overview: semantic-versioning adoption and the share of final artefacts show how big the migration is. The BLOCKER count shows whether it can start.
Findings by rule (Overview): each rule shows severity, basis (SCHEMA / STANDARD / GUIDELINE / OBSERVED) and kind of change. Click a rule to list its findings; the Rules tab explains why it matters and how to fix it.
Click any finding or plan row to see everything about that artefact in one place.
Platform: supported on the 3.1 rows means the service already serves SDMX-ML 3.1, SDMX-JSON 2.1 or SDMX-CSV 2.1.
Migration plan: sort by Tier and migrate tier 0 first.

Repository layout

sdmx_readiness/
  cli.py             command line (scan / endpoints / rules [--markdown | --review] / html / site)
  endpoints.py       endpoints.toml loading; which REST queries to run
  client.py          HTTP, authentication, raw-response cache
  parse.py           SDMX-ML 2.1 / 3.1 -> Artefact / Reference objects (+ DSD/MSD facts)
  model.py           Artefact, Reference, Inventory
  versions.py        version grammar (SDMX-ML 3.1 schema)
  rules/             readiness rules; each rule's documentation lives in its @rule(...)
    engine.py          Finding, Context, severity / basis / change definitions
    versioning.py      VER-*
    references.py      REF-*
    model_changes.py   MOD-*
    structures.py      DSD-*, MSD-*
    crossformat.py     XF-*
  migration.py       per-artefact migration plan
  capabilities.py    platform probes
  report.py          summary.md, CSVs, report.xlsx, meta.json
  html.py            explorable report.html (self-contained)
  site.py            static site from Markdown + reports
  scan.py            the scan pipeline (parallel fetch -> parse -> rules -> probes -> reports), used by CLI and web app
  netguard.py        refuses non-public / non-http URLs (web app)
  web/               web app: app.py (pages + JSON API), jobs.py (background scans), catalogue.py (endpoints.toml + GDS), pages.py (HTML)
tests/               unit tests on small SDMX-ML fixtures (one firing + one non-firing case per rule)
docs/                METHODOLOGY.md, RULES.md (generated), RULES_REVIEW.md, rules_review.xlsx (generated)
reports/             committed reports (without raw/), comparison with the original script
endpoints.toml       known endpoints
Dockerfile, compose.yaml   container image and local compose
deploy/              public VM: OpenTofu stack (infomaniak/), Caddy overlay, deploy.sh, runbook (README.md)

Development

uv sync --extra web && uv run pytest                        # web tests are skipped without the extra
uv run sdmx-readiness rules --markdown > docs/RULES.md        # after changing a rule
uv run sdmx-readiness rules --review docs/rules_review.xlsx \
  --evidence spc=reports/spc/2026-10-23/findings.csv \
  --evidence siscc-qa=reports/siscc-qa/2026-10-23/findings.csv  # review sheet for validators