sdmx-readiness

Check how ready an SDMx web service is for a direct move from SDMx 2.1 to SDMx 3.1:

• which structural artefacts (codelists, concept schemes, DSDs, dataflows, constraints, categorisations, ...) can move to semantic versioning;
• what changes or breaks in 3.1, including DSD structure such as the position of the time dimension;
• whether the service already speaks the SDMx 3.1 formats.

It started as the OECD script loop_artefacts_and_versions.py (first git commit). See docs/METHODOLOGY.md §12 and the side-by-side run in reports/comparison.

• What "readiness" means and how it is measured: docs/METHODOLOGY.md
• Every rule, with its source: docs/RULES.md (generated)
• Rules awaiting validation: docs/RULES_REVIEW.md + docs/rules_review.xlsx
• What .Stat Suite requires beyond SDMx (FREQ, TIME_PERIOD, lengths, versions, ...): docs/DOTSTAT_CONSTRAINTS.md
• How to add a rule or an endpoint: CONTRIBUTING.md
• SPC assessment: reports/spc/SPC_SDMX_3.1_readiness.md

Quick start

Requires Python ≥ 3.11 and uv.

uv sync                                   # install
uv run sdmx-readiness endpoints           # list configured endpoints
uv run sdmx-readiness scan -e spc         # scan SPC's public .Stat (< 1 min, ≈ 12 MB)

The report is written to reports/<endpoint>/<date>/. Open report.html in a browser first: it has filters, drill-down per artefact and CSV export, and works offline.

Scan your own endpoint

Either add a block to endpoints.toml:

[my-org]
label   = "My organisation - dissemination"
v1_base = "https://example.org/rest"      # SDMx REST 1.x -> SDMX-ML 2.1 (what you publish today); .Stat NSI: ".../rest"
v2_base = "https://example.org/rest/v2"   # SDMx REST 2.x -> SDMX-ML 3.1 (target), if the server offers it
auth    = "none"

and run uv run sdmx-readiness scan -e my-org, or scan ad hoc:

uv run sdmx-readiness scan --name my-org --v1-base https://example.org/rest --v2-base https://example.org/rest/v2

v1_base is what the rules run on. v2_base is used for the SDMx 3.1 platform probes, and for the 2.1-vs-3.1 comparison once the server serves SDMX-ML 3.1.

Authentication

Useful options

Output

sdmx-readiness html <scan folder> rebuilds report.html from these files without network access, e.g. after updating the tool.

Values come from the scanned server. A value that a spreadsheet would run as a formula (starting with =, +, -, @) is written to the CSVs with a leading ', and to report.xlsx as plain text, so a hostile server cannot plant formulas.

Browsable site

uv run sdmx-readiness site --out site      # docs + all reports as HTML
python -m http.server -d site 8000         # then open http://localhost:8000

The site renders every Markdown file (README, docs, reports, write-ups) to HTML with working links, and puts every scan's report.html under one reports index. It is static, so it can also be published as-is (e.g. GitLab Pages). Diagrams load mermaid.js from a CDN; offline, they show as text. site/ is a build output and is not committed.

Web app

sdmx-readiness serve runs the same scan behind a web page, so organisations can scan their endpoint without installing anything:

• the endpoint list combines endpoints.toml with the public services in the SDMx Global Discovery Service (refreshed daily, cached in <data>/gds.json); endpoints that need authentication are not offered. endpoints.toml also carries the SDMx Gateway MCP endpoints that GDS lacks (Fiji, Samoa, ABS, UNICEF) or lists without a REST v2 base (OECD); an endpoints.toml entry replaces a GDS entry with the same base URL;
• visitors can also type their own REST v1/v2 base URLs;
• a scan runs in the background with a live progress page and ends on the same report.html, report.xlsx and CSVs as the CLI; every scan of an endpoint is kept (newest 10) so progress can be followed over time;
• a JSON API (/api/docs) lets organisations script it, e.g. curl -X POST https://<host>/api/scans -H 'Content-Type: application/json' -d '{"endpoint": "spc"}' then poll the returned /api/scans/<key>/<stamp> until state is done;
• the methodology, rules and reference reports are served at /guide/.

uv sync --extra web
uv run sdmx-readiness site --out site          # optional: enables /guide/
uv run sdmx-readiness serve                    # http://127.0.0.1:8000

Hosting

The app is one process with no database: all state is in the data folder (<data>/scans/<endpoint>/<UTC stamp>/ with the report files and status.json). A container image builds the docs site in:

docker compose up -d --build                   # or: docker build -t sdmx-readiness . && docker run -p 8000:8000 -v sdmx-data:/data sdmx-readiness

The public instance runs on a dedicated Infomaniak Public Cloud VM: https://179-237-95-52.sslip.io/. deploy/README.md explains how it was created (OpenTofu), how to update it (deploy/deploy.sh) and how to operate it. The same deploy.sh works on any Ubuntu VM with Docker: Caddy (deploy/compose.vm.yaml) handles HTTPS, and the app port is not exposed. 2 vCPU and 4 GB are enough for the default of two scans at a time.

Settings (environment variables):

Safety, because the server fetches URLs chosen by visitors:

• Where it connects (netguard.py): only http(s) URLs whose host resolves to public addresses; checked before the scan, on every redirect, and again at connection time, where the fetcher connects to the address it checked (no DNS rebinding). No localhost, private ranges or cloud metadata addresses.
• How long and how much: a response must complete within the request deadline even if the server sends one byte at a time (a watchdog closes the connection); a scan stops sending queries at the scan deadline; responses are capped in size, more tightly for visitor-typed URLs, which also run one at a time in their own queue, so they cannot hold up scans of listed endpoints.
• This server's resources: the container has a memory limit (compose.yaml); raw responses are not kept; visitor-typed endpoints beyond the newest 50 are deleted; no scan starts when the disk is nearly full.
• What visitors download: report.html only inserts scan data as text; spreadsheet outputs neutralise formulas (see Output).

Every scan, including custom ones, is listed on the home page; scanned endpoints are public anyway.

How to read a report

In report.html:

1. Overview: semantic-versioning adoption and the share of final artefacts show how big the migration is. The BLOCKER count shows whether it can start.
2. Findings by rule (Overview): each rule shows severity, basis (SCHEMA / STANDARD / GUIDELINE / OBSERVED) and kind of change. Click a rule to list its findings; the Rules tab explains why it matters and how to fix it.
3. Click any finding or plan row to see everything about that artefact in one place.
4. Platform: supported on the 3.1 rows means the service already serves SDMX-ML 3.1, SDMX-JSON 2.1 or SDMX-CSV 2.1.
5. Migration plan: sort by Tier and migrate tier 0 first.

Repository layout

sdmx_readiness/
  cli.py             command line (scan / endpoints / rules [--markdown | --review] / html / site)
  endpoints.py       endpoints.toml loading; which REST queries to run
  client.py          HTTP, authentication, raw-response cache
  parse.py           SDMX-ML 2.1 / 3.1 -> Artefact / Reference objects (+ DSD/MSD facts)
  model.py           Artefact, Reference, Inventory
  versions.py        version grammar (SDMX-ML 3.1 schema)
  rules/             readiness rules; each rule's documentation lives in its @rule(...)
    engine.py          Finding, Context, severity / basis / change definitions
    versioning.py      VER-*
    references.py      REF-*
    model_changes.py   MOD-*
    structures.py      DSD-*, MSD-*
    crossformat.py     XF-*
  migration.py       per-artefact migration plan
  capabilities.py    platform probes
  report.py          summary.md, CSVs, report.xlsx, meta.json
  html.py            explorable report.html (self-contained)
  site.py            static site from Markdown + reports
  scan.py            the scan pipeline (parallel fetch -> parse -> rules -> probes -> reports), used by CLI and web app
  netguard.py        refuses non-public / non-http URLs (web app)
  web/               web app: app.py (pages + JSON API), jobs.py (background scans), catalogue.py (endpoints.toml + GDS), pages.py (HTML)
tests/               unit tests on small SDMX-ML fixtures (one firing + one non-firing case per rule)
docs/                METHODOLOGY.md, RULES.md (generated), RULES_REVIEW.md, rules_review.xlsx (generated)
reports/             committed reports (without raw/), comparison with the original script
endpoints.toml       known endpoints
Dockerfile, compose.yaml   container image and local compose
deploy/              public VM: OpenTofu stack (infomaniak/), Caddy overlay, deploy.sh, runbook (README.md)

Development

uv sync --extra web && uv run pytest                        # web tests are skipped without the extra
uv run sdmx-readiness rules --markdown > docs/RULES.md        # after changing a rule
uv run sdmx-readiness rules --review docs/rules_review.xlsx \
  --evidence spc=reports/spc/2026-10-23/findings.csv \
  --evidence siscc-qa=reports/siscc-qa/2026-10-23/findings.csv  # review sheet for validators