Methodology

How sdmx-readiness decides whether an SDMx endpoint is ready to move from SDMx 2.1 straight to SDMx 3.1, and why each choice was made. Read this before changing a rule or comparing two organisations' reports. The rules themselves are in RULES.md; open validation questions are in RULES_REVIEW.md.

1. What "SDMx 3.1 readiness" means here

SDMx 3.0 is not a target. Changes introduced in 3.0 still count, because 3.1 contains them. Readiness has two independent parts:

The tool does not produce a single score. Organisations differ in size and practice, and a percentage would hide what matters most: which artefacts need a decision.

Scope so far: public dissemination endpoints. Authenticated endpoints are supported (auth = "bearer" or "dotstat-adfs") but have not been the focus.

2. Pipeline

flowchart LR
  A[endpoints.toml] --> B[plan_queries<br/>one query per<br/>resource x format]
  B --> C[Fetcher<br/>HTTP + auth]
  C --> D[(raw/ + manifest.json)]
  D --> E[parse<br/>SDMX-ML 2.1 / 3.1]
  E --> F[Inventory 2.1<br/>primary]
  E --> G[Inventory 3.1<br/>if served]
  F --> H[rules]
  G --> H
  F --> I[migration plan]
  H --> I
  F --> J[capability probes]
  H --> K[report: report.html,<br/>summary.md, CSVs, report.xlsx]
  I --> K
  J --> K

--from-cache <previous output folder> restarts from raw/. A report can therefore be rebuilt (e.g. after a rule change) without calling the endpoint again.

3. What is fetched, and why

• SDMX-ML 2.1, every maintainable resource, all/all/all, no references=. This is what the organisation publishes today and what the rules are evaluated on. The original script asked for ?references=all once per artefact type. Since every type is fetched anyway, that only downloaded the same artefacts several times.

• SDMX-ML 3.1 from REST v2 (v2_base/structure/{type}/*/*/*), if the server offers it. Comparing it with 2.1 shows what the server's own conversion drops or re-versions (XF rules). On 2026-10-23 no scanned endpoint offered it: SPC, SIS-CC QA, the Swiss FSO and the SDMx Global Registry all answered 406. The XF rules are therefore inactive for now, and the 406 is reported as a platform gap. The 3.1 dataflow query is sent first; when it gets 406 or 415, the other 3.1 queries are not sent (recorded as unsupported, "not sent"). That keeps a scan to ~27 requests instead of 60, which matters for servers with quotas (the OECD Data API allows 60 per hour). A server that answers a 3.1 query with SDMX-ML 3.0 (Fusion Metadata Registry, Eurostat) is also unsupported.

• No SDMX-ML 3.0. It is not the target, and a 3.0 conversion says little about a 3.1 one.

• Codelists are fetched with detail=allstubs (id, agency, version, isFinal; no codes). SDMX-ML 2.1 codelists have no outgoing references, so nothing is lost; on SPC this is 92 kB instead of 24 MB. 3.1 codelists can extend other codelists, so they are fetched in full when 3.1 is the only format. --full-codelists turns stubs off.

• HTTP status meaning (client.outcome_for):

  Status	Outcome
  200	ok
  204, 404	empty (the NSI answers 404 "No Results Found" for an empty resource)
  400, 405, 406, 415, 422, 501	unsupported
  401, 403	forbidden
  anything else	error

  A class counts as fetched only when its query was ok or empty. REF-001 never reports a missing target whose class was not fetched.

• Up to 4 queries at once (--concurrency); results do not depend on it. 5xx and network errors are retried twice. HTTP 429 (rate limited) waits for Retry-After (60 s if absent, at most 120 s) and is retried up to 5 times.

• Size cap (--max-mb, default 1000 MB; 200 MB on the web app). Some servers send a gzip file as the body without Content-Encoding (Eurostat's dissemination API: its DSDs are ~2 GB once decompressed); such bodies are decompressed, and the cap applies to the decompressed size. A query over the cap is an error, and the scan is marked incomplete.

4. Parsing

sdmx_readiness/parse.py turns each message into Artefact and Reference objects.

• Structural references. A reference is any of:
  • a 2.1 <Ref> with an agencyID (a <Ref> without one points inside the same artefact and is ignored);
  • a 2.1 <URN> element;
  • a 3.1 element whose text is a URN.
• Item references (Concept, Category, ...) are mapped to their owning scheme, because versions belong to maintainable artefacts.
• 2.1 schema defaults are applied:
  • a missing version / maintainableParentVersion is 1.0;
  • a missing isFinal is false;
  • a missing ContentConstraint/@type is Actual;
  • a missing TextFormat/@textType is String;
  • a missing isMultiLingual is true (3.1 default: false).
• URNs inside annotations are kept with kind=annotation. They are not SDMx references, but .Stat uses them (DSD annotation METADATA -> MSD URN) and migration tools do not rewrite them.
• Finality in 3.1. SDMx 3.x has no isFinal. For 3.1 messages the parser reads the .Stat NSI annotation NonFinalArtefact=true, on the artefact itself and not on its items.
• DSD / MSD facts (2.1 only) are recorded as Artefact.props["dsd"] and props["msd"]. They cover dimension order and positions, concept roles, attribute attachments, groups, ReportingYearStartDay, textual attributes without isMultiLingual, and MSD targets and report structures. Keys are documented in parse._dsd_facts / _msd_facts.

5. Version grammar

sdmx_readiness/versions.py is the only place that knows the grammar. It follows the SDMX-ML 3.1 schema (SDMXCommonReferences.xsd):

SDMX-ML 2.1 accepted any dotted number ([0-9]+(\.[0-9]+)*). Versions such as 1.03 are therefore valid today and invalid in 3.1 (VER-001).

6. Severity, basis, change

Every rule carries three labels, defined in sdmx_readiness/rules/engine.py:

• Severity: what the owner must do.

  • BLOCKER: cannot be represented in 3.1 without a decision, a remodel or a loss.
  • WARNING: must change, but mechanically; or valid but unsafe / silently different.
  • INFO: planning facts and guidelines.

• Basis: how strong the evidence is. The standard documents disclaim normativity themselves:

  • Section 2: "This document is not normative but provides a detailed view of the information model on which the normative SDMX specifications are based".
  • Section 6 §3.1: "It is not normative, and it does not provide any rules for the use of the standards, such as those found in SDMX-ML: Schema and Documentation".

  Hence four levels:

  Basis	Meaning
  SCHEMA	Enforced or stated by the SDMX-ML 3.1 XSD
  STANDARD	A MUST/SHOULD in a 3.1 document that the schema does not enforce, e.g. the Section 6 semantic-versioning annex
  GUIDELINE	Non-normative guidance, a MAY, or our own derivation
  OBSERVED	Endpoint content or behaviour

• Change: mechanical, decision, remodel or none.

7. Conversions deliberately not reported

Every 2.1 DSD contains these constructs, and any converter must rewrite them:

• assignmentStatus -> usage
• PrimaryMeasure -> Measure
• AttributeRelationship None -> Dataflow
• AttributeRelationship PrimaryMeasure -> Observation
• dimension type attribute (removed)
• TimeDimension/@position (prohibited)

Reporting them would flag every DSD and say nothing about any one of them. Some of the value mappings are not documented in the standard; they are listed for validation in RULES_REVIEW.md §6.

8. Metrics (summary.md "Headline")

• Versioned artefacts: all maintainables except external stubs and the classes that have no version in SDMx 3.1, namely organisation schemes and Categorisations (version prohibited in the 3.1 XSD). Artefacts of other agencies are included; the "Inventory by agency" table separates them.
• Semantic versioning adoption = semver (stable + extended) / versioned artefacts.
• Final share = isFinal=true / versioned artefacts whose finality is known. Non-final artefacts become X.Y.0-draft, so this share is the best predictor of migration effort.
• Findings are counted per rule and per distinct artefact.

9. Migration plan

migration_plan.csv has one row per artefact; the columns are defined in sdmx_readiness/migration.py.

• Mechanical mapping (Section 6, 14.5): X.Y -> X.Y.0; isFinal=false -> X.Y.0-draft. Unknown finality counts as non-final, because the Section 6 FAQ says that without semver "pre-SDMX 3.0 final artefacts will be migrated as non-final and mutable".
• Latest minor per major (SDMx TWG guidance sdmx-twg/semver, branch SDMX-3.1; not normative): only the highest X.Y of each major X is migrated. Older minors are split into two groups:
  • still referenced: repoint the referrers, or migrate them too;
  • unreferenced: candidates to retire.
• Tier: the bottom-up order along structural references. Tier 0 references nothing on the endpoint. A semver artefact may only reference semver artefacts, so migration goes up the tree: codelists -> concept schemes -> DSDs -> dataflows.
• Unversioned classes (Categorisations, organisation schemes) get "drop version".

10. Platform probes

One sample dataflow is used, preferably one with an Actual constraint (i.e. one that holds data). It is requested once per media type, and only the response headers are read. The SDMx 3.1 media types come from sdmx-rest v2.2.2:

One SDMX-ML 2.1 probe is kept as a baseline. Results mean:

11. Known limitations

• Wildcard references are not resolved; REF-001 skips them.
• 3.1-only features are optional and not assessed: evolvingStructure, the dataflow DimensionConstraint, MetadataAttributeUsage.
• MSD content beyond targets and report structures is not checked.
• When 2.1 is available, codelist extensions (3.1 only) are not assessed: 3.1 codelists are stubs and the rules run on 2.1.
• SDMX-JSON structure messages are probed but not parsed.
• The XF rules have never run on real data, because no server offered SDMX-ML 3.1 at the time of writing.

12. Changes from loop_artefacts_and_versions.py

The original OECD script is in git history (first commit). The side-by-side run is in reports/comparison.

Rule mapping:

• original rule 1 -> VER-003, generalised to any non-semver target;
• original rules 2-3 -> VER-004 (stable -> draft) and VER-005 (final -> non-final).

13. Sources

• SDMX-ML schemas, tags v2.1 and v3.1.0: https://github.com/sdmx-twg/sdmx-ml. In particular:
  • SDMXCommonReferences.xsd
  • SDMXStructureDataStructure.xsd
  • SDMXStructureMetadataStructure.xsd
  • SDMXStructureBase.xsd
  • SDMXStructureConstraint.xsd
  • SDMXStructureCategorisation.xsd
  • SDMXStructureOrganisation.xsd
• SDMx 3.1 Section 1, Framework: https://sdmx.org/wp-content/uploads/SDMx_3-1-0_SECTION_1_FINAL.pdf
• SDMx 3.1 Section 2, Information Model: https://sdmx.org/wp-content/uploads/SDMX_3-1-0_SECTION_2_FINAL.pdf
• SDMx 3.1 Section 6, Technical notes: https://sdmx.org/wp-content/uploads/SDMX_3-1-0_SECTION_6_FINAL.pdf
• SDMx 3.1 Summary of major changes: https://sdmx.org/wp-content/uploads/SDMX_3-1-0_Major_Changes_FINAL-1_0.pdf
• SDMx 3.0 Summary of major changes: https://sdmx.org/wp-content/uploads/SDMX_3-0-0_Major_Changes_FINAL-1_0.pdf
• SDMx TWG, Versioning in SDMx (3.1 guidance): https://github.com/sdmx-twg/semver/blob/SDMX-3.1/Versioning in SDMX.md
• SDMx REST API, tag v2.2.2: https://github.com/sdmx-twg/sdmx-rest
• SDMx Guidelines for the design of DSDs 1.0: https://sdmx.org/wp-content/uploads/SDMX_Guidelines_for_DSDs_1.0.pdf