Earlier this summer, the Community convened its fourth in the series of 2025 workshops, this time focusing on Metadata-driven automation: Calculation and Orchestration with SDMX. Chaired by György Gyomai (OECD), the session brought together members, partners, and invited experts to explore how calculation and orchestration services could evolve within the SDMX ecosystem to support modern statistical production workflows.
Why calculation and orchestration matters
Over the years, SDMX has matured into a standard that supports the representation, exchange, and delivery of data and data structures. However, while APIs and data-at-rest solutions have developed significantly, progress in tools for computation and orchestration has lagged behind.
Yet, calculation needs arise across all stages of the data lifecycle:
- Data collection (validation checks and imputations);
- Data integration (transcoding, mapping, pivoting);
- Production (aggregations, indicator derivations, currency/frequency conversions);
- Dissemination (OLAP-style on-demand aggregations and transformations).
The workshop assessed how SDMX could address these requirements more systematically: through robust calculation engines, orchestration services, and their integration into the reference architecture.
Use cases and perspectives
The session began with OECD and NBB presenting concrete use cases that framed the problem: while the OECD shared the pilot work decoupling computation from production databases, reconfigured to run against SDMX data stores, NBB presented its VTL-based calculation engine, interfacing smoothly with the .Stat Suite spaces for storage.
These examples illustrate both the opportunities and challenges of embedding computation into SDMX workflows.
Three perspectives also enriched the discussion:
- Meaningful Data showcased ongoing work to strengthen the links between their VTL engine and SDMX registries, positioning VTL rules and transformation schemes at the center of interoperable computation.
- BIS demonstrated its metadata-driven orchestration, built around an FMR registry-supported ETL pipeline, and hinted at extensions to incorporate VTL and SDMX Transformation Scheme features for richer data processing.
- Discussions also touched on alternative calculation engines using custom grammar. While less standardised, these approaches may be needed in the short term to support user adoption, manage migration costs, and accommodate diverse calculation granularity needs.
Key takeaways
Several conclusions emerged from the exchanges:
- Significant progress has been made with VTL execution engines and the pysdmx package, opening new integration possibilities;
- Orchestration scenarios are emerging, with execution sequences increasingly automated, though often still manually triggered;
- Grammar variation (beyond VTL) may remain necessary during transition phases or large-scale migrations;
- To move forward, calculation and orchestration components should be positioned within the SDMX reference architecture and supported by extensions to the SDMX web service API.
Looking ahead
The workshop confirmed both the demand and feasibility of advancing calculation and orchestration capabilities in SDMX. The SDMX Technical Working Group (TWG) will continue developing proposals to extend APIs and tools accordingly, ensuring these features support a more dynamic and automated statistical production workflow.
“Calculation engines are the most complex and least available in the context of SDMX. The main question is whether to aim for one language to express calculation rules such as VTL, or be language agnostic, and architect SDMX tools–orchestrators and storage components–around a more generic calculation node, Python based, for example.” (György Gyomai, OECD)
As with the previous workshops, these discussions will guide concrete prototypes, co-investments, and community collaboration in the months to come, strengthening the SDMX foundation for future-ready statistical systems.
