Enhancing SDMX tools interoperability for improved organisational efficiency

Over past decade or more, countless tools that cannot talk to one another have emerged within and across the official statistics system for the purposes of collecting, processing, and disseminating Meta(data). The proliferation of different technology platforms, data definitions and institutional arrangements for managing, sharing, and using meta(data) has increasingly made it necessary to dedicate resources to manage the exchange and reuse of this data. Fortunately, the Statistical Data and Metadata eXchange (SDMX) standard provides an efficient means to enable different tools to communicate with each other, and exchange, interpret and actively use data. It allows different tools to work together seamlessly. This is what interoperability is about. There are (at least) three main types of interoperability, which include:

• Syntactic (or foundational) interoperability allows two or more systems to communicate and share data. It thus allows different types of software to work together. The SDMX REST standard defines the required protocol for our needs.
• Structural interoperability defines the data exchange format, which specifies the standards used to format messages sent from one system to another. The SDMX structure and data message formats define the required exchange formats for our needs.
• Semantic interoperability allows two or more systems to connect and exchange data that each system understands in a meaningful way.  The SDMX information model should allow for the required data interoperability.

While interoperability is a characteristic of a product or system to be capable of communicating with other products or systems that have a common standards-based interface, integration is the connection of two or more systems, enabling communication, usually with the use of “middleware” to translate each system’s data because these systems do not (yet) have a common standard-based interface. Together, interoperability and integration (whenever a common standard-based interface is not yet available) provide the key enablers for an organisation to become more data-driven and therefore crucial to data management strategies in every organisation.

An assessment of SDMX Tools interoperability

In order to understand better the current state of SDMX tools landscape and their ability to work together the Statistical Information System Collaboration Community (SIS-CC) undertook an assessment of existing tools for SDMX artefact edition and maintenance in 2022. The findings of the assessment, published in its report of June 2022, shed light on the current state and challenges regarding tools interoperability and integration.  Provided below is a concise overview of the main conclusions and recommendations from the findings as well as steps already being taken by the Community.

Assessment main findings and goals

The primary aim of the assessment was to evaluate the existing SDMX-compliant tools and assess their interoperability capabilities, specifically looking at tools with the capacity to edit and maintain SDMX artefacts and excluding the dissemination phase from the assessment. The report was based on declarative input by tool provider – so no direct assessment of the tools was done, only aggregation of information provided by tool providers complemented with inputs from those organisations who were using them. The report’s main conclusion is that while multiple SDMX tools are typically required to edit and maintain SDMX Meta(data) Structures, ensuring interoperability and seamless integration remains a challenge – but criteria could be forged so that, going forward, tool providers have clear set goals in order to achieve effective interoperability, to the advantage of users. In the mid-long term, these interoperability criteria could become official SDMX guidelines.

Main interoperability requirements

A number of interoperability requirements were identified in the report which have been described below along with their definitions, typical challenges, and proposed good practices.

Versioning and upgrade management

Definition: Ensuring proper handling of versioning and upgrades in SDMX tools.

Typical challenge: Incompatible versions of tools, especially if using different versions of an SDMX web service supporting different features, can lead to data inconsistencies, compatibility issues, and difficulties in collaboration between different users and organisations.

Good practice: Tool providers should implement a version control system, clearly document version dependencies, and establish a process for notifying users about updates and managing backward compatibility.

Access control

Definition: Managing user access to SDMX tools and data based on user roles and permissions.

Typical challenge: Inadequate access control can result in unauthorised data access, security breaches, and potential data misuse.

Good practice: Tool providers should implement robust and recognisable identification and access control mechanisms, including user authentication, role-based access permissions, and audit trails to ensure data security and privacy.

Documentation

Definition: Providing comprehensive and up-to-date documentation for SDMX tools, including user guides, technical specifications, and API documentation.

Typical challenge: Insufficient or outdated documentation can hinder users’ understanding of tool features and functionalities, leading to suboptimal usage and inefficient workflows.

Good practice: Tool providers should prioritise the creation and maintenance of detailed documentation, including clear explanations, examples, and tutorials, to enhance user onboarding, support, and self-sufficiency.

Tool governance

Definition: Establishing effective governance frameworks and processes for SDMX tools, including maintenance, support, and decision-making.

Typical challenge: Lack of proper tool governance can result in inconsistent tool usage, limited support, and difficulty in aligning tools with organisational objectives.

Good practice: Tool providers should establish governance structures to ensure regular updates, bug fixes, and user support. This can include designated tool administrators, feedback channels, and regular review processes to address user needs and maintain tool relevance.

Meeting these interoperability requirements is crucial for achieving smoother integration and enhancing organisational workflows in SDMX tools. By implementing the proposed good practices, tool providers can enhance collaboration, data consistency, security, and user satisfaction.

Moving forward: Collaboration and standardisation

To ensure better coordination among tool providers, the report proposes a community cooperation framework that promotes formal collaboration and coordination. Precisely expressing user needs, establishing task forces, conducting user research, and creating support groups are all vital elements of this framework. By involving tool providers and the SDMX community in the development process, user requirements can be better incorporated, and tools can be harmonised and complement each other’s functionalities.

Standardising the formulation of user requirements and providing SDMX interoperability guidelines are essential for facilitating tool integration. These guidelines can serve as a common reference for tool development and implementation, ensuring that tools align with user needs and adhere to interoperability standards.

Conclusion and next steps to address these challenges

This assessment of SDMX tools for interoperability reveals both challenges and opportunities for improving the implementation of SDMX. By addressing the identified challenges and adopting a community collaboration framework, organisations can enhance tool integration, streamline workflows, and maximise the benefits of SDMX. Standardising user requirements and providing guidance to tool vendors will help align tool capabilities with user needs. With the collective efforts of the SDMX community, greater interoperability, and efficiency in SDMX tool implementation and advancing organisational data management practices can be achieved.

Significant steps have already been taken to address some of the challenges identified in the assessment report.

One step taken has been the development of the Bank of International Settlements (BIS) managed open-source Fusion Metadata Registry Workbench (FMR Workbench). The FMR Workbench is a remote registry browser and metadata maintenance tool which brings the strengths and capabilities of the FMR user interface to any SDMX compliant registry, implementing an SDMX RESTful API. One significant use case for FMR Workbench users is the ability to connect to .Stat Suite data spaces for browsing and maintaining .Stat Suite structural metadata. This initial version of the FMR Workbench has been driven by the needs of Community member the National Bank of Belgium (NBB) – but benefits a much larger user base, to make the most of an integrated stack [.Stat + FMR]. Additionally, future enhancements are being coordinated through the SIS-CC FMR Workbench User Group, led by the NBB.

The ILO SDMX Constructor tool – part of the ILO SMART Toolkit – now offers a direct connection to the .Stat Suite Data Lifecycle Manager. This integration allows for pushing SDMX artefacts into a data space for validation and subsequent dissemination. The ILO stack primarily aims to cater for Ministries of Labour in the context of the ILO Labour Market Information Systems (LMIS) project. With this approach, users can smoothly use a well-integrated [SMART+.Stat].

The sdmx.io initiative led by the BIS aims to strengthen interoperability between SDMX tools drawing much of its acceptance criteria from the assessment report but also the experiences from the development of the FMR Workbench that piloted this new approach. sdmx.io is not a single project but an ecosystem of open-source tools, patterns, guidance, learning materials and other resources like pre-configured containerised environments that make the software quick and simple to deploy. To be included on sdmx.io is to be compliant, ensuring a minimum level of interoperability and ease of integration with at least one other tool listed (e.g. .Stat+FMR). These collection of open-source SDMX tool providers are cooperating to solve the most common official statistics use cases.

The SDMX Technical Workshop Group has also developed a Technology Compatibility Kit (TCK) specifically designed to certify the compliance and coverage of RESTful SDMX web services. This TCK provides a standardised approach to ensure the interoperability and compatibility of these web services, offering a valuable tool for tool providers to certify their compliance with the SDMX standard.

Through these initiatives, the SIS-CC and wider SDMX community is actively involved in addressing the challenges of tools interoperability and facilitating the integration of SDMX solutions. By collaborating with tool providers and developing compatible tools and standards, we can improve the overall efficiency and effectiveness of SDMX implementation.

Note(s): This assessment was carried out by the SIS-CC SDMX Tools Assessment Working Group (STAWG), represented by a diverse group drawn from International Organisations, National Statistics Offices, Central Banks, Sectorial Organisations, and both tool users and providers were engaged in an evaluation process. This resulted in an adopted assessment approach that relied both on the use of the evaluations carried out by these members and the implementation of a number of methods. This made the completion of the missing information possible in order to achieve the group’s objectives. It is important to note that the SDMX tools that were shortlisted for assessment were chosen based on SDMX compliance and feature coverage, interoperability with the .Stat Suite platform, openness, and popularity within the SIS-CC.

The findings from the report are declarative and by no means a complete and validated assessment of all SDMX available tools.

References:

• SIS-CC SDMX Tools Assessment Working Group Final Report – June 2022
• 9^th SDMX Global Conference presentation: Achieving tools interoperability – What impact on SDMX 3.0 implementation by Fadhila Najeh, FAO