An SDMX Registry is a repository for the maintenance, storage, and dissemination of a specific type of metadata called ‘Structural Metadata’. Structural metadata can be used to describe the structure of datasets in any statistical domain. It is also used to describe the collection and dissemination of data and associated metadata. Structural metadata can also be used to describe rules and restrictions for data providers reporting datasets, it can describe statistical process, and help categorise datasets.
The purpose of an SDMX Registry is to act as a single point of access for structural metadata within an organisation, or between organisations. Although an SDMX Registry does not store datasets they provide a mechanism to identify where the datasets are located, this means an SDMX Registry describes datasets independently of how the data are stored.
The Fusion Registry is an implementation of an SDMX Registry, as described by the SDMX Specification.
The SDMX Information Model separates the data from the metadata used to describe the data. An SDMX Registry is built for the purpose of maintaining and disseminating the metadata used to describe the data (known as “structural metadata”), leaving the job of the data repository to focus fully on data storage and dissemination. Adopting this architecture leads to a decoupled and cohesive system, which can reduce software complexity and lead to a more maintainable solution which is resilient to change.
By adopting a Registry-based SDMX solution, all of the metadata required to describe the datasets, as well as rules pertaining to data collection, and disseminating data is held in the Fusion Registry. The Fusion Registry is used to describe the datasets for all the known domains, but will support all future data domains. The metadata is multilingual, allowing future support for names and descriptions in new languages.
Whilst the Fusion Registry supports multiple database platforms, the interface to the Fusion Registry is via the SDMX Web Services. By using web services, information required by external clients is obtained via URLs and the response is provided as an ISO-compliant SDMX (XML) format (ISO 17369: SDMX 2013). As the web services and response formats conform to the SDMX ISO Specification, it is possible to replace the Fusion Registry with another Registry implementation with no impact to existing clients. Additionally, web services have de-couple external applications from the programming language the Fusion Registry is written in, the database platform it is using to store the information, and the operating system it is installed on. No coupling means any of these aspects can be changed with no impact to consumers of this information, this includes the website, internal, and external clients.
The Fusion Registry provides its own maintenance interface, meaning there is no maintenance overhead of having to write a separate application.
As the client applications consume structural metadata using SDMX compliant web services, they can make use of any open source software to obtain the metadata. SdmxSource (www.sdmxsource.org) is an example of a good open source solution for consuming SDMX data without having to be SDMX literate.
Additionally external clients can make use of any pre-existing SDMX software code available.
The SDMX standard defines 19 different Maintainable structure types, and additionally the Registry specification defines Registrations and Subscriptions. The SDMX standard has 2 basic expressions, SDMX-ML (using XML syntax) and SDMX-EDI (using EDI syntax based on the GESMES/TS statistical message). SDMX-ML has undergone 3 revisions since 2001; Version 1.0, 2.0 and 2.1.
The Fusion Registry has aimed to support as much of the SDMX standard as possible for both import and dissemination via the web services, and visualisation and maintenance via the User Interfaces (UI). Additionally it is recognised that other formats such as CSV and Excel can simplify the maintenance effort of the metadata in the Registry.
With 6 supported input and export formats, a key feature of the Fusion registry is to automatically determine what format is being submitted via the import process, simplifying the process for the end user.
The following two tables show the supported input and export formats for each structure type, and the coverage of the SDMX structures which can be maintained and viewed via the Fusion Registry UIs.
The HTML User Interface provides a dataflow-centric view of the information in the registry, and to view reports of the information contained in the registry. The HTML interface is not intended to replace the Flash interface; it is intended to compliment it. While the Flash interface provides view and maintenance capabilities broken down for each structure type, the HTML interface combines this information, to provide a dataflow centric view of the information in the registry. Additionally the HTML view offers: free text searching of the registry content; the ability to create reports on codelist and concept usage in the registry; web services query builder; excel template creation for structure maintenance.
The Flash User Interface provides users with the ability to both view, create, modify, and delete structures and registrations to and from the Fusion Registry.
The Flash User Interface provides multilingual support, allowing structures to be created and viewed against multiple languages.
The Fusion Registry 8.4 now supports the FusionXL plugin for Microsoft Excel®. The plugin provides support for connecting to one or more Fusion Registry to both browse, modify, create, and delete structural content from the Fusion Registry.
The Fusion Registry can be integrated with external structure repositories (including the SDMX Global Registry) with the inclusion of support for SDMX Endpoints. The Flash UI provides the capability to add the location of an SDMX web service URL. It is then possible to PULL structures from that endpoint, and/or PUSH structures (securely if required) to the endpoint.
The structures will be duplicated in both the Fusion Registry and the endpoint service, however the Fusion Registry indicates to the user that the structure is externally maintained by means of an icon. If the structure is modified locally, then the Flash GUI will change the icon to indicate that the structure is out of sync with the endpoint. Structures can be PUSHed back to the endpoint securely using the given security credentials to authenticate. It is even possible to PUSH an entire tree of structures maintained by the same organisation, for example it is possible to PUSH a dataflow and include all the referenced structures referenced by the dataflow that are maintained by the owning Agency of the dataflow. This allows a single organisation to PUSH all of their structures securely to the Global Registry in a single click, without including structures maintained by other Agencies (such as the Cross Domain Concepts maintained by SDMX).
Every structure in SDMX has a version associated with it, defaulting at 1.0. It is possible to submit changes to a structure without changing its version number. The Auto Increment feature processes a structure submission detecting any changes to existing structures, if a change is detected then the version number of the structure in the submission is incremented from the latest version of the same structure. In this way, a submission to the auto increment service will never result in overwriting of existing structures. The service will detect if the change is minor, which will trigger a minor version update (v1.0 to v1.1 for example), or if the change is major, which will trigger a major version change (v1.0 to v2.0 for example).
If the version of an existing structure is updated then any structures which cross reference the structure will undergo a minor version revision with their reference updated to the new version of the structure.
It is possible to upload a file to the auto increment service using the Flash UI or the Registry Web Service.
The Fusion Registry provides the ability for authenticated organisations to subscribe to structure and data events in the Fusion Registry. The Registry will notify users via email or HTTP POST when specific events occur. The Subscription Notification mechanism adheres to the SDMX specification for Registry Subscription and Notification.
The Fusion Registry maintains an RSS feed which is updated after each submission or deletion to the Registry. The RSS feed provides a fixed URL allowing the user to retrieve an SDMX document containing all the structures in the submission. The RSS feed is available at a public URL allowing any user to subscribe to the feed using any software capable of monitoring RSS feeds (including web browsers).
The Fusion Registry supports the ability to view the modification history of any registry structure. Any previous version can be viewed, and it is also possible to roll back to a previous version.
The Fusion Registry supports the ability to view transaction history for the Registry as a whole, download the contents of the Registry as they were for a particular point in time, and to roll back the contents to particular point in time.
The Fusion Registry provides a service which will automatically and periodically sweep a directory on the local file system, processing any file in the directory as a structure submission. The sweeper engine makes it possible to run the Fusion Registry in a dissemination environment without requiring the need to connect to a database (as it can run with the default in-memory database).
Fusion Audit is a standalone web application which captures audit events from other Fusion Products such as Fusion Registry, Fusion Matrix, and Fusion Security. As well as general audit events, Fusion Audit also captures and categorises SDMX specific events such as structure requests, submissions, and security events. It provides an interactive HTML UI which allow users to see a breakdown of information such as what structure types were queried, which browser was used, the request duration, and more. Captured with each event are the associated log events, making it possible to drill into any event to view more detailed information.
Fusion Audit can be configured to email reports on usage at given intervals.
NOTE: Fusion Audit is only available to organisations with a Fusion Reigstry Support Contract.
Fusion Security is a standalone web application providing for user management and authentication services. Fusion Security makes it possible to run a secure registry without exposing any user management functions. Fusion Security obtains organisation information from the Fusion Registry, allowing a single user to belong to multiple organisations if required.
Fusion Security provides an additional layer of security by enabling automatic account lockdown on repeated login failure attempts and lock down of accounts to specific IPs or IP ranges.
Fusion Security can be configured connect to Fusion Audit to capture all security events such as account creation, deletion, modification.
Metadata Technology started the development of an SDMX Registry following the publication of the version 2.0 standard in 2005 which included for the first time the SDMX Registry Specification. This registry development went through two major iterations (using different technologies) between 2005 and 2010.
In 2010 Metadata Technology started work on a syntax and version independent component architecture and a Java implementation of this architecture. The registry was completely re-developed in 2011 using these components and consequently it was able to support SDMX version 2.1 very soon after it was able to support the 2.1 standard. This registry was branded the “Fusion Registry” and as a result of the ability to support all SDMX versions and syntaxes the Fusion Registry was adopted by the SDMX sponsors as the tool to be used for the SDMX Global Registry.
The component architecture and Java implementation were made available by Metadata Technology as open source in 2013.
The modular component approach to the architecture has resulted in the ability to add quickly new features and this led to many major new features to the Fusion Registry since 2011. One major feature is the ability to interact with Excel spreadsheets for download, maintenance, and upload of structural metadata.
Fusion Registry 8.4 has seen significant performance enhancements in both import and export of SDMX-ML in version 2.1. The underlying reading and writing of structures in this version of SDMX has been completely re-written which both boosts performance and reduces memory requirements. This change will support a much higher number of concurrent requests, with minimal burden on memory requirements.
In terms of performance increase, we tested import and export with a 32Mb SDMX-2.1 Structure Document. The test was performed against our last public release of the Fusion Registry v8.2 to Fusion Registry v8.4. The results are as follows:
Fusion Registry 8.2
Fusion Registry 8.4
In previous versions of the Fusion Registry, a user could subscribe to changes, but only if the user had a Fusion Registry account and was logged in.
Fusion Registry version 8.4 allows both anonymous and authenticated users to subscribe to changes to the Fusion Registry content.
We have had a lot of requests to allow changes to a structure Id after it had been set and committed to the Fusion Registry. This is a major change to how the Registry works, as a change to a single Id can have knock on effects for other structures in the Fusion Registry which reference that structure.
In version 8.4, the user is able to modify any Id of any structure using the Maintenance Interface. The Fusion Registry will in turn update all references, to ensure referential integrity is maintained.
In version 8.4 it is possible to create a copy of a structure by using the Maintenance Interface. The user is able to change the agency Id, id, or version of the copied structure.
We have totally re-engineered our error reporting for data registrations, we now perform a full validation of the dataset header, outputting meaningful error messages to the user when we detect a dataset which is not semantically valid.
We drive the Fusion Registry developments from feedback from our customers, so we are always looking to the future to try and support as many use cases as possible whilst simplifying process for organisations.
Based on recent feedback we have rleaseed the 8.4 version of the Fsuion Registry which now supports maintenance via Microsoft Excel® using the FusionXL plugin. We have also put a lot of work into enhancing the performance and memory management of the Fusion Registry for SDMX 2.1 imports and exports.
We have a lot of ideas for new features, such as support for the SDMX VTL (Validation and Transformation Language), wizards to assist in structure creation, support for other types of Metadata (non-SDMX).
We are very happy to hear all user feedback, we have even prepared a feedback form which we would love you to fill in Open Feedback Form.