Overview

An SDMX Registry is a repository for the maintenance, storage, and dissemination of a specific type of metadata called ‘Structural Metadata’. Structural metadata can be used to describe the structure of datasets in any statistical domain. It is also used to describe the collection and dissemination of data and associated metadata. Structural metadata can also be used to describe rules and restrictions for data providers reporting datasets, it can describe statistical process, and help categorise datasets.

The purpose of an SDMX Registry is to act as a single point of access for structural metadata within an organisation, or between organisations. Although an SDMX Registry does not store datasets they provide a mechanism to identify where the datasets are located, this means an SDMX Registry describes datasets independently of how the data are stored.

The Fusion Registry is an implementation of an SDMX Registry, as described by the SDMX Specification.


Why Use an SDMX Registry?

The SDMX Information Model separates the data from the metadata used to describe the data. An SDMX Registry is built for the purpose of maintaining and disseminating the metadata used to describe the data (known as “structural metadata”), leaving the job of the data repository to focus fully on data storage and dissemination. Adopting this architecture leads to a decoupled and cohesive system, which can reduce software complexity and lead to a more maintainable solution which is resilient to change.

By adopting a Registry-based SDMX solution, all of the metadata required to describe the datasets, as well as rules pertaining to data collection, and disseminating data is held in the Fusion Registry. The Fusion Registry is used to describe the datasets for all the known domains, but will support all future data domains. The metadata is multilingual, allowing future support for names and descriptions in new languages.

Whilst the Fusion Registry supports multiple database platforms, the interface to the Fusion Registry is via the SDMX Web Services. By using web services, information required by external clients is obtained via URLs and the response is provided as an ISO-compliant SDMX (XML) format (ISO 17369: SDMX 2013). As the web services and response formats conform to the SDMX ISO Specification, it is possible to replace the Fusion Registry with another Registry implementation with no impact to existing clients. Additionally, web services have de-couple external applications from the programming language the Fusion Registry is written in, the database platform it is using to store the information, and the operating system it is installed on. No coupling means any of these aspects can be changed with no impact to consumers of this information, this includes the website, internal, and external clients.

The Fusion Registry provides its own maintenance interface, meaning there is no maintenance overhead of having to write a separate application.

As the client applications consume structural metadata using SDMX compliant web services, they can make use of any open source software to obtain the metadata. SdmxSource (www.sdmxsource.org) is an example of a good open source solution for consuming SDMX data without having to be SDMX literate.

Additionally external clients can make use of any pre-existing SDMX software code available.


Fusion Registry Features

High Coverage of Support for the SDMX-Information Model

The SDMX standard defines 19 different Maintainable structure types, and additionally the Registry specification defines Registrations and Subscriptions. The SDMX standard has 2 basic expressions, SDMX-ML (using XML syntax) and SDMX-EDI (using EDI syntax based on the GESMES/TS statistical message). SDMX-ML has undergone 3 revisions since 2001; Version 1.0, 2.0 and 2.1.

The Fusion Registry has aimed to support as much of the SDMX standard as possible for both import and dissemination via the web services, and visualisation and maintenance via the User Interfaces (UI). Additionally it is recognised that other formats such as CSV and Excel can simplify the maintenance effort of the metadata in the Registry.

With 6 supported input and export formats, a key feature of the Fusion registry is to automatically determine what format is being submitted via the import process, simplifying the process for the end user.

The following two tables show the supported input and export formats for each structure type, and the coverage of the SDMX structures which can be maintained and viewed via the Fusion Registry UIs.

Showing supported input and output formats
  1. SDMX-EDI imports are restricted to the rules imposed by the EDI standard, these can be more restrictive than SDMX. One such rule is that all structures in the EDI document must belong to the same maintenance Agency.
  2. CSV is supported for uploads only. You cannot ask the Fusion Registry to download a structure in CSV format. The XLSX format will provide a similar format to CSV.
  3. Agency Scheme, Data Provider Scheme and Data Consumer Scheme take a different semantic version prior to SDMX v2.1
  4. Note that in SDMX version 2.0 a Categorisation was embedded in the Category and Dataflow Structures. These constructs are supported in the Fusion Registry.
  5. For maintenance reasons, the Structure Set is limited to a single type of Map. Additionally the Structure Set is limited to a single Structure Map, and the two structures mapped must not exist in a Structure Map that is already contained in the Registry. The other types of Map can have multiple maps in the same Structure Set (e.g. one or more code list maps can exist) providing each is of the same type (e.g. Codelist Map).

Showing supported visualisation and maintenance formats

HTML User Interface

The HTML User Interface provides a dataflow-centric view of the information in the registry, and to view reports of the information contained in the registry. The HTML interface is not intended to replace the Flash interface; it is intended to compliment it. While the Flash interface provides view and maintenance capabilities broken down for each structure type, the HTML interface combines this information, to provide a dataflow centric view of the information in the registry. Additionally the HTML view offers: free text searching of the registry content; the ability to create reports on codelist and concept usage in the registry; web services query builder; excel template creation for structure maintenance.

Flash User Interface

The Flash User Interface provides users with the ability to both view, create, modify, and delete structures and registrations to and from the Fusion Registry.

The Flash User Interface provides multilingual support, allowing structures to be created and viewed against multiple languages.

More Information

Excel Support

The Fusion Registry 8.4 now supports the FusionXL plugin for Microsoft Excel®. The plugin provides support for connecting to one or more Fusion Registry to both browse, modify, create, and delete structural content from the Fusion Registry.

More Information

PUSH or PULL Structures to External SDMX Web Services

The Fusion Registry can be integrated with external structure repositories (including the SDMX Global Registry) with the inclusion of support for SDMX Endpoints. The Flash UI provides the capability to add the location of an SDMX web service URL. It is then possible to PULL structures from that endpoint, and/or PUSH structures (securely if required) to the endpoint.

The structures will be duplicated in both the Fusion Registry and the endpoint service, however the Fusion Registry indicates to the user that the structure is externally maintained by means of an icon. If the structure is modified locally, then the Flash GUI will change the icon to indicate that the structure is out of sync with the endpoint. Structures can be PUSHed back to the endpoint securely using the given security credentials to authenticate. It is even possible to PUSH an entire tree of structures maintained by the same organisation, for example it is possible to PUSH a dataflow and include all the referenced structures referenced by the dataflow that are maintained by the owning Agency of the dataflow. This allows a single organisation to PUSH all of their structures securely to the Global Registry in a single click, without including structures maintained by other Agencies (such as the Cross Domain Concepts maintained by SDMX).

More Information

Auto Increment Versions

Every structure in SDMX has a version associated with it, defaulting at 1.0. It is possible to submit changes to a structure without changing its version number. The Auto Increment feature processes a structure submission detecting any changes to existing structures, if a change is detected then the version number of the structure in the submission is incremented from the latest version of the same structure. In this way, a submission to the auto increment service will never result in overwriting of existing structures. The service will detect if the change is minor, which will trigger a minor version update (v1.0 to v1.1 for example), or if the change is major, which will trigger a major version change (v1.0 to v2.0 for example).

If the version of an existing structure is updated then any structures which cross reference the structure will undergo a minor version revision with their reference updated to the new version of the structure.

It is possible to upload a file to the auto increment service using the Flash UI or the Registry Web Service.

More Information

Subscription Notification

The Fusion Registry provides the ability for authenticated organisations to subscribe to structure and data events in the Fusion Registry. The Registry will notify users via email or HTTP POST when specific events occur. The Subscription Notification mechanism adheres to the SDMX specification for Registry Subscription and Notification.

More Information

RSS Feed

The Fusion Registry maintains an RSS feed which is updated after each submission or deletion to the Registry. The RSS feed provides a fixed URL allowing the user to retrieve an SDMX document containing all the structures in the submission. The RSS feed is available at a public URL allowing any user to subscribe to the feed using any software capable of monitoring RSS feeds (including web browsers).

More Information

Structure History/Rollback

The Fusion Registry supports the ability to view the modification history of any registry structure. Any previous version can be viewed, and it is also possible to roll back to a previous version.

More Information

Backup and Recovery

The Fusion Registry supports the ability to view transaction history for the Registry as a whole, download the contents of the Registry as they were for a particular point in time, and to roll back the contents to particular point in time.

More Information

Sweeper Engine

The Fusion Registry provides a service which will automatically and periodically sweep a directory on the local file system, processing any file in the directory as a structure submission. The sweeper engine makes it possible to run the Fusion Registry in a dissemination environment without requiring the need to connect to a database (as it can run with the default in-memory database).

More Information

Integration with Fusion Audit

Fusion Audit is a standalone web application which captures audit events from other Fusion Products such as Fusion Registry, Fusion Matrix, and Fusion Security. As well as general audit events, Fusion Audit also captures and categorises SDMX specific events such as structure requests, submissions, and security events. It provides an interactive HTML UI which allow users to see a breakdown of information such as what structure types were queried, which browser was used, the request duration, and more. Captured with each event are the associated log events, making it possible to drill into any event to view more detailed information.

Fusion Audit can be configured to email reports on usage at given intervals.

More Information

NOTE: Fusion Audit is only available to organisations with a Fusion Reigstry Support Contract.

Integration with Fusion Security

Fusion Security is a standalone web application providing for user management and authentication services. Fusion Security makes it possible to run a secure registry without exposing any user management functions. Fusion Security obtains organisation information from the Fusion Registry, allowing a single user to belong to multiple organisations if required.

Fusion Security provides an additional layer of security by enabling automatic account lockdown on repeated login failure attempts and lock down of accounts to specific IPs or IP ranges.

Fusion Security can be configured connect to Fusion Audit to capture all security events such as account creation, deletion, modification.

More Information


Provenance

Metadata Technology started the development of an SDMX Registry following the publication of the version 2.0 standard in 2005 which included for the first time the SDMX Registry Specification. This registry development went through two major iterations (using different technologies) between 2005 and 2010.

In 2010 Metadata Technology started work on a syntax and version independent component architecture and a Java implementation of this architecture. The registry was completely re-developed in 2011 using these components and consequently it was able to support SDMX version 2.1 very soon after it was able to support the 2.1 standard. This registry was branded the “Fusion Registry” and as a result of the ability to support all SDMX versions and syntaxes the Fusion Registry was adopted by the SDMX sponsors as the tool to be used for the SDMX Global Registry.

The component architecture and Java implementation were made available by Metadata Technology as open source in 2013.

The modular component approach to the architecture has resulted in the ability to add quickly new features and this led to many major new features to the Fusion Registry since 2011. One major feature is the ability to interact with Excel spreadsheets for download, maintenance, and upload of structural metadata.


What's new in Version 8.4?

Performance and Memory Enhancements

Fusion Registry 8.4 has seen significant performance enhancements in both import and export of SDMX-ML in version 2.1. The underlying reading and writing of structures in this version of SDMX has been completely re-written which both boosts performance and reduces memory requirements. This change will support a much higher number of concurrent requests, with minimal burden on memory requirements.

In terms of performance increase, we tested import and export with a 32Mb SDMX-2.1 Structure Document. The test was performed against our last public release of the Fusion Registry v8.2 to Fusion Registry v8.4. The results are as follows:

Action

Fusion Registry 8.2

Fusion Registry 8.4

Increase

Import

765s

17s

4400%

Export

30s

1.7s

1664%

Excel Plugin

With the Release of Fusion Registry 8.4 is the release of FusionXL our plugin for Microsoft Excel®. FusionXL allows a user to browse for and manage Fusion Registry content from within Microsoft Excel®.

Anonymous Subscriptions

In previous versions of the Fusion Registry, a user could subscribe to changes, but only if the user had a Fusion Registry account and was logged in.

Fusion Registry version 8.4 allows both anonymous and authenticated users to subscribe to changes to the Fusion Registry content.

Change Id of Structure

We have had a lot of requests to allow changes to a structure Id after it had been set and committed to the Fusion Registry. This is a major change to how the Registry works, as a change to a single Id can have knock on effects for other structures in the Fusion Registry which reference that structure.

In version 8.4, the user is able to modify any Id of any structure using the Maintenance Interface. The Fusion Registry will in turn update all references, to ensure referential integrity is maintained.

Copy Structures

In version 8.4 it is possible to create a copy of a structure by using the Maintenance Interface. The user is able to change the agency Id, id, or version of the copied structure.

Data Registrations – validation and Error Reporting

We have totally re-engineered our error reporting for data registrations, we now perform a full validation of the dataset header, outputting meaningful error messages to the user when we detect a dataset which is not semantically valid.


Change Log

8.4

Version 8.4.15 (Released: 2015/10/02)
  • [FR-373] - Critical fix for the Maintenance Tool addressed
Version 8.4.14 (Released: 2015/10/01)
  • [FR-359] - Server-side code for subscriptions addressed
  • [FR-360] - Now implements the new UI for the Subscription modal
  • [FR-94] - 404 error page added.
  • [FR-365] - SOAP issue - performing a Categorisation query now returns predictable results
  • [FR-366] - Web Service: submitting structures already in the Registry no longer returns a blank page
  • [FR-367] - HTML UI: Fixed minor issue in agency selector
Version 8.4.11 (Released: 2015/09/21)
  • Fixed issues with SOAP message
  • Fixed security issues with Cross-Site-Scripting
Version 8.4.10 (Released: 2015/07/27)
  • [FR-339] - Maintenance UI: Category Scheme now renders Arabic Catagory names
  • [FR-338] - HTML UI: Prevention of Javascript being injected into pages.
  • [FR-303] - Fixed issued with illegal Structure Urls
  • [FR-300] - Improvements to RSS feed
  • [FR-299] - HTML UI: Links to last updates do not show action in XML
  • [FR-290] - HTML UI: Export page now correctly deals with sub-agencies
  • [FR-287] - Now generates legal SDMX for MSD
  • [FR-286] - Fixed Registration Query issue
  • [FR-234] - Header of message now contains Receiver
  • [FR-231] - HTML UI: Error dialog now large enough to display all of error message
  • [FR-30] - HTML UI: Code Lists now sorted correctly
Version 8.4.6 (Released: 2015/06/10)
  • [FR-275] - Maintenance UI: Fixed indexing registrations
  • [FR-283] - Maintenance UI: Switching from "In Memory" database to "SQL Server" throws error
  • [FR-288] - Excel Plugin: Saving DSD sometimes throws an error
  • [FR-289] - Maintenance UI: Content Constraints issue fixed
Version 8.4.5 (Released: 2015/06/08)
  • [FR-271] - HTML UI: Does not display Arabic text
  • [FR-285] - Maintenance UI: Creating a Hierarchical Codelist fails
Version 8.4.3 (Released: 2015/06/02)
  • [FR-271] - HTML UI: Not displaying Arabic text
  • [FR-276] - SQL Server errors with NULL on save
Version 8.4.2 (Released: 2015/05/28)
  • Significant performance improvement in reading and writing SDMX 2.1 structures.
  • Support for Fusion XL - the Microsoft Excel Plugin.
  • Improved HTML User Interface.
  • Subscription to structural changes simplified.
  • [FTP-105] - Observation Attributes cause Failure
  • [FR-141] - Corrected issues in Soap
  • [FR-186] - Login Redirect setting added
  • [FR-210] - Database migration code added
  • [FR-215] - Maintenance UI: Add control to change the ID of a maintainable
  • [FR-224] - Move Agency Control
  • [FR-260] - Increase session timeout
  • [FR-270] - Maintenance UI: Attaching a constraint to a DataFlow throws Exception
  • Uses SdmxSource Version: 1.6.18

8.3

Version 8.3 (Released: 2015/03/17)
  • [FR-161] - Fusion Registry Excel Plugin - Versions are losing their suffix if ending .0
  • [FR-164] - HTML UI: Creation of HTML Subscription dialogs
  • [FR-167] - HTML UI: Spinner required on Login dialog
  • [FR-168] - HTML UI: Failure to login doesn't show an error.
  • [FR-172] - HTML UI: Add functionality so that a user can view their own subscriptions
  • [FR-173] - HTML-UI: Downloading XLSX stuctures improved
  • [FR-202] - HTML UI: Rest controls improved
  • [FR-176] - Maintenance UI: Remove Structure Subscription controls from Flex
  • [FR-178] - Maintenance UI: Email settings now have a clear control
  • Uses SdmxSource Version: 1.5.55

8.2

Version 8.2.1 (Released: 2014/11/20)
  • [FR-12] - HTML UI: Now shows Provision Agreement Constraint
  • [FR-68] - HTML UI: Now has the ability to Download a Structure Map
  • [FR-76] - Schema Generation Corrected
  • [FR-96] - New Feature: Support for Backup and Recovery added
  • [FR-98] - Support for latest web service query parameters added
  • [FR-103] - Maintenance UI: Can now register a datasource & index it
  • [FR-107] - HTML UI: Toggle Name/ID no longer resets the page view
  • [FR-115] - HTML UI: Structure Maps corrected
  • [FR-116] - HTML UI: A number of issues with screen re-sizing
  • [FR-121] - HTML UI: Show Name / Show ID doesn't persist across page changes
  • [FR-126] - HTML UI: Maintenance UI button is now "completely" clickable
  • [FR-134] - Maintenance UI: Endpoint controls added
  • [FR-151] - HTML UI: Login dialog now puts focus in Username field
  • [FR-152] - MQ Settings updated
  • [FR-155] - Further Fusion Audit support added
  • [FR-169] - HTML UI: Allows for Downloads in Excel Format
  • [FR-156] - HTML UI: Data Constraints now layed out correctly
  • [FR-148] - Bug fix: Allow non-admins to modify structures
  • [FR-142] - Registry can create XLSX files but creates them with the wrong extension
  • Uses SdmxSource Version: 1.4.4
Version 8.2 (Released: 2014/10/31)
  • [FR-28] - HTML UI: Overview page now updates after an upload
  • [FR-48] - HTML UI: now shows distinction between conditional and mandatory attributes
  • [FR-81] - Maintenance UI: Added control to clear date fields
  • [FR-97] - Maintenance UI: Performing a "Pull" updates the Overview screen
  • [FR-111] - Maintenance UI: Fixed NPE if XLSX Structure Map contains invalid structure
  • Uses SdmxSource Version: 1.4

8.1.6

Version 8.1.6 (Released: 2014/05/21)
  • [FR-22] - Fusion Registry fixed to work in Tomcat 8 - issue with servletContext.getRealPath
  • [FR-24] - Hibernate namespaces corrected
  • [FR-77] - HTML UI: "About" dialog fixed
  • [FR-83] - HTML UI: Bug in searching fixed
  • [FR-84] - HTML UI: Prevented XSS attack on search
  • [FR-85] - HTML UI: Corrected CSS buttons
  • [FR-93] - HTML UI: Fixed issue with drop-down for Agencies
  • [FR-49] - Maintenance UI: Dataflow page: Referenced Data Structure URN corrected
  • [FR-51] - Maintenance UI: Filter Codes in Codelist
  • [FR-54] - Maintenance UI: Modify Dataflow now selects correct DSD by default
  • [FR-57] - Maintenance UI: Editing a concept now selectes the correct version
  • [FR-58] - Maintenance UI: Concept Scheme now shows coded dimensions
  • [FR-59] - Maintenance UI: Home page imporved and now fits on small screens
  • [FR-60] - Maintenance UI: MSD page now has a control for setting the Target
  • [FR-61] - Maintenance UI: MSD Report Structure sometimes reports odd objects
  • [FR-87] - Maintenance UI: Fixed Array Index Out of Bounds when importing a CSV file
  • Uses SdmxSource Version: 1.2.8

8.0

Release Date : Not Public Release
  • Introduction of HTML interface
  • Reworking of Flex Maintenance Tool
  • Major performance improvements in parsing SDMX files
  • Users maintained by Fusion Security web application
  • #1661: Port 25 is always being used by the e-mail sender
  • #1657: Will not login using HTTPS
  • #1650: Modifying a DataProvider does not bring up the modification button
  • #1633: Hyphen (-) is not supported for an Id
  • #1627: URL for Registry Interface in 5.2 of User Guide is incorrect
  • #1603: Content Constraints are not always shown
  • #1602: Uploading a CSV requires the user to have permissions to modify all cross-referenced artefacts
  • #1600: CSV properties files are case-sensitive
  • #1599: When there is no selection in the Users page, Change Locale is still enabled
  • #1591: Data Consumers and Data Providers don't delete in the same way as all other structures
  • Uses SdmxSource Version: 1.1

The Future

We drive the Fusion Registry developments from feedback from our customers, so we are always looking to the future to try and support as many use cases as possible whilst simplifying process for organisations.

Based on recent feedback we have rleaseed the 8.4 version of the Fsuion Registry which now supports maintenance via Microsoft Excel® using the FusionXL plugin. We have also put a lot of work into enhancing the performance and memory management of the Fusion Registry for SDMX 2.1 imports and exports.

We have a lot of ideas for new features, such as support for the SDMX VTL (Validation and Transformation Language), wizards to assist in structure creation, support for other types of Metadata (non-SDMX).

We are very happy to hear all user feedback, we have even prepared a feedback form which we would love you to fill in Open Feedback Form.