Fusion Metadata Registry

SDMX Structural Metadata Registry

Fusion Metadata Registry is a free to use structural metadata registry for SDMX supporting EDI, SDMX 1.0, 2.0, 2.1 and 3.0.

  • Structural Metadata Registry.
  • Author and maintain SDMX structures using the web user interface or programmatically through the REST API.
  • Data validation - validates that data files are correctly structured.
  • SDMX structure mapping - transform SDMX data to a different DSD and coding.
  • SDMX data conversion - convert data files between SDMX data formats and versions.
  • Download and install the Java web application, or deploy as a container.

Sponsored by the Bank for International Settlements

The FMR enables organisations to 'get a grip of their metadata', by externalising their statistical structural metadata with the benefits of maintainability, re-use, standardisation, harmonisation and improved metadata and data governance, achieved by:

  • A controlled repository for SDMX structural metadata, which acts as a single source of truth for: automated metadata-driven data processing systems, metadata needed by data providers in data collections, structures for the purposes of standardisation, statistical data harmonisation, and communal benefit. Implements the SDMX 3.0 Registry specification.
  • Three integrated metadata-driven data processing services: data validation, data transformation (mapping data between structures) and conversion of data between SDMX data formats. Usable through the web UI or integrated into data automation processes using the REST API.
  • Metadata authoring and maintenance web UI, enabling metadata managers to browse, author and maintain their structural metadata model.

Fusion Metadata Registry 11 released

Fusion Metadata Registry version 11 implements the SDMX 3.0 specification.

Download get the Docker image

Backward compatibility with SDMX 2.1 and FMR 10 has been preserved where possible allowing upgrade to FMR 11 with few changes to operating procedures and intergated systems / processes.
Note that FMR 11 is not compatible with existing FMR 10 databases requiring a fresh install and export / import of the structural metadata content.

introductory video

Function Feature Fusion Metadata Registry 10 Fusion Metadata Registry 11
Internal Information Model SDMX 2.1
SDMX 3.0
Structural Metadata Formats SDMX-ML v3.0
SDMX-ML v2.1
SDMX-ML v2.0
SDMX-ML v1.0
SDMX-JSON v3.0 As of FMR 11.1.2: AgencyScheme Categorisation CategoryScheme Codelist ConceptScheme ContentConstraint DataConsumerScheme Dataflow DataProviderScheme DataStructureDefinition ProvisionAgreement RepresentationMap StructureMap
The remainder of the structures are on the roadmap
SDMX-JSON v2.1
EDI
Structural Metadata Management Load structural metadata from file
Load structural metadata from URL
Author and maintain structural metadata using the web interface
Author and maintain structural metadata using FusionXL addin for Excel
ACID structure repository
Structural metadata import / export as SDMX
Structural metadata import / export as Excel
SDMX structures REST API
Structure storage (limited by available memory)
Structure comparison
Structure validation on import
Structure referential integrity enforcement
Tools to correct referential integrity issues (fix 'zombie' structures)
Time dependant constraints
Structure-level validity
Item-level validity
Subscription / notification of changes to structures via email or HTTP POST
Structure update transaction journalling
Rollback structure changes to a specified journalled transaction
Structural metadata time travel On roadmap
SDMX Structures Agency Schemes
Constraints Converted to Data Constraints on import
Categorisations
Category Schemes
Codelists
Concept Schemes
Data Consumer Schemes
Data Provider Schemes
Data Structure Definitions
Dataflows
Hierarchical Codelists Converted to Hierarchies on import
Metadata Structure Definitions
Metadataflows
Organisation Unit Schemes
Processes
Provision Agreements
Reporting Taxonomies
Structure Sets Converted to Structure Maps on import
Structure Map
Representation Map
Organisation Scheme Map
Concept Scheme Map (SDMX 3.0) On roadmap
Category Scheme Map On roadmap
Reporting Taxonomy Map
Value List
Hierarchy
Hierarchy Association
Metadata Constraint On roadmap
Data Constraint
Metadata Constraint
Metadata Provision Agreement
Metadata Provider Scheme
Metadataset On roadmap
'Extended' Structures (structures that are not part of the formal SDMX specification) Validation Schemes
Reporting Templates
Publication Tables
Structural Metadata Storage Options MySQL
Oracle
Microsoft SQL Server
Identity and Access Management Microsoft Active Directory service integration
OpenLDAP directory service integration
Fusion Security directory service integration
Single-user mode (no directory service required)
Certificate authentication
SSO
Data Load data from file
Load data from URL (file or REST API)
SDMX-ML v3.0 structure specific data format (XML) On roadmap
SDMX-ML v2.1 structure specific data format (XML)
SDMX-ML v2.0 structure specific data format (XML)
SDMX-ML v2.1 generic data format (XML)
SDMX-ML v2.0 generic data format (XML)
SDMX-JSON v2.1 data format
SDMX-JSON v3.0 data format On roadmap
SDMX-CSV v2.1 data format
SDMX-CSV v3.0 data format (some limitations)
EDI data format
Excel FusionXL data format
Interactive data load using the web user interface
Data storage
Data processing REST API (validation, conversion, mapping)
Data storage REST API
Plugin interface for adding custom data readers
Data Registration SDMX data source registration On roadmap
Validation and indexing of registered data sources
Data Validation Structural compliance
Compliance of data values with the DSD
Compliance with constraint rules
Duplicate observations or series checks
Mandatory attributes
Validation Schemes - basic business rule validation expressions
Validation Schemes - advanced business rule validation expressions
Validation rules configuration - set what rules are applied and whether they prevent conversion or data publication
Plugin interface for adding custom validators
Data validation processing (load data, validate it against the defined structure and business rules, return a validation report plus valid and invalid series as separate output datasets)
Data validation REST API
VTL 2.0 Storage of VTL programs as SDMX structural metadata
Execution of VTL programs for validation or transformation of data
SDMX Structure Mapping Concept Scheme Maps (SDMX 2.1)
Codelist Maps
Data Structure Maps
Dataflow Maps
Regex pattern mapping rules
N-to-N data maps
N-to-1 data maps
1-to-N data maps
1-to-1 data maps
Mapped Datasets Mapped Datasets - like SQL views
Virtual Dataflows Virtual Dataflows describe a sub-cube of a dataflow On roadmap
Data Transformation Data transformation processing (load data, transform it to a different structure using SDMX Structure Mapping and return the result)
Data transformation REST API
Data Conversion Between supported SDMX formats and versions (SDMX-ML, SDMX-JSON, SDMX-CSV, EDI)
From Excel Reporting Template to SDMX formats
Quick convert to SDMX-ML 2.1 Structure Specific format
Conversion to and from Fusion Excel format for SDMX datasets On roadmap
Data Reporting Excel Reporting Templates - definition and generation
Excel Reporting Templates - validation and conversion of reported data
Excel Reporting Templates - load and storage of reported data
Excel Reporting Templates - configurable template security rules
Content Security / Access Controls Structure creation and maintenance controls (Agency privileges)
Data load and modification controls (Data Provider privileges)
Data access controls at the dataflow, series or observation level
Structure access controls at the structure or item level
Public mode - Registry content is readable by both unauthenticated guest and authenticated users
Private mode - Registry content is accessible only to authenticated users
Auditing SDMX structure transaction auditing
User activity auditing
Enterprise Deployment Load balancer support (clustering multiple Registry instances behind a load balancer for resilience and load tolerance)
Apache Kafka Integration Kafka 'Structure Notification' Producer for publishing changes to structural metadata on a definable topic
Client-side API IM-JS JavaScript client-side programming library
SDMX-UI TypeScript client-side programming library (higher-level abstraction) On roadmap
Command Line Interface Fusion Command Line Interface for managing structural metadata

Prioritised General Enhancements workstream

Complete
Publication tables
FMR 11.1.4
New presentational metadata artefact describing curated tables of statistics for publication on websites or dashboards (more)
Author, validate, map and transform data using Excel
FMR 11.1.5
Load data from Excel workbooks in FusionXL data format
Use the FusionXL Excel add in for FMR to author, validate and transform SDMX datasets
Improved data processing performance
FMR 11.1.6
Improved performance and reduced memory requirements when loading, validating, mapping and converting large datasets
On work plan
Reference metadata reports (metadataset)
Support for reference metadata report / metadataset artefacts
SDMX-JSON 3.0 structure format
Support for the SDMX 3.0 JSON transmission format for structural metadata
Improved error handling framework
Improved capture, handling and reporting of errors such as referential integrity violations
Structural metadata consistency checks and 'safe mode'
Strengthened checks on Registry start-up with the Registry placed in 'safe mode' if problems are detected
Performance improvements
Improved support for very large structures, improved performance when processing large datasets
Metadata time travel
Retrieve structual metadata as it was at a defined point in the past
Structure mapping default observation value
Option to to set a default ouput observation value for use where the input value is missing
SDMX-JSON 3.0 data format
Support for the SDMX 3.0 JSON transmission format for data excluding hierarchical reference metadata embedded within the data message
SDMX Data Registration
Support for registering SDMX data sources
Virtual dataflows
Virtual dataflows describe a sub-cube of data similar in concept to a SQL view

Data processing microservices workstream

On work plan
Mapping microservice
Decoupled stateless SDMX mapping microservice

See the full changelog for details of all new features and bug fixes.

11.1.6 - June 2022
  • New REST API web service parameter to control how duplicate obervations are handled on validation and transformation
  • Introduction of MVStore for temporary storage and consolidation of datasets during processing operations delivering improved performance and reduced memory demand
11.1.5 - June 2022
  • MySQL 8 support
  • Publication Tables: Option to output rows with no data
  • Publication Tables: Dimension Value based on another Dimenisons variable value
  • Publication Tables: CPMI - Dynamic Rows and columns
  • Load data from Excel workbooks in Fusion Excel data format
11.1.4 - May 2022
  • Publication tables - new presentational metadata artefact describing curated tables of statistics for publication on websites or dashboards (more)
11.1.3 - May 2022

maintenance release

11.1.2 - April 2022

maintenance release

11.1.1 - February 2022

maintenance release

11.1.0 - February 2022
  • Code refactored to use the new SDMX 3.0 io.sdmx Java packages
  • New validation rule to check that OBS_PRE_BREAK is consistent with OBS_STATUS
11.0.2 and 11.0.3 - February 2022

maintenance releases

11.0.1 - January 2022
  • SDMX 2.1 categorisations against hierarchical code lists applied to all relevant SDMX 3.0 hierarchies
11.0.0 - December 2021
  • Internal information model changed to SDMX 3.0 (FMR 10 uses the SDMX 2.1 information model internally)
  • Support for storage of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map
  • Support for SDMX 3.0 XML format for structures
  • Support for SDMX 3.0 CSV format for data
  • Automatic conversion of SDMX 2.1 structure sets to the SDMX 3.0 mapping model on load
  • Semantic versioning - support for three-digit version numbers
  • JNDI database connection support
More...
10.7 - May 2021
  • Option to send structures as stubs on Kafka
  • Data validation - new option to supply the SDMX Sender ID
  • Mappings can be applied to multiple datasets if they refer to the same dataset
  • Support the creation of the TIME_SERIES attributes from an EDI file
10.6
  • Improved Apache Kafka integration
    Fusion Metadata Registry acts as an Apache Kafka producer publishing changes to the SDMX structures on definable topics. This is useful in metadata-driven statistics production architectures where distributed applications like data validators need to hold a synchronised copy of the master structural metadata.
    In 10.6, the Kafka interface has been enhanced to support all connection properties, email alerts are sent if the producer terminates, filters have been added to select the structures to publish and publication failure recovery has been strengthened. more
10.5

What does Fusion Metadata Registry do?

Its primary function is an SDMX structural metadata registry.

FMR 10 implements the SDMX 2.1 registry specification.
FMR 11 implements the SDMX 3.0 registry specification.

In addition, FMR provides three built-in data processing functions:

  • SDMX data validation
  • SDMX data format conversion - convert datasets between the SDMX transmission formats (XML, JSON, CSV, EDI)
  • SDMX data mapping - transforms the structure of a dataset using SDMX structure map rules

Plus a function to generate Excel forms to simplify the task of authoring and reporting data for specific dataflows.

Key use cases

Structural metadata management
  • Centralise and control structural metadata
  • Author and maintain structural metadata
  • Drive statistics production using centrally-controlled metadata
  • Provide an SDMX-compliant a REST structure API
Statistics production
  • Validate data for compliance with structural metadata, constraints and business rules (not VTL)
  • Execute complex data mapping (SDMX structure maps, not VTL)
  • Convert data between formats (convert between SDMX formats)
Data collection
  • Collect data using metadata-driven Excel forms
  • Map incoming data into standardised structures
Data reporting
  • Prepare SDMX data for reporting
  • Validate data for compliance with the data collector's specifications prior to reporting
Capacity and Limits
  • No fixed limits on the number of users
  • No fixed limits on the volume of structures - in practice the volume of structures is limited by the available compute platform memory

License Terms
Public license agreement

Support
Technical support contracts are available - please contact us for details.

Availability
Free for all.

A self-contained Fusion Metadata Registry image complete with a MySQL database suitable for testing, personal use and light production work loads.
1. Install Docker Desktop
Execute the following commands using the Windows Command App, Windows PowerShell, Linux terminal or Mac terminal.
2. Download the Fusion Metadata Registry Docker image
docker pull metadatatechnology/fmr-mysql:latest
3. Create a container called 'fmr'
docker container create --name fmr --publish 8080:8080 metadatatechnology/fmr-mysql:latest
4. Start the container
docker start fmr

Point your browser at http://localhost:8080

The container will take between 1 and 2 minutes to start - so be patient!

There is a single Root User Account:
Username: root
Password: password

Log in to administer the system, and create, load and maintain SDMX structures.

5. When finished - stop the container
docker stop fmr

More on Docker Hub.

Install on Windows, Linux or Apple Mac

Fusion Metadata Registry is a Java web applications that will run on most Windows, Linux or Apple Mac machines with a minimum of 4GB memory.

There are three key pre-requisities:

  • A Java Runtime Environment (JRE or JDK) - Amazon Corretto 8 is a good free-to-use choice
  • A Java web application server - we recommend Apache Tomcat 8.0, 8.5 or 9.0
  • A database service - MySQL, Oracle or SQL Server are supported

Take a look at the quick start guide

Quick Start Guide