Fusion Metadata Registry
SDMX Structural Metadata RegistryFusion Metadata Registry is a free to use structural metadata registry for SDMX supporting EDI, SDMX 1.0, 2.0, 2.1 and 3.0.
- Structural Metadata Registry.
- Author and maintain SDMX structures using the web user interface or programmatically through the REST API.
- Data validation - validates that data files are correctly structured.
- SDMX structure mapping - transform SDMX data to a different DSD and coding.
- SDMX data conversion - convert data files between SDMX data formats and versions.
- Download and install the Java web application, or deploy as a
container.
Sponsored by the Bank for International Settlements 
The FMR enables organisations to 'get a grip of their metadata', by externalising their statistical structural metadata with the benefits of maintainability, re-use, standardisation, harmonisation and improved metadata and data governance, achieved by:
- A controlled repository for SDMX structural metadata, which acts as a single source of truth for: automated metadata-driven data processing systems, metadata needed by data providers in data collections, structures for the purposes of standardisation, statistical data harmonisation, and communal benefit. Implements the SDMX 3.0 Registry specification.
- Three integrated metadata-driven data processing services: data validation, data transformation (mapping data between structures) and conversion of data between SDMX data formats. Usable through the web UI or integrated into data automation processes using the REST API.
- Metadata authoring and maintenance web UI, enabling metadata managers to browse, author and maintain their structural metadata model.
Fusion Metadata Registry 11 released
Fusion Metadata Registry version 11 implements the SDMX 3.0 specification.
Download get the Docker image
Backward compatibility with SDMX 2.1 and FMR 10 has been preserved where possible allowing upgrade to FMR 11 with few changes to operating procedures and intergated systems / processes.
Note that FMR 11 is not compatible with existing FMR 10 databases requiring a fresh install and export / import of the structural metadata content.
Features
| Function | Feature | Fusion Metadata Registry 10 | Fusion Metadata Registry 11 |
|---|---|---|---|
| Internal Information Model | SDMX 2.1 | ||
| SDMX 3.0 | |||
| Structural Metadata Formats | SDMX-ML v3.0 | ||
| SDMX-ML v2.1 | |||
| SDMX-ML v2.0 | |||
| SDMX-ML v1.0 | |||
| SDMX-JSON v3.0 | As of FMR 11.1.2: AgencyScheme Categorisation CategoryScheme Codelist ConceptScheme ContentConstraint DataConsumerScheme Dataflow DataProviderScheme DataStructureDefinition ProvisionAgreement RepresentationMap StructureMap The remainder of the structures are on the roadmap |
||
| SDMX-JSON v2.1 | |||
| EDI | |||
| Structural Metadata Management | Load structural metadata from file | ||
| Load structural metadata from URL | |||
| Author and maintain structural metadata using the web interface | |||
| Author and maintain structural metadata using FusionXL addin for Excel | |||
| ACID structure repository | |||
| Structural metadata import / export as SDMX | |||
| Structural metadata import / export as Excel | |||
| SDMX structures REST API | |||
| Structure storage (limited by available memory) | |||
| Structure comparison | |||
| Structure validation on import | |||
| Structure referential integrity enforcement | |||
| Tools to correct referential integrity issues (fix 'zombie' structures) | |||
| Time dependant constraints | |||
| Structure-level validity | |||
| Item-level validity | |||
| Subscription / notification of changes to structures via email or HTTP POST | |||
| Structure update transaction journalling | |||
| Rollback structure changes to a specified journalled transaction | |||
| Structural metadata time travel | On roadmap | ||
| SDMX Structures | Agency Schemes | ||
| Constraints | Converted to Data Constraints on import | ||
| Categorisations | |||
| Category Schemes | |||
| Codelists | |||
| Concept Schemes | |||
| Data Consumer Schemes | |||
| Data Provider Schemes | |||
| Data Structure Definitions | |||
| Dataflows | |||
| Hierarchical Codelists | Converted to Hierarchies on import | ||
| Metadata Structure Definitions | |||
| Metadataflows | |||
| Organisation Unit Schemes | |||
| Processes | |||
| Provision Agreements | |||
| Reporting Taxonomies | |||
| Structure Sets | Converted to Structure Maps on import | ||
| Structure Map | |||
| Representation Map | |||
| Organisation Scheme Map | |||
| Concept Scheme Map (SDMX 3.0) | On roadmap | ||
| Category Scheme Map | On roadmap | ||
| Reporting Taxonomy Map | |||
| Value List | |||
| Hierarchy | |||
| Hierarchy Association | |||
| Metadata Constraint | On roadmap | ||
| Data Constraint | |||
| Metadata Constraint | |||
| Metadata Provision Agreement | |||
| Metadata Provider Scheme | |||
| Metadataset | On roadmap | ||
| 'Extended' Structures (structures that are not part of the formal SDMX specification) | Validation Schemes | ||
| Reporting Templates | |||
| Publication Tables | |||
| Structural Metadata Storage Options | MySQL | ||
| Oracle | |||
| Microsoft SQL Server | |||
| Identity and Access Management | Microsoft Active Directory service integration | ||
| OpenLDAP directory service integration | |||
| Fusion Security directory service integration | |||
| Single-user mode (no directory service required) | |||
| Certificate authentication | |||
| SSO | |||
| Data | Load data from file | ||
| Load data from URL (file or REST API) | |||
| SDMX-ML v3.0 structure specific data format (XML) | On roadmap | ||
| SDMX-ML v2.1 structure specific data format (XML) | |||
| SDMX-ML v2.0 structure specific data format (XML) | |||
| SDMX-ML v2.1 generic data format (XML) | |||
| SDMX-ML v2.0 generic data format (XML) | |||
| SDMX-JSON v2.1 data format | |||
| SDMX-JSON v3.0 data format | On roadmap | ||
| SDMX-CSV v2.1 data format | |||
| SDMX-CSV v3.0 data format | (some limitations) | ||
| EDI data format | |||
| Excel FusionXL data format | |||
| Interactive data load using the web user interface | |||
| Data storage | |||
| Data processing REST API (validation, conversion, mapping) | |||
| Data storage REST API | |||
| Plugin interface for adding custom data readers | |||
| Data Registration | SDMX data source registration | On roadmap | |
| Validation and indexing of registered data sources | |||
| Data Validation | Structural compliance | ||
| Compliance of data values with the DSD | |||
| Compliance with constraint rules | |||
| Duplicate observations or series checks | |||
| Mandatory attributes | |||
| Validation Schemes - basic business rule validation expressions | |||
| Validation Schemes - advanced business rule validation expressions | |||
| Validation rules configuration - set what rules are applied and whether they prevent conversion or data publication | |||
| Plugin interface for adding custom validators | |||
| Data validation processing (load data, validate it against the defined structure and business rules, return a validation report plus valid and invalid series as separate output datasets) | |||
| Data validation REST API | |||
| VTL 2.0 | Storage of VTL programs as SDMX structural metadata | ||
| Execution of VTL programs for validation or transformation of data | |||
| SDMX Structure Mapping | Concept Scheme Maps (SDMX 2.1) | ||
| Codelist Maps | |||
| Data Structure Maps | |||
| Dataflow Maps | |||
| Regex pattern mapping rules | |||
| N-to-N data maps | |||
| N-to-1 data maps | |||
| 1-to-N data maps | |||
| 1-to-1 data maps | |||
| Mapped Datasets | Mapped Datasets - like SQL views | ||
| Virtual Dataflows | Virtual Dataflows describe a sub-cube of a dataflow | On roadmap | |
| Data Transformation | Data transformation processing (load data, transform it to a different structure using SDMX Structure Mapping and return the result) | ||
| Data transformation REST API | |||
| Data Conversion | Between supported SDMX formats and versions (SDMX-ML, SDMX-JSON, SDMX-CSV, EDI) | ||
| From Excel Reporting Template to SDMX formats | |||
| Quick convert to SDMX-ML 2.1 Structure Specific format | |||
| Conversion to and from Fusion Excel format for SDMX datasets | On roadmap | ||
| Data Reporting | Excel Reporting Templates - definition and generation | ||
| Excel Reporting Templates - validation and conversion of reported data | |||
| Excel Reporting Templates - load and storage of reported data | |||
| Excel Reporting Templates - configurable template security rules | |||
| Content Security / Access Controls | Structure creation and maintenance controls (Agency privileges) | ||
| Data load and modification controls (Data Provider privileges) | |||
| Data access controls at the dataflow, series or observation level | |||
| Structure access controls at the structure or item level | |||
| Public mode - Registry content is readable by both unauthenticated guest and authenticated users | |||
| Private mode - Registry content is accessible only to authenticated users | |||
| Auditing | SDMX structure transaction auditing | ||
| User activity auditing | |||
| Enterprise Deployment | Load balancer support (clustering multiple Registry instances behind a load balancer for resilience and load tolerance) | ||
| Apache Kafka Integration | Kafka 'Structure Notification' Producer for publishing changes to structural metadata on a definable topic | ||
| Client-side API | IM-JS JavaScript client-side programming library | ||
| SDMX-UI TypeScript client-side programming library (higher-level abstraction) | On roadmap | ||
| Command Line Interface | Fusion Command Line Interface for managing structural metadata |
Prioritised General Enhancements workstream
CompleteUse the FusionXL Excel add in for FMR to author, validate and transform SDMX datasets
- Reference metadata reports (metadataset)
- Support for reference metadata report / metadataset artefacts
- SDMX-JSON 3.0 structure format
- Support for the SDMX 3.0 JSON transmission format for structural metadata
- Improved error handling framework
- Improved capture, handling and reporting of errors such as referential integrity violations
- Structural metadata consistency checks and 'safe mode'
- Strengthened checks on Registry start-up with the Registry placed in 'safe mode' if problems are detected
- Performance improvements
- Improved support for very large structures, improved performance when processing large datasets
- Metadata time travel
- Retrieve structual metadata as it was at a defined point in the past
- Structure mapping default observation value
- Option to to set a default ouput observation value for use where the input value is missing
- SDMX-JSON 3.0 data format
- Support for the SDMX 3.0 JSON transmission format for data excluding hierarchical reference metadata embedded within the data message
- SDMX Data Registration
- Support for registering SDMX data sources
- Virtual dataflows
- Virtual dataflows describe a sub-cube of data similar in concept to a SQL view
Data processing microservices workstream
On work plan- Mapping microservice
- Decoupled stateless SDMX mapping microservice
See the full changelog for details of all new features and bug fixes.
11.1.6 - June 2022- New REST API web service parameter to control how duplicate obervations are handled on validation and transformation
- Introduction of MVStore for temporary storage and consolidation of datasets during processing operations delivering improved performance and reduced memory demand
- MySQL 8 support
- Publication Tables: Option to output rows with no data
- Publication Tables: Dimension Value based on another Dimenisons variable value
- Publication Tables: CPMI - Dynamic Rows and columns
- Load data from Excel workbooks in Fusion Excel data format
- Publication tables - new presentational metadata artefact describing curated tables of statistics for publication on websites or dashboards (more)
maintenance release
11.1.2 - April 2022maintenance release
11.1.1 - February 2022maintenance release
11.1.0 - February 2022- Code refactored to use the new SDMX 3.0 io.sdmx Java packages
- New validation rule to check that OBS_PRE_BREAK is consistent with OBS_STATUS
maintenance releases
11.0.1 - January 2022- SDMX 2.1 categorisations against hierarchical code lists applied to all relevant SDMX 3.0 hierarchies
- Internal information model changed to SDMX 3.0 (FMR 10 uses the SDMX 2.1 information model internally)
- Support for storage of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map
- Support for SDMX 3.0 XML format for structures
- Support for SDMX 3.0 CSV format for data
- Automatic conversion of SDMX 2.1 structure sets to the SDMX 3.0 mapping model on load
- Semantic versioning - support for three-digit version numbers
- JNDI database connection support
10.7 - May 2021
- Option to send structures as stubs on Kafka
- Data validation - new option to supply the SDMX Sender ID
- Mappings can be applied to multiple datasets if they refer to the same dataset
- Support the creation of the TIME_SERIES attributes from an EDI file
- Improved Apache Kafka integration
Fusion Metadata Registry acts as an Apache Kafka producer publishing changes to the SDMX structures on definable topics. This is useful in metadata-driven statistics production architectures where distributed applications like data validators need to hold a synchronised copy of the master structural metadata.
In 10.6, the Kafka interface has been enhanced to support all connection properties, email alerts are sent if the producer terminates, filters have been added to select the structures to publish and publication failure recovery has been strengthened. more
- Transaction and user activity auditing
Auditing of changes to SDMX structural metadata artefacts and user activity more - Structure transaction rollback
Journalling of changes to the structural metadata content allowing administrators to rollback to a specific point in time more - Server integrity
Checks the referential integrity of the structural metadata allowing administrators to fix problems found like 'zombie' artefacts that have broken references more - Docker image for containerised deployment
Fusion Metadata Registry is now available as an image on Docker Hub for faster containerised deployment with minimal setup quick start
What does Fusion Metadata Registry do?
Its primary function is an SDMX structural metadata registry.
FMR 10 implements the SDMX 2.1 registry specification.
FMR 11 implements the SDMX 3.0 registry specification.
In addition, FMR provides three built-in data processing functions:
- SDMX data validation
- SDMX data format conversion - convert datasets between the SDMX transmission formats (XML, JSON, CSV, EDI)
- SDMX data mapping - transforms the structure of a dataset using SDMX structure map rules
Plus a function to generate Excel forms to simplify the task of authoring and reporting data for specific dataflows.
Key use cases
Structural metadata management
Capacity and Limits
- Centralise and control structural metadata
- Author and maintain structural metadata
- Drive statistics production using centrally-controlled metadata
- Provide an SDMX-compliant a REST structure API
- Validate data for compliance with structural metadata, constraints and business rules (not VTL)
- Execute complex data mapping (SDMX structure maps, not VTL)
- Convert data between formats (convert between SDMX formats)
- Collect data using metadata-driven Excel forms
- Map incoming data into standardised structures
- Prepare SDMX data for reporting
- Validate data for compliance with the data collector's specifications prior to reporting
- No fixed limits on the number of users
- No fixed limits on the volume of structures - in practice the volume of structures is limited by the available compute platform memory
License Terms
Public license agreement
Support
Technical support contracts are available - please contact us for details.
Availability
Free for all.
A self-contained Fusion Metadata Registry image complete with a MySQL database suitable for testing, personal use and light production work loads.
1. Install Docker Desktop
Execute the following commands using the Windows Command App, Windows PowerShell, Linux terminal or Mac terminal.
2. Download the Fusion Metadata Registry Docker image
docker pull metadatatechnology/fmr-mysql:latest
3. Create a container called 'fmr'
docker container create --name fmr --publish 8080:8080 metadatatechnology/fmr-mysql:latest
4. Start the container
docker start fmr
Point your browser at http://localhost:8080
The container will take between 1 and 2 minutes to start - so be patient!
There is a single Root User Account:
Username: root
Password: password
Log in to administer the system, and create, load and maintain SDMX structures.
5. When finished - stop the container
docker stop fmr
More on Docker Hub.
Install on Windows, Linux or Apple Mac
Fusion Metadata Registry is a Java web applications that will run on most Windows, Linux or Apple Mac machines with a minimum of 4GB memory.
There are three key pre-requisities:
- A Java Runtime Environment (JRE or JDK) - Amazon Corretto 8 is a good free-to-use choice
- A Java web application server - we recommend Apache Tomcat 8.0, 8.5 or 9.0
- A database service - MySQL, Oracle or SQL Server are supported
Take a look at the quick start guide
Quick Start Guide