Fusion Registry®10

SDMX Statistical Data Warehouse

Fusion Registry 10 is for central banks, international organisations, national statistics offices, development banks and regional statistics authorities who need an enterprise-grade statistical data warehouse for the collection, integration, storage, processing and publication of aggregated official statistics using the SDMX standard.

SDMX is used right down to its core, so Fusion Registry is not just a database with an SDMX interface but a true multi-dimensional aggregated statistical data warehouse designed by statisticians explicitly to meet the requirements of official statistics organisations.

Live Demo

SDMX is designed for statistics.
SDMX provides a structured and standardised way to describe statistical data in any domain, and a common language for exchanging both the data and its structural metadata between organisations and systems (information about how the data is or should be structured).

It allows statisticians to robustly describe the structure and rules for complex datasets, provides data providers with a clear description about what the data should look like, and data users with an unambiguous understanding of how to interpret it.

SDMX is a proven, open, non-proprietary standard sponsored by seven international organisations including the World Bank, the International Monetary Fund and the United Nations Statistics Division. It's also an International Standard ISO 17369:2013.

SDMX-ML v1.0, v2.0 and v2.1
Data sub-formats: v2.0 Generic, v2.0 Compact, v2.1 Generic, v2.1 Structure Specific
SDMX-EDI
SDMX-JSON 1.0 (data and structures)

SDMX 3.0 will be supported when it is released.

No
Fusion Registry 10 is closed-source with a proprietary commerical right to use license.

However, if you're a central bank or international organisation user of the Enterprise Edition, you can get access to the source code by joining the Development Collaboration Programme.

Yes
Yes, Metadata Technology in conjuction with our integration partners can design and build complete solutions to meet your official statistics requirements based on Fusion Registry 10.

No

Fusion Registry 10 is a only available as a commercial product - either installed on-premises, or delivered as a fully-managed cloud service (SaaS).

Yes

Fusion Registry 10 Cloud gives you a SDMX statistical data warehouse service in the cloud, installed, configured and ready to go. There's a live demonstration here.

Pricing information can be found here.

Structural Metadata is information about how the data should be structured in terms of its dimensionality, the concepts being measured and how they are classified.
SDMX provides a range of 'structural metadata artefacts' including Concept (e.g. frequency, country, gender), Codelist (classifications) and Data Structure Definition (describing the dimensionality of a particular dataset).
Statisticians use structural metadata artefacts to define their statistical data model in Fusion Registry in the same way that a SQL database administrator will use tables, columns and keys to define the schema for a relational database.
While a Data Structure Definition (DSD) sets out the structure or dimensionality of a dataset, a Dataflow describes a specific instance. For instance, a DSD for National Accounts could be used for GDP, Central Government Debt and CPI Dataflows - different datasets that all share the same dimensionality. Data consumers retrieve the data they are interested in by querying the relevant Dataflow. In some ways, Dataflows are equivalent to Tables in the SQL relational model.
Structural metadata is also used to describe rules to be applied when collecting and processing data. In particular, Content Constraints set what data a provider is authorised to report.

,

Reference Metadata (also called Referential Metadata) is human-readable information that can be attached to datasets, series, observations, concepts and other data and structural metadata elements. If you want to provide some explanatory notes to help users interpret a dataset, Reference Metadata is the thing to use. And it's not just restricted to text. Reference Metadata can be plain text, HTML or links to other documents and resources such as videos and images.

Time Series: Yes
Fusion Registry is designed primarily for aggregated time series.

Cross Sectional: Partial
Fusion Registry can model cross sectional data and manage it to an extent. However, many functions assume the presence of a time dimension so the best approach is to model cross sectional datasets with a time dimension, but give all observations the same survey time.

No: only aggregated data at present.
This is parimarily a limitation of the current 2.1 version of SDMX which is mainly concerned with aggregated data. Better support for microdata is planned in the forthcoming SDMX 3.0 information model including alternative measure cardinality on Data Structure Definitions, and more relaxed Codelists that don't need to comply with the current strict rules imposed by the standard.

No: not at present.
VTL 2.0 support is on the development roadmap.

Yes: using Metadata Technology's Time Series Expression Language.
Fusion Registry's time series calculation engine executes formulas in expressed in Metadata Technology's Time Series Expression Language on existing series in its statistical data repository to create new series. In this sense it follows the functional programming paradigm where the series / datasets referenced by an expression are immutable, and the result is always new series.

  • Transformation - SDMX Structure Maps can be used to transform data to different structures and classification schemes.
  • Calculation - Calculate new series using Metadata Technology's Time Series Expression Language.
  • Format Conversion - Convert data between formats including SDMX formats, non-SDMX formats like Excel.
  • Aggregation - Aggregate series to a higher level using definable functions.
  • Frequency Conversion - Convert time series to different frequencies using definable re-sampling and interpolation functions.
  • Structural Validation - Check data for compliance with the information model defined by the SDMX structural metadata. This can also check for other potential problems such as duplication observations.
  • Semantic Validation - Check data for compliance with definable business rules such as observations for individual classifications match the value reported for TOTAL.

Fusion Registry does not have a workflow engine so has no built-in way for users to define and execute processes consisting of sequences of tasks.
However, third party workflow tools can be easily integrated using Fusion Registry's rich REST API to execute tasks and receive results.
Shell scripts (.sh, .bat and Windows Powershell) can also be created using Fusion Command Line Interface allowing certain tasks such as loading data and metadata to be scripted.

No
Although all functions can be executed using the REST API, so tools like Postman or Curl can be used instead.

Registry Content: Yes
Registry content (names, labels, descriptions etc) can be loaded, displayed and retrieved in multiple languages, including those written right-to-left such as Arabic and Hebrew. Users can choose their preferred language - content will be displayed in that language if available, or the default language otherwise. The default is often English, but it doesn't need to be.

Fusion Registry Web User Interface: No
Text on Fusion Registry's web user interface - including the menu bar, buttons and informational messages displayed for various reasons - are always in English.

Fusion Data Browser Web User Interface: Yes
Legends, buttons, menu items and other text can all appear in a language of a your choice using configurable language translation JSON files.

Fusion Registry is a Java web application so it will run on any computing platform with a Java run-time environment and a suitable Java web application server such as Apache Tomcat. It has a web user interface so users require only a standard web browser.

It's also designed to be very scalable, so can be run on most computers and servers with a minimum of 8GB of memory. More memory and processing capacity will be required depending on the volume of data and usage. There's more on this in the What are the system requirements? section.

Fusion Registry provides a comprehensive REST web services API for integration with other systems. The API is compliant with the standard SDMX RESTful web services specification, and also provides a range of other functions:

  • Query for data - retrieve data for a specified selection of series from a dataset
  • Submit data content - load data into local Fusion Registry managed data stores, including the in-memory database
  • Delete data content - delete selected datasets, series and observations
  • Rollback data change - rollback data submissions
  • Data registration - registration of external data sources
  • Data reporting forms - generate Excel forms for reporting data
  • Freetext search - search the metadata and data content
  • Reference metadata - query for and submit reference metadata
  • Audit - query audit logs
  • Fusion Edge Server - take snapshots of the data warehouse for publication using Fusion Edge Servers
  • Content Security - query for and manage Content Security rules

Operating System - Windows or Linux. Other platforms including Apple Mac, HP-UX and IBM AIX should also work provided they have a suitable Java Runtime Environment (JRE) or Java Development Kit (JDK), and Java Web Application Server.

Java Runtime Environment - Java 8 and Java 10 are supported.

Java Web Application Server - Apache Tomcat 8.0, 8.5 or 9.0 are recommended. Jetty is also tested and proven. Other application servers like IBM WebSphere Application Server should also work.

Processor - A minimum of 4 cores.

Memory- Fusion Registry 10 is designed to be very memory efficient. 4GB is sufficient for testing and basic use cases. However, plan for between 8GB and 32GB for heavier workloads or where extensive use of the Fusion Data Store in-memory database is planned.

Web Server - A web server is recommended working in reverse-proxy mode for all production systems. Apache Web Server 2.4 is recommended although cloud deployment platforms such as Amazon Web Services and Microsoft Azure generally provide a reverse proxy service meaning that Apache is not required in those circumstances.

"Fusion Registry 10 distinguishes itself by providing a fully supported commercial alternative to open-source, freeware and community developed tools, making it suitable as a key component in mission-critical systems."

Key Features

SDMX 1.0,2.0,2.1 and EDI

All versions of SDMX are supported, including EDI. Preparations are already being made for version 3.0 which will also be supported when released.

SDMX Structural Metadata

The Enterprise Edition incorporates the same best-in-class SDMX structural metadata registry as used for the SDMX Global Registry. Author and manage all of the standard structures including Codelists, Concepts, Data Structure Definitions and Dataflows using Fusion Registry's intuitive web GUI.

Data Collection

Control what data are collected from which providers and when using SDMX Provision Agreements, Reporting Constraints and advance release calendars. Register URLs to automatically integrate data from data providers' SDMX REST APIs or SDMX files, submit data interactively using the web GUI in Excel, CSV or SDMX, or POST data programatically using the Fusion Registry SDMX REST API. Automatically generate custom Excel forms for data reporters to complete and submit.

Data Integration

Fusion Registry seamlessly integrates data from any number of local and remote sources into a single statistical data warehouse. Sources include local stores (data loaded interactively through the GUI or programatically through the REST API, distributed Fusion Registry services, Data Portal sources and ETL sources (primarily SQL databases). Data for a single dataset can even come from multiple different sources with a separate subset of series coming from each source.

Data Validation

Choose from the catalogue of built-in validation rules checking for a range of potential issues including structural non-compliance, incorrect coding and duplicate observations. Write your own rules using Metadata Technology's expression language, or add custom validators with the Fusion Registry plug-in framework.

Data Dissemination

Publish data using Fusion Registry's SDMX REST data API and high-performance query execution engine. Provide end users with direct access to the API, and / or use it as a data source to drive other dissemination services such the Fusion Data Browser, statistics web portals or mobile apps.

Data Storage

Choose where and how Fusion Registry stores your SDMX data. Options include standard SQL databases: MySQL, Oracle and SQL Server, and Metadata Technology's high-performance persitant in-memory database. You can even choose different stores for different datasets perhaps depending on their size, confidentiality or performance requirements.

REST API

Fusion Registry provides a fully-compliant SDMX REST API for both data and metadata. In addition, Metadata Technology's extended API provides programmatic access to a further range of useful functions including data validation, data mapping and free-text search.

Fusion Data Browser

Web tool for data search, visualisation and analysis.
Fusion Data Browser is a web application for data consumers to search and explore the catalogue of datasets, select series and cubes of interest, build visualisations using tables and charts, and retrieve selected data in a variety of formats including SDMX, Excel and CSV.
It can be used with any Fusion Registry dissemination interface so will work both with Fusion Registry environments and Fusion Edge Server clusters. That makes it equally useful as an internal tool for statistics departments and internal data consumers, or for external use on public data portals.

Reference Metadata

Reference Metadata (sometimes called Referential Metadata) can be thought of as comments or even rich content with images and markup that can be attached to any statistical element from Concepts and Dataflows to series and individual observations. Practical applications include recording qualitative information about datasets and datapoints, and creating glossaries of terms. The ENterprise Edition supports definition, authoring, collection and dissemination of Reference Metadata.

Data Portal

Automatically retrieve, download and seamlessly integrate copies of selected datasets from a range of public and private data APIs into Fusion Registry's statistical datawarehouse. Choose how fresh you want the data and Fusion Registry will update on your reqired schedule. Supporting APIs include Eurostat, ECB, IMF, OECD and BIS. Support for other data services can be added using Fusion Registry's plugin framework.

Data Structure Mapping

Transform datasets to different structures and coding schemes using Fusion Registry's SDMX Structure Mapping engine which includes support for Codelist maps, N-to-N series mapping and regex pattern matching.

Content Security

Fusion Registry's content security sub-system allows fine-grained control over exactly what metadata and data groups of users have access to. Rules can even be created down to individual codes and observations providing the opportunity to grant general access to datasets while keeping certain series or observations private.

IMF NSDP

IMF member countries subscribing to the eGDDS, SDDS and SDDS Plus dissemination standards are required to publish a National Summary Data Page (NSDP), a web page summarising a specific collection of key datasets. Fusion Registry can automatically generate a dynamic NSDP including the SDMX download links and tabular data browser mandatory for SDDS Plus.

IM-JS and SDMX-UI

IM-JS and SDMX-UI provide JavaScript and TypeScript libraries for building custom Fusion Registry data-driven web applications. The libraries simplify the process of connecting to a Fusion Registry REST API and retrieving data and metadata. SDMX-UI provides higher-level visual widgets including interactive pivot tables and metadata driven query builders.

Load Balancing / High Availability

High availability configurations are supported by clustering two or more Fusion Registry instances behind a load balancer. It's cache consistency monitor recognises when updates have been made to data or metadata and automatically purge the necessary cache entries across the entire cluster to avoid the risk of serving stale content.

Fusion Edge Server

Fusion Edge Server is a secure high-performance data dissemination server that works in tandem with Fusion Registry. It's designed for public dissemination use cases that need scalability, high-performance and resilience, and where it is desirable to isolate the master Fusion Registry. Fusion Edge Server has no user interface and provides a read-only SDMX REST API suitable for driving a range of dissemination services and tools.

Audit

Fusion Registry's audit sub-system maintains a comprehensive record of all metadata and data changes.

Fusion Registry 10 Deployment Options


Cloud

SDMX data warehouse delivered as a managed software as a service (SaaS)

A cloud hosted Fusion Registry 10 SDMX statistical data warehouse pre-configured, maintained and ready to go.

Annual subscription.

On-Premises

Custom designed SDMX data and metadata solutions

Metadata Technology will custom design solutions to precisely meet your SDMX data and metadata management needs using the Fusion Registry 10 toolset for on-premises deployment.

You can choose to implement the solutions yourself, if ncessary buying specific advice and services like modelling or SDMX training to support your project.

Alternatively Metadata Technology, in conjunction with our integration partners, can plan and manage your entire implementation project including detailed design, modelling of your data domains in SDMX, data collection and ETL processes, data migration, custom dashboards / dissemination portals, training and handover.

Compare Features

Cloud
Modules
Fusion Registry Core
Fusion Data Browser
Features
SDMX 1.0,2.0,2.1 & EDI
SDMX Structural Metadata
Data Validation
Data Structure Mapping
SDMX REST API
Extended REST API
Full Data Collection
Data Storage (all options)
Data Dissemination
Reference Metadata
Data Portal
Data Integration
Content Security
IM-JS and SDMX-UI
1GB of structural metadata expressed in SDMX-ML 2.1
100 registered users
100 registered data providers
Data Limits: 30 million observations
On-premises
Modules
Fusion Registry Core
Fusion Data Browser
Fusion Edge Server
Fusion Registry Command Line
Features
SDMX 1.0,2.0,2.1 & EDI
SDMX Structural Metadata
Data Validation
Data Structure Mapping
SDMX REST API
Extended REST API
Full Data Collection
Data Storage (all options)
Data Dissemination
Reference Metadata
Data Portal
Data Integration
Content Security
IM-JS and SDMX-UI
Load balancing / high availability
Audit
Unlimited structural metadata*
Unlimited registered users**
Unlimited registered data providers*
Unlimited data*
* While there are no software restrictions, the actual volume of structures and data that can be loaded and stored is dependent on the amount of memory allocated to the Fusion Registry instance
** Fusion Registry places no limits on the number of registered users, however the practical number of users is determined by the capacity of the Microsoft Active Directory or LDAP directory service used

The Full Feature List

Function Feature Cloud On-Premises
Internal Information Model SDMX 2.1
SDMX Input / Output Support SDMX-ML(1.0, 2.0, 2.1), SDMX-JSON(1.0), SDMX-EDI
Structural Metadata Management Author and maintain structural metadata using the web interface
Author and maintain structural metadata using FusionXL addin for Excel
ACID structure repository
Structural metadata import / export as SDMX
Structural metadata import / export as Excel
SDMX structures REST API
Structure storage (no limits)
Structure comparison
Structure validation on import
Structure referential integrity enforcement
Time dependant constraints
Structure-level validity
Item-level validity
Subscription / notification of changes to structures via email or HTTP POST
Structure update transaction journalling
Rollback structure changes to a specified journalled transaction
SDMX Structures Agency Schemes
Attachment Constraints
Categorisations
Category Schemes
Codelists
Concept Schemes
Data Consumer Schemes
Data Provider Schemes
Data Structure Definitions
Dataflows
Hierarchical Codelists
Metadata Structure Definitions
Metadataflows
Organisation Unit Schemes
Processes
Provision Agreements
Reporting Taxonomies
Structure Sets
Fuson Registry 'Extended' Structures Validation Schemes
Reporting Templates
Structural Metadata Storage Options MySQL
Oracle
Microsoft SQL Server
Identity and Access Management Microsoft Active Directory service integration
Active Directory / Kerberos SSO
OpenLDAP directory service integration
Fusion Security directory service integration
Single-user mode (no directory service required)
Certificate autentication
Data Load From File or URL SDMX Data Formats (SDMX-ML, SDMX-JSON, SDMX-CSV, EDI)
Excel
Fusion CSV (non-SDMX variant)
Interactive data load using the web user interface
Interactive store (publish) of loaded data through the web user interface
Programmatic data load and store using the REST API
Plugin interface for adding custom data readers
Data Collection Control how data is collected from providers using SDMX Provision Agreements
Data Providers Multiple Multiple
Reporting Constraints - control what data each provider can report
Registration - allows data providers to report by registering a file or SDMX REST URL for each dataset
Data ETL Query-for and retrieve data on-demand from SQL databases in the Fusion Registry defined schema (MySQL, Oracle and SQL Server)
Extract, map and load data from arbitrary SQL database schemas Unsupported in the current release
Big Data sources (e.g. HDFS) Unsupported in the current release
Data Portal (beta)
(retrieve and automatically integrate selected data from distributed data services on a defined schedule)
Fusion Registry data sources
SDMX compliant REST API data sources
Eurostat
Plugin interface for adding additional data sources
Data Integration - Supported Data Sources Fusion Registry managed local data stores
Distributed Fusion Registries
Data Portal Sources
Data ETL Sources
Data Validation Structural compliance
Compliance of data values with the DSD
Compliance with constraint rules
Duplicate observations or series checks
Mandatory attributes
Validation Schemes - basic business rule validation expressions
Validation Schemes - advanced business rule validation expressions
Validation rules configuration - set what rules are applied and whether they prevent conversion or data publication
Plugin interface for adding custom validators
Data validation processing (load data, validate it against the defined structure and business rules, return a validation report plus valid and invalid series as separate output datasets)
Data validation REST API
VTL VTL 2.0 Unsupported in the current release
SDMX Structure Maps Concept Scheme Maps
Codelist Maps
Data Structure Maps
Dataflow Maps
Regex pattern mapping rules
N-to-N data maps
N-to-1 data maps
1-to-N data maps
1-to-1 data maps
Mapped Datasets Mapped Datasets (like SQL 'views', create new datasets in the statistical data warehouse by applying SDMX Structure Maps to existing datasets)
Data Transformation Data transformation processing (load data, transform it to a different structure using SDMX Structure Mapping and return the result)
Data transformation REST API
Time Series Calculations / Formulae (beta) Calculated Datasets (like SQL 'views', create new datasets in the statistical data warehouse using functions / formulae written in Metadata Technology's expression language)
Data Query Calculations (use functions / formulae written in Metadata Technology's expression language in data queries to perform calculations on the results)
Data Storage SQL Databases (MySQL, Oracle, SQL Server)
Fusion Data Store high-performance in-memory database
Data Conversion Between SDMX formats and versions (SDMX-ML, SDMX-JSON, SDMX-CSV, EDI)
Excel
Fusion CSV (non-SDMX variant)
Data Reporting Excel Reporting Templates - definition and generation
Excel Reporting Templates - validation and load of reported data
Data Dissemination SDMX data REST API
Availability API
Aggregation queries - aggregate REST API query results on-the-fly
Non-SDMX REST query response formats (Excel, CSV, RDF, Fusion-JSON)
Query response data compression
Embeddable and shareable live charts and tables (create using Fusion Data Browser)
Tableau Data Connector
IMF compliant National Summary Data Page (NSDP)
Fusion Cache (server-side query result caching)
HTTP 304 'Not Modified' support (client-side caching)
Fusion Data Browser
(web data explorer for searching, querying, charting and retrieving data)
Configurable tree view for drilling into the data catalogue by concept
Series list search results with filters
Multi-dimensional query builder
Free-text search
Saved queries
Time series frequency conversion
Calculated measures (e.g. % change)
Series basket
Pivot table
Charts (time series, bar, pie, pyramid, heatmap)
Geospatial maps Unsupported in the current release
Export data (Excel, CSV, SDMX)
Create embeddable charts and tables
Fusion Edge Server
(secure high-performance data dissemination server)
SDMX Data REST API
Horizontal scaling with load balancer support for clusters of Edge Servers
Embargo - publication of data at a precise time
Selective publication - choose exactly which datasets to publish
Full refresh or incremental data updates
Content Security Structure creation and maintenance controls (Agency privileges)
Data load and modification controls (Data Provider privileges)
Data access controls at the dataflow, series or observation level
Structure access controls at the structure or item level
Enterprise Deployment Audit
Load balancer support (for load-balancing multiple Fusion Registry instances)
'Varnish' web cache / reverse proxy integration
Apache Kafka Integration Kafka 'Structure Notification' Producer for publishing changes to structural metadata on a definable topic
Client-side API IM-JS JavaScript client-side programming library
SDMX-UI TypeScript client-side programming library (higher-level abstraction)
Command Line Interface Fusion Command Line Interface
Under the covers

If you're interested, the schematic below shows how the Fusion Registry Core statistical data engine works at a logical level

Fusion Registry 10 components, modules and integrations

1. SDMX Metadata Registry
The central repository and maintenance tool for all of the SQL structural metadata including Codelists, Concepts and Data Structure Definitions.

2. Fusion Registry Core
Statistical data engine.
The main statistical data engine responsible for all data collection, integration processing and storage. It also includes an optimised query execution engine and SDMX data and metadata REST API allowing it to be used on its own for applications like internal statistical data warehouses.

3. SQL Database Storage
Fusion Registry Core can be configured to use one or more standard SQL databases for metadata and data storage. You'll need to provide your own database but MySQL, SQL Server and Oracle are all supported.

4. Fusion Edge Server
Secure high-performance data dissemination server.
Fusion Edge Server is a separate light-weight dissemination module that works in tandem with Fusion Registry. It's designed for public dissemination use cases that need scalability, high-performance and resilience, and where it is desirable to isolate the master Fusion Registry. Fusion Edge Server has no user interface and provides a read-only SDMX REST API suitable for driving a range of dissemination services and tools.

5. Fusion Data Browser
Web tool for data search, visualisation and analysis.
Fusion Data Browser is a web application suitable for both internal and external users to search and explore the catalogue of datasets, select series and cubes of interest, build visualisations using tables and charts, and retrieve selected data in a variety of formats including SDMX, Excel and CSV.

6. NSDP
IMF compliant data dissemination web page.
The National Summary Data Page (NSDP) is a web application that automatically summarises selected datasets on a single web page in a format compliant with the IMF's requirements for countries subscribing to the eGDDS, SDDS and SDDS Plus dissemination standards.

7. IM-JS and SDMX-UI
JavaScript and TypeScript libraries for developing custom data-driven web applications like data portals and dashboards.

8. Excel
Microsoft Excel can be used for (a) preparing and submitting data, (b) authoring and maintaining structures, and (c) analytics of published data.

9. Tableau
Connect Tableau directly to Fusion Registry's data dissemination services for analytics and for building rich visualisations / dashboards.

10. Active Directory or OpenLDAP Service
A Microsoft Active Directory or OpenLDAP directory service is required for authentication and role management.