The Fusion Matrix is a multi-functional application
supporting multiple uses cases for the dissemination of data and associated
The Fusion Matrix provides a scalable, performant, and
robust solution for the storage and retrieval of data and associated metadata for
any statistical domain. At the heart of
the Fusion Matrix is the SDMX Information model, which provides an organised,
decoupled model to describe data, where the associated metadata used to
interpret the data is maintained independently from the data itself.
The Fusion Matrix supports all SDMX versions for both
consumption and dissemination, and it has been designed so that it can support
future versions of the standard with little to no impact on the current
application. Additionally the design
lends itself to be easily adapted to adopt other standards and formats as
The Fusion Matrix provides fully SDMX compliant web services
making the data in the Fusion Matrix accessible for consumption by both human
and machines alike.
The Fusion Matrix offers a simple web front end to navigate
to, and view datasets. In addition the Fusion Matrix provides a simple, yet
powerful API to facilitate the rapid construction of bespoke data dissemination
website creation using the Fusion Matrix APIs.
Why Fusion Matrix?
Protecting Investment with SDMX
The Fusion Matrix is fully SDMX compliant. It is built upon the SDMX Source open source framework. The Fusion Matrix has no other bespoke internal model: it is SDMX through and through. For instance, as all the information held in the Fusion Matrix is described in SDMX, it is possible to export the entire contents of the Fusion Matrix as SDMX documents: data; structures; and metadata,. This ensures all of the information held in the Matrix is fully compatible with any other SDMX compliant application.
Adopting SDMX as the Information Model ensures that both you and your client applications are not tied to a particular tool for implementation. For instance by opening your data to data consumers via the SDMX web service API, the only coupling is to the SDMX Web Service.
Thus the business investment in not tightly coupled to particular tools, but rather to the SDMX standard, a widely used and increasing popular standard for Statistical Data and Metadata eXchange.
The Fusion Matrix is a highly metadata aware application. This means all aspects of data collection and dissemination can be controlled through metadata. The ability to connect to the Fusion Registry means that the Fusion Registry can be used as the metadata management application, leaving the Fusion Matrix to focus on data collection and dissemination.
The Fusion Matrix has been designed in such a way that it is possible to expose the data from existing data sources via its APIs in all of its supported formats, without the underlying data source needing to know about the available formats. Using metadata it is even possible for the Fusion Matrix to perform mapping on internal views of the data, exposing it externally conforming to a different dimensionality and coding scheme.
It is possible to use metadata to define which datasets data providers are allowed to supply, and even place restrictions on the allowable content of each dataset for each provider.
The Fusion Matrix is fast, very fast. The database table structures are built on demand, tailored to the definition for each data structure. Tailor-made tables means the table columns are the correct type and size and relevant indexes are built to make data retrieval as quick as possible.
The Fusion Matrix is also fast for data import. Initial imports for a dataset go via the bulk loader, which is capable of loading over 1 million observations a minute, making it blazingly fast to load large datasets.
High Concurrency & Availability
The Fusion Matrix has been designed to cope with large data volumes without the need to allocate vast amounts of memory to the application. Every data load and data query is streamed, this means very little information is held in memory at any given point. With very little information held in memory, the Fusion Matrix can support a high volume of traffic, without impacting performance.
It is also possible to run multiple instances of the Fusion Matrix, and load balance them. This provides not only a more efficient solution when handling large volumes of traffic, but importantly a more robust solution, when one instance is taken out of service, the others take up the load. A load balanced solution leads to greater availability, and lends itself to new deployments with little to no downtime for users.
The Fusion Matrix is built upon a solid open source framework which is well positioned to adapt to future changes to the standard. The decoupled nature of the underlying architecture lends itself to seamlessly plug in support for other data formats, such as the RDF Data Cube Vocabulary which is based on the SDMX Information Model.
HTML User Interface
The Fusion Matrix provides an HTML User Interface (UI) for data discovery and data display. The UI has been built from the Matrix API and is intended to demonstrate the power of a simple API. The UI provides a powerful pivot table, metadata view, data export, and the ability to store queries locally and publish them globally.
The HTML UI is also provides the means to author Reference Metadata. Reference Metadata is additional metadata that can be attached to metadata (such as a Dimension) or data (such as a specific observation). The metadata editor is a What You See Is What You Get (WYSIWYG) editor enabling the simple creation of HTML mark-up without having to edit the underlying HTML.
It should be noted that this HTML User Interface demonstrates what can be achieved using a combination of the Matrix API and SDMX web services. The Fusion Matrix supports rapid construction of dissemination applications by exposing simple, yet information rich APIs.
Manager User Interface
The Matrix Manager UI to administer the Matrix environment, which includes connection to other data sources, integration with external applications, importing data, structure, and metadata.
Command Line Application
The Matrix Command Line Application can be used to import data, structures, and metadata from the command line. It also provides the means to permanently delete data from the Matrix database. The Command Line Application can also be used to start the file sweeper application, which will periodically scan a directory structure for new data files. When new data files are placed in the directory structure, they will be automatically imported in the Fusion Matrix.
Fusion Browser Excel Plugin
The Fusion Browser Excel Plugin for the Fusion Matrix makes it possible to connect to one or more instances of the Fusion Matrix and browse the data available in the instance from within Microsoft Excel.
The Excel Plugin makes use of both the SDMX and Matrix Specific APIs exposed by the Fusion Matrix, and is just another example of how the APIs can be used to build a dissemination application, in this case the application is desktop based (Excel) as opposed to web based.
Fusion Matrix is not just a data dissemination system. It also incorporates a fully functional metadata authoring, maintenance, and repository system. This system can be used on its own or in conjunction with data dissemination. The metadata can relate to any data or structural metadata, even if this is held external to the Fusion Matrix.
The Fusion Matrix metadata repository supports the SDMX Information Model which means it supports metadata that is structured according an SDMX Metadata Structure Definition (MSD). The MSD is held in the Fusion Registry.
The Matrix-UI provides the ability to author reference metadata which can be attached to a Dimension, Concept, Code, Dimension Code (the code in the context of the dimension), Dataflow, Data Structure, or Observation.
The reference metadata is stored as SDMX Metadata Sets and made available to external applications through the SDMX-WS. It is possible to attach reference metadata to datasets and observations even if the underlying dataset is not held in the Fusion Matrix. This means that you can retrofit metadata to data held in external (to the Matrix) databases. The Matrix brokers a data query to both the database and metadata repository and the Matrix will add the metadata links to the relevant data points in the data response
SDMX Metadata Sets authored or derived from external systems can be loaded into the Fusion Matrix metadata repository and so are available for viewing, editing (creating a new version), and access by the data dissemination system.
When data has to be released at a specific point in time, and not a moment before, the Fusion Matrix offers the Embargo feature. The Fusion Matrix embargo solution takes into account both the security concerns of releasing the data too early, or indeed having the data taken before the embargo is lifted, as well as the timeliness of the release. Being able to ensure both that the data are unavailable, whilst maintaining the ability to make it available as soon as the embargo time passes poses an interesting challenge as there are many security risks which have to be mitigated. The Fusion Matrix provides an innovative solution to this problem, which maintains full confidentiality of embargoed observation values, whilst maintaining the ability to expose the data the very second the embargo time has passed. The Fusion Matrix embargo solution supports multiple datasets to be uploaded all with differing embargo times, without the need to run multiple databases and switch between them.
Revisions to Observations
Revisions, also referred to as Vintages, occur where an observation value for a particular time period is revised. A good example is Gross Domestic Product, where an initial released value for GDP will often be revised upwards or downwards in future points in time, as more information becomes available.
The Fusion Matrix will maintain all revised values to both observation and observation attributes, and make these revisions available via the SDMX web service. The Fusion Matrix HTML UI provides the means to view revisions for any observation.
Supports Other Data Formats
Whilst SDMX provides a great model for describing and exchanging data, we appreciate that many systems that provide the data or consume the data are not aware of SDMX. For this reason the Fusion Matrix also supports datasets in XSLX (Excel) and CSV format for both import and export.
As Fusion Matrix is built upon SDMX Source it is also possible to take advantage of the plugin nature of the framework, allowing easy support for new data formats with no modification to the Matrix software.
Mapping Between Data Structures
A less understood, but very powerful feature in SDMX is the ability to define a relationship between two data structures. With a mapping relationship defined, it is possible to convert a dataset conforming to one data structure, to a dataset with a different structure.
The Fusion Matrix can interpret and make use of SDMX Structure Maps, meaning it can expose data externally conforming to a different dimensionality and coding scheme to how it is stored internally. This is a very useful feature if you wish to expose data from existing data sources, which may not conform to the structure and coding scheme required for disseminationn. Fusion Matrix will expose the external structure for data discovery and query, convert the query to the conform to the internal structure and convert the resulting dataset to the external structure
Integration with Existing Data Sources
Whilst the Fusion Matrix provides its own self building database to store datasets, it is also possible to integrate the Fusion Matrix with existing data sources. The Fusion Matrix can connect to multiple data sources, and supports RDBMS databases such as Oracle, SQL Server, and MySQL. The Fusion Matrix will automatically generate a SQL query, based on your definition of how the data is structured. This makes it possible to create database views to service the query. Coupled with the ability to define structure maps, it is possible to expose data from a database in the least intrusive way possible.
If a data source is not directly accessible, for example if it exists outside of your organisation, then the Fusion Matrix is capable of obtaining data from SDMX compliant web services. This means putting a simple web service on a data store will make the data available to the Fusion Matrix. The Fusion Matrix supports brokering data queries to one or more external SDMX web services, and will consolidate the response from each source, and write it out as a single dataset, in a single data format, regardless of the format of data returned from the external datasource.
The ability to connect to multiple data sources across an organisation, or even between organisations, makes the Fusion Matrix the perfect data portal application for providing a single point of access to data across multiple data domains, where the data is retrievable in a consistent format, regardless of data source.
Integration with Fusion Registry
The Fusion Registry is a web application which is used to create, view, and maintain structural metadata. Structural Metadata includes all the metadata used to define what a dataset looks like, how it is coded, who the data providers are, what they can provide data for, how data is categorised, and much more. The Fusion Matrix must be supplied with structural metadata before it can be used to store or disseminate data. Without structural metadata, the Fusion Matrix has no concept of what data exists.
It is not the job of the Fusion Matrix to create or maintain Structural Metadata. This facility is best provided by an SDMX-compliant registry. The Fusion Matrix integrates with the Fusion Registry, a dedicated Structural Metadata repository, which provides an intuitive user interface to enable creation, maintenance, and viewing of Structural Metadata.
When the Fusion Matrix is integrated with the Fusion Registry the structures in the Matrix will be updated in real time, as they are changed in the Fusion Registry.
The Fusion Registry also supports a Data Registration. The registration is made by the data provider and contains the link to a data source in the form of the URL to an SDMX web service, or the URL of a data file. The registration is linked to all the other information required to process the data: the data provider, the data domain, and the data structure. When the Fusion Matrix is integrated with a Fusion Registry it will react to data registrations by indexing the data source, and making the data available to query through its interface. It should be noted that the Fusion Matrix does not replicate the data at the registered data source, it instead acts as a data portal, brokering a single data query to one or more registered data sources.
Information on Fusion Registry is available here
Integration with Fusion Security
Fusion Security is a standalone web application providing user management and authentication services. Fusion Security makes it possible to run a secure registry without exposing any user management functions. Fusion Security obtains organisation information from the Fusion Registry, allowing a single user to link to multiple organisations if required.
Fusion Security provides an additional layer of security by enabling automatic account lockdown on repeated login failure attempts and lock down of accounts to specific IPs or IP ranges.
Fusion Security can be configured to connect to Fusion Audit to capture all security events such as account creation, deletion, modification.
Integration with Fusion Audit
Fusion Matrix can integrate with Fusion Audit, which is another web application purpose built for capturing audit information related to SDMX queries and submissions.
SDMX Web Services
The Fusion Matrix supports the SDMX web services specification for data, structure, and metadata retrieval. Each version of SDMX is supported, from SDMX-EDI up to SDMX-2.1 and includes the new SDMX-JSON format to support web developers build data visualisations.
Providing an ISO compliant web service enables your data consumers to access your data using any existing software, or to easily write their own bespoke software using any SDMX libraries, including the SDMX Source Java libraries developed by Metadata Technology.
Custom Web Services supporting Data Discovery
Whilst SDMX is a powerful format for the exchange of metadata and and data. However, the learning curve for application developers can be steep as there is a lot of domain or SDMX-specific knowledge which can be difficult for web developers to assimilate. In addition the SDMX web services for query return structural metadata (used to drive data discovery, query, and visualisation) in a granular way, resulting in many interactions with the server. This can be complex for the development of the client side logic. In order to facilitate the construction of a data discovery application, the Fusion Matrix provides a few additional web service APIs, which repackage SDMX information making it specific to data discovery.
The Developers API includes an API into the Matrix Constraints Engine which is a complex algorithm used to prevent a user from constructing a query which results in no data being returned.