Overview

The Fusion Transformer Pro is the professional version of the free-to-use Fusion Transformer product.

It provides the capabilities to Validate, Transform, Map, Visualise, and Publish data in both SDMX and non-SDMX formats. By supporting non-SDMX formats such as CSV and Excel, it is possible to easily convert and publish datasets to SDMX formats. It is equally possible to convert SDMX datasets into native Excel and CSV formats.

As well as transformation between data formats, Fusion Transformer Pro offers a mapping service, which maps dimensions and coding schemes of an input dataset, to an output dataset. This allows a dataset to be uploaded, in any supported format, conforming to Data Structure Definition, and then transformed to conform to a mapped Data Structure Definition. After the mapping is complete, the new dataset can be exported in any supported format.

The Fusion Transformer Pro provides a data store, allowing uploaded datasets to be stored locally and made available in any supported format via the Fusion Transformer Pro web service.

The Fusion Transformer Pro is a web application offering a multi-user interface via a web browser. In addition it provides a command line application which supports direct integration into your own software products and services.

The Fusion Transformer Pro is extremely fast, and does not have high CPU or memory requirements, even when processing extremely large datasets. It has no external dependencies except for Java, and consequently it can run on a local desktop computer for single user use, or it can be hosted on a server to support multi-user access.

Professional Vs Free

Feature

Free

Professional

Transform SDMX Datasets

Transform CSV/Excel Datasets

-

Transform Structure Files

Uses Fusion Registry

Full Dataset Validation

-

Command Line Application

Web Interface

-

Multi-User Support

-

Data Visualisation

-

Data Store + Web Services

-

Fusion Registry Integration

-

Publish to Fusion Matrix

-

Fusion Audit Integration

-

Fusion Security Integration

-


Features Overview

Running as a Web Application

The Fusion Transformer Pro is run on a web server, by either deploying to a Tomcat server, or by launching the executable service. The application provides a web GUI which can be accessed via a web browser. The GUI offers the ability to load a dataset, view dataset information, including a validation report, view the dataset, export the dataset in any supported format, and publish the data to an external web service in any of the supported formats.

The web application can be connected to a Fusion Registry from which the Fusion Transformer Pro will obtain all of its structural metadata to drive its data validation, mapping, and transformation logic. When the contents of the Fusion Registry change, the Fusion Transformer Pro will update its definitions, allowing real time modification of structural metadata.

In addition the web application provides a service to which the command line can connect when running a validation or transformation request. This allows a single server to run validation and transformation logic for multiple clients in a distributed environment.

Command Line Client

The Fusion Transformer Pro offers a command line client application which connects to the web application via its web services.

The command line client brokers the request directly to your Fusion Transformer Pro web application which will process the request. Communication is automatically compressed to increase network request/response times.

More Information

Simple Set Up

As the Fusion Transformer Pro has no dependencies, there is no complicated installation process. The web application can be started by simply running a single batch file, at which point it will launch its own embedded web server, or it can be manually deployed to your own copy of Tomcat (Web Application Server).

Fusion Registry Integration

Unlike the Fusion Transformer standard edition, the Fusion Transformer Pro is capable of linking to a Fusion Registry to obtain its structural metadata. For performance reasons it will create a local in-memory cache of the contents, and it will keep in-sync with the Fusion Registry by monitoring the Fusion Registry RSS feed for changes.

This enables you to maintain your structural metadata, such as your Data Structure Definitions, Codelists, Concepts, Structure Maps etc, in a dedicated structure maintenance tool, whilst maintaining a validation and transformation service whose rules are driven directly from these structure definitions.

Support for CSV and XLSX

As well as support for all SDMX versions (standard feature in Fusion Transformer), the Fusion Transformer Pro also supports datasets described in both CSV and native XSLX formats. This is true for both import and export.

More Information

Extending Support for Data Formats

As the Fusion Transformer Pro is built using SdmxSource, our open source Java framework, it is an easy task to write new plugins to support additional data formats for reading and writing.

If you wish to support additional specific non-SDMX formats please contact us for more information on how to achieve this.

Auto Detect Data Formats and Definitions

The Fusion Transformer Pro will automatically detect data format and SDMX version. It is also capable of automatically detecting the linked Data Structure Definition, Dataflow, or Provision Agreement as defined in the dataset header of SDMX datasets.

In some datasets this information may be either not present (e.g. if reading from a non-SDMX format) or even incorrect. If processing a version 2.0 dataset the DSD version information is not available. In these cases it is possible to define explicitly the DSD/Flow/Provision to be linked to the dataset. Where this is done the exported dataset will be populated with this information.

Roll Up Datasets

The Fusion Transformer Pro will roll up a dataset by detecting if the same series is defined multiple times in various locations in the dataset. It will also detect if there are duplicate observations, and try to merge the observations if there is no conflict. This is particularly useful if reading SDMX-EDI data, which, with specific exceptions, separates the observation value from the observation attributes. For SDMX-EDI a typical conversion tool would repeat the key of the observation at the beginning of the file, with the reported attributes at the end of the file. With the Fusion Transformer Pro the rolled up input dataset will be consolidated before being exported.

Full Validation, with Understandable, Categorised Errors

The Fusion Transformer Pro provides a number of validators, which validate beyond that which a XML Schema is capable of doing.

The validators are fast, far faster than running a schema validation. Each error is categorised, such as syntax error, or duplicate observations, missing mandatory attributes. Above all, the error messages are meaningful, each error gives a description of the series or observation at fault, and a reason why it is at fault.

If validating from the web browser, then the validation report is presented to the user in the browser, with the ability to view all the errors in each category. If running from the command line application the report is output as a SDMX Metadata Set, this allows for further processing by external applications.

More Information

Data Transformation

The Fusion Transformer Pro can read from CSV, Excel, SDMX (multiple versions & formats) and output the data file to CSV, Excel and SDMX. The allows for non-SDMX datasets to be converted into SDMX, or SDMX to be converted to non-SDMX, it is also possible to convert between SDMX formats.

Data Transformation is a separate process from data validation, so it is possible to transform a dataset which is not strictly valid, if required.

More Information

Mapping

It is quite possible that your datasets, which may or may not be in SDMX format, may not even conform to the target SDMX Data Structure Definition. It may have fewer dimensions, and the dimensions may be coded using different coding schemes. In this case the requirement is to define a mapping between the dimensions and coding schemes of the incoming dataset, to the dimensions and coding schemes of the target dataset.

The SDMX solution is to provide a structure called a Structure Map which is used to describe this mapping. The Fusion Registry solution is to allow a Structure Map to be authored and maintained in Excel which can then be imported/validated by the registry. The Structure Map can be exported to Excel for viewing and amendment.

The Fusion Transformer Pro solution is to make use of this Structure Map to convert the structure and codes used in the incoming dataset to conform to that of the target dataset.

By combining mapping with transformation capabilities, it is possible to use the Fusion Transformer Pro to perform a number of streamed processes in a linear sequence without intervening files: read CSV format, change the structure to the target Data Structure Definition, and output the data file as an SDMX dataset.

More Information

Data Visualisation

The Web Interface of the Fusion Transformer Pro provides a simple HTML table of your dataset, allowing you to be able to view data for verification purposes. For SDMX datasets, exporting as Excel provides the means to view/modify the full dataset in a familiar tool.

More Information

Local Data Store

The Fusion Transformer Pro is able to store uploaded files on the local file system and make these files available for other users to view and export. In addition it also provides a web service for external applications to obtain stored data files in whichever format they choose.

More Information

Data Publication

The Fusion Transformer Pro can be used to publish datasets to a Fusion Matrix service. It will monitor the publication progress, as reported by the Fusion Matrix, and update the user interface in real time.

More Information

Fusion Audit Integration

The Fusion Transformer Pro can be configured to publish audit events to Fusion Audit. These events can be used to view which datasets users are validating, transforming, and which datasets have been published.