Leveraging the Pervasive Integration Engine API: Basic On-Demand Mapper

Contributor: Julie Hunt, Julie Hunt Consulting

Overview of Approaches to Creating Transformation Maps

Developers working with the Pervasive Data Integration Platform design tools are well-acquainted with the broad range of capabilities that they provide for any data integration challenges. Besides saving time by using Pervasive design tools, developers don't have to "reinvent the wheel" every time they do a new project, due to the many repeatable functions and features in the products. The Pervasive Data Integration Platform maintains project components in one place, documented and managed for control and ease of use.

But sometimes developers face data integration challenges that may have to be solved without the design tools interface. While the majority of data transformations are accomplished by executing transformation maps created in the Map Designer, there are situations in which the only method to accomplish a specific transformation is to write code outside of the design platform. The code can then leverage the powerful resources of the Pervasive Integration Engine API to perform transformation tasks.

In this article a real-world example of leveraging the Integration Engine API, in conjunction with Java code, will be detailed. The example covers the creation of a basic on-demand mapper that is set up in conjunction with any UI and is used to build Pervasive maps (map.xml, tf.xml). Since this mapper is intended for business users as the end users, the best UI to develop is a wizard-like interface. The maps created are simple, without complex transformation parameters. The engine API is integral to providing source and target connections and metadata acquisition. This article presents what is needed in Java code to complete these tasks.

The Pervasive Integration Engine

The Pervasive Integration Engine is a dedicated execution engine with no user interface components. This allows it to be invoked to perform automatic runtime data transformations quickly and easily. The Integration Engine API supports seamless integration with other programs and extensive control of transformation processes by running maps and processes created in Map Designer and Process Designer.

The API provides for launching transformation processing directly from any "calling" program (Java, .NET or other environments). In addition to execution of pre-designed transformations, it may be used to automatically create, map and run default transformations. The functionality provided by the API allows for tight control of connection parameters and data manipulation, including error handling, on-the-fly filtering and transformation overrides.

Engine SDK - Java and COM

Documentation is provided for the Engine SDK with the installation of Pervasive Data Integrator™. The SDK outlines development in either Java or COM environments. For most data integration projects, the Engine API is leveraged for runtime overrides of parameters in maps and processes that have been created with the Pervasive design tools.

Prerequisites for Working with the Basic On-Demand Mapper Sample in this Article

Skills:

• Strong knowledge of the Pervasive Data Integration Platform
  • Map Designer
  • Maps and their properties as created by the Map Designer (including map.xml and tf.xml created for each map)
  • Runtime engine and its API
  • Runtime deployment options for maps after they have been created
• Java development experience
  
• UI development and usage experience
• Knowledge of source and target data formats, structures and fields

Products:

• Pervasive Data Integrator installed (access to supporting documentation and transformation objects)
• Pervasive runtime engine with a configuration appropriate for the solution platform

A Basic On-Demand Mapper

The most obvious use for a basic on-demand mapper that creates maps without the Pervasive Data Integration design tools is for applications that enable business users to perform mapping activities through a wizard-like UI. This could be for migrating data from one application to another, or to maintain synchronization between two applications. This sort of mapper is useful for applications that need to create on-the-fly maps on a frequent basis where the business user knows the source and target data well enough to be able to easily perform the mapping.

This basic on-demand mapper is best suited for situations that are based on tightly defined data mapping without extensive transformation needs. This mapper is best used as a component of an overall solution where the component performs very specific mapping tasks. The small footprint of the Integration Engine makes it quite appealing as an embedded component in a solution or application.

The mapper described in this article is based on a combination of Java code and the functionality available with the Integration Engine API that allows developers to build out source and target connections, metadata, map and transformation tasks. There are no constraints on the type of UI that a developer could use, as long as it works well for the solution. If the UI is mainly for business users, then it should be simple and easily used.

Data mapping and connectivity could be bi-directional or uni-directional, depending on project requirements. There are no limits on the number of fields that can be acquired from connections to a source and target, except for the usual constraints (i.e., memory). But if the purpose is to provide a business user a wizard-like UI for mapping, then it makes sense to limit the number of fields handled by the on-demand mapper UI.

Once a map has been created and saved, the map is saved to disk and may then be run immediately or run at another time via any scheduler or automation options available to a developer.

Sample code has been provided as a download at the end of the article to illustrate one approach to a basic on-demand mapper that leverages the Pervasive Integration Engine API.

COMPONENTS OF THE SAMPLE SOLUTION

Connecting to Sources and Targets
When developing a UI for an on-demand mapper, the developer will need to understand all parameters that are required by the Integration Engine to connect to source and target data. Some parameters may be stored in a file that is accessed by the mapper code; some or all may be entered dynamically through the UI. Parameters could be tied to a specific login provided by the business user. Again, the developer has many options for providing information that is required to make source and target connections. The sample code shows examples of how parameters are used with the engine API.

While the only call to the engine API is to instantiate "DataJunction.ec," there is extensive coding to be done in Java to work with source and target data. To connect to sources and targets through the Integration Engine, the developer will write a Java wrapper around each adapter to be used, implementing the engine SDK. The wrapper code is detailed but not difficult to write.

Developers should expect complexity when coding wrappers around the adapters, so a careful read-through of the engine API documentation is highly recommended. The sample code download provides more detail on writing wrappers around Pervasive Integration Engine adapters.

Acquiring Metadata from Sources and Targets
Once connectivity has been established to source and target, the Java code must acquire source and target metadata. The Pervasive Integration Engine does not make the metadata directly available to the Java code. The metadata is held in memory, so code is written to read the metadata from memory and save it to disk. The saved metadata is in the form of a partial map or map.xml object, containing source and target connection information as well as the metadata for source and target data. The Java code then opens the partially populated map.xml to parse out the metadata. The extracted metadata is used to populate metadata for source and target in the UI, for the business user.

The sample Java code walks the developer through the methods used to perform the metadata acquisition steps.

Mapping in the UI
The business user must have a good understanding of the source and target data fields to properly perform the mapping in the UI. Additionally, the mapping in the UI should be simple and intuitive.

Once mapping is completed, the business user will click on a "Save" command to enact Java code to save the now fully-completed map.xml to disk. The sample code shows the new map.xml overwriting the original partially completed map.xml that was saved to disk by the Java code after acquiring it from memory. The Save command also invokes Java code to create the transformation object, "tf.xml," in order to run the transformation map with the integration engine.

The newly created map may be viewed in the Pervasive Map Designer to assist with the development phase, or may be viewed as an xml document if the developer is well familiar with the valid structure of map.xml and tf.xml.

Running Completed Maps
Completed maps comprised of valid map.xml and tf.xml can be run any number of ways. Code can be written to run the map immediately. Or the saved map objects can be run on a scheduled or automated basis, accomplished by any of the ways a developer would like to run an automated task. No matter what method is selected, running the maps created by the on-demand mapper will require the Integration Engine.

UML diagram
A UML diagram can be downloaded here. The UML diagram describes the classes used in the sample Java code for the basic on-demand mapper. The UML diagram presents the objects and their relationships to help developers understand and adapt the sample Java code to their projects.

Benefits of Creating a Basic On-Demand Mapper with Java and the Engine API
The most likely scenarios that benefit from the on-demand mapper include:

• Embedding in another application where a small footprint is important
• Map creation by business users (typically, high frequency scenarios)
• Migrating data from one application to another
• Synchronizing data between applications
• Need for customized UI, particularly for business users
• Need to reduce developer or IT involvement in some map creation and running activities

The over-arching benefit of combining Java code with the engine API is the opening of opportunities for runtime overrides, which provide greater control and flexibility for transformations and processes.

Limitations of This Approach
Using a mapper to create maps without the Pervasive Data Integration design tools has limitations:

• Very limited functionality compared to the breadth and depth of design possibilities available in the Map Designer and Process Designer
• Should always keep in mind that this is a work-around (but a very useful one) that works through the engine API
• This mapper does not produce the more complex maps that can be created with the sophisticated design tools (Map Designer and Process Designer)
• Limited access to metadata due to inherent function of the engine API
• Can't "see" database schemas or other metadata as can be done in the Map Designer
• Limited by how much a business user understands about mapping data from source to target
• This approach will only satisfy a narrow need and is not a universal approach to creating maps