Overview of HCL Link_a web based ETL & Integration tool

Abstract –

Today, data plays an important role for any organization, every organization processes a large volume of data which can be in different formats. They may have many applications not necessarily supporting the same data format to process the data for their day-to-day activity. To solve this problem, they would need a translator/connector or a middleware to exchange the data within internal/external applications/systems and here we have a powerful ETL & integration tool HCL Link. HCL link is also used by other HCLSoftware offerings such as UNICA, Commerce, Volt etc. to integrate with each other or the external systems.

The key points that can encourage the user to use HCL Link –

• A GUI tool – very simple to use
• HCL LINK Embedded Edition
• REST based Custom connectors
• Rich set of adapter’s support
• Service Builder support for REST-based endpoints

•  A GUI tool – very simple to use –

HCL Link – A robust platform that accelerates integration of the many applications and diverse data across organizations on-prem, cloud or hybrid environments​. It includes the Schema Designer, Map Designer, Flow Designer, Core, Runtime and Adapters. It’s a powerful web based ETL & integration tool and user friendly. It facilitates the simple way of uploading the files, creating connections/actions by using the rich set of adapters/connectors, creating maps for data transformation/transfer for the actions and orchestrate the maps and other nodes in the flow to execute together to get the desired goal.

The home UI page looks as below. It provides you the two banners i.e., Design and Deploy. Design can show the list of projects, create, and import projects and deploy banner can show the options to create the servers, packages, server groups, configuration variables and deploy packages.

Design Banner

• Under Design, click + icon to create a new project, enter the project name and click Ok. It will then show below screen

Quick Links – It provides you the links to quickly access Files, Connections/Actions, Schemas, Maps and Flows. Also alternatively, you have the separate tabs at the bottom of the page to go through these options.

Link design options for creating/modifying the project–

Files – Use this option to upload the input files so that you can use in in the map/flow. You can also download the files on your local, specially required if you want to see the output on your local system.

Connections – A connection identifies what system / technology is being connected to – e.g., File, JDBC, FTP, REST etc.​

Actions – An action identifies what specifically is being done using that connection – e.g. Inserting rows in database table, update the Contact objects in any third-party systems/applications, invoke a specific REST API, transfer a file to the third-party system.​

Schema – A schema describes the structure of the data as returned by or sent to a connector/adapter​. You can use the schema importer to create a schema from CSV, Copybook, XSD and JSON template etc.

Map – A map provides the logic that produces the desired output results​. For example, if you would need to map the XML data to CSV, a map can be used to do this.

Flow – To provide a way to orchestrate multiple nodes together to achieve the goal. It can include one or more nodes.

Deploy Banner

Link Deploy options –

Server Groups – We can create the server groups such as Prod, Dev and Test and the server can be associated with the server group.

Servers – We can create a server that contains the connection string such as the platform details Windows or Linux, user credentials, and server URL or IP/FQDN and the server group.

Configuration Variables – A configuration variable defines an alias that resolves at run time to a value that is specific to a deployment environment (Server Group). By associating the values of a configuration variable with different server groups, you can run the same map or flow in different environments, servers, and platforms without changing the map or flow.

Packages – We can create a package by selecting the project to package. We can then add the maps, flows and files in that package.

Deploy – You can select the package and the server’s name to deploy the package on the selected server.

Build – You can build the package before you deploy it on the server to see if there is any compile time error.

HCL Link Components (Once link installed, you can check for below components) –

HIP Client – It is responsible to render the UI page in browser

HIP Server (Design-Time) – It facilitates the users to create schema, map, and flow.

HIP Rest (Runtime) – It facilitates to run the map & flow to meet the requirement.

Mongo – It’s used to store the artifacts that you create by using design time (HIP-Server)

Redis – HIP-Rest/runtime uses REDIS to execute the request

• HCL LINK Embedded Edition

This is what we provide to other HCLSoftware teams to embed within their applications (e.g., Unica, Volt etc.), which includes HCL Link and Embeddability Extensions ​

Especially, UNICA applications i.e., Journey & Campaign combine HCL Link EE with their appropriate application-specific connectors (e.g., for Unica Link – Email connectors, AdTech, CRM, SMS, and the database connector) as part of their product packaging

HCL Link Embeddability Extensions (EmbedX) – An extension built on top of HCL Link to enable embedded usage by other HCLSoftware offerings, such as Unica, Volt, and Commerce etc. It enables partners and customers to easily create and use their own connectors​

The connectors consumed by Unica-Link Integration is depicted below–

Different hops involved to run Unica-Link request on HIP-REST(Runtime): –

• REST based Custom connectors

HCL Link can be easily used by the customer to create the connectors as per their need. This is what the custom connector is. For example, Unica customers can leverage it to create their own connector. For example – if a Unica customer if gets a requirement to connect to the new third party system for data processing, let’s say the new marketing application, they can then leverage link to create such connector. They would need to have the requirements for that custom connector and then to verify that appropriate REST APIs are provided by the third-party application to be able to meet those requirements.

Once they are ready with the requirement, they can create a connector project, define the REST endpoints and the connector descriptor file to define the connection & action properties.

Artifacts required to create a custom connector package which can be installed on the server–

1. Create a connect project file which contains maps/flows
2. A service definition file/configuration package where you define the REST endpoints
3. A connector descriptor file which defines the connection & action properties

Use packager command to create the connector package by packaging above artifacts. It can then be deployed on link hip-rest component

To package the connector into an output directory called ‘\output\test’, invoke the packager tool with -p (package) and -ep (export package) options:

C:\packager> packager \connectors\test \output\test -ep Test – p -h https://localhost:8443/ -u admin -pw ****

Above steps will create a connector package which can be deployed on the server and then installed so that the customer can start integrating Unica applications with the new connector for marketing through the new channel.

• Rich set of adapter’s support

HCL link supports a rich set of adapters to connect to different applications/systems. It makes it robust which can meet the varying customer requirements when they intend to integrate with many applications using the different communication protocols.

• Service Builder support for REST-based endpoints –

HCL Link includes Service Builder to create service definition file for the REST endpoints that will be used to connect to the other systems or applications. This is important for creating the UNICA-Link REST based connectors.

• Go to Link home page and click the service builder link to create a server definition and endpoints

• Create a new service by clicking on the ‘Create Service’.

• Provide the details such as service name, authentication type and parameters –

• You can go to the last stepper to save the service
• In the service, click Create Endpoint

• Enter the endpoint name, method type and other parameters required for the request/response payload and save.

• You can click Call API button under request tab to check if endpoint Is working ok.

These endpoints can be used in the map and flow.

Link basic concepts explaining through an example for the new user

The scenario is to convert the CSV input file to JSON. The CSV input file structure, let’s assume is as below –

Employee Name,Designation

• Go to the link UI and select + icon under Design banner to create a new project. You can then see many tabs at the bottom of the page which are explained below-

Files tab –

So, the step 1 is to upload the input file –

• Click the Files tab then + icon to upload a new file –

Connections tab –

• Click the Connections tab

• Click + icon to create a new connection. It will show you the list of adapters available to connect to the external or internal systems. Choosing File adapter as my source/target is file. Click next to provide the connection name and save it.

Schemas tab-

Schema helps to use the importer to easily create the data structure from the template to store the file data.

We would create CSV followed by JSON.

• Click on Schemas and then Import to Import from the file and select CSV from drop down box

• Select the input file name to create the structure and click Next to enter the schema name and select import.

Now, import JSON schema –

• Click XSD/JSON and browse for the JSON file and click Ok.

Connections tab for creating actions –

Creating a source action –

• Hover the mouse on the connection name and click 3 dots to get below

• Click New Action option to create a new action

• Select Source and click Next and browse for the input file
• Select the schema and schema type to match the input file, provide the name and save

Now creating the target action –

• Hover the mouse again on the connection name and click 3 dots

• Click New Action to create a target action and click Next

• Browse for the JSON template file and click Next. Enter the target action name and save.

Now, you can see two actions created i.e. the source action to read from the CSV file and the target action to write into JSON format.

Maps tab –

• Click on Maps tab and then + icon to create a new map. Enter the map name and click Ok-

• Now add the actions to the map in form of the input and output cards –

• After adding the actions – the map will look like below

• Create a functional map to map CSV to JSON

• Click build icon to build the map. Once build is successful, click the run button to run the map

• The input and the output result data will be as follows

Flows tab –

• Click on the flows tab and then + icon to create a new flow. Enter the name and click Ok

• Creating a flow having one decision node and a map node. The decision node will execute true terminal if Flag_to_Run_the_Map flow valuable is set to True else the false terminal.
• If Flag_to_Run_the_Map is True will execute the map node which will further result in JSON data. If it’s false, then map node won’t get executed.

• Executing the flow by setting Flag_to_Run_the_Map to True

• Executing the flow by setting Flag_to_Run_the_Map to False, would notice that the green tick mark doesn’t appear on the map node.