Create a Pipeline

Create a Pipeline

How pipelines work

Pipelines are the fundamental building blocks of your Hop projects.

Pipelines perform the heavy lifting: they read data from a variety of sources, perform a number of operations (combine, clean, enrich, transform etc) and write the data out to some target platform. A pipeline executes all of these operations in a predefined order and in parallel.

In the image below, a very simple pipeline reads data from a database, adds a message to the data and sends out an email. All of these operations are executed in a predefined order (read from the database, add the message, send the mail) and in parallel. The pipelines executes these transforms, Imagine our database table or query contains thousands of rows. The pipeline will start reading results from the query, pass those on to the ‘Add message’ transform. Once the message has been added, we’ll send a mail from the Mail transform. All of these will operate in parallel, so the Mail transform will already be sending mails while the table input is still reading records from the table or query.

Concepts

Pipelines consist of transforms connected by hops. In the mail example ‘Table input’, ‘Add message’ and ‘Mail’ are all transforms.

transforms are the basic operations in your pipeline. A pipeline typically consists of a lot of transforms that are chained together by hops. Transforms are granular, in the sense that each transform is designed and optimized to perform one and only one task. Although one transform by itself may not offer spectacular functionality, the combination of all transforms in a pipeline is makes your pipelines powerful.
hops link transforms together. When a transform finishes processing the data set it received, that data set is passed to the next transform through a hop. Hops are uni-directional (data can’t flow backwards). Hops only buffer and pass data around, the hop itself is transform-agnostic, it doesn’t know anything about the transforms it passes data from or to. Some transforms can read from or write to other transforms conditionally to or from a number of other transforms, but this a transform-specific configuration. The hop is unaware of it. Hops can be disabled by clicking on them, or through right-click → disable.

Create a pipeline

Create a new pipeline through the work item dialog. You’ll be presented with the dialog shown below.

When you are finished with your pipeline, save it. This can be done via the File menu, the icons or using CTLR s or Command s. For new pipelines a file browser is displayed to navigate towards the location you want to store the file.

Add Transform to your pipelines

Click anywhere in the pipeline canvas, the area where you’ll see the image below.

Upon clicking, you’ll be presented with the dialog shown below. The search box at the top of this dialog works for transform, name, tags (TODO) etc. Once you’ve found the transform you’re looking for, click on it to add it to your pipeline. An alternative to clicking is arrow key navigation + enter. Repeat this step now or whenever you want to add more transforms to your pipeline. Once you’ve added a transform to your pipeline, you can drag to reposition it.

Check the list of transforms to add to your pipeline for more details.

Add a ‘Generate Rows’ and a ‘Add Sequence’ transform, and your pipeline should like the one below.

The transform object can be configured through a single click on the object. The menu displayed below will be shown based on your transform object.

Action	Description
Detach transform	Detach the transform from the pipeline
Edit the transform	Edit the transform’s metadata
Copy transform to clipboard	Copies selected items to clipboard.
Create hop	Creates a new hop between two transforms.
Set the number of transforms	Starts several instances of a transform in parallel.
Preview output	Allows you to preview the results of the transform.
Debug output
Show the fields entering this transform	Shows metadata, like the field name and type for fields coming into the transform.
Show the fields exiting this transform	Shows metadata, like the field name and type for fields coming out of the transform.
Distribute rows	In case of more than one hop the data is distributed between the next transforms.
Copy rows	In case of more than one hop the daya is copied to the next transforms.
Specify transform partitioning	Specify how rows of data need to be grouped into partitions allowing parallel execution where similar rows need to end up on the same transform copy
Edit transform description	Add a description to the transform.
Transform error handling	Set the error handling for the transform, not available for all transforms.
Delete this transform	Delete selected transform from the canvas.
Edit Custom Logging	Edit the custom log settings for this transform. This will change the log level used for this transform.
Clear Custom Logging	Clear custom log settings. This will clear the log level used for this transform.
Sniff output	Take a look at 50 rows coming out of this transform. This will show a real-time table with a continuous output of the selected transform.
Set input data set	Defines which data to use instead of the active input transform, applies to the selected unit test
Clear input data set	Remove a defined data set from the selected unit test
Set golden data set	The input to this transform is taken and compared to the golden data set you are selecting.\nThe transform itself is not executed during testing
Clear golden data set	Remove a defined input data set from this transform unit test
Create data set	Create an empty dataset with the output fields of this transform
Write rows to data set	Run the current pipeline and write the data to a data set
Remove from test	When this unit test is run, do not include this transform
Include in test	Run the current pipeline and write the data to a data set
Bypass in tess	When this unit test is run, bypass this transform (replace with a dummy)
Remove bypass in test	Do not bypass this transform in the current pipeline during testing

Add a Hop between transforms

There are a number of ways to create a hop:

shift-drag: while holding down the shift key on your keyboard. Click on a transform, while holding down your primary mouse button, drag to the second transform. Release the primary mouse button and the shift key.
scroll-drag: scroll-click on a transform , while holding down your mouse’s scroll button, drag to the second transform. Release the scroll button.
click on a transform in your pipeline to open the ‘click anywhere’ dialog. Click the ‘Create hop’ image::getting-started/icons/HOP.svg[Create hop, 25px, align=”bottom”] button and select the transform you want to create the hop to.

Some transforms result in different types of hops.

Hop	Description
Result is TRUE	Specifies that the transform will be executed only when the result from the previous transform is true
Result is FALSE	pecifies that the transform will be executed only when the result from the previous transform is false
Main output of transform	The default hop between two transforms

Pipeline properties

Pipeline properties are a collection of properties that describe the pipeline and configure its behavior.

The properties dialog can be opened by double clicking on the pipeline canvas.

Following properties can be configured:

Pipeline
Parameters
Monitoring

The Pipeline tab allows you to specify general properties about the pipeline including:

Property	Description
Pipeline name	The name of the pipeline
Synchronize name with filename	If option is enabled the filename and pipeline name are synchronized.
Pipeline filename	The filename of the pipeline
Description	Short description of the pipeline
Extended description	Long extended description of the pipeline
Status	Draft or production status
Version	Description of the version
Created by	Displays the original creator of the pipeline
Created at	Displays the date and time when the pipeline was created.
Last modified by	Displays the last user that modified the pipeline
Last modified at	Displays the date and time when the pipeline was last modified.

The parameters tab allows you to specify parameters specific for the pipeline. Parameters are defined by a name, a default value and a description.

The monitoring tab allows you to specify the monitoring of the pipeline.

The options to set in this tab are:

Property	Description	Type
Enable transform performance monitoring	Enable performance monitoring for the transforms in this pipeline	boolean
Transform performance measurement interval (ms)	The interval (milliseconds) to monitor the performance for the transforms in this pipeline	integer
Maximum number of snapshots in memory	the number of performance monitoring snapshots to keep in memory for the transforms in this pipeline	integer