Hop Concepts

Core Concepts

Before we dive deeper, let’s take a minute to familiarize ourselves with the Hop lingo.

Metadata is by far the most important concept in all of Hop. Every item we’ll cover below is defined as metadata. All interactions between Hop and other components in your data architecture are done through metadata. Metadata is at the core of everything in Hop.

  • Pipelines are a collections of transforms, connected by hops. All transforms in a pipeline run in parallel.

  • Workflows are a collection of actions, connected by hops. All actions in a workflow run sequentially by default.

  • Projects are logical collections of hop code and configuration. Environments contain the environment-specific (e.g. dev, uat, prd) metadata.

Item types

Action

An Action is one operation performed in a Workflow. Actions are executed sequentially by default, with parallel execution as a configuration option. An Action returns a true or false exit code, which can be used (or ignored) in the Workflow’s execution.

Hop

A Hop links Actions in a Workflow or Transforms in a Pipeline. In Workflows, Hops operate based on the exit status of previous Actions, Hops in Pipelines pass data between Transforms.

Pipeline

Pipelines are the actual data workers. Operations in a Pipeline read, modify, enrich, clean and write data. Orchestration of Pipelines is done through othere Pipelines and/or Workflows.

Pipeline

Transform

A Transform is a unit of work performed in a Pipeline. Typical Transform operations are reading data from files, databases, performing lookups or joins, enriching, cleaning data and more. All transforms in a Pipeline are executed in parallel. Transforms process data and move batches of processed data on Hops for processing by subsequent Actions.

Workflow

A Workflow is a sequence of operations that are performed sequentially by default (with optional parallel execution). Workflows usually do not operate on the data directly, but perform orchestration tasks. Typical tasks in a Workflow consist of retrieving and archiving data, sending emails, error handling etc. )

Workflow

Projects and Environments

Project

Hop Projects are a conceptual grouping of configurations, variables, metadata objects and workflows and pipelines. Projects can inherit metadata from parent projects. A project contains one or more environments where the actual configuration is defined.
Example: a ‘Sales’ project contains a ‘customers’ database connection and a number of workflows and pipelines. The runtime configurations, database connection properties etc are defined in the ‘dev’, ‘uat’ and ‘prd’ environments.

Environment

Hop Environments are instances of projects that hold the actual runtime configurations and other metadata objects for a project.
Example: the ‘dev’ environment for the ‘Sales’ project specifies to read from host ‘10.0.0.1’ for the ‘customers’ database connection

Environment Examples