Advanced project and environment management

Configuration files

Global Project Configuration

The 2 main things that define a project are its name and its home folder. Projects and environments as such are defined in the central Hop configuration file hop-config.json. By default this file lives in the config/ folder of your Hop client distribution. You can change that folder by setting the HOP_CONFIG_FOLDER environment variable on your system.

In hop-config.json, you’ll find a "projectsConfig" section. By default it contains the following:

{
  "projectsConfig" : {
    "enabled" : true,
    "projectMandatory" : true,
    "environmentMandatory" : false,
    "defaultProject" : "default",
    "defaultEnvironment" : null,
    "standardParentProject" : "default",
    "standardProjectsFolder" : null,
    "projectConfigurations" : [ {
      "projectName" : "default",
      "projectHome" : "config/projects/default",
      "configFilename" : "project-config.json"
    }, {
      "projectName" : "samples",
      "projectHome" : "config/projects/samples",
      "configFilename" : "project-config.json"
    } ],
    "lifecycleEnvironments" : [ ],
    "projectLifecycles" : [ ]
  }
}

As you can see the standard Hop client distribution defines 2 projects: default and samples.

Project Configuration

Every project has extra metadata and settings stored in a project configuration file called `project-config.json` . For the samples project this would be `config/projects/samples/project-config.json` . Let’s take a look at it:

{
  "metadataBaseFolder" : "${PROJECT_HOME}/metadata",
  "unitTestsBasePath" : "${PROJECT_HOME}",
  "dataSetsCsvFolder" : "${PROJECT_HOME}/datasets",
  "enforcingExecutionInHome" : true,
  "parentProjectName" : "default",
  "config" : {
    "variables" : [ ]
  }
}

Variables

You can define variables on a project level as well. This makes it handy to reference things like input and output folders which are not sensitive to being checked into version control.

Parent projects

As you can see from the project configuration file (parentProjectName), a project can have a parent from which it will inherit all the metadata objects as well as all the variables that are defined in it.

Environment configuration

Hop enviroments and their home folders are stored in the hop configuration file ‘hop-config.json’. That file lives by default in the config folder of the Hop installation. System property HOP_CONFIG_FOLDER can also be used to point to a different folder

{
 "environmentConfig" : {
  "enabled" : true,
  "openingLastEnvironmentAtStartup" : true,
  "environmentConfigFilename" : "environment.json",
  "environmentFolders" : {
   "Project 1 - DEV" : "/projects/one/dev/",
   "Project 1 - UAT" : "/projects/one/uat/",
   "Project 1 - PRD" : "/projects/one/prd/",
   "Project 2 - DEV" : "/projects/two/dev/",
   "Project 2 - UAT" : "/projects/two/uat/",
   "Project 2 - PRD" : "/projects/two/prd/",
  }
}

Command Line Project Configuration

In addition to the Hop Gui and configuration files, all aspects of and operations on projects and environments can be managed through the Hop Conf command line tool.

Configuration on the command line

The `hop-conf` script offers many options to edit environment definitions.

Creating an environment

$ sh hop-conf.sh \
     --environment-create \
     --environment hop2 \
     --environment-project hop2 \
     --environment-purpose=Development \
     --environment-config-files=/home/user/projects/hop2-conf.json
Creating environment 'hop2'
Environment 'hop2' was created in Hop configuration file <path-to-hop>/config/hop-config.json
2021/02/01 16:37:02 - General - ERROR: Configuration file '/home/user/projects/hop2-conf.json' does not exist to read variables from.
Created empty environment configuration file : /home/user/projects/hop2-conf.json
  hop2
    Purpose: Development
    Configuration files:
    Project name: hop2
      Config file: /home/user/projects/hop2-conf.json

As you can see from the log, an empty file was created to set variables in:

{ }

Setting variables in an environment

This command adds a variable to the environment configuration file:

$ sh hop-conf.sh --config-file /home/user/projects/hop2-conf.json --config-file-set-variables DB_HOSTNAME=localhost,DB_PASSWORD=abcd
Configuration file '/home/user/projects/hop2-conf.json' was modified.

If we look at the file `hop2-conf.json` we’ll see that the variables were added:

{
  "variables" : [ {
    "name" : "DB_HOSTNAME",
    "value" : "localhost",
    "description" : ""
  }, {
    "name" : "DB_PASSWORD",
    "value" : "abcd",
    "description" : ""
  } ]
}

Please note that you can add descriptions for the variables as well with the `--describe-variable` option. Please run hop-conf without options to see all the possibilities.

Deleting an environment

The following deletes an environment from the Hop configuration file:

$ sh hop-conf.sh --environment-delete --environment hop2
Lifecycle environment 'hop2' was deleted from Hop configuration file <path-to-hop>/config/hop-config.json

Projects Plugin configuration

There are various options to configure the behavior of the `Projects` plugin itself. In Hop configuration file `hop-config.json` we can find the following options:

{
    "projectMandatory" : true,
    "environmentMandatory" : false,
    "defaultProject" : "default",
    "defaultEnvironment" : null,
    "standardParentProject" : "default",
    "standardProjectsFolder" : "/home/matt/test-stuff/"
}