Upgrading to GoCD 20.5.0 and higher

GoCD 20.5.0 introduced several changes to its database implementation in order to build a more flexible model that allows integrating with multiple databases. As part of these changes GoCD changed the technologies used for automated database migrations (from the unmaintained DBDeploy to Liquibase). These changes require a one-time migration of the GoCD database <= 20.4.0 to one compliant with GoCD 20.5.0 and beyond.

GoCD 20.5.0, while continuing to support H2 by default, provides an ability to use PostgreSQL and MySQL. As part of this one-time database migration users can also choose to move to a database of their choice. Possible migrations are:

  1. H2 => PostgreSQL [Recommended]
  2. PostgreSQL => PostgreSQL
  3. H2 => H2
  4. H2 => MySQL [Warning! See note below]

GoCD Supports following database versions

  • PostgreSQL (v9.6, v10, v11 and v12)
  • MySQL (v8.0)

Follow the instructions below to migrate your exisiting GoCD <= 20.4.0 database to a GoCD 20.5.0 (and beyond) compliant database. The time taken by this migration is dependent on the size of your database. While testing we have seen the migration taking few minutes to more than an hour based on the size of the database. Please test on a backup of your GoCD server to understand the time taken for your particular database.

Note:

  • GoCD, by default, supports H2. Support for PostgreSQL used to be provided through a commercial addon. All of GoCD’s functional tests run against both H2 and PostgreSQL databases. While, support for MySQL is added in 20.5.0 and a basic round of migration tests has been completed, the functional test suite does not regularly run against MySQL as a part of the build pipeline. This is something to be aware of if moving to MySQL.

  • The H2 version used in GoCD 20.5.0 has moved to use MVStore as the default storage subsystem. The Current State of MVStore (as of June 2020) as per H2 documentation is marked as experimental. While using the default H2 is fine to get started or experimenting, we would recommend using PostgreSQL for production instances of GoCD.

  • Strong recommendation: try this migration on a non-production instance or backup of GoCD before attempting it on the production instance.

Step 1: Upgrade to GoCD 20.4.0

You should be able to migrate from any older version of GoCD to 20.5.0. However, over the last few releases there have been multiple changes to GoCD around installers and agent communication which could involve necessary changes to your setup. Hence it is recommended to do a normal upgrade to GoCD 20.4.0 and start GoCD 20.4.0 before performing an upgrade to GoCD 20.5.0.

Step 2: Backup

Backup your GoCD server. Refer the Backup GoCD Server documentation for instructions.

Step 3: Stop GoCD Server

Stop your GoCD server, if it is running.

Step 4: Database Migration

  1. Download the latest stable version of the migrator tool from the GitHub releases section of the GoCD database migration tool’s repository.

  2. Uncompress it and cd into the directory.

  3. Run ./bin/gocd-database-migrator --help for usage instructions.

Prerequisites: Ensure you have Java 8+ installed on the machine which runs the migration.

4.1 Migrating data from H2 to H2

  1. The gocd-database-migrator requires the source-db-url which consists of the location of the GoCD H2 database. The location of the database depends on the distribution your GoCD server is running on. Please refer to GoCD installation documentation to identfiy the file location.

  2. Run the command (The below example is for a GoCD server running on Linux) -

    1. ./bin/gocd-database-migrator \
    2.   --insert \
    3.   --progress \
    4.   --source-db-url='jdbc:h2:/var/lib/go-server/db/h2db/cruise' \
    5.   --source-db-user='sa' \
    6.   --source-db-password='' \
    7.   --target-db-url='jdbc:h2:/var/lib/go-server/db/h2db/new_cruise' \
    8.   --target-db-user='sa' \
    9.   --target-db-password=''

    For GoCD server running on Windows refer the below example -

    1. bin\gocd-database-migrator.bat ^
    2. --insert ^
    3. --progress ^
    4. --source-db-url="jdbc:h2:C:\Program Files (x86)\Go Server\db\h2db\cruise" ^
    5. --source-db-user="sa" ^
    6. --source-db-password="" ^
    7. --target-db-url="jdbc:h2:C:\Program Files (x86)\Go Server\db\h2db\new_cruise" ^
    8. --target-db-user="sa" ^
    9. --target-db-password=""

    Note: The source-db-url and target-db-url contain just the prefixes of the file names (cruise and new_cruise), even though the actual files are named: cruise.h2.db and new_cruise.mv.db.

  3. Delete, take a backup of or move away the file /var/lib/go-server/db/h2db/cruise.h2.db.

  4. Replace the old database with the migrated database by moving the file /var/lib/go-server/db/h2db/new_cruise.mv.db to /var/lib/go-server/db/h2db/cruise.mv.db.

  5. Ensure that the file permissions and ownership of the new cruise.mv.db file are correct (same as that of the old cruise.h2.db file).

4.2 Migrating data from PostgreSQL to PostgreSQL

  1. Create an empty database in PostgreSQL. Refer the PostgreSQL docs for information on creating an empty database.

  2. Run the command by providing the right parameters for the required options. An example is shown below:

    1. ./bin/gocd-database-migrator \
    2.   --insert \
    3.   --progress \
    4.   --source-db-url='jdbc:postgresql://localhost:5432/cruise' \
    5.   --source-db-user='postgres' \
    6.   --source-db-password='pass' \
    7.   --target-db-url='jdbc:postgresql://localhost:5432/new_cruise' 
    8.   --target-db-user='postgres' \
    9.   --target-db-password='pass'

4.3 Migrating data from H2 to PostgreSQL

  1. Create an empty database in PostgreSQL. Refer the PostgreSQL docs for information on creating an empty database.

  2. Run the command by providing the right parameters for the required options,

    1. ./bin/gocd-database-migrator \
    2.   --insert \
    3.   --progress \
    4.   --source-db-url='jdbc:h2:/var/lib/go-server/db/h2db/cruise' \
    5.   --source-db-user='sa' \
    6.   --source-db-password='' \
    7.   --target-db-url='jdbc:postgresql://localhost:5432/new_cruise' 
    8.   --target-db-user='postgres' \
    9.   --target-db-password='pass'

4.4 Migrating data from H2 to MySQL

  1. Create an empty database in MySQL. Refer the MySQL docs for information on creating an empty database.

  2. Run the command by providing the right parameters for the required options,

    1. ./bin/gocd-database-migrator \
    2.   --insert \
    3.   --progress \
    4.   --source-db-url='jdbc:h2:/var/lib/go-server/db/h2db/cruise' \
    5.   --source-db-user='sa' \
    6.   --source-db-password='' \
    7.   --target-db-url='jdbc:mysql://localhost:3306/new_cruise'
    8.   --target-db-user='root' \
    9.   --target-db-password='password'

Step 5: Configure db.properties for your GoCD server

5.1 Enabling GoCD to use H2 Database

GoCD runs on H2 by default. Configuring the db.properties is not required. Just make sure that the directory <<GoCD_installation_directory>>/db/h2db/ does not contain cruise.h2.db and contains cruise.mv.db.

5.2 Enabling GoCD to use PostgreSQL or MySQL Database

A properties file with the name db.properties needs to be created in GoCD’s configuration directory. This file should contain information about the PostgreSQL or MySQL server, so that the GoCD Server can connect to it. Refer the GoCD Database Connection Properties documentation for more information about the format of this file and valid keys.

The location of GoCD’s configuration directory varies per operating system. Usually, on a Linux system using the RPM or Debian installers, this file will need to be at /etc/go/db.properties. The installation documentation provides information about the locations.

  • Sample configuration for db.properties for PostgreSQL:

    1. db.driver=org.postgresql.Driver
    2. db.url=jdbc:postgresql://localhost:5432/new_cruise
    3. db.user=postgres
    4. db.password=pass

  • Sample configuration for db.properties for MySQL:

    1. db.driver=com.mysql.cj.jdbc.Driver
    2. db.url=jdbc:mysql://localhost:3306/gocd
    3. db.user=root
    4. db.password=password

Step 6: Only for users using the (old) commercial PostgreSQL addon

With GoCD now providing support for PostgreSQL, the previously commercial PostgreSQL addon is no longer required. You will, however, need to perform the PostgreSQL to PostgreSQL migration mentioned above and follow the instructions to configure db.properties. Once done:

  • Remove the PostgreSQL addon jar from the addons directory (typically /var/lib/go-server/addons on Linux)

  • Remove the postgresqldb.properties file from the configuration directory (typically /etc/go on Linux).

Step 7: Upgrade GoCD Server

Upgrade your GoCD server to 20.5.0+ and start the server.

Troubleshooting

Possible issues you might see are:

Database is read-only

You might see a message such as this, after upgrade, in the GoCD server logs:

  1. Caused by: org.h2.jdbc.JdbcSQLNonTransientException: The database is read only; SQL statement:
  2. UPDATE PUBLIC.DATABASECHANGELOGLOCK SET LOCKED = TRUE, LOCKEDBY = '10.16.0.5 (10.16.0.5)', LOCKGRANTED = '2020-06-17 15:07:20.707' WHERE ID = 1 AND LOCKED = FALSE [90097-200]
  3.     at org.h2.message.DbException.getJdbcSQLException(DbException.java:505)
  4.     at org.h2.message.DbException.getJdbcSQLException(DbException.java:429)
  5.     at org.h2.message.DbException.get(DbException.java:205)

This can happen due to the H2 DB file (usually at /var/lib/go-server/db/h2db/cruise.mv.db on Linux) having the wrong permissions or ownership.

MySQL: Identifier case senitivity

You might see a message such as this in the GoCD server logs, if you are using MySQL:

  1. Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.commons.dbcp2.BasicDataSource]: Factory method 'getDataSource' threw exception; nested exception is java.sql.SQLException: Unable to migrate the database
  2.         at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
  3.         at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
  4.         ... 73 common frames omitted
  5. Caused by: java.sql.SQLException: Unable to migrate the database
  6.         at com.thoughtworks.go.server.database.migration.DatabaseMigrator.migrate(DatabaseMigrator.java:68)
  7.         at com.thoughtworks.go.server.database.Database.getDataSource(Database.java:63)
  8.         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  9.         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  10.         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  11.         at java.base/java.lang.reflect.Method.invoke(Unknown Source)
  12.         at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
  13.         ... 74 common frames omitted
  14. Caused by: liquibase.exception.MigrationFailedException: Migration failed for change set db-migration-scripts/initial/create-trigger.xml::107::gocd(generated):
  15.      Reason: liquibase.exception.DatabaseException: Table 'gocd.buildStateTransitions' doesn't exist [Failed SQL: (1146) CREATE TRIGGER lastTransitionedTimeUpdate
  16.                 AFTER INSERT ON buildStateTransitions
  17.                 FOR EACH ROW
  18.                 BEGIN
  19.                     UPDATE stages SET lastTransitionedTime = NEW.statechangetime WHERE stages.id = NEW.stageid;
  20.                 END]
  21.         at liquibase.changelog.ChangeSet.execute(ChangeSet.java:646)
  22.         at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:53)
  23.         at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:83)
  24.         at liquibase.Liquibase.update(Liquibase.java:202)
  25.         at liquibase.Liquibase.update(Liquibase.java:179)
  26.         at liquibase.Liquibase.update(Liquibase.java:175)
  27.         at com.thoughtworks.go.server.database.migration.DatabaseMigrator.migrate(DatabaseMigrator.java:54)
  28.         ... 80 common frames omitted
  29. Caused by: liquibase.exception.DatabaseException: Table 'gocd.buildStateTransitions' doesn't exist [Failed SQL: (1146) CREATE TRIGGER lastTransitionedTimeUpdate
  30.                 AFTER INSERT ON buildStateTransitions
  31.                 FOR EACH ROW
  32.                 BEGIN
  33.                     UPDATE stages SET lastTransitionedTime = NEW.statechangetime WHERE stages.id = NEW.stageid;
  34.                 END]
  35.         at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:402)
  36.         at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:59)
  37.         at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:131)
  38.         at liquibase.database.AbstractJdbcDatabase.execute(AbstractJdbcDatabase.java:1276)
  39.         at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1258)
  40.         at liquibase.changelog.ChangeSet.execute(ChangeSet.java:609)
  41.         ... 86 common frames omitted
  42. Caused by: java.sql.SQLSyntaxErrorException: Table 'gocd.buildStateTransitions' doesn't exist
  43.         at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
  44.         at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
  45.         at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
  46.         at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:764)
  47.         at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:648)
  48.         at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
  49.         at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
  50.         at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:398)
  51.         ... 91 common frames omitted

If you see this, the most probable cause is that your MySQL instance has case-sensitive identifiers turned on. GoCD needs case-insensitive identifiers and you will need to change your MySQL instance to enable that. Please note that, according to the documentation, it is not possible to change the lower_case_table_names variable once the MySQL instance is initialized. You might need to recreate the instance.