Upgrading Kong Gateway

Using this guide, prepare for a Kong Gateway upgrade and determine which Kong Gateway upgrade paths to use.

This guide walks you through four available upgrade strategies and recommends the best strategy for each Kong Gateway deployment mode. Additionally, it lists some fundamental factors that play important roles in the upgrade process, and explains how to back up and recover data.

This guide uses the following terms in the context of Kong Gateway:

• Upgrade: The overall process of switching from an older to a newer version of Kong Gateway.
• Migration: The migration of your data store data into a new environment. For example, the process of moving 2.8.x data from an old PostgreSQL instance to a new one for 3.4.x is referred to as database migration.

Note: If you are interested in upgrading between the Kong Gateway Enterprise 2.8.x and 3.4.x long-term support (LTS) versions, see the LTS upgrade guide.

Upgrade overview

A Kong Gateway upgrade requires two phases of work: preparing for the upgrade and applying the upgrade.

Preparation phase

1. Review version compatibility between your platform version and the version of that you are upgrading to:
   • OS version
   • Kubernetes version and Helm prerequisites
   • Database version
   • Dependency versions
2. Determine your upgrade path based on the release you’re starting from and the release you’re upgrading to.
3. Back up your database or your declarative configuration files.
4. Choose the right strategy for upgrading based on your deployment topology.
5. Review the breaking changes for the version you’re upgrading to.
6. Review any remaining upgrade considerations.
7. Test migration in a pre-production environment.

Performing the upgrade

The actual execution of the upgrade is dependent on the type of deployment you have with Kong Gateway. In this part of the upgrade journey, you will use the strategy you determined during the preparation phase.

1. Execute your chosen upgrade strategy on dev.
2. Move from dev to prod.
3. Smoke test.
4. Wrap up the upgrade or roll back and try again.

Now, let’s move on to preparation, starting with determining your upgrade path.

Preparation: Review upgrade paths

Kong Gateway adheres to a structured approach to versioning its products, which makes a distinction between major, minor, and patch versions.

The upgrade from 2.x to 3.x is a major upgrade. The lowest version that Kong Gateway 3.0.x supports migrating from is 2.1.x.

Kong Gateway does not support directly upgrading from 1.x to 3.x. If you are running 1.x, upgrade to 2.1.0 first at a minimum, then upgrade to 3.0.x from there.

While you can upgrade directly to the latest version, be aware of any breaking changes between the 2.x and 3.x series (both this version and prior versions) and in the open-source (OSS) and Enterprise Gateway changelogs. Since Kong Gateway is built on an open-source foundation, any breaking changes in OSS affect all Kong Gateway packages.

An upgrade path is subject to a wide spectrum of conditions, and there is not a one-size-fits-all way applicable to all users, depending on the deployment modes, custom plugins, technical capabilities, hardware capacities, SLA, and so on. You should discuss the upgrade process thoroughly and carefully with our engineers before you take any action.

We encourage you to stay updated with Kong Gateway releases, as that helps maintain a smooth upgrade path. The smaller the version gap is, the less complex the upgrade process becomes.

Guaranteed upgrade paths

By default, Kong Gateway has migration tests between adjacent versions and hence the following upgrade paths are guaranteed officially:

1. Between patch releases of the same major and minor version.
2. Between adjacent minor releases of the same major version.

3. Between LTS (Long Term Support) versions.

   Kong Gateway maintains LTS versions and guarantees upgrades between adjacent LTS versions. The current LTS in the 2.x series is 2.8, and the current LTS in the 3.x series is 3.4. If you want to upgrade between the 2.8 and 3.4 LTS versions, see the LTS Upgrade guide.

The following table outlines various upgrade path scenarios to 3.8.x depending on the Kong Gateway version you are currently using:

Preparation: Choose a backup strategy

The kong migrations commands used during upgrade and database migration are not reversible. Always back up your database or declarative configuration files before an upgrade.

There are two main types of backup for Kong Gateway entities:

• Database backup: PostgreSQL has native exporting and importing tools that are reliable and performant, and that ensure consistency when backing up or restoring data. If you’re running Kong Gateway in traditional or hybrid mode, you should always take a database-native backup.
• Declarative backup: Kong ships two declarative backup tools: decK and the Kong CLI, which support managing Kong Gateway entities in the declarative format. For traditional and hybrid mode deployments, use these tools to create secondary backups. For DB-less mode deployments, use the Kong CLI and manually manage declarative configuration files.

We highly recommend backing up your data using both methods if possible, as this offers you recovery flexibility.

The database-native tools are robust and can restore data instantly compared to the declarative tools. In case of data corruption, try to do a database-level restore first. Otherwise, bootstrap a new database and use declarative tools to restore configuration from backup files.

Review the Backup and Restore guide to prepare backups of your configuration. If you run into any issues and need to roll back, you can also reference that guide to restore your old data store.

Preparation: Choose an upgrade strategy based on deployment mode

Though you could define your own upgrade procedures, we recommend using one of the nominated strategies in this section. Any custom upgrade requirements may require a well-tailored upgrade strategy. For example, if you only want a small group of customer objects to be directed to the new version, use the Canary plugin and a load balancer that supports traffic interception.

Whichever upgrade strategy you choose, you should account for management downtime for Kong Gateway, as Admin API operations and database updates are not allowed during the upgrade process.

Based on your deployment type, we recommend one of the following upgrade strategies. Carefully read the descriptions for each option to choose the upgrade strategy that works best for your situation.

• Traditional or hybrid mode control planes:
  • Dual-cluster upgrade
  • In-place upgrade
  • Blue-green upgrade (not recommended)
• DB-less mode or hybrid mode data planes:
  • Rolling upgrade

Here’s a flowchart that breaks down how the decision process works:

flowchart TD
    A{Deployment type?} --> B(Traditional mode)
    A{Deployment type?} --> C(Hybrid mode)
    A{Deployment type?} --> D(DB-less mode)
    A{Deployment type?} --> E(Konnect DP)
    B ---> F{Enough hardware to 
    run another cluster?}
    C --> G(Upgrade CP first) & H(Upgrade DP second)
    D ----> K([Rolling upgrade])
    E ----> K
    G --> F
    F ---Yes--->I([Dual-cluster upgrade])
    F ---No--->J([In-place upgrade])
    H ---> K
    click K "/gateway/latest/upgrade/rolling-upgrade/"
    click I "/gateway/latest/upgrade/dual-cluster/"
    click J "/gateway/latest/upgrade/in-place/"

Figure 1: Choose an upgrade strategy based on your deployment type. For traditional mode, choose a dual-cluster upgrade if you have enough resources, or an in-place upgrade if you don’t have enough resources. For DB-less mode and Konnect DPs, use a rolling upgrade. For hybrid mode, use one of the traditional mode strategies for CPs, and the rolling upgrade for DPs.

See the following sections for breakdowns and links to each upgrade strategy guide.

Traditional mode

A traditional mode deployment is when all Kong Gateway components are running in one environment, and there is no control plane/data plane separation.

You have two options when upgrading Kong Gateway in traditional mode:

• Dual-cluster upgrade: A new Kong Gateway cluster of version Y is deployed alongside the current version X, so that two clusters serve requests concurrently during the upgrade process.
• In-place upgrade: An in-place upgrade reuses the existing database and has to shut down cluster X first, then configure the new cluster Y to point to the database.

We recommend using a dual-cluster upgrade if you have the resources to run another cluster concurrently. Use the in-place method only if resources are limited, as it will cause business downtime.

Important: While the blue-green upgrade strategy is an option, we do not recommend it. Support from Kong for upgrades using this strategy is limited. It is nearly impossible to fully cover all migration tests, because we have to cover all combinations, given the number of Kong Gateway versions, upgrade strategies, features adopted, and deployment modes. If you must use this strategy, only use it to upgrade between patch versions.

DB-less mode

In DB-less mode, each independent Kong Gateway node loads a copy of declarative Kong Gateway configuration data into memory without persistent database storage, so failure of some nodes doesn’t spread to other nodes.

Deployments in this mode should use the rolling upgrade strategy. You could parse the validity of the declarative YAML contents with version Y, using the deck gateway validate or the kong config parse command.

You must back up your current kong.yaml file before starting the upgrade.

Hybrid mode

Hybrid mode comprises of one or more control plane (CP) nodes, and one or more data plane (DP) nodes. CP nodes use a database to store Kong configuration data, whereas DP nodes don’t, since they get all of the needed information from the CP. The recommended upgrade process is a combination of different upgrade strategies for each type of node, CP or DP.

The major challenge with a hybrid mode upgrade is the communication between the CP and DP. As hybrid mode requires the minor version of the CP to be no less than that of the DP, you must upgrade CP nodes before DP nodes. The upgrade must be carried out in two phases:

1. First, upgrade the CP according to the recommendations in the section Traditional Mode, while DP nodes are still serving API requests.
2. Next, upgrade DP nodes using the recommendations from the section DB-less Mode. Point the new DP nodes to the new CP to avoid version conflicts.

The role decoupling feature between CP and DP enables DP nodes to serve API requests while upgrading CP. With this method, there is no business downtime.

Custom plugins (either your own plugins or third-party plugins that are not shipped with Kong Gateway) need to be installed on both the control plane and the data planes in hybrid mode. Install the plugins on the control plane first, and then the data planes.

See the following sections for a breakdown of the options for hybrid mode deployments.

Control planes

CP nodes must be upgraded before DP nodes. CP nodes serve an admin-only role and require database support. So, you can select from the same upgrade strategies nominated for traditional mode (dual-cluster or in-place), as described in figure 2 and figure 3 respectively.

Upgrading the CP nodes using the dual-cluster strategy:

flowchart TD
    DBA[(Current
    database)]
    DBB[(New 
    database)]
    CPX(Current control plane X)
    Admin(No admin 
    write operations)
    CPY(New control plane Y)
    DPX(fa:fa-layer-group Current data plane X nodes)
    API(API requests)
    DBA -.- CPX -."DP connects to either \nCP X...".- DPX
    Admin -.X.- CPX & CPY
    DBB --pg_restore--- CPY -."...OR to CP Y".- DPX
    API--> DPX
    style API stroke:none
    style DBA stroke-dasharray:3
    style CPX stroke-dasharray:3
    style Admin fill:none,stroke:none,color:#d44324
    linkStyle 2,3 stroke:#d44324,color:#d44324

Figure 2: The diagram shows a CP upgrade using the dual-cluster strategy. The new CP Y is deployed alongside the current CP X, while current DP nodes X are still serving API requests.

Upgrading the CP nodes using the in-place strategy:

flowchart 
    DBA[(Database)]
    CPX(Current control plane X \n #40;inactive#41;)
    Admin(No admin \n write operations)
    CPY(New control plane Y)
    DPX(fa:fa-layer-group Current data plane X nodes)
    API(API requests)
    DBA -..- CPX -."DP connects to either \nCP X...".- DPX
    Admin -.X.- CPX & CPY
    DBA --"kong migrations up \n kong migrations finish"--- CPY -."...OR to CP Y".- DPX
    API--> DPX
    style API stroke:none
    style CPX stroke-dasharray:3
    style Admin fill:none,stroke:none,color:#d44324
    linkStyle 2,3 stroke:#d44324,color:#d44324

Figure 3: The diagram shows a CP upgrade using the in-place strategy, where the current CP X is directly replaced by a new CP Y. The database is reused by the new CP Y, and the current CP X is shut down once all nodes are migrated.

From the two figures, you can see that DP nodes X remain connected to the current CP node X, or alternatively switch to the new CP node Y. Kong Gateway guarantees that new minor versions of CPs are compatible with old minor versions of the DP, so you can temporarily point DP nodes X to the new CP node Y. This lets you pause the upgrade process if needed, or conduct it over a longer period of time.

This setup is meant to be temporary, to be used only during the upgrade process. We do not recommend running a combination of new versions of CP nodes and old versions of DP nodes in a long-term production deployment.

After the CP upgrade, cluster X can be decommissioned. You can delay this task to the very end of the DP upgrade.

Data planes

Once the CP nodes are upgraded, you can move on to upgrade the DP nodes. The only supported upgrade strategy for DP upgrades is the rolling upgrade. The following diagrams, figure 4 and 5, are the counterparts of figure 2 and 3 respectively.

Using the dual-cluster strategy with a rolling upgrade workflow:

flowchart TD
    DBX[(Current \n database)]
    DBY[(New \n database)]
    CPX(Current control plane X)
    CPY(New control plane Y)
    DPX(Current data planes X)
    DPY(New data planes Y)
    API(API requests)
    LB(Load balancer)
    Admin(No admin \n write operations)
    Admin2(No admin \n write operations)
    subgraph A
        Admin -.X.- CPX
        DBX -.- CPX
        DBY --- CPY
        CPX -."Current DP connects to \neither CP X...".- DPX
        Admin2 -.X.- CPY
        CPY -."...OR to CP Y".- DPX
        DPX -.90%..- LB
        CPY --- DPY --10%---- LB
    end
    subgraph B
        API --> LB & LB & LB
    end
    linkStyle 0,4 stroke:#d44324,color:#d44324
    linkStyle 8,9 stroke:#b6d7a8
    style CPX stroke-dasharray:3,fill:#eff0f1ff,stroke:#c1c6cdff
    style DPX stroke-dasharray:3
    style DBX stroke-dasharray:3,fill:#eff0f1ff,stroke:#c1c6cdff
    style API stroke:none
    style A stroke:none,color:#fff
    style B stroke:none,color:#fff
    style Admin fill:none,stroke:none,color:#d44324
    style Admin2 fill:none,stroke:none,color:#d44324

Figure 4: The diagram shows a DP upgrade using the dual-cluster and rolling strategies. The new CP Y is deployed alongside with the current CP X, while current DP nodes X are still serving API requests. In the image, the background color of the current database and CP X is grey instead of white, signaling that the old CP is already upgraded and might have been decommissioned.

Using the in-place strategy strategy with a rolling upgrade workflow:

flowchart 
    DBA[(Database)]
    CPX(Current control plane X \n #40;inactive#41;)
    CPY(New control plane Y)
    DPX(Current data planes X)
    DPY(New data planes Y)
    API(API requests)
    LB(Load balancer)
    Admin(No admin \n write operations)
    Admin2(No admin \n write operations)
    subgraph A
        Admin -.X.- CPX
        DBA -.X.- CPX
        DBA --- CPY
        CPX -."Current DP connects to \neither CP X...".- DPX
        Admin2 -.X.- CPY
        CPY -."OR to CP Y".- DPX -.90%..- LB
        CPY --- DPY --10%---- LB 
    end
    subgraph B
        API --> LB & LB & LB
    end
    linkStyle 0,1,4 stroke:#d44324,color:#d44324
    linkStyle 8,9 stroke:#b6d7a8
    style CPX stroke-dasharray:3,fill:#eff0f1ff,stroke:#c1c6cdff
    style DPX stroke-dasharray:3
    style A stroke:none,color:#fff
    style B stroke:none,color:#fff
    style Admin fill:none,stroke:none,color:#d44324
    style Admin2 fill:none,stroke:none,color:#d44324

Figure 5: The diagram shows a DP upgrade using the in-place and rolling strategies. The diagram shows that the database is reused by the new CP Y, while current DP nodes X are still serving API requests.

Upgrades from 3.1.0.0 or 3.1.1.1

There is a special case if you deployed Kong Gateway in hybrid mode and the version you are using is 3.1.0.0 or 3.1.1.1. Kong removed the legacy WebSocket protocol between the CP and DP, replaced it with a new WebSocket protocol in 3.1.0.0, and added back the legacy one in 3.1.1.2. So, upgrade to 3.1.1.2 first before moving forward to later versions.

Additionally, the new WebSocket protocol has been completely removed since 3.2.0.0. See the following table for the version breakdown:

Preparation: Review breaking changes and changelogs

Review the breaking changes and changelogs for the release or releases that you are upgrading to. Make any preparations or adjustments as directed in the breaking changes.

Preparation: Upgrade considerations

There are some universal factors that may also influence the upgrade, regardless of your deployment mode.

Selecting a strategy for the target deployment mode doesn’t guarantee that you can start the upgrade immediately. You must also account for the following factors:

• During the upgrade process, no changes can be made to the database. Until the upgrade is completed:
  • Don’t write to the database via the Admin API.
  • Don’t operate on the database directly.
  • Don’t update configuration through Kong Manager, decK, or the kong config CLI.
• Review the compatibility between the new version Y and your existing platform. Factors may include, but are not limited to:
  • OS version
  • Kubernetes version and Helm prerequisites
  • Hardware resources
  • Database version
  • Dependency versions

• Carefully review all changelogs starting from your current version X, all the way up to the target version Y.

  • Look for any potential conflicts, especially for schema changes and functionality removal.

  • When configuring the new cluster, update kong.conf directly or via environment variables based on the changelog.

    Breaking changes in kong.conf in a minor version upgrade are infrequent, but do happen.

    For example, the parameter pg_ssl_version defaults to tlsv1 in 2.8.2.4, but in 3.2.2.1, tlsv1 is not a valid argument anymore. If you were depending on this setting, you would have to adjust your environment to fit one of the new options.

• If you have custom plugins, review the code against changelog and test the custom plugin using the new version Y.

• If you have modified any Nginx templates like nginx-kong.conf and nginx-kong-stream.conf, also make those changes to the templates for the new version Y. Refer to Nginx Directives for a detailed customization guide.
• If you’re using Kong Gateway Enterprise, make sure to apply the enterprise license to the new Gateway cluster.
• Always remember to take a backup.
• Cassandra DB support has been removed from Kong Gateway with 3.4.0.0. Migrate to PostgreSQL according to the Cassandra to PostgreSQL Migration Guidelines.

Perform the upgrade

Once you have reviewed everything and chosen a strategy, proceed to upgrade Kong Gateway using your chosen strategy:

• Dual-cluster upgrade
• In-place upgrade
• Blue-green upgrade
• Rolling upgrade