Helm Provenance and Integrity

Helm has provenance tools which help chart users verify the integrity and originof a package. Using industry-standard tools based on PKI, GnuPG, andwell-respected package managers, Helm can generate and verify signature files.

Overview

Integrity is established by comparing a chart to a provenance record. Provenancerecords are stored in provenance files, which are stored alongside a packagedchart. For example, if a chart is named myapp-1.2.3.tgz, its provenance filewill be myapp-1.2.3.tgz.prov.

Provenance files are generated at packaging time (helm package —sign …),and can be checked by multiple commands, notably helm install —verify.

The Workflow

This section describes a potential workflow for using provenance dataeffectively.

Prerequisites:

  • A valid PGP keypair in a binary (not ASCII-armored) format
  • The helm command line tool
  • GnuPG command line tools (optional)
  • Keybase command line tools (optional)NOTE: If your PGP private key has a passphrase, you will be prompted toenter that passphrase for any commands that support the —sign option.

Creating a new chart is the same as before:

  1. $ helm create mychart
  2. Creating mychart

Once ready to package, add the —sign flag to helm package. Also, specifythe name under which the signing key is known and the keyring containing thecorresponding private key:

  1. $ helm package --sign --key 'helm signing key' --keyring path/to/keyring.secret mychart

TIP: for GnuPG users, your secret keyring is in ~/.gnupg/secring.gpg. Youcan use gpg —list-secret-keys to list the keys you have.

Warning: the GnuPG v2 store your secret keyring using a new format kbx onthe default location ~/.gnupg/pubring.kbx. Please use the following commandto convert your keyring to the legacy gpg format:

  1. $ gpg --export-secret-keys >~/.gnupg/secring.gpg

At this point, you should see both mychart-0.1.0.tgz andmychart-0.1.0.tgz.prov. Both files should eventually be uploaded to yourdesired chart repository.

You can verify a chart using helm verify:

  1. $ helm verify mychart-0.1.0.tgz

A failed verification looks like this:

  1. $ helm verify topchart-0.1.0.tgz
  2. Error: sha256 sum does not match for topchart-0.1.0.tgz: "sha256:1939fbf7c1023d2f6b865d137bbb600e0c42061c3235528b1e8c82f4450c12a7" != "sha256:5a391a90de56778dd3274e47d789a2c84e0e106e1a37ef8cfa51fd60ac9e623a"

To verify during an install, use the —verify flag.

  1. $ helm install --verify mychart-0.1.0.tgz

If the keyring containing the public key associated with the signed chart isnot in the default location, you may need to point to the keyring with—keyring PATH as in the helm package example.

If verification fails, the install will be aborted before the chart is evenrendered.

Using Keybase.io credentials

The Keybase.io service makes it easy to establish a chainof trust for a cryptographic identity. Keybase credentials can be used to signcharts.

Prerequisites:

  • A configured Keybase.io account
  • GnuPG installed locally
  • The keybase CLI installed locally

Signing packages

The first step is to import your keybase keys into your local GnuPG keyring:

  1. $ keybase pgp export -s | gpg --import

This will convert your Keybase key into the OpenPGP format, and then import itlocally into your ~/.gnupg/secring.gpg file.

You can double check by running gpg —list-secret-keys.

  1. $ gpg --list-secret-keys
  2. /Users/mattbutcher/.gnupg/secring.gpg
  3. -------------------------------------
  4. sec   2048R/1FC18762 2016-07-25
  5. uid                  technosophos (keybase.io/technosophos) <technosophos@keybase.io>
  6. ssb   2048R/D125E546 2016-07-25

Note that your secret key will have an identifier string:

  1. technosophos (keybase.io/technosophos) <technosophos@keybase.io>

That is the full name of your key.

Next, you can package and sign a chart with helm package. Make sure you use atleast part of that name string in —key.

  1. $ helm package --sign --key technosophos --keyring ~/.gnupg/secring.gpg mychart

As a result, the package command should produce both a .tgz file and a.tgz.prov file.

Verifying packages

You can also use a similar technique to verify a chart signed by someone else’sKeybase key. Say you want to verify a package signed bykeybase.io/technosophos. To do this, use the keybase tool:

  1. $ keybase follow technosophos
  2. $ keybase pgp pull

The first command above tracks the user technosophos. Next keybase pgp pulldownloads the OpenPGP keys of all of the accounts you follow, placing them inyour GnuPG keyring (~/.gnupg/pubring.gpg).

At this point, you can now use helm verify or any of the commands with a—verify flag:

  1. $ helm verify somechart-1.2.3.tgz

Reasons a chart may not verify

These are common reasons for failure.

  • The .prov file is missing or corrupt. This indicates that something ismisconfigured or that the original maintainer did not create a provenancefile.
  • The key used to sign the file is not in your keyring. This indicate that theentity who signed the chart is not someone you’ve already signaled that youtrust.
  • The verification of the .prov file failed. This indicates that something iswrong with either the chart or the provenance data.
  • The file hashes in the provenance file do not match the hash of the archivefile. This indicates that the archive has been tampered with.If a verification fails, there is reason to distrust the package.

The Provenance File

The provenance file contains a chart’s YAML file plus several pieces ofverification information. Provenance files are designed to be automaticallygenerated.

The following pieces of provenance data are added:

  • The chart file (Chart.yaml) is included to give both humans and tools an easyview into the contents of the chart.
  • The signature (SHA256, just like Docker) of the chart package (the .tgz file)is included, and may be used to verify the integrity of the chart package.

  • The entire body is signed using the algorithm used by OpenPGP (see[https://keybase.io] for an emerging way of making crypto signing andverification easy).The combination of this gives users the following assurances:

  • The package itself has not been tampered with (checksum package .tgz).

  • The entity who released this package is known (via the GnuPG/PGP signature).The format of the file looks something like this:
  1. Hash: SHA512
  3. apiVersion: v2
  4. appVersion: 1.16.0
  5. description: Sample chart
  6. name: mychart
  7. type: application
  8. version: 0.1.0
  10. ...
  11. files:
  12.   mychart-0.1.0.tgz: sha256:d31d2f08b885ec696c37c7f7ef106709aaf5e8575b6d3dc5d52112ed29a9cb92
  13. -----BEGIN PGP SIGNATURE-----
  15. wsBcBAEBCgAQBQJdy0ReCRCEO7+YH8GHYgAAfhUIADx3pHHLLINv0MFkiEYpX/Kd
  16. nvHFBNps7hXqSocsg0a9Fi1LRAc3OpVh3knjPfHNGOy8+xOdhbqpdnB+5ty8YopI
  17. mYMWp6cP/Mwpkt7/gP1ecWFMevicbaFH5AmJCBihBaKJE4R1IX49/wTIaLKiWkv2
  18. cR64bmZruQPSW83UTNULtdD7kuTZXeAdTMjAK0NECsCz9/eK5AFggP4CDf7r2zNi
  19. hZsNrzloIlBZlGGns6mUOTO42J/+JojnOLIhI3Psd0HBD2bTlsm/rSfty4yZUs7D
  20. qtgooNdohoyGSzR5oapd7fEvauRQswJxOA0m0V+u9/eyLR0+JcYB8Udi1prnWf8=
  21. =aHfz
  22. -----END PGP SIGNATURE-----

Note that the YAML section contains two documents (separated by …\n). Thefirst file is the content of Chart.yaml. The second is the checksums, a mapof filenames to SHA-256 digests of that file’s content at packaging time.

The signature block is a standard PGP signature, which provides tamperresistance.

Chart Repositories

Chart repositories serve as a centralized collection of Helm charts.

Chart repositories must make it possible to serve provenance files over HTTP viaa specific request, and must make them available at the same URI path as thechart.

For example, if the base URL for a package ishttps://example.com/charts/mychart-1.2.3.tgz, the provenance file, if itexists, MUST be accessible athttps://example.com/charts/mychart-1.2.3.tgz.prov.

From the end user’s perspective, helm install —verify myrepo/mychart-1.2.3should result in the download of both the chart and the provenance file with noadditional user configuration or action.

Establishing Authority and Authenticity

When dealing with chain-of-trust systems, it is important to be able toestablish the authority of a signer. Or, to put this plainly, the system abovehinges on the fact that you trust the person who signed the chart. That, inturn, means you need to trust the public key of the signer.

One of the design decisions with Helm has been that the Helm projectwould not insert itself into the chain of trust as a necessary party. We don’twant to be “the certificate authority” for all chart signers. Instead, westrongly favor a decentralized model, which is part of the reason we choseOpenPGP as our foundational technology. So when it comes to establishingauthority, we have left this step more-or-less undefined in Helm 2 (a decisioncarried forward in Helm 3).

However, we have some pointers and recommendations for those interested in usingthe provenance system:

  • The Keybase platform provides a public centralizedrepository for trust information.
    • You can use Keybase to store your keys or to get the public keys of others.
    • Keybase also has fabulous documentation available
    • While we haven’t tested it, Keybase’s “secure website” feature could be usedto serve Helm charts.
  • The official Helm Charts project istrying to solve this problem for the official chart repository.
    • There is a long issue there detailing the currentthoughts.
    • The basic idea is that an official “chart reviewer” signs charts with her orhis key, and the resulting provenance file is then uploaded to the chartrepository.
    • There has been some work on the idea that a list of valid signing keys maybe included in the index.yaml file of a repository.Finally, chain-of-trust is an evolving feature of Helm, and some communitymembers have proposed adapting part of the OSI model for signatures. This is anopen line of inquiry in the Helm team. If you’re interested, jump on in.