Release Guide

Check Your Environment

To make sure you could successfully complete the release for DolphinScheduler, you should check your environment and make sure all conditions are met, if any or them are missing, you should install them and make sure them work.

  1. # JDK 1.8 above is requests
  2. java -version
  3. # Maven requests
  4. mvn -version
  5. # Python 3.6 above is requests, and you have to make keyword `python` work in your terminal and version match
  6. python --version

GPG Settings

Install GPG

Download installation package on official GnuPG website. The command of GnuPG 1.x version can differ a little from that of 2.x version. The following instructions take GnuPG-2.1.23 version for example.

After the installation, execute the following command to check the version number.

  1. gpg --version

Create Key

After the installation, execute the following command to create key.

This command indicates GnuPG-2.x can be used:

  1. gpg --full-gen-key

This command indicates GnuPG-1.x can be used:

  1. gpg --gen-key

Finish the key creation according to instructions, Notice: Please use Apache mails and its password for key creation.

  1. gpg (GnuPG) 2.0.12; Copyright (C) 2009 Free Software Foundation, Inc.
  2. This is free software: you are free to change and redistribute it.
  3. There is NO WARRANTY, to the extent permitted by law.
  5. Please select what kind of key you want:
  6.   (1) RSA and RSA (default)
  7.   (2) DSA and Elgamal
  8.   (3) DSA (sign only)
  9.   (4) RSA (sign only)
  10. Your selection? 1
  11. RSA keys may be between 1024 and 4096 bits long.
  12. What keysize do you want? (2048) 4096
  13. Requested keysize is 4096 bits
  14. Please specify how long the key should be valid.
  15.         0 = key does not expire
  16.      <n>  = key expires in n days
  17.      <n>w = key expires in n weeks
  18.      <n>m = key expires in n months
  19.      <n>y = key expires in n years
  20. Key is valid for? (0)
  21. Key does not expire at all
  22. Is this correct? (y/N) y
  24. GnuPG needs to construct a user ID to identify your key.
  26. Real name: ${Input username}
  27. Email address: ${Input email}
  28. Comment: ${Input comment}
  29. You selected this USER-ID:
  30.    "${Inputed username} (${Inputed comment}) <${Inputed email}>"
  32. Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
  33. You need a Passphrase to protect your secret key. # Input your Apache mail passwords

Check Generated Key

  1. gpg --list-keys

Execution Result:

  1. pub   4096R/85E11560 2019-11-15
  2. uid                  ${Username} (${Comment}) <{Email}>
  3. sub   4096R/A63BC462 2019-11-15

Among them, 85E11560 is public key ID.

Upload the Public Key to Key Server

The command is as follow:

  1. gpg --keyserver hkp://pool.sks-keyservers.net --send-key 85E11560

pool.sks-keyservers.net is randomly chosen from public key server. Each server will automatically synchronize with one another, so it would be okay to choose any one, a backup keys servers is gpg --keyserver hkp://keyserver.ubuntu.com --send-key <YOUR_KEY_ID>

Apache Maven Central Repository Release

Set settings-security.xml and settings.xml

In this section, we add Apache server maven configuration to prepare the release, we have to add settings-security.xml according to here firstly and then change your ~/.m2/settings.xml like below

  1. <settings>
  2.   <servers>
  3.     <server>
  4.       <id>apache.snapshots.https</id>
  5.       <username> <!-- APACHE LDAP username --> </username>
  6.       <password> <!-- APACHE LDAP encrypted password --> </password>
  7.     </server>
  8.     <server>
  9.       <id>apache.releases.https</id>
  10.       <username> <!-- APACHE LDAP username --> </username>
  11.       <password> <!-- APACHE LDAP encrypted password --> </password>
  12.     </server>
  13.   </servers>
  14. </settings>

Set Release in Environment

We will use the release version, your github name and your Apache username below several times, so it is better to store it to bash variable for easier use.

  1. VERSION=<THE-VERSION-YOU-RELEASE>
  2. GH_USERNAME=<YOUR-GITHUB-USERNAME>
  3. A_USERNAME=<YOUR-APACHE-USERNAME>

Note: We can use the variable directly in you bash after we set environment, without changing anything. For example, we can use command git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git to clone the release branch and it can be success by covert the "${VERSION}" to <THE-VERSION-YOU-RELEASE>. But you have to change <VERSION> manually in some of not bash step like vote mail, we using <VERSION> instead of "${VERSION}" to notice release manager they have to change by hand.

Create Release Branch

In this section, we dwonload source code from github and create new branch to release

  1. git clone -b "${VERSION}"-prepare https://github.com/apache/dolphinscheduler.git
  2. cd ~/dolphinscheduler/
  3. git pull
  4. git checkout -b "${VERSION}"-release
  5. git push origin "${VERSION}"-release

Pre-Release Check

  1. # make gpg command could be run in maven correct
  2. export GPG_TTY=$(tty)
  4. mvn release:prepare -Prelease,python -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true" -DautoVersionSubmodules=true -DdryRun=true -Dusername="${GH_USERNAME}"
  • -Prelease,python: choose release and python profile, which will pack all the source codes, jar files and executable binary packages, and Python distribute package.
  • -DautoVersionSubmodules=true: it can make the version number is inputted only once and not for each sub-module.
  • -DdryRun=true: dry run which means not to generate or submit new version number and new tag.

Prepare for the Release

First, clean local pre-release check information.

  1. mvn release:clean

Then, prepare to execute the release.

  1. mvn release:prepare -Prelease,python -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true" -DautoVersionSubmodules=true -DpushChanges=false -Dusername="${GH_USERNAME}"

It is basically the same as the previous rehearsal command, but deleting -DdryRun=true parameter.

  • -DpushChanges=fals: do not submit the edited version number and tag to GitHub automatically.

Note: You have to config your git user.name and user.password by command git config --global user.email "you@example.com" and git config --global user.name "Your Name" if you meet some mistake like Please tell me who you are. from git.

After making sure there is no mistake in local files, submit them to GitHub.

  1. git push -u origin "${VERSION}"-release
  2. git push origin --tags

Note1: In this step, you should use github token for password because native password no longer supported, you can see https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token for more detail about how to create token about it.

Note2: After the command done, it will auto-created release.properties file and *.Backup files, their will be need in the following command and DO NOT DELETE THEM

Deploy the Release

  1. mvn release:perform -Prelease,python -Darguments="-Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true" -DautoVersionSubmodules=true -Dusername="${GH_USERNAME}"

After that command is executed, the version to be released will be uploaded to Apache staging repository automatically. Go to apache staging repositories and login by Apache LDAP. then you can see the uploaded version, the content of Repository column is the ${STAGING.REPOSITORY}. Click Close to tell Nexus that the construction is finished, because only in this way, this version can be usable. If there is any problem in gpg signature, Close will fail, but you can see the failure information through Activity.

Apache SVN Repository Release

Checkout dolphinscheduler Release Directory

If there is no local work directory, create one at first.

  1. mkdir -p ~/ds_svn/dev/
  2. cd ~/ds_svn/dev/

After the creation, checkout dolphinscheduler release directory from Apache SVN.

  1. svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/dev/dolphinscheduler
  2. cd ~/ds_svn/dev/dolphinscheduler

Add gpg Public Key

Only the account in its first deployment needs to add that. It is alright for KEYS to only include the public key of the deployed account.

  1. gpg -a --export <YOUR-GPG-KEY-ID> >> KEYS

Add the Release Content to SVN Directory

Create folder by version number.

  1. mkdir -p ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
  2. mkdir -p ~/ds_svn/dev/dolphinscheduler/"${VERSION}"/python
  3. cd ~/ds_svn/dev/dolphinscheduler/"${VERSION}"

Add source code packages, binary packages and executable binary packages to SVN working directory.

  1. # Source and binary tarball for main code
  2. cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/*.tar.gz ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
  3. cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/*.tar.gz.asc ~/ds_svn/dev/dolphinscheduler/"${VERSION}"
  5. # Source and binary tarball for Python API
  6. cp -f ~/dolphinscheduler/dolphinscheduler-dist/target/python/* ~/ds_svn/dev/dolphinscheduler/"${VERSION}"/python

Generate sign files

  1. shasum -a 512 apache-dolphinscheduler-"${VERSION}"-src.tar.gz >> apache-dolphinscheduler-"${VERSION}"-src.tar.gz.sha512
  2. shasum -b -a 512 apache-dolphinscheduler-"${VERSION}"-bin.tar.gz >> apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.sha512
  3. cd python
  4. shasum -a 512 apache-dolphinscheduler-python-"${VERSION}".tar.gz >> apache-dolphinscheduler-python-"${VERSION}".tar.gz.sha512
  5. shasum -b -a 512 apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl >> apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl.sha512
  6. cd ../

Commit to Apache SVN

  1. cd ~/ds_svn/dev/dolphinscheduler
  2. svn add *
  3. svn --username="${A_USERNAME}" commit -m "release ${VERSION}"

Check Release

Check sha512 hash

  1. shasum -c apache-dolphinscheduler-"${VERSION}"-src.tar.gz.sha512
  2. shasum -c apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.sha512
  3. cd python
  4. shasum -c apache-dolphinscheduler-python-"${VERSION}".tar.gz.sha512
  5. shasum -c apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl.sha512
  6. cd ../

Check gpg Signature

First, import releaser’s public key. Import KEYS from SVN repository to local. (The releaser does not need to import again; the checking assistant needs to import it, with the user name filled as the releaser’s. )

  1. curl https://dist.apache.org/repos/dist/dev/dolphinscheduler/KEYS >> KEYS
  2. gpg --import KEYS
  3. gpg --edit-key "${A_USERNAME}"
  4.   > trust
  6. Please decide how far you trust this user to correctly verify other users' keys
  7. (by looking at passports, checking fingerprints from different sources, etc.)
  9.   1 = I don't know or won't say
  10.   2 = I do NOT trust
  11.   3 = I trust marginally
  12.   4 = I trust fully
  13.   5 = I trust ultimately
  14.   m = back to the main menu
  16. Your decision? 5
  18.   > save

Then, check the gpg signature.

  1. gpg --verify apache-dolphinscheduler-"${VERSION}"-src.tar.gz.asc
  2. gpg --verify apache-dolphinscheduler-"${VERSION}"-bin.tar.gz.asc
  3. cd python
  4. gpg --verify apache-dolphinscheduler-python-"${VERSION}".tar.gz.asc
  5. gpg --verify apache_dolphinscheduler-python-"${VERSION}"-py3-none-any.whl.asc
  6. cd ../

Note: You have to create gpg signature manually when you can not find your asc file, the command gpg --armor --detach-sign --digest-algo=SHA512 apache-dolphinscheduler-"${VERSION}"-bin.tar.gz and gpg --armor --detach-sign --digest-algo=SHA512 apache-dolphinscheduler-"${VERSION}"-src.tar.gz will create them

Check Released Files

Check source package

Decompress apache-dolphinscheduler-<VERSION>-src.tar.gz and python/apache-dolphinscheduler-python-<VERSION>.tar.gz then check the following items:

  • Check whether source tarball is oversized for including nonessential files
  • LICENSE and NOTICE files exist
  • Correct year in NOTICE file
  • There is only text files but no binary files
  • All source files have ASF headers
  • Codes can be compiled and pass the unit tests (mvn install)
  • The contents of the release match with what’s tagged in version control (diff -r a verify_dir tag_dir)
  • Check if there is any extra files or folders, empty folders for example

Check binary packages

Decompress apache-dolphinscheduler-<VERSION>-src.tar.gz and python/apache-dolphinscheduler-python-<VERSION>-bin.tar.gz to check the following items:

  • LICENSE and NOTICE files exist
  • Correct year in NOTICE file
  • Check the third party dependency license:
    • The software have a compatible license
    • All software licenses mentioned in LICENSE
    • All the third party dependency licenses are under licenses folder
    • If it depends on Apache license and has a NOTICE file, that NOTICE file need to be added to NOTICE file of the release

Call for a Vote

Update Release Notes

You should create a release note in GitHub by new release note. It should be done before vote mail because we need the release note in the mail. You could use command git log --pretty="- %s" <PREVIOUS-RELEASE-SHA>..<CURRENT-RELEASE-SHA> > changelog.md to creat the changelog(some log maybe not correct, you should filter them by yourself) and classify them and paste them to GitHub release note page

Vote procedure

  1. DolphinScheduler community vote: send the vote e-mail to dev@dolphinscheduler.apache.org. PMC needs to check the rightness of the version according to the document before they vote. After at least 72 hours and with at least 3 +1 and no -1 PMC member votes, it can come to the next stage of the vote.

  2. Announce the vote result: send the result vote e-mail to dev@dolphinscheduler.apache.org。

Vote Templates

DolphinScheduler Community Vote Template

Title：

  1. [VOTE] Release Apache DolphinScheduler <VERSION>

Body：

  1. Hello DolphinScheduler Community,
  3. This is a call for vote to release Apache DolphinScheduler version <VERSION>
  5. Release notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION>
  7. The release candidates: https://dist.apache.org/repos/dist/dev/dolphinscheduler/<VERSION>/
  9. Maven 2 staging repository: https://repository.apache.org/content/repositories/<VERSION>/org/apache/dolphinscheduler/
  11. Git tag for the release: https://github.com/apache/dolphinscheduler/tree/<VERSION>
  13. Release Commit ID: https://github.com/apache/dolphinscheduler/commit/<SHA-VALUE>
  15. Keys to verify the Release Candidate: https://dist.apache.org/repos/dist/dev/dolphinscheduler/KEYS
  17. Look at here for how to verify this release candidate: https://dolphinscheduler.apache.org/en-us/community/release.html
  19. The vote will be open for at least 72 hours or until necessary number of votes are reached.
  21. Please vote accordingly:
  23. [ ] +1 approve
  24. [ ] +0 no opinion
  25. [ ] -1 disapprove with the reason
  27. Checklist for reference:
  29. [ ] Download links are valid.
  30. [ ] Checksums and PGP signatures are valid.
  31. [ ] Source code artifacts have correct names matching the current release.
  32. [ ] LICENSE and NOTICE files are correct for each DolphinScheduler repo.
  33. [ ] All files have license headers if necessary.
  34. [ ] No compiled archives bundled in source archive.
  1. Announce the vote result:

Body：

  1. The vote to release Apache DolphinScheduler <VERSION> has passed.Here is the vote result,
  3. 4 PMC member +1 votes:
  5. xxx
  6. xxx
  7. xxx
  8. xxx
  10. 1 community +1 vote:
  11. xxx
  13. Thanks everyone for taking time to check this release and help us.

Finish the Release

Move source packages, binary packages from the dev directory to release directory

  1. svn mv https://dist.apache.org/repos/dist/dev/dolphinscheduler/"${VERSION}" https://dist.apache.org/repos/dist/release/dolphinscheduler/

Export you new gpg KEYS from dev to release(optional)

Only if the first time you release with this gpg KEY, including it is you first release or you change your KEY

  1. mkdir -p ~/ds_svn/release/
  2. cd ~/ds_svn/release/
  3. svn --username="${A_USERNAME}" co https://dist.apache.org/repos/dist/release/dolphinscheduler
  4. gpg -a --export <YOUR-GPG-KEY-ID> >> KEYS
  5. svn add *
  6. svn --username="${A_USERNAME}" commit -m "new key <YOUR-GPG-KEY-ID> add"

Update Document

Website should be present before you send the announce mail this section will tell you how to change the website. For example, the release version is <VERSION>, the following updates are required(note it will take effect immediately when the PR is merged):

  • Repository apache/dolphinscheduler-website:
    • download/en-us/download.md and download/zh-cn/download.md: add the download of the <VERSION> release package
    • scripts/conf.sh: Add new release version <VERSION> key-value pair to variable DEV_RELEASE_DOCS_VERSIONS
  • Repository apache/dolphinscheduler:
    • docs/configs/site.js:
      • docsLatest: update to <VERSION>
      • docs0: The text of two places of en-us/zh-cn needs to be updated to latest(<VERSION>)
      • docsxyz: Add a drop-down menu with key as docsxyz and text as <VERSION> in children of two places of en-us/zh-cn
    • docs/configs/index.md.jsx: Add <VERSION>: docsxyzConfig
    • docs/docs/en/history-versions.md and docs/docs/zh/history-versions.md: Add new <VERSION> release docs.
    • .github/ISSUE_TEMPLATE/bug-report.yml: DolphinScheduler’s GitHub bug-report issue template have Version selection bottom. So after we release DolphinScheduler we should and the new <VERSION> to bug-report.yml

Find DolphinScheduler in apache staging repositories and click Release

Send Announcement E-mail Community

You should send announcement E-mail after release process finished. The E-mail should send to dev@dolphinscheduler.apache.org and cc to announce@apache.org.

Announcement e-mail template as below：

Title：

  1. [ANNOUNCE] Release Apache DolphinScheduler <VERSION>

Body：

  1. Hi all,
  3. We are glad to announce the release of Apache DolphinScheduler <VERSION>. Once again I would like to express my thanks to your help.
  5. Dolphin Scheduler is a distributed and easy-to-extend visual workflow scheduler system,
  6. dedicated to solving the complex task dependencies in data processing, making the scheduler system out of the box for data processing.
  9. Download Links: https://dolphinscheduler.apache.org/en-us/download/download.html
  11. Release Notes: https://github.com/apache/dolphinscheduler/releases/tag/<VERSION>
  13. Website: https://dolphinscheduler.apache.org/
  15. DolphinScheduler Resources:
  16. - Issue: https://github.com/apache/dolphinscheduler/issues/
  17. - Mailing list: dev@dolphinscheduler.apache.org
  18. - Documents: https://dolphinscheduler.apache.org/zh-cn/docs/<VERSION>/user_doc/about/introduction.html