HOWTO

Here’s some miscellaneous documentation about using Calcite and its variousadapters.

Building from a source distribution

Prerequisite is Java (JDK 8, 9, 10, 11, or 12) on your path.

Unpack the source distribution .tar.gz file,cd to the root directory of the unpacked source,then build using the included maven wrapper:

  1. $ tar xvfz calcite-1.21.0-source.tar.gz
  2. $ cd calcite-1.21.0
  3. $ ./mvnw install

Running tests describes how to run more or fewertests.

Building from git

Prerequisites are gitand Java (JDK 8, 9, 10, 11, or 12) on your path.

Create a local copy of the github repository,cd to its root directory,then build using the included maven wrapper:

  1. $ git clone git://github.com/apache/calcite.git
  2. $ cd calcite
  3. $ ./mvnw install

Calcite includes a number of machine-generated codes. By default, these areregenerated on every build, but this has the negative side-effect of causinga re-compilation of the entire project when the non-machine-generated codehas not changed. To make sure incremental compilation still works as intended,provide the skipGenerate command line option with your maven command.If you invoke the clean lifecycle phase, you must not specify theskipGenerate option as it will not recompile the necessary code for the buildto succeed.

  1. $ mvn clean
  2. $ mvn package
  3. ... hacks ...
  4. $ mvn package -DskipGenerate

Running tests describes how to run more or fewertests.

If you already have Apache Maven

If you have already installed Maven and it is on your path, then youcan use mvn rather than ./mvnw in commands. You need Maven version3.5.2 or later.

Running tests

The test suite will run by default when you build, unless you specify-DskipTests:

  1. $ ./mvnw -DskipTests clean install
  2. $ ./mvnw test

There are other options that control which tests are run, and in whatenvironment, as follows.

  • -Dcalcite.test.db=DB (where db is h2, hsqldb, mysql, or postgresql) allows youto change the JDBC data source for the test suite. Calcite’s testsuite requires a JDBC data source populated with the foodmart dataset.
    • hsqldb, the default, uses an in-memory hsqldb database.
    • All others access a test virtual machine(see integration tests below).mysql and postgresql might be somewhat faster than hsqldb, but you needto populate it (i.e. provision a VM).
  • -Dcalcite.debug prints extra debugging information to stdout.
  • -Dcalcite.test.slow enables tests that take longer to execute. Forexample, there are tests that create virtual TPC-H and TPC-DS schemasin-memory and run tests from those benchmarks.
  • -Dcalcite.test.splunk enables tests that run against Splunk.Splunk must be installed and running.

Running integration tests

For testing Calcite’s external adapters, a test virtual machine should be used.The VM includes Cassandra, Druid, H2, HSQLDB, MySQL, MongoDB, and PostgreSQL.

Test VM requires 5GiB of disk space and it takes 30 minutes to build.

Note: you can use calcite-test-dataset to populate your own database, however it is recommended to use test VM so the test environment can be reproduced.

VM preparation

0) Install dependencies: Vagrant and VirtualBox

1) Clone https://github.com/vlsi/calcite-test-dataset.git at the same level as calcite repository.For instance:

  1. code
  2.   +-- calcite
  3.   +-- calcite-test-dataset

Note: integration tests search for ../calcite-test-dataset or ../../calcite-test-dataset. You can specify full path via calcite.test.dataset system property.

2) Build and start the VM:

  1. cd calcite-test-dataset && mvn install

VM management

Test VM is provisioned by Vagrant, so regular Vagrant vagrant up and vagrant halt should be used to start and stop the VM.The connection strings for different databases are listed in calcite-test-dataset readme.

Suggested test flow

Note: test VM should be started before you launch integration tests. Calcite itself does not start/stop the VM.

Command line:

  • Executing regular unit tests (does not require external data): no change. mvn test or mvn install.
  • Executing all tests, for all the DBs: mvn verify -Pit. it stands for “integration-test”. mvn install -Pit works as well.
  • Executing just tests for external DBs, excluding unit tests: mvn -Dtest=foo -DfailIfNoTests=false -Pit verify
  • Executing just MongoDB tests: cd mongo; mvn verify -Pit

From within IDE:

  • Executing regular unit tests: no change.
  • Executing MongoDB tests: run MongoAdapterIT.java as usual (no additional properties are required)
  • Executing MySQL tests: run JdbcTest and JdbcAdapterTest with setting -Dcalcite.test.db=mysql
  • Executing PostgreSQL tests: run JdbcTest and JdbcAdapterTest with setting -Dcalcite.test.db=postgresql

Integration tests technical details

Tests with external data are executed at maven’s integration-test phase.We do not currently use pre-integration-test/post-integration-test, however we could use that in future.The verification of build pass/failure is performed at verify phase.Integration tests should be named …IT.java, so they are not picked up on unit test execution.

Contributing

See the developers guide.

Getting started

See the developers guide.

Setting up an IDE for contributing

Setting up IntelliJ IDEA

To setup IntelliJ IDEA, follow the standard steps for the installation of IDEA and set up one of the JDK versions currently supported by Calcite.

Start with building Calcite from the command line.

Go to File > Open… and open up Calcite’s pom.xml file.When IntelliJ asks if you want to open it as a project or a file, select project.Also, say yes when it asks if you want a new window.IntelliJ’s Maven project importer should handle the rest.

There is a partially implemented IntelliJ code style configuration that you can import located on GitHub.It does not do everything needed to make Calcite’s style checker happy, butit does a decent amount of it.To import, go to Preferences > Editor > Code Style, click the gear next to “scheme”,then Import Scheme > IntelliJ IDEA Code Style XML.

Once the importer is finished, test the project setup.For example, navigate to the method JdbcTest.testWinAgg withNavigate > Symbol and enter testWinAgg. Run testWinAgg by right-clicking and selecting Run (or the equivalent keyboard shortcut).

If you encounter an error while running the JdbcTest.testWinAgg , run the following Maven command from the command line:

$ ./mvnw -DskipTests clean install

You should see "BUILD SUCCESS".

Once that is complete, proceed with running JdbcTest.testWinAgg.

Setting up NetBeans

From the main menu, select File > Open Project and navigate to a name of the project (Calcite) with a small Maven icon, and choose to open.(See this tutorial for an example of how to open a Maven project)Wait for NetBeans to finish importing all dependencies.

To ensure that the project is configured successfully, navigate to the method testWinAgg in org.apache.calcite.test.JdbcTest.Right-click on the method and select to Run Focused Test Method.NetBeans will run a Maven process, and you should see in the command output window a line with Running org.apache.calcite.test.JdbcTest followed by "BUILD SUCCESS".

Tracing

To enable tracing, add the following flags to the java command line:

-Dcalcite.debug=true

The first flag causes Calcite to print the Java code it generates(to execute queries) to stdout. It is especially useful if you are debuggingmysterious problems like this:

Exception in thread "main" java.lang.ClassCastException: Integer cannot be cast to Long at Baz$1$1.current(Unknown Source)

By default, Calcite uses the Log4j bindings for SLF4J. There is a provided configurationfile which outputs logging at the INFO level to the console in core/src/test/resources/log4j.properties.You can modify the level for the rootLogger to increase verbosity or change the levelfor a specific class if you so choose.

  1. # Change rootLogger level to WARN
  2. log4j.rootLogger=WARN, A1
  4. # Increase level to DEBUG for RelOptPlanner
  5. log4j.logger.org.apache.calcite.plan.RelOptPlanner=DEBUG
  7. # Increase level to TRACE for HepPlanner
  8. log4j.logger.org.apache.calcite.plan.hep.HepPlanner=TRACE

Debugging generated classes in Intellij

Calcite uses Janino to generate Javacode. The generated classes can be debugged interactively(see the Janino tutorial).

To debug generated classes, set two system properties when starting the JVM:

  • -Dorg.codehaus.janino.source_debugging.enable=true
  • -Dorg.codehaus.janino.source_debugging.dir=C:\tmp (This property is optional;if not set, Janino will create temporary files in the system’s default locationfor temporary files, such as /tmp on Unix-based systems.)

After code is generated, either go into Intellij and mark the folder thatcontains generated temporary files as a generated sources root or sources root,or directly set the value of org.codehaus.janino.source_debugging.dir to anexisting source root when starting the JVM.

CSV adapter

See the tutorial.

MongoDB adapter

First, download and install Calcite,and install MongoDB.

Note: you can use MongoDB from integration test virtual machine above.

Import MongoDB’s zipcode data set into MongoDB:

  1. $ curl -o /tmp/zips.json https://media.mongodb.org/zips.json
  2. $ mongoimport --db test --collection zips --file /tmp/zips.json
  3. Tue Jun  4 16:24:14.190 check 9 29470
  4. Tue Jun  4 16:24:14.469 imported 29470 objects

Log into MongoDB to check it’s there:

  1. $ mongo
  2. MongoDB shell version: 2.4.3
  3. connecting to: test
  4. > db.zips.find().limit(3)
  5. { "city" : "ACMAR", "loc" : [ -86.51557, 33.584132 ], "pop" : 6055, "state" : "AL", "_id" : "35004" }
  6. { "city" : "ADAMSVILLE", "loc" : [ -86.959727, 33.588437 ], "pop" : 10616, "state" : "AL", "_id" : "35005" }
  7. { "city" : "ADGER", "loc" : [ -87.167455, 33.434277 ], "pop" : 3205, "state" : "AL", "_id" : "35006" }
  8. > exit
  9. bye

Connect using themongo-model.jsonCalcite model:

  1. $ ./sqlline
  2. sqlline> !connect jdbc:calcite:model=mongodb/target/test-classes/mongo-model.json admin admin
  3. Connecting to jdbc:calcite:model=mongodb/target/test-classes/mongo-model.json
  4. Connected to: Calcite (version 1.x.x)
  5. Driver: Calcite JDBC Driver (version 1.x.x)
  6. Autocommit status: true
  7. Transaction isolation: TRANSACTION_REPEATABLE_READ
  8. sqlline> !tables
  9. +------------+--------------+-----------------+---------------+
  10. | TABLE_CAT  | TABLE_SCHEM  |   TABLE_NAME    |  TABLE_TYPE   |
  11. +------------+--------------+-----------------+---------------+
  12. | null       | mongo_raw    | zips            | TABLE         |
  13. | null       | mongo_raw    | system.indexes  | TABLE         |
  14. | null       | mongo        | ZIPS            | VIEW          |
  15. | null       | metadata     | COLUMNS         | SYSTEM_TABLE  |
  16. | null       | metadata     | TABLES          | SYSTEM_TABLE  |
  17. +------------+--------------+-----------------+---------------+
  18. sqlline> select count(*) from zips;
  19. +---------+
  20. | EXPR$0  |
  21. +---------+
  22. | 29467   |
  23. +---------+
  24. 1 row selected (0.746 seconds)
  25. sqlline> !quit
  26. Closing: org.apache.calcite.jdbc.FactoryJdbc41$CalciteConnectionJdbc41
  27. $

Splunk adapter

To run the test suite and sample queries against Splunk,load Splunk’s tutorialdata.zip data set as described inthe Splunk tutorial.

(This step is optional, but it provides some interesting data for the samplequeries. It is also necessary if you intend to run the test suite, using-Dcalcite.test.splunk=true.)

Implementing an adapter

New adapters can be created by implementing CalcitePrepare.Context:

  1. import org.apache.calcite.adapter.java.JavaTypeFactory;
  2. import org.apache.calcite.jdbc.CalcitePrepare;
  3. import org.apache.calcite.jdbc.CalciteSchema;
  5. public class AdapterContext implements CalcitePrepare.Context {
  6.   @Override
  7.   public JavaTypeFactory getTypeFactory() {
  8.     // adapter implementation
  10.     return typeFactory;
  11.   }
  13.   @Override
  14.   public CalciteSchema getRootSchema() {
  15.     // adapter implementation
  17.     return rootSchema;
  18.   }
  19. }

Testing adapter in Java

The example below shows how SQL query can be submitted toCalcitePrepare with a custom context (AdapterContext in thiscase). Calcite prepares and implements the query execution, using theresources provided by the Context. CalcitePrepare.PrepareResultprovides access to the underlying enumerable and methods forenumeration. The enumerable itself can naturally be some adapterspecific implementation.

  1. import org.apache.calcite.jdbc.CalcitePrepare;
  2. import org.apache.calcite.prepare.CalcitePrepareImpl;
  3. import org.junit.Test;
  5. public class AdapterContextTest {
  6.   @Test
  7.   public void testSelectAllFromTable() {
  8.     AdapterContext ctx = new AdapterContext();
  9.     String sql = "SELECT * FROM TABLENAME";
  10.     Class elementType = Object[].class;
  11.     CalcitePrepare.PrepareResult<Object> prepared =
  12.         new CalcitePrepareImpl().prepareSql(ctx, sql, null, elementType, -1);
  13.     Object enumerable = prepared.getExecutable();
  14.     // etc.
  16.   }
  17. }

Advanced topics for developers

The following sections might be of interest if you are adding featuresto particular parts of the code base. You don’t need to understandthese topics if you are just building from source and running tests.

JavaTypeFactory

When Calcite compares types (instances of RelDataType), it requires them to be the sameobject. If there are two distinct type instances that refer to thesame Java type, Calcite may fail to recognize that they match. It isrecommended to:

  • Use a single instance of JavaTypeFactory within the calcite context;
  • Store the types so that the same object is always returned for the same type.

Rebuilding generated Protocol Buffer code

Calcite’s Avatica Server component supports RPC serializationusing Protocol Buffers.In the context of Avatica, Protocol Buffers cangenerate a collection of messages defined by a schema. The libraryitself can parse old serialized messages using anew schema. This is highly desirable in an environment where theclient and server are not guaranteed to have the same version ofobjects.

Typically, the code generated by the Protocol Buffers library doesn’tneed to be re-generated only every build, only when the schema changes.

First, install Protobuf 3.0:

  1. $ wget https://github.com/google/protobuf/releases/download/v3.0.0-beta-1/protobuf-java-3.0.0-beta-1.tar.gz
  2. $ tar xf protobuf-java-3.0.0-beta-1.tar.gz && cd protobuf-3.0.0-beta-1
  3. $ ./configure
  4. $ make
  5. $ sudo make install

Then, re-generate the compiled code:

  1. $ cd avatica/core
  2. $ ./src/main/scripts/generate-protobuf.sh

Advanced topics for committers

The following sections are of interest to Calcite committers and inparticular release managers.

Merging pull requests (for Calcite committers)

These are instructions for a Calcite committer who has reviewed a pull requestfrom a contributor, found it satisfactory, and is about to merge it to master.Usually the contributor is not a committer (otherwise they would be committingit themselves, after you gave approval in a review).

If the PR has multiple commits, squash them into a single commit. Thecommit message should follow the conventions outined incontribution guidelines.If there are conflicts it is better to ask the contributor to take this step,otherwise it is preferred to do this manually since it saves time and alsoavoids unnecessary notification messages to many people on GitHub.

If the contributor is not a committer, add their name in parentheses at the endof the first line of the commit message.

If the merge is performed via command line (not through the GitHub webinterface), make sure the message contains a line “Close apache/calcite#YYY”,where YYY is the GitHub pull request identifier.

When the PR has been merged and pushed, be sure to update the JIRA case. Youmust:

  • resolve the issue (do not close it as this will be done by the releasemanager);
  • select “Fixed” as resolution cause;
  • mark the appropriate version (e.g., 1.21.0) in the “Fix version” field;
  • add a comment (e.g., “Fixed in …”) with a hyperlink pointing to the commitwhich resolves the issue (in GitHub or GitBox), and also thank the contributorfor their contribution.

Set up PGP signing keys (for Calcite committers)

Follow instructions here tocreate a key pair. (On macOS, I did brew install gpg andgpg —gen-key.)

Add your public key to theKEYSfile by following instructions in the KEYS file.(The KEYS file is not present in the git repo or in a release tarball because that would beredundant.)

Set up Maven repository credentials (for Calcite committers)

Follow the instructions here to add your credentials to your maven configuration.

Making a snapshot (for Calcite committers)

Before you start:

  • Set up signing keys as described above.
  • Make sure you are using JDK 8.
  • Make sure build and tests succeed with -Dcalcite.test.db=hsqldb (the default)
  1. # Tell GPG how to read a password from your terminal
  3. export GPG_TTY=$(tty)
  5. # Make sure that there are no junk files in the sandbox
  7. git clean -xn
  8. ./mvnw clean
  10. ./mvnw -Papache-release install

When the dry-run has succeeded, change install to deploy.

Making a release (for Calcite committers)

Before you start:

  • Set up signing keys as described above.
  • Make sure you are using JDK 8 (not 9 or 10).
  • Check that README and site/_docs/howto.md have the correct version number.
  • Check that NOTICE has the current copyright year.
  • Set version.major and version.minor in pom.xml.
  • Make sure build and tests succeed, including with -P it,it-oracle.
  • Make sure that ./mvnw javadoc:javadoc javadoc:test-javadoc succeeds(i.e. gives no errors; warnings are OK)
  • Generate a report of vulnerabilities that occur among dependencies,using -Ppedantic; if you like, run again with -DfailBuildOnCVSS=8 to seewhether serious vulnerabilities exist. Report to private@calcite.apache.orgif new critical vulnerabilities are found among dependencies.
  • Make sure that ./mvnw apache-rat:check succeeds. (It will be run as part ofthe release, but it’s better to trouble-shoot early.)
  • Decide the supported configurations of JDK, operating system andGuava. These will probably be the same as those described in therelease notes of the previous release. Document them in the releasenotes. To test Guava version x.y, specify -Dguava.version=x.y
  • Optional extra tests:
    • -Dcalcite.test.db=mysql
    • -Dcalcite.test.db=hsqldb
    • -Dcalcite.test.slow
    • -Dcalcite.test.mongodb
    • -Dcalcite.test.splunk
  • Trigger aCoverity scanby merging the latest code into the julianhyde/coverity_scan branch,and when it completes, make sure that there are no important issues.
  • Add release notes to site/_docs/history.md. Include the commit history,and say which versions of Java, Guava and operating systems the release istested against.
  • Make sure thatevery “resolved” JIRA case (including duplicates) hasa fix version assigned (most likely the version we arejust about to release)

Smoke-test sqlline with Spatial and Oracle function tables:

  1. $ ./sqlline
  2. > !connect jdbc:calcite:fun=spatial,oracle "sa" ""
  3. SELECT NVL(ST_Is3D(ST_PointFromText('POINT(-71.064544 42.28787)')), TRUE);
  4. +--------+
  6. | EXPR$0 |
  7. +--------+
  9. | false  |
  10. +--------+
  12. 1 row selected (0.039 seconds)
  13. > !quit

Create a release branch named after the release, e.g. branch-1.1, and push it to Apache.

  1. $ git checkout -b branch-X.Y
  2. $ git push -u origin branch-X.Y

We will use the branch for the entire the release process. Meanwhile,we do not allow commits to the master branch. After the release isfinal, we can use git merge —ff-only to append the changes on therelease branch onto the master branch. (Apache does not allow revertsto the master branch, which makes it difficult to clean up the kind ofmessy commits that inevitably happen while you are trying to finalizea release.)

Now, set up your environment and do a dry run. The dry run will notcommit any changes back to git and gives you the opportunity to verifythat the release process will complete as expected.

If any of the steps fail, clean up (see below), fix the problem, andstart again from the top.

  1. # Tell GPG how to read a password from your terminal
  3. export GPG_TTY=$(tty)
  5. # Make sure that there are no junk files in the sandbox
  7. git clean -xn
  8. ./mvnw clean
  10. # Do a dry run of the release:prepare step, which sets version numbers
  12. # (accept the default tag name of calcite-X.Y.Z).
  14. # Note X.Y.Z is the current version we're trying to release (e.g. 1.8.0),
  16. # and X.(Y+1).Z is the next development version (e.g. 1.9.0).
  18. ./mvnw -DdryRun=true -DskipTests -DreleaseVersion=X.Y.Z -DdevelopmentVersion=X.(Y+1).Z-SNAPSHOT -Papache-release -Darguments=-DskipTests release:prepare 2>&1 | tee /tmp/prepare-dry.log
  20. # If you have multiple GPG keys, you can select the key used to sign the release by adding `-Dgpg.keyname=${GPG_KEY_ID}` to `-Darguments`:
  22. ./mvnw -DdryRun=true -DskipTests -DreleaseVersion=X.Y.Z -DdevelopmentVersion=X.(Y+1).Z-SNAPSHOT -Papache-release -Darguments="-DskipTests -Dgpg.keyname=${GPG_KEY_ID}" release:prepare 2>&1 | tee /tmp/prepare-dry.log

Check the artifacts.Note that when performing the dry run SNAPSHOT will appear in any file or directory names given below.The version will be automatically changed when performing the release for real.

  • In the target directory should be these 3 files, among others:
    • apache-calcite-X.Y.Z-src.tar.gz
    • apache-calcite-X.Y.Z-src.tar.gz.asc
    • apache-calcite-X.Y.Z-src.tar.gz.sha256
  • Note that the file names start apache-calcite-.
  • In the source distro .tar.gz (currently there isno binary distro), check that all files belong to a directory calledapache-calcite-X.Y.Z-src.
  • That directory must contain files NOTICE, LICENSE,README, README.md
    • Check that the version in README is correct
    • Check that the copyright year in NOTICE is correct
  • Make sure that there is no KEYS file in the source distros
  • In each .jar (for examplecore/target/calcite-core-X.Y.Z.jar andmongodb/target/calcite-mongodb-X.Y.Z-sources.jar), checkthat the META-INF directory contains DEPENDENCIES, LICENSE,NOTICE and git.properties
  • In core/target/calcite-core-X.Y.Z.jar,check that org-apache-calcite-jdbc.properties ispresent and does not contain un-substituted ${…} variables
  • Check PGP, per this

Now, remove the -DdryRun flag and run the release for real.For this step you’ll have to add the Apache servers to ~/.m2/settings.xml.

  1. # Make sure that there are no junk files in the sandbox; performing a dry run may have generated
  3. # redundant files that do not need to be present in the release artifacts.
  5. git clean -xn
  6. ./mvnw clean
  8. # Prepare sets the version numbers, creates a tag, and pushes it to git
  10. # Note X.Y.Z is the current version we're trying to release, and X.Y+1.Z is the next development version.
  12. # For example, if I am currently building a release for 1.16.0, X.Y.Z would be 1.16.0 and X.Y+1.Z would be 1.17.0.
  14. ./mvnw -DdryRun=false -DskipTests -DreleaseVersion=X.Y.Z -DdevelopmentVersion=X.Y+1.Z-SNAPSHOT -Papache-release -Darguments=-DskipTests release:prepare 2>&1 | tee /tmp/prepare.log
  16. # If you have multiple GPG keys, you can select the key used to sign the release by adding `-Dgpg.keyname=${GPG_KEY_ID}` to `-Darguments`:
  18. ./mvnw -DdryRun=false -DskipTests -DreleaseVersion=X.Y.Z -DdevelopmentVersion=X.Y+1.Z-SNAPSHOT -Papache-release -Darguments="-DskipTests -Dgpg.keyname=${GPG_KEY_ID}" release:prepare 2>&1 | tee /tmp/prepare.log
  20. # Perform checks out the tagged version, builds, and deploys to the staging repository
  22. ./mvnw -DskipTests -Papache-release release:perform 2>&1 | tee /tmp/perform.log

Verify the staged artifacts in the Nexus repository:

Upload the artifacts via subversion to a staging area,https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-X.Y.Z-rcN:

  1. # Create a subversion workspace, if you haven't already
  3. mkdir -p ~/dist/dev
  4. pushd ~/dist/dev
  5. svn co https://dist.apache.org/repos/dist/dev/calcite
  6. popd
  8. # Move the files into a directory
  10. mkdir ~/dist/dev/calcite/apache-calcite-X.Y.Z-rcN
  11. mv apache-calcite-* ~/dist/dev/calcite/apache-calcite-X.Y.Z-rcN
  13. # Check in
  15. cd ~/dist/dev/calcite
  16. svn add apache-calcite-X.Y.Z-rcN
  17. svn ci

Cleaning up after a failed release attempt (for Calcite committers)

  1. # Make sure that the tag you are about to generate does not already
  3. # exist (due to a failed release attempt)
  5. git tag
  7. # If the tag exists, delete it locally and remotely
  9. git tag -d calcite-X.Y.Z
  10. git push origin :refs/tags/calcite-X.Y.Z
  12. # Remove modified files
  14. ./mvnw release:clean
  16. # Check whether there are modified files and if so, go back to the
  18. # original git commit
  20. git status
  21. git reset --hard HEAD

Validate a release

  1. # Check that the signing key (e.g. DDB6E9812AD3FAE3) is pushed
  3. gpg --recv-keys key
  5. # Check keys
  7. curl -O https://dist.apache.org/repos/dist/release/calcite/KEYS
  9. # Sign/check sha256 hashes
  11. # (Assumes your O/S has a 'shasum' command.)
  13. function checkHash() {
  14.   cd "$1"
  15.   for i in *.{pom,gz}; do
  16.     if [ ! -f $i ]; then
  17.       continue
  18.     fi
  19.     if [ -f $i.sha256 ]; then
  20.       if [ "$(cat $i.sha256)" = "$(shasum -a 256 $i)" ]; then
  21.         echo $i.sha256 present and correct
  22.       else
  23.         echo $i.sha256 does not match
  24.       fi
  25.     else
  26.       shasum -a 256 $i > $i.sha256
  27.       echo $i.sha256 created
  28.     fi
  29.   done
  30. }
  31. checkHash apache-calcite-X.Y.Z-rcN

Get approval for a release via Apache voting process (for Calcite committers)

Release vote on dev list

  1. To: dev@calcite.apache.org
  2. Subject: [VOTE] Release apache-calcite-X.Y.Z (release candidate N)
  4. Hi all,
  6. I have created a build for Apache Calcite X.Y.Z, release candidate N.
  8. Thanks to everyone who has contributed to this release.
  9. <Further details about release.> You can read the release notes here:
  10. https://github.com/apache/calcite/blob/XXXX/site/_docs/history.md
  12. The commit to be voted upon:
  13. https://gitbox.apache.org/repos/asf?p=calcite.git;a=commit;h=NNNNNN
  15. Its hash is XXXX.
  17. The artifacts to be voted on are located here:
  18. https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-X.Y.Z-rcN/
  20. The hashes of the artifacts are as follows:
  21. src.tar.gz.sha256 XXXX
  23. A staged Maven repository is available for review at:
  24. https://repository.apache.org/content/repositories/orgapachecalcite-NNNN
  26. Release artifacts are signed with the following key:
  27. https://people.apache.org/keys/committer/jhyde.asc
  29. Please vote on releasing this package as Apache Calcite X.Y.Z.
  31. The vote is open for the next 72 hours and passes if a majority of
  32. at least three +1 PMC votes are cast.
  34. [ ] +1 Release this package as Apache Calcite X.Y.Z
  35. [ ]  0 I don't feel strongly about it, but I'm okay with the release
  36. [ ] -1 Do not release this package because...
  39. Here is my vote:
  41. +1 (binding)
  43. Julian

After vote finishes, send out the result:

  1. Subject: [RESULT] [VOTE] Release apache-calcite-X.Y.Z (release candidate N)
  2. To: dev@calcite.apache.org
  4. Thanks to everyone who has tested the release candidate and given
  5. their comments and votes.
  7. The tally is as follows.
  9. N binding +1s:
  10. <names>
  12. N non-binding +1s:
  13. <names>
  15. No 0s or -1s.
  17. Therefore I am delighted to announce that the proposal to release
  18. Apache Calcite X.Y.Z has passed.
  20. Thanks everyone. We’ll now roll the release out to the mirrors.
  22. There was some feedback during voting. I shall open a separate
  23. thread to discuss.
  26. Julian

Use the Apache URL shortener to generateshortened URLs for the vote proposal and result emails. Examples:s.apache.org/calcite-1.2-vote ands.apache.org/calcite-1.2-result.

Publishing a release (for Calcite committers)

After a successful release vote, we need to push the releaseout to mirrors, and other tasks.

Choose a release date.This is based on the time when you expect to announce the release.This is usually a day after the vote closes.Remember that UTC date changes at 4pm Pacific time.

Promote the staged nexus artifacts.

  • Go to https://repository.apache.org/ and login
  • Under “Build Promotion” click “Staging Repositories”
  • In the line with “orgapachecalcite-xxxx”, check the box
  • Press “Release” button

Check the artifacts into svn.

  1. # Get the release candidate.
  3. mkdir -p ~/dist/dev
  4. cd ~/dist/dev
  5. svn co https://dist.apache.org/repos/dist/dev/calcite
  7. # Copy the artifacts. Note that the copy does not have '-rcN' suffix.
  9. mkdir -p ~/dist/release
  10. cd ~/dist/release
  11. svn co https://dist.apache.org/repos/dist/release/calcite
  12. cd calcite
  13. cp -rp ../../dev/calcite/apache-calcite-X.Y.Z-rcN apache-calcite-X.Y.Z
  14. svn add apache-calcite-X.Y.Z
  16. # Check in.
  18. svn ci

Svnpubsub will publish to therelease repo and propagate to themirrors within 24 hours.

If there are now more than 2 releases, clear out the oldest ones:

  1. cd ~/dist/release/calcite
  2. svn rm apache-calcite-X.Y.Z
  3. svn ci

The old releases will remain available in therelease archive.

You should receive an email from the Apache Reporter Service.Make sure to add the version number and date of the latest release at the site linked to in the email.

Update the site with the release note, the release announcement, and the javadoc of the new version.The javadoc can be generated only from a final version (not a SNAPSHOT) so checkout the most recenttag and start working there (git checkout calcite-X.Y.Z). Add a release announcement by copyingsite/_posts/2016-10-12-release-1.10.0.md.Generate the javadoc, and preview the site by following theinstructions in site/README.md. Check that the announcement,javadoc, and release note appear correctly and then publish the site following the instructionsin the same file. Now checkout again the release branch (git checkout branch-X.Y) and committhe release announcement.

Merge the release branch back into master (e.g., git merge —ff-only branch-X.Y) and alignthe master with the site branch (e.g., git merge —ff-only site).

In JIRA, search forall issues resolved in this release,and do a bulk update changing their status to “Closed”,with a change comment“Resolved in release X.Y.Z (YYYY-MM-DD)”(fill in release number and date appropriately).Uncheck “Send mail for this update”. Under the releases tabof the Calcite project mark the release X.Y.Z as released. If it does not already exist create alsoa new version (e.g., X.Y+1.Z) for the next release.

After 24 hours, announce the release by sending an email toannounce@apache.org.You can usethe 1.20.0 announcementas a template. Be sure to include a brief description of the project.

Publishing the web site (for Calcite committers)

See instructions insite/README.md.