4.4 The DataSource

Since Grails is built on Java technology setting up a data source requires some knowledge of JDBC (the technology that stands for Java Database Connectivity).

If you use a database other than H2 you need a JDBC driver. For example for MySQL you would need Connector/J.

Drivers typically come in the form of a JAR archive. It’s best to use the dependency resolution to resolve the jar if it’s available in a Maven repository, for example you could add a dependency for the MySQL driver like this:

dependencies {
    runtime 'mysql:mysql-connector-java:5.1.29'
}

Once you have the JAR resolved you need to get familiar with how Grails manages its database configuration. The configuration can be maintained in either grails-app/conf/application.groovy or grails-app/conf/application.yml. These files contain the dataSource definition which includes the following settings:

• driverClassName - The class name of the JDBC driver

• username - The username used to establish a JDBC connection

• password - The password used to establish a JDBC connection

• url - The JDBC URL of the database

• dbCreate - Whether to auto-generate the database from the domain model - one of 'create-drop', 'create', 'update' or 'validate'

• pooled - Whether to use a pool of connections (defaults to true)

• logSql - Enable SQL logging to stdout

• formatSql - Format logged SQL

• dialect - A String or Class that represents the Hibernate dialect used to communicate with the database. See the org.hibernate.dialect package for available dialects.

• readOnly - If true makes the DataSource read-only, which results in the connection pool calling setReadOnly(true) on each Connection

• transactional - If false leaves the DataSource’s transactionManager bean outside the chained BE1PC transaction manager implementation. This only applies to additional datasources.

• persistenceInterceptor - The default datasource is automatically wired up to the persistence interceptor, other datasources are not wired up automatically unless this is set to true

• properties - Extra properties to set on the DataSource bean. See the Tomcat Pool documentation. There is also a Javadoc format documentation of the properties.

• jmxExport - If false, will disable registration of JMX MBeans for all DataSources. By default JMX MBeans are added for DataSources with jmxEnabled = true in properties.

A typical configuration for MySQL in application.groovy may be something like:

dataSource {
    pooled = true
    dbCreate = "update"
    url = "jdbc:mysql://localhost:3306/my_database"
    driverClassName = "com.mysql.jdbc.Driver"
    dialect = org.hibernate.dialect.MySQL5InnoDBDialect
    username = "username"
    password = "password"
    properties {
       jmxEnabled = true
       initialSize = 5
       maxActive = 50
       minIdle = 5
       maxIdle = 25
       maxWait = 10000
       maxAge = 10 * 60000
       timeBetweenEvictionRunsMillis = 5000
       minEvictableIdleTimeMillis = 60000
       validationQuery = "SELECT 1"
       validationQueryTimeout = 3
       validationInterval = 15000
       testOnBorrow = true
       testWhileIdle = true
       testOnReturn = false
       jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
       defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
    }
}

dataSource {
    boolean pooled = true // type declaration results in ignored local variable
    ...
}

Example of advanced configuration using extra properties:

dataSource {
    pooled = true
    dbCreate = "update"
    url = "jdbc:mysql://localhost:3306/my_database"
    driverClassName = "com.mysql.jdbc.Driver"
    dialect = org.hibernate.dialect.MySQL5InnoDBDialect
    username = "username"
    password = "password"
    properties {
       // Documentation for Tomcat JDBC Pool
       // http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Common_Attributes
       // https://tomcat.apache.org/tomcat-7.0-doc/api/org/apache/tomcat/jdbc/pool/PoolConfiguration.html
       jmxEnabled = true
       initialSize = 5
       maxActive = 50
       minIdle = 5
       maxIdle = 25
       maxWait = 10000
       maxAge = 10 * 60000
       timeBetweenEvictionRunsMillis = 5000
       minEvictableIdleTimeMillis = 60000
       validationQuery = "SELECT 1"
       validationQueryTimeout = 3
       validationInterval = 15000
       testOnBorrow = true
       testWhileIdle = true
       testOnReturn = false
       ignoreExceptionOnPreLoad = true
       // http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#JDBC_interceptors
       jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
       defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED // safe default
       // controls for leaked connections
       abandonWhenPercentageFull = 100 // settings are active only when pool is full
       removeAbandonedTimeout = 120
       removeAbandoned = true
       // use JMX console to change this setting at runtime
       logAbandoned = false // causes stacktrace recording overhead, use only for debugging
       // JDBC driver properties
       // Mysql as example
       dbProperties {
           // Mysql specific driver properties
           // http://dev.mysql.com/doc/connector-j/en/connector-j-reference-configuration-properties.html
           // let Tomcat JDBC Pool handle reconnecting
           autoReconnect=false
           // truncation behaviour
           jdbcCompliantTruncation=false
           // mysql 0-date conversion
           zeroDateTimeBehavior='convertToNull'
           // Tomcat JDBC Pool's StatementCache is used instead, so disable mysql driver's cache
           cachePrepStmts=false
           cacheCallableStmts=false
           // Tomcat JDBC Pool's StatementFinalizer keeps track
           dontTrackOpenResources=true
           // performance optimization: reduce number of SQLExceptions thrown in mysql driver code
           holdResultsOpenOverStatementClose=true
           // enable MySQL query cache - using server prep stmts will disable query caching
           useServerPrepStmts=false
           // metadata caching
           cacheServerConfiguration=true
           cacheResultSetMetadata=true
           metadataCacheSize=100
           // timeouts for TCP/IP
           connectTimeout=15000
           socketTimeout=120000
           // timer tuning (disable)
           maintainTimeStats=false
           enableQueryTimeouts=false
           // misc tuning
           noDatetimeStringSync=true
       }
    }
}

More on dbCreate

Hibernate can automatically create the database tables required for your domain model. You have some control over when and how it does this through the dbCreate property, which can take these values:

• create - Drops the existing schema and creates the schema on startup, dropping existing tables, indexes, etc. first.

• create-drop - Same as create, but also drops the tables when the application shuts down cleanly.

• update - Creates missing tables and indexes, and updates the current schema without dropping any tables or data. Note that this can’t properly handle many schema changes like column renames (you’re left with the old column containing the existing data).

• validate - Makes no changes to your database. Compares the configuration with the existing database schema and reports warnings.

• any other value - does nothing

Setting the dbCreate setting to "none" is recommended once your schema is relatively stable and definitely when your application and database are deployed in production. Database changes are then managed through proper migrations, either with SQL scripts or a migration tool like Flyway or Liquibase. The Database Migration plugin uses Liquibase.

4.4.1 DataSources and Environments

The previous example configuration assumes you want the same config for all environments: production, test, development etc.

Grails' DataSource definition is "environment aware", however, so you can do:

dataSource {
    pooled = true
    driverClassName = "com.mysql.jdbc.Driver"
    dialect = org.hibernate.dialect.MySQL5InnoDBDialect
    // other common settings here
}
environments {
    production {
        dataSource {
            url = "jdbc:mysql://liveip.com/liveDb"
            // other environment-specific settings here
        }
    }
}

4.4.2 Automatic Database Migration

The dbCreate property of the DataSource definition is important as it dictates what Grails should do at runtime with regards to automatically generating the database tables from GORM classes. The options are described in the DataSource section:

• create

• create-drop

• update

• validate

• no value

In development mode dbCreate is by default set to "create-drop", but at some point in development (and certainly once you go to production) you’ll need to stop dropping and re-creating the database every time you start up your server.

It’s tempting to switch to update so you retain existing data and only update the schema when your code changes, but Hibernate’s update support is very conservative. It won’t make any changes that could result in data loss, and doesn’t detect renamed columns or tables, so you’ll be left with the old one and will also have the new one.

Grails supports migrations with Liquibase or Flyway via plugins.

• Database Migration

• Flyway

4.4.3 Transaction-aware DataSource Proxy

The actual dataSource bean is wrapped in a transaction-aware proxy so you will be given the connection that’s being used by the current transaction or Hibernate Session if one is active.

If this were not the case, then retrieving a connection from the dataSource would be a new connection, and you wouldn’t be able to see changes that haven’t been committed yet (assuming you have a sensible transaction isolation setting, e.g. READ_COMMITTED or better).

The "real" unproxied dataSource is still available to you if you need access to it; its bean name is dataSourceUnproxied.

You can access this bean like any other Spring bean, i.e. using dependency injection:

class MyService {
   def dataSourceUnproxied
   ...
}

or by pulling it from the ApplicationContext:

def dataSourceUnproxied = ctx.dataSourceUnproxied

4.4.4 Database Console

The H2 database console is a convenient feature of H2 that provides a web-based interface to any database that you have a JDBC driver for, and it’s very useful to view the database you’re developing against. It’s especially useful when running against an in-memory database.

You can access the console by navigating to http://localhost:8080/dbconsole in a browser. The URI can be configured using the grails.dbconsole.urlRoot attribute in application.groovy and defaults to '/dbconsole'.

The console is enabled by default in development mode and can be disabled or enabled in other environments by using the grails.dbconsole.enabled attribute in application.groovy. For example, you could enable the console in production like this:

environments {
    production {
        grails.serverURL = "http://www.changeme.com"
        grails.dbconsole.enabled = true
        grails.dbconsole.urlRoot = '/admin/dbconsole'
    }
    development {
        grails.serverURL = "http://localhost:8080/${appName}"
    }
    test {
        grails.serverURL = "http://localhost:8080/${appName}"
    }
}

Configuration

By default the console is configured for an H2 database which will work with the default settings if you haven’t configured an external database - you just need to change the JDBC URL to jdbc:h2:mem:devDb. If you’ve configured an external database (e.g. MySQL, Oracle, etc.) then you can use the Saved Settings dropdown to choose a settings template and fill in the url and username/password information from your application.groovy.

4.4.5 Multiple Datasources

By default all domain classes share a single DataSource and a single database, but you have the option to partition your domain classes into two or more data sources.

Configuring Additional DataSources

The default DataSource configuration in grails-app/conf/application.yml looks something like this:

dataSource:
    pooled: true
    jmxExport: true
    driverClassName: org.h2.Driver
    username: sa
    password:
environments:
    development:
        dataSource:
            dbCreate: create-drop
            url: jdbc:h2:mem:devDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
    test:
        dataSource:
            dbCreate: update
            url: jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
    production:
        dataSource:
            dbCreate: update
            url: jdbc:h2:prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
            properties:
               jmxEnabled: true
               initialSize: 5

This configures a single DataSource with the Spring bean named dataSource. To configure extra data sources, add a dataSources block (at the top level, in an environment block, or both, just like the standard DataSource definition) with a custom name. For example, this configuration adds a second DataSource, using MySQL in the development environment and Oracle in production:

dataSource:
    pooled: true
    jmxExport: true
    driverClassName: org.h2.Driver
    username: sa
    password:
dataSources:
    lookup:
        dialect: org.hibernate.dialect.MySQLInnoDBDialect
        driverClassName: com.mysql.jdbc.Driver
        username: lookup
        password: secret
        url: jdbc:mysql://localhost/lookup
        dbCreate: update
environments:
    development:
        dataSource:
            dbCreate: create-drop
            url: jdbc:h2:mem:devDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
    test:
        dataSource:
            dbCreate: update
            url: jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
    production:
        dataSource:
            dbCreate: update
            url: jdbc:h2:prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
            properties:
                jmxEnabled: true
                initialSize: 5
                ...
        dataSources:
            lookup:
                dialect: org.hibernate.dialect.Oracle10gDialect
                driverClassName: oracle.jdbc.driver.OracleDriver
                username: lookup
                password: secret
                url: jdbc:oracle:thin:@localhost:1521:lookup
                dbCreate: update

You can use the same or different databases as long as they’re supported by Hibernate.

If you need to inject the lookup datasource in a Grails artefact, you can do it like this:

DataSource dataSource_lookup

Configuring Domain Classes

If a domain class has no DataSource configuration, it defaults to the standard 'dataSource'. Set the datasource property in the mapping block to configure a non-default DataSource. For example, if you want to use the ZipCode domain to use the 'lookup' DataSource, configure it like this:

class ZipCode {
   String code
   static mapping = {
      datasource 'lookup'
   }
}

A domain class can also use two or more data sources. Use the datasources property with a list of names to configure more than one, for example:

class ZipCode {
   String code
   static mapping = {
      datasources(['lookup', 'auditing'])
   }
}

If a domain class uses the default DataSource and one or more others, use the special name 'DEFAULT' to indicate the default DataSource:

class ZipCode {
   String code
   static mapping = {
      datasources(['lookup', 'DEFAULT'])
   }
}

If a domain class uses all configured data sources, use the special value 'ALL':

class ZipCode {
   String code
   static mapping = {
      datasource 'ALL'
   }
}

Namespaces and GORM Methods

If a domain class uses more than one DataSource then you can use the namespace implied by each DataSource name to make GORM calls for a particular DataSource. For example, consider this class which uses two data sources:

class ZipCode {
   String code
   static mapping = {
      datasources(['lookup', 'auditing'])
   }
}

The first DataSource specified is the default when not using an explicit namespace, so in this case we default to 'lookup'. But you can call GORM methods on the 'auditing' DataSource with the DataSource name, for example:

def zipCode = ZipCode.auditing.get(42)
...
zipCode.auditing.save()

As you can see, you add the DataSource to the method call in both the static case and the instance case.

Hibernate Mapped Domain Classes

You can also partition annotated Java classes into separate datasources. Classes using the default datasource are registered in grails-app/conf/hibernate.cfg.xml. To specify that an annotated class uses a non-default datasource, create a hibernate.cfg.xml file for that datasource with the file name prefixed with the datasource name.

For example if the Book class is in the default datasource, you would register that in grails-app/conf/hibernate.cfg.xml:

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
          '-//Hibernate/Hibernate Configuration DTD 3.0//EN'
          'http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd'>
<hibernate-configuration>
   <session-factory>
      <mapping class='org.example.Book'/>
   </session-factory>
</hibernate-configuration>

and if the Library class is in the "ds2" datasource, you would register that in grails-app/conf/ds2_hibernate.cfg.xml:

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
          '-//Hibernate/Hibernate Configuration DTD 3.0//EN'
          'http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd'>
<hibernate-configuration>
   <session-factory>
      <mapping class='org.example.Library'/>
   </session-factory>
</hibernate-configuration>

The process is the same for classes mapped with hbm.xml files - just list them in the appropriate hibernate.cfg.xml file.

Services

Like Domain classes, by default Services use the default DataSource and PlatformTransactionManager. To configure a Service to use a different DataSource, use the static datasource property, for example:

class DataService {
   static datasource = 'lookup'
   void someMethod(...) {
      ...
   }
}

A transactional service can only use a single DataSource, so be sure to only make changes for domain classes whose DataSource is the same as the Service.

Note that the datasource specified in a service has no bearing on which datasources are used for domain classes; that’s determined by their declared datasources in the domain classes themselves. It’s used to declare which transaction manager to use.

If you have a Foo domain class in dataSource1 and a Bar domain class in dataSource2, if WahooService uses dataSource1, a service method that saves a new Foo and a new Bar will only be transactional for Foo since they share the same datasource. The transaction won’t affect the Bar instance. If you want both to be transactional you’d need to use two services and XA datasources for two-phase commit, e.g. with the Atomikos plugin.

Transactions across multiple data sources

Grails does not by default try to handle transactions that span multiple data sources.

You can enable Grails to use the Best Effort 1PC pattern for handling transactions across multiple datasources. To do so you must set the grails.transaction.chainedTransactionManagerPostProcessor.enabled setting to true in application.yml:

grails:
  transaction:
    chainedTransactionManagerPostProcessor:
      enabled: true

The Best Efforts 1PC pattern is fairly general but can fail in some circumstances that the developer must be aware of.

This is a non-XA pattern that involves a synchronized single-phase commit of a number of resources. Because the 2PC is not used, it can never be as safe as an XA transaction, but is often good enough if the participants are aware of the compromises.

The basic idea is to delay the commit of all resources as late as possible in a transaction so that the only thing that can go wrong is an infrastructure failure (not a business-processing error). Systems that rely on Best Efforts 1PC reason that infrastructure failures are rare enough that they can afford to take the risk in return for higher throughput. If business-processing services are also designed to be idempotent, then little can go wrong in practice.

The BE1PC implementation was added in Grails 2.3.6. . Before this change additional datasources didn’t take part in transactions initiated in Grails. The transactions in additional datasources were basically in auto commit mode. In some cases this might be the wanted behavior. One reason might be performance: on the start of each new transaction, the BE1PC transaction manager creates a new transaction to each datasource. It’s possible to leave an additional datasource out of the BE1PC transaction manager by setting transactional = false in the respective configuration block of the additional dataSource. Datasources with readOnly = true will also be left out of the chained transaction manager (since 2.3.7).

By default, the BE1PC implementation will add all beans implementing the Spring PlatformTransactionManager interface to the chained BE1PC transaction manager. For example, a possible JMSTransactionManager bean in the Grails application context would be added to the Grails BE1PC transaction manager’s chain of transaction managers.

You can exclude transaction manager beans from the BE1PC implementation with this configuration option:

grails:
  transaction:
    chainedTransactionManagerPostProcessor:
      enabled: true
      blacklistPattern: '.*'

The exclude matching is done on the name of the transaction manager bean. The transaction managers of datasources with transactional = false or readOnly = true will be skipped and using this configuration option is not required in that case.

XA and Two-phase Commit

When the Best Efforts 1PC pattern isn’t suitable for handling transactions across multiple transactional resources (not only datasources), there are several options available for adding XA/2PC support to Grails applications.

The Spring transactions documentation contains information about integrating the JTA/XA transaction manager of different application servers. In this case, you can configure a bean with the name transactionManager manually in resources.groovy or resources.xml file.