Ruby on Rails

This section investigates how to migrate multi-tenant Rails applications to a Citus storage backend. We’ll use the activerecord-multi-tenant Ruby gem for easier scale-out.

This Ruby gem has evolved from our experience working with customers scaling out their multi-tenant apps. It patches some restrictions that ActiveRecord and Rails currently have when it comes to automatic query building. It is based on the excellent acts_as_tenant library, and extends it for the particular use-case of a distributed multi-tenant database like Citus.

Preparing to scale-out a multi-tenant application

Initially you’ll often start out with all tenants placed on a single database node, and using a framework like Ruby on Rails and ActiveRecord to load the data for a given tenant when you serve a web request that returns the tenant’s data.

ActiveRecord makes a few assumptions about the data storage that limit your scale-out options. In particular, ActiveRecord introduces a pattern where you normalize data and split it into many distinct models each identified by a single id column, with multiple belongs_to relationships that tie objects back to a tenant or customer:

  1. # typical pattern with multiple belongs_to relationships
  3. class Customer < ActiveRecord::Base
  4.   has_many :sites
  5. end
  6. class Site < ActiveRecord::Base
  7.   belongs_to :customer
  8.   has_many :page_views
  9. end
  10. class PageView < ActiveRecord::Base
  11.   belongs_to :site
  12. end

The tricky thing with this pattern is that in order to find all page views for a customer, you’ll have to query for all of a customer’s sites first. This becomes a problem once you start sharding data, and in particular when you run UPDATE or DELETE queries on nested models like page views in this example.

There are a few steps you can take today, to make scaling out easier in the future:

1. Introduce a column for the tenant_id on every record that belongs to a tenant

In order to scale out a multi-tenant model, it’s essential you can locate all records that belong to a tenant quickly. The easiest way to achieve this is to simply add a tenant_id column (or “customer_id” column, etc) on every object that belongs to a tenant, and backfilling your existing data to have this column set correctly.

When you move to a distributed multi-tenant database like Citus in the future, this will be a required step - but if you’ve done this before, you can simply COPY over your data, without doing any additional data modification.

2. Use UNIQUE constraints which include the tenant_id

Unique and foreign-key constraints on values other than the tenant_id will present a problem in any distributed system, since it’s difficult to make sure that no two nodes accept the same unique value. Enforcing the constraint would require expensive scans of the data across all nodes.

To solve this problem, for the models which are logically related to a store (the tenant for our app), you should add store_id to the constraints, effectively scoping objects uniquely inside a given store. This helps add the concept of tenancy to your models, thereby making the multi-tenant system more robust.

For example, Rails creates a primary key by default, that only includes the id of the record:

  1. Indexes:
  2.     "page_views_pkey" PRIMARY KEY, btree (id)

You should modify that primary key to also include the tenant_id:

  1. ALTER TABLE page_views DROP CONSTRAINT page_views_pkey;
  2. ALTER TABLE page_views ADD PRIMARY KEY(id, customer_id);

An exception to this rule might be an email or username column on a users table (unless you give each tenant their own login page), which is why, once you scale out, we typically recommend these to be split out from your distributed tables and placed as a local table on the Citus coordinator node.

3. Include the tenant_id in all queries, even when you can locate an object using its own object_id

The easiest way to run a typical SQL query in a distributed system without restrictions is to always access data that lives on a single node, determined by the tenant you are accessing.

For this reason, once you use a distributed system like Citus, we recommend you always specify both the tenant_id and an object’s own ID for queries, so the coordinator can locate your data quickly, and can route the query to a single shard - instead of going to each shard in the system individually and asking the shard whether it knows the given object_id.

Updating the Rails Application

You can get started by including gem 'activerecord-multi-tenant' into your Gemfile, running bundle install, and then annotating your ActiveRecord models like this:

  1. class PageView < ActiveRecord::Base
  2.   multi_tenant :customer
  3.   # ...
  4. end

In this case customer is the tenant model, and your page_views table needs to have a customer_id column that references the customer the page view belongs to.

The activerecord-multi-tenant Ruby gem aims to make it easier to implement the above data changes in a typical Rails application.

Note

The library relies on the tenant id column to be present and non-null for all rows. However, it is often useful to have the library set the tenant id for new records, while backfilling missing tenant id values in existing records as a background task. This makes it easier to get started with activerecord-multi-tenant.

To support this, the library has a write-only mode, in which the tenant id column is not filtered in queries, but is set properly for new records. Include the following in a Rails initializer to enable it:

  1. MultiTenant.enable_write_only_mode

Once you are ready to enforce tenancy, add a NOT NULL constraint to your tenant_id column and simply remove the initializer line.

As mentioned in the beginning, by adding multi_tenant :customer annotations to your models, the library automatically takes care of including the tenant_id with all queries.

In order for that to work, you’ll always need to specify which tenant you are accessing, either by specifying it on a per-request basis:

  1. class ApplicationController < ActionController::Base
  2.   # Opt-into the "set_current_tenant" controller helpers by specifying this:
  3.   set_current_tenant_through_filter
  5.   before_filter :set_customer_as_tenant
  7.   def set_customer_as_tenant
  8.     customer = Customer.find(session[:current_customer_id])
  9.     set_current_tenant(customer) # Set the tenant
  10.   end
  11. end

Or by wrapping your code in a block, e.g. for background and maintenance tasks:

  1. customer = Customer.find(session[:current_customer_id])
  2. # ...
  3. MultiTenant.with(customer) do
  4.   site = Site.find(params[:site_id])
  6.   # Modifications automatically include tenant_id
  7.   site.update! last_accessed_at: Time.now
  9.   # Queries also include tenant_id automatically
  10.   site.page_views.count
  11. end

Once you are ready to use a distributed multi-tenant database like Citus, all you need is a few adjustments to your migrations, and you’re good to go:

  1. class InitialTables < ActiveRecord::Migration
  2.   def up
  3.     create_table :page_views, partition_key: :customer_id do |t|
  4.       t.references :customer, null: false
  5.       t.references :site, null: false
  7.       t.text :url, null: false
  8.       ...
  9.       t.timestamps null: false
  10.     end
  11.     create_distributed_table :page_views, :account_id
  12.   end
  14.   def down
  15.     drop_table :page_views
  16.   end
  17. end

Note the partition_key: :customer_id, something that’s added to Rails’ create_table by our library, which ensures that the primary key includes the tenant_id column, as well as create_distributed_table which enables Citus to scale out the data to multiple nodes.

Updating the Test Suite

If the test suite for your Rails application uses the database_cleaner gem to reset the test database between runs, be sure to use the “truncation” strategy rather than “transaction.” We have seen occasional failures during transaction rollbacks in the tests. The database_cleaner documentation has instructions for changing the cleaning strategy.

Continuous Integration

The easiest way to run a Citus cluster in continuous integration is by using the official Citus Docker containers. Here is how to do it on Circle CI in particular.

  1. Copy https://github.com/citusdata/docker/blob/master/docker-compose.yml into the Rails project, and name it citus-docker-compose.yml.

  2. Update the steps: section in .circleci/config.yml. This will start a coordinator and worker node:

    1. steps:
    2.   - setup_remote_docker:
    3.       docker_layer_caching: true
    4.   - run:
    5.       name: Install Docker Compose
    6.       command: |
    7.         curl -L https://github.com/docker/compose/releases/download/1.19.0/docker-compose-`uname -s`-`uname -m` > ~/docker-compose
    8.         chmod +x ~/docker-compose
    9.         mv ~/docker-compose /usr/local/bin/docker-compose
    11.   - checkout
    13.   - run:
    14.       name: Starting Citus Cluster
    15.       command: docker-compose -f citus-docker-compose.yml up -d

  3. Have your test suite connect to the database in Docker, which will be on localhost:5432.

Example Application

If you are interested in a more complete example, check out our reference app that showcases a simplified sample SaaS application for ad analytics.

dashboard in example application showing a campaign

As you can see in the screenshot, most data is associated to the currently logged in customer - even though this is complex analytical data, all data is accessed in the context of a single customer or tenant.