Diesel provides a lot of capabilities out of the box. However, it doesn’t necessarily provide everything your app may want to use. One of Diesel’s greatest strengths is that it can be extended to add new functionality.

In this guide we’ll look at several ways to hook into Diesel’s query builder, both to add new capabilities, and to introduce new abstractions.

This guide is only going to cover extending the query builder. How to add support for new SQL types will be covered in a future guide.

sql_function!

The easiest and most common way to extend Diesel’s query builder is by declaring a new SQL function. This can be used for functions defined by your database, or for built-in functions that Diesel doesn’t support out of the box.

Functions in SQL often have multiple signatures, making them difficult or impossible to represent in Rust. Because of this, Diesel only provides support for a small number of built-in SQL functions. Consider COALESCE. This function can take any number of arguments, and its return type changes based on whether any arguments are NOT NULL. While we can’t easily represent that in Rust, we can use sql_function! to declare it with the exact signature we’re using.

Example

  1. use diesel::sql_types::{Nullable, Text};
  2. sql_function! { fn coalesce(x: Nullable<Text>, y: Text) -> Text; }
  4. users.select(coalesce(hair_color, "blue"))

As this example shows, sql_function! converts its argument like other parts of the query builder. This means that the generated function can take both Diesel expressions, and Rust values to be sent with the query.

The macro takes one argument: a function definition. However, the types in the function signature are SQL types, not concrete Rust types. This is what allows us to pass both columns and Rust strings. If we defined this function manually, it would look like this:

Example

  1. pub fn coalesce<X, Y>(x: X, y: Y) -> coalesce::HelperType<X, Y>
  2. where
  3. X: AsExpression<Nullable<Text>>,
  4. Y: AsExpression<Text>,
  5. {
  6. ...
  7. }
  9. pub(crate) mod coalesce {
  10. pub type HelperType<X, Y> = ...;
  11. }

A helper type is generated with the same name as the function. This helper type handles Diesel’s argument conversion. This lets us write coalesce<hair_color, &str>. More information can be found in the API documentation

Using Custom SQL and How to Extend the Query DSL

Often times it’s useful to encapsulate a common SQL pattern. For example, if you’re doing pagination on your queries, PostgreSQL is capable of loading the total count in a single query. The query you would want to execute would look like this:

Example

  1. SELECT *, COUNT(*) OVER () FROM (subselect t) as paged_query_with LIMIT $1 OFFSET $2

However, as of version 2.0, Diesel doesn’t support window functions, or selecting from a subselect. Even if Diesel’s query builder supported those things, this is a case that is easier to reason about in terms of the SQL we want to generate.

Let’s look at how we would go about adding a paginate method to Diesel’s query builder, to generate that query. Let’s assume for the time being that we have a struct Paginated<T> already. We’ll look at the specifics of this struct shortly.

If you are creating a struct where you want to manually define the SQL, you will need to implement a trait called QueryFragment. The implementation will look like this:

src/pagination.rs

  1. impl<T> QueryFragment<Pg> for Paginated<T>
  2. where
  3. T: QueryFragment<Pg>,
  4. {
  5. fn walk_ast<'b>(&'b self, mut out: AstPass<'_, 'b, Pg>) -> QueryResult<()> {
  6. out.push_sql("SELECT *, COUNT(*) OVER () FROM (");
  7. self.query.walk_ast(out.reborrow())?;
  8. out.push_sql(") as paged_query_with LIMIT ");
  9. out.push_bind_param::<BigInt, _>(&self.per_page)?;
  10. out.push_sql(" OFFSET ");
  11. out.push_bind_param::<BigInt, _>(&self.offset)?;
  12. Ok(())
  13. }
  14. }

For details on what each method does, see the documentation for AstPass. One important question to ask whenever you implement QueryFragment is whether you are generating a query that is safe to cache. The way to answer this question is by asking “does this struct generate an unlimited number of potential SQL queries”? Typically that is only the case if the body of walk_ast contains a for loop. If your query is not safe to cache, you must call out.unsafe_to_cache_prepared.

Whenever you implement QueryFragment, you also need to implement QueryId. - We can use #[derive(QueryId)] for this. Since this struct represents a full query which can be executed, we will implement RunQueryDsl which adds methods like execute and load. Since this query has a return type, we’ll implement Query which states the return type as well.

src/pagination.rs

  1. impl<T: Query> Query for Paginated<T> {
  2. type SqlType = (T::SqlType, BigInt);
  3. }
  5. impl<T> RunQueryDsl<PgConnection> for Paginated<T> {}

Now that we’ve implemented all of these things, let’s look at how we would go about constructing this. We’ll want to add a paginate method to all Diesel queries, which specifies which page we’re on, as well as a per_page method which specifies the number of elements per page.

In order to add new methods to existing types, we can use a trait.

src/pagination.rs

  1. pub trait Paginate: Sized {
  2. fn paginate(self, page: i64) -> Paginated<Self>;
  3. }
  5. impl<T> Paginate for T {
  6. fn paginate(self, page: i64) -> Paginated<Self> {
  7. Paginated {
  8. query: self,
  9. per_page: DEFAULT_PER_PAGE,
  10. page,
  11. offset: (page - 1) * DEFAULT_PER_PAGE,
  12. }
  13. }
  14. }
  16. const DEFAULT_PER_PAGE: i64 = 10;
  18. #[derive(Debug, Clone, Copy, QueryId)]
  19. pub struct Paginated<T> {
  20. query: T,
  21. page: i64,
  22. per_page: i64,
  23. offset: i64,
  24. }
  26. impl<T> Paginated<T> {
  27. pub fn per_page(self, per_page: i64) -> Self {
  28. Paginated {
  29. per_page,
  30. offset: (self.page - 1) * per_page,
  31. ..self
  32. }
  33. }
  34. }

Now we can get the third page of a query with 25 elements per page like this:

Example

  1. let results: Vec<(User, i64)> = users::table
  2. .paginate(3)
  3. .per_page(25)
  4. .get_results(&conn)
  5. .expect("error");

With this code, we could load any query into a Vec<(T, i64)>, but we can do better. When doing pagination, you usually want the records and the total number of pages. We can write that method.

src/pagination.rs

  1. impl<T> Paginated<T> {
  2. pub fn load_and_count_pages<'a, U>(self, conn: &mut PgConnection) -> QueryResult<(Vec<U>, i64)>
  3. where
  4. Self: LoadQuery<'a, PgConnection, (U, i64)>,
  5. {
  6. let per_page = self.per_page;
  7. let results = self.load::<(U, i64)>(conn)?;
  8. let total = results.get(0).map(|x| x.1).unwrap_or(0);
  9. let records = results.into_iter().map(|x| x.0).collect();
  10. let total_pages = (total as f64 / per_page as f64).ceil() as i64;
  11. Ok((records, total_pages))
  12. }
  13. }

This is one of the rare cases where we want to define a function that takes a connection. One benefit of defining the function this way is that if we wanted to support backends other than PostgreSQL, we could have this function execute two queries.

You can find the full code for this example in the “advanced blog” example.

Custom Operators

If you’re adding support for a new type to Diesel, or working with a type that has incomplete support, you may wish to add support for the operators associated with that type. The term operator refers to anything that uses one of these syntaxes:

  • Infix (e.g. left OP right)
  • Prefix (e.g. OP expr)
  • Postfix (e.g. expr OP)

Diesel provides helper macros for defining each of these kinds of operators. In fact, Diesel uses these macros to declare nearly all of the operators supported by the main crate. The macros are diesel::infix_operator!, diesel::postfix_operator! and diesel::prefix_operator!.

All of these macros have the same signature. They take between two and four arguments.

The first is the name of the struct you want to represent this operator.

The second is the actual SQL for this operator.

The third argument is optional, and is the SQL type of the operator. If the SQL type is not specified, it will default to Bool. You can also pass the “magic” type ReturnBasedOnArgs, which will cause the SQL type to be the same as the type of its arguments. Diesel uses this to make the string concatenation operator || return Nullable<Text> if the arguments are nullable, or Text if they are not null.

The fourth argument (or third if you didn’t specify the SQL type) is the backend this operator is used for. If you don’t specify a backend, the operator can be used on all backends.

Let’s look at some example usage from Diesel:

Example

  1. // A simple operator. It returns `Bool` and works on all backends.
  2. diesel::infix_operator!(Eq, " = ");
  4. // Here we've specified the SQL type.
  5. // Since this operator is only used for ordering, and we don't want it used
  6. // elsewhere, we've made it `()` which is normally useless.
  7. diesel::postfix_operator!(Asc, " ASC", ());
  9. // Concat uses the magic `ReturnBasedOnArgs` return type
  10. // so it can work with both `Text` and `Nullable<Text>`.
  11. diesel::infix_operator!(Concat, " || ", ReturnBasedOnArgs);
  13. // This operator is PG specific, so we specify the backend
  14. diesel::infix_operator!(IsDistinctFrom, " IS DISTINCT FROM ", backend: Pg);
  16. // This operator is PG specific, and we are also specifying the SQL type.
  17. diesel::postfix_operator!(NullsFirst, " NULLS FIRST", (), backend: Pg);

Diesel provides a proof-of-concept crate showing how to add new SQL types called diesel_full_text_search. These are the operators as they are defined in that crate. You’ll notice all of the operators specify the backend, and many of them specify the return type.

src/lib.rs

  1. diesel::infix_operator!(Matches, " @@ ", backend: Pg);
  2. diesel::infix_operator!(Concat, " || ", TsVector, backend: Pg);
  3. diesel::infix_operator!(And, " && ", TsQuery, backend: Pg);
  4. diesel::infix_operator!(Or, " || ", TsQuery, backend: Pg);
  5. diesel::infix_operator!(Contains, " @> ", backend: Pg);
  6. diesel::infix_operator!(ContainedBy, " <@ ", backend: Pg);

However, just declaring the operator by itself isn’t very useful. This creates the types required by Diesel’s query builder, but doesn’t provide anything to help use the operator in real code. The structs created by these macros will have a new method, but that’s not typically how you work with Diesel’s query builder.

  • Infix operators are usually methods on the left hand side.
  • Postfix operators are usually methods on the expression.
  • Prefix operators are usually bare functions.

For operators that you create with methods, you would typically create a trait for this. For example, here’s how the .eq method gets defined by Diesel.

src/expression_methods/global_expression_methods.rs

  1. pub trait ExpressionMethods: Expression + Sized {
  2. fn eq<T>(self, other: T) -> dsl::Eq<Self, T>
  3. where
  4. Self::SqlType: SqlType,
  5. T: AsExpression<Self::SqlType>,
  6. {
  7. Grouped(Eq::new(self, other.as_expression()))
  8. }
  9. }
  11. impl<T: Expression> ExpressionMethods for T {}

It’s important to note that these methods are where you should put any type constraints. The structs defined by diesel::*_operator! don’t know or care about what the types of the arguments should be. The = operator requires that both sides be of the same type, so we represent that in the type of ExpressionMethods::eq.

You’ll also notice that our argument is AsExpression, not Expression. This allows Rust values to be passed as well as Diesel expressions. For example, we can do text_col.eq(other_text_col), or text_col.eq("Some Rust string").

If the operator is specific to only one SQL type, we can represent that in our trait.

src/expression_methods/global_expression_methods.rs

  1. pub trait BoolExpressionMethods
  2. where
  3. Self: Expression<SqlType = Bool> + Sized,
  4. {
  5. fn and<T, ST>(self, other: T) -> dsl::And<Self, T, ST>
  6. where
  7. Self::SqlType: SqlType,
  8. ST: SqlType + TypedExpressionType,
  9. T: AsExpression<ST>,
  10. And<Self, T::Expression>: Expression,
  11. {
  12. Grouped(And::new(self, other.as_expression()))
  13. }
  14. }
  16. impl<T> BoolExpressionMethods for T
  17. where
  18. T: Expression,
  19. T::SqlType: BoolOrNullableBool,
  20. {
  21. }

Prefix operators are usually defined as bare functions. The code is very similar, but without the trait. Here’s how not is defined in Diesel.

src/expression/not.rs

  1. pub fn not<T>(expr: T) -> not<T>
  2. where
  3. T: Expression,
  4. <T as Expression>::SqlType: BoolOrNullableBool,
  5. {
  6. super::operators::Not::new(Grouped(expr))
  7. }

In this case we’re using Grouped (which is currently undocumented in Diesel and only used internally) to add parenthesis around our argument. This ensures that the operator precedence in SQL matches what’s expected. For example, we would expect not(true.and(false)) to return true. However, SELECT NOT TRUE AND FALSE returns FALSE. Diesel does the same thing with .or.

It’s also a best practice to expose a “helper type” for your method, which does the same type conversion as the method itself. Nobody wants to write Eq<text_col, <&str as AsExpression<Text>>::Expression>. Instead, we provide a type that lets you write Eq<text_col, &str>.

src/expression/helper_types.rs

  1. pub type Eq<Lhs, Rhs> = Grouped<super::operators::Eq<Lhs, AsExpr<Rhs, Lhs>>>;

For defining these types, you’ll usually want to make use of SqlTypeOf, AsExpr, and AsExprOf.