SQL Expressions

How do I render SQL expressions as strings, possibly with bound parameters inlined?

The “stringification” of a SQLAlchemy Core statement object orexpression fragment, as well as that of an ORM Query object,in the majority of simple cases is as simple as usingthe str() builtin function, as below when use it with the printfunction (note the Python print function also calls str() automaticallyif we don’t use it explicitly):

  1. >>> from sqlalchemy import table, column, select
  2. >>> t = table('my_table', column('x'))
  3. >>> statement = select([t])
  4. >>> print(str(statement))
  5. SELECT my_table.x
  6. FROM my_table

The str() builtin, or an equivalent, can be invoked on ORMQuery object as well as any statement such as that ofselect(), insert() etc. and also any expression fragment, suchas:

  1. >>> from sqlalchemy import column
  2. >>> print(column('x') == 'some value')
  3. x = :x_1

Stringifying for Specific Databases

A complication arises when the statement or fragment we are stringifyingcontains elements that have a database-specific string format, or when itcontains elements that are only available within a certain kind of database.In these cases, we might get a stringified statement that is not in the correctsyntax for the database we are targeting, or the operation may raise aUnsupportedCompilationError exception. In these cases, it isnecessary that we stringify the statement using theClauseElement.compile() method, while passing along an Engineor Dialect object that represents the target database. Such asbelow, if we have a MySQL database engine, we can stringify a statement interms of the MySQL dialect:

  1. from sqlalchemy import create_engine
  2.  
  3. engine = create_engine("mysql+pymysql://scott:tiger@localhost/test")
  4. print(statement.compile(engine))

More directly, without building up an Engine object we caninstantiate a Dialect object directly, as below where weuse a PostgreSQL dialect:

  1. from sqlalchemy.dialects import postgresql
  2. print(statement.compile(dialect=postgresql.dialect()))

When given an ORM Query object, in order to get at theClauseElement.compile()method we only need access the statementaccessor first:

  1. statement = query.statement
  2. print(statement.compile(someengine))

Rendering Bound Parameters Inline

Warning

Never use this technique with string content received fromuntrusted input, such as from web forms or other user-input applications.SQLAlchemy’s facilities to coerce Python values into direct SQL stringvalues are not secure against untrusted input and do not validate the typeof data being passed. Always use bound parameters when programmaticallyinvoking non-DDL SQL statements against a relational database.

The above forms will render the SQL statement as it is passed to the PythonDBAPI, which includes that bound parameters are not rendered inline.SQLAlchemy normally does not stringify bound parameters, as this is handledappropriately by the Python DBAPI, not to mention bypassing boundparameters is probably the most widely exploited security hole inmodern web applications. SQLAlchemy has limited ability to do thisstringification in certain circumstances such as that of emitting DDL.In order to access this functionality one can use the literal_bindsflag, passed to compile_kwargs:

  1. from sqlalchemy.sql import table, column, select
  2.  
  3. t = table('t', column('x'))
  4.  
  5. s = select([t]).where(t.c.x == 5)
  6.  
  7. print(s.compile(compile_kwargs={"literal_binds": True}))  # **do not use** with untrusted input!!!

the above approach has the caveats that it is only supported for basictypes, such as ints and strings, and furthermore if a bindparam()without a pre-set value is used directly, it won’t be able tostringify that either.

To support inline literal rendering for types not supported, implementa TypeDecorator for the target type which includes aTypeDecorator.process_literal_param() method:

  1. from sqlalchemy import TypeDecorator, Integer
  2.  
  3.  
  4. class MyFancyType(TypeDecorator):
  5.     impl = Integer
  6.  
  7.     def process_literal_param(self, value, dialect):
  8.         return "my_fancy_formatting(%s)" % value
  9.  
  10. from sqlalchemy import Table, Column, MetaData
  11.  
  12. tab = Table('mytable', MetaData(), Column('x', MyFancyType()))
  13.  
  14. print(
  15.     tab.select().where(tab.c.x > 5).compile(
  16.         compile_kwargs={"literal_binds": True})
  17. )

producing output like:

  1. SELECT mytable.x
  2. FROM mytable
  3. WHERE mytable.x > my_fancy_formatting(5)

I’m using op() to generate a custom operator and my parenthesis are not coming out correctly

The Operators.op() method allows one to create a custom database operatorotherwise not known by SQLAlchemy:

  1. >>> print(column('q').op('->')(column('p')))
  2. q -> p

However, when using it on the right side of a compound expression, it doesn’tgenerate parenthesis as we expect:

  1. >>> print((column('q1') + column('q2')).op('->')(column('p')))
  2. q1 + q2 -> p

Where above, we probably want (q1 + q2) -> p.

The solution to this case is to set the precedence of the operator, usingthe Operators.op.precedence parameter, to a highnumber, where 100 is the maximum value, and the highest number used by anySQLAlchemy operator is currently 15:

  1. >>> print((column('q1') + column('q2')).op('->', precedence=100)(column('p')))
  2. (q1 + q2) -> p

We can also usually force parenthesization around a binary expression (e.g.an expression that has left/right operands and an operator) using theColumnElement.self_group() method:

  1. >>> print((column('q1') + column('q2')).self_group().op('->')(column('p')))
  2. (q1 + q2) -> p

Why are the parentheses rules like this?

A lot of databases barf when there are excessive parenthesis or whenparenthesis are in unusual places they doesn’t expect, so SQLAlchemy does notgenerate parenthesis based on groupings, it uses operator precedence and if theoperator is known to be associative, so that parenthesis are generatedminimally. Otherwise, an expression like:

  1. column('a') & column('b') & column('c') & column('d')

would produce:

  1. (((a AND b) AND c) AND d)

which is fine but would probably annoy people (and be reported as a bug). Inother cases, it leads to things that are more likely to confuse databases or atthe very least readability, such as:

  1. column('q', ARRAY(Integer, dimensions=2))[5][6]

would produce:

  1. ((q[5])[6])

There are also some edge cases where we get things like "(x) = 7" and databasesreally don’t like that either. So parenthesization doesn’t naivelyparenthesize, it uses operator precedence and associativity to determinegroupings.

For Operators.op(), the value of precedence defaults to zero.

What if we defaulted the value of Operators.op.precedence to 100,e.g. the highest? Then this expression makes more parenthesis, but isotherwise OK, that is, these two are equivalent:

  1. >>> print (column('q') - column('y')).op('+', precedence=100)(column('z'))
  2. (q - y) + z
  3. >>> print (column('q') - column('y')).op('+')(column('z'))
  4. q - y + z

but these two are not:

  1. >>> print column('q') - column('y').op('+', precedence=100)(column('z'))
  2. q - y + z
  3. >>> print column('q') - column('y').op('+')(column('z'))
  4. q - (y + z)

For now, it’s not clear that as long as we are doing parenthesization based onoperator precedence and associativity, if there is really a way to parenthesizeautomatically for a generic operator with no precedence given that is going towork in all cases, because sometimes you want a custom op to have a lowerprecedence than the other operators and sometimes you want it to be higher.

It is possible that maybe if the “binary” expression above forced the use ofthe self_group() method when op() is called, making the assumption thata compound expression on the left side can always be parenthesized harmlessly.Perhaps this change can be made at some point, however for the time beingkeeping the parenthesization rules more internally consistent seems to bethe safer approach.