TDengine Python Connector

taospy is the official Python connector for TDengine. taospy provides a rich API that makes it easy for Python applications to use TDengine. taospy wraps both the native interface and REST interface of TDengine, which correspond to the taos and taosrest modules of the taospy package, respectively. In addition to wrapping the native and REST interfaces, taospy also provides a set of programming interfaces that conforms to the Python Data Access Specification (PEP 249). It is easy to integrate taospy with many third-party tools, such as SQLAlchemy and pandas.

The direct connection to the server using the native interface provided by the client driver is referred to hereinafter as a “native connection”; the connection to the server using the REST interface provided by taosAdapter is referred to hereinafter as a “REST connection”.

The source code for the Python connector is hosted on GitHub.

Supported platforms

  • The supported platforms for the native connection are the same as the ones supported by the TDengine client.
  • REST connections are supported on all platforms that can run Python.

Version selection

We recommend using the latest version of taospy, regardless of the version of TDengine.

Supported features

  • Native connections support all the core features of TDengine, including connection management, SQL execution, bind interface, subscriptions, and schemaless writing.
  • REST connections support features such as connection management and SQL execution. (SQL execution allows you to: manage databases, tables, and supertables, write data, query data, create continuous queries, etc.).

Installation

Preparation

  1. Install Python. Python >= 3.7 is recommended. If Python is not available on your system, refer to the Python BeginnersGuide to install it.
  2. Install pip. In most cases, the Python installer comes with the pip utility. If not, please refer to pip documentation to install it. If you use a native connection, you will also need to Install Client Driver. The client install package includes the TDengine client dynamic link library (libtaos.so or taos.dll) and the TDengine CLI.

Install via pip

Uninstalling an older version

If you have installed an older version of the Python Connector, please uninstall it beforehand.

  1. pip3 uninstall taos taospy
Python - 图1note

Earlier TDengine client software includes the Python connector. If the Python connector is installed from the client package’s installation directory, the corresponding Python package name is taos. So the above uninstall command includes taos, and it doesn’t matter if it doesn’t exist.

To install taospy

  • Install from PyPI
  • Install from GitHub

Install the latest version of:

  1. pip3 install taospy

You can also specify a specific version to install:

  1. pip3 install taospy==2.3.0
  1. pip3 install git+https://github.com/taosdata/taos-connector-python.git

Verify

  • native connection
  • REST connection

For native connection, you need to verify that both the client driver and the Python connector itself are installed correctly. The client driver and Python connector have been installed properly if you can successfully import the taos module. In the Python Interactive Shell, you can type.

  1. import taos

For REST connections, verifying that the taosrest module can be imported successfully can be done in the Python Interactive Shell by typing.

  1. import taosrest
Python - 图2tip

If you have multiple versions of Python on your system, you may have various pip commands. Be sure to use the correct path for the pip command. Above, we installed the pip3 command, which rules out the possibility of using the pip corresponding to Python 2.x versions. However, if you have more than one version of Python 3.x on your system, you still need to check that the installation path is correct. The easiest way to verify this is to type pip3 install taospy again in the command, and it will print out the exact location of taospy, for example, on Windows.

  1. C:\> pip3 install taospy
  2. Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
  3. Requirement already satisfied: taospy in c:\users\username\appdata\local\programs\python\python310\lib\site-packages (2.3.0)

Establishing a connection

Connectivity testing

Before establishing a connection with the connector, we recommend testing the connectivity of the local TDengine CLI to the TDengine cluster.

  • native connection
  • REST connection

Ensure that the TDengine instance is up and that the FQDN of the machines in the cluster (the FQDN defaults to hostname if you are starting a stand-alone version) can be resolved locally, by testing with the ping command.

  1. ping <FQDN>

Then test if the cluster can be appropriately connected with TDengine CLI:

  1. taos -h <FQDN> -p <PORT>

The FQDN above can be the FQDN of any dnode in the cluster, and the PORT is the serverPort corresponding to this dnode.

For REST connections, make sure the cluster and taosAdapter component, are running. This can be tested using the following curl command.

  1. curl -u root:taosdata http://<FQDN>:<PORT>/rest/sql -d "select server_version()"

The FQDN above is the FQDN of the machine running taosAdapter, PORT is the port taosAdapter listening, default is 6041. If the test is successful, it will output the server version information, e.g.

  1. {
  2.   "code": 0,
  3.   "column_meta": [
  4.     [
  5.       "server_version()",
  6.       "VARCHAR",
  7.       7
  8.     ]
  9.   ],
  10.   "data": [
  11.     [
  12.       "3.0.0.0"
  13.     ]
  14.   ],
  15.   "rows": 1
  16. }

Using connectors to establish connections

The following example code assumes that TDengine is installed locally and that the default configuration is used for both FQDN and serverPort.

  • native connection
  • REST connection
  1. import taos
  3. conn: taos.TaosConnection = taos.connect(host="localhost",
  4.                                          user="root",
  5.                                          password="taosdata",
  6.                                          database="test",
  7.                                          port=6030,
  8.                                          config="/etc/taos",  # for windows the default value is C:\TDengine\cfg
  9.                                          timezone="Asia/Shanghai")  # default your host's timezone
  11. server_version = conn.server_info
  12. print("server_version", server_version)
  13. client_version = conn.client_info
  14. print("client_version", client_version)  # 3.0.0.0
  16. conn.close()
  18. # possible output:
  19. # 3.0.0.0
  20. # 3.0.0.0

view source code

All arguments of the connect() function are optional keyword arguments. The following are the connection parameters specified.

  • host : The FQDN of the node to connect to. There is no default value. If this parameter is not provided, the firstEP in the client configuration file will be connected.
  • user : The TDengine user name. The default value is root.
  • password : TDengine user password. The default value is taosdata.
  • port : The starting port of the data node to connect to, i.e., the serverPort configuration. The default value is 6030, which will only take effect if the host parameter is provided.
  • config : The path to the client configuration file. On Windows systems, the default is C:\TDengine\cfg. The default is /etc/taos/ on Linux/macOS.
  • timezone : The timezone used to convert the TIMESTAMP data in the query results to python datetime objects. The default is the local timezone.
Python - 图3warning

config and timezone are both process-level configurations. We recommend that all connections made by a process use the same parameter values. Otherwise, unpredictable errors may occur.

Python - 图4tip

The connect() function returns a taos.TaosConnection instance. In client-side multi-threaded scenarios, we recommend that each thread request a separate connection instance rather than sharing a connection between multiple threads.

  1. from taosrest import connect, TaosRestConnection, TaosRestCursor
  3. conn: TaosRestConnection = connect(url="http://localhost:6041",
  4.                                    user="root",
  5.                                    password="taosdata",
  6.                                    timeout=30)

view source code

All arguments to the connect() function are optional keyword arguments. The following are the connection parameters specified.

  • url： The URL of taosAdapter REST service. The default is http://localhost:6041.
  • user: TDengine user name. The default is root.
  • password: TDengine user password. The default is taosdata.
  • timeout: HTTP request timeout. Enter a value in seconds. The default is socket._GLOBAL_DEFAULT_TIMEOUT. Usually, no configuration is needed.

Example program

Basic Usage

  • native connection
  • REST connection
TaosConnection class

The TaosConnection class contains both an implementation of the PEP249 Connection interface (e.g., the cursor() method and the close() method) and many extensions (e.g., the execute(), query(), schemaless_insert(), and subscribe() methods).

execute method

  1. conn = taos.connect()
  2. # Execute a sql, ignore the result set, just get affected rows. It's useful for DDL and DML statement.
  3. conn.execute("DROP DATABASE IF EXISTS test")
  4. conn.execute("CREATE DATABASE test")
  5. # change database. same as execute "USE db"
  6. conn.select_db("test")
  7. conn.execute("CREATE STABLE weather(ts TIMESTAMP, temperature FLOAT) TAGS (location INT)")
  8. affected_row: int = conn.execute("INSERT INTO t1 USING weather TAGS(1) VALUES (now, 23.5) (now+1m, 23.5) (now+2m, 24.4)")
  9. print("affected_row", affected_row)
  10. # output:
  11. # affected_row 3

view source code

query method

  1. # Execute a sql and get its result set. It's useful for SELECT statement
  2. result: taos.TaosResult = conn.query("SELECT * from weather")
  4. # Get fields from result
  5. fields: taos.field.TaosFields = result.fields
  6. for field in fields:
  7.     print(field)  # {name: ts, type: 9, bytes: 8}
  9. # output:
  10. # {name: ts, type: 9, bytes: 8}
  11. # {name: temperature, type: 6, bytes: 4}
  12. # {name: location, type: 4, bytes: 4}
  14. # Get data from result as list of tuple
  15. data = result.fetch_all()
  16. print(data)
  17. # output:
  18. # [(datetime.datetime(2022, 4, 27, 9, 4, 25, 367000), 23.5, 1), (datetime.datetime(2022, 4, 27, 9, 5, 25, 367000), 23.5, 1), (datetime.datetime(2022, 4, 27, 9, 6, 25, 367000), 24.399999618530273, 1)]
  20. # Or get data from result as a list of dict
  21. # map_data = result.fetch_all_into_dict()
  22. # print(map_data)
  23. # output:
  24. # [{'ts': datetime.datetime(2022, 4, 27, 9, 1, 15, 343000), 'temperature': 23.5, 'location': 1}, {'ts': datetime.datetime(2022, 4, 27, 9, 2, 15, 343000), 'temperature': 23.5, 'location': 1}, {'ts': datetime.datetime(2022, 4, 27, 9, 3, 15, 343000), 'temperature': 24.399999618530273, 'location': 1}]

view source code

Python - 图5tip

The queried results can only be fetched once. For example, only one of fetch_all() and fetch_all_into_dict() can be used in the example above. Repeated fetches will result in an empty list.

Use of TaosResult class

In the above example of using the TaosConnection class, we have shown two ways to get the result of a query: fetch_all() and fetch_all_into_dict(). In addition, TaosResult also provides methods to iterate through the result set by rows (rows_iter) or by data blocks (blocks_iter). Using these two methods will be more efficient in scenarios where the query has a large amount of data.

blocks_iter method

  1. import taos
  3. conn = taos.connect()
  4. conn.execute("DROP DATABASE IF EXISTS test")
  5. conn.execute("CREATE DATABASE test")
  6. conn.select_db("test")
  7. conn.execute("CREATE STABLE weather(ts TIMESTAMP, temperature FLOAT) TAGS (location INT)")
  8. # prepare data
  9. for i in range(2000):
  10.     location = str(i % 10)
  11.     tb = "t" + location
  12.     conn.execute(f"INSERT INTO {tb} USING weather TAGS({location}) VALUES (now+{i}a, 23.5) (now+{i + 1}a, 23.5)")
  14. result: taos.TaosResult = conn.query("SELECT * FROM weather")
  16. block_index = 0
  17. blocks: taos.TaosBlocks = result.blocks_iter()
  18. for rows, length in blocks:
  19.     print("block ", block_index, " length", length)
  20.     print("first row in this block:", rows[0])
  21.     block_index += 1
  23. conn.close()
  25. # possible output:
  26. # block  0  length 1200
  27. # first row in this block: (datetime.datetime(2022, 4, 27, 15, 14, 52, 46000), 23.5, 0)
  28. # block  1  length 1200
  29. # first row in this block: (datetime.datetime(2022, 4, 27, 15, 14, 52, 76000), 23.5, 3)
  30. # block  2  length 1200
  31. # first row in this block: (datetime.datetime(2022, 4, 27, 15, 14, 52, 99000), 23.5, 6)
  32. # block  3  length 400
  33. # first row in this block: (datetime.datetime(2022, 4, 27, 15, 14, 52, 122000), 23.5, 9)

view source code

Use of the TaosCursor class

The TaosConnection class and the TaosResult class already implement all the functionality of the native interface. If you are familiar with the interfaces in the PEP249 specification, you can also use the methods provided by the TaosCursor class.

Use of TaosCursor

  1. import taos
  3. conn = taos.connect()
  4. cursor = conn.cursor()
  6. cursor.execute("DROP DATABASE IF EXISTS test")
  7. cursor.execute("CREATE DATABASE test")
  8. cursor.execute("USE test")
  9. cursor.execute("CREATE STABLE weather(ts TIMESTAMP, temperature FLOAT) TAGS (location INT)")
  11. for i in range(1000):
  12.     location = str(i % 10)
  13.     tb = "t" + location
  14.     cursor.execute(f"INSERT INTO {tb} USING weather TAGS({location}) VALUES (now+{i}a, 23.5) (now+{i + 1}a, 23.5)")
  16. cursor.execute("SELECT count(*) FROM weather")
  17. data = cursor.fetchall()
  18. print("count:", data[0][0])
  19. cursor.execute("SELECT tbname, * FROM weather LIMIT 2")
  20. col_names = [meta[0] for meta in cursor.description]
  21. print(col_names)
  22. rows = cursor.fetchall()
  23. print(rows)
  25. cursor.close()
  26. conn.close()
  28. # output:
  29. # count: 2000
  30. # ['tbname', 'ts', 'temperature', 'location']
  31. # row_count: -1
  32. # [('t0', datetime.datetime(2022, 4, 27, 14, 54, 24, 392000), 23.5, 0), ('t0', datetime.datetime(2022, 4, 27, 14, 54, 24, 393000), 23.5, 0)]

view source code

Python - 图6note

The TaosCursor class uses native connections for write and query operations. In a client-side multi-threaded scenario, this cursor instance must remain thread exclusive and cannot be shared across threads for use, otherwise, it will result in errors in the returned results.

Use of TaosRestCursor class

The TaosRestCursor class is an implementation of the PEP249 Cursor interface.

Use of TaosRestCursor

  1. # create STable
  2. cursor: TaosRestCursor = conn.cursor()
  3. cursor.execute("DROP DATABASE IF EXISTS power")
  4. cursor.execute("CREATE DATABASE power")
  5. cursor.execute("CREATE STABLE power.meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (location BINARY(64), groupId INT)")
  7. # insert data
  8. cursor.execute("""INSERT INTO power.d1001 USING power.meters TAGS(California.SanFrancisco, 2) VALUES ('2018-10-03 14:38:05.000', 10.30000, 219, 0.31000) ('2018-10-03 14:38:15.000', 12.60000, 218, 0.33000) ('2018-10-03 14:38:16.800', 12.30000, 221, 0.31000)
  9.     power.d1002 USING power.meters TAGS(California.SanFrancisco, 3) VALUES ('2018-10-03 14:38:16.650', 10.30000, 218, 0.25000)
  10.     power.d1003 USING power.meters TAGS(California.LosAngeles, 2) VALUES ('2018-10-03 14:38:05.500', 11.80000, 221, 0.28000) ('2018-10-03 14:38:16.600', 13.40000, 223, 0.29000)
  11.     power.d1004 USING power.meters TAGS(California.LosAngeles, 3) VALUES ('2018-10-03 14:38:05.000', 10.80000, 223, 0.29000) ('2018-10-03 14:38:06.500', 11.50000, 221, 0.35000)""")
  12. print("inserted row count:", cursor.rowcount)
  14. # query data
  15. cursor.execute("SELECT * FROM power.meters LIMIT 3")
  16. # get total rows
  17. print("queried row count:", cursor.rowcount)
  18. # get column names from cursor
  19. column_names = [meta[0] for meta in cursor.description]
  20. # get rows
  21. data: list[tuple] = cursor.fetchall()
  22. print(column_names)
  23. for row in data:
  24.     print(row)
  26. # output:
  27. # inserted row count: 8
  28. # queried row count: 3
  29. # ['ts', 'current', 'voltage', 'phase', 'location', 'groupid']
  30. # [datetime.datetime(2018, 10, 3, 14, 38, 5, 500000, tzinfo=datetime.timezone(datetime.timedelta(seconds=28800), '+08:00')), 11.8, 221, 0.28, 'california.losangeles', 2]
  31. # [datetime.datetime(2018, 10, 3, 14, 38, 16, 600000, tzinfo=datetime.timezone(datetime.timedelta(seconds=28800), '+08:00')), 13.4, 223, 0.29, 'california.losangeles', 2]
  32. # [datetime.datetime(2018, 10, 3, 14, 38, 5, tzinfo=datetime.timezone(datetime.timedelta(seconds=28800), '+08:00')), 10.8, 223, 0.29, 'california.losangeles', 3]

view source code

  • cursor.execute: Used to execute arbitrary SQL statements.
  • cursor.rowcount : For write operations, returns the number of successful rows written. For query operations, returns the number of rows in the result set.
  • cursor.description : Returns the description of the field. Please refer to TaosRestCursor for the specific format of the description information.
Use of the RestClient class

The RestClient class is a direct wrapper for the REST API. It contains only a sql() method for executing arbitrary SQL statements and returning the result.

Use of RestClient

  1. from taosrest import RestClient
  3. client = RestClient("http://localhost:6041", user="root", password="taosdata")
  4. res: dict = client.sql("SELECT ts, current FROM power.meters LIMIT 1")
  5. print(res)
  7. # output:
  8. # {'status': 'succ', 'head': ['ts', 'current'], 'column_meta': [['ts', 9, 8], ['current', 6, 4]], 'data': [[datetime.datetime(2018, 10, 3, 14, 38, 5, tzinfo=datetime.timezone(datetime.timedelta(seconds=28800), '+08:00')), 10.3]], 'rows': 1}

view source code

For a more detailed description of the sql() method, please refer to RestClient.

Used with pandas

  • native connection
  • REST connection
  1. import pandas
  2. from sqlalchemy import create_engine
  4. engine = create_engine("taos://root:taosdata@localhost:6030/power")
  5. df = pandas.read_sql("SELECT * FROM meters", engine)
  7. # print index
  8. print(df.index)
  9. # print data type  of element in ts column
  10. print(type(df.ts[0]))
  11. print(df.head(3))
  13. # output:
  14. # RangeIndex(start=0, stop=8, step=1)
  15. # <class 'pandas._libs.tslibs.timestamps.Timestamp'>
  16. #                        ts  current  ...               location  groupid
  17. # 0 2018-10-03 14:38:05.500     11.8  ...  california.losangeles        2
  18. # 1 2018-10-03 14:38:16.600     13.4  ...  california.losangeles        2
  19. # 2 2018-10-03 14:38:05.000     10.8  ...  california.losangeles        3

view source code

  1. import pandas
  2. from sqlalchemy import create_engine
  4. engine = create_engine("taosrest://root:taosdata@localhost:6041")
  5. df: pandas.DataFrame = pandas.read_sql("SELECT * FROM power.meters", engine)
  7. # print index
  8. print(df.index)
  9. # print data type  of element in ts column
  10. print(type(df.ts[0]))
  11. print(df.head(3))
  13. # output:
  14. # RangeIndex(start=0, stop=8, step=1)
  15. # <class 'pandas._libs.tslibs.timestamps.Timestamp'>
  16. #                                 ts  current  ...               location  groupid
  17. # 0 2018-10-03 06:38:05.500000+00:00     11.8  ...  california.losangeles        2
  18. # 1 2018-10-03 06:38:16.600000+00:00     13.4  ...  california.losangeles        2
  19. # 2        2018-10-03 06:38:05+00:00     10.8  ...  california.losangeles        3

view source code

Other sample programs

Example program linksExample program content
bind_multi.pyparameter binding, bind multiple rows at once
bind_row.pybind_row.py
insert_lines.pyInfluxDB line protocol writing
json_tag.pyUse JSON type tags
tmq.pyTMQ subscription

Other notes

Exception handling

All errors from database operations are thrown directly as exceptions and the error message from the database is passed up the exception stack. The application is responsible for exception handling. For example:

  1. import taos
  3. try:
  4.     conn = taos.connect()
  5.     conn.execute("CREATE TABLE 123")  # wrong sql
  6. except taos.Error as e:
  7.     print(e)
  8.     print("exception class: ", e.__class__.__name__)
  9.     print("error number:", e.errno)
  10.     print("error message:", e.msg)
  11. except BaseException as other:
  12.     print("exception occur")
  13.     print(other)
  15. # output:
  16. # [0x0216]: syntax error near 'Incomplete SQL statement'
  17. # exception class:  ProgrammingError
  18. # error number: -2147483114
  19. # error message: syntax error near 'Incomplete SQL statement'

view source code

About nanoseconds

Due to the current imperfection of Python’s nanosecond support (see link below), the current implementation returns integers at nanosecond precision instead of the datetime type produced by ms and us, which application developers will need to handle on their own. And it is recommended to use pandas’ to_datetime(). The Python Connector may modify the interface in the future if Python officially supports nanoseconds in full.

  1. https://stackoverflow.com/questions/10611328/parsing-datetime-strings-containing-nanoseconds
  2. https://www.python.org/dev/peps/pep-0564/

Important Update

[Release Notes] (https://github.com/taosdata/taos-connector-python/releases)

API Reference

Frequently Asked Questions

Welcome to ask questions or report questions.