Resource Planning

It is important to plan computing and storage resources if using TDengine to build an IoT, time-series or Big Data platform. How to plan the CPU, memory and disk resources required, will be described in this chapter.

Memory Requirement of Server Side

By default, the number of vgroups created for each database is the same as the number of CPU cores. This can be configured by the parameter maxVgroupsPerDb. Each vnode in a vgroup stores one replica. Each vnode consumes a fixed amount of memory, i.e. blocks * cache. In addition, some memory is required for tag values associated with each table. A fixed amount of memory is required for each cluster. So, the memory required for each DB can be calculated using the formula below:

  1. Database Memory Size = maxVgroupsPerDb * replica * (blocks * cache + 10MB) + numOfTables * (tagSizePerTable + 0.5KB)

For example, assuming the default value of maxVgroupPerDB is 64, the default value of cache is 16M, the default value of blocks is 6, there are 100,000 tables in a DB, the replica number is 1, total length of tag values is 256 bytes, the total memory required for this DB is: 64 * 1 * (16 * 6 + 10) + 100000 * (0.25 + 0.5) / 1000 = 6792M.

In the real operation of TDengine, we are more concerned about the memory used by each TDengine server process taosd.

  1.     taosd_memory = vnode_memory + mnode_memory + query_memory

In the above formula:

  1. “vnode_memory” of a taosd process is the memory used by all vnodes hosted by this taosd process. It can be roughly calculated by firstly adding up the total memory of all DBs whose memory usage can be derived according to the formula for Database Memory Size, mentioned above, then dividing by number of dnodes and multiplying the number of replicas.
  1.     vnode_memory = (sum(Database Memory Size) / number_of_dnodes) * replica

  1. “mnode_memory” of a taosd process is the memory consumed by a mnode. If there is one (and only one) mnode hosted in a taosd process, the memory consumed by “mnode” is “0.2KB * the total number of tables in the cluster”.

  2. “query_memory” is the memory used when processing query requests. Each ongoing query consumes at least “0.2 KB * total number of involved tables”.

Please note that the above formulas can only be used to estimate the minimum memory requirement, instead of maximum memory usage. In a real production environment, it’s better to reserve some redundance beyond the estimated minimum memory requirement. If memory is abundant, it’s suggested to increase the value of parameter blocks to speed up data insertion and data query.

Memory Requirement of Client Side

For the client programs using TDengine client driver taosc to connect to the server side there is a memory requirement as well.

The memory consumed by a client program is mainly about the SQL statements for data insertion, caching of table metadata, and some internal use. Assuming maximum number of tables is N (the memory consumed by the metadata of each table is 256 bytes), maximum number of threads for parallel insertion is T, maximum length of a SQL statement is S (normally 1 MB), the memory required by a client program can be estimated using the below formula:

  1. M = (T * S * 3 + (N / 4096) + 100)

For example, if the number of parallel data insertion threads is 100, total number of tables is 10,000,000, then the minimum memory requirement of a client program is:

  1. 100 * 3 + (10000000 / 4096) + 100 = 2741 (MBytes)

So, at least 3GB needs to be reserved for such a client.

CPU Requirement

The CPU resources required depend on two aspects:

  • Data Insertion Each dnode of TDengine can process at least 10,000 insertion requests in one second, while each insertion request can have multiple rows. The difference in computing resource consumed, between inserting 1 row at a time, and inserting 10 rows at a time is very small. So, the more the number of rows that can be inserted one time, the higher the efficiency. Inserting in batch also imposes requirements on the client side which needs to cache rows to insert in batch once the number of cached rows reaches a threshold.
  • Data Query High efficiency query is provided in TDengine, but it’s hard to estimate the CPU resource required because the queries used in different use cases and the frequency of queries vary significantly. It can only be verified with the query statements, query frequency, data size to be queried, and other requirements provided by users.

In short, the CPU resource required for data insertion can be estimated but it’s hard to do so for query use cases. In real operation, it’s suggested to control CPU usage below 50%. If this threshold is exceeded, it’s a reminder for system operator to add more nodes in the cluster to expand resources.

Disk Requirement

The compression ratio in TDengine is much higher than that in RDBMS. In most cases, the compression ratio in TDengine is bigger than 5, or even 10 in some cases, depending on the characteristics of the original data. The data size before compression can be calculated based on below formula:

  1. Raw DataSize = numOfTables * rowSizePerTable * rowsPerTable

For example, there are 10,000,000 meters, while each meter collects data every 15 minutes and the data size of each collection is 128 bytes, so the raw data size of one year is: 10000000 * 128 * 24 * 60 / 15 * 365 = 44.8512(TB). Assuming compression ratio is 5, the actual disk size is: 44.851 / 5 = 8.97024(TB).

Parameter keep can be used to set how long the data will be kept on disk. To further reduce storage cost, multiple storage levels can be enabled in TDengine, with the coldest data stored on the cheapest storage device. This is completely transparent to application programs.

To increase performance, multiple disks can be setup for parallel data reading or data inserting. Please note that an expensive disk array is not necessary because replications are used in TDengine to provide high availability.

Number of Hosts

A host can be either physical or virtual. The total memory, total CPU, total disk required can be estimated according to the formulae mentioned previously. Then, according to the system resources that a single host can provide, assuming all hosts have the same resources, the number of hosts can be derived easily.

Quick Estimation for CPU, Memory and Disk Please refer to Resource Estimate.