Appendix 1: Configuration Parameters
Before starting to use IoTDB, you need to config the configuration files first. For your convenience, we have already set the default config in the files.
In total, we provide users three kinds of configurations module:
environment configuration file (iotdb-env.bat, iotdb-env.sh). The default configuration file for the environment configuration item. Users can configure the relevant system configuration items of JAVA-JVM in the file.
system configuration file (iotdb-engine.properties).
iotdb-engine.properties: The default configuration file for the IoTDB engine layer configuration item. Users can configure the IoTDB engine related parameters in the file, such as JDBC service listening port (rpc_port), unsequence data storage directory (unsequence_data_dir), etc. What’s more, Users can configure the information about the TsFile, such as the data size written to the disk per time(group_size_in_byte).
- log configuration file (
logback.xml)
The configuration files of the three configuration items are located in the IoTDB installation directory: $IOTDB_HOME/conf folder.
Hot Modification Configuration
For the convenience of users, IoTDB server provides users with hot modification function, that is, modifying some configuration parameters in iotdb engine. Properties during the system operation and applying them to the system immediately. In the parameters described below, these parameters whose way of Effective is trigger support hot modification.
Trigger way: The client sends the command load configuration to the IoTDB server. See Chapter 4 for the usage of the client.
IoTDB Environment Configuration File
The environment configuration file is mainly used to configure the Java environment related parameters when IoTDB Server is running, such as JVM related configuration. This part of the configuration is passed to the JVM when the IoTDB Server starts. Users can view the contents of the environment configuration file by viewing the iotdb-env.sh (or iotdb-env.bat) file.
The detail of each variables are as follows:
| Name | MAX_HEAP_SIZE |
|---|
| Description | The maximum heap memory size that IoTDB can use at startup. |
| Type | String |
| Default | On Linux or MacOS, the default is one quarter of the memory. On Windows, the default value for 32-bit systems is 512M, and the default for 64-bit systems is 2G. |
| Effective | After restart system |
| Name | HEAP_NEWSIZE |
|---|
| Description | The minimum heap memory size that IoTDB can use at startup. |
| Type | String |
| Default | On Linux or MacOS, the default is min{cores * 100M, one quarter of MAX_HEAP_SIZE}. On Windows, the default value for 32-bit systems is 512M, and the default for 64-bit systems is 2G. |
| Effective | After restart system |
| Name | JMX_LOCAL |
|---|
| Description | JMX monitoring mode, configured as yes to allow only local monitoring, no to allow remote monitoring |
| Type | Enum String: “true”, “false” |
| Default | true |
| Effective | After restart system |
| Name | JMX_PORT |
|---|
| Description | JMX listening port. Please confirm that the port is not a system reserved port and is not occupied |
| Type | Short Int: [0,65535] |
| Default | 31999 |
| Effective | After restart system |
| Name | JMX_IP |
|---|
| Description | JMX listening address. Only take effect if JMX_LOCAL=false. 0.0.0.0 is never allowed |
| Type | String |
| Default | 127.0.0.1 |
| Effective | After restart system |
JMX Authorization
We STRONGLY RECOMMENDED you CHANGE the PASSWORD for the JMX remote connection.
The user and passwords are in ${IOTDB_CONF}/conf/jmx.password.
The permission definitions are in ${IOTDB_CONF}/conf/jmx.access.
IoTDB System Configuration File
File Layer
| Name | compressor |
|---|
| Description | Data compression method |
| Type | Enum String : “UNCOMPRESSED”, “SNAPPY” |
| Default | UNCOMPRESSED |
| Effective | Trigger |
| Name | group_size_in_byte |
|---|
| Description | The data size written to the disk per time |
| Type | Int32 |
| Default | 134217728 |
| Effective | Trigger |
| Name | page_size_in_byte |
|---|
| Description | The maximum size of a single page written in memory when each column in memory is written (in bytes) |
| Type | Int32 |
| Default | 65536 |
| Effective | Trigger |
- max_number_of_points_in_page
| Name | max_number_of_points_in_page |
|---|
| Description | The maximum number of data points (timestamps - valued groups) contained in a page |
| Type | Int32 |
| Default | 1048576 |
| Effective | Trigger |
| Name | max_degree_of_index_node |
|---|
| Description | The maximum degree of the metadata index tree (that is, the max number of each node’s children) |
| Type | Int32 |
| Default | 256 |
| Effective | Only allowed to be modified in first start up |
| Name | max_string_length |
|---|
| Description | The maximum length of a single string (number of character) |
| Type | Int32 |
| Default | 128 |
| Effective | Trigger |
| Name | time_series_data_type |
|---|
| Description | Timestamp data type |
| Type | Enum String: “INT32”, “INT64” |
| Default | Int64 |
| Effective | Trigger |
| Name | time_encoder |
|---|
| Description | Encoding type of time column |
| Type | Enum String: “TS_2DIFF”,“PLAIN”,“RLE” |
| Default | TS_2DIFF |
| Effective | Trigger |
| Name | value_encoder |
|---|
| Description | Encoding type of value column |
| Type | Enum String: “TS_2DIFF”,“PLAIN”,“RLE” |
| Default | PLAIN |
| Effective | Trigger |
| Name | float_precision |
|---|
| Description | The precision of the floating point number.(The number of digits after the decimal point) |
| Type | Int32 |
| Default | The default is 2 digits. Note: The 32-bit floating point number has a decimal precision of 7 bits, and the 64-bit floating point number has a decimal precision of 15 bits. If the setting is out of the range, it will have no practical significance. |
| Effective | Trigger |
| Name | bloomFilterErrorRate |
|---|
| Description | The false positive rate of bloom filter in each TsFile. Bloom filter checks whether a given time series is in the tsfile before loading metadata. This can improve the performance of loading metadata and skip the tsfile that doesn’t contain specified time series. If you want to learn more about its mechanism, you can refer to: wiki page of bloom filter (opens new window). |
| Type | float, (0, 1) |
| Default | 0.05 |
| Effective | After restart system |
Engine Layer
| Name | rpc_address |
|---|
| Description | The jdbc service listens on the address. |
| Type | String |
| Default | “0.0.0.0” |
| Effective | After restart system |
| Name | rpc_port |
|---|
| Description | The jdbc service listens on the port. Please confirm that the port is not a system reserved port and is not occupied. |
| Type | Short Int : [0,65535] |
| Default | 6667 |
| Effective | After restart system |
- rpc_thrift_compression_enable
| Name | rpc_thrift_compression_enable |
|---|
| Description | Whether enable thrift’s compression (using GZIP). |
| Type | Boolean |
| Default | false |
| Effective | After restart system |
- rpc_advanced_compression_enable
| Name | rpc_advanced_compression_enable |
|---|
| Description | Whether enable thrift’s advanced compression. |
| Type | Boolean |
| Default | false |
| Effective | After restart system |
| Name | time_zone |
|---|
| Description | The time zone in which the server is located, the default is Beijing time (+8) |
| Type | Time Zone String |
| Default | +08:00 |
| Effective | Trigger |
| Name | base_dir |
|---|
| Description | The IoTDB system folder. It is recommended to use an absolute path. |
| Type | String |
| Default | data |
| Effective | After restart system |
| Name | data_dirs |
|---|
| Description | The directories of data files. Multiple directories are separated by comma. The starting directory of the relative path is related to the operating system. It is recommended to use an absolute path. If the path does not exist, the system will automatically create it. |
| Type | String[] |
| Default | data/data |
| Effective | Trigger |
| Name | wal_dir |
|---|
| Description | Write Ahead Log storage path. It is recommended to use an absolute path. |
| Type | String |
| Default | data/wal |
| Effective | After restart system |
| Name | enable_wal |
|---|
| Description | Whether to enable the pre-write log. The default value is true(enabled), and false means closed. |
| Type | Bool |
| Default | true |
| Effective | Trigger |
| Name | enable_mem_control |
|---|
| Description | enable memory control to avoid OOM |
| Type | Bool |
| Default | true |
| Effective | After restart system |
| Name | memtable_size_threshold |
|---|
| Description | max memtable size |
| Type | Long |
| Default | 1073741824 |
| Effective | when enable_mem_control is false & After restart system |
- avg_series_point_number_threshold
| Name | avg_series_point_number_threshold |
|---|
| Description | max average number of point of each series in memtable |
| Type | Int32 |
| Default | 10000 |
| Effective | After restart system |
| Name | tsfile_size_threshold |
|---|
| Description | max tsfile size |
| Type | Long |
| Default | 536870912 |
| Effective | After restart system |
| Name | enable_partition |
|---|
| Description | Whether enable time partition for data, if disabled, all data belongs to partition 0 |
| Type | Bool |
| Default | false |
| Effective | Only allowed to be modified in first start up |
| Name | partition_interval |
|---|
| Description | Time range for dividing storage group, time series data will be divided into groups by this time range |
| Type | Int64 |
| Default | 604800 |
| Effective | Only allowed to be modified in first start up |
- concurrent_writing_time_partition
| Name | concurrent_writing_time_partition |
|---|
| Description | This config decides how many time partitions in a storage group can be inserted concurrently
For example, your partitionInterval is 86400 and you want to insert data in 5 different days, |
| Type | Int32 |
| Default | 1 |
| Effective | After restart system |
| Name | multi_dir_strategy |
|---|
| Description | IoTDB’s strategy for selecting directories for TsFile in tsfile_dir. You can use a simple class name or a full name of the class. The system provides the following three strategies:
1. SequenceStrategy: IoTDB selects the directory from tsfile_dir in order, traverses all the directories in tsfile_dir in turn, and keeps counting;
2. MaxDiskUsableSpaceFirstStrategy: IoTDB first selects the directory with the largest free disk space in tsfile_dir;
3. MinFolderOccupiedSpaceFirstStrategy: IoTDB prefers the directory with the least space used in tsfile_dir;
4. UserDfineStrategyPackage (user-defined policy)
You can complete a user-defined policy in the following ways:
1. Inherit the cn.edu.tsinghua.iotdb.conf.directories.strategy.DirectoryStrategy class and implement its own Strategy method;
2. Fill in the configuration class with the full class name of the implemented class (package name plus class name, UserDfineStrategyPackage);
3. Add the jar file to the project. |
| Type | String |
| Default | MaxDiskUsableSpaceFirstStrategy |
| Effective | Trigger |
| Name | tsfile_size_threshold |
|---|
| Description | When a TsFile size on the disk exceeds this threshold, the TsFile is closed and open a new TsFile to accept data writes. The unit is byte and the default value is 2G. |
| Type | Int64 |
| Default | 536870912 |
| Effective | After restart system |
| Name | tag_attribute_total_size |
|---|
| Description | The maximum persistence size of tags and attributes of each time series. |
| Type | Int32 |
| Default | 700 |
| Effective | Only allowed to be modified in first start up |
| Name | enable_partial_insert |
|---|
| Description | Whether continue to write other measurements if some measurements are failed in one insertion. |
| Type | Bool |
| Default | true |
| Effective | After restart system |
| Name | mtree_snapshot_interval |
|---|
| Description | The least interval line numbers of mlog.txt when creating a checkpoint and saving snapshot of MTree. Unit: line numbers |
| Type | Int32 |
| Default | 100000 |
| Effective | After restart system |
| Name | mlog_buffer_size |
|---|
| Description | size of log buffer in each metadata operation plan(in byte) |
| Type | Int32 |
| Default | 1048576 |
| Effective | After restart system |
| Name | sync_mlog_period_in_ms |
|---|
| Description | The cycle when metadata log is periodically forced to be written to disk(in milliseconds). If force_mlog_period_in_ms = 0 it means force metadata log to be written to disk after each refreshment |
| Type | Int64 |
| Default | 100 |
| Effective | After restart system |
| Name | flush_wal_threshold |
|---|
| Description | After the WAL reaches this value, it is flushed to disk, and it is possible to lose at most flush_wal_threshold operations. |
| Type | Int32 |
| Default | 10000 |
| Effective | Trigger |
| Name | force_wal_period_in_ms |
|---|
| Description | The period during which the log is periodically forced to flush to disk(in milliseconds) |
| Type | Int32 |
| Default | 10 |
| Effective | Trigger |
| Name | fetch_size |
|---|
| Description | The amount of data read each time in batch (the number of data strips, that is, the number of different timestamps.) |
| Type | Int32 |
| Default | 10000 |
| Effective | After restart system |
| Name | merge_concurrent_threads |
|---|
| Description | THe max threads which can be used when unsequence data is merged. The larger it is, the more IO and CPU cost. The smaller the value, the more the disk is occupied when the unsequence data is too large, the reading will be slower. |
| Type | Int32 |
| Default | 0 |
| Effective | After restart system |
| Name | enable_stat_monitor |
|---|
| Description | Whether to enable background statistics |
| Type | Boolean |
| Default | false |
| Effective | After restart system |
| Name | concurrent_flush_thread |
|---|
| Description | The thread number used to perform the operation when IoTDB writes data in memory to disk. If the value is less than or equal to 0, then the number of CPU cores installed on the machine is used. The default is 0. |
| Type | Int32 |
| Default | 0 |
| Effective | After restart system |
| Name | tsfile_storage_fs |
|---|
| Description | The storage file system of Tsfile and related data files. Currently LOCAL file system and HDFS are supported. |
| Type | String |
| Default | LOCAL |
| Effective | Only allowed to be modified in first start up |
| Name | core_site_path |
|---|
| Description | Absolute file path of core-site.xml if Tsfile and related data files are stored in HDFS. |
| Type | String |
| Default | /etc/hadoop/conf/core-site.xml |
| Effective | After restart system |
| Name | hdfs_site_path |
|---|
| Description | Absolute file path of hdfs-site.xml if Tsfile and related data files are stored in HDFS. |
| Type | String |
| Default | /etc/hadoop/conf/hdfs-site.xml |
| Effective | After restart system |
| Name | hdfs_ip |
|---|
| Description | IP of HDFS if Tsfile and related data files are stored in HDFS. If there are more than one hdfs_ip in configuration, Hadoop HA is used. |
| Type | String |
| Default | localhost |
| Effective | After restart system |
| Name | hdfs_port |
|---|
| Description | Port of HDFS if Tsfile and related data files are stored in HDFS |
| Type | String |
| Default | 9000 |
| Effective | After restart system |
| Name | hdfs_nameservices |
|---|
| Description | Nameservices of HDFS HA if using Hadoop HA |
| Type | String |
| Default | hdfsnamespace |
| Effective | After restart system |
| Name | hdfs_ha_namenodes |
|---|
| Description | Namenodes under DFS nameservices of HDFS HA if using Hadoop HA |
| Type | String |
| Default | nn1,nn2 |
| Effective | After restart system |
- dfs_ha_automatic_failover_enabled
| Name | dfs_ha_automatic_failover_enabled |
|---|
| Description | Whether using automatic failover if using Hadoop HA |
| Type | Boolean |
| Default | true |
| Effective | After restart system |
- dfs_client_failover_proxy_provider
| Name | dfs_client_failover_proxy_provider |
|---|
| Description | Proxy provider if using Hadoop HA and enabling automatic failover |
| Type | String |
| Default | org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider |
| Effective | After restart system |
| Name | hdfs_use_kerberos |
|---|
| Description | Whether use kerberos to authenticate hdfs |
| Type | String |
| Default | false |
| Effective | After restart system |
- kerberos_keytab_file_path
| Name | kerberos_keytab_file_path |
|---|
| Description | Full path of kerberos keytab file |
| Type | String |
| Default | /path |
| Effective | After restart system |
| Name | kerberos_principal |
|---|
| Description | Kerberos pricipal |
| Type | String |
| Default | your principal |
| Effective | After restart system |
- authorizer_provider_class
| Name | authorizer_provider_class |
|---|
| Description | the class name of the authorization service |
| Type | String |
| Default | org.apache.iotdb.db.auth.authorizer.LocalFileAuthorizer |
| Effective | After restart system |
| Other available values | org.apache.iotdb.db.auth.authorizer.OpenIdAuthorizer |
| Name | openID_url |
|---|
| Description | the openID server if OpenIdAuthorizer is enabled |
| Type | String (a http url) |
| Default | no |
| Effective | After restart system |
| Name | thrift_max_frame_size |
|---|
| Description | the max bytes in a RPC request/response |
| Type | long |
| Default | 67108864 (should >= 8 1024 1024) |
| Effective | After restart system |
Automatic Schema Creation and Type Inference
- enable_auto_create_schema
| Name | enable_auto_create_schema |
|---|
| Description | whether auto create the time series when a non-existed time series data comes |
| Type | true or false |
| Default | true |
| Effective | After restart system |
- default_storage_group_level
| Name | default_storage_group_level |
|---|
| Description | Storage group level when creating schema automatically is enabled. For example, if we receives a data point from root.sg0.d1.s2, we will set root.sg0 as the storage group if storage group level is 1. (root is level 0) |
| Type | integer |
| Default | 1 |
| Effective | After restart system |
- boolean_string_infer_type
| Name | boolean_string_infer_type |
|---|
| Description | To which type the values “true” and “false” should be reslved |
| Type | BOOLEAN or TEXT |
| Default | BOOLEAN |
| Effective | After restart system |
- integer_string_infer_type
| Name | integer_string_infer_type |
|---|
| Description | To which type an integer string like “67” in a query should be resolved |
| Type | INT32, INT64, DOUBLE, FLOAT or TEXT |
| Default | DOUBLE |
| Effective | After restart system |
| Name | nan_string_infer_type |
|---|
| Description | To which type the value NaN in a query should be resolved |
| Type | DOUBLE, FLOAT or TEXT |
| Default | FLOAT |
| Effective | After restart system |
- floating_string_infer_type
| Name | floating_string_infer_type |
|---|
| Description | To which type a floating number string like “6.7” in a query should be resolved |
| Type | DOUBLE, FLOAT or TEXT |
| Default | FLOAT |
| Effective | After restart system |
| Name | enable_partition |
|---|
| Description | whether enable data partition. If disabled, all data belongs to partition 0 |
| Type | BOOLEAN |
| Default | false |
| Effective | After restart system |
| Name | partition_interval |
|---|
| Description | time range for partitioning data inside each storage group, the unit is second |
| Type | LONG |
| Default | 604800 |
| Effective | After restart system |
Enable GC log
GC log is off by default. For performance tuning, you may want to collect the GC info.
To enable GC log, just add a parameter “printgc” when you start the server.
nohup sbin/start-server.sh printgc >/dev/null 2>&1 &
Or
sbin\start-server.bat printgc
GC log is stored at IOTDB_HOME/logs/gc.log. There will be at most 10 gc.log.* files and each one can reach to 10MB.
我的书签
添加书签
移除书签
