4.5. Resource Groups

Resource groups place limits on resource usage, and can enforce queueing policies onqueries that run within them or divide their resources among sub groups. A querybelongs to a single resource group, and consumes resources from that group (and its ancestors).Except for the limit on queued queries, when a resource group runs out of a resourceit does not cause running queries to fail; instead new queries become queued.A resource group may have sub groups or may accept queries, but may not do both.

The resource groups and associated selection rules are configured by a manager which is pluggable.Add an etc/resource-groups.properties file with the following contents to enablethe built-in manager that reads a JSON config file:

  1. resource-groups.configuration-manager=file
  2. resource-groups.config-file=etc/resource_groups.json

Change the value of resource-groups.config-file to point to a JSON config file,which can be an absolute path, or a path relative to the Presto data directory.

Resource Group Properties

  • name (required): name of the group. May be a template (see below).

  • maxQueued (required): maximum number of queued queries. Once this limit is reachednew queries will be rejected.

  • hardConcurrencyLimit (required): maximum number of running queries.

  • softMemoryLimit (required): maximum amount of distributed memory thisgroup may use before new queries become queued. May be specified asan absolute value (i.e. 1GB) or as a percentage (i.e. 10%) of the cluster’s memory.

  • softCpuLimit (optional): maximum amount of CPU time thisgroup may use in a period (see cpuQuotaPeriod) before a penalty will be applied tothe maximum number of running queries. hardCpuLimit must also be specified.

  • hardCpuLimit (optional): maximum amount of CPU time thisgroup may use in a period.

  • schedulingPolicy (optional): specifies how queued queries are selected to run,and how sub groups become eligible to start their queries. May be one of three values:

  • fair (default): queued queries are processed first-in-first-out, and sub groups
    must take turns starting new queries (if they have any queued).

  • weighted: queued queries are selected stochastically in proportion to their priority
    (specified via the query_priority session property). Sub groups are selected
    to start new queries in proportion to their schedulingWeight.

  • query_priority: all sub groups must also be configured with query_priority.
    Queued queries will be selected strictly according to their priority.

  • schedulingWeight (optional): weight of this sub group. See above.Defaults to 1.

  • jmxExport (optional): If true, group statistics are exported to JMX for monitoring.Defaults to false.

  • queuedTimeLimit (optional): maximum amount of time a query may be in the queue for this groupbefore it is marked as failed.

  • runningTimeLimit (optional): maximum amount of time a query in this group can execute(not including queued time) before it is marked as failed.

  • subGroups (optional): list of sub groups.

Selector Properties

  • user (optional): regex to match against user name. Defaults to .*

  • source (optional): regex to match against source string. Defaults to .*

  • queryType (optional): string to match against the type of the query submitted. The query type can be:

    • DATA_DEFINITION: Queries that alter/create/drop the metadata of schemas/tables/views, and that manageprepared statements, privileges, sessions, and transactions.
    • DELETE: DELETE queries.
    • DESCRIBE: DESCRIBE, DESCRIBE INPUT, DESCRIBE OUTPUT, and SHOW queries.
    • EXPLAIN: EXPLAIN queries.
    • INSERT: INSERT and CREATE TABLE AS SELECT queries.
    • SELECT: SELECT queries.
  • group (required): the group these queries will run in.

Global Properties

  • cpuQuotaPeriod (optional): the period in which cpu quotas are enforced.
    Selectors are processed sequentially and the first one that matches will be used.In the example configuration below, there are five resource group templates.In the adhoc_${USER} group, ${USER} will be expanded to the name of theuser that submitted the query. ${SOURCE} is also supported, which expandsto the source submitting the query. The source name can be set as follows:
- CLI: use the —source option.- JDBC: set the ApplicationName client info property on the Connection instance.

There are three selectors that define which queries run in which resource group:

- The first selector places queries from bob into the admin group.- The second selector states that all data definition queries that come from a source that includes “pipeline”should run in the user’s personal data definition group, which belongs to thegloba.data_definition parent group.- The third selector states that all queries that come from a source that includes “pipeline”should run in the user’s personal pipeline group, which belongs to the global.pipelineparent group.- The last selector is a catch all, which puts all queries into the user’s adhoc group.

All together these selectors implement the policy that bob is an admin andall other users are subject to the following limits:

- Users are allowed to have up to 2 adhoc queries running. Additionally, they may run one pipeline.- No more than 5 “pipeline” queries may run at once.- No more than 100 total queries may run at once, unless they’re from the admin.
  1. {
  2.   "rootGroups": [
  3.     {
  4.       "name": "global",
  5.       "softMemoryLimit": "80%",
  6.       "hardConcurrencyLimit": 100,
  7.       "maxQueued": 1000,
  8.       "schedulingPolicy": "weighted",
  9.       "jmxExport": true,
  10.       "subGroups": [
  11.         {
  12.           "name": "data_definition_${USER}",
  13.           "softMemoryLimit": "10%",
  14.           "hardConcurrencyLimit": 3,
  15.           "maxQueued": 10,
  16.           "schedulingWeight": 1
  17.         },
  18.         {
  19.           "name": "adhoc_${USER}",
  20.           "softMemoryLimit": "10%",
  21.           "hardConcurrencyLimit": 2,
  22.           "maxQueued": 1,
  23.           "schedulingWeight": 9,
  24.           "schedulingPolicy": "query_priority"
  25.         },
  26.         {
  27.           "name": "pipeline",
  28.           "softMemoryLimit": "20%",
  29.           "hardConcurrencyLimit": 5,
  30.           "maxQueued": 100,
  31.           "schedulingWeight": 1,
  32.           "jmxExport": true,
  33.           "subGroups": [
  34.             {
  35.               "name": "pipeline_${USER}",
  36.               "softMemoryLimit": "10%",
  37.               "hardConcurrencyLimit": 1,
  38.               "maxQueued": 100,
  39.               "schedulingPolicy": "query_priority"
  40.             }
  41.           ]
  42.         }
  43.       ]
  44.     },
  45.     {
  46.       "name": "admin",
  47.       "softMemoryLimit": "100%",
  48.       "hardConcurrencyLimit": 200,
  49.       "maxQueued": 100,
  50.       "schedulingPolicy": "query_priority",
  51.       "jmxExport": true
  52.     }
  53.   ],
  54.   "selectors": [
  55.     {
  56.       "user": "bob",
  57.       "group": "admin"
  58.     },
  59.     {
  60.       "source": ".*pipeline.*",
  61.       "queryType": "DATA_DEFINITION",
  62.       "group": "global.data_definition_${USER}"
  63.     },
  64.     {
  65.       "source": ".*pipeline.*",
  66.       "group": "global.pipeline.pipeline_${USER}"
  67.     },
  68.     {
  69.       "group": "global.adhoc_${USER}"
  70.     }
  71.   ],
  72.   "cpuQuotaPeriod": "1h"
  73. }

原文: https://prestodb.io/docs/current/admin/resource-groups.html