1.2. Presto Concepts

Overview

To understand Presto you must first understand the terms and conceptsused throughout the Presto documentation.

While it’s easy to understand statements and queries, as an end-useryou should have familiarity with concepts such as stages and splits totake full advantage of Presto to execute efficient queries. As aPresto administrator or a Presto contributor you should understand howPresto’s concepts of stages map to tasks and how tasks contain a setof drivers which process data.

This section provides a solid definition for the core conceptsreferenced throughout Presto, and these sections are sorted from mostgeneral to most specific.

Server Types

There are two types of Presto servers: coordinators and workers. Thefollowing section explains the difference between the two.

Coordinator

The Presto coordinator is the server that is responsible for parsingstatements, planning queries, and managing Presto worker nodes. It isthe “brain” of a Presto installation and is also the node to which aclient connects to submit statements for execution. Every Prestoinstallation must have a Presto coordinator alongside one or morePresto workers. For development or testing purposes, a singleinstance of Presto can be configured to perform both roles.

The coordinator keeps track of the activity on each worker andcoordinates the execution of a query. The coordinator createsa logical model of a query involving a series of stages which is thentranslated into a series of connected tasks running on a cluster ofPresto workers.

Coordinators communicate with workers and clients using a REST API.

Worker

A Presto worker is a server in a Presto installation which is responsiblefor executing tasks and processing data. Worker nodes fetch data fromconnectors and exchange intermediate data with each other. The coordinatoris responsible for fetching results from the workers and returning thefinal results to the client.

When a Presto worker process starts up, it advertises itself to the discoveryserver in the coordinator, which makes it available to the Presto coordinatorfor task execution.

Workers communicate with other workers and Presto coordinatorsusing a REST API.

Data Sources

Throughout this documentation, you’ll read terms such as connector,catalog, schema, and table. These fundamental concepts cover Presto’smodel of a particular data source and are described in the followingsection.

Connector

A connector adapts Presto to a data source such as Hive or arelational database. You can think of a connector the same way youthink of a driver for a database. It is an implementation of Presto’sSPI which allows Presto to interactwith a resource using a standard API.

Presto contains several built-in connectors: a connector forJMX, a Systemconnector which provides access to built-in system tables,a Hive connector, and aTPCH connector designed to serve TPC-H benchmarkdata. Many third-party developers have contributed connectors so thatPresto can access data in a variety of data sources.

Every catalog is associated with a specific connector. If you examinea catalog configuration file, you will see that each contains amandatory property connector.name which is used by the catalogmanager to create a connector for a given catalog. It is possibleto have more than one catalog use the same connector to access twodifferent instances of a similar database. For example, if you havetwo Hive clusters, you can configure two catalogs in a single Prestocluster that both use the Hive connector, allowing you to query datafrom both Hive clusters (even within the same SQL query).

Catalog

A Presto catalog contains schemas and references a data source via aconnector. For example, you can configure a JMX catalog to provideaccess to JMX information via the JMX connector. When you run a SQLstatement in Presto, you are running it against one or more catalogs.Other examples of catalogs include the Hive catalog to connect to aHive data source.

When addressing a table in Presto, the fully-qualified table name isalways rooted in a catalog. For example, a fully-qualified table nameof hive.test_data.test would refer to the test table in thetest_data schema in the hive catalog.

Catalogs are defined in properties files stored in the Prestoconfiguration directory.

Schema

Schemas are a way to organize tables. Together, a catalog and schemadefine a set of tables that can be queried. When accessing Hive or arelational database such as MySQL with Presto, a schema translates tothe same concept in the target database. Other types of connectors maychoose to organize tables into schemas in a way that makes sense forthe underlying data source.

Table

A table is a set of unordered rows which are organized into named columnswith types. This is the same as in any relational database. The mappingfrom source data to tables is defined by the connector.

Query Execution Model

Presto executes SQL statements and turns these statements into queriesthat are executed across a distributed cluster of coordinator and workers.

Statement

Presto executes ANSI-compatible SQL statements. When the Prestodocumentation refers to a statement, it is referring to statements asdefined in the ANSI SQL standard which consists of clauses,expressions, and predicates.

Some readers might be curious why this section lists separate conceptsfor statements and queries. This is necessary because, in Presto,statements simply refer to the textual representation of a SQLstatement. When a statement is executed, Presto creates a query alongwith a query plan that is then distributed across a series of Prestoworkers.

Query

When Presto parses a statement, it converts it into a query and createsa distributed query plan which is then realized as a series ofinterconnected stages running on Presto workers. When you retrieveinformation about a query in Presto, you receive a snapshot of everycomponent that is involved in producing a result set in response to astatement.

The difference between a statement and a query is simple. A statementcan be thought of as the SQL text that is passed to Presto, while a queryrefers to the configuration and components instantiated to executethat statement. A query encompasses stages, tasks, splits, connectors,and other components and data sources working in concert to produce aresult.

Stage

When Presto executes a query, it does so by breaking up the executioninto a hierarchy of stages. For example, if Presto needs to aggregatedata from one billion rows stored in Hive, it does so by creating aroot stage to aggregate the output of several other stages all ofwhich are designed to implement different sections of a distributedquery plan.

The hierarchy of stages that comprises a query resembles a tree.Every query has a root stage which is responsible for aggregatingthe output from other stages. Stages are what the coordinator uses tomodel a distributed query plan, but stages themselves don’t run onPresto workers.

Task

As mentioned in the previous section, stages model a particularsection of a distributed query plan, but stages themselves don’texecute on Presto workers. To understand how a stage is executed,you’ll need to understand that a stage is implemented as a series oftasks distributed over a network of Presto workers.

Tasks are the “work horse” in the Presto architecture as a distributedquery plan is deconstructed into a series of stages which are thentranslated to tasks which then act upon or process splits. A Prestotask has inputs and outputs, and just as a stage can be executed inparallel by a series of tasks, a task is executing in parallel with aseries of drivers.

Split

Tasks operate on splits which are sections of a larger dataset. Stages at the lowest level of a distributed query plan retrievedata via splits from connectors, and intermediate stages at a higherlevel of a distributed query plan retrieve data from other stages.

When Presto is scheduling a query, the coordinator will query aconnector for a list of all splits that are available for a table.The coordinator keeps track of which machines are running which tasksand what splits are being processed by which tasks.

Driver

Tasks contain one or more parallel drivers. Drivers act upon data andcombine operators to produce output that is then aggregated by a taskand then delivered to another task in another stage. A driver is asequence of operator instances, or you can think of a driver as aphysical set of operators in memory. It is the lowest level ofparallelism in the Presto architecture. A driver has one input andone output.

Operator

An operator consumes, transforms and produces data. For example, a tablescan fetches data from a connector and produces data that can be consumedby other operators, and a filter operator consumes data and produces asubset by applying a predicate over the input data.

Exchange

Exchanges transfer data between Presto nodes for different stages ofa query. Tasks produce data into an output buffer and consume datafrom other tasks using an exchange client.