Security Model

Prometheus is a sophisticated system with many components and many integrationswith other systems. It can be deployed in a variety of trusted and untrustedenvironments.

This page describes the general security assumptions of Prometheus and theattack vectors that some configurations may enable.

As with any complex system, it is not possible to guarantee that there are nobugs. If you find a security bug please report it privately to the maintainerslisted in the MAINTAINERS.md of the relevant repository and CC[email protected]. We will fix the issue and coordinate arelease date with you, acknowledging your effort and mentioning you by nameif you want.

Prometheus

It is presumed that untrusted users have access to the Prometheus HTTP endpointand logs. They have access to all time series information contained in thedatabase, plus a variety of operational/debugging information.

It is also presumed that only trusted users have the ability to change thecommand line, configuration file, rule files and other aspects of the runtimeenvironment of Prometheus and other components.

Which targets Prometheus scrapes, how often and with what other settings isdetermined entirely via the configuration file. The administrator maydecide to use information from service discovery systems, which combined withrelabelling may grant some of this control to anyone who can modify data inthat service discovery system.

Scraped targets may be run by untrusted users. It should not by default bepossible for a target to expose data that impersonates a different target. Thehonor_labels option removes this protection, as can certain relabellingsetups.

As of Prometheus 2.0, the —web.enable-admin-api flag controls access to theadministrative HTTP API which includes functionality such as deleting timeseries. This is disabled by default. If enabled, administrative and mutatingfunctionality will be accessible under the /api/*/admin/ paths. The—web.enable-lifecycle flag controls HTTP reloads and shutdowns ofPrometheus. This is also disabled by default. If enabled they will beaccessible under the /-/reload and /-/quit paths.

In Prometheus 1.x, /-/reload and using DELETE on /api/v1/series areaccessible to anyone with access to the HTTP API. The /-/quit endpoint isdisabled by default, but can be enabled with the -web.enable-remote-shutdownflag.

The remote read feature allows anyone with HTTP access to send queries to theremote read endpoint. If for example the PromQL queries were ending up directlyrun against a relational database, then anyone with the ability to send queriesto Prometheus (such as via Grafana) can run arbitrary SQL against thatdatabase.

Alertmanager

Any user with access to the Alertmanager HTTP endpoint has access to its data.They can create and resolve alerts. They can create, modify and deletesilences.

Where notifications are sent to is determined by the configuration file. Withcertain templating setups it is possible for notifications to end up at analert-defined destination. For example if notifications use an alert label asthe destination email address, anyone who can send alerts to the Alertmanagercan send notifications to any email address. If the alert-defined destinationis a templatable secret field, anyone with access to either Prometheus orAlertmanager will be able to view the secrets.

Any secret fields which are templatable are intended for routing notificationsin the above use case. They are not intended as a way for secrets to beseparated out from the configuration files using the template file feature. Anysecrets stored in template files could be exfiltrated by anyone able toconfigure receivers in the Alertmanager configuration file. For example inlarge setups, each team might have an alertmanager configuration file fragmentwhich they fully control, that are then combined into the full finalconfiguration file.

Pushgateway

Any user with access to the Pushgateway HTTP endpoint can create, modify anddelete the metrics contained within. As the Pushgateway is usually scraped withhonor_labels enabled, this means anyone with access to the Pushgateway cancreate any time series in Prometheus.

The —web.enable-admin-api flag controls access to theadministrative HTTP API, which includes functionality such as wiping all the existingmetric groups. This is disabled by default. If enabled, administrativefunctionality will be accessible under the /api/*/admin/ paths.

Exporters

Exporters generally only talk to one configured instance with a preset set ofcommands/requests, which cannot be expanded via their HTTP endpoint.

There are also exporters such as the SNMP and Blackbox exporters that taketheir targets from URL parameters. Thus anyone with HTTP access to theseexporters can make them send requests to arbitrary endpoints. As they alsosupport client-side authentication, this could lead to a leak of secrets suchas HTTP Basic Auth passwords or SNMP community strings. Challenge-responseauthentication mechanisms such as TLS are not affected by this.

Client Libraries

Client libraries are intended to be included in users' applications.

If using a client-library-provided HTTP handler, it should not be possible formalicious requests that reach that handler to cause issues beyond thoseresulting from additional load and failed scrapes.

Authentication, Authorization, and Encryption

Prometheus and its components do not provide any server-sideauthentication, authorization or encryption. If you require this, it isrecommended to use a reverse proxy.

As administrative and mutating endpoints are intended to be accessed via simpletools such as cURL, there is no built inCSRF protection asthat would break such use cases. Accordingly when using a reverse proxy, youmay wish to block such paths to prevent CSRF.

For non-mutating endpoints, you may wish to set CORSheaders such asAccess-Control-Allow-Origin in your reverse proxy to preventXSS.

If you are composing PromQL queries that include input from untrusted users(e.g. URL parameters to console templates, or something you built yourself) whoare not meant to be able to run arbitrary PromQL queries make sure anyuntrusted input is appropriately escaped to prevent injection attacks. Forexample up{job="<user_input>"} would become up{job=""} orsome_metric{zzz=""} if the <user_input> was "} or some_metric{zzz=".

For those using Grafana note that dashboard permissions are not data sourcepermissions,so do not limit a user's ability to run arbitrary queries in proxy mode.

Various Prometheus components support client-side authentication andencryption. If TLS client support is offered, there is often also an optioncalled insecure_skip_verify which skips SSL verification.

Secrets

Non-secret information or fields may be available via the HTTP API and/or logs.

In Prometheus, metadata retrieved from service discovery is not consideredsecret. Throughout the Prometheus system, metrics are not considered secret.

Fields containing secrets in configuration files (marked explicitly as such inthe documentation) will not be exposed in logs or via the HTTP API. Secretsshould not be placed in other configuration fields, as it is common forcomponents to expose their configuration over their HTTP endpoint.

Secrets from other sources used by dependencies (e.g. the AWS_SECRET_KEYenvironment variable as used by EC2 service discovery) may end up exposed due tocode outside of our control or due to functionality that happens to exposewherever it is stored.

Denial of Service

There are some mitigations in place for excess load or expensive queries.However, if too many or too expensive queries/metrics are provided componentswill fall over. It is more likely that a component will be accidentally takenout by a trusted user than by malicious action.

It is the responsibility of the user to ensure they provide components withsufficient resources including CPU, RAM, disk space, IOPS, file descriptors,and bandwidth.

It is recommended to monitor all components for failure, and to have themautomatically restart on failure.

Libraries

This document considers vanilla binaries built from the stock source code.Information presented here does not apply if you modify Prometheus source code,or use Prometheus internals (beyond the official client library APIs) in yourown code.

Build Process

The build pipeline for Prometheus runs on third-party providers to which manymembers of the Prometheus development team and the staff of those providershave access. If you are concerned about the exact provenance of your binaries,it is recommended to build them yourself rather than relying on thepre-built binaries provided by the project.

External audits

CNCF sponsored an external security audit bycure53 which ran from April 2018 to June 2018.

For more details, please read thefinal report of the audit.