Performance Analyzer

Performance Analyzer is an agent and REST API that allows you to query numerous performance metrics for your cluster, including aggregations of those metrics, independent of the Java Virtual Machine (JVM). PerfTop is the default command line interface (CLI) for displaying those metrics.

To download PerfTop, see Download on the PerfTop release page.

You can also install it using npm:

  1. npm install -g @aws/opensearch-perftop

PerfTop screenshot

For enabling Performance Analyzer with tarball installations of OpenSearch, see Configure Performance Analyzer for Tarball Installation.

Get started with PerfTop

The basic syntax is:

  1. ./opensearch-perf-top-<operating_system> --dashboard <dashboard>.json --endpoint <endpoint>

If you’re using npm, the syntax is similar:

  1. opensearch-perf-top --dashboard <dashboard> --endpoint <endpoint>

If you’re running PerfTop from a node (i.e. locally), specify port 9600:

  1. ./opensearch-perf-top-linux --dashboard dashboards/<dashboard>.json --endpoint localhost:9600

Otherwise, just specify the OpenSearch endpoint:

  1. ./opensearch-perf-top-macos --dashboard dashboards/<dashboard>.json --endpoint my-cluster.my-domain.com

PerfTop has four pre-built dashboards in the dashboards directory, but you can also create your own.

You can also load the pre-built dashboards (ClusterOverview, ClusterNetworkMemoryAnalysis, ClusterThreadAnalysis, or NodeAnalysis) without the JSON files, such as --dashboard ClusterThreadAnalysis.

PerfTop has no interactivity. Start the application, monitor the dashboard, and press Esc, Q, or Ctrl + C to quit.

Other options

  • For NodeAnalysis and similar custom dashboards, you can add the --nodename <node_name> argument if you want your dashboard to display metrics for only a single node.
  • For troubleshooting, add the --logfile <log-file>.txt argument.

Performance Analyzer configuration

Storage

Performance Analyzer uses /dev/shm for temporary storage. During heavy workloads on a cluster, Performance Analyzer can use up to 1 GB of space.

Docker, however, has a default /dev/shm size of 64 MB. To change this value, you can use the docker run --shm-size 1gb flag or a similar setting in Docker Compose.

If you’re not using Docker, check the size of /dev/shm using df -h. The default value is probably plenty, but if you need to change its size, add the following line to /etc/fstab:

  1. tmpfs /dev/shm tmpfs defaults,noexec,nosuid,size=1G 0 0

Then remount the file system:

  1. mount -o remount /dev/shm

Security

Performance Analyzer supports encryption in transit for requests. It currently does not support client or server authentication for requests. To enable encryption in transit, edit performance-analyzer.properties in your $OPENSEARCH_HOME directory:

  1. vi $OPENSEARCH_HOME/config/opensearch-performance-analyzer/performance-analyzer.properties

Change the following lines to configure encryption in transit. Note that certificate-file-path must be a certificate for the server, not a root CA:

  1. https-enabled = true
  3. #Setup the correct path for certificates
  4. certificate-file-path = specify_path
  6. private-key-file-path = specify_path

Enable Performance Analyzer for RPM/YUM installations

If you installed OpenSearch from an RPM distribution, you can start and stop Performance Analyzer with systemctl:

  1. # Start OpenSearch Performance Analyzer
  2. sudo systemctl start opensearch-performance-analyzer.service
  3. # Stop OpenSearch Performance Analyzer
  4. sudo systemctl stop opensearch-performance-analyzer.service

Configure Performance Analyzer for tarball installations

In a tarball installation, Performance Analyzer collects data when it is enabled. But in order to read that data using the REST API on port 9600, you must first manually launch the associated reader agent process:

  1. Make Performance Analyzer accessible outside of the host machine

    1. cd /usr/share/opensearch # navigate to the OpenSearch home directory
    2. cd config/opensearch-performance-analyzer/
    3. vi performance-analyzer.properties

    Uncomment the line #webservice-bind-host and set it to 0.0.0.0:

    1. # ======================== OpenSearch performance analyzer plugin config =========================
    3. # NOTE: this is an example for Linux. Please modify the config accordingly if you are using it under other OS.
    5. # WebService bind host; default to all interfaces
    6. webservice-bind-host = 0.0.0.0
    8. # Metrics data location
    9. metrics-location = /dev/shm/performanceanalyzer/
    11. # Metrics deletion interval (minutes) for metrics data.
    12. # Interval should be between 1 to 60.
    13. metrics-deletion-interval = 1
    15. # If set to true, the system cleans up the files behind it. So at any point, we should expect only 2
    16. # metrics-db-file-prefix-path files. If set to false, no files are cleaned up. This can be useful, if you are archiving
    17. # the files and wouldn't like for them to be cleaned up.
    18. cleanup-metrics-db-files = true
    20. # WebService exposed by App's port
    21. webservice-listener-port = 9600
    23. # Metric DB File Prefix Path location
    24. metrics-db-file-prefix-path = /tmp/metricsdb_
    26. https-enabled = false
    28. #Setup the correct path for certificates
    29. certificate-file-path = specify_path
    31. private-key-file-path = specify_path
    33. # Plugin Stats Metadata file name, expected to be in the same location
    34. plugin-stats-metadata = plugin-stats-metadata
    36. # Agent Stats Metadata file name, expected to be in the same location
    37. agent-stats-metadata = agent-stats-metadata

  2. Make the CLI executable:

    1. sudo chmod +x ./bin/performance-analyzer-agent-cli

  3. Launch the agent CLI:

    1. OPENSEARCH_HOME="$PWD" OPENSEARCH_PATH_CONF="$PWD/config" ./bin/performance-analyzer-agent-cli

  4. In a separate window, enable the Performance Analyzer plugin:

    1. curl -XPOST localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}'

    If you receive the curl: (52) Empty reply from server error, you are likely protecting your cluster with the security plugin and you need to provide credentials. Modify the following command to use your username and password:

    1. curl -XPOST https://localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k

  5. Finally, enable the Root Cause Analyzer (RCA) framework

    1. curl -XPOST localhost:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}'

    Similar to step 4, if you run into curl: (52) Empty reply from server, run the command below to enable RCA

    1. curl -XPOST https://localhost:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k

Table of contents