Monitoring a Greenplum System

You can monitor a Greenplum Database system using a variety of tools included with the system or available as add-ons. SNMP support allows Greenplum to be integrated with popular system management frameworks.

Observing the Greenplum Database system day-to-day performance helps administrators understand the system behavior, plan workflow, and troubleshoot problems. This chapter discusses tools for monitoring database performance and activity.

Also, be sure to review Recommended Monitoring and Maintenance Tasks for monitoring activities you can script to quickly detect problems in the system.

Parent topic: Managing a Greenplum System

Monitoring Database Activity and Performance

Greenplum Database includes an optional system monitoring and management database, gpperfmon, that administrators can enable. The gpperfmon_install command-line utility creates the gpperfmon database and enables data collection agents that collect and store query and system metrics in the database. Administrators can query metrics in the gpperfmon database. See the documentation for the gpperfmon database in the Greenplum Database Reference Guide.

Tanzu Greenplum Command Center, an optional web-based interface, graphically displays the metrics collected in the gpperfmon database and provides additional system management tools. Download the Greenplum Command Center package from VMware Tanzu Network and view the documentation at the Greenplum Command Center Documentation web site.

Monitoring System State

As a Greenplum Database administrator, you must monitor the system for problem events such as a segment going down or running out of disk space on a segment host. The following topics describe how to monitor the health of a Greenplum Database system and examine certain state information for a Greenplum Database system.

• Enabling System Alerts and Notifications
• Checking System State
• Checking Disk Space Usage
• Checking for Data Distribution Skew
• Viewing Metadata Information about Database Objects
• Viewing Session Memory Usage Information
• Viewing Query Workfile Usage Information

Enabling System Alerts and Notifications

You can configure a Greenplum Database system to trigger SNMP (Simple Network Management Protocol) alerts or send email notifications to system administrators if certain database events occur. These events include:

• All PANIC-level error conditions
• All FATAL-level error conditions
• ERROR-level conditions that are “internal errors” (for example, SIGSEGV errors)
• Database system shutdown and restart
• Segment failure and recovery
• Standby master out-of-sync conditions
• Master host manual shutdown or other software problem (in certain failure scenarios, Greenplum Database cannot send an alert or notification)

This topic includes the following sub-topics:

• Using SNMP with a Greenplum Database System
• Enabling Email Notifications

Note that SNMP alerts and email notifications report the same event information. There is no difference in the event information that either tool reports. For information about the SNMP event information, see Greenplum Database SNMP OIDs and Error Codes.

Using SNMP with a Greenplum Database System

Greenplum Database supports SNMP to monitor the state of a Greenplum Database system using MIBs (Management Information Bases). MIBs are collections of objects that describe an SNMP-manageable entity — in this case, a Greenplum Database system.

The Greenplum Database SNMP support allows a Network Management System to obtain information about the hardware, operating system, and Greenplum Database from the same port (161) and IP address. It also enables the auto-discovery of Greenplum Database instances.

Prerequisites

Before setting up SNMP support on Greenplum Database, ensure SNMP is installed on the master host. If the snmpd file is not present in the /usr/sbin directory, then SNMP is not installed on the system. Depending on the platform on which you are running Greenplum Database, install the following:

1 SNMP is installed by default on SUSE platforms.

The snmp.conf configuration file is located in /etc/snmp/.

Pre-installation Tasks

After you establish that SNMP is on the master host, log in as root, open a text editor, and edit the path\_to/snmp/snmpd.conf file. To use SNMP with Greenplum Database, the minimum configuration change required to the snmpd.conf file is specifying a community name. For example:

rocommunity public

Note: Replace public with the name of your SNMP community. You should also configure syslocation and syscontact. Configure other SNMP settings as required for your environment and save the file.

For more information about the snmpd.conf file, enter:

man snmpd.conf

Note: On SUSE Linux platforms, make sure to review and configure security settings in the snmp.conf file so snmpd accepts connections from sub-agents and returns all available Object IDs (OIDs).

After you finish configuring the snmpd.conf file, start the system snmpd daemon:

# /sbin/chkconfig snmpd on

Then, verify the system snmpd daemon is running. Enter:

# snmpwalk -v 1 -c <community_name> localhost .1.3.6.1.2.1.1.1.0

For example:

# snmpwalk -v 1 -c public localhost .1.3.6.1.2.1.1.1.0

If this command returns “Timeout: No Response from localhost“, then the system snmpd daemon is not running. If the daemon is running, output similar to the following displays:

SNMPv2-MIB::sysDescr.0 = STRING: Linux *hostname* 
2.6.18-92.el5 #1 SMP Tue Jun 10 18:51:06 EDT 2016 x86_64

Setting up SNMP Notifications

1. To configure a Greenplum Database system to send SNMP notifications when alerts occur, set the following parameters on the Greenplum Database master host with the gpconfig utility:

   • gp_snmp_community: Set this parameter to the community name you specified for your environment.

   • gp_snmp_monitor_address: Enter the hostname:port of your network monitor application. Typically, the port number is 162. If there are multiple monitor addresses, separate them with a comma.

   • gp_snmp_use_inform_or_trap: Enter either trap or inform. Trap notifications are SNMP messages sent from one application to another (for example, between Greenplum Database and a network monitoring application). These messages are unacknowledged by the monitoring application, but generate less network overhead.

     Inform notifications are the same as trap messages, except the application sends an acknowledgement to the application that generated the alert. In this case, the monitoring application sends acknowledgement messages to Greenplum Database-generated trap notifications. While inform messages create more overhead, they inform Greenplum Database the monitoring application has received the traps.

     The following example commands set the server configuration parameters with the Greenplum Database gpconfig utility:

     $ gpconfig -c gp_snmp_community -v public --masteronly 
     $ gpconfig -c gp_snmp_monitor_address -v mdw:162 --masteronly 
     $ gpconfig -c gp_snmp_use_inform_or_trap -v trap --masteronly

2. To test SNMP notifications, you can use the snmptrapd trap receiver. As root, enter:

   # /usr/sbin/snmptrapd -m ALL -Lf ~/<filename>.log

   -Lf indicates that traps are logged to a file. -Le indicates that traps are logged to stderr instead. -m ALL loads all available MIBs (you can also specify individual MIBs if required).

Enabling Email Notifications

Complete the following steps to enable Greenplum Database to send email notifications to system administrators whenever certain database events occur.

1. Open $MASTER_DATA_DIRECTORY/postgresql.conf in a text editor.

2. In the EMAIL ALERTS section, uncomment the following parameters and enter the appropriate values for your email server and domain. For example:

   gp_email_smtp_server='smtp.company.com:25'
   gp_email_smtp_userid='gpadmin@example.com'
   gp_email_smtp_password='mypassword'
   gp_email_from='Greenplum Database <gpadmin@example.com>'
   gp_email_to='dba@example.com;John Smith <jsmith@example.com>'

   You may create specific email accounts or groups in your email system that send and receive email alerts from the Greenplum Database system. For example:

   gp_email_from='GPDB Production Instance <gpdb@example.com>'
   gp_email_to='gpdb_dba_group@example.com'

   You can also specify multiple email addresses for both gp_email parameters. Use a semi-colon ( ; ) to separate each email address. For example:

   gp_email_to='gpdb_dba_group@example.com;admin@example.com'

3. Save and close the postgresql.conf file.

4. Reload the Greenplum Database postgresql.conf file:

   $ gpstop -u

Testing Email Notifications

The Greenplum Database master host must be able to connect to the SMTP email server you specify for the gp_email_smtp_server parameter. To test connectivity, use the ping command:

$ ping <<my_email_server>>

If the master host can contact the SMTP server, run this psql command to log into a database and test email notification with the gp_elog function:

$ psql -d <<testdb>> -U gpadmin -c "SELECT gp_elog('Test GPDB Email',true);"

The address you specified for the gp_email_to parameter should receive an email with Test GPDB Email in the subject line.

Note: If you have difficulty sending and receiving email notifications, verify the security settings for you organization’s email server and firewall.

Checking System State

A Greenplum Database system is comprised of multiple PostgreSQL instances (the master and segments) spanning multiple machines. To monitor a Greenplum Database system, you need to know information about the system as a whole, as well as status information of the individual instances. The gpstate utility provides status information about a Greenplum Database system.

Viewing Master and Segment Status and Configuration

The default gpstate action is to check segment instances and show a brief status of the valid and failed segments. For example, to see a quick status of your Greenplum Database system:

$ gpstate

To see more detailed information about your Greenplum Database array configuration, use gpstate with the -s option:

$ gpstate -s

Viewing Your Mirroring Configuration and Status

If you are using mirroring for data redundancy, you may want to see the list of mirror segment instances in the system, their current synchronization status, and the mirror to primary mapping. For example, to see the mirror segments in the system and their status:

$ gpstate -m

To see the primary to mirror segment mappings:

$ gpstate -c

To see the status of the standby master mirror:

$ gpstate -f

Checking Disk Space Usage

A database administrator’s most important monitoring task is to make sure the file systems where the master and segment data directories reside do not grow to more than 70 percent full. A filled data disk will not result in data corruption, but it may prevent normal database activity from continuing. If the disk grows too full, it can cause the database server to shut down.

You can use the gp_disk_free external table in the gp_toolkit administrative schema to check for remaining free space (in kilobytes) on the segment host file systems. For example:

=# SELECT * FROM gp_toolkit.gp_disk_free 
   ORDER BY dfsegment;

Checking Sizing of Distributed Databases and Tables

The gp_toolkit administrative schema contains several views that you can use to determine the disk space usage for a distributed Greenplum Database database, schema, table, or index.

For a list of the available sizing views for checking database object sizes and disk space, see the Greenplum Database Reference Guide.

Viewing Disk Space Usage for a Database

To see the total size of a database (in bytes), use the gp_size_of_database view in the gp_toolkit administrative schema. For example:

=> SELECT * FROM gp_toolkit.gp_size_of_database 
   ORDER BY sodddatname;

Viewing Disk Space Usage for a Table

The gp_toolkit administrative schema contains several views for checking the size of a table. The table sizing views list the table by object ID (not by name). To check the size of a table by name, you must look up the relation name (relname) in the pg_class table. For example:

=> SELECT relname AS name, sotdsize AS size, sotdtoastsize 
   AS toast, sotdadditionalsize AS other 
   FROM gp_toolkit.gp_size_of_table_disk as sotd, pg_class 
   WHERE sotd.sotdoid=pg_class.oid ORDER BY relname;

For a list of the available table sizing views, see the Greenplum Database Reference Guide.

Viewing Disk Space Usage for Indexes

The gp_toolkit administrative schema contains a number of views for checking index sizes. To see the total size of all index(es) on a table, use the gp_size_of_all_table_indexes view. To see the size of a particular index, use the gp_size_of_index view. The index sizing views list tables and indexes by object ID (not by name). To check the size of an index by name, you must look up the relation name (relname) in the pg_class table. For example:

=> SELECT soisize, relname as indexname
   FROM pg_class, gp_toolkit.gp_size_of_index
   WHERE pg_class.oid=gp_size_of_index.soioid 
   AND pg_class.relkind='i';

Checking for Data Distribution Skew

All tables in Greenplum Database are distributed, meaning their data is divided evenly across all of the segments in the system. Unevenly distributed data may diminish query processing performance. A table’s distribution policy is determined at table creation time. For information about choosing the table distribution policy, see the following topics:

• Viewing a Table’s Distribution Key
• Viewing Data Distribution
• Checking for Query Processing Skew

The gp_toolkit administrative schema also contains a number of views for checking data distribution skew on a table. For information about how to check for uneven data distribution, see the Greenplum Database Reference Guide.

Viewing a Table’s Distribution Key

To see the columns used as the data distribution key for a table, you can use the \d+ meta-command in psql to examine the definition of a table. For example:

=# `\d+ sales
`                Table "retail.sales"
 Column      |     Type     | Modifiers | Description
-------------+--------------+-----------+-------------
 sale_id     | integer      |           |
 amt         | float        |           |
 date        | date         |           |
Has OIDs: no
Distributed by: (sale_id)

Viewing Data Distribution

To see the data distribution of a table’s rows (the number of rows on each segment), you can run a query such as:

=# SELECT gp_segment_id, count(*) 
   FROM <table_name> GROUP BY gp_segment_id;

A table is considered to have a balanced distribution if all segments have roughly the same number of rows.

Checking for Query Processing Skew

When a query is being processed, all segments should have equal workloads to ensure the best possible performance. If you identify a poorly-performing query, you may need to investigate further using the EXPLAIN command. For information about using the EXPLAIN command and query profiling, see Query Profiling.

Query processing workload can be skewed if the table’s data distribution policy and the query predicates are not well matched. To check for processing skew, you can run a query such as:

=# SELECT gp_segment_id, count(*) FROM <table_name>
   WHERE <column>='<value>' GROUP BY gp_segment_id;

This will show the number of rows returned by segment for the given WHERE predicate.

Avoiding an Extreme Skew Warning

You may receive the following warning message while executing a query that performs a hash join operation:

Extreme skew in the innerside of Hashjoin

This occurs when the input to a hash join operator is skewed. It does not prevent the query from completing successfully. You can follow these steps to avoid skew in the plan:

1. Ensure that all fact tables are analyzed.
2. Verify that any populated temporary table used by the query is analyzed.
3. View the EXPLAIN ANALYZE plan for the query and look for the following:
   • If there are scans with multi-column filters that are producing more rows than estimated, then set the gp_selectivity_damping_factor server configuration parameter to 2 or higher and retest the query.
   • If the skew occurs while joining a single fact table that is relatively small (less than 5000 rows), set the gp_segments_for_planner server configuration parameter to 1 and retest the query.
4. Check whether the filters applied in the query match distribution keys of the base tables. If the filters and distribution keys are the same, consider redistributing some of the base tables with different distribution keys.
5. Check the cardinality of the join keys. If they have low cardinality, try to rewrite the query with different joining columns or or additional filters on the tables to reduce the number of rows. These changes could change the query semantics.

Viewing Metadata Information about Database Objects

Greenplum Database tracks various metadata information in its system catalogs about the objects stored in a database, such as tables, views, indexes and so on, as well as global objects such as roles and tablespaces.

Viewing the Last Operation Performed

You can use the system views pg_stat_operations and pg_stat_partition_operations to look up actions performed on an object, such as a table. For example, to see the actions performed on a table, such as when it was created and when it was last vacuumed and analyzed:

=> SELECT schemaname as schema, objname as table, 
   usename as role, actionname as action, 
   subtype as type, statime as time 
   FROM pg_stat_operations 
   WHERE objname='cust';
 schema | table | role | action  | type  | time
--------+-------+------+---------+-------+--------------------------
  sales | cust  | main | CREATE  | TABLE | 2016-02-09 18:10:07.867977-08
  sales | cust  | main | VACUUM  |       | 2016-02-10 13:32:39.068219-08
  sales | cust  | main | ANALYZE |       | 2016-02-25 16:07:01.157168-08
(3 rows)

Viewing the Definition of an Object

To see the definition of an object, such as a table or view, you can use the \d+ meta-command when working in psql. For example, to see the definition of a table:

=> \d+ <mytable>

Viewing Session Memory Usage Information

You can create and use the session_level_memory_consumption view that provides information about the current memory utilization for sessions that are running queries on Greenplum Database. The view contains session information and information such as the database that the session is connected to, the query that the session is currently running, and memory consumed by the session processes.

• Creating the session_level_memory_consumption View
• The session_level_memory_consumption View

Creating the session_level_memory_consumption View

To create the session_level_memory_consumption view in a Greenplum Database, run the script $GPHOME/share/postgresql/contrib/gp_session_state.sql once for each database. For example, to install the view in the database testdb, use this command:

$ psql -d testdb -f $GPHOME/share/postgresql/contrib/gp_session_state.sql

The session_level_memory_consumption View

The session_level_memory_consumption view provides information about memory consumption and idle time for sessions that are running SQL queries.

When resource queue-based resource management is active, the column is_runaway indicates whether Greenplum Database considers the session a runaway session based on the vmem memory consumption of the session’s queries. Under the resource queue-based resource management scheme, Greenplum Database considers the session a runaway when the queries consume an excessive amount of memory. The Greenplum Database server configuration parameter runaway_detector_activation_percent governs the conditions under which Greenplum Database considers a session a runaway session.

The is_runaway, runaway_vmem_mb, and runaway_command_cnt columns are not applicable when resource group-based resource management is active.

Viewing Query Workfile Usage Information

The Greenplum Database administrative schema gp_toolkit contains views that display information about Greenplum Database workfiles. Greenplum Database creates workfiles on disk if it does not have sufficient memory to execute the query in memory. This information can be used for troubleshooting and tuning queries. The information in the views can also be used to specify the values for the Greenplum Database configuration parameters gp_workfile_limit_per_query and gp_workfile_limit_per_segment.

These are the views in the schema gp_toolkit:

• The gp_workfile_entries view contains one row for each operator using disk space for workfiles on a segment at the current time.
• The gp_workfile_usage_per_query view contains one row for each query using disk space for workfiles on a segment at the current time.
• The gp_workfile_usage_per_segment view contains one row for each segment. Each row displays the total amount of disk space used for workfiles on the segment at the current time.

For information about using gp_toolkit, see Using gp_toolkit.

Viewing the Database Server Log Files

Every database instance in Greenplum Database (master and segments) runs a PostgreSQL database server with its own server log file. Daily log files are created in the pg_log directory of the master and each segment data directory.

Log File Format

The server log files are written in comma-separated values (CSV) format. Some log entries will not have values for all log fields. For example, only log entries associated with a query worker process will have the slice_id populated. You can identify related log entries of a particular query by the query’s session identifier (gp_session_id) and command identifier (gp_command_count).

The following fields are written to the log:

Searching the Greenplum Server Log Files

Greenplum Database provides a utility called gplogfilter can search through a Greenplum Database log file for entries matching the specified criteria. By default, this utility searches through the Greenplum Database master log file in the default logging location. For example, to display the last three lines of each of the log files under the master directory:

$ gplogfilter -n 3

To search through all segment log files simultaneously, run gplogfilter through the gpssh utility. For example, to display the last three lines of each segment log file:

$ gpssh -f seg_host_file

=> source /usr/local/greenplum-db/greenplum_path.sh
=> gplogfilter -n 3 /gpdata/gp*/pg_log/gpdb*.log

Using gp_toolkit

Use the Greenplum Database administrative schema gp_toolkit to query the system catalogs, log files, and operating environment for system status information. The gp_toolkit schema contains several views you can access using SQL commands. The gp_toolkit schema is accessible to all database users. Some objects require superuser permissions. Use a command similar to the following to add the gp_toolkit schema to your schema search path:

=> ALTER ROLE myrole SET search_path TO myschema,gp_toolkit;

For a description of the available administrative schema views and their usages, see the Greenplum Database Reference Guide.

Greenplum Database SNMP OIDs and Error Codes

When a Greenplum Database system is configured to trigger SNMP alerts or send email notifications to system administrators if certain database events occur, the alerts and notifications containObject IDs (OIDs) and SQL error codes.

• Greenplum Database SNMP OIDs
• SQL Standard Error Codes

For information about enabling Greenplum Database to use SNMP, see Enabling System Alerts and Notifications

Greenplum Database SNMP OIDs

This is the Greenplum Database OID hierarchy structure:

iso(1)
identified-organization(3)
dod(6)
internet(1)
private(4)
enterprises(1)
gpdbMIB(31327)
gpdbObjects(1)
gpdbAlertMsg(1)

gpdbAlertMsg

1.3.6.1.4.1.31327.1.1: STRING: alert message text

gpdbAlertSeverity

1.3.6.1.4.1.31327.1.2: INTEGER: severity level

gpdbAlertSeverity can have one of the following values:
```
gpdbSevUnknown(0)
gpdbSevOk(1)
gpdbSevWarning(2)
gpdbSevError(3)
gpdbSevFatal(4)
gpdbSevPanic(5)
gpdbSevSystemDegraded(6)
gpdbSevSystemDown(7)
```

gpdbAlertSqlstate

1.3.6.1.4.1.31327.1.3: STRING: SQL standard error codes

For a list of codes, see SQL Standard Error Codes.

gpdbAlertDetail

1.3.6.1.4.1.31327.1.4: STRING: detailed alert message text

gpdbAlertSqlStmt

1.3.6.1.4.1.31327.1.5: STRING: SQL statement generating this alert if applicable

gpdbAlertSystemName

1.3.6.1.4.1.31327.1.6: STRING: hostname

SQL Standard Error Codes

The following table lists all the defined error codes. Some are not used, but are defined by the SQL standard. The error classes are also shown. For each error class there is a standard error code having the last three characters 000. This code is used only for error conditions that fall within the class but do not have any more-specific code assigned.

The PL/pgSQL condition name for each error code is the same as the phrase shown in the table, with underscores substituted for spaces. For example, code 22012, DIVISION BY ZERO, has condition name DIVISION_BY_ZERO. Condition names can be written in either upper or lower case.

Note: PL/pgSQL does not recognize warning, as opposed to error, condition names; those are classes 00, 01, and 02.