Usage metrics#

Usage metrics collection enables Presto to log information about all nodes in the cluster. This feature is mostly used for Mission Control and other management solutions. Users can use the resulting log information for their own monitoring and operational purposes.

Configuration#

Usage metrics collection is configured by a number of properties, that you can add in etc/config.properties to configure details.

Usage Metrics Configuration Properties#

Property name

Description

Default value

usage-metrics.log.path

Base path of the usage log files

var/log/presto/usage-metrics.log

usage-metrics.log.max-size

Maximum size of a single usage log file

100 MB, minimum 1MB

usage-metrics.log.max-history

Maximum number of usage log files

1000

usage-metrics.cloud-watch-logs-directory

Path to the directory on the node in which log files are stored, for example, /var/log/presto/cloudwatch. Amazon CloudWatch is automatically enabled by the Presto CFT and the usage metrics data is available in usage-metrics-* log streams.

usage-metrics.gathering.initial-delay

The initial delay to before tracking usage, allowing the cluster to start up before metrics gathering starts.

1 min

usage-metrics.gathering.interval

Length of the interval between usage metric log entry creation.

1 min

usage-metrics.gatherer-threads

Number of threads used to gather and write all usage information.

100, minimum 10

usage-metrics.cluster-usage-resource.enabled

Expose the usage metrics, aggregated for all nodes since cluster start, on a REST API end point on the coordinator at /v1/cluster/usage

false

Logging details and analysis#

The logged information includes details about number of nodes, number of cores, start time, cpu time, and other data about the cluster and nodes in the cluster. The log uses JSON format for each entry.

The JSON information can be inspected manually, ingested in any logging analysis or management tool, and also parsed with the metrics parser tools available from Starburst. The parser is a command line application that can be run directly or with the java.

mv starburst-usage-metrics-parser-*-executable.jar usage-metrics-parser
chmod a+x usage-metrics-parser
./usage-metrics-parser
# or
java -jar usage-metrics-parser

Save the logs to local empty folder and use the parser with appropriate arguments:

usage-metrics-parser --from <from> --to <to> <path>

The parameters <from>/<to> define date-time range and <path> points to folder with the log files. The resulting output shows the analysis results:

cluster restarts: 1, cpu time: 265.55s, available cpu time: 93130.57s, cpu utilization: 0.29%, min cores: 16, max cores: 16

Persisting Metrics#

Understanding the usage of your cluster requires persisting the usage metrics. By default the metrics are not persisted.

The simplest mechanism to persist metrics is to configure the path of the log file to a location that is separate from the filesystem of the coordinator. Typically this is a network location that is mounted.

The CFT setup automatically configures persisting the metrics in CloudWatch. Users of Amazon EKS can also use CloudWatch as well.

Logs exported from CloudWatch are decorated and have to be pre-processed to limit information to the raw format displayed in the preceding section to process it with the metrics parser.

Log management solutions integrated with other cloud platforms, or available separately, allow persisting the metrics by capturing the log file or regularly inspecting the REST endpoint.