6.12. Event Logger#

The event logger allows query completion events and all related information to be logged into a PostgreSQL database.

Requirements#

  • PostgreSQL database with network connectivity from the coordinator
  • PostgreSQL JDBC driver JAR copied into plugin/eventlistener on all nodes

Installation and Configuration#

The event logger is implemented as a event listener implementation and can be enabled by creating a configuration file called etc/event-listener.properties. The name needs to be set to event-logger. The configuration includes the JDBC connection URL, username and password for accessing the PostgreSQL database.

event-listener.name=event-logger
jdbc.url=jdbc:postgresql://localhost:5432/postgres
jdbc.user=test_psql
jdbc.password=test12

The user needs to have sufficient rights to create tables and insert data in the configured schema.

Logged Information#

For each completed query, a row is created in the completed_queries table. It captures everything that Presto emits – query, user, metadata, stats, performance related attributes, resource consumption, start time, end time, and much more.

The following columns are populated with the relevant information:

  • query_id
  • catalog
  • schema
  • principal
  • user_agent
  • client_info
  • source
  • environment
  • remote_client_address
  • server_version
  • usr
  • query_state
  • query
  • query_plan
  • total_rows
  • total_bytes
  • output_rows
  • output_bytes
  • written_rows
  • written_bytes
  • cpu_time_ms
  • wall_time_ms
  • queued_time_ms
  • peak_user_memory_bytes
  • peak_total_non_revocable_memory_bytes
  • peak_task_user_memory
  • peak_task_total_memory
  • physical_input_bytes
  • physical_input_rows
  • internal_network_bytes
  • internal_network_rows
  • cumulative_memory
  • completed_splits
  • plan_node_stats_and_costs
  • stage_gc_statistics
  • cpu_time_distribution
  • operator_summaries
  • resource_waiting_time
  • analysis_time
  • execution_time
  • create_time
  • end_time
  • accessed_metadata

Analyzing the Event Log#

To analyze and query the event log, you can create a catalog with the PostgreSQL connector using the same JDBC connection parameters. This allows you to use the Presto CLI or any other application connected to the catalog to create queries, dashboards, perform ad hoc analysis and more.

Example use cases:

  • measure query performance numbers and trends
  • understand impact of different cluster configurations
  • enable cluster workload management and resource consumption