feat(log): export the log records via OpenTelemetry by armru · Pull Request #343 · cloudnative-pg/machinery

armru · 2026-10-28T11:33:08Z

Collecting the logs of a process today needs something reading its
output from outside, usually a DaemonSet on every node, which is not
available everywhere.

This adds WithOpenTelemetry() to Flags.ConfigureLogging: when
OTEL_EXPORTER_OTLP_ENDPOINT (or its logs variant) is set, the records
are also pushed via OTLP to an OpenTelemetry collector, configured with
the standard OTEL_* variables, using http/protobuf unless grpc is
requested. The log destination is unchanged.

The export is a zap core teed with the existing one inside the sampler,
so both outputs get the same records at the same log level. Structs are
exported as maps with the keys of the JSON output, and the custom debug
and trace levels get the DEBUG severity. The records are batched in the
background and dropped, never blocking, when the collector cannot keep
up. ShutdownOpenTelemetry flushes the queue before exiting, and
ForwardToOpenTelemetry exports the JSON records that a spawned process
wrote on the shared log destination, with the time, level and fields
they were written with.

A process that already configures OpenTelemetry for its traces and
metrics passes its own provider with WithOpenTelemetryLoggerProvider
instead, so the records share the resource and the exporter selection
of its other signals, and the caller owns the shutdown.

The CloudNativePG side is in cloudnative-pg/cloudnative-pg#11783, the
Klio one in cloudnative-pg/klio#562.

Refs cloudnative-pg/cloudnative-pg#11085
Refs cloudnative-pg/klio#549

…s written The instance manager decoded the records that wal-archive and wal-restore write on the log pipe, even with the export disabled, and exported them with the time it forwarded them at. It now hands the JSON line to ForwardToOpenTelemetry, from cloudnative-pg/machinery#343, so the records keep their own time and nothing is decoded while the export is disabled. The documentation now lists the timestamp among the exported fields, and names the stack trace of a record without an error. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

cloudnative-pg/machinery#343 gained an option for processes that build their own log provider, and WithOpenTelemetry now reaches the export through the same helper. Pin its new head, so this branch builds and runs against the code that is going to be merged. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

armru · 2026-11-03T14:07:41Z

Klio can reuse this framework with one additive option, which is now part
of this PR: WithOpenTelemetryLoggerProvider tees the records into a
provider the caller builds. Klio needs it because its traces and metrics
build their resource and pick their exporters themselves, and the logs
must share both. WithOpenTelemetry() is unchanged for CloudNativePG.

Verified in cloudnative-pg/klio#562 with unit tests and the
OpenTelemetry e2e features on kind.

Klio exports its traces and metrics through OpenTelemetry, but its logs only reach the standard output, so collecting them still needs a node agent or a sidecar. Every Klio process now builds a log provider with the resource and the exporter selection of its other signals, and hands it to the logging of machinery before configuring it, so the records written on the standard output are exported too. OTEL_LOGS_EXPORTER enables the export and is unset by default, so a configuration written for the traces and the metrics does not start sending the logs. Each command exports its own records and flushes them before exiting, including the short-lived commands that the plugin sidecar spawns, while `retention get` now runs without the OTEL_* variables like `get-metadata`, as its output is captured. The operator moves from the controller-runtime logger to the one of machinery, which the server and the sidecar already use: it now logs in JSON at the info level by default, and keeps accepting the --zap-* arguments along with --log-level. This needs cloudnative-pg/machinery#343, pinned to its head until it is released. Closes #549 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

…s written The instance manager decoded the records that wal-archive and wal-restore write on the log pipe, even with the export disabled, and exported them with the time it forwarded them at. It now hands the JSON line to ForwardToOpenTelemetry, from cloudnative-pg/machinery#343, so the records keep their own time and nothing is decoded while the export is disabled. The documentation now lists the timestamp among the exported fields, and names the stack trace of a record without an error. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

cloudnative-pg/machinery#343 gained an option for processes that build their own log provider, and WithOpenTelemetry now reaches the export through the same helper. Pin its new head, so this branch builds and runs against the code that is going to be merged. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Klio exports its traces and metrics through OpenTelemetry, but its logs only reach the standard output, so collecting them still needs a node agent or a sidecar. Every Klio process now builds a log provider with the resource and the exporter selection of its other signals, and hands it to the logging of machinery before configuring it, so the records written on the standard output are exported too. OTEL_LOGS_EXPORTER enables the export and is unset by default, so a configuration written for the traces and the metrics does not start sending the logs. Each command exports its own records and flushes them before exiting, including the short-lived commands that the plugin sidecar spawns, while `retention get` now runs without the OTEL_* variables like `get-metadata`, as its output is captured. The operator moves from the controller-runtime logger to the one of machinery, which the server and the sidecar already use: it now logs in JSON at the info level by default, and keeps accepting the --zap-* arguments along with --log-level. This needs cloudnative-pg/machinery#343, pinned to its head until it is released. Closes #549 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Collecting the logs of a process today requires something reading its standard output from outside, usually a DaemonSet tailing the container logs on every node, which is not available in every environment. This adds an optional `WithOpenTelemetry()` argument to `Flags.ConfigureLogging`. When the standard `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` environment variable is set, the records are also pushed via OTLP to an OpenTelemetry collector, while the log destination keeps getting them unchanged. The exporter takes its configuration from the standard `OTEL_*` variables, using `http/protobuf` unless `grpc` is requested, as the OpenTelemetry specification prescribes. The records reach the zap bridge of OpenTelemetry through a core teed with the existing one, inside the duplicate-message sampler, so both outputs get the same records at the same log level. The core maps the custom debug and trace levels to the zap debug level, which the bridge understands, and it turns reflected values into their JSON form, so a struct such as a PostgreSQL log record is exported as a map with the keys it has on the log destination instead of a Go-formatted string. A batch processor exports the records in the background, dropping them instead of blocking the caller when the collector cannot keep up. `ShutdownOpenTelemetry` exports the queued records and stops the export before the process exits, and `OpenTelemetryLogger` writes only to the export, for records already written to the log destination by another process. Refs cloudnative-pg/cloudnative-pg#11085 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

OpenTelemetryLogger exported the records written on the log destination by a spawned process through a logr logger, so they took the time they were forwarded at, and every caller had to decode them first. ForwardToOpenTelemetry takes the JSON line instead: the record keeps the time, level and logger name it was written with, its integers stay integers, and nothing is decoded while the export is disabled. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

A process that already configures OpenTelemetry for its metrics and traces builds their resource and picks their exporters from the environment itself. With WithOpenTelemetry its log records would leave through a second provider, without that resource and ignoring OTEL_LOGS_EXPORTER, so a backend could not correlate them with the other signals of the same process. WithOpenTelemetryLoggerProvider tees the records into a provider the caller creates instead, keeping the level filter and the field mapping of WithOpenTelemetry. The caller owns that provider, so ShutdownOpenTelemetry leaves it running. Refs cloudnative-pg/klio#549 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

…ests The exporter prefers a signal-specific variable such as OTEL_EXPORTER_OTLP_LOGS_ENDPOINT over the generic endpoint the tests point at their receiver. Dagger sets those variables in the containers it runs, so in CI the records went to its own collector and the tests found nothing. Clear every OTEL_* variable before each test. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Reflected values were exported through json.Unmarshal into any, which turns every number into a float64: the integers above 2^53 are rounded, and the others reach the OpenTelemetry bridge as doubles. The records forwarded from other processes kept the numbers of the top level only, so the nested ones were exported as strings. Decode both with UnmarshalCaseSensitivePreserveInts of sigs.k8s.io/json, which gives an int64 when the number fits and a float64 otherwise, at any depth, as the Kubernetes objects are decoded. The tests of the forwarded records only checked small numbers, which a plain float64 decoding also gets right. Forward a record with an integer beyond 2^53 at the top level, in an object and in a list, and check that it is exported as an exact integer. Assisted-by: Claude Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

The OpenTelemetry export of the log records started whenever an OTLP endpoint was set, so a collector configured for the traces and the metrics could not be kept from receiving the logs. OTEL_SDK_DISABLED=true, the standard switch that turns off every signal, and OTEL_LOGS_EXPORTER=none, the standard way to turn off the export of the logs only, now disable it even when the endpoint is set. As the specification asks, "true" is matched case-insensitively and the exporter name is interpreted in a case-insensitive manner, so "None" and "NONE" work as well. Assisted-by: Claude Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com>

armru requested a review from a team as a code owner October 28, 2026 11:33

This was referenced Oct 28, 2026

feat: export the logs via OpenTelemetry cloudnative-pg/cloudnative-pg#11783

Open

feat(logs): propagate OTel log endpoint to instance pods cloudnative-pg/cloudnative-pg#11089

Draft

feat(log): add OpenTelemetry log for direct OTLP export #290

Draft

maxlengdell approved these changes Oct 30, 2026

View reviewed changes

gbartolini mentioned this pull request Oct 31, 2026

Evaluate reuse of machinery's OpenTelemetry log export to complete native OTEL support (logs, metrics, traces) cloudnative-pg/klio#549

Open

armru force-pushed the dev/11085 branch from 32c682f to 8265ac9 Compare November 3, 2026 14:05

armru mentioned this pull request Nov 3, 2026

feat: export the logs via OpenTelemetry cloudnative-pg/klio#562

Open

fcanovai force-pushed the dev/11085 branch from 4d494a4 to 0f58adc Compare November 10, 2026 17:10

fcanovai approved these changes Nov 11, 2026

View reviewed changes

armru and others added 6 commits November 11, 2026 13:57

gbartolini force-pushed the dev/11085 branch from 0f58adc to 050de2c Compare November 11, 2026 12:57

Navigation Menu

Repository navigation

feat(log): export the log records via OpenTelemetry - #343

feat(log): export the log records via OpenTelemetry#343
armru wants to merge 6 commits into
mainfrom
dev/11085

armru commented Oct 28, 2026 •

edited

Loading

Uh oh!

armru commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

armru commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

armru commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

armru commented Oct 28, 2026 •

edited

Loading