Log ingestion: you are probably paying to store logs you will never read

The default way to adopt log management is to ship everything and search it later. It is the path every vendor’s quickstart puts you on, and it is the reason log bills surprise people: ingestion is priced by volume, so “ship everything” is a spending decision disguised as a configuration default.

The uncomfortable part is that most of that volume is never read. Nobody greps last Tuesday’s 200 OK access lines. They are shipped, indexed, retained and billed on the chance that someone might, and that chance is close to zero for the overwhelming majority of lines a healthy system produces.

There is a better division of labour available, and it does not involve giving up any visibility. Some logs are worth storing because you will read them during an incident. Others are only worth counting — you never need the line, you need to know how often it happened. Those are two different jobs, and conflating them is what makes log platforms expensive.

The two questions to ask of every log source

Before adding a source, answer two questions separately, because they have separate answers:

1. Will a human read these lines during an incident? If yes, ship them. Stack traces, error logs, audit trails and anything you would want to reconstruct a sequence of events from belong in storage.
2. Do I want to alert on how often this appears? If yes, count it. That is a metric, and it does not require the line to be stored at all.

Most sources answer yes to exactly one of these. Access logs are the clearest case: you want the request rate and the 5xx rate — both metrics — and you almost never want to read an individual successful request. Application error logs are the opposite: the count matters for alerting, and the line matters enormously once the alert fires.

Counting without shipping

This is the part people miss, so it is worth being concrete. In Glouton, log collection and log-to-metric counting share one pipeline and one configuration block, but neither requires the other. A receiver that defines metrics keeps producing them with shipping switched off entirely.

log:
  opentelemetry:
    receivers:
      app:
        include:
          - /var/log/myapp/*.log
        send_logs: false
        metrics:
          - metric: app_errors_count
            regex: '\[error\]'

That configuration reads the application log, counts lines matching \[error\] per second, publishes app_errors_count as a metric — and ships nothing. No log line leaves the host. You get an alertable error rate, a 13-month history of it, and a zero-byte ingestion bill for that source.

The knob is send_logs, which defaults to true. Three ways to turn shipping off, depending on scope:

The metrics are unaffected in all three cases. That is the whole point: you are not trading visibility for cost, you are choosing the cheaper representation of the same signal.

What “count it” gets you that a log search does not

An error rate as a metric is not a worse version of a log query. For the questions you actually ask on call, it is a better one.

• You can alert on it. Alerting on a metric crossing a threshold is a first-class operation. Alerting on log volume means running a search on a schedule, which is slower, more expensive and more fragile.
• It survives longer. Logs are retained for 30 days. Metrics on the Professional plan are kept for 13 months at full resolution, so “is this worse than the last deploy?” and “is this worse than last quarter?” become the same question. On Starter both are 30 days, and this particular argument does not apply — the other two still do.
• It graphs next to everything else. An error rate on the same dashboard as CPU, memory and request latency is how you notice that errors climb ten minutes after memory does. A log search in another tab is not.

The line itself still matters — but it matters after the alert fires, for a narrow window, on one host. That is a much smaller volume than shipping everything continuously against the possibility.

Drop it before it costs anything

For the sources you do ship, filtering belongs at the agent. A line dropped by a global filter is never transmitted, so it never counts toward ingestion. Filtering after the data has arrived saves you nothing at all.

Two mechanisms, both applying to every source — auto-discovered services, file receivers, container logs and OTLP:

log.opentelemetry.global_filters:
  exclude:
    match_type: regexp
    severity_texts:
      - 'debug'
      - 'trace'

Excluding debug and trace in production is the single highest-yield filter most systems have available, and it is close to free in information terms: if debug logging mattered in production you would have turned it on deliberately.

For anything the severity field cannot express, there are OTTL expressions:

log.opentelemetry.global_filters:
  log_record:
    - "Hour(Now()) < 7 or Hour(Now()) > 19"

That one drops records outside working hours. Whether it is a good idea depends entirely on your system — it is a poor fit for anything with an overnight batch window, and a reasonable one for an internal tool nobody uses at night. The mechanism is the point, not that particular condition.

The cardinality trap

Log-to-metric definitions accept an attributes list, which produces one series per distinct value seen for that key. It is genuinely useful and it is the easiest way to make a mess.

metrics:
  - metric: http_requests_count
    attributes:
      - key: http.response.status_code
        default_value: "unknown"

HTTP status codes are a good attribute: the set is small, bounded and known in advance. You get roughly a dozen series and each one is meaningful.

Request paths, user identifiers, session tokens and trace IDs are the opposite. They are unbounded by construction, and grouping by one turns a single metric into as many series as you have distinct values. The rule I use: if you cannot write down the complete list of possible values, it is a label on a log line, not an attribute on a metric.

When the grouping is fixed rather than data-dependent, use labels instead — static key: value pairs stamped on every sample, with no cardinality risk at all.

One parsing detail that costs people an afternoon: . and - in an attribute key become _ before the metric reaches the panel. The example above is queried as http_response_status_code, not http.response.status_code.

Let the built-in formats do the parsing

If you are shipping logs from something common, do not write the regex. Glouton ships parsers for nginx, Apache, PostgreSQL, MySQL, MariaDB, Redis, Valkey, Kafka, HAProxy, MongoDB, RabbitMQ, plain JSON and Go’s slog, in host and container variants.

A format extracts structured fields — HTTP method, status code, client IP — so they arrive as attributes you can filter and group by, rather than as a string you have to regex at query time. Every field you get from a built-in parser is a field you are not paying to re-derive on every search.

A starting configuration

Turning everything on and then narrowing down is a reasonable way to discover what your systems produce, as long as you actually do the narrowing. Enable auto-discovery, look at the volume histogram for a day, then decide per source whether it is a ship or a count.

log.opentelemetry.auto_discovery.container_and_service_enable: true
log.opentelemetry.auto_discovery.journald_enable: true

Glouton picks up configuration changes without a restart, so iterating on filters is a matter of editing a file in /etc/glouton/conf.d/ and watching what happens. Log management is a paid feature, so this needs a plan that includes it — the agent will collect nothing on a free account no matter how the receivers are configured.

What I would settle on for a typical web application:

• Application error logs — shipped. You will read these.
• Access logs — counted, not shipped. Request rate and 5xx rate as metrics; the individual successful request is not worth storing.
• Database logs — shipped, with a built-in format so slow queries arrive parsed.
• Debug and trace — dropped at the agent, globally.
• Anything with an unbounded identifier in it — shipped if you need it, never used as a metric attribute.

That is not a smaller amount of visibility than shipping everything. It is the same visibility, with the expensive representation reserved for the lines where storage is actually what you need.

The shape of the decision

Log platforms bill you for volume, so the only durable way to control the cost is to send less. Every technique above is a version of the same idea: decide, per source, whether you want the line or the number, and only pay for lines when the answer is the line.

Getting that division right is not a cost-cutting exercise at the expense of visibility. Counting is often the better answer independently of price, because an error rate you can alert on and graph for a year is more operationally useful than a log search you have to remember to run.