Grafana Loki: Log Aggregation Without the Index Bill

Loki takes a deliberately different approach from full-text indexing log platforms: it stores compressed log chunks and indexes only the labels that describe the stream. That makes it cheap to run and simple to operate, at the cost of relying on good label design and query patterns. This article covers the architecture, a working local deployment, and the LogQL queries that replace grep when you cannot SSH into the server.

Why Loki Is Not "Another Elasticsearch"

The design decision that defines Loki is the one it refuses to make. Where Elasticsearch, OpenSearch and Splunk tokenise every log line at ingest time and build a searchable inverted index over the content, Loki ingests the line as an opaque blob and indexes only a small set of labels attached to the stream. Nothing inside the log line is indexed - it is compressed into chunks that live on object storage, and the index exists purely so the query engine knows which chunks to fetch and decompress.

The practical consequences are easy to state and easy to underestimate:

• Cost scales with volume, not with number of fields. A log line with 40 JSON keys costs roughly the same as a plain text line of the same byte length, because no key is parsed at ingest.
• Object storage is the tier of record. Chunks land in S3, GCS, Azure Blob or a local filesystem; Loki's own state is tiny. A retention of two years costs object-storage prices, not SSD prices.
• Query latency depends on how well you prune. A tight label selector plus a time range touches few chunks and returns in milliseconds; a query that has to scan a week of a busy stream will take seconds and burn memory doing it.
• Cardinality is the real cost driver. Labels create index entries and, more importantly, they fragment the stream. A label whose values change per request turns one stream into millions and destroys the very thing that made ingestion cheap.

That trade is worth taking for high-volume application logs and, especially, for network device syslog - where the useful operations are "show me everything from edge-router-01 in the last 15 minutes" and "alert me when authentication failures spike", not "find every occurrence of this arbitrary string across five years of history".

How Loki Stores Logs

• Agent - a collector such as Grafana Alloy (or Promtail on older deployments) tails files or the systemd journal, attaches labels, and pushes streams over HTTP.
• Loki - ingests and stores log data in compressed chunks; the index holds only the label set, which is why it stays small.
• Grafana - queries and visualises; queries can also come from LogCLI or the HTTP API.

Because the whole log line is still searchable, labels exist to narrow the data scanned, not to make the search possible. Over-labelling is the standard first mistake: a label with unbounded values (a user ID, a request ID) creates a stream per value and destroys performance.

The Write and Read Paths, Component by Component

In a single-binary deployment everything below runs in one process; in microservices mode each is a separate deployment. Knowing the names pays off the first time you debug a gap in the data.

Agent  ->  distributor  ->  ingester  ->  object storage (chunks)
                                   |-->  index (TSDB / BoltDB-shipper / S3)
query path:  querier  --->  ingester (recent, in-memory) + storage (older)
             query-frontend / query-scheduler  ->  parallelise and queue
             ruler  ->  evaluates LogQL alert + recording rules
             compactor  ->  merges index, applies retention, deletes old chunks

• Distributor validates each push, enforces rate limits and per-tenant stream limits, and fans the batch out to ingesters based on a hash of the label set (replication factor 3 by default).
• Ingester buffers the most recent data in memory and WAL, and flushes chunks to object storage once they are full, idle, or old enough (chunk_idle_period, max_chunk_age).
• Querier merges results from ingesters (for recent data) and the store (for flushed chunks), applying the LogQL pipeline in order.
• Query frontend / scheduler splits queries by time, queues them, retries failures and caches results - the single biggest lever for dashboard performance.
• Compactor merges small index files, retains data per stream and enforces the retention period. Without it, retention silently does nothing.
• Ruler evaluates the rule files you load - the component that turns "logs" into "alerts".

Label Design: The Rules That Keep Loki Fast

Almost every Loki performance complaint traces back to label choices made at ingest, so decide them deliberately:

1. Keep the label set small - aim for ten or fewer. Each label multiplies the number of streams, and streams are the unit of index growth.
2. Never label high-cardinality data. Request IDs, user IDs, session IDs, source ports, full URLs and free-form error messages belong in the log line, not in a label. Attach them as structured metadata if you need them addressable.
3. Prefer unbounded values in the line, bounded values in the label. host, job, environment, cluster, device_role are good; filename extracted from a log path is usually bad.
4. Extract at query time, free of charge. Loki 3.x parses JSON, logfmt and pattern-encoded bodies on read, so the fields you occasionally need cost nothing until you ask for them.
5. Watch the metrics. loki_ingester_streams_created_total per tenant and loki_distributor_ingester_append_failures_total are the early warnings; the max_streams_per_user limit is what stops a mislabelled deployment from taking the cluster down.

A Working Docker Deployment

git clone https://github.com/grafana/loki-fundamentals.git -b getting-started
cd loki-fundamentals
docker compose up -d

Verify each component:

http://localhost:12345/graph   # Alloy UI
http://localhost:3000          # Grafana
http://localhost:3100/metrics  # Loki metrics

Alloy is configured to tail all container logs, so data should already be arriving; Grafana's Logs Drilldown view is the fastest way to confirm it. Logs Drilldown is worth learning early: it shows per-service log volume, error rates and detected fields without writing a single query, and it is the view you will reach for when something has broken and you do not yet know what to search for.

Two checks confirm the pipeline end to end:

# is Loki receiving data?
curl -s 'http://localhost:3100/loki/api/v1/labels'
# what do the streams look like?
curl -s 'http://localhost:3100/loki/api/v1/series' --data-urlencode 'match[]={job=~".+"}' | head

Promtail in Existing Fleets

# /etc/promtail/config.yml (short form)
server: { http_listen_port: 9080 }
clients:
  - url: http://loki.example.com:3100/loki/api/v1/push
scrape_configs:
  - job_name: syslog
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog
          host: edge-router-01
          __path__: /var/log/*.log

Promtail discovers targets, attaches labels and pushes batches; it exposes /ready and /metrics, which is all you need for health monitoring of the pipeline itself. Alloy is the successor and the better choice for new builds - it can run the same loki.source.file and loki.write components alongside Prometheus scraping and OpenTelemetry intake, in one agent.

# Alloy equivalent
loki.source.file "syslog" {
  targets    = [{ __path__ = "/var/log/*.log" }]
  forward_to = [loki.write.default.receiver]
}
loki.write "default" {
  endpoint { url = "http://loki.example.com:3100/loki/api/v1/push" }
}

LogQL Essentials

{job="syslog", host="edge-router-01"} |= "BGP"
{job="syslog"} |~ "LINK-3-UPDOWN" | json
sum(rate({job="syslog"} |= "authentication failed" [5m])) by (host)

The final form is the one that turns logs into alerting: a log-derived metric, evaluated by Loki's ruler, which can then route to Alertmanager. That gives you "alert me when authentication failures spike on any host" without shipping a separate metric pipeline.

LogQL is two languages in one. Log queries return lines; metric queries wrap an aggregation around a log query and return time series you can graph or alert on.

A log query always starts with a stream selector in braces, then applies filters left to right:

# line filters: |= contains, != does not, |~ regex, !~ not-regex
{job="syslog", host="edge-router-01"} |= "BGP" != "KEEPALIVE"

# parsers: json, logfmt, regexp, pattern, unpack
{job="nginx"} | json | status >= 500
{job="app"} | logfmt | duration > 2s
{job="syslog"} | pattern `<_> %LEVEL% %MSG%` | LEVEL="ERROR"

# label filter follows a parser and prunes on the extracted value
{job="app"} | json | level=~"error|fatal" | response_time > 1000

Metric queries add a range and an aggregate. The three you will use most:

# error rate per host, as a per-second rate over 5 minutes
sum by (host) (rate({job="syslog"} |= "authentication failed" [5m]))

# count of matching lines in the window
count_over_time({job="app"} | json | level="error" [5m])

# extract a numeric field and aggregate it (unwrap, for latency etc.)
avg by (path) (avg_over_time({job="nginx"} | json | unwrap request_time [5m]))
quantile_over_time(0.99, {job="nginx"} | json | unwrap request_time | __error__="" [5m])

Two habits make queries behave: always give the narrowest label set you can (single host beats job=~".+"), and put cheap line filters before expensive parsers - Loki applies the pipeline in order and a rejected line never reaches the parser.

Alerting and Recording Rules

Rules live in YAML files loaded by the ruler and look like Prometheus rules with LogQL expressions:

groups:
  - name: syslog-security
    rules:
      - alert: HighAuthFailures
        expr: |
          sum by (host) (rate({job="syslog"} |= "authentication failed" [5m])) > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Auth failures on {{ $labels.host }}"
      - record: job:syslog_errors:rate5m
        expr: sum by (host, job) (rate({job="syslog"} |~ "ERROR|CRIT" [5m]))

The recording rule is not a luxury at scale: a dashboard that refreshes an expensive regex aggregation every 30 seconds will hammer the cluster, whereas the pre-computed series costs one evaluation per interval. Alertmanager then handles grouping, inhibition and routing - see Alertmanager routing and silences for that half of the story.

Retention, Storage and Sizing

# loki.yaml (excerpt)
limits_config:
  retention_period: 30d
  max_label_names_per_series: 15
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
compactor:
  working_directory: /loki/compactor
  retention_enabled: true          # without this, retention_period is ignored
  delete_request_store: s3
storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
  aws:
    s3: s3://eu-west-1/loki-chunks
    bucketnames: loki-chunks

The rule of thumb for chunk storage is roughly ten to fifteen bytes per log line after compression, depending on how repetitive the text is. A network estate pushing 20 GB/day of raw syslog therefore lands near 2-4 GB/day on disk - the reason Loki deployments stay cheap. Set a retention_period per tenant, enable the compactor, and size the querier's memory rather than the disk: queries are where the pressure lives.

Structured Metadata and Bloom Filters

Two newer features soften the cardinality trade-off. Structured metadata attaches high-cardinality key/value pairs (trace IDs, pod names, request IDs) to a log line without adding them to the stream's label set: streams stay few, but a query can still filter on the value. Bloom filters build a per-chunk probabilistic index over tokenised content, so a query with a specific unusual term can skip chunks that certainly do not contain it - cutting the scan for needle-in-haystack searches from the whole range to a small fraction of chunks. Both are set on the agent side and configured per tenant in Loki.

Troubleshooting a Loki Pipeline

• No data in Grafana. Check loki_ingester_chunks_created_total rising at the distributor, then the agent's loki_write_sent_bytes_total on :12345/metrics. If the agent is silent, the path glob or journal unit is wrong. If the agent is pushing but Loki returns 429, you have hit a rate limit and loki_discarded_samples_total names the reason.
• Queries time out. Look for a broad selector: {job=~".+"} over days will always be slow. Narrow the labels, shrink the range, or add a Bloom filter.
• "too many outstanding requests". Raise the query-frontend queues or reduce concurrency; the scheduler is the throttle, not a bug.
• Old data never disappears. Retention requires the compactor with retention_enabled: true and a delete-request store configured. Setting the period alone does nothing.
• Streams explode. Read loki_ingester_streams_created_total per tenant and find the label with unbounded values - it is almost always a request ID, a URL, or a message fragment that someone promoted to a label.

Where Loki Fits

Use Loki for application and device logs where label-based filtering is enough and volume is high; keep full-text-indexed platforms for the rare case where you must search arbitrary strings across years of data. Network devices fit naturally: forward syslog as described in rsyslog central log server setup, tune device-side levels with IOS logging levels, and pair it with flow data for traffic questions - NetFlow v9 on IOS. For pure device health polling, Zabbix SNMP monitoring remains the lighter-weight option, and if you are deciding between storage backends for the metrics side of the same stack, the trade-offs in VictoriaMetrics single-node versus cluster apply to Loki's own SSD-versus-microservices question too.

原文链接：https://grafana.com/docs/loki/latest/get-started/