Fluent Bit to Elasticsearch: A Production Log Pipeline

Fluent Bit is the lightest way to move logs from servers and network devices into Elasticsearch, but the default configuration assumes a friendly environment: a reachable cluster, no TLS, no authentication and infinite retries. Production needs more. This article builds a pipeline with parsers, filters, index naming, buffering behaviour and TLS, explains the settings that cause data loss when they are wrong, and shows how to verify that events actually land in the index you expect.

Pipeline shape

1. Input — tail for files, systemd for journald, tcp/syslog for network devices.
2. Parser — turn unstructured lines into fields.
3. Filter — enrich, rewrite, drop noise.
4. Buffer — memory or filesystem, with limits.
5. Output — the Elasticsearch plugin, with retry and TLS settings.

A working configuration

[SERVICE]
    Flush         5
    Daemon        Off
    Log_Level     info
    Parsers_File  parsers.conf
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020
    storage.path  /var/log/flb-storage/
    storage.sync  normal
    storage.metrics on

[INPUT]
    Name              tail
    Path              /var/log/nginx/access.log
    Tag               web.nginx
    Parser            nginx
    DB                /var/log/flb-nginx.db
    Mem_Buf_Limit     32MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[INPUT]
    Name              systemd
    Tag               host.systemd
    Systemd_Filter    _SYSTEMD_UNIT=sshd.service
    Read_From_Tail    On

[FILTER]
    Name              record_modifier
    Match             *
    Record            hostname ${HOSTNAME}
    Record            env production

[FILTER]
    Name              grep
    Match             host.systemd
    Exclude           MESSAGE connected

[OUTPUT]
    Name              es
    Match             *
    Host              elasticsearch.logging.svc
    Port              9200
    HTTP_User         ${ES_USERNAME}
    HTTP_Passwd       ${ES_PASSWORD}
    TLS               On
    TLS.Verify        On
    TLS.CA_File       /etc/ssl/certs/es-ca.crt
    Logstash_Format   On
    Logstash_Prefix   fluentbit
    Logstash_DateFormat %Y.%m.%d
    Replace_Dots      On
    Retry_Limit       5
    Time_Key          @timestamp
    Suppress_Type_Name On

Line-by-line, the settings that decide reliability:

Parsing network device syslog

[PARSER]
    Name        cisco_syslog
    Format      regex
    Regex       ^<(?<pri>\d+)>(?<seq>\d+): (?<ts>[^:]+): %(?<facility>[A-Z0-9-]+)-(?<severity>\d)-(?<mnemonic>[A-Z0-9_]+): (?<message>.*)$
    Time_Key    ts

[INPUT]
    Name        syslog
    Tag         net.cisco
    Parser      cisco_syslog
    Listen      0.0.0.0
    Port        5140

Use a high port and forward from the network devices so you do not need root for a privileged port. Keep device syslog at debug levels only during troubleshooting — a single switch with debug enabled can produce more events than an entire application fleet.

Avoiding data loss

• Filesystem buffering for anything you must not lose. Memory-only buffering discards on restart.
• Do not set Retry_Limit 1. A rolling Elasticsearch restart then silently drops logs.
• Backpressure, not overflow. Mem_Buf_Limit plus filesystem storage means the input pauses; without it, events are lost.
• Monitor the pipeline itself. Fluent Bit exposes metrics on port 2020 — scraped by Prometheus, an output_errors counter is the earliest warning of a broken pipeline.
• Pin mappings. Use index templates so a new field type does not reject whole documents later.

Verification

# Pipeline metrics
curl -s http://localhost:2020/api/v1/metrics | jq '.output'

# Did the indices get created?
curl -s "http://elasticsearch:9200/_cat/indices/fluentbit-*?v&h=index,docs.count,store.size"

# Search for a specific host in the last hour
curl -s "http://elasticsearch:9200/fluentbit-*/_search" -H 'Content-Type: application/json' -d '
{ "query": { "bool": { "must": [
  { "term": { "hostname.keyword": "web-01" } },
  { "range": { "@timestamp": { "gte": "now-1h" } } } ] } },
  "size": 1 }' | jq '.hits.total'

# Error lines from Fluent Bit itself
journalctl -u fluent-bit -f | grep -i "\[error\]"

Scaling notes

• One Fluent Bit agent per host, forwarding to a small cluster of aggregators if you need central parsing; agents should stay dumb and fast.
• Keep Flush at 5 seconds or lower for interactive search expectations; raise it if you are optimising throughput over freshness.
• Use index lifecycle management to roll indices daily and delete after your retention window — unbounded indices are the most common cause of an Elasticsearch cluster falling over.

Performance tuning

A Fluent Bit pipeline that is correct but slow becomes a source of log loss as buffers fill. The levers, in order of impact:

A useful heuristic: measure events per second per host first, then decide whether the problem is the agent or the cluster. If Fluent Bit metrics show a low output rate and a healthy Elasticsearch bulk rate, the agent is the bottleneck. If the agent is pushing and Elasticsearch is rejecting bulks, the problem is downstream and the index lifecycle policy is usually the culprit.

Security and compliance considerations

• Never log secrets. Add filters to redact authorisation headers, tokens and passwords before shipping. Logs are widely readable; a token in a log is a credential leak.
• TLS everywhere. Transport encryption with certificate verification, not just TLS without validation.
• Access control on the index. Log data often contains personal data; restrict who can query it and apply a retention policy that matches your obligations.
• Immutable retention where required. If you need tamper resistance, write logs to object storage with retention locks rather than relying on the searchable index alone.
• Separate pipelines by sensitivity. Do not mix firewall authentication logs and application debug logs in one ungoverned index.

A deployment sequence that avoids blind spots

1. Stand up one agent on a non-critical host with console output only; validate parsing on real lines.
2. Add the Elasticsearch output with filesystem buffering and check that documents arrive with the expected field types.
3. Create the index template and lifecycle policy before volume increases.
4. Roll out to the fleet in batches, checking output_errors after each batch.
5. Build a small dashboard showing events per second per host — a host that stops logging is usually a symptom of something else.
6. Test resilience: stop Elasticsearch for ten minutes and confirm no events are lost once it returns.

Related articles

An alternative pipeline built on Loki and LogQL is covered in Grafana Loki log aggregation and LogQL. On the source side, device logging configuration is explained in Cisco IOS logging levels, buffer and trap configuration. If your logs come from containers, start with Docker Compose depends_on and healthcheck startup order to make sure services exist before the pipeline looks for their logs.

原文链接：https://docs.fluentbit.io/manual/data-pipeline/outputs/elasticsearch