Troubleshooting Fluentd Log Unreadable Warnings

When using Fluentd in a Kubernetes setup, you may encounter warnings indicating that certain log files are unreadable. This can prevent logs from being forwarded correctly. Below, we outline a sample configuration and troubleshooting steps to resolve these issues.

Sample Fluentd Configuration

Here’s a basic configuration for a Fluentd daemon set that reads logs from container directories:

<source>
    @type tail
    path /var/log/containers/sample*.log
    time_format %Y-%m-%dT%H:%M:%S.%NZ
    tag sample.*
    format json
    read_from_head true
</source>

<match sample.**>
    @type forward
    heartbeat_type tcp
    send_timeout 60s
    recover_wait 10s
    hard_timeout 60s
    <server>
        name worker-node2
        host 10.32.0.15
        port 24224
        weight 60
    </server>
</match>

Common Warning Messages

You might see warnings like the following in your logs:

2018-08-03 06:36:53 +0000 [warn]: /var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5d6328e7d.log unreadable. It is excluded and would be examined next time.

These messages indicate that Fluentd cannot access the specified log files, which can be due to permission issues or incorrect mount configurations.

Checking Permissions

Ensure that the log files have the correct permissions. You can verify this by running:

ls -lrt /var/log/containers/

You should see output similar to:

lrwxrwxrwx Jun 25 06:25 sample-77g68_kube-system_kube-proxy-9f3c3951c32ee.log -> /var/log/pods/aa1f8d5b-746f-11e8-95c0-005056b9ff3a/sample/7.log

DaemonSet YAML Configuration

Here is an example of a DaemonSet configuration that includes the necessary volume mounts:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging1
spec:
  template:
    spec:
      containers:
      - name: fluentd
        image: fluentd:latest
        volumeMounts:
        - name: fluentd-config
          mountPath: /fluentd/etc/
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlogcontainers
          mountPath: /var/log/containers
          readOnly: true
      volumes:
      - name: fluentd-config
        configMap:
          name: fluentd-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlogcontainers
        hostPath:
          path: /var/log/containers

Conclusion

If you have verified that the permissions are correct and the DaemonSet is configured properly, yet still encounter unreadable log warnings, consider checking the log file paths and ensuring that the Fluentd service has the necessary access rights. Adjusting these configurations should help in resolving the issue.