Customizing Fluentd Configuration in GKE

When using Google Kubernetes Engine (GKE) with the Stackdriver logging agent, you may want to customize the Fluentd configuration to accurately parse logs from your applications. This guide outlines the steps to achieve this.

Steps to Modify Fluentd Configuration

1. Retrieve the Fluentd ConfigMap: Start by exporting the existing Fluentd configuration to a YAML file for editing.

   kubectl get configmap fluentd-gcp-config -n kube-system -o yaml > fluentd-config.yaml

2. Add a Log Parser: Open the fluentd-config.yaml file and insert a new <filter> section that corresponds to your logging format (e.g., log4js). This allows Fluentd to parse the logs correctly, assigning appropriate severity levels and components.

3. Create a New ConfigMap: After editing, create a new ConfigMap with your updated configuration. You can name it something like fluentd-cm-custom:

   kubectl create configmap fluentd-cm-custom --from-file=fluentd-config.yaml -n kube-system

4. Update the Fluentd DaemonSet: Next, you need to modify the Fluentd DaemonSet to use your new ConfigMap. Use the following command to edit the DaemonSet:

   kubectl edit daemonset fluentd-gcp -n kube-system

   In the editor, change the configMap reference to fluentd-cm-custom.

5. Handle Reconciliation Issues: If you notice that your changes are reverted, it may be due to GKE's reconciliation mode. The DaemonSet and ConfigMap may have the annotation addonmanager.kubernetes.io/mode: Reconcile, which causes GKE to overwrite custom configurations. To prevent this, you may need to disable this mode or manage the logging agent outside of GKE's default settings.

Conclusion

By following these steps, you can successfully customize the Fluentd configuration in your GKE cluster to better suit your logging needs. Ensure to monitor the logs after making changes to verify that the parsing works as intended.