Setting Up Fluentd Logging in Docker

In this guide, we will configure a Docker setup that includes an HTTP application, Fluentd for logging, Elasticsearch for storage, and Kibana for visualization. The goal is to keep Fluentd private within the Docker network while allowing the application to send logs to it.

Docker Compose Configuration

We will define our services in a docker-compose.yml file. Here’s how the configuration looks:

Application Service

The application service uses the Fluentd logging driver to send logs:

web:
  image: httpd
  container_name: httpd
  ports:
    - "80:80"
  logging:
    driver: "fluentd"
    options:
      fluentd-address: fluentd:24224
      tag: httpd.access

Fluentd Service

The Fluentd service will not expose its port to the host network, keeping it private:

fluentd:
  build: ./fluentd
  image: fluentd
  container_name: fluentd
  links:
    - "elasticsearch"

Elasticsearch and Kibana Services

You can also define your Elasticsearch and Kibana services as needed, ensuring they are linked appropriately.

Important Notes

• Networking: By using fluentd:24224 as the address in the logging options, we ensure that the application can communicate with Fluentd over the Docker network.
• Linking: The links option is used to establish a connection between the services, but it is important to note that the logging driver may initialize before the links are fully established. If you encounter issues like failed to initialize logging driver: dial tcp: lookup fluentd, it may be due to timing issues in service startup.

Conclusion

This configuration allows you to keep your Fluentd instance private while still enabling your application to log to it effectively. Make sure to test your setup to confirm that logs are being sent and stored correctly in Elasticsearch and visualized in Kibana.