Introduction

In a setup involving EC2, AWS, Docker, Consul, and NGINX, maintaining consistent service discovery can be challenging. This guide outlines a typical configuration and addresses potential pitfalls.

Environment Setup

In this scenario, multiple services are deployed on individual EC2 instances, each running a series of containers in the following order:

  1. cAdvisor (for monitoring)
  2. Node Exporter (for monitoring)
  3. Consul (operating in agent mode)
  4. Registrator
  5. Your application service
  6. A custom container that runs both NGINX and Consul-Template

Custom Container Configuration

The custom container is built using the following Dockerfile:

FROM nginx:1.9

# Install Curl for downloading files
RUN apt-get update -qq && apt-get -y install curl

# Download and install Consul Template
RUN curl -L https://github.com/hashicorp/consul-template/releases/download/v0.10.0/consul-template_0.10.0_linux_amd64.tar.gz | tar -C /usr/local/bin --strip-components 1 -zxf -

# Create directory for Consul Template files
RUN mkdir /etc/consul-templates
COPY ./app.conf.tmpl /etc/consul-templates/app.conf

# Clear existing NGINX configuration files
RUN rm /etc/nginx/conf.d/*

# Set default Consul address
ENV CONSUL consul:8500

# Start NGINX and Consul Template
CMD /usr/sbin/nginx -c /etc/nginx/nginx.conf && consul-template -consul=$CONSUL -template "/etc/consul-templates/app.conf:/etc/nginx/conf.d/app.conf:/usr/sbin/nginx -s reload"

NGINX Configuration Template

The app.conf file used by Consul-Template is structured as follows:

{{range services}}
  upstream {{.Name}} {
    least_conn;{{range service .Name}}
    server {{.Address}}:{{.Port}};{{end}}
  }
{{end}}

server {
  listen 80 default_server;
  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  location / {
    proxy_pass http://cart/cart/;
  }

  location /cart {
    proxy_pass http://cart/cart;
  }

  {{range services}}
  location /api/{{.Name}} {
    proxy_read_timeout 180;
    proxy_pass http://{{.Name}}/{{.Name}};
  }
  {{end}}
}

Common Issues

Despite the initial successful startup, you may encounter an issue where Consul-Template reports that no servers are available for a specific service. This results in an empty upstream section for that service, leading to errors in the logs:

2015/12/04 07:09:34 [emerg] 77#77: no servers are inside upstream in /etc/nginx/conf.d/app.conf:336
nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/app.conf:336
2015/12/04 07:09:34 [ERR] (runner) error running command: exit status 1
Consul Template returned errors:
1 error(s) occurred:

* exit status 1

When this happens, NGINX will stop accepting requests.

Potential Causes and Solutions

It’s possible that NGINX crashes due to the lack of available upstream servers, but since Consul-Template continues to run, the Docker container does not restart. To resolve this, consider the following:

  • Check Service Registration: Ensure that all services are correctly registered with Consul and are healthy.
  • NGINX Error Handling: Implement error handling in NGINX to manage upstream failures gracefully.
  • Container Restart Policy: Configure a restart policy for your Docker container to ensure that it restarts if NGINX fails.

Conclusion

By following the above guidelines and troubleshooting steps, you can enhance the reliability of service discovery in your Dockerized environment using Consul and NGINX on AWS EC2 instances. If issues persist, further investigation into service health and network configurations may be required.