Managing Istio Sidecar Termination in Kubernetes Jobs and CronJobs

When integrating Istio into your Kubernetes environment, you may encounter issues with Jobs and CronJobs not terminating as expected due to the presence of the istio-proxy sidecar. This article outlines a workaround to ensure that your jobs complete successfully while adhering to security regulations through mTLS connections.

The Problem

Kubernetes Jobs and CronJobs are designed to run tasks to completion. However, when the istio-proxy sidecar is injected, it can prevent the pod from terminating if it continues to run after the main application container has finished. This behavior is due to the sidecar's lifecycle being independent of the primary container, which can lead to indefinite hanging states.

Proposed Workaround

While there are ongoing discussions in the Istio and Kubernetes communities regarding this issue (see Istio Issue #6324 and Kubernetes Issue #25908), a practical solution involves using a Kubernetes ServiceAccount, Role, and RoleBinding to allow the main container to send a termination signal to the sidecar once it has completed its task.

Configuration Steps

1. Create a ServiceAccount This ServiceAccount will be used to grant permissions for executing commands in the pod.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: terminate-sidecar-example-service-account

2. Define a Role The Role will allow the ServiceAccount to get and delete pods, as well as execute commands within them.

   kind: Role
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: terminate-sidecar-example-role
   rules:
     - apiGroups: [""]
       resources: ["pods"]
       verbs: ["get", "delete"]
     - apiGroups: [""]
       resources: ["pods/exec"]
       verbs: ["create"]

3. Create a RoleBinding This binds the ServiceAccount to the Role, allowing it to perform the defined actions.

   kind: RoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: terminate-sidecar-example-rolebinding
   subjects:
     - kind: ServiceAccount
       name: terminate-sidecar-example-service-account
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: terminate-sidecar-example-role

4. Configure the CronJob Finally, set up the CronJob to use the ServiceAccount and include the command to terminate the sidecar upon successful completion of the main container.

   apiVersion: batch/v1beta1
   kind: CronJob
   metadata:
     name: terminate-sidecar-example-cronjob
     labels:
       app: terminate-sidecar-example
   spec:
     schedule: "30 2 * * *"
     jobTemplate:
       metadata:
         labels:
           app: terminate-sidecar-example
       spec:
         template:
           metadata:
             labels:
               app: terminate-sidecar-example
             annotations:
               sidecar.istio.io/inject: "true"
           spec:
             serviceAccountName: terminate-sidecar-example-service-account
             containers:
             - name: your-container-name
               image: your-image-name
               command:
                 - "/bin/ash"
                 - "-c"
               args:
                 - "node index.js && kubectl exec -n ${POD_NAMESPACE} ${POD_NAME} -c istio-proxy -- bash -c 'sleep 5 && /bin/kill -s TERM 1'"
               env:
                 - name: POD_NAME
                   valueFrom:
                     fieldRef:
                       fieldPath: metadata.name
                 - name: POD_NAMESPACE
                   valueFrom:
                     fieldRef:
                       fieldPath: metadata.namespace

Conclusion

This workaround allows you to gracefully terminate the istio-proxy sidecar after your main container has completed its task, ensuring that your Jobs and CronJobs do not hang indefinitely. While there are ongoing efforts to develop more robust solutions, this approach provides a practical interim measure.