Introduction

In this guide, we will explore how to set up Open Policy Agent (OPA) as an admission controller for a federated Kubernetes cluster deployed on Google Kubernetes Engine (GKE). This setup is particularly useful for managing dynamic workloads across different regions.

Background

When creating a Federated Deployment in Kubernetes, the default behavior is to evenly distribute replicas across all registered clusters. For instance, if you have three clusters and specify spec.replicas = 9, each cluster will receive three replicas. However, in scenarios where workloads vary significantly by region, such as in our case, this approach may not suffice.

Use Case

Consider a scenario with three clusters, each located in a different region:

• Europe (eu): 1000 requests per second (rps), nodes labeled with region=eu
• United States (us): 200 rps, nodes labeled with region=us
• Asia (asia): 100 rps, nodes labeled with region=asia

The goal is to deploy a single application that can dynamically adjust the number of pods across these clusters based on the specified load.

Pod Requirements

Each pod should:

• Handle 100 rps
• Require 2 vCPUs and 2GB of RAM
• Be scheduled on a node with anti-affinity to ensure it does not share a node with another pod of the same application

Configuring Open Policy Agent

To achieve this dynamic distribution, you can configure OPA to enforce policies that dictate how pods are allocated across the clusters. Below is an example policy that could be used:

package kubernetes.admission

# Define the policy for pod placement based on region
allow {
    input.request.kind.kind == "Pod"
    count(input.request.object.spec.containers) == 1
    region := input.request.object.metadata.labels.region
    count(input.request.object.spec.containers) == 1
    required_resources := input.request.object.spec.containers[0].resources.requests
    required_resources["cpu"] == "2"
    required_resources["memory"] == "2Gi"
    # Ensure the pod can handle the specified rps
    rps := get_rps(region)
    rps >= 100
}

# Function to get rps based on region
get_rps(region) = rps {
    rps = 1000  # For eu
    region == "eu"
}
get_rps(region) = rps {
    rps = 200   # For us
    region == "us"
}
get_rps(region) = rps {
    rps = 100   # For asia
    region == "asia"
}

This policy checks the region of the pod being created and ensures that it meets the resource requirements and can handle the specified requests per second.

Conclusion

By implementing OPA as an admission controller, you can effectively manage pod distribution across your federated Kubernetes clusters based on dynamic workloads. For further details, refer to the Open Policy Agent documentation and the Kubernetes Federation documentation.

Feel free to reach out for additional resources or clarifications regarding this setup.