Topology-Aware Traffic Routing

Motivation

The enablement of highly available shoot control-planes requires multi-zone seed clusters. A garden runtime cluster can also be a multi-zone cluster. The topology-aware routing is introduced to reduce costs and to improve network performance by avoiding the cross availability zone traffic, if possible. The cross availability zone traffic is charged by the cloud providers and it comes with higher latency compared to the traffic within the same zone. The topology-aware routing feature enables topology-aware routing for Services deployed in a seed or garden runtime cluster. For the clients consuming these topology-aware services, kube-proxy favors the endpoints which are located in the same zone where the traffic originated from. In this way, the cross availability zone traffic is avoided.

How it works

The topology-aware routing feature relies on the Kubernetes ServiceTrafficDistribution feature.

The ServiceTrafficDistribution allows expressing preferences for how traffic should be routed to Service endpoints. For more details, see upstream documentation of the feature.

The PreferSameZone strategy allows traffic to be routed to Service endpoints in topology-aware and predictable manner.

How to make a Service topology-aware

How to make a Service topology-aware in Kubernetes 1.32 to 1.33

In Kubernetes 1.32 to 1.33, ServiceTrafficDistribution is being used to make a Service topology-aware. The .spec.trafficDistribution field has to be set to PreferClose:

apiVersion: v1
kind: Service
spec:
  trafficDistribution: PreferClose

How to make a Service topology-aware in Kubernetes 1.34 and later

The value PreferClose has been deprecated in favor of PreferSameZone and PreferSameNode. PreferSameZone is an alias for the existing PreferClose to clarify its semantics. For more information, read the details in the Kubernetes deprecation announcement. In Kubernetes 1.34 and later, ServiceTrafficDistribution is still used to make a Service topology-aware. The .spec.trafficDistribution field should be set to PreferSameZone:

apiVersion: v1
kind: Service
spec:
  trafficDistribution: PreferSameZone

Prerequisites for making a Service topology-aware

1. The Pods backing the Service should be spread on most of the available zones. This constraint should be ensured with appropriate scheduling constraints (topology spread constraints, (anti-)affinity). Enabling the feature for a Service with a single backing Pod or Pods all located in the same zone does not lead to a benefit.
2. The component should be scaled up by VerticalPodAutoscaler. In case of an overload (a large portion of the of the traffic is originating from a given zone), the VerticalPodAutoscaler should provide better resource recommendations for the overloaded backing Pods.
3. Check the Considerations for using traffic distribution control.

Topology-aware Services in the Seed cluster

etcd-main-client and etcd-events-client

The etcd-main-client and etcd-events-client Services are topology-aware. They are consumed by the kube-apiserver.

kube-apiserver

The kube-apiserver Service is topology-aware if the shoot uses layer 4 load-balancing. If it is using layer 7 load-balancing it is not. It is consumed by the controllers running in the Shoot control plane.
Layer 7 load-balancing is active when IstioTLSTermination feature gate is active on the Seed and the Shoot did not opt out. Please see this documentation for more details.

Note: The istio-ingressgateway component routes traffic in topology-aware manner - if possible, it routes traffic to the target kube-apiserver Pods in the same zone. If there is no healthy kube-apiserver Pod available in the same zone, the traffic is routed to any of the healthy Pods in the other zones. This behaviour is unconditionally enabled.

gardener-resource-manager

The gardener-resource-manager Service that is part of the Shoot control plane is topology-aware. The resource-manager serves webhooks and the Service is consumed by the kube-apiserver for the webhook communication.

vpa-webhook

The vpa-webhook Service that is part of the Shoot control plane is topology-aware. It is consumed by the kube-apiserver for the webhook communication.

Topology-aware Services in the garden runtime cluster

virtual-garden-etcd-main-client and virtual-garden-etcd-events-client

The virtual-garden-etcd-main-client and virtual-garden-etcd-events-client Services are topology-aware. virtual-garden-etcd-main-client is consumed by virtual-garden-kube-apiserver and gardener-apiserver, virtual-garden-etcd-events-client is consumed by virtual-garden-kube-apiserver.

virtual-garden-kube-apiserver

The virtual-garden-kube-apiserver Service is topology-aware if it uses layer 4 load-balancing. If it is using layer 7 load-balancing it is not. It is consumed by virtual-garden-kube-controller-manager, gardener-controller-manager, gardener-scheduler, gardener-admission-controller, extension admission components, gardener-dashboard and other components. Layer 7 load-balancing is active when IstioTLSTermination feature gate is active in gardener-operator. Please see this documentation for more details.

gardener-apiserver

The gardener-apiserver Service is topology-aware. It is consumed by virtual-garden-kube-apiserver. The aggregation layer in virtual-garden-kube-apiserver proxies requests sent for the Gardener API types to the gardener-apiserver.

gardener-admission-controller

The gardener-admission-controller Service is topology-aware. It is consumed by virtual-garden-kube-apiserver and gardener-apiserver for the webhook communication.

How to enable the topology-aware routing for a Seed cluster?

For a Seed cluster the topology-aware routing functionality can be enabled in the Seed specification:

apiVersion: core.gardener.cloud/v1beta1
kind: Seed
# ...
spec:
  settings:
    topologyAwareRouting:
      enabled: true

The topology-aware routing setting can be only enabled for a Seed cluster with more than one zone. gardenlet enables topology-aware Services only for Shoot control planes with failure tolerance type zone (.spec.controlPlane.highAvailability.failureTolerance.type=zone). Control plane Pods of non-HA Shoots and HA Shoots with failure tolerance type node are pinned to single zone. For more details, see High Availability Of Deployed Components.

How to enable the topology-aware routing for a garden runtime cluster?

For a garden runtime cluster the topology-aware routing functionality can be enabled in the Garden resource specification:

apiVersion: operator.gardener.cloud/v1alpha1
kind: Garden
# ...
spec:
  runtimeCluster:
    settings:
      topologyAwareRouting:
        enabled: true

The topology-aware routing setting can be only enabled for a garden runtime cluster with more than one zone.