Kubernetes Pod network connectivity issue with Istio enabled

A K8s Pod can return an Error when trying to reach the network before the Istio Sidecar is Running.

Many applications execute commands or checks during startup, which require network connectivity. This can cause application containers to hang or restart if the istio-proxy sidecar container is not ready.

This article demonstrates how to use Istio’s annotation holdApplicationUntilProxyStarts to avoid network connectivity issues.

Test Environment

The test environment had the following tech stack:

macOS Monterey version 12.3.1
Apple M1 Pro
minikube version v1.26.0
K8s version v1.22.7
Istio version 1.13.3

Check the official documentation on how to install Istio

Before you begin

1. Start minikube with the specific K8s version:

minikube start — kubernetes-version=v1.22.7

2. Deploy the httpbin sample application:

Make sure your current directory is the istio directory.
Start the httpbin sample.

kubectl apply -f samples/httpbin/httpbin.yaml

3. Review the httpbin deployment:

$ kubectl get services httpbin 
NAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
httpbin   ClusterIP   10.105.138.79   <none>        8000/TCP   32d

$ kubectl get pods httpbin-74fb669cc6-4qs24                                                                                    
NAME                       READY   STATUS    RESTARTS         AGE
httpbin-74fb669cc6-4qs24   2/2     Running   0                32d

Scenario 1 — Pod with Error

Now that Istio and httpbin are deployed and working we can deploy our Pod pod-test-connectivity. This is the Pod yaml file:

apiVersion: v1
kind: Pod
metadata:
  name: "pod-test-connectivity"
spec:
  containers:
  - name: wget
    image: busybox
    command: ['wget']
    args: [ '-qO-', 'httpbin:8000']
restartPolicy: Never

This Pod will run a wget command to GET the httpbin application.

Create the Pod:

$ kubectl apply -f pod-connectivity.yaml                      
pod/pod-test-connectivity created

Check Pod status:

$ kubectl get pods pod-test-connectivity                  
NAME                    READY   STATUS   RESTARTS   AGE
pod-test-connectivity   1/2     Error    0          25s

The Pod is in Error status. Looking at the logs we can see:

$ kubectl logs pod-test-connectivity    
wget: can't connect to remote host (10.105.138.79): Connection refused

Delete the Pod:

$ kubectl delete -f pod-connectivity.yaml                      
pod/pod-test-connectivity deleted

Scenario 2 — Pod Running Successfully

This time, we will add the Istio annotation to the Pod manifest file:

apiVersion: v1
kind: Pod
metadata:
  name: "pod-test-connectivity"
  annotations:
    proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
spec:
  containers:
  - name: curl
    image: curlimages/curl
    command: ["/bin/sh"]
    args:
    - "-c"
    - |
    curl httpbin:8000
    curl -s -X POST 127.0.0.1:15000/quitquitquit #to exit the istio-proxy sidecar
restartPolicy: Never

Create the Pod:

$ kubectl apply -f pod-connectivity.yaml                      
pod/pod-test-connectivity created

Check Pod status:

$ kubectl get pods pod-test-connectivity
NAME                    READY   STATUS      RESTARTS   AGE
pod-test-connectivity   0/2     Completed   0          7s

Now our Pod completed the curl without any problems. Check the logs to confirm:

$ kubectl logs pod-test-connectivity

...
<div class="wrapper">
        <section class="block col-12 block-desktop col-12-desktop">
            <div>
            
 <h2>Other Utilities</h2>
 
 <ul>
                    <li>
                        <a href="/forms/post">HTML form</a> that posts to /post /forms/post</li>
                </ul>
                
<br />
                <br />
            </div>
        </section>
    </div>
</div>
</body>
</html>OK

Using the annotation holdApplicationUntilProxyStarts prevents the Pod to execute any commands before the istio-proxy is Running . Therefore, when the Pod executes the curl it already has network connectivity.

Reference

https://istio.io/latest/docs/ops/common-problems/injection/#pod-or-containers-start-with-network-issues-if-istio-proxy-is-not-ready

Rafael Natali