Comandi diagnostici — Istio / OSSM 2.x

Reference dei comandi usati in troubleshooting su OpenShift Service Mesh 2.x. Namespace di controllo assunto istio-system, SMCP basic.

Stato del control plane

oc get smcp -n istio-system
oc get smcp basic -n istio-system -o jsonpath='{.status.conditions}' | jq
oc get smcp basic -n istio-system -o jsonpath='{.spec.version}{"\n"}'

oc get smmr default -n istio-system -o jsonpath='{.status.configuredMembers}{"\n"}' | tr ',' '\n'
oc get smmr default -n istio-system -o jsonpath='{.status.configuredMembers}{"\n"}' | tr ',' '\n' | wc -l

oc get pod -n istio-system
oc get events -n istio-system --sort-by=.metadata.creationTimestamp
oc logs -n istio-system -l app=istiod --tail=300 | grep -Ei 'error|rejected|timeout|push'

Sincronizzazione xDS

Il primo comando da lanciare su qualunque problema di routing: mostra se i proxy hanno la configurazione aggiornata.

istioctl -i istio-system proxy-status

Stato	Significato
SYNCED	Il proxy ha l’ultima configurazione
NOT SENT	istiod non ha inviato nulla (spesso normale se non c’è config da inviare)
STALE	Push inviato ma non confermato → proxy bloccato o rete verso istiod degradata

istioctl -i istio-system analyze -n <namespace>
istioctl -i istio-system analyze --all-namespaces

Configurazione effettiva del proxy

istioctl -i istio-system proxy-config listeners <pod> -n <namespace>
istioctl -i istio-system proxy-config routes    <pod> -n <namespace>
istioctl -i istio-system proxy-config clusters  <pod> -n <namespace>
istioctl -i istio-system proxy-config endpoints <pod> -n <namespace>
istioctl -i istio-system proxy-config secret    <pod> -n <namespace>

Dump completo (grande — sempre su file):

istioctl -i istio-system proxy-config all <pod> -n <namespace> -o json > /tmp/dump.json

Verifica di quale route serve un host specifico:

istioctl -i istio-system proxy-config routes <pod> -n <namespace> --name http.8080 -o json

Envoy admin via pilot-agent

L’admin interface di Envoy è sulla porta 15000, raggiungibile dall’interno del container tramite pilot-agent request.

oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET server_info
oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET ready
oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET memory
oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET clusters
oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET listeners
oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET certs
oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET config_dump

Porte di riferimento:

Porta	Funzione
15000	Envoy admin interface
15001	Outbound capture
15006	Inbound capture
15020	Agent — merged metrics (/stats/prometheus)
15021	Health check (/healthz/ready)
15090	Envoy Prometheus telemetry

Statistiche e memoria

oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET stats | grep -E 'server\.(memory_allocated|memory_heap_size|memory_physical_size|total_connections|concurrency|uptime)'

oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET stats | grep -E 'cluster_manager\.active_clusters|listener_manager\.total_listeners_active|update_(success|failure|rejected)'

Cardinalità delle metriche — se supera ~50k righe, le stats stesse pesano sulla heap:

oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET stats | wc -l
oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET config_dump | wc -c

Codici di risposta upstream per cluster:

oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request GET stats | grep -E 'upstream_rq_(5xx|4xx|pending_failure_eject|timeout)'

Riduzione della cardinalità via SMCP:

spec:
  proxy:
    runtime:
      container:
        env:
          ISTIO_META_PROXY_STATS_MATCHER_INCLUSION_PREFIXES: "cluster_manager,listener_manager,server,http"

Certificati e mTLS

istioctl -i istio-system proxy-config secret <pod> -n <namespace> -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 -d | openssl x509 -noout -text

Scadenza rapida di tutti i certificati del mesh:

istioctl -i istio-system proxy-config secret <pod> -n <namespace>

Verifica del CA root usato dal control plane:

oc get cm istio-ca-root-cert -n <namespace> -o jsonpath='{.data.root-cert\.pem}' | openssl x509 -noout -dates -subject

Policy mTLS effettiva:

oc get peerauthentication -A
oc get destinationrule -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,HOST:.spec.host,TLS:.spec.trafficPolicy.tls.mode'

Log dinamici

Alzare il livello di log senza restart (torna a info al riavvio del pod):

oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request POST 'logging?level=debug'
oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request POST 'logging?level=info'

Solo alcuni logger (molto meno rumoroso):

oc exec -n istio-system <pod> -c istio-proxy -- pilot-agent request POST 'logging?connection=debug&router=debug&http=debug'

Log delle richieste in tempo reale:

oc logs -n istio-system <pod> -c istio-proxy -f | grep -v '15021'

Lato istiod:

istioctl -i istio-system admin log --level ads:debug

Risorse e restart

oc get pod -n istio-system -l app=istio-ingressgateway -o custom-columns='NAME:.metadata.name,RC:.status.containerStatuses[*].restartCount,REASON:.status.containerStatuses[*].lastState.terminated.reason,EXIT:.status.containerStatuses[*].lastState.terminated.exitCode,AT:.status.containerStatuses[*].lastState.terminated.finishedAt'

oc adm top pod -n istio-system --containers
oc get deploy istio-ingressgateway -n istio-system -o jsonpath='{.spec.template.spec.containers[?(@.name=="istio-proxy")].resources}{"\n"}'

oc get events -A --field-selector reason=OOMKilling
oc debug node/<node> -- chroot /host dmesg -T | grep -i -A3 'oom-kill'

Patch sulla SMCP

Danger

L’operator riconcilia i Deployment del mesh: ogni modifica diretta a deploy/istio-ingressgateway o deploy/istiod viene sovrascritta. Tutte le modifiche passano dalla SMCP.

Risorse del gateway:

oc patch smcp basic -n istio-system --type=merge -p '{"spec":{"gateways":{"ingress":{"runtime":{"container":{"resources":{"requests":{"cpu":"500m","memory":"1Gi"},"limits":{"cpu":"2","memory":"4Gi"}}}}}}}}'

Concurrency dei worker Envoy (di default segue i core del nodo, non il CPU limit):

oc patch smcp basic -n istio-system --type=merge -p '{"spec":{"gateways":{"ingress":{"runtime":{"pod":{"metadata":{"annotations":{"proxy.istio.io/config":"{\"concurrency\":4}"}}}}}}}}'

Repliche e HPA:

oc patch smcp basic -n istio-system --type=merge -p '{"spec":{"gateways":{"ingress":{"runtime":{"deployment":{"replicas":4}}}}}}'

Verifica dell’applicazione:

oc get smcp basic -n istio-system -o jsonpath='{.spec.gateways.ingress.runtime}{"\n"}' | jq
oc rollout status deploy/istio-ingressgateway -n istio-system --timeout=5m

Traffico e routing

oc get gateway,virtualservice,destinationrule,serviceentry -A
oc get gateway,virtualservice,destinationrule,serviceentry -A --no-headers | wc -l

Test end-to-end dall’interno del mesh:

oc exec -n <namespace> <pod> -c istio-proxy -- curl -sS -o /dev/null -w '%{http_code}\n' http://<service>.<namespace>.svc.cluster.local:<port>/

Header di debug utili sulle richieste in ingresso:

curl -sS -D- -o /dev/null -H 'x-envoy-force-trace: true' https://<host>/

I flag di risposta Envoy (response_flags nell’access log) più frequenti:

Flag	Significato
UH	Nessun host sano nell’upstream cluster
UF	Fallita la connessione all’upstream
UO	Overflow del circuit breaker
NR	Nessuna route configurata per la richiesta
URX	Superato il limite di retry o di richieste
DC	Downstream disconnesso