Connect AKS to Azure Container Registry and Fix ImagePullBackOff

Your deployment is applied, the pods are stuck, and kubectl get pods shows ErrImagePull followed by ImagePullBackOff. The image is definitely in Azure Container Registry, you can pull it from your laptop, yet the cluster cannot. Nine times out of ten the cause is one of a handful of things: the cluster identity has no pull permission, the tag does not exist, or the node cannot reach the registry over the network.

This guide covers the correct way to connect Azure Kubernetes Service (AKS) to ACR, how to verify it, the cross-subscription, cross-tenant and private-network variants, and a repeatable flow for diagnosing pull failures from the pod events. For registry fundamentals such as login servers and SKUs, see Introduction to Azure Container Registry.

How AKS authenticates to ACR

Pods do not pull images; the kubelet on each node does. On AKS with managed identity (the default), the nodes of all node pools run with a user-assigned managed identity called the kubelet identity, created in the node resource group (MC_rg-apps_aks-apps_<region>) and named like aks-apps-agentpool. When the kubelet pulls crappsdemo.azurecr.io/orders-api:1.4.0, it uses that identity to get an Entra ID token and exchanges it for a registry token.

So "connecting AKS to ACR" is nothing more than an Azure RBAC role assignment: the kubelet identity needs AcrPull on the registry. No Kubernetes secret, no password, nothing to rotate. This is also why you should keep the registry admin user disabled: the admin account is one shared credential with full push rights that cannot be scoped or individually audited, and once the kubelet identity has AcrPull there is no workload that needs it.

Note what the kubelet identity is not: it is not the cluster (control plane) identity, and it is not a workload identity used by your application code. Granting AcrPull to either of those does nothing for image pulls.

Attach the registry with --attach-acr

For an existing cluster:

az aks update \
  --resource-group rg-apps \
  --name aks-apps \
  --attach-acr crappsdemo

For a new cluster, pass the same flag at creation time:

az aks create \
  --resource-group rg-apps \
  --name aks-apps \
  --node-count 3 \
  --generate-ssh-keys \
  --attach-acr crappsdemo

--attach-acr accepts either the registry name (when it is in the current subscription) or its full resource ID. Behind the scenes it creates an AcrPull role assignment for the kubelet identity scoped to the registry. Because it writes a role assignment, the person or pipeline running it needs Owner, User Access Administrator or Role Based Access Control Administrator on the registry; Contributor is not enough, and the command fails with an authorization error on Microsoft.Authorization/roleAssignments/write.

To remove it later, use --detach-acr crappsdemo.

Doing it manually or in IaC

In Bicep or Terraform you usually create the role assignment yourself. The CLI equivalent shows exactly what --attach-acr does:

KUBELET_ID=$(az aks show -g rg-apps -n aks-apps \
  --query identityProfile.kubeletidentity.objectId -o tsv)
ACR_ID=$(az acr show -g rg-apps -n crappsdemo --query id -o tsv)

az role assignment create \
  --assignee-object-id $KUBELET_ID \
  --assignee-principal-type ServicePrincipal \
  --role AcrPull \
  --scope $ACR_ID

Scope the assignment to the registry, not the resource group or subscription. A kubelet identity with subscription-wide AcrPull can pull from every registry in that subscription, which is rarely intended.

Verify with az aks check-acr

Before deploying anything, let AKS test the path end to end:

az aks check-acr \
  --resource-group rg-apps \
  --name aks-apps \
  --acr crappsdemo.azurecr.io

The command runs a short-lived validation pod on the cluster that checks DNS resolution of the login server, network reachability, and whether the kubelet identity can obtain a registry token. It reports each check separately, which makes it the fastest first step when pulls fail too. Pass the login server (crappsdemo.azurecr.io), not just the registry name. If your nodes run a restricted egress path, remember that the validation pod image itself must be pullable.

You can also confirm the role assignment directly:

az role assignment list --assignee $KUBELET_ID --all \
  --query "[].{role:roleDefinitionName, scope:scope}" -o table

Cross-subscription and cross-tenant registries

Different subscription, same tenant

This is common: a shared registry in a platform subscription serving clusters in workload subscriptions. Pass the full resource ID so the CLI does not look for the registry in the current subscription:

az aks update -g rg-apps -n aks-apps \
  --attach-acr /subscriptions/<shared-sub-id>/resourceGroups/rg-shared/providers/Microsoft.ContainerRegistry/registries/crappsdemo

The caller needs role-assignment rights on the registry in the other subscription. If the cluster team does not have those, the platform team can run the manual az role assignment create against the kubelet identity's object ID instead.

Different tenant

Managed identities cannot be granted roles in another Entra tenant, so --attach-acr does not work across tenants. Options, in order of preference:

1. Bring the images to a registry in the cluster's tenant with az acr import (a pipeline step that copies tagged images), then attach that registry normally.
2. Use a multitenant app registration that has a service principal in the registry's tenant with AcrPull, and supply its credentials as an imagePullSecret. This reintroduces a secret you must rotate.
3. Use a repository-scoped ACR token as an imagePullSecret (shown below).

Importing is usually the cleanest: the cluster only ever talks to a registry in its own tenant and network, and you control exactly which images cross the boundary.

Private registries: private endpoint and DNS

When the registry has public network access disabled, nodes must reach it through a private endpoint (Premium SKU). The part that breaks is almost always DNS: a registry has two hostnames that must resolve to private IPs, the login server crappsdemo.azurecr.io and the regional data endpoint crappsdemo.<region>.data.azurecr.io that serves layer blobs.

ACR_ID=$(az acr show -g rg-apps -n crappsdemo --query id -o tsv)
VNET_ID=$(az network vnet show -g rg-apps -n vnet-apps --query id -o tsv)

az network private-dns zone create -g rg-apps -n privatelink.azurecr.io

az network private-dns link vnet create -g rg-apps \
  --zone-name privatelink.azurecr.io \
  --name link-vnet-apps \
  --virtual-network $VNET_ID \
  --registration-enabled false

az network private-endpoint create -g rg-apps \
  --name pe-crappsdemo \
  --vnet-name vnet-apps \
  --subnet snet-private-endpoints \
  --private-connection-resource-id $ACR_ID \
  --group-ids registry \
  --connection-name pe-crappsdemo-conn

az network private-endpoint dns-zone-group create -g rg-apps \
  --endpoint-name pe-crappsdemo \
  --name default \
  --private-dns-zone privatelink.azurecr.io \
  --zone-name privatelink-azurecr-io

az acr update -n crappsdemo --public-network-enabled false

The DNS zone group creates and maintains the A records for both hostnames. Check them with az network private-dns record-set a list -g rg-apps -z privatelink.azurecr.io -o table; you should see an entry for crappsdemo and one for crappsdemo.<region>.data.

Pitfalls that cause pulls to hang or time out:

• The AKS VNet uses custom DNS servers that do not forward azurecr.io queries to Azure DNS (168.63.129.16) from a VNet linked to the zone, so the private records are never consulted.
• The cluster is in a spoke VNet not linked to the zone, so nodes resolve the public IP and are refused.
• Geo-replicated registries need data endpoint records for every replica region; after adding a replica, confirm the new data endpoint record exists.

imagePullSecrets for clusters outside AKS

Clusters that have no Azure managed identity, such as on-premises Kubernetes, another cloud, or local kind clusters, need a credential. Do not use the admin user. Create a repository-scoped token with pull-only rights instead:

az acr token create \
  --registry crappsdemo \
  --name k8s-onprem-pull \
  --scope-map _repositories_pull \
  --query "credentials.passwords[0].value" -o tsv

For narrower access, create your own scope map with az acr scope-map create and --repository orders-api content/read. Then store it in the cluster:

kubectl create secret docker-registry acr-pull \
  --namespace orders \
  --docker-server=crappsdemo.azurecr.io \
  --docker-username=k8s-onprem-pull \
  --docker-password='<token-password>'

kubectl patch serviceaccount default -n orders \
  -p '{"imagePullSecrets":[{"name":"acr-pull"}]}'

Patching the service account means every pod using it gets the secret without editing each manifest. Token passwords can be generated with an expiry (az acr token credential generate --expiration-in-days), so put rotation on a calendar or automate it.

A systematic ImagePullBackOff troubleshooting flow

ImagePullBackOff is not an error in itself; it is Kubernetes waiting before retrying a failed pull, with an increasing delay. The real error is in the pod events:

kubectl get pods -n orders
kubectl describe pod orders-api-7c9f6d8b5-x2kqp -n orders
kubectl get events -n orders --field-selector reason=Failed --sort-by=.lastTimestamp

Read the Failed to pull image message in full and match it:

Step 1: is the reference correct?

Copy the image string from kubectl describe and check it character by character. Registry host names and repository names must be lowercase, and a missing namespace prefix (orders-api versus team/orders-api) produces not found. Then confirm the tag exists:

az acr repository show-tags -n crappsdemo --repository orders-api --orderby time_desc --top 10 -o table
az acr manifest list-metadata -r crappsdemo -n orders-api \
  --query "[?tags!=null].{digest:digest, tags:tags, updated:lastUpdateTime}" -o table

If you deploy by SHA tags from CI, the most common "not found" is a deployment that ran before the push finished, or a push that failed silently.

Step 2: is it authentication?

A 401 means the kubelet reached the registry but presented no valid token. Run az aks check-acr and inspect role assignments for the kubelet identity. Role assignments can take several minutes to propagate; a pull that fails immediately after --attach-acr may succeed on its own a little later. If you need a faster retry, delete the pod rather than waiting out the backoff.

Step 3: is it the network?

Test from inside the cluster, from the same network path the kubelet uses:

kubectl run nettest -it --rm --restart=Never \
  --image=mcr.microsoft.com/azure-cli --command -- bash

# inside the pod
getent hosts crappsdemo.azurecr.io
curl -sS -o /dev/null -w "%{http_code}\n" https://crappsdemo.azurecr.io/v2/

A 401 from /v2/ is the good outcome: DNS works, TLS works and the registry answered, so any remaining problem is authorization. A private IP from getent hosts confirms the private endpoint path. A public IP on a private-only registry points at DNS; a hang points at egress rules. The kubelet pulls from the node itself, which normally shares DNS servers and egress with pods; if the results look inconsistent, repeat the test from a node with kubectl debug node/<node-name>.

Firewall and egress considerations

If the cluster uses outbound type userDefinedRouting through Azure Firewall or another appliance, image pulls need explicit egress rules:

• The login server, crappsdemo.azurecr.io.
• The data endpoint for each region you pull from, crappsdemo.<region>.data.azurecr.io. Enable dedicated data endpoints with az acr update -n crappsdemo --data-endpoint-enabled true so you can allow specific FQDNs instead of broad storage wildcards.
• mcr.microsoft.com and *.data.mcr.microsoft.com, which AKS system components pull from.

The AzureContainerRegistry service tag can be used in NSG and firewall network rules, but it covers all registries in the region, so FQDN application rules are tighter.

In the opposite direction, if the registry has IP network rules (Premium), the cluster's public egress IP must be allowed. Find it with:

for id in $(az aks show -g rg-apps -n aks-apps \
  --query "networkProfile.loadBalancerProfile.effectiveOutboundIPs[].id" -o tsv); do
  az network public-ip show --ids "$id" --query ipAddress -o tsv
done

Outbound IPs change if you recreate the cluster or change the outbound type, so a private endpoint is more durable than IP allow lists.

Related guides

• Introduction to Azure Container Registry
• Push Docker Images to ACR with GitHub Actions
• Securing Azure Container Registry
• Clean Up ACR with Retention and az acr purge

Conclusion

AKS to ACR integration is a single AcrPull role assignment for the kubelet identity, and az aks update --attach-acr creates it for you. Verify it with az aks check-acr before your first deployment, keep the admin user disabled, and use repository-scoped tokens only for clusters that have no Azure identity. When a pod lands in ImagePullBackOff, skip the guessing: read the event message from kubectl describe pod, then decide whether you are looking at a wrong reference (not found, platform mismatch), an authorization problem (401, 403), or a network problem (DNS failure, timeout). Each class has a small set of causes, and the commands above narrow it down in minutes.