Debugging workloads¶

A useful place to start when you have problems getting your pods running is the troubleshooting guide.

It can also be useful to check out the official documentation from Kubernetes. Keep in mind that in the kubernetes documentation, the term "application" is a more general concept that a NAIS application.

• Troubleshoot Applications
• Debug Running Pods

kubectl debug¶

You can run an ephemeral container in a pod using the kubectl debug command.

The following example starts a shell in a new ephemeral container in the my-pod-name pod using the nais debug image:

kubectl debug -it my-pod-name --image="europe-north1-docker.pkg.dev/nais-io/nais/images/debug:latest" --profile=restricted

Once the ephemeral container is created, you will be presented with a shell prompt where you can try debugging the issue.

kubectl attach¶

If you want to attach to a running process inside a running container you need to use kubectl attach. You can read more about the command over at kubernetes.io/docs.

# Switch to raw terminal mode; sends stdin to 'bash' in ruby-container from pod mypod
# and sends stdout/stderr from 'bash' back to the client
kubectl attach mypod -c ruby-container -i -t

Debugging Memory Leaks¶

If you experience memory leaks in Java processes you can get heap dumps either automatically on OOM or on-demand.

Automatically on OOM¶

Set JAVA_OPTS to -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp

The /tmp volume is maintained through restarts, so if your app is restarting because of OOM, the heap dumps can be retrieved from there.

Manually on-demand¶

You can use jmap to create a heap dump of a running Java process.

Find a pod and exec jmap in it (assuming PID 1 is the Java process):

kubectl get pods -l app=[app-name]
kubectl exec [pod-name] -- jmap -dump:all,file=/tmp/heap.hprof 1

Getting the heap dump¶

You can use kubectl cp to get the files from the pod to your local computer:

kubectl cp [pod-name]:/tmp/heap.hprof ./heap.hprof

You can inspect the heap dumps with tools like JProfiler, VisualVM or IntelliJ.

FAQ¶

I get an HTTP 503 Service Unavailable error when visiting the ingress for my application, why?¶

My application gets an HTTP 504 Gateway Timeout error when attempting to communicate with another application, why?¶