I have a deployment with multiple replica pods behind a service. Sometimes, I want to know which of these pods is serving a request, i.e. to view the logs when there is some strange behaviour. Is there a way to add an HTTP header like X-Pod: my-app-5954c566c7-48s97 to each request served by an ingress using the NGINX Ingress Controller, for example through annotations?
Add a comment
|
2 Answers
Update
After enabling Ingress snippets in my MicroK8s cluster by adding allow-snippet-annotations: "true", I found that my original approach to setting headers via the ingress didn't work as expected. The closest result I achieved was retrieving the pod name for the Ingress controller, which is not the intended behavior.
The short answer is: No, you can't get the individual pod's hostname via the Ingress controller. You need to set this header inside your application, as the Ingress controller doesn't have direct knowledge of individual pod details.
Normally, the HOSTNAME environment variable is available inside all pods. You can verify it with:
kubectl -n my-ns exec -it my-pod-name -- sh -c 'echo $HOSTNAME'
Therefore, if you're using a framework like Node.js, you can set the X-Pod header directly in the application like this:
Node.js example:
const hostname = process.env.HOSTNAME
app.get("/my-endpoint", (req, res) => {
res.set("X-Pod", hostname)
...
})
Old answer
Yes, you can achieve this by using annotation and snippet. $HOSTNAME will contain the pod name.
kind: Ingress
metadata:
name: some-ingress
annotations:
nginx.ingress.kubernetes.io/server-snippet: |
proxy_set_header X-Pod $HOSTNAME;
Comments
If you deploy a ingress-nginx controller, it will create a default ConfigMap with the name of your ingress controller. This is the one you want to modify.
There would be something like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: {name-of-the-helm-chart}-nginx-ingress-controller
namespace: {namespace-where-the-nginx-ingress-is-deployed}
date:
...
The config of log-format-upstream in the data should be added:
data:
allow-snippet-annotations: "true"
enable-real-ip: "true"
log-format-upstream: "[$time_local] $remote_addr - $remote_user - $server_name $host to: $upstream_addr: $request $status upstream_response_time $upstream_response_time msec $msec request_time $request_time"
Note using $upstream_addr that would be the IP address of pod.
Another example would be:
log-format-upstream: '$remote_addr $host $remote_user [$time_local] $request $status $body_bytes_sent $http_referer $http_user_agent $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id
But if you want your ingress controller to log the pod name by X-My-Pod-Name value, you can set this HEADER in your service. Like for example having the manifest of service updated to have env var:
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
then for example in python, something like:
resp.headers['X-My-Pod-Name'] = os.environ.get('POD_NAME')
and finally you can show it in logs by $http_x_my_pod_name:
log-format-upstream: $remote_addr - $remote_user [$time_local] "$request" $status
$body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_my_pod_name"
