Troubleshooting Tips for Monitoring Go Applications in Kubernetes

Troubleshooting tips for monitoring Go applications in Kubernetes

This guide helps you diagnose and fix common issues when monitoring Go applications in Kubernetes using the APM Insight Go agent.

The Go agent is not discovering the application

  1. SymptomsYou do not see any logs containing the matched Go process, which means the agent is unable to detect your Go application.
  2. Required checks:
    1. Verify that the GO_APPS value in the ConfigMap matches your Go binary name:
kubectl get configmap apm-apps-config -n monitoring -o yaml
    1. Check the actual binary name inside the application pod to ensure it matches the ConfigMap entry:
kubectl exec -it my-go-app-xxxxx -- ps aux | grep go
    1. Confirm that the binary has been copied to the hostPath directory used for instrumentation:
kubectl exec -it -n monitoring exporter-agent-xxxxx -- \
 ls -lh /var/lib/apm-binaries/bin/
    1. Ensure the application pod and the exporter-agent pod are running on the same node so that the agent can discover the process:
  kubectl get pods -A -o wide | grep -E "my-go-app|exporter-agent"

The eBPF probes are not attaching

  1. Symptoms: The instrumentor logs contain errors such as Failed to attach uprobe, indicating that the agent could not attach the extended Berkeley Packet Filter (eBPF) probes to the Go binary.
  2. Required checks:
    1. Verify that the kernel version is 5.8 or higher, as eBPF instrumentation requires a compatible kernel:
kubectl exec -n monitoring exporter-agent-xxxxx -- uname -r
    1. Check whether the Go binary is stripped. The binary must contain symbol tables like .gopclntab for eBPF attachment:
kubectl exec -n monitoring exporter-agent-xxxxx -- \
readelf -S /var/lib/apm-binaries/bin/my-go-app | grep -E ".gopclntab|.debug"
    1. Confirm that the BPF filesystem is mounted correctly inside the agent pod:
kubectl exec -n monitoring exporter-agent-xxxxx -- \ 
 mount | grep bpf
    1. List the loaded eBPF programs to verify probe attachment:
kubectl exec -n monitoring exporter-agent-xxxxx -- \
 bpftool prog list

Permission denied errors are occurring

  1. Symptoms: The agent logs display messages such as Permission denied or Operation not permitted, indicating insufficient privileges.
  2. Required checks:
    1. Verify that the DaemonSet is running in privileged mode:
kubectl get daemonset exporter-agent -n monitoring -o yaml | grep privileged
    1. Confirm that the security context is configured correctly to allow eBPF operations:
kubectl get daemonset exporter-agent -n monitoring -o yaml | grep -A 10 securityContext
    1. Ensure that hostPID: true is enabled so the agent can inspect processes across the node:
kubectl get daemonset exporter-agent -n monitoring -o yaml | grep hostPID

No spans are exported from the application

  1. Symptoms: The instrumentor logs show No spans received, indicating that the application is not generating telemetry or the agent is not exporting it.
  2. Required checks:
    1. Ensure the application is actually receiving traffic on the expected port:
kubectl exec -it my-go-app-xxxxx -- netstat -tlnp | grep 8080
    1. Verify that the exporter agent can communicate with the back end by checking its logs:
kubectl exec -n monitoring exporter-agent-xxxxx -- \
 tail -f /opt/S247DataExporter/ logs/service /service.log
    1. Confirm that the license key in the secret is correct and valid:
kubectl get secret apm-license-key -n monitoring -o yaml

How to upgrade the Go agent

You can perform a rolling update of the agent through the DaemonSet.
# Update image version
kubectl set image daemonset/exporter-agent \
 exporter-agent=site24x7/apminsight-go-agent:latest \
 -n monitoring

Monitor the rollout to ensure the update is applied successfully:
# Monitor rollout
kubectl rollout status daemonset/exporter-agent -n monitoring

How to uninstall the Go agent

To uninstall the agent, follow these steps:
Delete the DaemonSet:
kubectl delete daemonset exporter-agent -n monitoring

Delete the ConfigMap and Secret:
kubectl delete configmap apm-apps-config -n monitoring
kubectl delete secret apm-license-key -n monitoring

Related article
    • Related Articles

    • Troubleshooting tips for monitoring Go applications in Docker

      If you encounter issues with Go application monitoring in Docker, follow these troubleshooting steps to identify and resolve common problems quickly. The container fails to start Symptom: The APM container terminates immediately after starting. ...
    • Basic troubleshooting tips for the APM Insight Node.js agent

      If you are experiencing issues after installing the APM Insight Node.js agent, follow these troubleshooting tips. Verify agent installation and connectivity To verify the agent installation status, you can execute the following command from the root ...
    • How to uninstall the Site24x7 Kubernetes monitoring agent

      Execute the below commands to uninstall the Site24x7 Kubernetes monitoring agent from your cluster: Deletion command for the DaemonSet agent deployed using a YAML file: kubectl delete -f site24x7-agent.yaml or kubectl delete ds,sa site24x7-agent -n ...
    • Troubleshooting tips for log collection errors

      The following is a list of possible log collection errors and the solutions to fix them: Error message Reason Solution Upload Limit Reached You might have reached the maximum upload limit configured for this log type in the current billing cycle. ...
    • List of log files generated and maintained by the Site24x7 server monitoring agent

      Here is the list of log files generated by the Linux and Windows server monitoring agent. The log files are stored in the same directory where the agent source file is installed. By default they are at: For Linux : ...