1

I'm trying to implement continuous profiling for our microservices running on ECS with Amazon Linux 2 hosts, but I'm running into persistent issues when trying to run profiling agents. I've tried several different approaches, and they all fail with the same error:

CannotStartContainerError: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /proc/sys/net/ipv4/

Environment Details

Host OS: Amazon Linux 2 (Latest Image) Container orchestration: AWS ECS Deployment method: Terraform

What I've Tried

I've attempted to implement the following profiling solutions:

Parca Agent:

{

"name": "container",

"image": "ghcr.io/parca-dev/parca-agent:v0.16.0",

"essential": true,

"privileged": true,

"mountPoints": [

{ "sourceVolume": "proc", "containerPath": "/proc", "readOnly": false },

{ "sourceVolume": "sys", "containerPath": "/sys", "readOnly": false },

{ "sourceVolume": "cgroup", "containerPath": "/sys/fs/cgroup", "readOnly": false },

{ "sourceVolume": "hostroot", "containerPath": "/host", "readOnly": true }

],

"command": ["--server-address=http://parca-server:7070", "--node", "--threads", "--cpu-time"]

},

OpenTelemetry eBPF Profiler:

{

"name": "container",

"image": "otel/opentelemetry-ebpf-profiler-dev:latest",

"essential": true,

"privileged": true,

"mountPoints": [

{ "sourceVolume": "proc", "containerPath": "/proc", "readOnly": false },

{ "sourceVolume": "sys", "containerPath": "/sys", "readOnly": false },

{ "sourceVolume": "cgroup", "containerPath": "/sys/fs/cgroup", "readOnly": false },

{ "sourceVolume": "hostroot", "containerPath": "/host", "readOnly": true }

],

"linuxParameters": {

"capabilities": { "add": ["ALL"] }

}

}

Doesnt Matter what i try, I always get the same error :

CannotStartContainerError: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /proc/sys/net/ipv4/ What

I've Already Tried:

  • Setting privileged: true Mounting /proc, /sys, /sys/fs/cgroup with readOnly: false
  • Adding ALL Linux capabilities to the task definition and at the service level
  • Tried different network modes: host, bridge, and awsvpc
  • Tried running as root user with user: "root" and "0:0" Disabled no-new-privileges security option

Is there a known limitation with Amazon Linux 2 that prevents containers from accessing /proc/sys/net/ipv4/ even with privileged mode?

Are there any specific kernel parameters or configurations needed for ECS hosts to allow profiling agents to work properly?

Has anyone successfully run eBPF-based profilers or other kernel-level profiling tools on ECS with Amazon Linux 2?

I would really like some help, im new to SRE and this is for my own knowledge

Thanks in Advance

Pd: No, migrating to K8s is not an option.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.