4

I use docker container with mysql offical images to create more than 11 database container, (container1 to container11). after setting up, all container running fine until container9. At container10, it only starts up about 1 mins and stop again. using docker logs to check container but I do not see anything. stop container9, and restart container10. It runs fine again. The situation seems only happen when I have 9 mysql container and trying to raise up 10th. If I stop one of them, and raise it up again. Then there is no problem. Is it bugs? or I miss some setting for docker bridge?

root@ec8dcb82f64d:/dev/shm# docker restart f4801b57c4cc
f4801b57c4cc
root@ec8dcb82f64d:/dev/shm# docker ps -a | grep f4801b57c4cc
f4801b57c4cc        mysql/mysql-server:5.7             "/entrypoint.sh my..."   2 weeks ago         Exited (1) 3 seconds ago                                                             db
root@ec8dcb82f64d:/dev/shm# docker logs f4801b57c4cc
Initializing database
Database initialized
MySQL init process in progress...
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.

/entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*


MySQL init process done. Ready for start up.

root@ec8dcb82f64d:/dev/shm#
8
  • How much memory does you system have? Commented Aug 2, 2017 at 18:30
  • 64g, 52g used, 11G free Commented Aug 2, 2017 at 19:11
  • Are you running this inside a docker container? So docker in docker? Commented Aug 3, 2017 at 4:15
  • yes, and it also happen when I start docker on my aws ec2 Commented Aug 3, 2017 at 12:40
  • Do a df -h inside the docker image when the failure happens. Commented Aug 3, 2017 at 12:44

2 Answers 2

16

I think I hit the solution after one week even though I do not really understand what happens. the following is what I tried and so far, I can bring up mysql container up to 20 with no problems.

1: Try to create a dummpy mysql container for testing

$ docker run -e MYSQL_ROOT_PASSWORD=password mysql
Unable to find image 'mysql:latest' locally
latest: Pulling from library/mysql
...
Initializing database
2017-08-09T17:58:30.034595Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-08-09T17:58:30.039274Z 0 [Warning] InnoDB: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
2017-08-09T17:58:30.039294Z 0 [Warning] InnoDB: io_setup() attempt 1.
2017-08-09T17:58:30.539495Z 0 [Warning] InnoDB: io_setup() attempt 2.
2017-08-09T17:58:31.039701Z 0 [Warning] InnoDB: io_setup() attempt 3.
2017-08-09T17:58:31.539902Z 0 [Warning] InnoDB: io_setup() attempt 4.
2017-08-09T17:58:32.040115Z 0 [Warning] InnoDB: io_setup() attempt 5.
2017-08-09T17:58:32.540330Z 0 [ERROR] InnoDB: io_setup() failed with EAGAIN after 5 attempts.
2017-08-09T17:58:32.540378Z 0 [ERROR] InnoDB: Cannot initialize AIO sub-system
2017-08-09T17:58:32.540390Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2017-08-09T17:58:32.540401Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2017-08-09T17:58:32.540408Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-08-09T17:58:32.540412Z 0 [ERROR] Failed to initialize plugins.
2017-08-09T17:58:32.540415Z 0 [ERROR] Aborting

and hit the error code for io_setup() failed with EAGAIN

2: Examines the current value of aio-max-nr

$ sysctl fs.aio-max-nr
fs.aio-max-nr = 65536

3: Increase the value of aio-max-nr to 2097152

$ sudo sysctl -w fs.aio-max-nr=2097152

3: Start mysql service

4: Try to create more mysql-containers and bring up original one with no problems

Sign up to request clarification or add additional context in comments.

1 Comment

Thanks bro, I got io_setup() error on my 26th mysql container, now it runs smoothly
-1

Run docker events in background and then try starting your 10th container. You will surely see what's going wrong. Given below is an example to start an exited container and error while starting it. Commands below in sequence:

~$ sudo docker events &
[1] 9414

~$ sudo docker start 48137950f1b7
2017-08-03T00:01:18.971406558+05:30 network connect c79096ff0fef046d24b2a23907b3cc82c4df0838db2475909f8fa9f796a0418e (container=48137950f1b714797143529d63ec7221d3cbcd38bb6c8d20a241b06ddbd3d27a, name=bridge, type=bridge)
2017-08-03T00:01:19.305063392+05:30 container start 48137950f1b714797143529d63ec7221d3cbcd38bb6c8d20a241b06ddbd3d27a (image=ubuntu, name=modest_northcutt)
48137950f1b7
2017-08-03T00:01:19.305915636+05:30 container die 48137950f1b714797143529d63ec7221d3cbcd38bb6c8d20a241b06ddbd3d27a (exitCode=0, image=ubuntu, name=modest_northcutt)

Hope you will figure out.

1 Comment

network connect 0303...dec (container=<container id>, name=bridge, type=bridge) volume mount 5a...117 (container=<container id>, destination=/var/lib/mysql, driver=local, propagation=, read/write=true) container start <container id> (image=mysql/mysql-server:5.7, name=<container name>) container die <container id> (exitCode=1, image=mysql/mysql-server:5.7, name=<container name>) network disconnect 0303...dec (container=<container id>, name=bridge, type=bridge) volume unmount 5a...117 (container=<container id>, driver=local) expect all process, I cannot see any further information

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.