I am trying to run celery with the command celeryworker [1] based on this configuration [2], but I get the error [3] when I launch the program. My program is running in medusa1-blank1, and the rabbitmq-server is running in hadoop-medusa-1. You can see in [1] that the $HOST_NAME variable is the medusa1-blank1 and that the celeryconfig.py contains the host address where rabbitmq-server is running.
I looked to my configuration, and I cannot find any error in it. I would like that the log could be more verbose to understand what is going on, but I also don't think that it is possible to do that. Since it looks that the error is not in my code, I am completely clueless in understanding what is going on. Any help to try to debug this?
[1] Script that I use to run with celery
#!/bin/bash
set -xv
# This scripts runs celery in the server host
export C_FORCE_ROOT="true"
HOST_NAME=`hostname`
echo "------------------------"
echo "Initialize celery at $HOST_NAME"
echo "------------------------"
celery worker -n $HOST_NAME -E --loglevel=DEBUG --concurrency=20 -f ./logs/celerydebug.log --config=celeryconfig -Q $HOST_NAME
# celery worker -n medusa1-blank1 -E --loglevel=DEBUG --concurrency=20 -f ./logs/celerydebug.log --config=celeryconfig -Q medusa1-blank1
[2] Configuration that I use:
(medusa-env)xubuntu@medusa1-blank1:~/Programs/medusa-1.0$ cat celeryconfig.py
import os
import sys
# add hadoop python to the env, just for the running
sys.path.append(os.path.dirname(os.path.basename(__file__)))
# broker configuration
BROKER_URL = "amqp://celeryuser:celery@hadoop-medusa-1/celeryvhost"
CELERY_RESULT_BACKEND = "amqp"
CELERY_RESULT_PERSISTENT = True
TEST_RUNNER = 'celery.contrib.test_runner.run_tests'
# for debug
# CELERY_ALWAYS_EAGER = True
# module loaded
CELERY_IMPORTS = ("manager.mergedirs", "manager.system", "manager.utility", "manager.pingdaemon", "manager.hdfs")
[3] Error that I have:
[2016-03-07 10:24:09,482: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2016-03-07 10:24:09,484: DEBUG/MainProcess] | Worker: Building graph...
[2016-03-07 10:24:09,484: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Queues (intra), Pool, Autoscaler, Autoreloader, StateDB, Beat, Consumer}
[2016-03-07 10:24:09,487: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2016-03-07 10:24:09,487: DEBUG/MainProcess] | Consumer: Building graph...
[2016-03-07 10:24:09,491: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Agent, Events, Mingle, Tasks, Control, Heart, Gossip, event loop}
[2016-03-07 10:24:09,491: WARNING/MainProcess] /home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/apps/worker.py:161: CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.
The pickle serializer is a security concern as it may give attackers
the ability to execute any command. It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.
If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::
CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
You must only enable the serializers that you will actually use.
warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
[2016-03-07 10:24:09,493: ERROR/MainProcess] Unrecoverable error: AttributeError("'NoneType' object has no attribute 'rstrip'",)
Traceback (most recent call last):
File "/home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
self.blueprint.start(self)
File "/home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
self.on_start()
File "/home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/apps/worker.py", line 169, in on_start
string(self.colored.cyan(' \n', self.startup_info())),
File "/home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/apps/worker.py", line 230, in startup_info
results=self.app.backend.as_uri(),
File "/home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/backends/base.py", line 117, in as_uri
else maybe_sanitize_url(self.url).rstrip("/"))
AttributeError: 'NoneType' object has no attribute 'rstrip'
hostname(medusa1-blank1), but it is not the ip address whererabbitmq-serveris running (hadoop-medusa-1).celery worker -E --loglevel=DEBUG --concurrency=20 -f ./logs/celerydebug.log --config=celeryconfig