0

I am trying to run celery with the command celeryworker [1] based on this configuration [2], but I get the error [3] when I launch the program. My program is running in medusa1-blank1, and the rabbitmq-server is running in hadoop-medusa-1. You can see in [1] that the $HOST_NAME variable is the medusa1-blank1 and that the celeryconfig.py contains the host address where rabbitmq-server is running.

I looked to my configuration, and I cannot find any error in it. I would like that the log could be more verbose to understand what is going on, but I also don't think that it is possible to do that. Since it looks that the error is not in my code, I am completely clueless in understanding what is going on. Any help to try to debug this?

[1] Script that I use to run with celery

#!/bin/bash
set -xv

# This scripts runs celery in the server host

export C_FORCE_ROOT="true"
HOST_NAME=`hostname`

echo "------------------------"
echo "Initialize celery at $HOST_NAME"
echo "------------------------"
celery worker -n $HOST_NAME -E --loglevel=DEBUG --concurrency=20 -f ./logs/celerydebug.log --config=celeryconfig -Q $HOST_NAME 
# celery worker -n medusa1-blank1 -E --loglevel=DEBUG --concurrency=20 -f ./logs/celerydebug.log --config=celeryconfig -Q medusa1-blank1

[2] Configuration that I use:

(medusa-env)xubuntu@medusa1-blank1:~/Programs/medusa-1.0$ cat celeryconfig.py
import os
import sys

# add hadoop python to the env, just for the running
sys.path.append(os.path.dirname(os.path.basename(__file__)))

# broker configuration
BROKER_URL = "amqp://celeryuser:celery@hadoop-medusa-1/celeryvhost"

CELERY_RESULT_BACKEND = "amqp"
CELERY_RESULT_PERSISTENT = True
TEST_RUNNER = 'celery.contrib.test_runner.run_tests'

# for debug
# CELERY_ALWAYS_EAGER = True

# module loaded
CELERY_IMPORTS = ("manager.mergedirs", "manager.system", "manager.utility", "manager.pingdaemon", "manager.hdfs")

[3] Error that I have:

[2016-03-07 10:24:09,482: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2016-03-07 10:24:09,484: DEBUG/MainProcess] | Worker: Building graph...
[2016-03-07 10:24:09,484: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Queues (intra), Pool, Autoscaler, Autoreloader, StateDB, Beat, Consumer}
[2016-03-07 10:24:09,487: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2016-03-07 10:24:09,487: DEBUG/MainProcess] | Consumer: Building graph...
[2016-03-07 10:24:09,491: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Agent, Events, Mingle, Tasks, Control, Heart, Gossip, event loop}
[2016-03-07 10:24:09,491: WARNING/MainProcess] /home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/apps/worker.py:161: CDeprecationWarning: 
Starting from version 3.2 Celery will refuse to accept pickle by default.

The pickle serializer is a security concern as it may give attackers
the ability to execute any command.  It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.

If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::

    CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

You must only enable the serializers that you will actually use.


  warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
[2016-03-07 10:24:09,493: ERROR/MainProcess] Unrecoverable error: AttributeError("'NoneType' object has no attribute 'rstrip'",)
Traceback (most recent call last):
  File "/home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "/home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    self.on_start()
  File "/home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/apps/worker.py", line 169, in on_start
    string(self.colored.cyan(' \n', self.startup_info())),
  File "/home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/apps/worker.py", line 230, in startup_info
    results=self.app.backend.as_uri(),
  File "/home/xubuntu/Programs/medusa-1.0/medusa-env/local/lib/python2.7/site-packages/celery/backends/base.py", line 117, in as_uri
    else maybe_sanitize_url(self.url).rstrip("/"))
AttributeError: 'NoneType' object has no attribute 'rstrip'
4
  • what the result without -n $HOST_NAME ? Commented Mar 7, 2016 at 15:53
  • It is the name of the hostname (medusa1-blank1), but it is not the ip address where rabbitmq-server is running (hadoop-medusa-1). Commented Mar 7, 2016 at 16:14
  • Try to run celery without this option :) celery node name will be create as celery@hostname (by default) Commented Mar 7, 2016 at 16:16
  • I get the same error when I run this command celery worker -E --loglevel=DEBUG --concurrency=20 -f ./logs/celerydebug.log --config=celeryconfig Commented Mar 7, 2016 at 16:20

2 Answers 2

1

Don't know which version you're using, but found this bug report:

https://github.com/celery/celery/issues/3094

bottom line, roll back for now.

Sign up to request clarification or add additional context in comments.

Comments

0

In my minimum configuration file would be:

CELERY_IMPORTS = ...
AMPQ_USERNAME = os.getenv('AMQP_USERNAME', '...')
AMPQ_PASSWORD = os.getenv('AMQP_PASSWORD', '...')
AMQP_HOST = os.getenv('AMQP_HOST', '172.17.42.1')
AMQP_PORT = int(os.getenv('AMQP_PORT', '5672'))
DEFAULT_BROKER_URL = 'amqp://%s:%s@%s:%d'\
                     % (AMPQ_USERNAME, AMPQ_PASSWORD, AMQP_HOST, AMQP_PORT)

CELERY_RESULT_BACKEND = 'amqp://%s:%s@%s:%d'\
                     % (AMPQ_USERNAME, AMPQ_PASSWORD, AMQP_HOST, AMQP_PORT)

BROKER_API = DEFAULT_BROKER_URL

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.