13

I've been pulling my hair out trying to figure this one out, hoping someone else has already encountered this and knows how to solve it :)

I'm trying to build a very simple Flask endpoint that just needs to call a long running, blocking php script (think while true {...}). I've tried a few different methods to async launch the script, but the problem is my browser never actually receives the response back, even though the code for generating the response after running the script is executed.

I've tried using both multiprocessing and threading, neither seem to work:

# multiprocessing attempt
@app.route('/endpoint')
def endpoint():
  def worker():
    subprocess.Popen('nohup php script.php &', shell=True, preexec_fn=os.setpgrp)

  p = multiprocessing.Process(target=worker)
  print '111111'
  p.start()
  print '222222'
  return json.dumps({
    'success': True
  })

# threading attempt
@app.route('/endpoint')
def endpoint():
  def thread_func():
    subprocess.Popen('nohup php script.php &', shell=True, preexec_fn=os.setpgrp)

  t = threading.Thread(target=thread_func)
  print '111111'
  t.start()
  print '222222'
  return json.dumps({
    'success': True
  })

In both scenarios I see the 111111 and 222222, yet my browser still hangs on the response from the endpoint. I've tried p.daemon = True for the process, as well as p.terminate() but no luck. I had hoped launching a script with nohup in a different shell and separate processs/thread would just work, but somehow Flask or uWSGI is impacted by it.

Update

Since this does work locally on my Mac when I start my Flask app directly with python app.py and hit it directly without going through my Nginx proxy and uWSGI, I'm starting to believe it may not be the code itself that is having issues. And because my Nginx just forwards the request to uWSGI, I believe it may possibly be something there that's causing it.

Here is my ini configuration for the domain for uWSGI, which I'm running in emperor mode:

[uwsgi]
protocol = uwsgi
max-requests = 5000
chmod-socket = 660
master = True
vacuum = True
enable-threads = True
auto-procname = True
procname-prefix = michael-
chdir = /srv/www/mysite.com
module = app
callable = app
socket = /tmp/mysite.com.sock
Improve this question
12
  • Is it even possible in flask to run php? Commented Sep 19, 2018 at 6:10
  • yes, you just run a shell command :) Commented Sep 20, 2018 at 4:05
  • I have to say I'm a bit puzzled. I replicated your setup locally and everything looks about just fine. The only weird thing that happens is for the threading attempt to only starts the process whence called twice, otherwise the HTTP server always answers in a minimal amount of time... Have you tried curl? Commented Sep 22, 2018 at 8:36
  • 1
    Yes, I did try local curl as well. Are you sure the php script you call is an infinite while (true) {}? Commented Sep 22, 2018 at 21:27
  • Have you tried running flask with uWSGI and at least 2 threads? Commented Sep 22, 2018 at 22:38

3 Answers 3

Reset to default
6
+25

This kind of stuff is the actual and probably main use case for Python Celery (https://docs.celeryproject.org/). As a general rule, do not run long-running jobs that are CPU-bound in the wsgi process. It's tricky, it's inefficient, and most important thing, it's more complicated than setting up an async task in a celery worker. If you want to just prototype you can set the broker to memory and not using an external server, or run a single-threaded redis on the very same machine.

This way you can launch the task, call task.result() which is blocking, but it blocks in an IO-bound fashion, or even better you can just return immediately by retrieving the task_id and build a second endpoint /result?task_id=<task_id> that checks if result is available:

result = AsyncResult(task_id, app=app)
if result.state == "SUCCESS":
   return result.get()
else:
   return result.state  # or do something else depending on the state

This way you have a non-blocking wsgi app that does what is best suited for: short time CPU-unbound calls that have IO calls at most with OS-level scheduling, then you can rely directly to the wsgi server workers|processes|threads or whatever you need to scale the API in whatever wsgi-server like uwsgi, gunicorn, etc. for the 99% of workloads as celery scales horizontally by increasing the number of worker processes.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I'm quite familiar with jobs but I'm trying to do this without introducing more dependencies. But thanks for the suggestion!
1

This approach works for me, it calls the timeout command (sleep 10s) in the command line and lets it work in the background. It returns the response immediately. 

@app.route('/endpoint1')
def endpoint1():
    subprocess.Popen('timeout 10', shell=True)
    return 'success1'

However, not testing on WSGI server, but just locally.

Improve this answer

1 Comment

Yes, as a matter of fact I also have no issues running locally but the challenge is running on my server with uWSGI seems to be problematic. In case you missed it, I mentioned this in the update of my question as well as the chat thread under the question :)
0

Would it be enough to use a background task? Then you only need to import threading e.g.

import threading
import ....
    
def endpoint():
    """My endpoint."""
    try:
        t = BackgroundTasks()
        t.start()
    except RuntimeError as exception:
        return f"An error occurred during endpoint: {exception}", 400
    return "successful.", 200
    return "successfully started.", 200

class BackgroundTasks(threading.Thread):
    def run(self,*args,**kwargs):
        ...do long running stuff
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.