Python and multithreading + function

Question

I have a function that is going to request sites. I want to multithread this to make it faster of course, however, after trying many methods ranging from concurrent.futures to threading module, they either do not work, stop working after a while or simply are too unreliable. Are there any methods of multithreading relating to one function that is reliable and fast?

My Code:

import requests
import concurrent.futures

def example_request():
  requests.get("www.bing.com")

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
  while True:
     executor.submit(example_request)

Hope someone can help thanks!!!!!!

I am using python 2.7, i am using threading module. and it works for me. i can post an answer if this can help?! — AdamF
– AdamF, Commented May 14, 2020 at 6:32
This seems to work but is not reliable. Can you provide your threading module example? — xnarf
– xnarf, Commented May 14, 2020 at 6:34
sure. I have post it. it works for me. my solution also solves and handle cases that some exception raised by foo and re raised it in the caller thread (main thread !!). if you care. If you found it useful please vote for my solution. thanks. — AdamF
– AdamF, Commented May 14, 2020 at 6:40

xnarf · Accepted Answer · 2020-05-14 07:13:37Z

1

This is a great and easy way to multi thread :)

import threading
from threading import Thread

# Adding in definition provides more reliability :)
def example_request():
  while True:
     try:
       requests.get("www.bing.com")
     except:
       print("Error...")

# Threading phase
for _ in range(5):
  Thread(target=example_request).start()

This example is way more simpler, has thread exception catcher and is pretty fast and reliable. For more speed just increase the range length ;)

answered May 14, 2020 at 7:13

xnarf

1009 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

AdamF · Accepted Answer · 2020-05-14 06:38:22Z

class InformantThread(Thread):
  """Informant thread can report or inform the caller thread regarding any exception in the call stack of the spawned thread by
     saving the exception and re raise it in join section. this is the best way for notifying the caller regarding exceptions during
     target executing.
  """
  def __init__(self, *args, **kwargs):
    super(InformantThread, self).__init__(*args, **kwargs)
    self.exc = None

  def run(self):
    try:
      if hasattr(self, '_Thread__target'):
        # Thread uses name mangling in python 2.
        self.ret = self._Thread__target(*self._Thread__args, **self._Thread__kwargs)
      else:
        # for python 3
        self.ret = self._target(*self._args, **self._kwargs)
    except (BaseException, Exception) as e:
      self.exc = e

  def join(self, timeout=None):
    super(InformantThread, self).join(timeout)
    if self.exc:
      raise self.exc

def foo():
    while(True):
        print 'Hello World!'
        time.sleep(1)

Caling My InformantThread for executing foo():

  i_thread = InformantThread(target=foo)
  ## If we dispatch it as normal thread (not daemon), at time out it will prevent main thread
  # (calling thread) to exit !! hence the declaration as daemon.
  i_thread.daemon=True
  i_thread.start()
  i_thread.join(float(60)) #60 seconds time out
  if i_thread .is_alive():
    raise Exception("Timeout has expired")

Collectives™ on Stack Overflow

Python and multithreading + function

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related