Asynchronous Web Scraping, Python

Question

Im trying to run this function every 2seconds forever :

import requests
from bs4 import BeautifulSoup
import asyncio

async def scrape():
    test = []
    r = requests.get(coin_desk)
    soup = BeautifulSoup(r.text, features='xml') 
    title = soup.find_all('title')[2]
    await asyncio.sleep(2)
    for x in title:
        test.append(x)
        print(test)

if name == 'main':

try:
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(scrape())
except (KeyboardInterrupt, SystemExit):
    pass

If i use

run_forever()

instead of

run_until_complete(scrape())

nothing gets printed out, it just runs forever and skips the function it seems.

Do you get any error messages with an exception / stack trace? Have you tried adding other prints to see which lines you actually reach before it exits? — dskrypa
– dskrypa, Commented Nov 18, 2022 at 22:03
No error messages, when i try to use run_forever() nothing gets printed out. It just runs forever and skips the function completly. — Snuskbetty
– Snuskbetty, Commented Nov 18, 2022 at 22:27
the async routine (scrape) was never placed into the runtime/evloop. run_until_complete includes the parameter to queue a routine for you — Christian
– Christian, Commented Nov 18, 2022 at 22:48

Christian · Accepted Answer · 2022-11-18 22:53:37Z

0

The async routine (scrape) was never placed into the event loop.

You can add it with

loop.create_task(routine)

See: What does asyncio.create_task() do?

answered Nov 18, 2022 at 22:53

Christian

9821 gold badge12 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Asynchronous Web Scraping, Python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related