Hi I'm trying to scrape all the data points that are in this url https://m-selig.ae.illinois.edu/ads/coord/a18.dat
import pandas as pd
import requests
from bs4 import BeautifulSoup
url = "https://m-selig.ae.illinois.edu/ads/coord/a18.dat"
page = requests.get(url)
x = BeautifulSoup(page.content, 'html.parser')
df = pd.DataFrame(x)
df.to_excel("air_foil.xlsx")
I've tried this code but x is just a long list that consist of one element.
First of all you need to get this data:
r = requests.get("https://m-selig.ae.illinois.edu/ads/coord/a18.dat")
print(r.tetx)
you will see what inside (string).
Then you need create a list and put to Dataframe:
df = pd.DataFrame([el.split() for el in r.text.split("\r\n")[1:]])
If you are going to use pandas, you can just use pd.read_table(url) or pd.read_csv(url), e.g.
import pandas as pd
url = "https://m-selig.ae.illinois.edu/ads/coord/a18.dat"
df = pd.read_csv(url, header=None, skiprows=1, sep=' ', engine='python')
print(df)
print(df.dtypes)
df = pd.read_table(url, header=None, skiprows=1, sep=' ', engine='python')
print(df)
print(df.dtypes)
df.to_excel('test.xlsx', index=False, header=False)
