Grab data using python

Question

Here website will list some data based on dropdown-filter, so i am trying to fetch those data by passing static dropdown values, but i think due to view state i am unable to grab those data.

Anyone have any idea how to grab asp.net website data which is using viewstate?

i am getting below error

Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.

Python Script

import requests
from bs4 import BeautifulSoup

def get_viewstate():
url = "http://xlnindia.gov.in/frm_G_Cold_S_Query.aspx?ST=GJ"
req = requests.get(url)
data = req.text

bs = BeautifulSoup(data)
return bs.find("input", {"id": "__VIEWSTATE"}).attrs['value']

url = "http://xlnindia.gov.in/frm_G_Cold_S_Query.aspx?ST=GJ"
data = {"__VIEWSTATE": get_viewstate(),"ST":'GJ', "ddldistrict":'AMR', "ddltaluka":'' ,"btnSearch":'Search'}
req = requests.post(url, data)

bs = BeautifulSoup(req.text)
print(bs.prettify())

Give Selenium a try?

OneCricketeer
– OneCricketeer

2017-05-26 11:53:47 +00:00
Commented May 26, 2017 at 11:53 — OneCricketeer
– OneCricketeer, Commented May 26, 2017 at 11:53

MD. Khairul Basar · Accepted Answer · 2017-05-26 14:09:45Z

1

I don't think you can do this with requests but you can easily do this with selenium.

To install selenium - pip install selenium or pip3 install selenium.
Then download the latest Chromedriver from this link for your machine and copy the driver to your working directory.

You can read the selenium documentation here.

import time
from selenium import webdriver

url = "http://xlnindia.gov.in/frm_G_Cold_S_Query.aspx?ST=GJ"
browser = webdriver.Chrome()
browser.get(url)

#change this if you want to change the state from Gujrat to something else
#or you can change the state simply by changing the "?ST=GJ" part of the URL
#state = browser.find_element_by_id("ddlState")
#state.send_keys("BR")

district = browser.find_element_by_id("ddldistrict")
district.send_keys("AMR")

#Skip this if you want to include all categories into the result
category = browser.find_element_by_id("ddlCategory")
category.send_keys("R")

button = browser.find_element_by_id("btnSearch")
button.click()

time.sleep(10)
browser.save_screenshot(browser.title + ".JPEG")
html = browser.page_source
print(html)

browser.close()
browser.quit()

NOTE
If you want to use headless browser with selenium, use PhantomJS. To learn how to do this with PhantomJS read this.

edited May 26, 2017 at 14:09

answered May 26, 2017 at 13:28

MD. Khairul Basar

5,11015 gold badges43 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Juned Ansari Over a year ago

hey MD. Khairul Basar can you help me to store each field value into mysql database?

MD. Khairul Basar Over a year ago

@JunedAnsari Can give it a try.

Juned Ansari Over a year ago

Basar, finally i did to store data into mysqldatabase, now can you help me to fetch dropdown values dynamically, instead of specifying static value, it is based on ajax next dropdown value fills. i am trying to store whole states dropdown values in array like wise

Juned Ansari Over a year ago

please check stackoverflow.com/questions/44277239/…

Collectives™ on Stack Overflow

Grab data using python

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related