0

I am trying to handle a website programmatically. Lets say I visit the page www.example.com/something. On the website there is a button which I am pressing. The code of the button looks something like this:

<form action="/something" method="POST" enctype="text/plain">
   <input type="submit"  class="button" value="Click me" >
</form>

Pressing this button updates the information on the website.

Now I would like to do this procedure programatically to receive the content of the updated website after pressing the button.

Can someone lead me to the right direction on how to do this? preferably in C#.

Thank you in advance!

Edit:

I used Fiddler to capture the HTTP request and response, it looks like this:

POST /something HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://example.com/something
Cookie: cookie1=cookiecontent; cookie2=cookiecontent
Connection: keep-alive
Content-Type: text/plain
Content-Length: 0

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 05 Dec 2013 23:36:31 GMT
Content-Length: 2202

Although the requests includes cookies they don't appear to be relevant. I decompressed the received content with fiddler and found the wanted data to be included in the response.

I am not very experienced in HTTP requests and am therefore hoping that someone can help me convertion this into a C# http request to receive the content.

Improve this question
2
  • Could you clarify please... are you trying to use a C# program (not a browser) to mine data from that page? Commented Dec 5, 2013 at 23:12
  • Yes, that is what I am trying to do. I want my C# program to receive the data which is present on the website after clicking this button. Commented Dec 5, 2013 at 23:14

4 Answers 4

Reset to default
1

If the website in question is open and doesn't do any sort of cookie generation to validate requests (there are plenty of sites like this) then you can just use System.Net.WebRequest or similar to post the required form data, then examine the response. See this MSDN page for an example.

If the page does use cookies and so on you'll have to get a bit more creative. In some cases you can issue one web request to get the first page, examine the results for cookies and hidden form values and use those in your POST.

If all else fails then the Selenium WebDriver library will give you almost complete browser emulation with full access to the DOM. It's a bit more complex than using a WebRequest, but will work for pretty much everything you can use a web browser for.

Regardless of which method you use, Fiddler is a good debugging tool. Use it to compare what your C# code is doing to what the web browser is doing to see if there's anything your code isn't getting right.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks, System.Net.WebRequest from your MSDN link did the trick.
1

Since it's a submit button then simulating the resulting HTTP Request would be easier than simulating a click. First, I would use a program like Fiddler to inspect what is being sent when you submit the form. Then I would replicate that request, just changing the values that I need changing, using HTTPWebRequest. You can find an example here.

The resultant HTTPWebResponse can then be parsed for data. Using something like HtmlAgilityPack makes that part easier.

Improve this answer

3 Comments

That was my first idea too but since I am not very familiar with http requests I was hoping that there is another and, from my perspective, easier way. I added the information I got using fiddler. maybe you could give me some information or examples on how to convert this request into a C# http request.
@Lotzki Just read the linked Code Project article. It should tell you exactly what you need to do to replicate the request.
I ended up using the link from Corey. Your link appears to have all the needed information. However, in my case, his version was more straightforward.
0

You can do what you want with http://www.seleniumhq.org/projects/webdriver/. It is possible to do web automation with c# in a console program. I am using it for ui integration testing and it works fairly well

Improve this answer

Comments

0

I would look into searching for a browser automation framework. I would usually do this in Python and have not used .Net for this, but a quick Google search yields quite a few results.

Included within these:

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.