Capture JavaScript generated content

Question

The web page is using JavaScript to put content on the page, so then I got plain HTML there is no data on the page. I need somerthing that will not only read the HTML, but also will execute and apply JavaScript to DOM, and only then return result as HTML text.

The task is exactly same as in this question, but I'm looking a solution for .NET.

If you want javascript-generated content, then you need a browser engine to actually "run" the page and you can then examine the resulting DOM. — jfriend00
– jfriend00, Commented Aug 8, 2013 at 4:37
@jfriend00 well, that's apparent solution, but I'm looking for something like lobobrowser.org/cobra/java-html-parser.jsp (a solution for similar task in Java) — pabdulin
– pabdulin, Commented Aug 8, 2013 at 4:41

Community · Accepted Answer · 2017-05-23 12:20:23Z

1

I'll be surprised if you find anything like that for .NET. I would use PhantomJS to open the page and interact with the DOM. It's a highly scriptable headless WebKit browser and will do exactly what you want with ease. See How to print html source to console with phantomjs for an example.

var page = require('webpage').create();
page.open('http://google.com', function () {
    console.log(page.content);
    phantom.exit();
});

You'll have to install PhantomJS and then launch a separate process to run your script, but PhantomJS will probably do a much better job than anything you can find written for .NET.

edited May 23, 2017 at 12:20

CommunityBot

11 silver badge

answered Aug 8, 2013 at 5:17

Trevor Dixon

24.6k13 gold badges77 silver badges114 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

pabdulin Over a year ago

I will look to it a little deeper, but so far I can't get what I want with it. Code in the example still returns pure html. I guess my problem with mine particular site is more complex than I thought. The data is downloaded in AJAX requests, and PhantomJS doesn't catches that, of course. I think it's easier to just look into AJAX calls and use them directly. Anyway thanks, PhantomJS is neat tool, maybe it will be useful for me some other time.

Trevor Dixon Over a year ago

PhantomJS executes all the AJAX calls and has mechanisms for examining those requests (see onResourceRequested). But if all the data you're interested in is in an AJAX request, it would be better to do like you said and just make that HTTP request yourself from your program.

pabdulin Over a year ago

I guess I'll accept whis as an answer, since it's relative to the question and nobody proposed a better solution anyway.

Collectives™ on Stack Overflow

Capture JavaScript generated content

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related