I want to get the html source of a webpage genereted by javascript using Curl(PHP)
I tried the curl but I get just a javascript code :(
Can I use ruby to resolve my problem ?!
Add a comment
|
4 Answers
The javascript is executed by the browser to generate the HTML. If you make a request with CURL it will just show you the actual HTML content.You would need a Javascript engine to process the Javascript after receiving the response body.
3 Comments
cowls
Take a look at this question, it might help you. But Id be interest to know why you want to do this, its most likely not the correct approach to a problem :) stackoverflow.com/questions/10514604/…
AHmedRef
10x but how i can process the javascript after receiving the response body ?
cowls
Assuming the sites are well built it should be ok to do this without processing the javascript. E.g. the google bot indexes the web but doesnt process javascript. Also you should make sure you are allowed to use that content..
just look at any web inspector tools (in chrome just ctrl+shift+i). here you can see the changes that the javascript has on the page reflected. I dont think curl or any curl-like-tool can do this.
1 Comment
AHmedRef
can i do 'it using other langages or tools for example RUBY ?
This is a tough problem because the JavaScript has to run to get the right code. What I would say is download all the code locally and then add in an ajax call to the code, so it can ajax the source back to you after all the js has run. Then run the code in a browser.
If you need to do this a bunch of times you could queue these pages that needed to be loaded into a db and load all the pages using php. Then once the js has ajax'd the code back to the server it can refresh and pull the next page off the queue.
Let me know if you need me to clarify anything.
2 Comments
AHmedRef
can i do it using ruby or something else ?!!
Ian Overton
It's not really a question on which server side language to use. The reason it doesn't work is because JavaScript is a client side language, so it has to run on the client side in one way or another before it's going to give you the right answer. You could look into using server side javascript.
This can be done by headless browser activity like phantom js a great way to create your own logic whatever you want then get result array in console for php you can try activity here https://github.com/jonnnnyw/php-phantomjs & also https://github.com/ariya/phantomjs
