Python: Issue getting updating html created via JavaScript calls in browser

Question

I am using Python to pull the HTML of a website to get satellite locations. Of course since I am not actually accessing the site via a browser I am not retrieving any html that would be populated by javascript calls.

import urllib.request
page = urllib.request.urlopen('http://n2yo.com/?s=20217')
file = open("textFile", "wb")
satelliteText = page.read()
file.write(satelliteText)
file.close()

I've explored libraries like Windmill that literally run a browser so that you can get that javascript created html, but I am using a Raspberry Pi. I'd rather not install an additional browser.

Is there anyway that I can make the ajax get calls myself that the website is making and retrieve just the data I need?

Jordan Soltman · Accepted Answer · 2015-11-27 20:35:53Z

1

Looking at this source here: http://www.n2yo.com/js/passes.js it appears that it is calling http://www.n2yo.com/inc/all.php to get the data. By reading through passes.js carefully you should be able to figure out how to parse it.

answered Nov 27, 2015 at 20:35

Jordan Soltman

3,95322 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

dyllandry Over a year ago

In passes.js, besides the 'satid', they seem to create a 'satrec' by calling twoline2satrec, and then throw the 'satrec' into satellite.propagate which returns the position and velocity of the satellite. I can't find these functions they call in any of the source files within the dev tools. They have a file called satellite.min.js that seems to have all of their mathematic formulas in it, but I cannot correlate any with 'propogate' or 'twoline2satrec'. I guess It makes sense that they let our browsers do all the math to take the load off of the server.

dyllandry Over a year ago

I really appreciate your help Jordan. I believe I have a lead on this but it isn't much. If you toss satellite.min.js into www.dirtymarkup.com on line 1508 I think the function I am interested in begins. Since all their code seems to be 'minified' and very dense it is pretty hard to interpret and follow. Though, I think all the substrs and parsing going on in lines 1519 - 1559 hold the gold.

Jordan Soltman Over a year ago

You are getting closer. satellite is coming from satellite-js which is available here: github.com/shashwatak/satellite-js. The github outlines things pretty well and has the source (non-minified)

dyllandry Over a year ago

Thanks to you I just finished my objective. I got a hold of someone at the website and they gave me a different way to get the data. Really, the same data you helped me get. From there I looked at that satellite lib you referenced, and then the python library it is built from. Turns out all I had to do was download the library and use it to interpret those weird lines of data I was getting from .../inc/all.php . Because of the way the other guy was having me get the data I also needed a python soap library called suds. Thanks so much for your help. I could not have done it with out you! :)

Collectives™ on Stack Overflow

Python: Issue getting updating html created via JavaScript calls in browser

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related