Accessing a Javascript variable inside an HTML file

Question

I'm doing a little bit of reverse engineering on the Rapportive API in Gmail.

I make this request

import requests
url ='https://api.linkedin.com/uas/js/xdrpc.html'
r = requests.get(url)
print r.text

The response is an empty HTML file that has a lot of Javascript in it. On line 3661, it sets the RequestHeader for the subsequent call to Rapportive:

ak.setRequestHeader("oauth_token", ae);

Is there a way I can request that page and then return ae?

The only thing in the file is JS, or there is JS mixed inside an HTML body? — Matthew Herbst
– Matthew Herbst, Commented Jan 1, 2016 at 0:30
Only JS. You can actually run that code and see the response. The Rapportive API is really flexible. — dropWizard
– dropWizard, Commented Jan 1, 2016 at 0:31
You need to execute all the Javascript, because that's what sets ae. — Barmar
– Barmar, Commented Jan 1, 2016 at 0:46
I thought by running the page, it runs the javascript and sets the oauth header? How can I intercept it? — dropWizard
– dropWizard, Commented Jan 1, 2016 at 0:48
requests.get() is not a Javascript interpreter, it just performs the HTTP request to download the contents of the file. — Barmar
– Barmar, Commented Jan 1, 2016 at 1:53

Felypp Oliveira · Accepted Answer · 2016-01-02 01:54:41Z

1

I think you can try:

Get the page as you already does;
Remove all non-javascript elements from the response page;
Prepend a javascript (described below) in the page's javascript to override some code;
Execute it with eval('<code>');
Check if the token has been set correctly;

I'm proposing the following code to override the XMLHttpRequest.setRequestHeader functionality to be able to get the token:

// this will keep the token
var headerToken; 

// create a backup method
XMLHttpRequest.prototype.setRequestHeaderBkp = 
XMLHttpRequest.prototype.setRequestHeader; 

// override the "setRequestHeader" method
XMLHttpRequest.prototype.setRequestHeader = function(key, val)
{
  if ('oauth_token' === key)
    headerToken = val;

  this.setRequestHeaderBkp(key, val);
}

answered Jan 2, 2016 at 1:54

Felypp Oliveira

2,1971 gold badge17 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

dropWizard Over a year ago

I'll give this a try!

Felypp Oliveira Over a year ago

@MorganAllen any news about the problem?

James Buck · Accepted Answer · 2016-01-01 01:03:29Z

0

If you are just interested in retrieving the token can't you just do a regex match:

var str = '<script>var a = 1;...ak.setRequestHeader("oauth_token", ae);...</script>';
var token = str.match(/setRequestHeader\("oauth_token",\s*([^)]+)/)[1];

Although this assumes ae is the actual string value. If it's a variable this approach wouldn't work as easily.

Edit: If it's a variable you could do something like:

str.replace(/\w+\.setRequestHeader\([^,]+,\s*([^)]+)\s*\);/, 'oauthToken = \1';

Before running the JavaScript returned from the page, then the global oauthToken (notice the missing 'var') will contain the value of the token, assuming the the evaluation of the code is run in the same scope as the caller.

edited Jan 1, 2016 at 1:03

answered Jan 1, 2016 at 0:49

James Buck

1,6409 silver badges14 bronze badges

4 Comments

dropWizard Over a year ago

Well, I'm hoping to be able to request the page, parse it somehow and access token.

Valijon Over a year ago

You are extracting simple string ae from response text. What it is necessary, it's ae value in javascript context. Imagine: var ae = "foo"; ak.setRequestHeader("oauth_token", ae);. You code will extract ae string, not foo value.

James Buck Over a year ago

@Valijon in my edit we are replacing the statement ak.setRequestHeader("oauth_token", ae); with oauthToken = ae;. Therefore after the code has been run, the global variable we just declared should in theory contain ae's value.

Valijon Over a year ago

I'm lost. We are speaking different things. Try to adapt your code to python syntax and check if you are right. Because, I don't see how are you going to get ae value from string...

Collectives™ on Stack Overflow

Accessing a Javascript variable inside an HTML file

2 Answers 2

2 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related