Google Apps Script UrlFetchApp and Ajax Data

Question

I am trying to fetch data behind a login in Google Apps script. My code below works and fetches both the login and data page (after logging in) without issues.

function fetch() {
  var loginurl = "https://somedomain.com/login";
  var dataurl = "https://somedomain.com/orders#status=0&p=1&per=100";
  var params = {
    "method": "post",
    "payload": {
      "UserName": "myusername",
      "Password": "mypassword",
      "action": "Log In",
      "testcookie": 1
    },
    "followRedirects": false
  };

  var response = UrlFetchApp.fetch(loginurl, params);

  if (response.getResponseCode() == 302) {
    var headers = response.getAllHeaders();
    var cookies = typeof headers['Set-Cookie'] == 'string' ? [ headers['Set-Cookie'] ] : headers['Set-Cookie'];
    for (var i = 0; i < cookies.length; i++) {
      cookies[i] = cookies[i].split(';')[0];
    };
    params = {"method": "get","headers": {"Cookie": cookies.join(';')}};
    response = UrlFetchApp.fetch(dataurl, params);
    Logger.log(response.getContentText());
  } else {
    Logger.log(getResponseCode());
  }
}

My problem is the fetch response for the data page does not include the actual data I need since it is not fully loaded by the website (ajax) when the response is sent back. I can even see the "ajaxLoading" div is visible in the ContentText of the response.

Is there any kind of work-around for this?

Are you trying to web scrape?

ReyAnthonyRenacia
– ReyAnthonyRenacia

2018-02-15 15:20:06 +00:00
Commented Feb 15, 2018 at 15:20 — ReyAnthonyRenacia
– ReyAnthonyRenacia, Commented Feb 15, 2018 at 15:20
Yes, but managed to solve it.

inkt
– inkt

2018-02-16 22:02:04 +00:00
Commented Feb 16, 2018 at 22:02 — inkt
– inkt, Commented Feb 16, 2018 at 22:02

inkt · Accepted Answer · 2018-02-16 22:03:56Z

0

Solved: The dataurl in my example above was taking my parameters and posting to a totally different url, and i was able to call that url directly for the data I was after.

answered Feb 16, 2018 at 22:03

inkt

111 silver badge1 bronze badge

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Google Apps Script UrlFetchApp and Ajax Data

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related