I am working on a web scraper in VBA.
I have a website with a JavaScript form and I don't know how to access a table from the search results of that JavaScript form.
I know how to navigate and pull out the required info from a normal HTML site. I already put in my searching keywords and click the "search" button all through VBA.
After my search (e.g. "SN857X00PE") the search results are displayed in a table:
SN857X00PE/01 StudioLine 9702 - 9709
SN857X00PE/38 StudioLine 9711 - 9801
SN857X00PE/42 StudioLine 9802 - 9804
SN857X00PE/46 StudioLine 9805 - 9806
I'm looking to access all the left serial numbers (e.g. SN857X00PE/01, SN857X00PE/38 etc.).
When I go into the Firefox debugger I see many .js files and an index.xhtml.
I can find everything I need in the index.xhtml file (code included below with example search SN857X00PE) but when I access the HTML through IE.Document.getElementById("body").InnerHtml it doesn't show me the content of the index.xhtml file but instead the content of the TP4.js file (code below).
As you can see below, the TP4.js file doesn't include any useful info about the search results or any way to access them.
Is there a way to access the search result table of the JavaScript form? If I am able to put keywords in it and make the search result inside VBA it should be possible the access the results too. I'd like IE.Documents to point to the content of the index.xhtml file instead of the default TP4.js file if that's possible.
When I go to the Firefox inspector to see the HTML the javascript website is producing in the end it looks so easy to access my information. Is there a way to get direct access to the nice and clean "endresult-html" after the browser finishes compiling all the JavaScript?
The website is tradeplace(DOT)com but the Javascript form is hidden behind a login.
Here an overview on how the site looks like when it displays the search results. On the right I am showing the table with the search results inside the index.xhtml that I am trying to access but I don't know how to reach it, since I only have access to the contents of the tp4.js file.
Due to max character limit, I couldn't include the whole HTML code, here are some parts I think could be important/relevant, so you can get a general idea how the site looks:
index.xhtml:
<?xml version='1.0' encoding='UTF-8' ?>
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head id="head">
<title>Tradeplace Marketplace</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link href="/tp4-web/css/jquery.theme.css" rel="stylesheet" type="text/css" />
<script type="text/javascript" src="/tp4-web/javax.faces.resource/jquery/jquery.js.xhtml?ln=primefaces&v=6.0"></script>
<script type="text/javascript" src="/tp4-web/javax.faces.resource/core.js.xhtml?ln=primefaces&v=6.0"></script>
<script type="text/javascript" src="/tp4-web/javax.faces.resource/jsf.js.xhtml?ln=javax.faces"></script>
<link rel="stylesheet" type="text/css" href="/tp4-web/javax.faces.resource/components.css.xhtml?ln=primefaces&v=6.0" />
<script type="text/javascript" src="/tp4-web/javax.faces.resource/components.js.xhtml?ln=primefaces&v=6.0"></script>
<script type="text/javascript" src="/tp4-web/javax.faces.resource/jquery/jquery-plugins.js.xhtml?ln=primefaces&v=6.0"></script>
<script type="text/javascript" src="/tp4-web/javax.faces.resource/primefaces-extensions.js.xhtml?ln=primefaces-extensions&v=6.0.0"></script>
<script type="text/javascript">
if (window.PrimeFaces) {
PrimeFaces.settings.locale = 'en_US';
}
</script>
<link href="/tp4-web/css/tp4.css" rel="stylesheet" type="text/css" />
<script type="text/javascript" src="/tp4-web/js/tp4.js"></script>
</head>
<body id="body">
<script type="text/javascript">
$(document).ready(function() {
TP4.initPageLayout();
TP4.enableSubmenus();
});
$(window).unload(function() {
TP4.hideLoadingScreen();
});
</script>
<div id="page">
<div id="header">
<div id="logoline" class="clearfix">
<img id="mainLogo" src="/tp4-web/images/sparetable/logo-tradeplace.png" alt="tradeplace" />
<img id="sparesLogo" src="/tp4-web/images/sparetable/spares.png" alt="spares" />
<table id="oemLogo">
<tr>
<td valign="middle"><img id="logoLine:brandImage" src="/tp4-web/pages/streamImage.xhtml?type=1&identifier=100001&pfdrid_c=true" alt="" />
</td>
</tr>
</table>
</div>
------------------------------------------------------------------------- ... SKIPPING SOME CODE UP TO THE RELEVANT CODE PART -------------------------------------------------------------------------
<tr class="data alt">
<td>
<a href="/tp4-web/pages/secure/product/productSearchDetail.xhtml?productNumber=SN857X00PE%2F01&formattedProductNumber=SN857X00PE%2F01&windowId=909" onclick="TP4.showLoadingScreen(this)">
SN857X00PE/01
</a>
</td>
<td>
StudioLine
</td>
<td>9702 - 9709
</td>
<td valign="middle" class="nowrap">
<span>
<a href="/tp4-web/pages/secure/product/productSearchDetail.xhtml?productNumber=SN857X00PE%2F01&formattedProductNumber=SN857X00PE%2F01&windowId=909" onclick="TP4.showLoadingScreen(this)">
<img alt="Dokumente" title="Dokumente" src="/tp4-web/images/icons/docs.gif" /></a>
</span>
<span>
<a href="/tp4-web/pages/secure/product/part/productPartList.xhtml?productNumber=SN857X00PE%2F01&formattedProductNumber=SN857X00PE%2F01&windowId=909" onclick="TP4.showLoadingScreen(this)">
<img alt="Ersatzteilliste" title="Ersatzteilliste" src="/tp4-web/images/icons/spare.gif" /></a>
</span>
<span>
<a href="/tp4-web/pages/secure/product/explodedView.xhtml?productNumber=SN857X00PE%2F01&formattedProductNumber=SN857X00PE%2F01&typeId1=A02&typeId2=A02E32&typeId3=A02E32V&windowId=909" onclick="TP4.showLoadingScreen(this)">
<img alt="Explosionszeichnung" title="Explosionszeichnung" src="/tp4-web/images/icons/explos.gif" /></a>
</span>
<span>
<a href="/tp4-web/pages/secure/accessory/accessoryForProduct.xhtml?commercialItemNumber=SN857X00PE%2F01&commercialItemNumberFormatted=SN857X00PE%2F01&windowId=909" onclick="TP4.showLoadingScreen(this)">
<img alt="Zubehör" title="Zubehör" src="/tp4-web/images/icons/commercial.gif" /></a>
</span>
</td>
</tr>
-------------------------------------------------------------- SKIPPING CODE AGAIN TO THE END -------------------------------------------------------------
</div>
<div id="ajaxStatus"></div>
<script id="ajaxStatus_s" type="text/javascript">
$(function() {
PrimeFaces.cw("AjaxStatus", "ajaxStatusWidget", {
id: "ajaxStatus",
start: function() {
TP4.showLoadingScreen();
},
success: function() {
TP4.hideLoadingScreen();
}
});
});
</script>
<div id="loadingDialog" class="ui-dialog ui-widget ui-widget-content ui-corner-all ui-shadow ui-hidden-container">
<div class="ui-dialog-content ui-widget-content">
<table id="loadingScreen">
<tr>
<td>
<p><b>Bitte warten - Datenbeschaffung...</b></p>
</td>
<td><img src="/tp4-web/images/ajax-loading.gif" /></td>
</tr>
</table>
</div>
</div>
<script id="loadingDialog_s" type="text/javascript">
$(function() {
PrimeFaces.cw("Dialog", "loadingDialogWidget", {
id: "loadingDialog",
resizable: false,
modal: true
});
});
</script>
</body>
<script type="text/javascript">
(function(i, s, o, g, r, a, m) {
i['GoogleAnalyticsObject'] = r;
i[r] = i[r] || function() {
(i[r].q = i[r].q || []).push(arguments)
}, i[r].l = 1 * new Date();
a = s.createElement(o),
m = s.getElementsByTagName(o)[0];
a.async = 1;
a.src = g;
m.parentNode.insertBefore(a, m)
})(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga');
ga('create', 'UA-55961901-1', 'auto');
ga('set', '&uid', '27d62c9d4ec32f32a829bed7142036c05d9516ac93c8935d18acf1fdc3d59145');
ga('send', 'pageview', {
'title': 'ProductSearchResult'
});
</script>
</html>TP4.js
PrimeFaces.widget.ExtImageAreaSelect && (PrimeFaces.widget.ExtImageAreaSelect.prototype.cancelSelection = function () {
this.instance && this.instance.cancelSelection()
});
TP4 = {
Constants: {
PAGE_ELEMENT_SELECTOR: '#page',
HEADER_ELEMENT_SELECTOR: '#header',
FOOTER_ELEMENT_SELECTOR: '#footer',
BASKET_RELOAD_LINK_ID: '#reloadBasketLink',
BASKET_POPUP_WIDTH: '460',
BASKET_POPUP_HEIGHT: '400',
SPECIAL_NOTICE_PART_POPUP_WIDTH: '380',
SPECIAL_NOTICE_PART_POPUP_HEIGHT: '365',
SPECIAL_NOTICE_DOCUMENT_LIST_POPUP_WIDTH: '600',
SPECIAL_NOTICE_DOCUMENT_LIST_POPUP_HEIGHT: '400',
SPECIAL_NOTICE_DOCUMENT_POPUP_WIDTH: '500',
SPECIAL_NOTICE_DOCUMENT_POPUP_HEIGHT: '200',
CUSTOMER_INFO_POPUP_WIDTH: '480',
CUSTOMER_INFO_POPUP_HEIGHT: '405',
SMALL_POPUP_WIDTH: '360',
SMALL_POPUP_HEIGHT: '330',
MEDIUM_POPUP_WIDTH: '800',
MEDIUM_POPUP_HEIGHT: '600',
NON_MODAL_DIALOG_Z_INDEX: 500
},
Context: {
currentDialogZIndex: 0
},
showLoadingScreen: function (a) {
a ? !a.href || a.href && '#' == a.href ? TP4.showLoadingScreenAfterTimeout() : a.href && PF('loadingDialogWidget').show() : PF('loadingDialogWidget').show()
},
showLoadingScreenAfterTimeout: function () {
setTimeout(function () {
PF('loadingDialogWidget').show()
}, 100)
},
hideLoadingScreen: function () {
PF('loadingDialogWidget').hide()
},
initPageLayout: function () {
$(TP4.Constants.PAGE_ELEMENT_SELECTOR).css({
'padding-top': $(TP4.Constants.HEADER_ELEMENT_SELECTOR).height(),
'padding-bottom': $(TP4.Constants.FOOTER_ELEMENT_SELECTOR).height()
})
},
adjustElementHeightToAvailableHeight: function (a, b) {
var c = $(a),
d = $(TP4.Constants.HEADER_ELEMENT_SELECTOR),
g = $(TP4.Constants.FOOTER_ELEMENT_SELECTOR),
e = $(window).height();
0 < d.length && (e -= d.height());
0 < g.length && (e -= g.height());
null != b && (e -= b);
c.height(e)
},
setElementCssHeightToContentHeight: function (a) {
a = $(a);
var b = a.height();
a.css('height', b + 'px')
},
setDefaultCommand: function (a, b) {
a.keydown(function (a) {
var d = $.ui.keyCode;
if (a.which ==
d.ENTER || a.which == d.NUMPAD_ENTER) b.click(),
a.preventDefault()
})
},
scrollTo: function (a) {
window.location.hash = '#' + a
},
scrollToTop: function () {
$('html, body').animate({
scrollTop: 0
}, 0)
},
scrollToBottom: function (a) {
$('html, body').animate({
scrollTop: $(document).height()
}, a)
},
adjustMarginForTwoColumnLayout: function (a, b) {
$(b).css({
'margin-left': $(a).width()
})
},
restoreInputFieldValueFromHtml: function (a) {
a.each(function (b) {
$(this).val(a[b].defaultValue)
})
},
disableLoadingScreenModality: function () {
PF('loadingDialogWidget').cfg.modal = !1
},
enableLoadingScreenModality: function () {
PF('loadingDialogWidget').cfg.modal = !0
},
moveDialogBehindModalContainer: function (a) {
TP4.Context.currentDialogZIndex = a.jq.css('z-index');
a.jq.css('z-index', TP4.Constants.NON_MODAL_DIALOG_Z_INDEX)
},
moveDialogBeforeModalContainer: function (a) {
a.jq.css('z-index', TP4.Context.currentDialogZIndex)
},
synchronizeDialogModality: function (a) {
TP4.disableLoadingScreenModality();
TP4.moveDialogBehindModalContainer(a)
},
resetDialogModalityToDefault: function (a) {
TP4.enableLoadingScreenModality();
TP4.moveDialogBeforeModalContainer(a)
},
activateMaxLengthHandlingOnTextareas: function () {
$('textarea[maxlength]').keyup(function () {
var a = parseInt($(this).attr('maxlength'));
$(this).val().length > a && $(this).val($(this).val().substr(0, $(this).attr('maxlength')))
})
},
resizeTo: function (a, b, c) {
var d = - 1 < navigator.userAgent.toLowerCase().indexOf('chrome');
null != a && (d ? setTimeout(function () {
a.resizeTo(b, c)
}, 4) : a.resizeTo(b, c))
},
enableSubmenus: function () {
$('li').hover(function () {
var a = $(this);
a.closest('ul').children('li.hover').removeClass('hover');
a.addClass('hover');
a.attr('isHovered', !0)
}, function () {
var a = $(this);
a.removeAttr('isHovered');
setTimeout(function () {
a.attr('isHovered') || a.removeClass('hover')
}, 350)
})
},
redirect: function (a) {
window.location.href = a
},
overwriteIfZero: function (a) {
0 < a.value.length && '0' == a.value.substring(0, 1) && (a.value = a.value.substring(1), TP4.overwriteIfZero(a))
},
Part: {
showChildList: function (a) {
$('#childList' + a).show();
$('#showChildList' + a).hide();
$('#hideChildList' + a).show()
},
hideChildList: function (a) {
$('#childList' +
a).hide();
$('#showChildList' + a).show();
$('#hideChildList' + a).hide()
},
setAllDefaultCommands: function () {
TP4.setDefaultCommand($('#inputTextDescription'), $('#commandLinkDescription'));
TP4.setDefaultCommand($('#inputTextPartNumber'), $('#commandLinkPartNumber'));
TP4.setDefaultCommand($('#inputTextPicPos'), $('#commandLinkPicPos'))
}
},
Basket: {
openAddToBasketPopUp: function (a) {
TP4.PopUp.openPopUpWithSize(a.href, a.target, TP4.Constants.BASKET_POPUP_WIDTH, TP4.Constants.BASKET_POPUP_HEIGHT, 0, 0);
return !1
},
initializeDatePicker: function (a) {
a = $('#requestDeliveryDate').datepicker({
dateFormat: 'yy-mm-dd',
changeMonth: !0,
changeYear: !0,
yearRange: '-90:+0',
showOn: 'button',
buttonImageOnly: !0,
buttonImage: a + '/images/calendar.gif',
minDate: 0
});
a.attr('readonly', !0);
a.datepicker('option', 'showAnim', 'show')
},
setAllDefaultCommands: function () {
TP4.setDefaultCommand($('#basketOverviewItemListWrapper input'), $('#updateBasket'))
}
},
SpecialNotices: {
openSpecialNoticePartPopUp: function (a) {
TP4.PopUp.openPopUpWithSize(a.href, a.target, TP4.Constants.SPECIAL_NOTICE_PART_POPUP_WIDTH, TP4.Constants.SPECIAL_NOTICE_PART_POPUP_HEIGHT, 0, 0);
return !1
},
openSpecialNoticeDocumentListPopUp: function (a) {
TP4.PopUp.openPopUpWithSize(a.href, a.target, TP4.Constants.SPECIAL_NOTICE_DOCUMENT_LIST_POPUP_WIDTH, TP4.Constants.SPECIAL_NOTICE_DOCUMENT_LIST_POPUP_HEIGHT, 0, 0);
return !1
},
openSpecialNoticeDocumentPopUp: function (a) {
TP4.PopUp.openPopUpWithSize(a.href, a.target, TP4.Constants.SPECIAL_NOTICE_DOCUMENT_POPUP_WIDTH, TP4.Constants.SPECIAL_NOTICE_DOCUMENT_POPUP_HEIGHT, 0, 0);
return !1
},
initializeDocumentView: function (a) {
if (a) {
a = $('#specialNoticeHeader');
var b = 0;
0 < a.length && (b = a.height());
$('#specialNoticesPdf').css('visibility', 'visible');
$.browser.webkit ? setTimeout(function () {
TP4.adjustElementHeightToAvailableHeight('#specialNoticesDocumentContainer', b)
}, 100) : TP4.adjustElementHeightToAvailableHeight('#specialNoticesDocumentContainer', b)
}
},
afterAddToBasket: function (a, b) {
b && TP4.SpecialNotices.initializeDocumentView(a);
TP4.SpecialNotices.triggerBasketRefreshInMainWindow();
scroll(0, 0)
},
triggerBasketRefreshInMainWindow: function () {
TP4.SpecialNotices.triggerBasketRefresh(window.opener)
},
triggerBasketRefresh: function (a) {
var b = a.$(TP4.Constants.BASKET_RELOAD_LINK_ID);
0 < b.length ? (a.TP4.showLoadingScreen(), a.location.href = b.attr('href')) : a.opener && TP4.SpecialNotices.triggerBasketRefresh(a.opener)
}
},
PopUp: {
openSmallPopUpForLink: function (a) {
TP4.PopUp.openPopUpWithSize(a.href, a.target, TP4.Constants.SMALL_POPUP_WIDTH, TP4.Constants.SMALL_POPUP_HEIGHT, 0, 0);
return !1
},
-----------------------------------------------------------------
SKIPPING CODE TO THE END
-------------------------------------------------------------------
Infoline: {
openCustomerInfoPopUpForLink: function (a) {
TP4.PopUp.openPopUpWithSize(a.href, a.target, TP4.Constants.CUSTOMER_INFO_POPUP_WIDTH, TP4.Constants.CUSTOMER_INFO_POPUP_HEIGHT, 0, 0);
return !1
},
triggerInfolineRefreshInMainWindow: function () {
var a = window.opener.$(TP4.Constants.BASKET_RELOAD_LINK_ID);
0 < a.length ? (window.opener.location.href = a.attr('href'), window.opener.TP4.showLoadingScreen()) : (window.opener.TP4.showLoadingScreen(), window.opener.location.search = TP4.Infoline.getOpenerUrlWithEmptyWindowId())
},
getOpenerUrlWithEmptyWindowId: function () {
for (var a = {
}, b = window.opener.location.search.substring(1), c = /([^&=]+)=([^&]*)/g, d; d = c.exec(b); ) a[decodeURIComponent(d[1])] = decodeURIComponent(d[2]);
a.windowId = '';
return $.param(a)
}
},
History: {
initializeDatePicker: function (a) {
var b = $('#dateFrom, #dateTo').datepicker({
dateFormat: 'yy-mm-dd',
changeMonth: !0,
changeYear: !0,
yearRange: '-90:+0',
showOn: 'button',
buttonImageOnly: !0,
buttonImage: a + '/images/calendar.gif',
onSelect: function (a) {
var d = 'dateFrom' == this.id ? 'minDate' :
'maxDate',
g = $(this).data('datepicker');
a = $.datepicker.parseDate(g.settings.dateFormat || $.datepicker._defaults.dateFormat, a, g.settings);
b.not(this).datepicker('option', d, a)
}
});
b.attr('readonly', !0);
b.datepicker('option', 'showAnim', 'show')
}
},
FAQ: {
Context: {
selectedIssueId: - 1,
selectedQuestionsIssueId: - 1,
selectedQuestionAnchor: ''
},
setSelectedIssueId: function (a) {
TP4.FAQ.Context.selectedIssueId = a;
TP4.FAQ.Context.selectedQuestionsIssueId = a
},
setSelectedQuestionsIssueId: function (a) {
TP4.FAQ.Context.selectedQuestionsIssueId = a;
- 1 === TP4.FAQ.Context.selectedIssueId && (TP4.FAQ.Context.selectedIssueId = a)
},
setSelectionQuestionAnchor: function (a) {
TP4.FAQ.Context.selectedQuestionAnchor = 'question' + a
},
areQuestionsAlreadyLoaded: function () {
if (TP4.FAQ.Context.selectedIssueId === TP4.FAQ.Context.selectedQuestionsIssueId) return TP4.FAQ.scrollToAnchor(),
!1;
TP4.FAQ.Context.selectedIssueId = TP4.FAQ.Context.selectedQuestionsIssueId
},
scrollToAnchor: function () {
TP4.scrollTo(TP4.FAQ.Context.selectedQuestionAnchor)
},
moveOnClickFromInnerLinkToTreeNodeContainer: function (a, b) {
$(a).find('.ui-treenode-leaf').each(function () {
var a = $(this).find(b),
d = a.attr('onclick');
$(this).attr('onclick', d);
a.attr('onclick', null)
})
}
}
};
PrimeFaces.widget.Poll = PrimeFaces.widget.BaseWidget.extend({
init: function (a) {
this.cfg = a;
this.id = this.cfg.id;
this.active = !1;
this.cfg.autoStart && this.start()
},
refresh: function (a) {
this.isActive() && this.stop();
this.init(a)
},
start: function () {
this.timer = setInterval(this.cfg.fn, 1000 * this.cfg.frequency);
this.active = !0
},
stop: function () {
clearInterval(this.timer);
this.active = !1
},
handleComplete: function (a, b, c) {
c.stop && this.stop()
},
isActive: function () {
return this.active
}
});
