Scraping text generated by script with PHP Simple HTML DOM Parser

Question

I am trying to get the following text "Huggies Pure Baby Wipes 4 x 64 per pack" shown in the code below.

<div class="offerList-item-description-title">
    <div id="result-title-5" class="offerList-item-description-title">
        <script type="text/javascript">
            document.write(getContents('wF8UD9Jj8:6D !FC6 q23J (:A6D c I ec A6C A24\<'));
        </script>Huggies Pure Baby Wipes 4 x 64 per pack
    </div>
</div>

I have tried using code such as:

foreach($element -> find('.offerList-item-description-title') as $title)
{
    foreach($element -> find('text') as $text){
        echo $text;
    }
}

But just get returned an empty string, any suggestions?

Thanks.

I'm not familiar with this package, but I'd say that $element -> find('text') is your problem. There is no text tag. I would think that instead of the second foreach you'd want something like $title->innertext — Tim Morton
– Tim Morton, Commented Mar 14, 2018 at 13:26

Faraz Irfan · Accepted Answer · 2018-03-15 07:55:02Z

1

If you are aware your HTML returned by your scraper does not contain Javascript rendered code, like in your case text is generated by javascript that's why you are getting empty response. What you need is a headless browser like PhantomJS you can use PHP wrapper of PhantomJS http://jonnnnyw.github.io/php-phantomjs/.

This will solve your problem. It has following features:

Load webpages through the PhantomJS headless browser
View detailed response data including page content, headers, status code etc.
Handle redirects
View javascript console errors

Hope this helps.

answered Mar 15, 2018 at 7:55

Faraz Irfan

1,3263 gold badges10 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

RickL · Accepted Answer · 2018-03-14 13:45:34Z

0

I'm not sure what code your using in your example (and I suspect the getContents function result gets in the way of your method for retrieving the text) but if you wrap the text you're after in a <span> like so:

<div class="offerList-item-description">
    <div id="result-title-5" class="offerList-item-description-title">
        <script type="text/javascript">
            document.write(getContents('wF8UD9Jj8:6D !FC6 q23J (:A6D c I ec A6C A24\<'));
        </script><span>Huggies Pure Baby Wipes 4 x 64 per pack</span>
    </div>
</div>

you can retrieve it using javascript:

<script>
    var $title = document.getElementsByClassName("offerList-item-description-title");
    for (var i = 0; i < $title.length; i++) {
        var span = $title[i].getElementsByTagName("span");
        var $text = span[0].innerText || span[0].textContent;
        //echo $text;
        console.log("==> " + $text);
    }
</script>

answered Mar 14, 2018 at 13:45

RickL

3,42410 gold badges42 silver badges41 bronze badges

1 Comment

T Simple Over a year ago

Unfortunately we cant add <span> to the script because its scraped directly off the website. Unless of course it is possible to add a <span> to the scraped <div>. We're using Siple Html Dom Parser for scraping.

Collectives™ on Stack Overflow

Scraping text generated by script with PHP Simple HTML DOM Parser

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related