python selenium chromedriver on amazon linux / aws Lambda

Question

I've converted my locally running scraper to be usable on aws lambda. I followed using the guides, tools, and examples:

https://medium.com/@marco.luethy/running-headless-chrome-on-aws-lambda-fa82ad33a9eb

https://github.com/adieuadieu/serverless-chrome/releases

https://github.com/21Buttons/pychromeless/blob/master/src/webdriver_wrapper.py

https://robertorocha.info/setting-up-a-selenium-web-scraper-on-aws-lambda-with-python/

import os
from selenium import webdriver

 # Chrome Driver Client
    class ChromeClient:

        # set up setting of web driver
        def __init__(self, logger):
            # setup logger
            self._tmp_folder = '/tmp/'
            try:
                if not os.path.exists(self._tmp_folder):
                    os.makedirs(self._tmp_folder)

                if not os.path.exists(self._tmp_folder + '/user-data'):
                    os.makedirs(self._tmp_folder + '/user-data')

                if not os.path.exists(self._tmp_folder + '/data-path'):
                    os.makedirs(self._tmp_folder + '/data-path')

                if not os.path.exists(self._tmp_folder + '/cache-dir'):
                    os.makedirs(self._tmp_folder + '/cache-dir')
            except Exception as ex:
                logger.error('problem with {0}'.format(ex))
            try:

                chrome_options = webdriver.ChromeOptions()
                chrome_options.add_argument('--headless')
                chrome_options.add_argument('--no-sandbox')
                chrome_options.add_argument('--disable-gpu')
                chrome_options.add_argument('--window-size=1280x1696')
                chrome_options.add_argument('--user-data-dir=/tmp/user-data')
                chrome_options.add_argument('--hide-scrollbars')
                chrome_options.add_argument('--enable-logging')
                chrome_options.add_argument('--log-level=0')
                chrome_options.add_argument('--v=99')
                chrome_options.add_argument('--single-process')
                chrome_options.add_argument('--data-path=/tmp/data-path')
                chrome_options.add_argument('--ignore-certificate-errors')
                chrome_options.add_argument('--homedir=/tmp')
                chrome_options.add_argument('--disk-cache-dir=/tmp/cache-dir')
                chrome_options.add_argument('user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36')
                chrome_options.binary_location = os.getcwd() + "/bin/headless-chromium"
            except Exception as ex:
                logger.error('problem with {0}'.format(ex))

            try:
                logger.info('__init__(), Init Chrome Driver')
                print('make me some chrome')
                try:
                    self.driver = webdriver.Chrome(chrome_options=chrome_options)
                except Exception as ex:
                    logger.error('chrome problem with {0}'.format(ex))
                self.driver.maximize_window()
                self.driver.delete_all_cookies()
                self.driver.set_page_load_timeout(120)
                self.driver.implicitly_wait(5)
                self.driver.delete_all_cookies()
            except Exception as ex:
                logger.error('problem with {0}'.format(ex))

Every time it gets to webdriver.Chrome(chrome_options=chrome_options) I see the error:

Service chromedriver unexpectedly exited. Status code was: 127

Any ideas?

any help with this? stackoverflow.com/questions/49323099/… — Paolo
– Paolo, Commented May 24, 2020 at 23:40

Ethan Drower · Accepted Answer · 2021-07-28 14:46:01Z

1

Looks to be about what I'm struggling with too. One of my first issues was that I didn't have the proper chromium browser binary uploaded to Lambda with the package (it will live in /opt/* when you do upload it) so make sure that the binary is referenced properly if it's uploaded.

answered Jul 28, 2021 at 14:46

Ethan Drower

741 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

GeneralBear Over a year ago

that's interesting, i'm well beyond some other solution but it's good to know for future projects, thanks

subash bharath Km · Accepted Answer · 2022-02-24 15:18:36Z

1

there is a version conflicts with respect to chromedriver and binary file ....https://github.com/soumilshah1995/Selenium-on-AWS-Lambda-Python3.7/blob/main/chrome_headless.zip you can add this zip file as layer to your lambda function you wont get 127 status code

answered Feb 24, 2022 at 15:18

subash bharath Km

111 bronze badge

Collectives™ on Stack Overflow

python selenium chromedriver on amazon linux / aws Lambda

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related