chrome profile error: session not created: DevToolsActivePort file doesn't exist

We are using latest chrome version 137.0.7151.68, earlier version used is 135.0.7049.115, with selenium while trying to open an existing chrome profile and automate opening tabs and browsing especially youtube.

Facing error as "chrome profile error: session not created: DevToolsActivePort file doesn't exist"

"
number_of_sites, driver = load_pages(options.local_IPaddress, options.sleep_time, options.result_dir, browser_obj, timestamp_g, options.premium)
                              ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Documents\Chrometesting\browsers\browser_test.py", line 63, in load_pages
    driver = browser_obj.create()
  File "C:\Users\Documents\Chrometesting\browsers\browser.py", line 63, in create
    driver = webdriver.Chrome(options=self.options)
  File "C:\Users\Documents\Chrometesting\.venv\Lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 45, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        browser_name=DesiredCapabilities.CHROME["browserName"],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        keep_alive=keep_alive,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\Documents\Chrometesting\.venv\Lib\site-packages\selenium\webdriver\chromium\webdriver.py", line 66, in __init__
    super().__init__(command_executor=executor, options=options)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Documents\Chrometesting\.venv\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 250, in __init__
    self.start_session(capabilities)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "C:\Users\Documents\Chrometesting\.venv\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 342, in start_session
    response = self.execute(Command.NEW_SESSION, caps)["value"]
               ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Documents\Chrometesting\.venv\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 429, in execute
    self.error_handler.check_response(response)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Users\Documents\Chrometesting\.venv\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 232, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: DevToolsActivePort file doesn't exist
Stacktrace:
        GetHandleVerifier [0x0x7ff68ee5fea5+79173]
        GetHandleVerifier [0x0x7ff68ee5ff00+79264]
        (No symbol) [0x0x7ff68ec19e5a]
        (No symbol) [0x0x7ff68ec5ca69]
        (No symbol) [0x0x7ff68ec565fe]
        (No symbol) [0x0x7ff68ec52b1d]
        (No symbol) [0x0x7ff68eca67de]
        (No symbol) [0x0x7ff68eca5f70]
        (No symbol) [0x0x7ff68ec98743]
        (No symbol) [0x0x7ff68ec614c1]
        (No symbol) [0x0x7ff68ec62253]
        GetHandleVerifier [0x0x7ff68f12a2dd+3004797]
        GetHandleVerifier [0x0x7ff68f12472d+2981325]
        GetHandleVerifier [0x0x7ff68f143380+3107360]
        GetHandleVerifier [0x0x7ff68ee7aa2e+188622]
        GetHandleVerifier [0x0x7ff68ee822bf+219487]
        GetHandleVerifier [0x0x7ff68ee68df4+115860]
        GetHandleVerifier [0x0x7ff68ee68fa9+116297]
        GetHandleVerifier [0x0x7ff68ee4f558+11256]
        BaseThreadInitThunk [0x0x7ff91ccb269d+29]
        RtlUserThreadStart [0x0x7ff91db8a9f8+40]
"

Tried out approaches to fix the issue:

1. Creating a temporary directory using the user profile data: Profile shown as unverified sign-in and causes further execution failures.
2. Options tried out to configure chrome browser to access user profile: --no-sandbox, --disable-dev-shm-usage, --remote-debugging-port=9222

We access the chrome profile from the system as defined in the following code

 if self.premium:
        base_path = "AppData\\Local\\Google\\Chrome\\User Data"
        home_directory = os.path.expanduser("~")
        user_data_directory = os.path.join(home_directory, base_path)
        user_data_directory = os.path.normpath(user_data_directory).replace(os.sep, '/')
        profile_dirs = []
        profile_dir = None

        # Locate the last accessed Chrome Profile
        for entry in os.listdir(user_data_directory):
            if entry == "Default" or entry.startswith("Profile"):
                full_path = os.path.join(user_data_directory, entry)
                modified_time = os.path.getmtime(full_path)
                profile_dirs.append((modified_time, entry))
        profile_dir = max(profile_dirs)[1]

If any solution for the same will be helpful.