Bypassing Captcha when Chrome opened in remote debugging mode

[ds4573]

I've used SeleniumBase OK in two situations independently - it's great.

1. Log onto a website and bypass Cloudflare
2. Connect to an existing Chrome Window already opened via remote debugging (--remote-debugging-port=9222 etc etc)

Has already managed to combine the two? eg

1. Open Chrome with remote debugging
2. Use SeleniumBase to connect to open Chrome
   ---- so far so good
   3.When I then try to log on to the required website, Cloudflare stops things. I assue that it detects Chrome in remote debugging mode and there's no way around this.

Any ideas / suggestions? The application might seem weird, but I have a system using SeleniumVBA in Excel that no longer works because CloudFlare blocks login. I can rewrite everything to use SeleniumBase but am trying to avoid the work. If I can get logged in via SeleniumBase to the open Chrome window, I can then connect into this with SeleniumVBA and its business as usual.

Look forward ot any suggestion, Dain

[mdmintz]

Taking control of existing browsers via remote-debugging-port is covered in https://www.youtube.com/watch?v=vt2zsdiNh3U.

Note that if you take control of an existing browser that wasn't stealthy to begin with, then you probably still won't be stealthy.

Plenty of examples of Cloudflare-bypass in the SeleniumBase/examples/cdp_mode folder:

• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_gitlab.py
• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_planetmc.py
• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_ahrefs.py
• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_cf.py

[ds4573]

Thanks MIke, appreaciate the quick response.
Launching Chrome in remote debugger mode via command prompt is not stealthy.
So is there actually a way to do it to make it stealthy? Can SeleniumBase be used to do this?
Regards, Dain

[mdmintz]

All these examples are bypassing Cloudflare:

• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_gitlab.py
• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_planetmc.py
• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_ahrefs.py
• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_cf.py

(And the remote-debugging-port is used to to connect the CDP-Driver to the browser that was spun up with the stealthy version of Selenium. That's why you can call sb.reconnect() from those examples and call regular Selenium WebDriver methods again.)

[ds4573]

Sorry Mike, you're dealing with a Python-challenged 72 year old Australian. I'm OK with Excel VBA, you are operating at about 10 levels higher in the programming world.

But guess what, I got it to f***ing work. Turned out to be easy.

I ran the gitlab example with the URL changed to the page I want to work with, got the page open with Captcha bypassed, and left the page open by using a checkpoint() at the end of the code. Then I ran my Excel VBA code using port 9222 (guessed the number) and it hooked in no problem.

One question, the Chrome window closes when the code is complete. Is there any way of leaving it open (other than checkpoint() or similar?

If you're ever around the Sunshine Coast, Queensland, Australia, I'll buy you a beer.

Dain

[mdmintz]

breakpoint() is the way to keep the browser open at the end of the script.

[ds4573]

Next day..... Not sure under GitHub etiquette whether I should continue this discussion or start a new one with a different subject.
I need to distribute stuff to a few computer challenged usres who wont have Pyhton on their computers. So I want to convert the py file to exe with pyinstaller.
I tried doing this with my PY code, and ran into problems. So I tried doing the same thing with a couple of yours - raw_cf.py and raw_plantetmc.py, and got the same error.
When I run the executables on my Windows11 PC, I get this error

File "raw_cf.py", line 4, in
File "contextlib.py", line 141, in enter
File "seleniumbase\plugins\sb_manager.py", line 1400, in SB
File "seleniumbase\fixtures\base_case.py", line 15766, in setUp
File "seleniumbase\fixtures\base_case.py", line 4308, in get_new_driver
File "seleniumbase\core\browser_launcher.py", line 3464, in get_driver
File "seleniumbase\core\browser_launcher.py", line 5372, in get_local_driver
File "shutil.py", line 454, in copy2
FileNotFoundError: [WinError 3] The system cannot find the path specified
[PYI-6672:ERROR] Failed to execute script 'raw_cf' due to unhandled exception!

Any ideas? I wouldnt have a clue about whether this is something specific to my PC, or a general Python issue, or Pyinstaller not doing its job properly, something to do with SeleniumBase.

Regards, Dain

PS Also tried Py2Exe to build the file, same result

[mdmintz]

Try with the pure CDP formats (sb_cdp), eg:

• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_cdp_copilot.py
• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_cdp_methods.py
• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/raw_cdp_gitlab.py

Pyinstaller may have issues with the formats that use chromedriver. sb_cdp doesn't need or use it.

[ds4573]

Answer is getting complicated but here goes

1. I used pyinstaller on the raw_cpd_methods type files, I get a new set of errors

Traceback (most recent call last):
File "raw_cdp_methods.py", line 9, in
File "bdb.py", line 116, in trace_dispatch
File "bdb.py", line 215, in dispatch_opcode
File "pdb.py", line 448, in user_line
File "pdb.py", line 625, in interaction
File "pdb.py", line 511, in _cmdloop
File "cmd.py", line 111, in cmdloop
AttributeError: module 'readline' has no attribute 'backend'
[PYI-15488:ERROR] Failed to execute script 'raw_cdp_methods' due to unhandled exception!

In any case, even when I run as a PY file, these methods don't help me as they dont seem to generate a Chrome remote debugging window (happy to be corrected here if this is an incorrect statement)

2. I tried the old files (sb.activate_cdp_mode(url) type method) on a "newish" computer which didn't have SeleniumBase, Pyinstaller or Python installed. Also no Chromedriver. Once things were set up, I compiled the raw_github.py examples from yesterday. When I run it, it generates a lot of message about missing drivers

Warning: uc_driver not found. Getting it now:

*** chromedriver to download = 142.0.7444.175 (Previous Version)

Downloading chromedriver-win64.zip from:
https://storage.googleapis.com/chrome-for-testing-public/142.0.7444.175/win64/chromedriver-win64.zip ...
Download Complete!

Extracting ['chromedriver.exe'] from chromedriver-win64.zip ...
Unzip Complete!

The file [uc_driver.exe] was saved to:
C:\Users\dains\AppData\Local\Temp_MEI114202\seleniumbase\drivers
uc_driver.exe

Making [uc_driver.exe 142.0.7444.175] executable ...
[uc_driver.exe 142.0.7444.175] is now ready for use!

Then things work - webpages open and Captcha is bypassed - but then things eventually fail with all the same messages as shown in section 1 above.

3. I then went back to my main computer and temproarily got rid of my chromedriver file.

When I complied and ran, there was the same behaviour as in section 2. Same missing drivers, installing uc_driver etc
Then the pages loaded, Captcha bypassed, then things failed with the same set of messages as shown in section 1.

So suddenly there are two sets of issues -the one relating to presence or not of Chromedriver, the second relating to the messages in section 1.

The weird thing is despite all of the above, I have a workable solution - if get rid of the chromedriver, and put in a very long sleep function, I can get my page open, Capcha bypassed, and do the required scraping via my Excel progam.

Dain