ImportError: cannot import name 'to_bytes' from 'urllib3.util.util' with requests==2.32.4 and urllib3==2.5.0

[zromick]

Summary: ImportError: cannot import name 'to_bytes' from 'urllib3.util.util' with requests==2.32.4 and urllib3==2.5.0

Expected Result

The requests library (version 2.32.4) should function correctly with urllib3 version 2.5.0, without encountering ImportError for to_bytes. Our CI pipeline should successfully install all dependencies and build the Docker image.

Actual Result

Our CI pipeline fails during the pdm install step with the following traceback:

#16 11.64 ✖ Install PACKAGE 0.8.1 failed
#16 11.66 pdm.termui: Error occurs adding PACKAGE:
#16 11.66 Traceback (most recent call last):
#16 11.66 File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
#16 11.66 result = self.fn(*self.args, **self.kwargs)
#16 11.66 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 228, in install_candidate
#16 11.66 self.manager.install(can)
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/pdm/installers/manager.py", line 33, in install
#16 11.66 prepared.build(),
#16 11.66 ^^^^^^^^^^^^^^^^
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 397, in build
#16 11.66 self._obtain(allow_all=False)
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 451, in _obtain
#16 11.66 self._unpack(validate_hashes=not allow_all)
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 458, in _unpack
#16 11.66 with self.environment.get_finder() as finder:
#16 11.66 File "/usr/local/lib/python3.11/contextlib.py", line 137, in enter
#16 11.66 return next(self.gen)
#16 11.66 ^^^^^^^^^^^^^^
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/pdm/environments/base.py", line 145, in get_finder
#16 11.66 finder = PDMPackageFinder(
#16 11.66 ^^^^^^^^^^^^^^^^^
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/pdm/models/finder.py", line 71, in init
#16 11.66 super().init(session, **kwargs)
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/unearth/finder.py", line 129, in init
#16 11.66 _check_legacy_session(session)
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/unearth/finder.py", line 52, in _check_legacy_session
#16 11.66 from requests import Session
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/requests/init.py", line 43, in
#16 11.66 import urllib3
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/urllib3/init.py", line 15, in
#16 11.66 from ._base_connection import _TYPE_BODY
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/urllib3/_base_connection.py", line 5, in
#16 11.66 from .util.connection import _TYPE_SOCKET_OPTIONS
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/urllib3/util/init.py", line 5, in
#16 11.66 from .request import SKIP_HEADER, SKIPPABLE_HEADERS, make_headers
#16 11.66 File "/srv/.venv/lib/python3.11/site-packages/urllib3/util/request.py", line 9, in
#16 11.66 from .util import to_bytes
#16 11.66 ImportError: cannot import name 'to_bytes' from 'urllib3.util.util' (/srv/.venv/lib/python3.11/site-packages/urllib3/util/util.py)
#16 11.66 ✔ Install urllib3 2.5.0 successful
#16 11.68 ✖ Install PACKAGE 0.4.1 failed
#16 11.68 pdm.termui: Error occurs adding PACKAGE:
#16 11.68 Traceback (most recent call last):
#16 11.68 File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
#16 11.68 result = self.fn(*self.args, **self.kwargs)
#16 11.68 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 228, in install_candidate
#16 11.68 self.manager.install(can)
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/pdm/installers/manager.py", line 33, in install
#16 11.68 prepared.build(),
#16 11.68 ^^^^^^^^^^^^^^^^
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 397, in build
#16 11.68 self._obtain(allow_all=False)
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 429, in _obtain
#16 11.68 with self.environment.get_finder(sources, env_spec=env_spec) as finder:
#16 11.68 File "/usr/local/lib/python3.11/contextlib.py", line 137, in enter
#16 11.68 return next(self.gen)
#16 11.68 ^^^^^^^^^^^^^^
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/pdm/environments/base.py", line 145, in get_finder
#16 11.68 finder = PDMPackageFinder(
#16 11.68 ^^^^^^^^^^^^^^^^^
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/pdm/models/finder.py", line 71, in init
#16 11.68 super().init(session, **kwargs)
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/unearth/finder.py", line 129, in init
#16 11.68 _check_legacy_session(session)
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/unearth/finder.py", line 52, in _check_legacy_session
#16 11.68 from requests import Session
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/requests/init.py", line 43, in
#16 11.68 import urllib3
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/urllib3/init.py", line 15, in
#16 11.68 from ._base_connection import _TYPE_BODY
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/urllib3/_base_connection.py", line 7, in
#16 11.68 from .util.url import Url
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/urllib3/util/init.py", line 5, in
#16 11.68 from .request import SKIP_HEADER, SKIPPABLE_HEADERS, make_headers
#16 11.68 File "/srv/.venv/lib/python3.11/site-packages/urllib3/util/request.py", line 9, in
#16 11.68 from .util import to_bytes
#16 11.68 ImportError: cannot import name 'to_bytes' from 'urllib3.util.util' (/srv/.venv/lib/python3.11/site-packages/urllib3/util/util.py)
#16 11.69 ✖ Install websockets 15.0.1 failed

... (subsequent errors for other packages) ...

This error occurs specifically when urllib3 is resolved to version 2.5.0. When urllib3 is resolved to 2.4.0 (with the same requests version), the build passes.

Reproduction Steps

This issue is observed in a PDM-managed Python project within a Docker build environment.

Project Setup:
PACKAGE/pyproject.toml contains:
[project]
name = "PACKAGE"
version = "0.4.1"
dependencies = [
"requests>=2.31.0",
"ray[default]==2.41",
"PACKAGE", # Placeholder for common package
"PACKAGE", # Placeholder for core package
# ... other dependencies
]
requires-python = "==3.11.*"

The main application project (another PACKAGE) depends on this PACKAGE.
Dependency Resolution (PDM):
When pdm lock is run and resolves requests to 2.32.4 and urllib3 to 2.5.0 (as per the attached pdm.lock in the original issue description), the subsequent pdm install fails.
When pdm lock resolves requests to 2.32.4 and urllib3 to 2.4.0 (as per the attached pdm.lock from 3 weeks ago), the pdm install succeeds.
Minimal Python Code (Illustrative, as the error occurs during package loading): While the error occurs during package installation/loading, the underlying incompatibility is demonstrated by the import failure. A direct requests import might not immediately trigger it, but it's where the urllib3 dependency is consumed.
import requests
The error occurs during the internal loading of requests' dependencies,
specifically when it tries to import 'to_bytes' from urllib3.util.util.
This suggests an incompatibility between requests 2.32.4 and urllib3 2.5.0
regarding the location or existence of 'to_bytes'.

System Information
{
"chardet": {
"version": null
},
"charset_normalizer": {
"version": "3.4.2"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "3.10"
},
"implementation": {
"name": "CPython",
"version": "3.11.11"
},
"platform": {
"release": "24.5.0",
"system": "Darwin"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.32.4"
},
"system_ssl": {
"version": "30500000"
},
"urllib3": {
"version": "2.5.0"
},
"using_charset_normalizer": true,
"using_pyopenssl": false
}

[nateprewitt]

Hi @zromick,

This seems like a potential issue in your environment, perhaps stemming from pdm. We can look at the stacktrace you provided and see the failure mode is entirely self contained in urllib3, it's not related to Requests.

https://github.com/psf/requests/issues/16 11.68 File "/srv/.venv/lib/python3.11/site-packages/requests/init.py", line 43, in
https://github.com/psf/requests/issues/16 11.68 import urllib3
https://github.com/psf/requests/issues/16 11.68 File "/srv/.venv/lib/python3.11/site-packages/urllib3/init.py", line 15, in
https://github.com/psf/requests/issues/16 11.68 from ._base_connection import _TYPE_BODY
https://github.com/psf/requests/issues/16 11.68 File "/srv/.venv/lib/python3.11/site-packages/urllib3/_base_connection.py", line 7, in
https://github.com/psf/requests/issues/16 11.68 from .util.url import Url
https://github.com/psf/requests/issues/16 11.68 File "/srv/.venv/lib/python3.11/site-packages/urllib3/util/init.py", line 5, in
https://github.com/psf/requests/issues/16 11.68 from .request import SKIP_HEADER, SKIPPABLE_HEADERS, make_headers
https://github.com/psf/requests/issues/16 11.68 File "/srv/.venv/lib/python3.11/site-packages/urllib3/util/request.py", line 9, in
https://github.com/psf/requests/issues/16 11.68 from .util import to_bytes

The act of importing urllib3 is triggering this call chain, which would mean urllib 2.5.0 is non-functioning. Since we know there's wide usage without problems in that releases, something else is going wrong.

Looking at urllib3 2.5.0, we can see to_bytes is importable from urllib3.util.util. This is likely something to approach pdm about as it's either an issue with the packages it's finding or a configuration issue with the tool. I don't believe we'll be able to provide much assistance there.

[zromick]

Thanks, will look into it. I appreciate your response.