Fix input plane attempt retry handling. #3565
Open
+281
−68
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thomasjpfan
reviewed
Sep 18, 2025
davidxia
added a commit
that referenced
this pull request
Sep 18, 2025
> Checks for functions with a high McCabe complexity https://docs.astral.sh/ruff/rules/complex-structure/ Motivated by bugs like the one introduced in #3456 and fixed in #3565. Signed-off-by: David Xia <[email protected]>
This was referenced Sep 18, 2025
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
danielshaar
force-pushed
the
dshaar/fix-input-plane-retries
branch
from
7772425 to
41f9c58
Compare
thomasjpfan
approved these changes
Sep 19, 2025
Use [parameterized pytests][1] to be DRY. [1]: https://docs.pytest.org/en/stable/example/parametrize.html Signed-off-by: David Xia <[email protected]>
davidxia
reviewed
Sep 19, 2025
Comment on lines
-319
to
-320
| # AttemptAwait will return a failure until this is 0. It is decremented by 1 each time AttemptAwait is called. | ||
| self.attempt_await_failures_remaining = 0 |
There was a problem hiding this comment.
I refactored unit tests to not require this attribute.
| @@ -91,7 +322,7 @@ def test_map(client, servicer, slow_put_inputs): | |||
|
|
|||
| @pytest.mark.parametrize("slow_put_inputs", [False, True]) | |||
| @pytest.mark.timeout(120) | |||
| def test_map_inputplane(client, servicer, slow_put_inputs): | |||
| def test_map_input_plane(client, servicer, slow_put_inputs): | |||
There was a problem hiding this comment.
This change and below are unrelated and simply fix camel-casing of "input plane" to be consistent.
| assert servicer.attempt_await_failures_remaining == 0 | ||
|
|
||
|
|
||
| def test_retry_limit_on_internal_error(client: Client, servicer: MockClientServicer, monkeypatch): |
| @@ -35,25 +32,3 @@ def test_lookup_foo(client: Client, servicer: MockClientServicer): | |||
| assert f.remote() == "foo" | |||
| assert f._get_metadata().input_plane_url is not None | |||
| assert f._get_metadata().input_plane_region == "us-east" | |||
|
|
|||
|
|
|||
| def test_retry_on_internal_error(client: Client, servicer: MockClientServicer): | |||
|
I manually tested many of the unit test cases against prod. The client behaved as I expected. |
thomasjpfan
reviewed
Sep 19, 2025
| ), | ||
| ], | ||
| ) | ||
| def test_remote_input_plane( |
There was a problem hiding this comment.
Can we reduce the duration of some of these tests?
60.53s call test/function_test.py::test_remote_input_plane[long running]
7.43s call test/function_test.py::test_remote_input_plane[honor user retry policy]
7.41s call test/function_test.py::test_remote_input_plane[internal failure does not use up user retries]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe your changes
We have a bug in which we create excessive input plane attempts due to accidentally submitting AttemptRetry on every AttemptAwait timeout of 55s. We haven't caught it because most input plane calls take less than a minute. This will be patched server-side.
Also fixes a separate bug where internal failures can eat into client specified retries.
SVC-863