Exceeded resource group quota limitation if request tokens exceeded 500ms #8349
Description
Bug Report
What did you do?
User use resource control.
What did you expect to see?
no error report
What did you see instead?
meet exceed resource group quota limitation, but RU usage is below the RU settings.
| Event 1 | 17:40:19.765 | 17:40:19.765 Coming a request and finding that the local tokens are insufficient, send a notification to the thread that acquires tokens. |
|---|---|---|
| Event 2 | 17:40:19.765 ~ 17:40:20.198 | 17:40:20.198 The request keep retrying, but the local tokens haven't refreshed yet, continuously logging the same events as before during this retry period. After retrying 500ms timeout,the report failed error to the applications. |
| Event 3 | 17:40:20.263 ~17:40:20.265 | 17:40:20.263 The thread responsible for fetching Tokens received the notification and started to send requests for tokens.17:40:20.265 Obtain new tokens authorization |
See the above table. Theoretically, Event 1 should immediately trigger Event 3. After Event 3 succeeds, then enough tokens are obtained during the retry period of Event 2, the request can continue. However, it is possible that since the current event-driven system is similar to a single-threaded event loop, in some cases, the processing delay of a certain message exceeds 500 ms, leading to a failure in obtaining tokens and resulting in an error.
What version of PD are you using (pd-server -V)?
7.5.2