[router] grpc router bootstraps

[slin1237]

Modifications

1. add connection mode in worker trait
2. add grpc router place holders
3. re-organize http proxy router to its own folder
4. add place holder for rerank and embeddings

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Summary of Changes

Hello @slin1237, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request initiates the integration of gRPC router support by introducing a ConnectionMode for workers and creating placeholder gRPC router implementations. It also refactors the existing HTTP proxy router components into a dedicated module for better organization and adds new placeholder endpoints for embeddings and rerank functionalities.

Highlights

• Worker Connection Mode: A new ConnectionMode enum has been introduced to distinguish between HTTP and gRPC worker communication protocols, and the Worker trait and WorkerMetadata have been updated to incorporate this new mode.
• gRPC Router Bootstrapping: Placeholder implementations for gRPC routers (GrpcRouter and GrpcPDRouter) have been added, laying the foundation for future gRPC support.
• HTTP Router Refactoring: Existing HTTP proxy router components, including pd_router, router, and pd_types, have been reorganized into a dedicated http submodule for improved code structure and clarity.
• New Router Endpoints: The RouterTrait has been extended to include route_embeddings and route_rerank methods, with todo!() placeholders in current implementations, indicating future support for these functionalities.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request is a solid step towards introducing gRPC support, with a well-executed refactoring of the router modules into http and grpc subdirectories. The addition of ConnectionMode to the Worker trait is a key part of this groundwork. The new placeholder implementations for gRPC routers and the embeddings/rerank endpoints are appropriate for this stage. My main suggestion is to replace the todo!() macros in the new HTTP router methods with a 501 Not Implemented response. This will prevent potential panics and ensure the server fails gracefully if these endpoints are called before they are fully implemented.