Skip to content

Minimize the impact on the lag of existing changefeeds when creating a new changefeed. #1942

@asddongmen

Description

@asddongmen

Core Requirements

Optimize the puller task scheduling mechanism to prevent changefeeds (CFs) with a large volume of incremental scan tasks from starving (i.e., blocking or delaying) other incremental scan tasks generated by region changes. The optimization must meet two key goals:

  1. Ensure delay stability for all changefeeds (no significant lag fluctuations across CFs);
  2. Avoid compromising the efficiency of existing incremental scan operations.

Core Priority Design Principles

Priority rules shall be formulated based on the intrinsic characteristics of tasks, with the following specifications:

  1. Highest Priority Scenario:
    RegionTasks triggered by the need for region reconnection (e.g., due to region migration, node failure, or other region-level changes) shall be assigned the highest priority.
  2. General Priority Rule:
    For all other RegionTasks (i.e., those not related to reconnection), priority shall increase with task waiting time. This design aims to approximate a "first-in-first-out (FIFO)" scheduling effect, thereby preventing long-term accumulation of pending tasks.

Metadata

Metadata

Assignees

Labels

type/enhancementThe issue or PR belongs to an enhancement.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions