Skip to content

Conversation

Jun-Howie
Copy link
Collaborator

# ❗there are glitches with vllm 0.10.1.1, still looking for resolutions❗
# ❗downgrade vllm for now ❗
pip install vllm==0.9.2 transformers==4.53.0

SITE_PACKAGES=$(pip -V | awk '{print $4}' | sed 's/\/pip$//')
# ❗patch up AWQ MoE quant config, otherwise some modules cannot be properly loaded❗
cp awq_marlin.py "$SITE_PACKAGES/vllm/model_executor/layers/quantization/awq_marlin.py"
# ❗patch up for fp32 e_score_correction_bias, see https://www.github.com/vllm-project/vllm/pull/23640❗
cp deepseek_v2.py "$SITE_PACKAGES/vllm/model_executor/models/deepseek_v2.py"

@XprobeBot XprobeBot added this to the v1.x milestone Aug 29, 2025
Copy link
Contributor

@qinxuye qinxuye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qinxuye qinxuye changed the title FEAT :Support DeepSeek-V3.1 Quantization and tool FEAT: [Model] Support DeepSeek-V3.1 Quantization and tool Aug 29, 2025
@qinxuye qinxuye merged commit 954c544 into xorbitsai:main Aug 29, 2025
4 of 13 checks passed
@Jun-Howie Jun-Howie deleted the DeepSeek-V3.1 branch August 30, 2025 01:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants