Add mx_fp4 path

[drisspg]

Stacked PRs:

• ->Add mx_fp4 path #2201

---

Add mx_fp4 path

BF16 70B MLP: https://fburl.com/aeqm5s4v~ 1300 us
MXFP8 70B MLP: https://fburl.com/uxgoju4r ~ 723us
MXFP4 70B MLP: https://fburl.com/u95f6f39 ~ 600us

Looks like we need to do some more kernel tuning since the gemm kernel is only about 70% the execution time instead of of 50%

Benchmarks

Baseline Model
Throughput: 56.68 requests/s, 24053.96 total tokens/s, 11590.20 output tokens/s
Total num prompt tokens: 225190
Total num output tokens: 209407

python /home/drisspg/meta/vllm/benchmarks/benchmark_throughput.py \
 --backend vllm \
 --model "Qwen/Qwen2-7B-Instruct" \
 --dataset-name sharegpt \
 --dataset-path /home/drisspg/meta/scripts/data/ShareGPT_V3_unfiltered_cleaned_split.json \
 --num-prompts 1024 \
 --disable-log-stats \
 --gpu-memory-utilization=0.9 \
 --seed 42

MXFP8:
Throughput: 50.52 requests/s, 21443.10 total tokens/s, 10332.18 output tokens/s
Total num prompt tokens: 225190
Total num output tokens: 209407

python /home/drisspg/meta/vllm/benchmarks/benchmark_throughput.py \
 --backend vllm \
 --model "/home/drisspg/meta/scripts/data/mxfp8-Qwen2-7B-Instruct" \
 --dataset-name sharegpt \
 --dataset-path /home/drisspg/meta/scripts/data/ShareGPT_V3_unfiltered_cleaned_split.json \
 --num-prompts 1024 \
 --disable-log-stats \
 --gpu-memory-utilization=0.9 \
 --seed 42

MXFP4:

Throughput: 56.64 requests/s, 24039.96 total tokens/s, 11583.46 output tokens/s
Total num prompt tokens:  225190
Total num output tokens:  209407

python /home/drisspg/meta/vllm/benchmarks/benchmark_throughput.py
--backend vllm
--model "/home/drisspg/meta/scripts/data/mxfp4-Qwen2-7B-Instruct"
--dataset-name sharegpt
--dataset-path /home/drisspg/meta/scripts/data/ShareGPT_V3_unfiltered_cleaned_split.json
--num-prompts 1024
--disable-log-stats
--gpu-memory-utilization=0.9
--seed 42

70b

Base
Throughput: 26.28 requests/s, 11154.41 total tokens/s, 5374.66 output tokens/s
Total num prompt tokens: 225190
Total num output tokens: 209407

VLLM_USE_V1=1 VLLM_DISABLE_COMPILE_CACHE=1 python /home/drisspg/meta/vllm/benchmarks/benchmark_throughput.py \                          3m 43s  nightly
 --backend vllm \
 --model "Qwen/Qwen2.5-72B" \
 --dataset-name sharegpt \
 --dataset-path /home/drisspg/meta/scripts/data/ShareGPT_V3_unfiltered_cleaned_split.json \
 --num-prompts 1024 \
 --disable-log-stats \
 --gpu-memory-utilization=0.9 \
 --tensor-parallel-size 8 \
 --seed 42

MXFP4
Throughput: 25.96 requests/s, 11018.18 total tokens/s, 5309.02 output tokens/s
Total num prompt tokens: 225190
Total num output tokens: 209407

 VLLM_USE_V1=1 VLLM_DISABLE_COMPILE_CACHE=1 python /home/drisspg/meta/vllm/benchmarks/benchmark_throughput.py \ 
 --backend vllm \
 --model "data/mxfp4-Qwen2.5-72B" \
 --dataset-name sharegpt \
 --dataset-path /home/drisspg/meta/scripts/data/ShareGPT_V3_unfiltered_cleaned_split.json \
 --num-prompts 1024 \
 --disable-log-stats \
 --gpu-memory-utilization=0.9 \
 --tensor-parallel-size 8 \
 --seed 42

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2201

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [PREEMPTIVE] Removal of ephemeral variants on scale-config.yml

This comment was automatically generated by Dr. CI and updates every 15 minutes.