Add dynamic quantization support to gemlite layout

[mobicham]

This PR adds dynamic quantization support to the GemLite layout. The new config param is mode:

• mode="weight_only" - no activation quantization (default).
• mode="dynamic" - uses int8 dynamic quantization for A8W8 and fp8 for A8Wn.

import torch

model_id = "meta-llama/Llama-3.1-8B-Instruct"
from transformers import AutoModelForCausalLM, TorchAoConfig
from torchao.quantization import GemliteUIntXWeightOnlyConfig

quant_gemlite = GemliteUIntXWeightOnlyConfig(
    bit_width=8, group_size=None, mode="dynamic" #A8W8
)
quant_config = TorchAoConfig(
    quant_type=quant_gemlite,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="cuda",
    quantization_config=quant_config,
)

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2327

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit a94c9cf with merge base dd22777 ():

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Run Regression Tests / test-nightly (CPU Nightly, linux.4xlarge, --pre torch --index-url https://download.pytorch.org/wh... / linux-job (gh) (trunk failure)
  test/quantization/pt2e/test_x86inductor_fusion.py::TestPatternMatcher::test_smooth_quant_with_int_mm_has_bias_True_float32_per_channel_quant_True_dynamic_True
• Run Regression Tests / test-nightly (CUDA Nightly, linux.g5.12xlarge.nvidia.gpu, --pre torch --index-url https://downloa... / linux-job (gh) (trunk failure)
  test/quantization/pt2e/test_x86inductor_fusion.py::TestPatternMatcher::test_smooth_quant_with_int_mm_has_bias_True_float32_per_channel_quant_True_dynamic_True

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[jerryzh168]

I think by "static" you meant weight only?

[mobicham]

Anything else to have this merged? Thank you!

[jerryzh168]

we can merge now, the test failures does not look related