Isn't it better to enable use_quantized_matmul?
#2
by wtrida19933 - opened
It should make generation much faster as far as I understand.
I tried to enable it in current config.json, but got
sdnq/quant_utils.py", line 182, in prepare_weight_for_matmul
weight = weight.t_().contiguous().t_()
^^^^^^^^^^^
RuntimeError: t_() expects a tensor with <= 2 dimensions, but self is 3D
I suppose it's not a runtime param and should be enabled during quantization.
it is a runtime param and not everyone has triton installed, that's why it is usually something you set in the script and not in the model config.
Not sure what you did but I run it with something like this:
pipe.transformer = apply_sdnq_options_to_model(pipe.transformer, use_quantized_matmul=True)
pipe.text_encoder = apply_sdnq_options_to_model(pipe.text_encoder, use_quantized_matmul=True)
and it works fine.