Convert QLoRA trained model to AWQ

[vidhyat98]

Hi there!
I have an MPT model that was fine tuned with QLoRA. I merged the QLoRA weights to the original model and saved it in 16bit (because save_pretrained() of transformers doesn't allow to save 4bit versions yet).
I am trying to convert this 16bit model to AWQ. After loading the model, when I apply model.quantize(tokenizer, quant_config=quant_config), I get an error saying:
TypeError: forward() got an unexpected keyword argument 'output_attentions'

How do I solve this?
I am using "0.1.6" version of AutoAWQ and downgraded transformers to 4.34. Torch and other libraries are of the version mentioned in the setup.py file.

[casper-hansen]

I see now that there is an incompatibility. MPT models used to have seamless compatibility, but they pushed updates to the model repositories that are not pushed into Huggingface transformers.

I will look into if there is a workaround for this.

[vidhyat98]

Thanks, I'll be eagerly looking forward to your response! This AWQ conversion is a crucial step in my project :)

[casper-hansen]

If I were you, an easy thing to try is going many versions back to see if you can find one that works with transformers+AutoAWQ.

[vidhyat98]

Hi! I tried with various versions of transformers from 4.30 to 4.36 against different autoawq versions (upto 0.1.7). Still getting the error..
This is mosaicml/mpt-30b-instruct being used as the base model.