error: torch.isnan(scales).sum() == 0

[namtranase]

Thank you for the awesome repo. I have successfully run and tried it on the LLama model.
The problem occurs when I try to run the MPT model (7B), here are some issues, I hope u can help me with this:

1. With the model including bias, this line will report the mismatch between bias and scales -> I play around with it and fixed it
2. With the first one fixed, I can run the MPT model with small layers (I tested with 2 layers), but when testing with full 32 layers, I got the nan error in this line and it from this line
   Thank you a lot for the awesome repo.

[casper-hansen]

I just pushed a fix for MPT models in #206. Other than that, I am not sure what the issue is here?

[namtranase]

Thank you for your reply, I fixed it by changing the proper dataset. I will close the issues

[namtranase]

Hi @casper-hansen, thank you a lot for your repo. I learned a lot. I tried to apply awq to llama.cpp, here is my PR. If you have time, please give me your comment on it.
PR