Restoring SlimPajama Preprocessing Code

[aflah02]

Addresses #1837 (comment)

[aflah02]

Hi @Andrei-Aksionov
Not sure why the tests are failing as I just copied over a file

[Andrei-Aksionov]

Today was a release of LitServe and apparently it broke something.
Try to pin the version of it in pyproject.toml:

"litserve>=0.2.4",           # litgpt.deploy

It will do for now, later I'll try to fix it and unpin the version.

cc @aniketmaurya

[aflah02]

Today was a release of LitServe and apparently it broke something. Try to pin the version of it in pyproject.toml:

"litserve>=0.2.4",           # litgpt.deploy

It will do for now, later I'll try to fix it and unpin the version.

cc @aniketmaurya

This still fails. Should I pin it to some other version?

[aniketmaurya]

seems like some other issue with LitGPT? @Andrei-Aksionov

[Andrei-Aksionov]

seems like some other issue with LitGPT?

I doubt it.
This morning it was fine.
And in this PR only one file is added.

I'll investigate it later.

[Andrei-Aksionov]

Hey @aflah02
The name of your branch (in your fork) tricked by github CLI :)
So I accidentally made changed to our main.
(And had to deal with it.)

Could you create one more PR? 😊
The same content and also with pinned LitServe.

I unpined it today to confirm that the issue with tests was indeed caused by LitServe.
With pinned version all workflows (apart from LitGPT + Thunder) were green.

[aflah02]

Ah got it
Thanks!
Just to be sure, should I use the pin you suggested or the one @aniketmaurya did?

[Andrei-Aksionov]

Yes