End-to-end training of DeepSeek V3

[EugenHotaj]

Any plans to have a real training script for DSV3?

Right now run.py only has the forward pass on some dummy data on DSV2 so it's unclear how much of DSV3 is supported and whether it actually works.

For the DSV3 forward pass, I was able to run using 32 H200s but had to lower config.max_seq_len quite a bit, otherwise was OOMing when setting up symmetric memory. Would love to be able to train DSV3!

[lessw2020]

Hi @EugenHotaj - yes a full DS v3 training script is coming.

The current PR's are part of an iterative process...more is coming soon!

[EugenHotaj]

@lessw2020 amazing, looking forward to it!