Skip to content

Commit 266f36a

Browse files
authored
Update README.md
1 parent 93851cb commit 266f36a

File tree

1 file changed

+1
-3
lines changed

1 file changed

+1
-3
lines changed

README.md

+1-3
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@
22

33
Code to reproduce results for the paper "Faster Causal Attention Over Large Sequences Through Sparse Flash Attention"
44

5-
Arxiv link.
6-
75
# Setup
86

97
To install the required python dependencies, first run:
@@ -25,7 +23,7 @@ In the command above we set the Triton library to the commit used in our experim
2523

2624
# Reproducing our LM experiments on OpenWebText2
2725

28-
**GPU requirements:** Prefeerably, you need at least one A100. Some of our experiments use data-parallelism with up to 3 A100s. You should have no problem running those experiments on any GPU supporting `bfloat16`, you might have to change the model parameters to adapt to the memory available.
26+
**GPU requirements:** Preferably, you need at least one A100. Some of our experiments use data-parallelism with up to 3 A100s. You should have no problem running those experiments on any GPU supporting `bfloat16`, you might have to change the model parameters to adapt to the memory available.
2927

3028
Go in the `openwebtext2-experiments` folder and run the `script/train-LMs.sh` command.
3129

0 commit comments

Comments
 (0)