You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: docs/en/get_started/quick_start.md
+36-38
Original file line number
Diff line number
Diff line change
@@ -22,7 +22,43 @@ For larger parameterized models like Llama-7B, refer to other examples provided
22
22
In OpenCompass, each evaluation task consists of the model to be evaluated and the dataset. The entry point for evaluation is `run.py`. Users can select the model and dataset to be tested either via command line or configuration files.
23
23
24
24
`````{tabs}
25
+
````{tab} Command Line (Custom HF Model)
26
+
27
+
For HuggingFace models, users can set model parameters directly through the command line without additional configuration files. For instance, for the `facebook/opt-125m` model, you can evaluate it with the following command:
28
+
29
+
```bash
30
+
python run.py --datasets siqa_gen winograd_ppl \
31
+
--hf-type base \
32
+
--hf-path facebook/opt-125m
33
+
```
25
34
35
+
Note that in this way, OpenCompass only evaluates one model at a time, while other ways can evaluate multiple models at once.
36
+
37
+
```{caution}
38
+
`--hf-num-gpus` does not stand for the actual number of GPUs to use in evaluation, but the minimum required number of GPUs for this model. [More](faq.md#how-does-opencompass-allocate-gpus)
39
+
```
40
+
41
+
:::{dropdown} More detailed example
42
+
:animate: fade-in-slide-down
43
+
```bash
44
+
python run.py --datasets siqa_gen winograd_ppl \
45
+
--hf-type base \ # HuggingFace model type, base or chat
46
+
--hf-path facebook/opt-125m \ # HuggingFace model path
47
+
--tokenizer-path facebook/opt-125m \ # HuggingFace tokenizer path (if the same as the model path, can be omitted)
48
+
--tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True \ # Arguments to construct the tokenizer
49
+
--model-kwargs device_map='auto' \ # Arguments to construct the model
50
+
--max-seq-len 2048 \ # Maximum sequence length the model can accept
51
+
--max-out-len 100 \ # Maximum number of tokens to generate
52
+
--min-out-len 100 \ # Minimum number of tokens to generate
53
+
--batch-size 64 \ # Batch size
54
+
--hf-num-gpus 1 # Number of GPUs required to run the model
55
+
```
56
+
```{seealso}
57
+
For all HuggingFace related parameters supported by `run.py`, please read [Launching Evaluation Task](../user_guides/experimentation.md#launching-an-evaluation-task).
58
+
```
59
+
:::
60
+
61
+
````
26
62
````{tab} Command Line
27
63
28
64
Users can combine the models and datasets they want to test using `--models` and `--datasets`.
@@ -74,44 +110,6 @@ If you want to evaluate other models, please check out the "Command Line (Custom
74
110
75
111
````
76
112
77
-
````{tab} Command Line (Custom HF Model)
78
-
79
-
For HuggingFace models, users can set model parameters directly through the command line without additional configuration files. For instance, for the `facebook/opt-125m` model, you can evaluate it with the following command:
80
-
81
-
```bash
82
-
python run.py --datasets siqa_gen winograd_ppl \
83
-
--hf-type base \
84
-
--hf-path facebook/opt-125m
85
-
```
86
-
87
-
Note that in this way, OpenCompass only evaluates one model at a time, while other ways can evaluate multiple models at once.
88
-
89
-
```{caution}
90
-
`--hf-num-gpus` does not stand for the actual number of GPUs to use in evaluation, but the minimum required number of GPUs for this model. [More](faq.md#how-does-opencompass-allocate-gpus)
91
-
```
92
-
93
-
:::{dropdown} More detailed example
94
-
:animate: fade-in-slide-down
95
-
```bash
96
-
python run.py --datasets siqa_gen winograd_ppl \
97
-
--hf-type base \ # HuggingFace model type, base or chat
98
-
--hf-path facebook/opt-125m \ # HuggingFace model path
99
-
--tokenizer-path facebook/opt-125m \ # HuggingFace tokenizer path (if the same as the model path, can be omitted)
100
-
--tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True \ # Arguments to construct the tokenizer
101
-
--model-kwargs device_map='auto' \ # Arguments to construct the model
102
-
--max-seq-len 2048 \ # Maximum sequence length the model can accept
103
-
--max-out-len 100 \ # Maximum number of tokens to generate
104
-
--min-out-len 100 \ # Minimum number of tokens to generate
105
-
--batch-size 64 \ # Batch size
106
-
--hf-num-gpus 1 # Number of GPUs required to run the model
107
-
```
108
-
```{seealso}
109
-
For all HuggingFace related parameters supported by `run.py`, please read [Launching Evaluation Task](../user_guides/experimentation.md#launching-an-evaluation-task).
110
-
```
111
-
:::
112
-
113
-
114
-
````
115
113
````{tab} Configuration File
116
114
117
115
In addition to configuring the experiment through the command line, OpenCompass also allows users to write the full configuration of the experiment in a configuration file and run it directly through `run.py`. The configuration file is organized in Python format and must include the `datasets` and `models` fields.
0 commit comments