README.md

HumanEval

python3 run.py --models hf_internlm2_7b --datasets deprecated_humaneval_gen_d2537e --debug
python3 run.py --models hf_internlm2_chat_7b --datasets humaneval_gen_8e312c --debug

Base Models

model	pass@1
llama-7b-turbomind	12.80
llama-13b-turbomind	15.24
llama-30b-turbomind	9.15
llama-65b-turbomind	7.32
llama-2-7b-turbomind	14.02
llama-2-13b-turbomind	15.24
llama-2-70b-turbomind	15.24
llama-3-8b-turbomind	28.05
llama-3-70b-turbomind	28.05
internlm2-1.8b-turbomind	30.49
internlm2-7b-turbomind	48.17
internlm2-20b-turbomind	51.83
qwen-1.8b-turbomind	16.46
qwen-7b-turbomind	23.78
qwen-14b-turbomind	23.78
qwen-72b-turbomind	66.46
qwen1.5-0.5b-hf	8.54
qwen1.5-1.8b-hf	23.17
qwen1.5-4b-hf	41.46
qwen1.5-7b-hf	53.05
qwen1.5-14b-hf	57.32
qwen1.5-32b-hf	70.12
qwen1.5-72b-hf	65.85
qwen1.5-moe-a2-7b-hf	45.73
mistral-7b-v0.1-hf	14.02
mistral-7b-v0.2-hf	9.15
mixtral-8x7b-v0.1-hf	24.39
mixtral-8x22b-v0.1-hf	16.46
yi-6b-hf	14.63
yi-34b-hf	17.07
deepseek-7b-base-hf	18.29
deepseek-67b-base-hf	23.17

Chat Models

model	pass@1
qwen1.5-0.5b-chat-hf	9.15
qwen1.5-1.8b-chat-hf	15.85
qwen1.5-4b-chat-hf	30.49
qwen1.5-7b-chat-hf	40.85
qwen1.5-14b-chat-hf	50.00
qwen1.5-32b-chat-hf	57.93
qwen1.5-72b-chat-hf	60.37
qwen1.5-110b-chat-hf	65.24
internlm2-chat-1.8b-hf	33.54
internlm2-chat-1.8b-sft-hf	34.15
internlm2-chat-7b-hf	56.71
internlm2-chat-7b-sft-hf	61.59
internlm2-chat-20b-hf	67.68
internlm2-chat-20b-sft-hf	67.68
llama-3-8b-instruct-hf	55.49
llama-3-70b-instruct-hf	70.73
llama-3-8b-instruct-lmdeploy	57.93
llama-3-70b-instruct-lmdeploy	70.73
mistral-7b-instruct-v0.1-hf	32.32
mistral-7b-instruct-v0.2-hf	29.27
mixtral-8x7b-instruct-v0.1-hf	34.15