Skip to content

add MMLU benchmark val / test #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Jul 14, 2023
Merged

add MMLU benchmark val / test #1

merged 11 commits into from
Jul 14, 2023

Conversation

@LZHgrla
Copy link
Contributor

@LZHgrla LZHgrla commented Jul 13, 2023 •
edited
Loading

MMLU benchmark val / test. This implementation follows the approach of the original implementation and QLoRA.

Model Split Method Original Impl. Ours
LLaMA 7B val zero-shot 32.3 33.3
val five-shot 33.6 33.2
test zero-shot 32.5 32.9
test five-shot 35.1 35.6

Note: We use the preprocessed MMLU dataset from QLoRA, instead of the original dataset.

@LZHgrla LZHgrla requested a review from pppppM July 13, 2023 03:01
@LZHgrla LZHgrla merged commit e840b6c into InternLM:main Jul 14, 2023
@LZHgrla LZHgrla deleted the lzh/add_mmlu branch July 21, 2023 07:10
HIT-cwh added a commit that referenced this pull request Aug 28, 2023
@HIT-cwh @LZHgrla
[Doc] Add dataset pipeline doc & Improve Doc (#38)
569d916 
* add dataset pipeline doc

* add dataset pipeline doc

* fix bugs

* fix bugs

* refine doc

* fix bugs

* Update README.md

* Update README.md

* update docs (#1)

* Update README.md

* fix pre-commit

* rename xTuner to XTuner

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* fix pre-commit

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update finetune.md

* Update finetune.md

* Update chat.md

* fix pre-commit

* add zh_cn chat and finetune doc

* Update chat.md

* Update README.md

* del tool_usage

* Update README.md

* Update chat.md

* Update chat.md

* Update README.md

* Update README.md

* Update README_zh-CN.md

* Update README.md

* Update README_zh-CN.md

* fix pre-commit

* Update README_zh-CN.md

* Update README.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* refactor data pipeline doc

* add colorist llama2

* fix incremental pretraining doc

---------

Co-authored-by: LZHgrla <[email protected]>
Co-authored-by: LZHgrla <[email protected]>
@apachemycat apachemycat mentioned this pull request Jun 26, 2024
Open
llkn-2 pushed a commit to llkn-2/xtuner that referenced this pull request Jul 31, 2024
@LZHgrla
add MMLU benchmark val / test (InternLM#1)
12b149b 
* add mmlu dataset configs

* add mmlu metric

* fix bugs

* implement predict for sft model

* remove dummy file

* clean code

* modify prefix for mmlu test

* add test.py

* add mmlu val/test for gunaco config

* use float16 for gunaco

* add METAINFO and add logger
llkn-2 pushed a commit to llkn-2/xtuner that referenced this pull request Jul 31, 2024
@HIT-cwh @LZHgrla
[Doc] Add dataset pipeline doc & Improve Doc (InternLM#38)
e56b62e 
* add dataset pipeline doc

* add dataset pipeline doc

* fix bugs

* fix bugs

* refine doc

* fix bugs

* Update README.md

* Update README.md

* update docs (InternLM#1)

* Update README.md

* fix pre-commit

* rename xTuner to XTuner

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* fix pre-commit

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update chat.md

* Update finetune.md

* Update finetune.md

* Update chat.md

* fix pre-commit

* add zh_cn chat and finetune doc

* Update chat.md

* Update README.md

* del tool_usage

* Update README.md

* Update chat.md

* Update chat.md

* Update README.md

* Update README.md

* Update README_zh-CN.md

* Update README.md

* Update README_zh-CN.md

* fix pre-commit

* Update README_zh-CN.md

* Update README.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* Update README_zh-CN.md

* refactor data pipeline doc

* add colorist llama2

* fix incremental pretraining doc

---------

Co-authored-by: LZHgrla <[email protected]>
Co-authored-by: LZHgrla <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pppppM pppppM Awaiting requested review from pppppM

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@LZHgrla