Commit
·
48d7bf7
1
Parent(s):
2eb2324
Update README.md
Browse files
README.md
CHANGED
|
@@ -10,29 +10,28 @@ base_model:
|
|
| 10 |
- common-pile/comma-v0.1-2t
|
| 11 |
pipeline_tag: text-generation
|
| 12 |
---
|
| 13 |
-
# Munin-7B-Open-pt
|
| 14 |
-
|
| 15 |
-
Munin-7B-open-pt is a 7 billion parameter language model continually pre-trained from [Comma v0.1-2T](https://huggingface.co/common-pile/comma-v0.1-2t) using 30B tokens using a mix of the [DynaWord](https://huggingface.co/datasets/danish-foundation-models/danish-dynaword) and the [Comma v0.1 dataset](https://huggingface.co/datasets/common-pile/comma_v0.1_training_dataset), both comprising only public domain and openly licensed data.
|
| 16 |
|
| 17 |
-
Munin-7B-
|
| 18 |
|
| 19 |
-
|
| 20 |
|
| 21 |
-
|
|
|
|
| 22 |
|
| 23 |
## Training details
|
| 24 |
|
| 25 |
-
Munin-7B-open-pt has been trained using the [maester](https://github.com/rlrs/maester/tree/main/3aca26960eaa1a16250b3feda40303c240ba4ca1) framework developed as part of
|
| 26 |
|
| 27 |
-
The training was performed in three stages, with data mix (open-stageK.py) and maester (open-stageK.toml) configuration files available in each subfolder. The datasets can be created using the create_dataset.py script provided in this repository.
|
| 28 |
|
| 29 |
The characteristics of the three pre-training stages are detailed in the following table:
|
| 30 |
|
| 31 |
| Stage | Batch size | Steps | HF path | Data mix | Comments |
|
| 32 |
|-|-|-|-|-|-|
|
| 33 |
-
| stage1 | 262,144 tok | 37,852| [subfolder="stage1"](https://huggingface.co/danish-foundation-models/munin-7b-open-pt/tree/main/stage1) | 2/3 [
|
| 34 |
-
| stage2 | 524,288 tok | 18,926 | [subfolder="stage2"](https://huggingface.co/danish-foundation-models/munin-7b-open-pt/tree/main/stage2) | 2/3 [
|
| 35 |
-
| stage3 | 524,288 tok | 18,926 | [subfolder="stage3"](https://huggingface.co/danish-foundation-models/munin-7b-open-pt/tree/main/stage3) | 2/3 [
|
|
|
|
| 36 |
|
| 37 |
## Limitations
|
| 38 |
|
|
@@ -41,10 +40,20 @@ It will likely have poor performance on other languages or programming languages
|
|
| 41 |
|
| 42 |
As a base model, Munin-7B-Open-pt has not been aligned for safety and may, for example, reflect social biases present in its training data or potentially provide toxic or harmful information.
|
| 43 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
## Project partners & funding
|
| 45 |
-
|
|
|
|
| 46 |
|
| 47 |
Funding was provided by the [Danish Ministry of Digital Affairs](https://www.english.digmin.dk/) and the [Danish Ministry of Higher Education and Science](https://ufm.dk/en).
|
| 48 |
|
|
|
|
|
|
|
| 49 |
## How to cite
|
|
|
|
| 50 |
Coming soon.
|
|
|
|
| 10 |
- common-pile/comma-v0.1-2t
|
| 11 |
pipeline_tag: text-generation
|
| 12 |
---
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
+
# Munin-7B-Open-pt
|
| 15 |
|
| 16 |
+
Munin-7B-open-pt is a 7 billion parameter language model continually pre-trained from [Comma v0.1-2T](https://huggingface.co/common-pile/comma-v0.1-2t) using 30B tokens utilizing a mix of [Danish Dynaword](https://huggingface.co/datasets/danish-foundation-models/danish-dynaword) and the [Comma v0.1 dataset](https://huggingface.co/datasets/common-pile/comma_v0.1_training_dataset), both comprising only public domain and openly licensed data.
|
| 17 |
|
| 18 |
+
Munin-7B-o
|
| 19 |
+
pen-pt is a base model that can serve as a starting point for fine-tuning and post-training. It has not been instruction-tuned and cannot directly be expected to function as a chat model.
|
| 20 |
|
| 21 |
## Training details
|
| 22 |
|
| 23 |
+
Munin-7B-open-pt has been trained using the [maester](https://github.com/rlrs/maester/tree/main/3aca26960eaa1a16250b3feda40303c240ba4ca1) framework developed as part of [Danish Foundation Models](https://foundationmodels.dk/). All training was performed on a single 8x NVIDIA B200 node (the first of its kind in Denmark) as part of the [SDU UCloud](https://cloud.sdu.dk/) research cloud.
|
| 24 |
|
| 25 |
+
The training was performed in three stages, with data mix (open-stageK.py) and maester (open-stageK.toml) configuration files available in each subfolder. The datasets can be created using the `create_dataset.py` script provided in this repository.
|
| 26 |
|
| 27 |
The characteristics of the three pre-training stages are detailed in the following table:
|
| 28 |
|
| 29 |
| Stage | Batch size | Steps | HF path | Data mix | Comments |
|
| 30 |
|-|-|-|-|-|-|
|
| 31 |
+
| stage1 | 262,144 tok | 37,852| [subfolder="stage1"](https://huggingface.co/danish-foundation-models/munin-7b-open-pt/tree/main/stage1) | 2/3 [Dynaword](https://huggingface.co/datasets/danish-foundation-models/danish-dynaword/tree/9e230b35e31a510e5ab909112ad5bfc9463b2c23); <br> 1/3 [Common-Pile](https://huggingface.co/common-pile/comma_v0.1_training_dataset/5afc546db324e7f39f297ba757c9a60547151e7c) | Excludes depbank, jvj, nordjyllandnews, synne for Dynaword; <br> uses subsets and weighting from [Comma-v0.1-2T](https://huggingface.co/common-pile/comma-v0.1-2t) cooldown phase for Common-Pile ; LR schedule with 1000 steps warmup, constant 1e-5, 1000 steps cooldown |
|
| 32 |
+
| stage2 | 524,288 tok | 18,926 | [subfolder="stage2"](https://huggingface.co/danish-foundation-models/munin-7b-open-pt/tree/main/stage2) | 2/3 [Dynaword](https://huggingface.co/datasets/danish-foundation-models/danish-dynaword/tree/9e230b35e31a510e5ab909112ad5bfc9463b2c23); <br> 1/3 [Common-Pile](https://huggingface.co/common-pile/comma_v0.1_training_dataset/5afc546db324e7f39f297ba757c9a60547151e7c) | Excludes depbank, jvj, nordjyllandnews, synne for Dynaword; <br> uses subsets and weighting from [Comma-v0.1-2T](https://huggingface.co/common-pile/comma-v0.1-2t) cooldown phase for Common-Pile; LR schedule with 500 steps warmup, constant 1e-5, 500 steps cooldown |
|
| 33 |
+
| stage3 | 524,288 tok | 18,926 | [subfolder="stage3"](https://huggingface.co/danish-foundation-models/munin-7b-open-pt/tree/main/stage3) | 2/3 [Dynaword](https://huggingface.co/datasets/danish-foundation-models/danish-dynaword/tree/9e230b35e31a510e5ab909112ad5bfc9463b2c23); <br> 1/3 [Common-Pile](https://huggingface.co/common-pile/comma_v0.1_training_dataset/5afc546db324e7f39f297ba757c9a60547151e7c) | Excludes depbank, jvj, nordjyllandnews, synne for Dynaword; <br> uses subsets and weighting from [Comma-v0.1-2T](https://huggingface.co/common-pile/comma-v0.1-2t) cooldown phase for Common-Pile; LR schedule with 500 steps warmup, square root decay from 1e-5 |
|
| 34 |
+
|
| 35 |
|
| 36 |
## Limitations
|
| 37 |
|
|
|
|
| 40 |
|
| 41 |
As a base model, Munin-7B-Open-pt has not been aligned for safety and may, for example, reflect social biases present in its training data or potentially provide toxic or harmful information.
|
| 42 |
|
| 43 |
+
|
| 44 |
+
## License
|
| 45 |
+
|
| 46 |
+
The model is made available under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) open source license. It may therefore be used, modified, distributed, and sublicensed for any purpose, including commercial use, without the licensee having to release their own derivative works under the same permissive terms, provided that users retain copyright and license notices and document any modifications they make.
|
| 47 |
+
|
| 48 |
+
|
| 49 |
## Project partners & funding
|
| 50 |
+
|
| 51 |
+
The development of Munin-7B-Open-pt was performed in a close collaboration between [Aarhus University](https://chc.au.dk/), the [Alexandra Institute](https://alexandra.dk/), and the [University of Southern Denmark](https://www.sdu.dk/en/forskning/machine-learning) as part of [Danish Foundation Models](https://foundationmodels.dk/).
|
| 52 |
|
| 53 |
Funding was provided by the [Danish Ministry of Digital Affairs](https://www.english.digmin.dk/) and the [Danish Ministry of Higher Education and Science](https://ufm.dk/en).
|
| 54 |
|
| 55 |
+
|
| 56 |
+
|
| 57 |
## How to cite
|
| 58 |
+
|
| 59 |
Coming soon.
|