tutorials/modalities_in_15_mins/modalities_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![a](res/banner.jpg)\n",
    "\n",
    "<h1 style=\"text-align: center;\">Getting into Modalities in 15mins</h1>\n",
    "\n",
    "<hr/>\n",
    "\n",
    "**Let's train a dense model with Modalities involving the following steps:**\n",
    "\n",
    "1. Data Preprocessing (Indexation, Tokenization)\n",
    "2. Model Pretraining (GPT Model)\n",
    "3. Monitoring (Weights&Biases)\n",
    "\n",
    "\n",
    "**Folder structure:**\n",
    "\n",
    "Throughout the tutorial, we will use the Jupyter Notebook `modalities_demo.ipynb` to guide us through the process. The notebook is located in the root directory of the tutorial, along with the `configs` and `data` directories. The `configs` directory contains configuration files for the model pretraining and tokenization, while the `data` directory contains subdirectories for storing checkpoints, preprocessed data, raw data, and tokenizer-related files.\n",
    "\n",
    "```text\n",
    "└── getting_started_15mins                 # Root directory for the tutorial\n",
    "    ├── modalities_demo.ipynb              # Jupyter Notebook used for the tutorial.\n",
    "    ├── configs                      \n",
    "    │   ├── pretraining_config.yaml        # Config file for the model pretraining\n",
    "    │   └── tokenization_config.yaml       # Config file for tokenization\n",
    "    └── data                         \n",
    "        ├── checkpoints                    # Dir where model and optimizer checkpoints  are stored.\n",
    "        │   └── <checkpoints>        \n",
    "        ├── preprocessed                   # Dir containing preprocessed training and eval data.\n",
    "        │   └── <files>              \n",
    "        ├── raw                      \n",
    "        │   └── fineweb_edu_num_docs_483606.jsonl   # JSONL file with raw data for training and eval.\n",
    "        └── tokenizer                \n",
    "            ├── tokenizer.json             # JSON file defining the tokenizer model.\n",
    "            └── tokenizer_config.json      # Config file specifying all tokenizer settings\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepraration steps\n",
    "\n",
    "Firstly, we need to install Modalities via pip\n",
    "\n",
    "```bash\n",
    "pip install modalities\n",
    "```\n",
    "\n",
    "and download the raw training data. \n",
    "We are going to use a  subset (500k documents) of the FineWeb-Edu dataset, as it is already cleaned, filtered and deduplicated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-09-08 22:57:37--  https://huggingface.co/datasets/ModalitiesTeam/FW_EDU_SUBSET_500k_docs/resolve/main/fineweb_edu_num_docs_483606.jsonl?download=true\n",
      "Resolving huggingface.co (huggingface.co)... 3.160.39.87, 3.160.39.100, 3.160.39.99, ...\n",
      "Connecting to huggingface.co (huggingface.co)|3.160.39.87|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://cdn-lfs-us-1.huggingface.co/repos/ba/a7/baa78bea1dc6ab4d3f8662c658f843bb98cc6e927f41f08161e3d8bde8bedb92/c26caf15c1d6b76ea9eca2e5fb3f931d2703fe3e4062f84d6542e6844a074079?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27fineweb_edu_num_docs_483606.jsonl%3B+filename%3D%22fineweb_edu_num_docs_483606.jsonl%22%3B&Expires=1726088257&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyNjA4ODI1N319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2JhL2E3L2JhYTc4YmVhMWRjNmFiNGQzZjg2NjJjNjU4Zjg0M2JiOThjYzZlOTI3ZjQxZjA4MTYxZTNkOGJkZThiZWRiOTIvYzI2Y2FmMTVjMWQ2Yjc2ZWE5ZWNhMmU1ZmIzZjkzMWQyNzAzZmUzZTQwNjJmODRkNjU0MmU2ODQ0YTA3NDA3OT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=j030rEo5Xob0mfCQHZekg2o7dx5YwWoXXgMYdKfJ3S-FOf%7EbiHXbXHn2bctN7bdIhMEp9Ps%7EsNXuw5NbUNpP4neRkmWFRUpyuSJXJwzZL4Zg3-RzMD0OkJ7FLA3Lr3X6OCKDvtg4sb6vIxKS6kqcGzq0c8KpV49QvAOUZa7Uo2e%7E8eZkcILrYE9ZKUI2dMX5ZgVxrorLjXAjkVV69PMU4p8XFWKO59y6N%7EZ%7E4tUDNbANIEJyCoFB6yolnQSMZxFzY646%7EtG3cSvwqPewkmBw7mFgcD0sORBrHyJyx7JOafTWHZbcPMyRn3Gd67G87LoECwd6jzRUim7J4qR8DiqDLA__&Key-Pair-Id=K24J24Z295AEI9 [following]\n",
      "--2024-09-08 22:57:37--  https://cdn-lfs-us-1.huggingface.co/repos/ba/a7/baa78bea1dc6ab4d3f8662c658f843bb98cc6e927f41f08161e3d8bde8bedb92/c26caf15c1d6b76ea9eca2e5fb3f931d2703fe3e4062f84d6542e6844a074079?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27fineweb_edu_num_docs_483606.jsonl%3B+filename%3D%22fineweb_edu_num_docs_483606.jsonl%22%3B&Expires=1726088257&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyNjA4ODI1N319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2JhL2E3L2JhYTc4YmVhMWRjNmFiNGQzZjg2NjJjNjU4Zjg0M2JiOThjYzZlOTI3ZjQxZjA4MTYxZTNkOGJkZThiZWRiOTIvYzI2Y2FmMTVjMWQ2Yjc2ZWE5ZWNhMmU1ZmIzZjkzMWQyNzAzZmUzZTQwNjJmODRkNjU0MmU2ODQ0YTA3NDA3OT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=j030rEo5Xob0mfCQHZekg2o7dx5YwWoXXgMYdKfJ3S-FOf%7EbiHXbXHn2bctN7bdIhMEp9Ps%7EsNXuw5NbUNpP4neRkmWFRUpyuSJXJwzZL4Zg3-RzMD0OkJ7FLA3Lr3X6OCKDvtg4sb6vIxKS6kqcGzq0c8KpV49QvAOUZa7Uo2e%7E8eZkcILrYE9ZKUI2dMX5ZgVxrorLjXAjkVV69PMU4p8XFWKO59y6N%7EZ%7E4tUDNbANIEJyCoFB6yolnQSMZxFzY646%7EtG3cSvwqPewkmBw7mFgcD0sORBrHyJyx7JOafTWHZbcPMyRn3Gd67G87LoECwd6jzRUim7J4qR8DiqDLA__&Key-Pair-Id=K24J24Z295AEI9\n",
      "Resolving cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)... 3.160.39.93, 3.160.39.128, 3.160.39.127, ...\n",
      "Connecting to cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)|3.160.39.93|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2543881064 (2,4G) [binary/octet-stream]\n",
      "Saving to: ‘fineweb_edu_num_docs_483606.jsonl’\n",
      "\n",
      "fineweb_edu_num_doc 100%[===================>]   2,37G   271MB/s    in 8,4s    \n",
      "\n",
      "2024-09-08 22:57:46 (289 MB/s) - ‘fineweb_edu_num_docs_483606.jsonl’ saved [2543881064/2543881064]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!cd data/raw && wget https://huggingface.co/datasets/ModalitiesTeam/FW_EDU_SUBSET_500k_docs/resolve/main/fineweb_edu_num_docs_483606.jsonl?download=true -O fineweb_edu_num_docs_483606.jsonl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Disclaimer:**\n",
    "\n",
    "Don't run modalities in jupyter notebooks!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "But this time for demonstration purposes:\n",
    "\n",
    "<img src=\"res/notebooks_1.png\" alt=\"Alt text\" style=\"width:30%;\"/>\n",
    "\n",
    "<small> credits: Joel Grus - I don't like Notebooks</small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preprocessing\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before training the model, we will preprocess the raw data. In the first step, we will create an index of the data that stores the starting byte position and byte length of every document. The index will be used to efficiently index the JSONL file during the tokenization in the second step. \n",
    "\n",
    "The raw JSONL dataset and has the following properties:\n",
    "\n",
    "* Subset of FineWeb-Edu (~500k documents) encoded as JSONL file\n",
    "* already cleaned, filtered and deduplicated\n",
    "\n",
    "Each line in the JSONL is a proper JSON object containing a single document. \n",
    "```json\n",
    "{\n",
    "   \"text\":\"What is the difference between 50 Ohm and 75 Ohm Coax? [...]\",\n",
    "   \"id\":\"<urn:uuid:57e09efe-1c29-49f8-a086-e1bb5dd552c9>\",\n",
    "   \"dump\":\"CC-MAIN-2021-39\",\n",
    "   \"url\":\"http://cablesondemandblog.com/wordpress1/2014/03/\",\n",
    "   \"file_path\":\"s3://commoncrawl/crawl-data/[...]20210918002307-00380.warc.gz\",\n",
    "   \"language\":\"en\",\n",
    "   \"language_score\":0.9309850335121155,\n",
    "   \"token_count\":2355,\n",
    "   \"score\":3.625,\n",
    "   \"int_score\":4\n",
    "}\n",
    "```\n",
    "\n",
    "While the meta data is generally interesing and can be used to further filter the dataset, we are only interested in the text field for now, providing us with the actual training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexation\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of the indexation process is to determine the starting byte position and length of each document in the raw data file.\n",
    "\n",
    "Architecturally, as shown in the diagram below, a reader process reads the raw data file line by line and writes the starting byte position and length of each document to the queue. For each line in the queue, the processor first validates the JSON object and then writes the starting byte position and length of the document to the index file.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"res/modalities_indexation_bright.svg\" alt=\"Alt text\" style=\"width:80%;\"/>\n",
    "\n",
    "We run the indexation with the command shown below. \n",
    "\n",
    "The `modalities data create_raw_index` command triggers the process of creating the index from the raw data.\n",
    "The `--index_path argument` specifies the location where the generated index file will be saved. In this example, the index will be stored at `data/preprocessed/fineweb_edu_num_docs_483606.idx`.\n",
    "The last part, i.e., `data/raw/fineweb_edu_num_docs_483606.jsonl` is the input file in JSONL (JSON Lines) format containing the raw data. The command will process this file to create the index.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reading raw data from data/raw/fineweb_edu_num_docs_483606.jsonl\n",
      "writing index to data/preprocessed/fineweb_edu_num_docs_483606.idx\n",
      "Processed Lines: 483606it [00:12, 39021.78it/s]\n",
      "Created index of length 483606\n"
     ]
    }
   ],
   "source": [
    "!modalities data create_raw_index --index_path data/preprocessed/fineweb_edu_num_docs_483606.idx \\\n",
    "                                               data/raw/fineweb_edu_num_docs_483606.jsonl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Throughput optimized tokenization\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the we have the raw JSONL dataset indexed, we can proceed with the tokenization. \n",
    "\n",
    "In Modalities, tokenization is the process of converting raw text data into a sequence of tokens that can be used as input to the model. This is achieved by scaling up the number of processors performing the tokenization on batches of documents in parallel, as shown in the diagram below. Typically, we use one processor per CPU core to maximize throughput and adapt the queue sizes and batches sizes for optimal throughput. \n",
    "\n",
    "The processors place the tokenized documents as byte streams in the queue from which the writer reads and writes the tokenized documents to the output file.\n",
    "\n",
    "<img src=\"res/modalities_tokenization_bright.svg\" alt=\"Alt text\" style=\"width:100%;\"/>\n",
    "\n",
    "The tokenized dataset file is heavily optimized for efficient indexing. As layed out in the diagram below, the header specifies the size of the data segment and size of a single token in bytes. With this information at hand, the file format is self-contained and does not need any additional information to be read. The data segment contains the concatenated byte streams of the tokenized documents.\n",
    "The documents are indexed by their starting byte position and length stored in the index segment. This allows for efficient random access to the tokenized documents in O(1) time complexity.\n",
    "\n",
    "Additionally, the shuffling of the data can be performed independently of the actual documents, as only the index can be shuffled which has a much lower memory-footprint. Internally, we implemented a numpy array-like view on top of the data segment. \n",
    "\n",
    "<img src=\"res/modalities_file_format_bright.svg\" alt=\"Alt text\" style=\"width:70%;\"/>\n",
    "\n",
    "\n",
    "We define the tokenization config as printed out below. It defines the tokenizer component including all the necessary settings to make it fully reproducible. Under settings we additionally define the performance optimization settings, such as number of CPUs to use and queue sizes, as well as, the input and output file paths.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display\n",
    "\n",
    "def display_markdown(file_path):\n",
    "    with open(file_path, 'r') as file:\n",
    "        code = file.read()\n",
    "    display(Markdown(f'```yaml\\n{code}\\n```'))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "```yaml\n",
       "settings:\n",
       "  src_path: data/raw/fineweb_edu_num_docs_483606.jsonl\n",
       "  dst_path: data/preprocessed/fineweb_edu_num_docs_483606.pbin\n",
       "  index_path: data/preprocessed/fineweb_edu_num_docs_483606.idx\n",
       "  jq_pattern: .text\n",
       "  num_cpus: ${node_env:num_cpus}\n",
       "  eod_token: <|endoftext|>\n",
       "  processing_batch_size: 10\n",
       "  raw_samples_queue_size: 300\n",
       "  processed_samples_queue_size: 300\n",
       "\n",
       "tokenizer:\n",
       "  component_key: tokenizer\n",
       "  variant_key: pretrained_hf_tokenizer\n",
       "  config:\n",
       "    pretrained_model_name_or_path: data/tokenizer\n",
       "    padding: false\n",
       "    truncation: false\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenization_config_path = \"configs/tokenization_config.yaml\"\n",
    "display_markdown(tokenization_config_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Instantiated <class 'modalities.tokenization.tokenizer_wrapper.PreTrainedHFTokenizer'>: tokenizer\n",
      "Processed batches:   0%|                             | 0/483606 [00:00<?, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1644 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2355 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1045 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1286 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3313 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1263 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1645 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1991 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1473 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1747 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1202 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1498 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1360 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1112 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1217 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (14596 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2481 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1123 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1234 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1359 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1155 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1640 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1234 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1042 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1476 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1138 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches:   0%|                 | 210/483606 [00:00<03:53, 2069.06it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (3360 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1514 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3551 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (15452 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1075 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1538 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2761 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1100 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3021 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2928 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (13706 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1497 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1613 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4078 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1117 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3580 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1497 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1031 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2106 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4242 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1589 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2719 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1732 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1059 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1563 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1446 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1231 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1799 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1646 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1586 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches:   0%|                 | 880/483606 [00:00<01:43, 4655.25it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1855 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1183 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2511 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1294 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1459 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1586 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2185 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2238 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2707 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2003 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1083 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3262 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1841 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (7199 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4562 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1634 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2071 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (10222 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2820 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1231 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2927 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (13424 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1285 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (7559 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1766 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (5839 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches:   1%|▏              | 4310/483606 [00:00<00:31, 15224.54it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1460 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1350 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2404 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1302 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1107 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1696 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1085 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1256 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2044 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1618 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2987 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3304 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1178 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (7291 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1532 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1538 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2753 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1070 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1154 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches:   2%|▏              | 7820/483606 [00:00<00:25, 18727.89it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1401 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1797 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1951 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2061 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1230 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1161 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3140 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3368 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1311 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (5465 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1692 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1468 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1275 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1179 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1417 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1790 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3828 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1052 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (5471 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2351 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4605 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1085 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2849 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1387 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1459 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1352 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1208 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches:   2%|▎              | 9560/483606 [00:00<00:37, 12606.68it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1261 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1546 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1775 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1731 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1712 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1530 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1092 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1689 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4103 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches:   2%|▎             | 11160/483606 [00:00<00:37, 12685.57it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1968 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2429 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2339 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1782 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1673 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1584 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (9074 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1304 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1446 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2225 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1251 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1443 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2480 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1294 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1202 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1226 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches:   5%|▊             | 26000/483606 [00:01<00:12, 35611.57it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1241 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1062 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1099 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1358 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1816 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2098 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1476 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1115 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches:   7%|▉             | 34110/483606 [00:01<00:10, 42200.80it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1167 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4104 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1115 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1729 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1515 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1996 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (5550 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1551 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1156 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1264 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1267 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4679 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1781 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1828 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1145 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches:   8%|█             | 38250/483606 [00:01<00:13, 32230.68it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1542 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1070 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1055 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (6228 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2843 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2381 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2647 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1076 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (7932 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (5034 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1309 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4327 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1255 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1257 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1188 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1216 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1408 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3032 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1432 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2144 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches:   9%|█▏            | 41640/483606 [00:01<00:15, 28644.10it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (1031 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1974 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (7133 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1612 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1358 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (3425 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1517 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1455 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1132 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1066 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (11885 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4068 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1568 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (4547 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2496 > 1024). Running this sequence through the model will result in indexing errors\n",
      "Processed batches: 100%|█████████████| 483606/483606 [00:15<00:00, 31210.64it/s]\n"
     ]
    }
   ],
   "source": [
    "!modalities data pack_encoded_data configs/tokenization_config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Modalities, we scale up the training via Fully Sharded Data Parallel (FSDP), as defined in the paper [Zhao, Yanli, et al. \"Pytorch fsdp: experiences on scaling fully sharded data parallel.\" arXiv preprint arXiv:2304.11277 (2023).](https://arxiv.org/pdf/2304.11277)\n",
    "\n",
    "**Goal:** Maximizing the token throughput during training by trading off communication overhead for a lower memory footprint. \n",
    "\n",
    "* Before training model is split into FSDP units and each FSDP unit is sharded across all ranks\n",
    "* Each rank is a data parallel process receiving only a subset of the data\n",
    "* Each rank materializes one FSDP unit at a time during the forward pass by receving the sharded weights from its peers\n",
    "\n",
    "<img src=\"res/fsdp_bright.svg\" alt=\"Alt text\" style=\"width:90%;\"/>\n",
    "\n",
    "\n",
    "adopted from Zhao, Yanli, et al. \"Pytorch fsdp: experiences on scaling fully sharded data parallel.\" arXiv preprint arXiv:2304.11277 (2023)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While FSDP happens under the hood of Modalities the user can still parameterize the training process via the `pretraining_config.yaml` file. In fact, the training config is specified in a way that every component during training, e.g, dataset, dataloader, model, etc., are fully reproducible. On the one hand, this leads to larger, somewhat more complex config files, however it also allows to fully reproduce the training process. Especially in the field of LLMs, where the training process is expensive, complex and involves excessive amounts of ablations, this is a crucial feature to keep track of the entire configuration of the system in a reproducible manner. \n",
    "\n",
    "The config file is shown in the print out below. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenization_config_path = \"configs/pretraining_config.yaml\"\n",
    "display_markdown(tokenization_config_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below you find the command for running the distributed training with modalities across multiple 4 GPUs on a single node. Let's break it down into its components:\n",
    "\n",
    "* `CUDA_VISIBLE_DEVICES=0,1,2,3`: This environment variable specifies which GPUs will be used for the job. In this case, GPUs with IDs 0, 1, 2, 3 are selected for training.\n",
    "\n",
    "* `torchrun`: This is a utility from PyTorch used to launch distributed training. It automatically manages multiple processes for distributed training.\n",
    "\n",
    "* `--rdzv-endpoint localhost:29515`: Specifies the rendezvous endpoint. Here, localhost is the machine's address, and 29515 is the port. The rendezvous endpoint coordinates the processes involved in distributed training.\n",
    "\n",
    "* `--nnodes 1`: Specifies the number of nodes to be used in the distributed setup. Since this is a single-node setup, 1 is used.\n",
    "\n",
    "* `--nproc_per_node 4`: This argument tells torchrun how many processes to launch on each node. In this case, 4 processes are launched per node, corresponding to the 4 GPUs (IDs 0, 1, 2, 3) specified by `CUDA_VISIBLE_DEVICES`.\n",
    "\n",
    "* `$(which modalities) run`: This part dynamically finds the path to the modalities executable and runs it. The run command triggers the main process to start the training.\n",
    "\n",
    "* `--config_file_path configs/pretraining_config.yaml`: The `--config_file_path` argument provides the path to the configuration file for the training job. In this example, the configuration is provided in `configs/pretraining_config.yaml`, which includes settings like model architecture, optimizer, dataset, dataloader and other training components.\n",
    "\n",
    "\n",
    "Once executed, the training process will start, and you will see the training logs in the terminal. The logs will include information about the training progress, such as the loss values, learning rate, and other metrics. Additionally, you can monitor the training process using Weights & Biases, which modalities automatically logs. Make sure that you are logged into your Weights & Biases account to track the training metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W0908 22:59:08.277000 140048090269504 torch/distributed/run.py:757] \n",
      "W0908 22:59:08.277000 140048090269504 torch/distributed/run.py:757] *****************************************\n",
      "W0908 22:59:08.277000 140048090269504 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. \n",
      "W0908 22:59:08.277000 140048090269504 torch/distributed/run.py:757] *****************************************\n",
      "Instantiated <class 'modalities.models.components.layer_norms.RMSLayerNorm'>: model_raw -> config -> attention_norm\n",
      "Instantiated <class 'modalities.models.components.layer_norms.RMSLayerNorm'>: model_raw -> config -> ffn_norm\n",
      "Instantiated <class 'modalities.models.components.layer_norms.RMSLayerNorm'>: model_raw -> config -> lm_head_norm\n",
      "Instantiated <class 'modalities.models.gpt2.gpt2_model.GPT2LLM'>: model_raw\n",
      "Instantiated <class 'modalities.nn.model_initialization.composed_initialization.ModelInitializerWrapper'>: model -> config -> model_initializer\n",
      "Instantiated <class 'modalities.models.gpt2.gpt2_model.GPT2LLM'>: model\n",
      "Unsharded number of parameters on rank 0: 6934784\n",
      "\n",
      "Wrapped layer classes: [<class 'modalities.models.gpt2.gpt2_model.GPT2Block'>]\n",
      "\n",
      "Unsharded number of parameters on rank 1: 6934784\n",
      "Unsharded number of parameters on rank 2: 6934784\n",
      "Unsharded number of parameters on rank 3: 6934784\n",
      "Sharded number of parameters on rank 1:1733696\n",
      "Sharded number of parameters on rank 2:1733696\n",
      "Sharded number of parameters on rank 0:1733696\n",
      "Instantiated <class 'torch.distributed.fsdp.fully_sharded_data_parallel.FullyShardedDataParallel'>: wrapped_model\n",
      "=> optimizer groups:\n",
      "linear (28 modules with 122,880 parameters): weight_decay = 0.1\n",
      "embedding (1 modules with 1,609,792 parameters): weight_decay = 0.0\n",
      "layernorm (10 modules with 1,024 parameters): weight_decay = 0.0\n",
      "=> all (39 modules with 1,733,696 parameters)\n",
      "Instantiated <class 'torch.optim.adamw.AdamW'>: optimizer\n",
      "Instantiated <class 'torch.optim.lr_scheduler.OneCycleLR'>: scheduler\n",
      "Instantiated <class 'modalities.loss_functions.CLMCrossEntropyLoss'>: loss_fn\n",
      "Sharded number of parameters on rank 3:1733696\n",
      "Instantiated <class 'modalities.dataloader.dataset.PackedMemMapDatasetContinuous'>: train_dataset\n",
      "Instantiated <class 'torch.utils.data.distributed.DistributedSampler'>: train_dataloader -> config -> batch_sampler -> config -> sampler\n",
      "Instantiated <class 'torch.utils.data.sampler.BatchSampler'>: train_dataloader -> config -> batch_sampler\n",
      "Instantiated <class 'modalities.models.gpt2.collator.GPT2LLMCollateFn'>: collate_fn\n",
      "Instantiated <class 'modalities.dataloader.dataloader.LLMDataLoader'>: train_dataloader\n",
      "Instantiated <class 'modalities.logging_broker.subscriber_impl.batch_progress_subscriber.DummyProgressSubscriber'>: batch_progress_subscriber\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mmichael-fromm\u001b[0m (\u001b[33mmodalities\u001b[0m). Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: wandb version 0.17.9 is available!  To upgrade, please run:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:  $ pip install wandb --upgrade\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Tracking run with wandb version 0.16.6\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run data is saved locally in \u001b[35m\u001b[1m/raid/fhgiais/opengptx/max_lue/modalities/examples/modalities_in_15_mins/wandb_storage/wandb/run-20240908_225919-lahhdqwg\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run \u001b[1m`wandb offline`\u001b[0m to turn off syncing.\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Syncing run \u001b[33m2024-09-08__22-59-10_7d9fc15e\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: ⭐️ View project at \u001b[34m\u001b[4mhttps://wandb.ai/modalities/ai_24_demo\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: 🚀 View run at \u001b[34m\u001b[4mhttps://wandb.ai/modalities/ai_24_demo/runs/lahhdqwg\u001b[0m\n",
      "Instantiated <class 'modalities.logging_broker.subscriber_impl.results_subscriber.WandBEvaluationResultSubscriber'>: evaluation_subscriber\n",
      "Instantiated <class 'modalities.checkpointing.checkpoint_saving_strategies.SaveKMostRecentCheckpointsStrategy'>: checkpoint_saving -> config -> checkpoint_saving_strategy\n",
      "Instantiated <class 'function'>: checkpoint_saving -> config -> checkpoint_saving_execution -> config -> get_num_tokens_from_num_steps_callable\n",
      "Instantiated <class 'modalities.checkpointing.fsdp.fsdp_checkpoint_saving.FSDPCheckpointSaving'>: checkpoint_saving -> config -> checkpoint_saving_execution\n",
      "Instantiated <class 'modalities.checkpointing.checkpoint_saving.CheckpointSaving'>: checkpoint_saving\n",
      "Instantiated <class 'modalities.training.gradient_clipping.fsdp_gradient_clipper.FSDPGradientClipper'>: gradient_clipper\n",
      "Initialize Model at 2024-09-08 22:59:21.967533.\n",
      "Model initialized at 2024-09-08 22:59:21.970057.\n",
      "Start model training at 2024-09-08 22:59:21.970135.\n",
      "step: 5 | train samples/s: 1793.7 | train mfu: 0.0 | lr mean: 0.0003768849710468203 | train loss avg: 10.81 | train loss last: 10.75 | consumed tokens: 327680.0 | grad norm avg: 2.62 | grad norm last: 2.63 | \n",
      "step: 10 | train samples/s: 2883.1 | train mfu: 0.01 | lr mean: 0.0005999986315146089 | train loss avg: 10.52 | train loss last: 10.38 | consumed tokens: 655360.0 | grad norm avg: 1.41 | grad norm last: 0.99 | \n",
      "step: 15 | train samples/s: 13628.7 | train mfu: 0.04 | lr mean: 0.0005999510758556426 | train loss avg: 10.13 | train loss last: 9.94 | consumed tokens: 983040.0 | grad norm avg: 1.0 | grad norm last: 1.0 | \n",
      "step: 20 | train samples/s: 13796.0 | train mfu: 0.04 | lr mean: 0.0005998355336487293 | train loss avg: 9.68 | train loss last: 9.5 | consumed tokens: 1310720.0 | grad norm avg: 1.0 | grad norm last: 1.01 | \n",
      "step: 25 | train samples/s: 13558.8 | train mfu: 0.04 | lr mean: 0.0005996520631015301 | train loss avg: 9.28 | train loss last: 9.14 | consumed tokens: 1638400.0 | grad norm avg: 1.03 | grad norm last: 1.01 | \n",
      "step: 30 | train samples/s: 13801.5 | train mfu: 0.04 | lr mean: 0.0005994007224217057 | train loss avg: 8.92 | train loss last: 8.78 | consumed tokens: 1966080.0 | grad norm avg: 1.01 | grad norm last: 0.99 | \n",
      "step: 35 | train samples/s: 13562.7 | train mfu: 0.04 | lr mean: 0.0005990815116092563 | train loss avg: 8.61 | train loss last: 8.5 | consumed tokens: 2293760.0 | grad norm avg: 1.0 | grad norm last: 1.0 | \n",
      "step: 40 | train samples/s: 13821.2 | train mfu: 0.04 | lr mean: 0.0005986946052871644 | train loss avg: 8.35 | train loss last: 8.25 | consumed tokens: 2621440.0 | grad norm avg: 0.98 | grad norm last: 0.98 | \n",
      "step: 45 | train samples/s: 13797.7 | train mfu: 0.04 | lr mean: 0.0005982400616630912 | train loss avg: 8.13 | train loss last: 8.08 | consumed tokens: 2949120.0 | grad norm avg: 0.94 | grad norm last: 0.91 | \n",
      "step: 50 | train samples/s: 13239.3 | train mfu: 0.04 | lr mean: 0.0005977179971523583 | train loss avg: 7.95 | train loss last: 7.86 | consumed tokens: 3276800.0 | grad norm avg: 0.88 | grad norm last: 0.88 | \n",
      "step: 55 | train samples/s: 12865.4 | train mfu: 0.04 | lr mean: 0.0005971285281702876 | train loss avg: 7.8 | train loss last: 7.74 | consumed tokens: 3604480.0 | grad norm avg: 0.88 | grad norm last: 0.8 | \n",
      "step: 60 | train samples/s: 13756.9 | train mfu: 0.04 | lr mean: 0.000596471771132201 | train loss avg: 7.72 | train loss last: 7.67 | consumed tokens: 3932160.0 | grad norm avg: 0.73 | grad norm last: 0.64 | \n",
      "step: 65 | train samples/s: 13824.8 | train mfu: 0.04 | lr mean: 0.000595747958868742 | train loss avg: 7.59 | train loss last: 7.53 | consumed tokens: 4259840.0 | grad norm avg: 0.6 | grad norm last: 0.57 | \n",
      "step: 70 | train samples/s: 13805.8 | train mfu: 0.04 | lr mean: 0.0005949572660028934 | train loss avg: 7.5 | train loss last: 7.52 | consumed tokens: 4587520.0 | grad norm avg: 0.84 | grad norm last: 0.67 | \n",
      "step: 75 | train samples/s: 13744.5 | train mfu: 0.04 | lr mean: 0.0005940998671576381 | train loss avg: 7.43 | train loss last: 7.41 | consumed tokens: 4915200.0 | grad norm avg: 0.7 | grad norm last: 0.77 | \n",
      "step: 80 | train samples/s: 13622.9 | train mfu: 0.04 | lr mean: 0.0005931759369559586 | train loss avg: 7.37 | train loss last: 7.38 | consumed tokens: 5242880.0 | grad norm avg: 0.54 | grad norm last: 0.43 | \n",
      "step: 85 | train samples/s: 13848.9 | train mfu: 0.04 | lr mean: 0.0005921858246438205 | train loss avg: 7.3 | train loss last: 7.24 | consumed tokens: 5570560.0 | grad norm avg: 0.64 | grad norm last: 0.48 | \n",
      "step: 90 | train samples/s: 13813.5 | train mfu: 0.04 | lr mean: 0.0005911296466365457 | train loss avg: 7.25 | train loss last: 7.23 | consumed tokens: 5898240.0 | grad norm avg: 0.6 | grad norm last: 0.71 | \n",
      "step: 95 | train samples/s: 13841.5 | train mfu: 0.04 | lr mean: 0.0005900077521800995 | train loss avg: 7.18 | train loss last: 7.12 | consumed tokens: 6225920.0 | grad norm avg: 0.94 | grad norm last: 0.9 | \n",
      "step: 100 | train samples/s: 13746.5 | train mfu: 0.04 | lr mean: 0.0005888204323127866 | train loss avg: 7.12 | train loss last: 7.09 | consumed tokens: 6553600.0 | grad norm avg: 0.67 | grad norm last: 0.44 | \n",
      "step: 105 | train samples/s: 13186.2 | train mfu: 0.04 | lr mean: 0.0005875679198652506 | train loss avg: 7.07 | train loss last: 7.04 | consumed tokens: 6881280.0 | grad norm avg: 0.59 | grad norm last: 0.55 | \n",
      "step: 110 | train samples/s: 13775.7 | train mfu: 0.04 | lr mean: 0.000586250564083457 | train loss avg: 7.01 | train loss last: 6.98 | consumed tokens: 7208960.0 | grad norm avg: 0.99 | grad norm last: 2.23 | \n",
      "step: 115 | train samples/s: 13887.7 | train mfu: 0.04 | lr mean: 0.0005848687142133713 | train loss avg: 6.97 | train loss last: 6.95 | consumed tokens: 7536640.0 | grad norm avg: 1.4 | grad norm last: 1.82 | \n",
      "step: 120 | train samples/s: 13642.4 | train mfu: 0.04 | lr mean: 0.000583422661293298 | train loss avg: 6.93 | train loss last: 6.89 | consumed tokens: 7864320.0 | grad norm avg: 0.72 | grad norm last: 0.36 | \n",
      "step: 125 | train samples/s: 13816.9 | train mfu: 0.04 | lr mean: 0.0005819128709845245 | train loss avg: 6.87 | train loss last: 6.83 | consumed tokens: 8192000.0 | grad norm avg: 0.65 | grad norm last: 0.59 | \n",
      "step: 130 | train samples/s: 13792.5 | train mfu: 0.04 | lr mean: 0.0005803396343253553 | train loss avg: 6.84 | train loss last: 6.82 | consumed tokens: 8519680.0 | grad norm avg: 0.51 | grad norm last: 0.4 | \n",
      "step: 135 | train samples/s: 13831.7 | train mfu: 0.04 | lr mean: 0.0005787033587694168 | train loss avg: 6.78 | train loss last: 6.77 | consumed tokens: 8847360.0 | grad norm avg: 0.46 | grad norm last: 0.46 | \n",
      "step: 140 | train samples/s: 13794.7 | train mfu: 0.04 | lr mean: 0.0005770045099779963 | train loss avg: 6.74 | train loss last: 6.67 | consumed tokens: 9175040.0 | grad norm avg: 0.61 | grad norm last: 0.43 | \n",
      "step: 145 | train samples/s: 13884.0 | train mfu: 0.04 | lr mean: 0.0005752434954047203 | train loss avg: 6.73 | train loss last: 6.71 | consumed tokens: 9502720.0 | grad norm avg: 0.69 | grad norm last: 0.62 | \n",
      "step: 150 | train samples/s: 13596.7 | train mfu: 0.04 | lr mean: 0.0005734206642955542 | train loss avg: 6.68 | train loss last: 6.66 | consumed tokens: 9830400.0 | grad norm avg: 0.49 | grad norm last: 0.56 | \n",
      "step: 155 | train samples/s: 13461.4 | train mfu: 0.04 | lr mean: 0.000571536598727107 | train loss avg: 6.66 | train loss last: 6.65 | consumed tokens: 10158080.0 | grad norm avg: 0.55 | grad norm last: 0.45 | \n",
      "step: 160 | train samples/s: 13821.7 | train mfu: 0.04 | lr mean: 0.0005695917643606663 | train loss avg: 6.62 | train loss last: 6.55 | consumed tokens: 10485760.0 | grad norm avg: 0.6 | grad norm last: 0.84 | \n",
      "step: 165 | train samples/s: 13821.7 | train mfu: 0.04 | lr mean: 0.0005675865686498582 | train loss avg: 6.63 | train loss last: 6.62 | consumed tokens: 10813440.0 | grad norm avg: 0.9 | grad norm last: 0.8 | \n",
      "step: 170 | train samples/s: 13840.4 | train mfu: 0.04 | lr mean: 0.0005655215936712921 | train loss avg: 6.59 | train loss last: 6.59 | consumed tokens: 11141120.0 | grad norm avg: 0.66 | grad norm last: 0.64 | \n",
      "step: 175 | train samples/s: 13919.5 | train mfu: 0.04 | lr mean: 0.0005633973050862551 | train loss avg: 6.59 | train loss last: 6.55 | consumed tokens: 11468800.0 | grad norm avg: 0.61 | grad norm last: 0.65 | \n",
      "step: 180 | train samples/s: 13797.6 | train mfu: 0.04 | lr mean: 0.0005612142267636955 | train loss avg: 6.53 | train loss last: 6.53 | consumed tokens: 11796480.0 | grad norm avg: 0.8 | grad norm last: 0.91 | \n",
      "step: 185 | train samples/s: 13868.3 | train mfu: 0.04 | lr mean: 0.0005589729989878833 | train loss avg: 6.54 | train loss last: 6.56 | consumed tokens: 12124160.0 | grad norm avg: 0.78 | grad norm last: 1.0 | \n",
      "step: 190 | train samples/s: 13836.3 | train mfu: 0.04 | lr mean: 0.0005566740874201059 | train loss avg: 6.51 | train loss last: 6.52 | consumed tokens: 12451840.0 | grad norm avg: 0.82 | grad norm last: 0.66 | \n",
      "step: 195 | train samples/s: 13825.8 | train mfu: 0.04 | lr mean: 0.0005543181323446333 | train loss avg: 6.51 | train loss last: 6.48 | consumed tokens: 12779520.0 | grad norm avg: 0.76 | grad norm last: 0.84 | \n",
      "step: 200 | train samples/s: 13770.9 | train mfu: 0.04 | lr mean: 0.0005519057158380747 | train loss avg: 6.47 | train loss last: 6.46 | consumed tokens: 13107200.0 | grad norm avg: 0.65 | grad norm last: 0.58 | \n",
      "step: 205 | train samples/s: 13391.3 | train mfu: 0.04 | lr mean: 0.0005494374199770391 | train loss avg: 6.46 | train loss last: 6.45 | consumed tokens: 13434880.0 | grad norm avg: 0.55 | grad norm last: 0.59 | \n",
      "step: 210 | train samples/s: 13824.3 | train mfu: 0.04 | lr mean: 0.0005469138850457966 | train loss avg: 6.46 | train loss last: 6.5 | consumed tokens: 13762560.0 | grad norm avg: 0.69 | grad norm last: 1.02 | \n",
      "step: 215 | train samples/s: 2879.1 | train mfu: 0.01 | lr mean: 0.0005443357513286173 | train loss avg: 6.43 | train loss last: 6.41 | consumed tokens: 14090240.0 | grad norm avg: 0.76 | grad norm last: 0.98 | \n",
      "step: 220 | train samples/s: 13766.6 | train mfu: 0.04 | lr mean: 0.0005417036591097713 | train loss avg: 6.4 | train loss last: 6.43 | consumed tokens: 14417920.0 | grad norm avg: 0.7 | grad norm last: 0.64 | \n",
      "step: 225 | train samples/s: 13845.4 | train mfu: 0.04 | lr mean: 0.0005390182486735284 | train loss avg: 6.37 | train loss last: 6.4 | consumed tokens: 14745600.0 | grad norm avg: 0.81 | grad norm last: 1.08 | \n",
      "step: 230 | train samples/s: 13876.9 | train mfu: 0.04 | lr mean: 0.0005362802185118198 | train loss avg: 6.34 | train loss last: 6.35 | consumed tokens: 15073280.0 | grad norm avg: 0.72 | grad norm last: 0.69 | \n",
      "step: 235 | train samples/s: 13580.0 | train mfu: 0.04 | lr mean: 0.0005334902671165764 | train loss avg: 6.33 | train loss last: 6.26 | consumed tokens: 15400960.0 | grad norm avg: 0.79 | grad norm last: 1.22 | \n",
      "step: 240 | train samples/s: 13800.3 | train mfu: 0.04 | lr mean: 0.0005306490929797292 | train loss avg: 6.35 | train loss last: 6.35 | consumed tokens: 15728640.0 | grad norm avg: 0.76 | grad norm last: 0.68 | \n",
      "step: 245 | train samples/s: 13872.8 | train mfu: 0.04 | lr mean: 0.000527757394593209 | train loss avg: 6.33 | train loss last: 6.33 | consumed tokens: 16056320.0 | grad norm avg: 0.81 | grad norm last: 0.84 | \n",
      "step: 250 | train samples/s: 13789.3 | train mfu: 0.04 | lr mean: 0.0005248159286566079 | train loss avg: 6.32 | train loss last: 6.3 | consumed tokens: 16384000.0 | grad norm avg: 0.76 | grad norm last: 0.54 | \n",
      "step: 255 | train samples/s: 13263.1 | train mfu: 0.04 | lr mean: 0.0005218254518695176 | train loss avg: 6.32 | train loss last: 6.38 | consumed tokens: 16711680.0 | grad norm avg: 0.88 | grad norm last: 0.93 | \n",
      "step: 260 | train samples/s: 13856.1 | train mfu: 0.04 | lr mean: 0.0005187866045162082 | train loss avg: 6.28 | train loss last: 6.28 | consumed tokens: 17039360.0 | grad norm avg: 0.88 | grad norm last: 0.58 | \n",
      "step: 265 | train samples/s: 13879.0 | train mfu: 0.04 | lr mean: 0.0005157003179192543 | train loss avg: 6.26 | train loss last: 6.24 | consumed tokens: 17367040.0 | grad norm avg: 0.89 | grad norm last: 0.66 | \n",
      "step: 270 | train samples/s: 13868.6 | train mfu: 0.04 | lr mean: 0.000512567232362926 | train loss avg: 6.22 | train loss last: 6.24 | consumed tokens: 17694720.0 | grad norm avg: 0.9 | grad norm last: 1.13 | \n",
      "step: 275 | train samples/s: 13938.9 | train mfu: 0.04 | lr mean: 0.0005093881627544761 | train loss avg: 6.23 | train loss last: 6.21 | consumed tokens: 18022400.0 | grad norm avg: 0.86 | grad norm last: 0.88 | \n",
      "step: 280 | train samples/s: 13875.1 | train mfu: 0.04 | lr mean: 0.0005061639822088182 | train loss avg: 6.21 | train loss last: 6.23 | consumed tokens: 18350080.0 | grad norm avg: 0.7 | grad norm last: 0.77 | \n",
      "step: 285 | train samples/s: 13836.0 | train mfu: 0.04 | lr mean: 0.0005028954474255443 | train loss avg: 6.21 | train loss last: 6.23 | consumed tokens: 18677760.0 | grad norm avg: 0.91 | grad norm last: 1.2 | \n",
      "step: 290 | train samples/s: 13711.7 | train mfu: 0.04 | lr mean: 0.000499583373311907 | train loss avg: 6.18 | train loss last: 6.18 | consumed tokens: 19005440.0 | grad norm avg: 0.67 | grad norm last: 0.62 | \n",
      "step: 295 | train samples/s: 13833.5 | train mfu: 0.04 | lr mean: 0.0004962285747751594 | train loss avg: 6.19 | train loss last: 6.19 | consumed tokens: 19333120.0 | grad norm avg: 0.71 | grad norm last: 0.71 | \n",
      "step: 300 | train samples/s: 13755.4 | train mfu: 0.04 | lr mean: 0.0004928319831378758 | train loss avg: 6.18 | train loss last: 6.2 | consumed tokens: 19660800.0 | grad norm avg: 0.73 | grad norm last: 0.78 | \n",
      "step: 305 | train samples/s: 13241.1 | train mfu: 0.04 | lr mean: 0.0004893943550996482 | train loss avg: 6.18 | train loss last: 6.16 | consumed tokens: 19988480.0 | grad norm avg: 1.03 | grad norm last: 1.05 | \n",
      "step: 310 | train samples/s: 13650.4 | train mfu: 0.04 | lr mean: 0.00048591665108688176 | train loss avg: 6.16 | train loss last: 6.18 | consumed tokens: 20316160.0 | grad norm avg: 0.9 | grad norm last: 1.01 | \n",
      "step: 315 | train samples/s: 13894.9 | train mfu: 0.04 | lr mean: 0.00048239962779916823 | train loss avg: 6.15 | train loss last: 6.14 | consumed tokens: 20643840.0 | grad norm avg: 0.88 | grad norm last: 1.01 | \n",
      "step: 320 | train samples/s: 13879.4 | train mfu: 0.04 | lr mean: 0.00047884430387057364 | train loss avg: 6.14 | train loss last: 6.13 | consumed tokens: 20971520.0 | grad norm avg: 0.94 | grad norm last: 0.93 | \n",
      "step: 325 | train samples/s: 13903.7 | train mfu: 0.04 | lr mean: 0.0004752514942083508 | train loss avg: 6.13 | train loss last: 6.12 | consumed tokens: 21299200.0 | grad norm avg: 0.94 | grad norm last: 0.78 | \n",
      "step: 330 | train samples/s: 13851.5 | train mfu: 0.04 | lr mean: 0.00047162213013507426 | train loss avg: 6.11 | train loss last: 6.09 | consumed tokens: 21626880.0 | grad norm avg: 0.92 | grad norm last: 1.06 | \n",
      "step: 335 | train samples/s: 13694.4 | train mfu: 0.04 | lr mean: 0.0004679570847656578 | train loss avg: 6.11 | train loss last: 6.05 | consumed tokens: 21954560.0 | grad norm avg: 1.0 | grad norm last: 0.94 | \n",
      "step: 340 | train samples/s: 13742.6 | train mfu: 0.04 | lr mean: 0.0004642573185265064 | train loss avg: 6.1 | train loss last: 6.1 | consumed tokens: 22282240.0 | grad norm avg: 0.84 | grad norm last: 0.9 | \n",
      "step: 345 | train samples/s: 13789.4 | train mfu: 0.04 | lr mean: 0.00046052373363636434 | train loss avg: 6.08 | train loss last: 6.06 | consumed tokens: 22609920.0 | grad norm avg: 0.68 | grad norm last: 0.63 | \n",
      "step: 350 | train samples/s: 13809.7 | train mfu: 0.04 | lr mean: 0.0004567572905216366 | train loss avg: 6.09 | train loss last: 6.1 | consumed tokens: 22937600.0 | grad norm avg: 0.83 | grad norm last: 0.93 | \n",
      "step: 355 | train samples/s: 13357.9 | train mfu: 0.04 | lr mean: 0.00045295897871255875 | train loss avg: 6.06 | train loss last: 6.07 | consumed tokens: 23265280.0 | grad norm avg: 0.78 | grad norm last: 0.74 | \n",
      "step: 360 | train samples/s: 13816.9 | train mfu: 0.04 | lr mean: 0.0004491297004278749 | train loss avg: 6.06 | train loss last: 6.07 | consumed tokens: 23592960.0 | grad norm avg: 0.83 | grad norm last: 0.73 | \n",
      "step: 365 | train samples/s: 13858.4 | train mfu: 0.04 | lr mean: 0.0004452704160939902 | train loss avg: 6.06 | train loss last: 6.06 | consumed tokens: 23920640.0 | grad norm avg: 0.75 | grad norm last: 0.65 | \n",
      "step: 370 | train samples/s: 13571.6 | train mfu: 0.04 | lr mean: 0.00044138208613730967 | train loss avg: 6.04 | train loss last: 6.01 | consumed tokens: 24248320.0 | grad norm avg: 0.7 | grad norm last: 0.71 | \n",
      "step: 375 | train samples/s: 13776.8 | train mfu: 0.04 | lr mean: 0.00043746581650339067 | train loss avg: 6.05 | train loss last: 6.05 | consumed tokens: 24576000.0 | grad norm avg: 0.76 | grad norm last: 0.76 | \n",
      "step: 380 | train samples/s: 13825.9 | train mfu: 0.04 | lr mean: 0.00043352239299565554 | train loss avg: 6.04 | train loss last: 6.06 | consumed tokens: 24903680.0 | grad norm avg: 0.7 | grad norm last: 0.61 | \n",
      "step: 385 | train samples/s: 13762.0 | train mfu: 0.04 | lr mean: 0.00042955289245583117 | train loss avg: 6.05 | train loss last: 6.02 | consumed tokens: 25231360.0 | grad norm avg: 0.8 | grad norm last: 0.72 | \n",
      "step: 390 | train samples/s: 13809.3 | train mfu: 0.04 | lr mean: 0.0004255584208294749 | train loss avg: 6.03 | train loss last: 6.03 | consumed tokens: 25559040.0 | grad norm avg: 0.88 | grad norm last: 1.02 | \n",
      "step: 395 | train samples/s: 13852.8 | train mfu: 0.04 | lr mean: 0.0004215398512315005 | train loss avg: 6.04 | train loss last: 6.1 | consumed tokens: 25886720.0 | grad norm avg: 0.89 | grad norm last: 0.96 | \n",
      "step: 400 | train samples/s: 13861.2 | train mfu: 0.04 | lr mean: 0.0004174982605036348 | train loss avg: 6.0 | train loss last: 6.0 | consumed tokens: 26214400.0 | grad norm avg: 0.71 | grad norm last: 0.71 | \n",
      "step: 405 | train samples/s: 13468.9 | train mfu: 0.04 | lr mean: 0.00041343457996845245 | train loss avg: 5.98 | train loss last: 6.02 | consumed tokens: 26542080.0 | grad norm avg: 0.73 | grad norm last: 0.81 | \n",
      "step: 410 | train samples/s: 13798.2 | train mfu: 0.04 | lr mean: 0.00040934994467534125 | train loss avg: 5.98 | train loss last: 5.95 | consumed tokens: 26869760.0 | grad norm avg: 0.78 | grad norm last: 0.62 | \n",
      "step: 415 | train samples/s: 13815.4 | train mfu: 0.04 | lr mean: 0.00040524531505070627 | train loss avg: 5.98 | train loss last: 5.99 | consumed tokens: 27197440.0 | grad norm avg: 0.76 | grad norm last: 0.73 | \n",
      "step: 420 | train samples/s: 13802.3 | train mfu: 0.04 | lr mean: 0.0004011217097286135 | train loss avg: 5.96 | train loss last: 5.95 | consumed tokens: 27525120.0 | grad norm avg: 0.82 | grad norm last: 0.99 | \n",
      "step: 425 | train samples/s: 13549.3 | train mfu: 0.04 | lr mean: 0.0003969802928622812 | train loss avg: 5.96 | train loss last: 5.98 | consumed tokens: 27852800.0 | grad norm avg: 0.79 | grad norm last: 0.82 | \n",
      "step: 430 | train samples/s: 13729.1 | train mfu: 0.04 | lr mean: 0.0003928219375666231 | train loss avg: 5.96 | train loss last: 5.94 | consumed tokens: 28180480.0 | grad norm avg: 0.81 | grad norm last: 0.82 | \n",
      "step: 435 | train samples/s: 13653.0 | train mfu: 0.04 | lr mean: 0.00038864780799485743 | train loss avg: 5.95 | train loss last: 5.92 | consumed tokens: 28508160.0 | grad norm avg: 0.79 | grad norm last: 0.87 | \n",
      "step: 440 | train samples/s: 13762.2 | train mfu: 0.04 | lr mean: 0.00038445889367721975 | train loss avg: 5.94 | train loss last: 5.98 | consumed tokens: 28835840.0 | grad norm avg: 0.86 | grad norm last: 0.91 | \n",
      "step: 445 | train samples/s: 13841.6 | train mfu: 0.04 | lr mean: 0.0003802563005592674 | train loss avg: 5.96 | train loss last: 5.91 | consumed tokens: 29163520.0 | grad norm avg: 0.8 | grad norm last: 0.77 | \n",
      "step: 450 | train samples/s: 13800.1 | train mfu: 0.04 | lr mean: 0.000376040959963575 | train loss avg: 5.94 | train loss last: 5.97 | consumed tokens: 29491200.0 | grad norm avg: 0.82 | grad norm last: 0.83 | \n",
      "step: 455 | train samples/s: 13503.2 | train mfu: 0.04 | lr mean: 0.0003718141233548522 | train loss avg: 5.91 | train loss last: 5.92 | consumed tokens: 29818880.0 | grad norm avg: 0.77 | grad norm last: 0.82 | \n",
      "step: 460 | train samples/s: 13809.2 | train mfu: 0.04 | lr mean: 0.00036757675115950406 | train loss avg: 5.93 | train loss last: 5.95 | consumed tokens: 30146560.0 | grad norm avg: 0.84 | grad norm last: 0.9 | \n",
      "step: 465 | train samples/s: 13897.8 | train mfu: 0.04 | lr mean: 0.000363329891115427 | train loss avg: 5.93 | train loss last: 5.91 | consumed tokens: 30474240.0 | grad norm avg: 0.74 | grad norm last: 0.8 | \n",
      "step: 470 | train samples/s: 13863.7 | train mfu: 0.04 | lr mean: 0.00035907464916817844 | train loss avg: 5.92 | train loss last: 5.94 | consumed tokens: 30801920.0 | grad norm avg: 0.75 | grad norm last: 0.72 | \n",
      "step: 475 | train samples/s: 13828.8 | train mfu: 0.04 | lr mean: 0.0003548121021594852 | train loss avg: 5.92 | train loss last: 5.95 | consumed tokens: 31129600.0 | grad norm avg: 0.76 | grad norm last: 0.66 | \n",
      "step: 480 | train samples/s: 13860.1 | train mfu: 0.04 | lr mean: 0.00035054326872341335 | train loss avg: 5.91 | train loss last: 5.93 | consumed tokens: 31457280.0 | grad norm avg: 0.68 | grad norm last: 0.75 | \n",
      "step: 485 | train samples/s: 13869.6 | train mfu: 0.04 | lr mean: 0.0003462693130131811 | train loss avg: 5.89 | train loss last: 5.91 | consumed tokens: 31784960.0 | grad norm avg: 0.74 | grad norm last: 0.63 | \n",
      "step: 490 | train samples/s: 13802.3 | train mfu: 0.04 | lr mean: 0.000341991224559024 | train loss avg: 5.93 | train loss last: 5.92 | consumed tokens: 32112640.0 | grad norm avg: 0.72 | grad norm last: 0.88 | \n",
      "step: 495 | train samples/s: 13764.6 | train mfu: 0.04 | lr mean: 0.0003377101384103298 | train loss avg: 5.89 | train loss last: 5.86 | consumed tokens: 32440320.0 | grad norm avg: 0.74 | grad norm last: 0.69 | \n",
      "step: 500 | train samples/s: 13815.1 | train mfu: 0.04 | lr mean: 0.00033342710230499506 | train loss avg: 5.9 | train loss last: 5.86 | consumed tokens: 32768000.0 | grad norm avg: 0.83 | grad norm last: 0.9 | \n",
      "step: 505 | train samples/s: 13492.3 | train mfu: 0.04 | lr mean: 0.00032914322218857706 | train loss avg: 5.9 | train loss last: 5.88 | consumed tokens: 33095680.0 | grad norm avg: 0.85 | grad norm last: 0.83 | \n",
      "step: 510 | train samples/s: 13747.7 | train mfu: 0.04 | lr mean: 0.0003248595166951418 | train loss avg: 5.88 | train loss last: 5.9 | consumed tokens: 33423360.0 | grad norm avg: 0.82 | grad norm last: 0.75 | \n",
      "step: 515 | train samples/s: 13876.8 | train mfu: 0.04 | lr mean: 0.00032057714997790754 | train loss avg: 5.85 | train loss last: 5.89 | consumed tokens: 33751040.0 | grad norm avg: 0.78 | grad norm last: 0.88 | \n",
      "step: 520 | train samples/s: 13840.2 | train mfu: 0.04 | lr mean: 0.0003162971115671098 | train loss avg: 5.85 | train loss last: 5.84 | consumed tokens: 34078720.0 | grad norm avg: 0.75 | grad norm last: 0.69 | \n",
      "step: 525 | train samples/s: 13848.1 | train mfu: 0.04 | lr mean: 0.0003120205656159669 | train loss avg: 5.87 | train loss last: 5.89 | consumed tokens: 34406400.0 | grad norm avg: 0.68 | grad norm last: 0.74 | \n",
      "step: 530 | train samples/s: 13883.0 | train mfu: 0.04 | lr mean: 0.0003077485307585448 | train loss avg: 5.87 | train loss last: 5.87 | consumed tokens: 34734080.0 | grad norm avg: 0.69 | grad norm last: 0.67 | \n",
      "step: 535 | train samples/s: 13786.8 | train mfu: 0.04 | lr mean: 0.0003034820838365704 | train loss avg: 5.83 | train loss last: 5.81 | consumed tokens: 35061760.0 | grad norm avg: 0.67 | grad norm last: 0.7 | \n",
      "step: 540 | train samples/s: 13803.9 | train mfu: 0.04 | lr mean: 0.000299222330795601 | train loss avg: 5.87 | train loss last: 5.84 | consumed tokens: 35389440.0 | grad norm avg: 0.75 | grad norm last: 0.7 | \n",
      "step: 545 | train samples/s: 13862.8 | train mfu: 0.04 | lr mean: 0.00029497031937353313 | train loss avg: 5.84 | train loss last: 5.84 | consumed tokens: 35717120.0 | grad norm avg: 0.67 | grad norm last: 0.62 | \n",
      "step: 550 | train samples/s: 13877.4 | train mfu: 0.04 | lr mean: 0.0002907270973082632 | train loss avg: 5.82 | train loss last: 5.77 | consumed tokens: 36044800.0 | grad norm avg: 0.72 | grad norm last: 0.67 | \n",
      "step: 555 | train samples/s: 13521.4 | train mfu: 0.04 | lr mean: 0.000286493799649179 | train loss avg: 5.82 | train loss last: 5.84 | consumed tokens: 36372480.0 | grad norm avg: 0.65 | grad norm last: 0.61 | \n",
      "step: 560 | train samples/s: 13875.1 | train mfu: 0.04 | lr mean: 0.0002822714450303465 | train loss avg: 5.81 | train loss last: 5.81 | consumed tokens: 36700160.0 | grad norm avg: 0.69 | grad norm last: 0.61 | \n",
      "step: 565 | train samples/s: 13932.0 | train mfu: 0.04 | lr mean: 0.0002780611102934927 | train loss avg: 5.82 | train loss last: 5.85 | consumed tokens: 37027840.0 | grad norm avg: 0.72 | grad norm last: 0.64 | \n",
      "step: 570 | train samples/s: 13860.1 | train mfu: 0.04 | lr mean: 0.0002738638431765139 | train loss avg: 5.82 | train loss last: 5.82 | consumed tokens: 37355520.0 | grad norm avg: 0.76 | grad norm last: 0.96 | \n",
      "step: 575 | train samples/s: 13928.2 | train mfu: 0.04 | lr mean: 0.0002696807205211371 | train loss avg: 5.79 | train loss last: 5.75 | consumed tokens: 37683200.0 | grad norm avg: 0.71 | grad norm last: 0.8 | \n",
      "step: 580 | train samples/s: 13352.0 | train mfu: 0.04 | lr mean: 0.0002655127609614283 | train loss avg: 5.81 | train loss last: 5.8 | consumed tokens: 38010880.0 | grad norm avg: 0.71 | grad norm last: 0.67 | \n",
      "step: 585 | train samples/s: 13954.7 | train mfu: 0.04 | lr mean: 0.0002613610413391143 | train loss avg: 5.82 | train loss last: 5.83 | consumed tokens: 38338560.0 | grad norm avg: 0.73 | grad norm last: 0.65 | \n",
      "step: 590 | train samples/s: 13921.9 | train mfu: 0.04 | lr mean: 0.00025722660939209163 | train loss avg: 5.79 | train loss last: 5.78 | consumed tokens: 38666240.0 | grad norm avg: 0.67 | grad norm last: 0.71 | \n",
      "step: 595 | train samples/s: 13930.2 | train mfu: 0.04 | lr mean: 0.0002531105128582567 | train loss avg: 5.81 | train loss last: 5.82 | consumed tokens: 38993920.0 | grad norm avg: 0.7 | grad norm last: 0.72 | \n",
      "step: 600 | train samples/s: 13643.8 | train mfu: 0.04 | lr mean: 0.00024901374126784503 | train loss avg: 5.77 | train loss last: 5.79 | consumed tokens: 39321600.0 | grad norm avg: 0.78 | grad norm last: 0.84 | \n",
      "step: 605 | train samples/s: 13658.6 | train mfu: 0.04 | lr mean: 0.00024493737146258354 | train loss avg: 5.79 | train loss last: 5.76 | consumed tokens: 39649280.0 | grad norm avg: 0.81 | grad norm last: 0.84 | \n",
      "step: 610 | train samples/s: 13908.6 | train mfu: 0.04 | lr mean: 0.00024088243662845343 | train loss avg: 5.78 | train loss last: 5.76 | consumed tokens: 39976960.0 | grad norm avg: 0.81 | grad norm last: 0.8 | \n",
      "step: 615 | train samples/s: 13907.8 | train mfu: 0.04 | lr mean: 0.00023684989719185978 | train loss avg: 5.79 | train loss last: 5.84 | consumed tokens: 40304640.0 | grad norm avg: 0.8 | grad norm last: 0.86 | \n",
      "step: 620 | train samples/s: 13742.2 | train mfu: 0.04 | lr mean: 0.00023284082999452949 | train loss avg: 5.78 | train loss last: 5.79 | consumed tokens: 40632320.0 | grad norm avg: 0.7 | grad norm last: 0.63 | \n",
      "step: 625 | train samples/s: 13528.8 | train mfu: 0.04 | lr mean: 0.00022885622456669807 | train loss avg: 5.76 | train loss last: 5.79 | consumed tokens: 40960000.0 | grad norm avg: 0.71 | grad norm last: 0.67 | \n",
      "step: 630 | train samples/s: 13924.1 | train mfu: 0.04 | lr mean: 0.00022489707043860108 | train loss avg: 5.75 | train loss last: 5.75 | consumed tokens: 41287680.0 | grad norm avg: 0.72 | grad norm last: 0.7 | \n",
      "step: 635 | train samples/s: 13937.5 | train mfu: 0.04 | lr mean: 0.00022096438624430448 | train loss avg: 5.79 | train loss last: 5.8 | consumed tokens: 41615360.0 | grad norm avg: 0.69 | grad norm last: 0.71 | \n",
      "step: 640 | train samples/s: 13909.2 | train mfu: 0.04 | lr mean: 0.00021705914696212858 | train loss avg: 5.77 | train loss last: 5.77 | consumed tokens: 41943040.0 | grad norm avg: 0.68 | grad norm last: 0.57 | \n",
      "step: 645 | train samples/s: 13915.5 | train mfu: 0.04 | lr mean: 0.0002131823421223089 | train loss avg: 5.77 | train loss last: 5.75 | consumed tokens: 42270720.0 | grad norm avg: 0.69 | grad norm last: 0.69 | \n",
      "step: 650 | train samples/s: 13893.6 | train mfu: 0.04 | lr mean: 0.00020933493215125054 | train loss avg: 5.77 | train loss last: 5.77 | consumed tokens: 42598400.0 | grad norm avg: 0.7 | grad norm last: 0.76 | \n",
      "step: 655 | train samples/s: 13380.3 | train mfu: 0.04 | lr mean: 0.000205517906579189 | train loss avg: 5.76 | train loss last: 5.71 | consumed tokens: 42926080.0 | grad norm avg: 0.83 | grad norm last: 0.83 | \n",
      "step: 660 | train samples/s: 13735.9 | train mfu: 0.04 | lr mean: 0.00020173222583252937 | train loss avg: 5.76 | train loss last: 5.76 | consumed tokens: 43253760.0 | grad norm avg: 0.73 | grad norm last: 0.74 | \n",
      "step: 665 | train samples/s: 13887.5 | train mfu: 0.04 | lr mean: 0.0001979788503376767 | train loss avg: 5.75 | train loss last: 5.73 | consumed tokens: 43581440.0 | grad norm avg: 0.73 | grad norm last: 0.8 | \n",
      "step: 670 | train samples/s: 13804.3 | train mfu: 0.04 | lr mean: 0.00019425868231337517 | train loss avg: 5.73 | train loss last: 5.74 | consumed tokens: 43909120.0 | grad norm avg: 0.78 | grad norm last: 0.88 | \n",
      "step: 675 | train samples/s: 13903.6 | train mfu: 0.04 | lr mean: 0.00019057268218602985 | train loss avg: 5.74 | train loss last: 5.76 | consumed tokens: 44236800.0 | grad norm avg: 0.83 | grad norm last: 0.7 | \n",
      "step: 680 | train samples/s: 13887.8 | train mfu: 0.04 | lr mean: 0.00018692179583013058 | train loss avg: 5.74 | train loss last: 5.77 | consumed tokens: 44564480.0 | grad norm avg: 0.78 | grad norm last: 0.76 | \n",
      "step: 685 | train samples/s: 13941.9 | train mfu: 0.04 | lr mean: 0.0001833069254644215 | train loss avg: 5.71 | train loss last: 5.7 | consumed tokens: 44892160.0 | grad norm avg: 0.69 | grad norm last: 0.66 | \n",
      "step: 690 | train samples/s: 13860.9 | train mfu: 0.04 | lr mean: 0.00017972900241147727 | train loss avg: 5.72 | train loss last: 5.66 | consumed tokens: 45219840.0 | grad norm avg: 0.72 | grad norm last: 0.65 | \n",
      "step: 695 | train samples/s: 13948.6 | train mfu: 0.04 | lr mean: 0.00017618888523429632 | train loss avg: 5.73 | train loss last: 5.71 | consumed tokens: 45547520.0 | grad norm avg: 0.65 | grad norm last: 0.65 | \n",
      "step: 700 | train samples/s: 13798.7 | train mfu: 0.04 | lr mean: 0.0001726875052554533 | train loss avg: 5.74 | train loss last: 5.72 | consumed tokens: 45875200.0 | grad norm avg: 0.65 | grad norm last: 0.65 | \n",
      "step: 705 | train samples/s: 13631.3 | train mfu: 0.04 | lr mean: 0.00016922569193411618 | train loss avg: 5.72 | train loss last: 5.78 | consumed tokens: 46202880.0 | grad norm avg: 0.7 | grad norm last: 0.73 | \n",
      "step: 710 | train samples/s: 13879.6 | train mfu: 0.04 | lr mean: 0.00016580440569669008 | train loss avg: 5.72 | train loss last: 5.75 | consumed tokens: 46530560.0 | grad norm avg: 0.67 | grad norm last: 0.68 | \n",
      "step: 715 | train samples/s: 13673.9 | train mfu: 0.04 | lr mean: 0.00016242443234659731 | train loss avg: 5.72 | train loss last: 5.73 | consumed tokens: 46858240.0 | grad norm avg: 0.64 | grad norm last: 0.72 | \n",
      "step: 720 | train samples/s: 13956.2 | train mfu: 0.04 | lr mean: 0.00015908663044683635 | train loss avg: 5.7 | train loss last: 5.69 | consumed tokens: 47185920.0 | grad norm avg: 0.7 | grad norm last: 0.69 | \n",
      "step: 725 | train samples/s: 13931.2 | train mfu: 0.04 | lr mean: 0.0001557918731123209 | train loss avg: 5.72 | train loss last: 5.68 | consumed tokens: 47513600.0 | grad norm avg: 0.75 | grad norm last: 0.74 | \n",
      "step: 730 | train samples/s: 13909.0 | train mfu: 0.04 | lr mean: 0.00015254097525030375 | train loss avg: 5.72 | train loss last: 5.69 | consumed tokens: 47841280.0 | grad norm avg: 0.76 | grad norm last: 0.82 | \n",
      "step: 735 | train samples/s: 13945.5 | train mfu: 0.04 | lr mean: 0.00014933473721612245 | train loss avg: 5.71 | train loss last: 5.73 | consumed tokens: 48168960.0 | grad norm avg: 0.66 | grad norm last: 0.64 | \n",
      "step: 740 | train samples/s: 13686.5 | train mfu: 0.04 | lr mean: 0.00014617398846894503 | train loss avg: 5.69 | train loss last: 5.68 | consumed tokens: 48496640.0 | grad norm avg: 0.68 | grad norm last: 0.73 | \n",
      "step: 745 | train samples/s: 13914.5 | train mfu: 0.04 | lr mean: 0.0001430595148121938 | train loss avg: 5.74 | train loss last: 5.69 | consumed tokens: 48824320.0 | grad norm avg: 0.72 | grad norm last: 0.86 | \n",
      "step: 750 | train samples/s: 13895.2 | train mfu: 0.04 | lr mean: 0.00013999210204929113 | train loss avg: 5.72 | train loss last: 5.73 | consumed tokens: 49152000.0 | grad norm avg: 0.75 | grad norm last: 0.92 | \n",
      "step: 755 | train samples/s: 13500.3 | train mfu: 0.04 | lr mean: 0.0001369725214317441 | train loss avg: 5.71 | train loss last: 5.75 | consumed tokens: 49479680.0 | grad norm avg: 0.65 | grad norm last: 0.56 | \n",
      "step: 760 | train samples/s: 13901.7 | train mfu: 0.04 | lr mean: 0.00013400152965914458 | train loss avg: 5.72 | train loss last: 5.73 | consumed tokens: 49807360.0 | grad norm avg: 0.66 | grad norm last: 0.64 | \n",
      "step: 765 | train samples/s: 13903.4 | train mfu: 0.04 | lr mean: 0.00013107989798299968 | train loss avg: 5.71 | train loss last: 5.72 | consumed tokens: 50135040.0 | grad norm avg: 0.65 | grad norm last: 0.57 | \n",
      "step: 770 | train samples/s: 13923.5 | train mfu: 0.04 | lr mean: 0.00012820832489524037 | train loss avg: 5.72 | train loss last: 5.74 | consumed tokens: 50462720.0 | grad norm avg: 0.6 | grad norm last: 0.54 | \n",
      "step: 775 | train samples/s: 13923.8 | train mfu: 0.04 | lr mean: 0.0001253875670954585 | train loss avg: 5.71 | train loss last: 5.7 | consumed tokens: 50790400.0 | grad norm avg: 0.67 | grad norm last: 0.64 | \n",
      "step: 780 | train samples/s: 13640.0 | train mfu: 0.04 | lr mean: 0.00012261830852366984 | train loss avg: 5.7 | train loss last: 5.71 | consumed tokens: 51118080.0 | grad norm avg: 0.63 | grad norm last: 0.66 | \n",
      "step: 785 | train samples/s: 13949.5 | train mfu: 0.04 | lr mean: 0.00011990125494776294 | train loss avg: 5.67 | train loss last: 5.65 | consumed tokens: 51445760.0 | grad norm avg: 0.62 | grad norm last: 0.66 | \n",
      "step: 790 | train samples/s: 13891.4 | train mfu: 0.04 | lr mean: 0.00011723709758371115 | train loss avg: 5.67 | train loss last: 5.69 | consumed tokens: 51773440.0 | grad norm avg: 0.64 | grad norm last: 0.67 | \n",
      "step: 795 | train samples/s: 13916.9 | train mfu: 0.04 | lr mean: 0.00011462650581961498 | train loss avg: 5.7 | train loss last: 5.7 | consumed tokens: 52101120.0 | grad norm avg: 0.62 | grad norm last: 0.54 | \n",
      "step: 800 | train samples/s: 13937.2 | train mfu: 0.04 | lr mean: 0.00011207012721570209 | train loss avg: 5.67 | train loss last: 5.63 | consumed tokens: 52428800.0 | grad norm avg: 0.67 | grad norm last: 0.59 | \n",
      "step: 805 | train samples/s: 13578.5 | train mfu: 0.04 | lr mean: 0.00010956860933220014 | train loss avg: 5.7 | train loss last: 5.7 | consumed tokens: 52756480.0 | grad norm avg: 0.62 | grad norm last: 0.56 | \n",
      "step: 810 | train samples/s: 13879.2 | train mfu: 0.04 | lr mean: 0.00010712258517742157 | train loss avg: 5.69 | train loss last: 5.69 | consumed tokens: 53084160.0 | grad norm avg: 0.6 | grad norm last: 0.6 | \n",
      "step: 815 | train samples/s: 13809.1 | train mfu: 0.04 | lr mean: 0.00010473267320776358 | train loss avg: 5.72 | train loss last: 5.73 | consumed tokens: 53411840.0 | grad norm avg: 0.62 | grad norm last: 0.6 | \n",
      "step: 820 | train samples/s: 5484.8 | train mfu: 0.02 | lr mean: 0.00010239947005175054 | train loss avg: 5.7 | train loss last: 5.71 | consumed tokens: 53739520.0 | grad norm avg: 0.6 | grad norm last: 0.64 | \n",
      "step: 825 | train samples/s: 13872.4 | train mfu: 0.04 | lr mean: 0.00010012355778599158 | train loss avg: 5.68 | train loss last: 5.62 | consumed tokens: 54067200.0 | grad norm avg: 0.61 | grad norm last: 0.67 | \n",
      "step: 830 | train samples/s: 13928.0 | train mfu: 0.04 | lr mean: 9.790552576305345e-05 | train loss avg: 5.67 | train loss last: 5.68 | consumed tokens: 54394880.0 | grad norm avg: 0.58 | grad norm last: 0.5 | \n",
      "step: 835 | train samples/s: 13947.9 | train mfu: 0.04 | lr mean: 9.574591967975721e-05 | train loss avg: 5.67 | train loss last: 5.71 | consumed tokens: 54722560.0 | grad norm avg: 0.56 | grad norm last: 0.48 | \n",
      "step: 840 | train samples/s: 13945.0 | train mfu: 0.04 | lr mean: 9.364527795696631e-05 | train loss avg: 5.67 | train loss last: 5.64 | consumed tokens: 55050240.0 | grad norm avg: 0.59 | grad norm last: 0.55 | \n",
      "step: 845 | train samples/s: 13963.0 | train mfu: 0.04 | lr mean: 9.160415356745943e-05 | train loss avg: 5.71 | train loss last: 5.67 | consumed tokens: 55377920.0 | grad norm avg: 0.62 | grad norm last: 0.56 | \n",
      "step: 850 | train samples/s: 13898.8 | train mfu: 0.04 | lr mean: 8.962302672443911e-05 | train loss avg: 5.68 | train loss last: 5.7 | consumed tokens: 55705600.0 | grad norm avg: 0.61 | grad norm last: 0.66 | \n",
      "step: 855 | train samples/s: 13372.4 | train mfu: 0.04 | lr mean: 8.770242129685357e-05 | train loss avg: 5.67 | train loss last: 5.68 | consumed tokens: 56033280.0 | grad norm avg: 0.59 | grad norm last: 0.6 | \n",
      "step: 860 | train samples/s: 13776.7 | train mfu: 0.04 | lr mean: 8.584281749790534e-05 | train loss avg: 5.68 | train loss last: 5.66 | consumed tokens: 56360960.0 | grad norm avg: 0.62 | grad norm last: 0.64 | \n",
      "step: 865 | train samples/s: 13882.8 | train mfu: 0.04 | lr mean: 8.404467371292412e-05 | train loss avg: 5.7 | train loss last: 5.65 | consumed tokens: 56688640.0 | grad norm avg: 0.56 | grad norm last: 0.56 | \n",
      "step: 870 | train samples/s: 13788.0 | train mfu: 0.04 | lr mean: 8.23084483272396e-05 | train loss avg: 5.68 | train loss last: 5.66 | consumed tokens: 57016320.0 | grad norm avg: 0.59 | grad norm last: 0.6 | \n",
      "step: 875 | train samples/s: 13903.5 | train mfu: 0.04 | lr mean: 8.063457789830863e-05 | train loss avg: 5.66 | train loss last: 5.62 | consumed tokens: 57344000.0 | grad norm avg: 0.6 | grad norm last: 0.61 | \n",
      "step: 880 | train samples/s: 13879.3 | train mfu: 0.04 | lr mean: 7.902348443167284e-05 | train loss avg: 5.69 | train loss last: 5.69 | consumed tokens: 57671680.0 | grad norm avg: 0.57 | grad norm last: 0.52 | \n",
      "step: 885 | train samples/s: 13914.5 | train mfu: 0.04 | lr mean: 7.7475568105001e-05 | train loss avg: 5.68 | train loss last: 5.71 | consumed tokens: 57999360.0 | grad norm avg: 0.6 | grad norm last: 0.61 | \n",
      "step: 890 | train samples/s: 13844.9 | train mfu: 0.04 | lr mean: 7.59912290959619e-05 | train loss avg: 5.67 | train loss last: 5.7 | consumed tokens: 58327040.0 | grad norm avg: 0.61 | grad norm last: 0.64 | \n",
      "step: 895 | train samples/s: 13810.5 | train mfu: 0.04 | lr mean: 7.457083120243624e-05 | train loss avg: 5.67 | train loss last: 5.67 | consumed tokens: 58654720.0 | grad norm avg: 0.6 | grad norm last: 0.54 | \n",
      "step: 900 | train samples/s: 13867.2 | train mfu: 0.04 | lr mean: 7.321473822230473e-05 | train loss avg: 5.66 | train loss last: 5.66 | consumed tokens: 58982400.0 | grad norm avg: 0.57 | grad norm last: 0.58 | \n",
      "step: 905 | train samples/s: 13639.2 | train mfu: 0.04 | lr mean: 7.192329212557524e-05 | train loss avg: 5.68 | train loss last: 5.7 | consumed tokens: 59310080.0 | grad norm avg: 0.58 | grad norm last: 0.6 | \n",
      "step: 910 | train samples/s: 13854.8 | train mfu: 0.04 | lr mean: 7.06968130543828e-05 | train loss avg: 5.68 | train loss last: 5.7 | consumed tokens: 59637760.0 | grad norm avg: 0.63 | grad norm last: 0.58 | \n",
      "step: 915 | train samples/s: 13897.4 | train mfu: 0.04 | lr mean: 6.953561387490481e-05 | train loss avg: 5.66 | train loss last: 5.7 | consumed tokens: 59965440.0 | grad norm avg: 0.65 | grad norm last: 0.72 | \n",
      "step: 920 | train samples/s: 13891.5 | train mfu: 0.04 | lr mean: 6.843998562544584e-05 | train loss avg: 5.67 | train loss last: 5.67 | consumed tokens: 60293120.0 | grad norm avg: 0.6 | grad norm last: 0.65 | \n",
      "step: 925 | train samples/s: 13793.2 | train mfu: 0.04 | lr mean: 6.741019751643762e-05 | train loss avg: 5.68 | train loss last: 5.66 | consumed tokens: 60620800.0 | grad norm avg: 0.58 | grad norm last: 0.6 | \n",
      "step: 930 | train samples/s: 13819.1 | train mfu: 0.04 | lr mean: 6.644651875831187e-05 | train loss avg: 5.65 | train loss last: 5.64 | consumed tokens: 60948480.0 | grad norm avg: 0.55 | grad norm last: 0.51 | \n",
      "step: 935 | train samples/s: 13917.4 | train mfu: 0.04 | lr mean: 6.554918945766985e-05 | train loss avg: 5.66 | train loss last: 5.7 | consumed tokens: 61276160.0 | grad norm avg: 0.56 | grad norm last: 0.5 | \n",
      "step: 940 | train samples/s: 13899.0 | train mfu: 0.04 | lr mean: 6.471843516919762e-05 | train loss avg: 5.68 | train loss last: 5.65 | consumed tokens: 61603840.0 | grad norm avg: 0.57 | grad norm last: 0.6 | \n",
      "step: 945 | train samples/s: 13913.4 | train mfu: 0.04 | lr mean: 6.395446689566597e-05 | train loss avg: 5.68 | train loss last: 5.7 | consumed tokens: 61931520.0 | grad norm avg: 0.56 | grad norm last: 0.59 | \n",
      "step: 950 | train samples/s: 13961.9 | train mfu: 0.04 | lr mean: 6.325746653601527e-05 | train loss avg: 5.68 | train loss last: 5.71 | consumed tokens: 62259200.0 | grad norm avg: 0.58 | grad norm last: 0.62 | \n",
      "step: 955 | train samples/s: 13664.1 | train mfu: 0.04 | lr mean: 6.262762326514348e-05 | train loss avg: 5.7 | train loss last: 5.7 | consumed tokens: 62586880.0 | grad norm avg: 0.55 | grad norm last: 0.58 | \n",
      "step: 960 | train samples/s: 13795.5 | train mfu: 0.04 | lr mean: 6.206508260220289e-05 | train loss avg: 5.67 | train loss last: 5.7 | consumed tokens: 62914560.0 | grad norm avg: 0.58 | grad norm last: 0.52 | \n",
      "step: 965 | train samples/s: 13870.7 | train mfu: 0.04 | lr mean: 6.15699973423034e-05 | train loss avg: 5.65 | train loss last: 5.64 | consumed tokens: 63242240.0 | grad norm avg: 0.59 | grad norm last: 0.6 | \n",
      "step: 970 | train samples/s: 13859.4 | train mfu: 0.04 | lr mean: 6.114249117672443e-05 | train loss avg: 5.64 | train loss last: 5.66 | consumed tokens: 63569920.0 | grad norm avg: 0.59 | grad norm last: 0.53 | \n",
      "step: 975 | train samples/s: 13886.0 | train mfu: 0.04 | lr mean: 6.078266596887261e-05 | train loss avg: 5.67 | train loss last: 5.65 | consumed tokens: 63897600.0 | grad norm avg: 0.57 | grad norm last: 0.54 | \n",
      "step: 980 | train samples/s: 13868.4 | train mfu: 0.04 | lr mean: 6.049061266821809e-05 | train loss avg: 5.67 | train loss last: 5.66 | consumed tokens: 64225280.0 | grad norm avg: 0.56 | grad norm last: 0.55 | \n",
      "step: 985 | train samples/s: 13903.4 | train mfu: 0.04 | lr mean: 6.0266407672315836e-05 | train loss avg: 5.65 | train loss last: 5.62 | consumed tokens: 64552960.0 | grad norm avg: 0.57 | grad norm last: 0.5 | \n",
      "step: 990 | train samples/s: 13890.9 | train mfu: 0.04 | lr mean: 6.0110109188826755e-05 | train loss avg: 5.64 | train loss last: 5.65 | consumed tokens: 64880640.0 | grad norm avg: 0.58 | grad norm last: 0.51 | \n",
      "step: 995 | train samples/s: 13834.1 | train mfu: 0.04 | lr mean: 6.002174995956011e-05 | train loss avg: 5.68 | train loss last: 5.68 | consumed tokens: 65208320.0 | grad norm avg: 0.58 | grad norm last: 0.56 | \n",
      "step: 1000 | train samples/s: 14056.3 | train mfu: 0.04 | lr mean: 6.000135908834636e-05 | train loss avg: 5.65 | train loss last: 5.63 | consumed tokens: 65536000.0 | grad norm avg: 0.59 | grad norm last: 0.6 | \n",
      "Training done at 2024-09-08 22:59:45.225184.\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: | 0.087 MB of 0.087 MB uploaded\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run history:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: train consumed tokens ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:   train grad norm avg █▅▃▁▅▁▂▃▂▃▄▃▄▄▃▄▃▄▄▃▃▃▂▂▃▂▃▃▂▂▂▂▂▂▂▂▂▁▂▂\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:  train grad norm last ▃▃▂▁█▁▃▃▃▂▂▂▃▃▂▃▂▃▃▃▂▂▂▂▃▂▃▂▂▂▂▂▂▂▂▂▂▂▁▂\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:         train lr mean ███████▇▇▇▇▇▇▆▆▆▆▅▅▅▄▄▄▄▃▃▃▃▂▂▂▂▂▁▁▁▁▁▁▁\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:  train train loss avg █▆▄▃▃▃▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: train train loss last █▆▄▄▃▃▂▂▂▂▂▂▂▂▂▂▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:       train train mfu ▁███████████████████████████████▃███████\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: train train samples/s ▁███████████████████████████████▃███████\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run summary:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: train consumed tokens 65536000.0\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:   train grad norm avg 0.58829\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:  train grad norm last 0.59563\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:         train lr mean 6e-05\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:  train train loss avg 5.64687\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: train train loss last 5.63281\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:       train train mfu 0.03856\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: train train samples/s 14056.3418\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: 🚀 View run \u001b[33m2024-09-08__22-59-10_7d9fc15e\u001b[0m at: \u001b[34m\u001b[4mhttps://wandb.ai/modalities/ai_24_demo/runs/lahhdqwg\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: ⭐️ View project at: \u001b[34m\u001b[4mhttps://wandb.ai/modalities/ai_24_demo\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Synced 6 W&B file(s), 0 media file(s), 1 artifact file(s) and 0 other file(s)\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Find logs at: \u001b[35m\u001b[1m./wandb_storage/wandb/run-20240908_225919-lahhdqwg/logs\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "! CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --rdzv-endpoint localhost:29515 \\\n",
    "                                        --nnodes 1 \\\n",
    "                                        --nproc_per_node 4 \\\n",
    "                                        $(which modalities) run --config_file_path configs/pretraining_config.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}