{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MWihWfGuVkRP"
      },
      "source": [
        "# ANLI Dataset Filtering\n",
        "\n",
        "This Google Colab notebook is inspired by the [*Start Your NLI Research*](https://github.com/facebookresearch/anli/blob/main/mds/start_your_nli_research.md) instructions located on the [ANLI](https://github.com/facebookresearch/anli) GitHub repo. This is intended to be run on a [Google Colab Pro](https://colab.research.google.com/signup) or [Pro+](https://colab.research.google.com/signup) account leveraging a GPU-backed runtime."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4PAZ13O77C2w"
      },
      "source": [
        "## Connect to Google Drive\n",
        "\n",
        "We will connect to [Google Drive](https://drive.google.com) to store weights and data within the cloud. This is needed because Google Colab has a maximum 24-hour runtime, even with Pro and Pro+ accounts. After running the cell below, you will be prompted to connect with your Google Drive account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nln7suF-69LL"
      },
      "outputs": [],
      "source": [
        "# Mount into drive\n",
        "from google.colab import drive\n",
        "drive.mount(\"/content/drive\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B7Dy2XhY7_c9"
      },
      "source": [
        "This cell created an `ANLI Project Data` folder within your `Colab Notebooks` folder."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Cea7z6I869OY"
      },
      "outputs": [],
      "source": [
        "%mkdir -p /content/drive/MyDrive/Colab\\ Notebooks/ANLI\\ Project\\ Data\n",
        "%mkdir -p /content/drive/MyDrive/Colab\\ Notebooks/ANLI\\ Project\\ Data/scripts\n",
        "%mkdir -p /content/drive/MyDrive/Colab\\ Notebooks/ANLI\\ Project\\ Data/checkpoints"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BMBDQPTvX2Yt"
      },
      "source": [
        "## GPU Allocation\n",
        "\n",
        "It is a good idea to capture what kind of GPU we have allocated to us. The following commands do this in a summarized and a verbose manner."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I_1_x5JOYHP3"
      },
      "outputs": [],
      "source": [
        "!nvidia-smi -L"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PekoZAVzYHSj"
      },
      "outputs": [],
      "source": [
        "!nvidia-smi -q"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZrB7azJfVsqC"
      },
      "source": [
        "## Project Setup\n",
        "\n",
        "### Code Setup\n",
        "\n",
        "First,  we need to download the [ANLI](https://github.com/facebookresearch/anli) repo and build the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9vvrCFC5Vq8T"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/facebookresearch/anli.git 2>/dev/null\n",
        "!source anli/setup.sh"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r0MUm0J5WgjE"
      },
      "source": [
        "Then, we'll change directory into the source code directory, `anli`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xo4L-XjwEcQD"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import sys\n",
        "\n",
        "try:\n",
        "    os.chdir('anli/')\n",
        "except FileNotFoundError as e:\n",
        "    print(f\"Could not change directory: {str(e)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cAFxj96xYVtP"
      },
      "source": [
        "Finally, as far as code goes, we'll need the `transformers` module from the popular [Hugging Face](https://huggingface.co/docs/transformers/index) open-source NLP company and `sentencepiece` which is needed to support experiments with the xlnet model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wmVDK6niYUYi"
      },
      "outputs": [],
      "source": [
        "!pip install transformers sentencepiece"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QX-Pz088WuXO"
      },
      "source": [
        "#### Environment Variables\n",
        "\n",
        "Before moving onto dataset setup, we'll set some environment variables to prepare for the Bash and Python scripts that follow."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "39E8dSShPDcS"
      },
      "outputs": [],
      "source": [
        "%env PYTHONPATH='/env/python:/content/anli/src:/content/anli/utest:/content/anli/src/dataset_tools'\n",
        "%env MASTER_ADDR=localhost"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sXU1zxgOXKT-"
      },
      "source": [
        "### Dataset Setup\n",
        "\n",
        "We can't train a model without data, so this will download the SNLI, MNLI, FEVER, and NLI datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UYf3Epg2SMaG"
      },
      "outputs": [],
      "source": [
        "!bash ./script/download_data.sh"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SDjY0zN0XtXu"
      },
      "source": [
        "Now, we'll transform the dataset into a format that the ANLI project expects."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KN6qem_WFxlf"
      },
      "outputs": [],
      "source": [
        "!python ./src/dataset_tools/build_data.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Pk1JIFy-PGx"
      },
      "source": [
        "## Update Training Script"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FE-Krd7BAKC1"
      },
      "source": [
        "If you've placed a modified `training.py` script in your GDrive `Colab Notebooks/ANLI Project Data/scripts/` directory, uncomment and run the following line so that your updated script will be used in the ***Model Training*** section."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b8nAruWu_3dW"
      },
      "outputs": [],
      "source": [
        "#!cp /content/drive/MyDrive/Colab\\ Notebooks/ANLI\\ Project\\ Data/scripts/training.py ./src/nli/training.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dI0C8fWcAii0"
      },
      "source": [
        "Alternatively, if you are storing an updated `training.py` in a public GitHub/GitLab repo, uncomment and run the following line after updating the URL to point to the *raw* file. In a browser, this will look like a plaintext version of the file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zUUVh9k8-SA4"
      },
      "outputs": [],
      "source": [
        "#!curl https://https://raw.githubusercontent.com/username/project/main/src/nli/training.py -o ./src/nli/training.py"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Update Data"
      ],
      "metadata": {
        "id": "usAZ5Dr7zgWp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "If any custom datasets are used for training or evaluation, the following lines bring them from Google Drive into Colab."
      ],
      "metadata": {
        "id": "TXawrKnBB3NY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#%mkdir -p experiments"
      ],
      "metadata": {
        "id": "RhTsSQRk0FV0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#!cp -R /content/drive/MyDrive/Colab\\ Notebooks/ANLI\\ Project\\ Data/data/* ./experiments"
      ],
      "metadata": {
        "id": "Gcwqhgx1zgoC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8DK8PO33Y8At"
      },
      "source": [
        "## Model Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5qZUOS-jZL2P"
      },
      "source": [
        "Note that a list of supported models and extra, undocumented command line arguments are located in `/content/anli/src/nli/training.py`. Comments are below where changes have been made from [*Start Your NLI Research*](https://github.com/facebookresearch/anli/blob/main/mds/start_your_nli_research.md) instructions.\n",
        "\n",
        "During training, model checkpoints will be automatically saved in a `saved_models` directory.\n",
        "\n",
        "***Changelog***\n",
        "\n",
        "* `-g 1`: This was changed to 1, since we only have one GPU.\n",
        "* `--single_gpu`: This was added to suppress PyTorch Multiprocessing logic from kicking in.\n",
        "* `--experiment_name`: The name of the experiment. During training, model checkpoints will be saved in `saved_models/{TRAINING_START_TIME}_[experiment_name]`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tU8pKQHcHtC2"
      },
      "outputs": [],
      "source": [
        "!python ./src/nli/training.py \\\n",
        "    --model_class_name \"roberta-large\" \\\n",
        "    -n 1 \\\n",
        "    -g 1 \\\n",
        "    --single_gpu \\\n",
        "    -nr 0 \\\n",
        "    --max_length 156 \\\n",
        "    --gradient_accumulation_steps 1 \\\n",
        "    --per_gpu_train_batch_size 16 \\\n",
        "    --per_gpu_eval_batch_size 16 \\\n",
        "    --save_prediction \\\n",
        "    --train_data snli_train:none,mnli_train:none \\\n",
        "    --train_weights 1,1 \\\n",
        "    --eval_data snli_dev:none \\\n",
        "    --eval_frequency 2000 \\\n",
        "    --experiment_name \"roberta-large|snli|nli\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L70Ca7T8FqxL"
      },
      "source": [
        "**Make sure to queue this command alongside your training cell.** This will copy saved checkpoints from Colab to your Google Drive."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XbfUqimYa0tY"
      },
      "outputs": [],
      "source": [
        "!cp -R ./saved_models/* /content/drive/MyDrive/Colab\\ Notebooks/ANLI\\ Project\\ Data/checkpoints/"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "background_execution": "on",
      "collapsed_sections": [],
      "machine_shape": "hm",
      "name": "ANLI on Google Colab.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}