{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LHCHnheBcwYs"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dbamman/anlp25/blob/main/3.embeddings/Lexical_Semantics.ipynb)\n",
    "\n",
    "**N.B.** Once it's open on Colab, remember to save a copy (by e.g. clicking `Copy to Drive` above).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tTNSdu8qV1lt"
   },
   "source": [
    "## Homework 3: Lexical Semantics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NDmHpjTJziCs"
   },
   "source": [
    "In this homework, we will explore lexical semantics in the context of slang and FastText, an alternative to word2vec (Part 1); and, how to represent a sentence with individual word vectors, so we can measure the similarity between a pair of sentences (Part 2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bv5qAYY2vxLH"
   },
   "source": [
    "### Part 1: Slang and word similarity with FastText"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UpqZQI31YG-S"
   },
   "source": [
    "Slang presents an interesting linguistic phenomenon that involves non-standard word forms. For this question, you will explore how FastText, an alternative to Word2Vec, handles the lexical semantics of slang and informal language.\n",
    "\n",
    "First, **familiarize yourself with the slang dataset that we are using**, introduced in [\"Toward Informal Language Processing: Knowledge of Slang in Large Language Models\" (Sun et al., NAACL 2024)](https://aclanthology.org/2024.naacl-long.94/). The full dataset includes annotations indicating whether a sentence from OpenSubtitles (typically a line from a movie) contains a slang term, and you can find some example sentences and terms here:\n",
    "\n",
    "https://raw.githubusercontent.com/dbamman/anlp25/main/data/slang_examples.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the 10 most common slang words?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, **train a [FastText model](https://radimrehurek.com/gensim/models/fasttext.html#gensim.models.fasttext.FastText) using `gensim` and `FastText`** on our slang data derived from the dataset described above, which you can download here:\n",
    "\n",
    "https://raw.githubusercontent.com/dbamman/anlp25/main/data/slang_corpus.txt\n",
    "\n",
    "For preprocessing, in the `txt` file, each token is already separated by whitespace, so you don't need to worry about tokenization. Treat each line in the file as one \"sentence\". For training, use the following parameters: embedding size of 400, context window of 5, frequency threshold of 5, and use 5 workers to train for 5 epochs.\n",
    "\n",
    "Note: we are using the `FastText` _implementation_ included in the `gensim` library! **Don't use the `fasttext` library.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the trained models:\n",
    "\n",
    "**Q1.** Pick a slang term, and in about 100 words, discuss:\n",
    "\n",
    "- what the most similar words are to the slang term of your choosing (as measured by the model), and\n",
    "- whether the result is aligned with your understanding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fy1QSpMktM_W"
   },
   "source": [
    "**Q2.** Train a separate word2vec (not FastText) model using the same dataset, and in about 100 words, compare the two approaches used to estimate word vectors. Here are some potential topics:\n",
    "\n",
    "- Look up the token `gonna` in both word2vec and FastText models. What does this tell you?\n",
    "- What is the high level difference between word2vec and FastText?\n",
    "- Evaluate the quality of the trained embeddings through intrinsic evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GtbVFoKquMFI"
   },
   "source": [
    "### Part 2. From words to sentences\n",
    "\n",
    "So far we've been working with word vectors, but in real-world scenarios, we often want to work with not just a word, but a sequence (like a sentence), which will explore later in the semester. However, with what we have learned so far, how do you represent a sentence? One approach is to look up the word vectors for individual words in the sentence and then *average* them, which we will explore in this question. We will be using pre-trained GloVe vectors [cf. SLP 6.8.3] we used in class. Download them here:\n",
    "\n",
    "https://raw.githubusercontent.com/dbamman/anlp25/main/data/glove.6B.100d.100K.txt\n",
    "\n",
    "**Q3.** Load the pretrained embeddings with `gensim`'s `load_word2vec_format` (see the lab notebooks), and create a function that takes a pair of sentences as input, and outputs the similarity of the two sentences measured by cosine -- the sentence pair you can use for sanity check is provided below.\n",
    "\n",
    "Find a pair of sentences where the similarity is high, but mean different (or opposite) things. Find a pair of sentences where the similarity is low, but you think the meanings are similar. In a paragraph, discuss why we might see these results given how we construct sentence embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sanity check: compare the similarity between:\n",
    "#       sentences[0], sentences[1]\n",
    "#       sentences[0], sentences[2]\n",
    "\n",
    "sentences = [\n",
    "    ['the', 'cat', 'is', 'on', 'the', 'mat'],\n",
    "    ['a', 'computer', 'is', 'in', 'the', 'store'],\n",
    "    ['the', 'cat', 'sits', 'on', 'the', 'mat']\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zYiKUIr5cwYw"
   },
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WqWFmNaBPYUq"
   },
   "source": [
    "## To submit\n",
    "\n",
    "Congratulations on finishing this homework!\n",
    "Please follow the instructions below to download the notebook file (`.ipynb`) and its printed version (`.pdf`) for submission on bCourses -- remember **all cells must be executed**."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "_TfNi4MzsTLk"
   ],
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}