{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "enppnUXzd6nT"
   },
   "source": [
    "Access this notebook on Google Colab: [bit.ly/anlp25-ner](https://bit.ly/anlp25-ner)\n",
    "\n",
    "In this notebook, we'll explore named entity recognition through the lens of toponym resolution, using NER to extract a list of geopolitical place names in a text, and then plotting those locations on a map (using the Folium mapping library -- see [here](https://blog.prototypr.io/interactive-maps-with-python-part-1-aa1563dbe5a9) for a Folium tutorial).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FiD3qsCid6nX"
   },
   "outputs": [],
   "source": [
    "!pip install folium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xalHX8KOd6nY"
   },
   "outputs": [],
   "source": [
    "!pip install wikipedia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bNAMLZJVd6nZ"
   },
   "outputs": [],
   "source": [
    "import folium\n",
    "import wikipedia\n",
    "import spacy\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "f1wUvdJHd6nZ"
   },
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_sm', disable=['parser'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Yo4vZoShd6nZ"
   },
   "source": [
    "There are several good APIs for resolving place names to their latitude/longitude (such as [Nominatim](https://wiki.openstreetmap.org/wiki/Nominatim) from OpenStreetMap and Google's [Geocoding API](https://developers.google.com/maps/documentation/geocoding/)).  Those are typically rate-limited or not free, so for this notebook let's use a simple georeferencer using data from [GeoNames](http://download.geonames.org/export/dump/) -- we'll assign each mention of a geopolitical entity placename to the city with the same name; in cases of ambiguity (e.g., Cambridge, MA vs. Cambridge UK), we'll select the city with the greatest population."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "S36OsyfOd6na"
   },
   "outputs": [],
   "source": [
    "def read_geonames(city_filename, country_filename):\n",
    "    cities=[]\n",
    "    countries=[]\n",
    "    with open(city_filename) as file:\n",
    "        for idx,line in enumerate(file):\n",
    "            cols=line.rstrip().split(\"\\t\")\n",
    "            name=cols[1].lower()\n",
    "            lat=float(cols[4])\n",
    "            long=float(cols[5])\n",
    "            population=int(cols[14])\n",
    "\n",
    "            cities.append((name, population, lat, long))\n",
    "\n",
    "    with open(country_filename) as file:\n",
    "        for idx,line in enumerate(file):\n",
    "            if line.startswith(\"#\"):\n",
    "                continue\n",
    "            cols=line.rstrip().split(\"\\t\")\n",
    "            name=cols[4].lower()\n",
    "            countries.append(name)\n",
    "\n",
    "    return cities, set(countries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ipikXpujeKDZ"
   },
   "outputs": [],
   "source": [
    "!wget https://github.com/dbamman/anlp25/raw/refs/heads/main/data/cities500.txt\n",
    "!wget https://github.com/dbamman/anlp25/raw/refs/heads/main/data/countryInfo.txt\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "p5E6e7Afd6na"
   },
   "outputs": [],
   "source": [
    "cities, countries=read_geonames(\"cities500.txt\", \"countryInfo.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hLmEwD2vd6nb"
   },
   "outputs": [],
   "source": [
    "def resolve_toponyms(locations, cities, countries, doc):\n",
    "    \"\"\" Resolve a counter of GPE entities to their latitude/longitude coordinates\n",
    "    Input:\n",
    "        - locations: counter mapping GPE entities to their count in a text\n",
    "        - cities: list of cities containing (placename, population, lat, long) tuples\n",
    "        - countries: set of country names\n",
    "        - doc: spacy-processed document containing all tokens, entities, etc.\n",
    "\n",
    "    Output: dict mapping each GPE entity to (lat, long) tuple \"\"\"\n",
    "\n",
    "    coordinates={}\n",
    "\n",
    "    new_geo={}\n",
    "\n",
    "    for (placename, population, lat, long) in cities:\n",
    "        if placename in countries:\n",
    "            continue\n",
    "\n",
    "        # for placenames that refer to multiple cities, just keep the city with biggest population\n",
    "        if placename in new_geo:\n",
    "            _, cur_pop, _, _=new_geo[placename]\n",
    "            if population > cur_pop:\n",
    "                new_geo[placename]=(placename, population, lat, long)\n",
    "        else:\n",
    "            new_geo[placename]=(placename, population, lat, long)\n",
    "\n",
    "\n",
    "    for entity in locations:\n",
    "        if entity in new_geo:\n",
    "            coordinates[entity]=(new_geo[entity][2], new_geo[entity][3])\n",
    "\n",
    "    return coordinates\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "s_mHlDZyd6nb"
   },
   "outputs": [],
   "source": [
    "def map_toponyms(text, cities, countries):\n",
    "    doc=nlp(text)\n",
    "\n",
    "    locations=Counter()\n",
    "    for entity in doc.ents:\n",
    "        # We'll select just entities that are tagged geopolitical entities (which include cities)\n",
    "        if entity.label_ == \"GPE\":\n",
    "            locations[entity.text.lower()]+=1\n",
    "\n",
    "\n",
    "    coordinates=resolve_toponyms(locations, cities, countries, doc)\n",
    "\n",
    "    center=None\n",
    "    maxentity=None\n",
    "    maxcount=0\n",
    "    for entity in coordinates:\n",
    "        if locations[entity] > maxcount:\n",
    "            maxcount=locations[entity]\n",
    "            center=[coordinates[entity][0], coordinates[entity][1]]\n",
    "\n",
    "            maxentity=entity\n",
    "\n",
    "    # Create map centered on the most frequently mentioned city\n",
    "    folium_map = folium.Map(location=center,\n",
    "                            zoom_start=3,\n",
    "                            tiles=\"CartoDB dark_matter\")\n",
    "\n",
    "    # Add locadtions to map\n",
    "    for entity in coordinates:\n",
    "        radius=locations[entity]\n",
    "        marker = folium.CircleMarker(location=[coordinates[entity][0], coordinates[entity][1]], radius=radius, fill=True, popup=entity)\n",
    "        marker.add_to(folium_map)\n",
    "\n",
    "    return folium_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kitT2v_ad6nb"
   },
   "source": [
    "Let's test our method by pulling articles from Wikipedia and plotting the placenames mentioned in them.  Explore this -- try inputting other Wikipedia articles and visualizing the places.  Let us all know if you find an interesting one!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-SznhBw1d6nb"
   },
   "outputs": [],
   "source": [
    "ucb = wikipedia.page(\"University of California, Berkeley\")\n",
    "ww2 = wikipedia.page(\"World War II\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-N9WUF-1d6nc"
   },
   "outputs": [],
   "source": [
    "folium_map=map_toponyms(ww2.content, cities, countries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mxuEWfdyd6nc"
   },
   "outputs": [],
   "source": [
    "folium_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lMFERvhYd6nd"
   },
   "outputs": [],
   "source": [
    "!wget https://github.com/dbamman/anlp25/raw/refs/heads/main/data/twain_innocents_abroad.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-_RRQ3LDd6nd"
   },
   "source": [
    "Now let's try it with the full text of a book (Mark Twain's travelogue *Innocents Abroad*).  Running this through spacy will take a minute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sFbS6vGad6nd"
   },
   "outputs": [],
   "source": [
    "with open(\"twain_innocents_abroad.txt\") as file:\n",
    "    data=file.read()\n",
    "folium_map=map_toponyms(data, cities, countries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZRdT4Pewd6nd"
   },
   "outputs": [],
   "source": [
    "folium_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qJH5Elf1d6ne"
   },
   "source": [
    "Download a text of your own from [Project Gutenberg](https://www.gutenberg.org) and run it through the pipeline above (Project Gutenberg has many works of literature published before 1925).  Generate a visualization for it and be prepared to share your visualization in class.  Are the locations centered around the main setting of the work?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Zu27Nqd4d6ne"
   },
   "outputs": [],
   "source": [
    "!wget --no-check-certificate https://www.gutenberg.org/files/1342/1342-0.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "q6cGGlJBd6ne"
   },
   "outputs": [],
   "source": [
    "with open(\"1342-0.txt\") as file:\n",
    "    data=file.read()\n",
    "map_toponyms(data, cities, countries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2uOtIzPQfkXW"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}