02-ElasticSearch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ElasticSearch\n",
    "\n",
    "In this notebook we'll implement a simple search using ElasticSearch.\n",
    "Because ES does not support polish language natively, we'll use polish analyzer plugin to add polish language support.\n",
    "\n",
    "There are many ways we can add this support. The simplest is to set a polish analyzer to selected properties. \n",
    "\n",
    "We can also set a custom tokenizers. That would let us have more control over the way the fields are tokenized. Some words like \"e-mail\" might be split into \"e\" and \"mail\" which is not something we want. \n",
    "\n",
    "There is also a plugin created by Allegro, which might work better for some parts. It's also possible to add a dictionary of synonyms.\n",
    "\n",
    "In later notebooks we'll look at how each of this methods works and what result can we achieve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'test_index'})"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from elasticsearch import Elasticsearch\n",
    "\n",
    "es = Elasticsearch('http://localhost:9200')\n",
    "\n",
    "mapping = {\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"content\": {\n",
    "                \"type\": \"text\",\n",
    "                \"analyzer\": \"polish\"\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "es.options(ignore_status=404).indices.delete(index='test_index')\n",
    "es.options(ignore_status=400).indices.create(index='test_index', body=mapping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze the query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matematyka\n",
      "super\n",
      "podobać\n",
      "trójkąt\n",
      "równania\n"
     ]
    }
   ],
   "source": [
    "text = \"Matematyka jest super! Bardzo mi się podobają trójkąty i równania.\"\n",
    "\n",
    "analysis = es.indices.analyze(index='test_index', body={\n",
    "    'analyzer': 'polish',\n",
    "    'text': text\n",
    "})\n",
    "\n",
    "for token in analysis['tokens']:\n",
    "    print(token['token'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add data to elastic search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens for document 0 :\n",
      "historia\n",
      "polski\n",
      "ciekawy\n",
      "mieszko\n",
      "pierwszey\n",
      "władca\n",
      "polski\n",
      "966\n",
      "przyjąć\n",
      "chrzesć\n",
      "polski\n",
      "stała\n",
      "kraj\n",
      "chrześcijański\n",
      "\n",
      "Tokens for document 1 :\n",
      "maria\n",
      "curia\n",
      "skłodowski\n",
      "wybitny\n",
      "polski\n",
      "naukowczynić\n",
      "otrzymać\n",
      "dwie\n",
      "nagroda\n",
      "nobl\n",
      "pierwy\n",
      "kobieta\n",
      "otrzymać\n",
      "nagroda\n",
      "chemia\n",
      "pasja\n",
      "odkryć\n",
      "pierwiastek\n",
      "promieniotwórczy\n",
      "\n",
      "Tokens for document 2 :\n",
      "trygonometria\n",
      "działem\n",
      "matematyka\n",
      "zajmować\n",
      "on\n",
      "badać\n",
      "zależność\n",
      "międyć\n",
      "bok\n",
      "kąt\n",
      "trójkąt\n",
      "trygonometria\n",
      "ważny\n",
      "nawigacja\n",
      "dzięk\n",
      "móc\n",
      "określić\n",
      "położen\n",
      "statek\n",
      "morze\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sentences = [\n",
    "    \"Historia Polski jest bardzo ciekawa. Mieszko I był pierwszym władcą Polski. W roku 966 przyjął chrzest. W ten sposób Polska stała się krajem chrześcijańskim.\",\n",
    "    \"Maria Curie-Skłodowska była wybitną polską naukowczynią. Otrzymała dwie Nagrody Nobla. Była pierwszą kobietą, która otrzymała tę nagrodę. Chemia była jej pasją. Odkryła pierwiastki promieniotwórcze.\",\n",
    "    \"Trygonometria jest działem matematyki. Zajmuje się ona badaniem zależności między bokami i kątami trójkątów. Trygonometria jest bardzo ważna w nawigacji. Dzięki niej możemy określić położenie statku na morzu.\",\n",
    "]\n",
    "\n",
    "for i, sentence in enumerate(sentences):\n",
    "    es.index(index='test_index', id=i, body={'content': sentence})\n",
    "\n",
    "response = es.search(index='test_index', body={\"query\": {\"match_all\": {}}})\n",
    "\n",
    "# For each document in the response, analyze its 'content' field\n",
    "for hit in response['hits']['hits']:\n",
    "    document = hit['_source']\n",
    "    content = document['content']\n",
    "    \n",
    "    # Analyze the content\n",
    "    analysis = es.indices.analyze(index='test_index', body={\n",
    "        'analyzer': 'polish',\n",
    "        'text': content\n",
    "    })\n",
    "    \n",
    "    # Print the tokens\n",
    "    print(\"Tokens for document\", hit['_id'], \":\")\n",
    "    for token in analysis['tokens']:\n",
    "        print(token['token'])\n",
    "    print()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Liczba wyników: 1\n",
      "Dokument: Trygonometria jest działem matematyki. Zajmuje się ona badaniem zależności między bokami i kątami trójkątów. Trygonometria jest bardzo ważna w nawigacji. Dzięki niej możemy określić położenie statku na morzu.\n",
      "Wynik: 0.9768399\n",
      "\n"
     ]
    }
   ],
   "source": [
    "query = {\n",
    "    \"query\": {\n",
    "        \"match\": {\n",
    "            \"content\": {\n",
    "                \"query\": \"matematyka\",\n",
    "                \"analyzer\": \"polish\"\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "results = es.search(index='test_index', body=query)\n",
    "\n",
    "print('Liczba wyników:', results['hits']['total']['value'])\n",
    "for hit in results['hits']['hits']:\n",
    "    print('Dokument:', hit['_source']['content'])\n",
    "    print('Wynik:', hit['_score'])\n",
    "    print()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}