notebooks/pfocr_fetch.R.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e6105050-eb52-4ef9-8a6b-05c6880100ac",
   "metadata": {},
   "source": [
    "# fetch list of pathway figures from NCBI\n",
    "\n",
    "Search PubMed for figures with titles or captions containing pathway-related terms and download the figure list. The actual downloading of figures is in get_figures.ipynb. Note this is the notebook version of what used to be a script: [pfocr_fetch.R](https://github.com/wikipathways/pathway-figure-ocr/blob/51f6af1697531814b4cd654584cd5759cb6f2879/pfocr_fetch.R)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1333cb7f-384c-482f-80a0-fd79ab053bb9",
   "metadata": {},
   "source": [
    "NOTE: query qualifier for figure captions [CAPT] is clearly broken and only hits on a fraction of caption titles.\n",
    "the `imagesdocsum` report type does a better job of actually searching captions, e.g.:\n",
    "- https://www.ncbi.nlm.nih.gov/pmc/?term=(signaling+pathway)+AND+(2019+[pdat])&report=imagesdocsum&dispmax=100 \n",
    "(11349 hits with \"signaling pathway\" in every caption title or caption body)\n",
    "- https://www.ncbi.nlm.nih.gov/pmc/?term=(signaling+pathway[CAPT])+AND+(2019+[pdat])&report=imagesdocsum&dispmax=100\n",
    "(244 hits with \"signaling pathway\" ONLY in caption titles)\n",
    "- https://www.ncbi.nlm.nih.gov/pmc/?term=(signaling+pathway[CAPT])+AND+(2019+[pdat])\n",
    "(2775 hits when \"report=imagesdocsum\" is excluded)\n",
    "\n",
    "NOTE: the `imagesdocsum` report is not supported by NCBI's eutils, so we'll have to go with HTML scraping. \n",
    "The pagination of pmc output is not apparent, however...\n",
    "\n",
    "## Example queries for what is possible\n",
    "- https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=asthma[mesh]+AND+leukotrienes[mesh]+AND+2009[pdat]&usehistory=y&retmax=500&retStart=0\n",
    "- https://www.ncbi.nlm.nih.gov/pmc/?term=signaling+pathway+AND+2018+[pdat]&report=imagesdocsum&dispmax=100\n",
    "- https://www.ncbi.nlm.nih.gov/pmc/?term=((((((((((signaling+pathway)+OR+regulatory+pathway)+OR+disease+pathway)+OR+drug+pathway)+OR+metabolic+pathway)+OR+biosynthetic+pathway)+OR+synthesis+pathway)+OR+cancer+pathway)+OR+response+pathway)+OR+cycle+pathway)+AND+(\\%222019/01/01\\%22[PUBDATE]+%3A+\\%223000\\%22[PUBDATE])&report=imagesdocsum&dispmax=100#\n",
    "\n",
    "## Network query:\n",
    "- https://www.ncbi.nlm.nih.gov/pmc/?term=((network)+OR+PPI)+AND+(%222019/01/01%22[PUBDATE]+%3A+%223000%22[PUBDATE])&report=imagesdocsum&dispmax=100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "30739efc-c249-429e-bca7-4804aacfe0a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'https://www.ncbi.nlm.nih.gov/pmc/?term=(((((((((((signaling+pathway)+OR+signalling+pathway)+OR+regulatory+pathway)+OR+disease+pathway)+OR+drug+pathway)+OR+metabolic+pathway)+OR+biosynthetic+pathway)+OR+synthesis+pathway)+OR+cancer+pathway)+OR+response+pathway)+OR+cycle+pathway)+AND+(\"2018/01/01\"[PUBDATE]+%3A+\"3000/01/01\"[PUBDATE])&amp;report=imagesdocsum&amp;dispmax=100'"
      ],
      "text/latex": [
       "'https://www.ncbi.nlm.nih.gov/pmc/?term=(((((((((((signaling+pathway)+OR+signalling+pathway)+OR+regulatory+pathway)+OR+disease+pathway)+OR+drug+pathway)+OR+metabolic+pathway)+OR+biosynthetic+pathway)+OR+synthesis+pathway)+OR+cancer+pathway)+OR+response+pathway)+OR+cycle+pathway)+AND+(\"2018/01/01\"{[}PUBDATE{]}+\\%3A+\"3000/01/01\"{[}PUBDATE{]})\\&report=imagesdocsum\\&dispmax=100'"
      ],
      "text/markdown": [
       "'https://www.ncbi.nlm.nih.gov/pmc/?term=(((((((((((signaling+pathway)+OR+signalling+pathway)+OR+regulatory+pathway)+OR+disease+pathway)+OR+drug+pathway)+OR+metabolic+pathway)+OR+biosynthetic+pathway)+OR+synthesis+pathway)+OR+cancer+pathway)+OR+response+pathway)+OR+cycle+pathway)+AND+(\"2018/01/01\"[PUBDATE]+%3A+\"3000/01/01\"[PUBDATE])&amp;report=imagesdocsum&amp;dispmax=100'"
      ],
      "text/plain": [
       "[1] \"https://www.ncbi.nlm.nih.gov/pmc/?term=(((((((((((signaling+pathway)+OR+signalling+pathway)+OR+regulatory+pathway)+OR+disease+pathway)+OR+drug+pathway)+OR+metabolic+pathway)+OR+biosynthetic+pathway)+OR+synthesis+pathway)+OR+cancer+pathway)+OR+response+pathway)+OR+cycle+pathway)+AND+(\\\"2018/01/01\\\"[PUBDATE]+%3A+\\\"3000/01/01\\\"[PUBDATE])&report=imagesdocsum&dispmax=100\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "################## QUERY BUILDER\n",
    "\n",
    "## Pathway types:\n",
    "query.terms <- c(\n",
    "  \"signaling+pathway\", \"signalling+pathway\", \"regulatory+pathway\",\n",
    "  \"disease+pathway\", \"drug+pathway\", \"metabolic+pathway\",\n",
    "  \"biosynthetic+pathway\", \"synthesis+pathway\", \"cancer+pathway\",\n",
    "  \"response+pathway\", \"cycle+pathway\"\n",
    ")\n",
    "\n",
    "query.date.from <- \"2018/01/01\"\n",
    "query.date.to <- \"3000/01/01\"\n",
    "\n",
    "term <- paste0(\n",
    "  \"term=\", paste(rep(\"(\", length(query.terms)), collapse = \"\"), paste(lapply(\n",
    "    query.terms,\n",
    "    function(x) {\n",
    "      paste0(x, \")\")\n",
    "    }\n",
    "  ), collapse = \"+OR+\"), \"+AND+(\\\"\", query.date.from, \"\\\"[PUBDATE]+%3A+\\\"\", query.date.to,\n",
    "  \"\\\"[PUBDATE])\"\n",
    ")\n",
    "\n",
    "query.url <- paste0(\n",
    "  \"https://www.ncbi.nlm.nih.gov/pmc/?\", term, \"&report=imagesdocsum\",\n",
    "  \"&dispmax=100\"\n",
    ")\n",
    "query.url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "64f8cfc6-4a97-4539-9555-6e5f87f5b66b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading required package: xml2\n",
      "\n",
      "── \u001b[1mAttaching packages\u001b[22m ─────────────────────────────────────── tidyverse 1.3.0 ──\n",
      "\n",
      "\u001b[32m✔\u001b[39m \u001b[34mggplot2\u001b[39m 3.3.3     \u001b[32m✔\u001b[39m \u001b[34mpurrr  \u001b[39m 0.3.4\n",
      "\u001b[32m✔\u001b[39m \u001b[34mtibble \u001b[39m 3.0.4     \u001b[32m✔\u001b[39m \u001b[34mdplyr  \u001b[39m 1.0.2\n",
      "\u001b[32m✔\u001b[39m \u001b[34mtidyr  \u001b[39m 1.1.2     \u001b[32m✔\u001b[39m \u001b[34mstringr\u001b[39m 1.4.0\n",
      "\u001b[32m✔\u001b[39m \u001b[34mreadr  \u001b[39m 1.4.0     \u001b[32m✔\u001b[39m \u001b[34mforcats\u001b[39m 0.5.0\n",
      "\n",
      "[conflicted] Will prefer \u001b[34mdplyr::filter\u001b[39m over any other package\n",
      "\n",
      "[conflicted] Will prefer \u001b[34mdplyr::select\u001b[39m over any other package\n",
      "\n",
      "[conflicted] Will prefer \u001b[34mdplyr::mutate\u001b[39m over any other package\n",
      "\n"
     ]
    }
   ],
   "source": [
    "################ PMC SCRAPER\n",
    "library(conflicted)\n",
    "library(processx)\n",
    "library(RSelenium)\n",
    "library(rvest)\n",
    "library(xml2)\n",
    "library(tidyverse)\n",
    "conflict_prefer(\"filter\", \"dplyr\")\n",
    "conflict_prefer(\"select\", \"dplyr\")\n",
    "conflict_prefer(\"mutate\", \"dplyr\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8dc3cfce-499b-414f-a71b-a463551fd057",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'/home/ariutta/Documents/pathway-figure-ocr/data/imagesdocsum_pathway_queries/20210513'"
      ],
      "text/latex": [
       "'/home/ariutta/Documents/pathway-figure-ocr/data/imagesdocsum\\_pathway\\_queries/20210513'"
      ],
      "text/markdown": [
       "'/home/ariutta/Documents/pathway-figure-ocr/data/imagesdocsum_pathway_queries/20210513'"
      ],
      "text/plain": [
       "[1] \"/home/ariutta/Documents/pathway-figure-ocr/data/imagesdocsum_pathway_queries/20210513\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# set dir for saving results as tsv\n",
    "data_dir_base <- \"../data/imagesdocsum_pathway_queries/\"\n",
    "# data_dir_suffix <- \"20210429\"\n",
    "data_dir_suffix <- \"20210513\"\n",
    "data_dir <- paste0(data_dir_base, data_dir_suffix)\n",
    "\n",
    "if (basename(getwd()) != \"notebooks\") {\n",
    "  # update this to whatever is appropriate for your system\n",
    "  setwd(\"/home/ariutta/Documents/pathway-figure-ocr/notebooks\")\n",
    "}\n",
    "\n",
    "if (\"pfocr_fetch.R.ipynb\" %in% list.files()) {\n",
    "  if (!dir.exists(data_dir)) {\n",
    "    dir.create(data_dir, showWarnings = TRUE, recursive = TRUE)\n",
    "  }\n",
    "  setwd(data_dir)\n",
    "} else {\n",
    "  print(\"Error: wrong directory\")\n",
    "}\n",
    "\n",
    "getwd()\n",
    "\n",
    "cat(query.url, file = \"query.txt\")\n",
    "write.table(data.frame(\n",
    "  \"figureid\", \"pmcid\", \"filename\", \"fignumber\", \"figtitle\",\n",
    "  \"papertitle\", \"figcaption\", \"figlink\", \"reftext\"\n",
    "),\n",
    "file = \"pmc.df.all.tsv\", append = FALSE,\n",
    "sep = \"\\t\", quote = FALSE, col.names = FALSE, row.names = FALSE\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "24fa4bc9-e27d-4bf0-ad16-64dce41f8fba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# adapted from https://www.tidyverse.org/blog/2018/09/processx-3.2.0/\n",
    "start_program <- function(command, args, message, timeout = 5, ...) {\n",
    "  timeout <- as.difftime(timeout, units = \"secs\")\n",
    "  deadline <- Sys.time() + timeout\n",
    "  px <- process$new(command, args, stdout = \"|\", stderr = \"|\", ...)\n",
    "  print(px)\n",
    "  while (px$is_alive() && (now <- Sys.time()) < deadline) {\n",
    "    poll_time <- as.double(deadline - now, units = \"secs\") * 1000\n",
    "    px$poll_io(as.integer(poll_time))\n",
    "    out_lines <- px$read_output_lines()\n",
    "    err_lines <- px$read_error_lines()\n",
    "    if (any(grepl(message, c(out_lines, err_lines)))) {\n",
    "      return(px)\n",
    "    }\n",
    "  }\n",
    "\n",
    "  px$kill()\n",
    "  stop(\"Cannot start \", command)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9f64062e-0215-42dd-8d6a-294e8069c8ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROCESS 'selenium-server', running, pid 22124.\n"
     ]
    }
   ],
   "source": [
    "# TODO: don't start it if it's already running\n",
    "proc <- start_program(\"selenium-server\", c(\"-port\", \"4445\"), \"running on port\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "51462ed4-e133-4a59-997a-2aed7756f94a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"Connecting to remote server\"\n",
      "$acceptInsecureCerts\n",
      "[1] FALSE\n",
      "\n",
      "$browserName\n",
      "[1] \"firefox\"\n",
      "\n",
      "$browserVersion\n",
      "[1] \"68.8.0\"\n",
      "\n",
      "$`moz:accessibilityChecks`\n",
      "[1] FALSE\n",
      "\n",
      "$`moz:buildID`\n",
      "[1] \"20200613165304\"\n",
      "\n",
      "$`moz:geckodriverVersion`\n",
      "[1] \"0.26.0\"\n",
      "\n",
      "$`moz:headless`\n",
      "[1] TRUE\n",
      "\n",
      "$`moz:processID`\n",
      "[1] 22174\n",
      "\n",
      "$`moz:profile`\n",
      "[1] \"/run/user/1000/rust_mozprofile5mFuws\"\n",
      "\n",
      "$`moz:shutdownTimeout`\n",
      "[1] 60000\n",
      "\n",
      "$`moz:useNonSpecCompliantPointerOrigin`\n",
      "[1] FALSE\n",
      "\n",
      "$`moz:webdriverClick`\n",
      "[1] TRUE\n",
      "\n",
      "$pageLoadStrategy\n",
      "[1] \"normal\"\n",
      "\n",
      "$platformName\n",
      "[1] \"linux\"\n",
      "\n",
      "$platformVersion\n",
      "[1] \"4.19.116-hardened\"\n",
      "\n",
      "$rotatable\n",
      "[1] FALSE\n",
      "\n",
      "$setWindowRect\n",
      "[1] TRUE\n",
      "\n",
      "$strictFileInteractability\n",
      "[1] FALSE\n",
      "\n",
      "$timeouts\n",
      "$timeouts$implicit\n",
      "[1] 0\n",
      "\n",
      "$timeouts$pageLoad\n",
      "[1] 300000\n",
      "\n",
      "$timeouts$script\n",
      "[1] 30000\n",
      "\n",
      "\n",
      "$unhandledPromptBehavior\n",
      "[1] \"dismiss and notify\"\n",
      "\n",
      "$webdriver.remote.sessionid\n",
      "[1] \"e040dafc-9472-4c2f-868d-6fa23bcd40f4\"\n",
      "\n",
      "$id\n",
      "[1] \"e040dafc-9472-4c2f-868d-6fa23bcd40f4\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "remDr <- remoteDriver(\n",
    "  remoteServerAddr = \"localhost\", port = 4445L, browserName = \"firefox\",\n",
    "  extraCapabilities = list(\n",
    "    `moz:firefoxOptions` = list(args = list(\"--headless\"))\n",
    "  )\n",
    ")\n",
    "remDr$open()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9549b311-2dcc-4d56-8526-e3e00081dafa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol>\n",
       "\t<li>'(((((((((((signaling pathway) OR signalling pathway) OR regulatory pat - PMC - NCBI'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate}\n",
       "\\item '(((((((((((signaling pathway) OR signalling pathway) OR regulatory pat - PMC - NCBI'\n",
       "\\end{enumerate}\n"
      ],
      "text/markdown": [
       "1. '(((((((((((signaling pathway) OR signalling pathway) OR regulatory pat - PMC - NCBI'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[[1]]\n",
       "[1] \"(((((((((((signaling pathway) OR signalling pathway) OR regulatory pat - PMC - NCBI\"\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## go to query result\n",
    "remDr$navigate(query.url)\n",
    "# confirm you got there remDr$screenshot(display = TRUE)\n",
    "remDr$getTitle()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4b3958e2-e371-470b-aeb0-9da49fb02b54",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "1246"
      ],
      "text/latex": [
       "1246"
      ],
      "text/markdown": [
       "1246"
      ],
      "text/plain": [
       "[1] 1246"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Collect all pages!\n",
    "df.all <- data.frame(\n",
    "  figid = character(),\n",
    "  pmcid = character(),\n",
    "  filename = character(),\n",
    "  number = character(),\n",
    "  figtitle = character(),\n",
    "  papertitle = character(),\n",
    "  caption = character(),\n",
    "  figlink = character(), reftext = character()\n",
    ")\n",
    "\n",
    "page.count <- xml2::read_html(remDr$getPageSource()[[1]]) %>%\n",
    "  rvest::html_nodes(\".title_and_pager\") %>%\n",
    "  rvest::html_node(\".pagination\") %>%\n",
    "  rvest::html_nodes(\"a\") %>%\n",
    "  rvest::html_attr(\"page\")\n",
    "page.count <- as.integer(page.count[4])\n",
    "page.count"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee6d77d8-b00c-4de2-bb67-049fc622e1e1",
   "metadata": {},
   "source": [
    "If the following cell gives an error, do the following:\n",
    "1. set `restarting_where_left_off` to `TRUE`\n",
    "2. set `start_page` to reflect the last page completed\n",
    "3. run this cell and all below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9d36bf26-cf7f-4249-9e2f-0ef23338f833",
   "metadata": {},
   "outputs": [],
   "source": [
    "restarting_where_left_off <- FALSE\n",
    "if (restarting_where_left_off) {\n",
    "  remDr$goBack()\n",
    "  start_page <- 402\n",
    "} else {\n",
    "  start_page <- 1\n",
    "}\n",
    "print(start_page)\n",
    "\n",
    "pages_completed_file <- file(paste0(getwd(), \"/\", \"pages_completed.txt\"))\n",
    "\n",
    "for (i in start_page:page.count) {\n",
    "\n",
    "  ## Parse page\n",
    "  page.source <- xml2::read_html(remDr$getPageSource()[[1]])\n",
    "  filename <- page.source %>%\n",
    "    rvest::html_nodes(\".rprt_img\") %>%\n",
    "    rvest::html_node(\"img\") %>%\n",
    "    rvest::html_attr(\"src-large\") %>%\n",
    "    str_match(\"bin/(.*\\\\.jpg)\") %>%\n",
    "    as.data.frame() %>%\n",
    "    select(2) %>%\n",
    "    as.matrix() %>%\n",
    "    as.character()\n",
    "  number <- page.source %>%\n",
    "    rvest::html_nodes(\".rprt_img\") %>%\n",
    "    rvest::html_node(\"img\") %>%\n",
    "    rvest::html_attr(\"alt\")\n",
    "  titles <- page.source %>%\n",
    "    rvest::html_nodes(\".rprt_img\") %>%\n",
    "    rvest::html_node(xpath = \"..\") %>%\n",
    "    rvest::html_node(\".rprt_cont\") %>%\n",
    "    rvest::html_node(\".title\") %>%\n",
    "    rvest::html_text() %>%\n",
    "    str_split(\"\\\\s+From: \", simplify = TRUE)\n",
    "  titles_text <- page.source %>%\n",
    "    rvest::html_nodes(\".rprt_img\") %>%\n",
    "    rvest::html_node(xpath = \"..\") %>%\n",
    "    rvest::html_node(\".rprt_cont\") %>%\n",
    "    rvest::html_node(\".title\") %>%\n",
    "    rvest::html_text()\n",
    "  # print(titles_text)\n",
    "  # print(titles)\n",
    "  papertitle <- titles[, 2] %>% str_trim()\n",
    "  caption <- page.source %>%\n",
    "    rvest::html_nodes(\".rprt_img\") %>%\n",
    "    rvest::html_node(xpath = \"..\") %>%\n",
    "    rvest::html_node(\".rprt_cont\") %>%\n",
    "    rvest::html_node(\".supp\") %>%\n",
    "    rvest::html_text()\n",
    "  figlink <- page.source %>%\n",
    "    rvest::html_nodes(\".rprt_img\") %>%\n",
    "    rvest::html_attr(\"image-link\")\n",
    "  reftext <- page.source %>%\n",
    "    rvest::html_nodes(\".rprt_img\") %>%\n",
    "    rvest::html_node(xpath = \"..\") %>%\n",
    "    rvest::html_node(\".rprt_cont\") %>%\n",
    "    rvest::html_node(\".aux\") %>%\n",
    "    rvest::html_text() %>%\n",
    "    str_remove(fixed(\"CitationFull text\"))\n",
    "  pmcid <- page.source %>%\n",
    "    rvest::html_nodes(\".rprt_img\") %>%\n",
    "    rvest::html_node(xpath = \"..\") %>%\n",
    "    rvest::html_node(\".rprt_cont\") %>%\n",
    "    rvest::html_node(\".title\") %>%\n",
    "    rvest::html_node(\"a\") %>%\n",
    "    rvest::html_attr(\"href\") %>%\n",
    "    str_match(\"PMC\\\\d+\") %>%\n",
    "    as.character()\n",
    "\n",
    "  ## Extract best figure title from analysis of provided, number, title and caption\n",
    "  temp.df <- data.frame(n = number, t = titles[, 1], c = caption, stringsAsFactors = FALSE) %>%\n",
    "    mutate(t = str_trim(str_remove(\n",
    "      t, fixed(\n",
    "        as.character(\n",
    "          if_else(\n",
    "            number != \"\",\n",
    "            number,\n",
    "            \"a string just to suppress the empty search patterns warning message\"\n",
    "          )\n",
    "        )\n",
    "      )\n",
    "    ))) %>%\n",
    "    mutate(c = if_else(is.na(c), t, c)) %>%\n",
    "    mutate(t = str_trim(str_remove(\n",
    "      t,\n",
    "      \"\\\\.$\"\n",
    "    ))) %>%\n",
    "    mutate(t = if_else(t == \"\", c, t)) %>%\n",
    "    mutate(t = if_else(!is.na(str_match(\n",
    "      t,\n",
    "      \"\\\\. .*\"\n",
    "    )), str_remove(t, \"\\\\. .*\"), t)) %>%\n",
    "    mutate(t = str_trim(str_remove(\n",
    "      t,\n",
    "      \"\\\\.+$\"\n",
    "    ))) %>%\n",
    "    mutate(t = str_trim(str_remove(t, \"^\\\\.\"))) %>%\n",
    "    mutate(c = str_trim(str_replace(\n",
    "      c,\n",
    "      \"\\\\.\\\\.\", \"\\\\.\"\n",
    "    ))) %>%\n",
    "    mutate(n = str_trim(str_replace(n, \"\\\\.$\", \"\")))\n",
    "  number <- as.character(temp.df[, 1])\n",
    "  figtitle <- as.character(temp.df[, 2])\n",
    "  caption <- as.character(temp.df[, 3])\n",
    "\n",
    "  ## Prepare df and write to R.object and tsv\n",
    "  df <- data.frame(\n",
    "    pmcid, filename, number, figtitle, papertitle, caption, figlink,\n",
    "    reftext\n",
    "  ) %>%\n",
    "    mutate(figid = paste(pmcid, filename, sep = \"__\")) %>%\n",
    "    select(\n",
    "      figid,\n",
    "      pmcid, filename, number, figtitle, papertitle, caption, figlink, reftext\n",
    "    )\n",
    "\n",
    "  df.all <- rbind(df.all, df)\n",
    "\n",
    "  write.table(df,\n",
    "    file = \"pmc.df.all.tsv\", append = TRUE, sep = \"\\t\", quote = TRUE,\n",
    "    col.names = FALSE, row.names = FALSE, fileEncoding = \"UTF-8\"\n",
    "  )\n",
    "\n",
    "  if (i < page.count - 1) {\n",
    "    next.page.button <- remDr$findElement(using = \"xpath\", \"//*[@class='active page_link next']\")\n",
    "    next.page.button$clickElement()\n",
    "    # remDr$screenshot(display = TRUE)\n",
    "  }\n",
    "\n",
    "  writeLines(c(paste(i, \"of\", page.count)), pages_completed_file)\n",
    "  # print(paste(i, \"of\", page.count))\n",
    "}\n",
    "\n",
    "close(pages_completed_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "18aa092c-bc6a-450d-b808-09cbbbb155eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "## At the end of the day...\n",
    "df.all <- unique(df.all)\n",
    "saveRDS(df.all, file = \"pmc.df.all.rds\")\n",
    "# df.all <- readRDS('pmc.df.all.rds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "365d8e26-7e1e-4c2e-94c1-188897055dd0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol>\n",
       "\t<li>NULL</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate}\n",
       "\\item NULL\n",
       "\\end{enumerate}\n"
      ],
      "text/markdown": [
       "1. NULL\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[[1]]\n",
       "NULL\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "TRUE"
      ],
      "text/latex": [
       "TRUE"
      ],
      "text/markdown": [
       "TRUE"
      ],
      "text/plain": [
       "[1] TRUE"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "TRUE"
      ],
      "text/latex": [
       "TRUE"
      ],
      "text/markdown": [
       "TRUE"
      ],
      "text/plain": [
       "[1] TRUE"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "FALSE"
      ],
      "text/latex": [
       "FALSE"
      ],
      "text/markdown": [
       "FALSE"
      ],
      "text/plain": [
       "[1] FALSE"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Close up shop\n",
    "remDr$closeall()\n",
    "\n",
    "proc$is_alive()\n",
    "proc$kill()\n",
    "proc$is_alive()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2618cc6-7e2c-4b4b-8de5-cd1e858a3cae",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R - mypkgs",
   "language": "R",
   "name": "ir_mypkgs"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.0.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}