Skip to content

Commit 0f00a6f

Browse files
bnavacbnavac
authored
801 Rewrite Webscraper to use SIS (#849)
1 parent da80d9b commit 0f00a6f

14 files changed

+898
-7
lines changed

‎rpi_data/modules/READEME.md

+90
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Package requirements
2+
pandas
3+
bs4
4+
requests
5+
lxml
6+
regex
7+
pyyaml
8+
selenium
9+
10+
# Course Parser
11+
Hopefully this will be the last one.
12+
The relevant files in the folder are csv_to_course.py, course.py, headless_login.py, new_parse.py, and parse_runner.py
13+
The other files in the folder are legacy code that were used in the old web scraper. For now they will remain in here as there are some pieces of code that could be useful if future edge cases pop up.
14+
15+
# How to run
16+
run parse_runner.py with a term as the command argument. The term is formatted as termYEAR. If the specified csv doesn't exist it'll do a full parse, if it does it'll immediately start updating.
17+
18+
# Common issues with SIS scraping
19+
20+
------------------------------------------------------------------------------------------------------------------------------------------
21+
Sel | CRN | Maj | Cod | Sec | Cmp | Crd | Nme | Dys | Tme | Cap | Act | Rem | WLC | WLA | WLR | XLC | XLA | XLR | Prof | Date | Loc | Attr
22+
------------------------------------------------------------------------------------------------------------------------------------------
23+
SR |99341|CSCI |1100 |01 | T | 4.00| CSI | MR |12-150| 24 | 3 | 21 | 0 | 0 | 0 | 0 | 0 | 0 |Stur |01-04 |TBA |Intro
24+
------------------------------------------------------------------------------------------------------------------------------------------
25+
While some details have been truncated to fit, this is an example of what we expect a course to be from sis. And for the most part, many courses on sis follow this format.
26+
But, SIS is not perfect, and there are often many mistakes in courses.
27+
The first main one (though this is moreso a design decision than a mistake) is that many parts of courses may be empty, for example
28+
------------------------------------------------------------------------------------------------------------------------------------------
29+
| | | | | | | | T |10-1150| | | | | | | | | |TBA |01-04 |TBA |Intro
30+
------------------------------------------------------------------------------------------------------------------------------------------
31+
This is the lab block for the above cs1 course. As you may notice, most of the details are missing, and so it is impossible to build out a course just from this information.
32+
However, since this appears directly below the cs1 lecture block in sis, we will parse this course immedieatly after parsing the lecture block.
33+
So, we keep a copy of the previous course that we parsed, and use that to fill in information about lab and test blocks.
34+
35+
Another common issue is the use of colspan, for example, Biomed 6940 in spring 2024, which looks like this in SIS
36+
------------------------------------------------------------------------------------------------------------------------------------------
37+
SR |90453|BMED |6940 |01 | T | 1-9 | REB | TBA | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |TBA |01-04 |TBA |
38+
------------------------------------------------------------------------------------------------------------------------------------------
39+
However, is parsed as
40+
------------------------------------------------------------------------------------------------------------------------------------------
41+
SR |90453|BMED |6940 |01 | T | 1-9 | REB | TBA | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |TBA |01-04 |TBA |
42+
-----------------------------------------------------------------------------------------------------------------------------------------
43+
because of the use of colspan in the days column. This means that our indecies are off when we start formatting and processing stuff, which crashes the web scraper. We get around this by inserting a TBA for the value of the colspan that we see.
44+
45+
We also split some fields into two different fields, namely the start and end times a class (eg. 2:00pm - 3:50pm) and the start and end dates of the classes (eg. 01/08-04/24).
46+
However there are often times where courses may have these fields as TBA or blank, eg admin 1030 which looks like this
47+
48+
------------------------------------------------------------------------------------------------------------------------------------------
49+
SR |93972|ADMN |1030 |01 | T | 0.00 | AXPA | | TBA |1000| 443| 557| 0 | 0 | 0 | 0 | 0 | 0 |Cary |01-04 |TBA |
50+
------------------------------------------------------------------------------------------------------------------------------------------
51+
52+
As you can see, the time field is TBA, which we can't easily divide into two seperate fields. At the moment we just add in two TBA's in this case. However in the event that another web scraper is needed or it breaks, this may be a source of failure.
53+
54+
These are the some of the more common offenders, but other issues can pop up, so generally, a row should always have exactly 21 things in it before we begin processing. Many issues that pop up with the webscraper are related to the rows not having a length of 21.
55+
56+
# Common Issues with catalog scraping
57+
58+
Unfortunately, SIS scraping is relatively simple compared to catalog scraping, which has many issues.
59+
Though most of these issues will probably (hopefully) dissapear when the catalog api is implemented.
60+
However I am under the assumption that speedup is all we can except from the api.
61+
So, SIS will not give us everything that we want, in particular the prerequsites and corequisites of a course, in order to get that we will need to scrap from the catalog, in particular, this link
62+
https://sis.rpi.edu/rss/bwckctlg.p_disp_course_detail?cat_term_in=?&subj_code_in=?&crse_numb_in=?
63+
Where you would replace the ?'s with a basevalue (the integer representation of a semester - Spring 2024 -> 202401), Major, and course code.
64+
65+
Because there is notablly less information to parse, there are less issues with the catalog at present, though some of the issues are more severe.
66+
![Alt text](image-1.png)
67+
68+
This is as close to the ideal course that one can find, there is a clear list of prerequisites and corequisites, as well as a description. (Though there is a slight issues where it's listed as "Prerequisites/Corequisites: Prerequisite:" instead of "Prerequisites/Corequisites: Prerequisites:" like other courses, but it's pretty good beyond that).
69+
70+
However, there are many courses that do not follow this, for example,
71+
72+
![Alt text](image-3.png)
73+
74+
Even though capstone is listed as having prerequsites or corequisites, it only has prerequisites, and that is difficult for a computer to distinguish, namely because it is missing the "Prerequites:" or "Corequisites" that other courses will have. For reasons that will be mentioned below, this is not too big of an issue with prerequisites, as there is a consistent way to get those, but getting corequisites consistently is difficult.
75+
76+
![Alt text](image-2.png)
77+
78+
This is another case of weird prerequisite and corequisite formatting, where is it difficult to parse the two.
79+
80+
![Alt text](image-4.png)
81+
82+
This is RCOS for next semester, however if you did not already know the course code, it would be very impossible to tell that this was RCOS. So, when parsed, there will be no prerequisites, corequisites, or description for the course, even though this is not actually true.
83+
84+
It is worth mentioning that there are two prerequisites, one called prerequisites, ie "Prerequisites/Corequisites: Prerequisites: CSCI 1200 and Introduction to Calculus (MATH 1010 or MATH 1500 or MATH 1020 or MATH 2010); MATH 1020 is strongly recommended.", and another called raw, or raw prerequistes in the database. Raw is
85+
86+
![Alt text](image-5.png)
87+
88+
In the webpage. When you click on a course in the explore page, this is the information that is displayed as the prerequisites of a course. Notable, raw is actually reliable and so many of the issues with prerequistes and corequisites mentioned here are mostly dealing with corequisites.
89+
90+
However, aside from raw, all of the other situtations are unique problems that do not have, or have limited solutions in the webscraper, this is especially true with the corequisite problem.

‎rpi_data/modules/course.py

+160
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
import time
2+
import pdb
3+
import copy
4+
from typing import overload
5+
class Course:
6+
name:str
7+
credits:int
8+
days:str
9+
stime:int
10+
etime:int
11+
profs:str
12+
loc:str
13+
max:int
14+
curr:int
15+
rem:int
16+
dept:str
17+
sdate:str
18+
enddate:str
19+
sem:str
20+
crn:int
21+
code:int
22+
section:int
23+
short:str
24+
long:str
25+
frequency:str
26+
desc:str
27+
raw:str
28+
pre:list
29+
co:list
30+
major:str
31+
school:str
32+
lec:str
33+
#Info will be an array of strings:
34+
# [crn, major, code, section, credits, name, days, stime, etime, max, curr, rem, profs, sdate, enddate, loc]
35+
36+
def __init__(self, info):
37+
self.crn = info[0]
38+
self.major = info[1]
39+
self.code = info[2]
40+
self.section = info[3]
41+
self.credits = info[4]
42+
self.name = info[5]
43+
self.days = info[6]
44+
self.stime = info[7]
45+
self.etime = info[8]
46+
self.max = info[9]
47+
self.curr = info[10]
48+
self.rem = info[11]
49+
self.profs = info[12]
50+
self.sdate = info[13]
51+
self.enddate = info[14]
52+
self.loc = info[15]
53+
self.long = self.processName(self.name)
54+
self.frequency = ""
55+
self.short = self.major + '-' + self.code
56+
self.lec = "LEC"
57+
self.desc = ""
58+
self.raw = ""
59+
self.pre = list()
60+
self.co = list()
61+
self.school = ""
62+
self.sem = ""
63+
64+
def processName(self, name:str) -> str:
65+
tmp = name.split()
66+
for i in range(0, len(tmp), 1):
67+
if not tmp[i].isalpha():
68+
continue
69+
tmp[i]= tmp[i][:1].upper() + tmp[i][1:].lower()
70+
return ' '.join(tmp)
71+
def addSemester(self, semester):
72+
self.sem = semester.upper()
73+
def addReqs(self, pre:list=[], co:list=[], raw:str="", desc: str=""):
74+
self.desc = desc
75+
self.raw = raw
76+
self.pre = copy.deepcopy(pre)
77+
self.co = copy.deepcopy(co)
78+
79+
def addReqsFromList(self, info: list=[]):
80+
self.pre = info[0]
81+
self.co = info[1]
82+
self.raw = info[2]
83+
self.desc = info[3]
84+
def print(self):
85+
for attr, value in self.__dict__.items():
86+
print(attr, " : ", value)
87+
#Turn the class back into a list.
88+
#Because of the diffs in how we store vs how we want it to be, need to do a lot of swapping
89+
#Maybe there's a diff way than doing this, hopefully there is
90+
def decompose(self) -> list[str]:
91+
retList = []
92+
retList.append(self.name)
93+
retList.append(self.lec)
94+
retList.append(self.credits)
95+
retList.append(self.days)
96+
retList.append(self.stime)
97+
retList.append(self.etime)
98+
retList.append(self.profs)
99+
retList.append(self.loc)
100+
retList.append(self.max)
101+
retList.append(self.curr)
102+
retList.append(self.rem)
103+
retList.append(self.major)
104+
retList.append(self.sdate)
105+
retList.append(self.enddate)
106+
retList.append(self.sem)
107+
retList.append(self.crn)
108+
retList.append(self.code)
109+
retList.append(self.section)
110+
retList.append(self.short)
111+
retList.append(self.long)
112+
retList.append(self.desc)
113+
retList.append(self.raw)
114+
retList.append(self.frequency)
115+
retList.append(self.pre)
116+
retList.append(self.co)
117+
retList.append(self.school)
118+
return retList
119+
120+
def list_to_class(self, row):
121+
self.name = row[0]
122+
self.lec = row[1]
123+
self.credits = row[2]
124+
self.days = row[3]
125+
self.stime = row[4]
126+
self.etime = row[5]
127+
self.profs = row[6]
128+
self.loc = row[7]
129+
self.max = row[8]
130+
self.curr = row[9]
131+
self.rem = row[10]
132+
self.major = row[11]
133+
self.sdate = row[12]
134+
self.enddate = row[13]
135+
self.sem = row[14]
136+
self.crn = row[15]
137+
self.code = row[16]
138+
self.section = row[17]
139+
self.short = row[18]
140+
self.long = row[19]
141+
self.desc = row[20]
142+
self.raw = row[21]
143+
self.pre = row[23]
144+
self.co = row[24]
145+
self.school = row[25]
146+
147+
def addSchool(self, school):
148+
self.school = school
149+
def __lt__(self, other):
150+
#Note that we will maybe need to compare times? Idk how to handle the case where the classes
151+
#are the same (ie lab, lecture, test) so at the moment the lecture appears last.
152+
# So far we just sort in reverse order.
153+
if self.major > other.major:
154+
return self.major > other.major
155+
if self.code > other.code:
156+
return self.code > other.code
157+
return self.section > other.section
158+
159+
def __str__(self):
160+
return self.name

‎rpi_data/modules/csv_to_course.py

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
import csv
2+
from course import Course
3+
import os
4+
import pdb
5+
__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
6+
7+
# This file takes our csv formatting and turns it into a course class type. If something goes wrong it's because you changed one of those two.
8+
9+
def parse_csv(filename):
10+
courses = list()
11+
i = 0
12+
with open(os.path.join(__location__, filename), 'r', encoding="utf8") as f:
13+
reader = csv.reader(f)
14+
for row in reader:
15+
if i == 0:
16+
i += 1
17+
continue
18+
temp = Course(["" for _ in range(16)])
19+
temp.name = row[0]
20+
temp.lec = row[1]
21+
temp.credits = row[2]
22+
temp.days = row[3]
23+
temp.stime = row[4]
24+
temp.etime = row[5]
25+
temp.profs = row[6]
26+
temp.loc = row[7]
27+
temp.max = row[8]
28+
temp.curr = row[9]
29+
temp.rem = row[10]
30+
temp.major = row[11]
31+
temp.sdate = row[12]
32+
temp.enddate = row[13]
33+
temp.sem = row[14]
34+
temp.crn = row[15]
35+
temp.code = row[16]
36+
temp.section = row[17]
37+
temp.short = row[18]
38+
temp.long = row[19]
39+
temp.desc = row[20]
40+
temp.raw = row[21]
41+
#empty column here
42+
temp.pre = row[23]
43+
temp.co = row[24]
44+
temp.school = row[25]
45+
courses.append(temp)
46+
return courses

‎rpi_data/modules/headless_login.py

+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
import selenium as sel
2+
from selenium import webdriver
3+
from selenium.webdriver.common.by import By
4+
from selenium.webdriver.support.ui import Select
5+
from selenium.webdriver.firefox.options import Options
6+
from selenium.webdriver.support.ui import WebDriverWait
7+
from selenium.webdriver.support import expected_conditions as EC
8+
import time
9+
import os
10+
import sys
11+
12+
# Remember to add enviromental variables named rcsid and rcspw with your account info!!!
13+
#
14+
# THINGS THAT CAN POTENTIALLY GO WRONG HERE AND HOW TO FIX THEM:
15+
#
16+
# - If the RPI login website changes at all, it's very likely that the login will break. Fixing might involve changing what element selenium looks for.
17+
# - DUO likes to change things. If they implement another 2FA type or add extra buttons for some reason you'll have to add more checks and button presses
18+
# - Selenium errors can occur if your internet is slow or if you have multiple browser instances open, so try to avoid this
19+
#
20+
# - You need to install firefox (I hate Google Chrome, and you should too). If you change it to be a Chrome instance instead it probably won't work from my experience
21+
# - To fix these things you can comment this line: "options.add_argument("--headless")" in the parse_runner file to see what goes wrong if python doesn't throw anything
22+
# - Try restarting python/vscode or even your computer if it's throwing something really weird for no reason
23+
# - You can try sending me a message on discord @gcm as a last resort
24+
25+
26+
def login(driver):
27+
URL = "http://sis.rpi.edu"
28+
driver.get(URL) # uses a selenium webdriver to go to the sis website, which then redirects to the rcs auth website
29+
username_box = driver.find_element(by=By.NAME, value = "j_username") # creates a variable which contains an element type, so that we can interact with it, j_username is the username text box
30+
password_box = driver.find_element(by=By.NAME, value = "j_password") # j_password is the password box
31+
submit = driver.find_element(by=By.NAME, value = "_eventId_proceed") # _eventId_proceed is the submit button
32+
username = os.environ.get("rcsid", "NONEFOUND")
33+
password = os.environ.get("rcspw", "NONEFOUND")
34+
if (username == "NONEFOUND" or password == "NONEFOUND"):
35+
print("username or password not found, check environment variables or input them manually")
36+
username = input("Enter username: ")
37+
password = input("Enter password: ")
38+
username_box.send_keys(username) # enters the username
39+
password_box.send_keys(password) # enters the password
40+
submit.click() # click the submit button
41+
while ("duosecurity" not in driver.current_url): # if you entered details incorrectly, the loop will be entered as you aren't on the duo verfication website (redo what we did before)
42+
print("User or Password Incorrect.")
43+
username_box = driver.find_element(by=By.NAME, value = "j_username") # we have to redefine the variables because the webpage reloads
44+
password_box = driver.find_element(by=By.NAME, value = "j_password")
45+
submit = driver.find_element(by=By.NAME, value = "_eventId_proceed")
46+
username = input("Enter Username: ")
47+
password = input("Enter Password: ")
48+
username_box.clear() # the username box by default has your previous username entered, so we clear it
49+
username_box.send_keys(username)
50+
password_box.send_keys(password)
51+
submit.click()
52+
while len(driver.find_elements(By.XPATH, '/html/body/div/div/div[1]/div/div[2]/div[7]/a'))==0:
53+
time.sleep(.1)
54+
options = driver.find_element(By.XPATH, '/html/body/div/div/div[1]/div/div[2]/div[7]/a')
55+
options.click()
56+
while len(driver.find_elements(By.XPATH, '/html/body/div/div/div[1]/div/div[1]/ul/li[1]/a')) == 0:
57+
time.sleep(.1)
58+
duo_option = driver.find_element(By.XPATH, '/html/body/div/div/div[1]/div/div[1]/ul/li[1]/a')
59+
duo_option.click()
60+
while len(driver.find_elements(By.XPATH, '/html/body/div/div/div[1]/div/div[2]/div[3]')) == 0:
61+
time.sleep(.1)
62+
print("Your DUO code: "+ driver.find_element(by= By.XPATH, value = "/html/body/div/div/div[1]/div/div[2]/div[3]").text) # print the duo code
63+
while len(driver.find_elements(By.XPATH, '//*[@id="trust-browser-button"]'))==0: # we need to press the trust browser button, so we wait until that shows up
64+
time.sleep(.1)
65+
trust_button = driver.find_element(By.XPATH, '//*[@id="trust-browser-button"]') #find and click it
66+
trust_button.click()
67+
time.sleep(3)
68+
if (driver.current_url == "https://sis.rpi.edu/rss/twbkwbis.P_GenMenu?name=bmenu.P_MainMnu"): # check if we're in the right place
69+
return "Success"
70+
else:
71+
print("login failed")
72+
return "Failure"

‎rpi_data/modules/image-1.png

124 KB
Loading

‎rpi_data/modules/image-2.png

130 KB
Loading

‎rpi_data/modules/image-3.png

139 KB
Loading

‎rpi_data/modules/image-4.png

27 KB
Loading

‎rpi_data/modules/image-5.png

16.5 KB
Loading

‎rpi_data/modules/image.png

141 KB
Loading

0 commit comments

Comments
 (0)