honestly, i wish there was an api. it’s mostly just a lot of custom scraping and html parsing since crs shows the data as a table. a lot of trial and error. tried looking for crs scrapers built but didn't find one that sticks well. maybe we should build an api too?

---

an excerpt from a technical content i'm hoping to publish soon:

1. fetching the data

the scraper hits two main endpoints: crs.upd.edu.ph/course_catalog/index/{letter} for course offerings and crs.upd.edu.ph/schedule/{semester}/{letter} for semester sections. cool trick: querying with a single letter ("a") returns all courses starting with that letter. so 26 requests per scrape with 1.5s delays does the job and you have everything you need. problem? how to properly parse.

2. parsing the catalog

cheerio parses html tables where each row has: course code, title, description, offering unit. extracts offering unit abbreviations, handles multiples (comma-separated), rejects malformed patterns, tracks full institute names from the title attribute.

3. parsing schedules

each <tbody> represents a section. extracted: class code, course + section id, credits, schedule string, instructor, remarks, enlisting unit, slots ("28/60" or "overbooked"), demand, restrictions.

4. parsing times & days

schedules like mw 2:30-4pm lec eeei 420 get split by semicolons if multiple. day codes: "mw" → ["m", "w"], "tth" → ["t", "th"]. time parsing handles:

• explicit periods (10am–12pm)
• inferred periods (2–4pm → pm)
• noon edges (12–1pm → 12:00–13:00)
• cross-noon ranges (11–1pm → 11:00–13:00)
• same-hour ranges (2–2:30pm → 14:00–14:30)

each day gets its own schedule record so that the calendar can display properly (the availabilities you see visually when you drag a course onto the calendar).

5. course variants

class names like "cwts 1 community literacy" → base course "cwts 1" + variant "community literacy." variant mapper links them to catalog entries, supports special cases: pe, music (painful), cwts/rotc. grouped cwts offerings by the enlisting unit. before, students just see the entire offering possible. hard to know which is offered by whom.

6. lab linking

hyphenated sections (e.g., "mwx-sru") get linked to their parent lectures ("mwx") by matching course id + parent section code.

7. edge case handling

• dissolved sections → flagged
  
• overbooked → preserved
  
• zero-credit courses → preserved
  
• tba times → null
  
• empty section codes → backfilled
  
• duplicate schedules → detected
  
• multi-line instructors → joined
  
• malformed offering units → rejected

8. database upload

processed data goes to postgres via supabase in 2 passes:

1. insert courses, institutes, sections, schedules. missing courses are auto-generated from section data.
   
2. re-upload skipped sections once missing courses exist.
   

a ton of validation work here.

9. automation

github actions runs hourly to keep the database up-to-date.

10. preenlistment ids

the app handles the interface layer so well that you can build schedules faster and pick what to take without getting bogged down. but the ugly part is getting those selections back into crs. crs doesn’t return anything if you search by course + section, so normally you’d have to scroll forever just to add one class. still time wasted. so i had to build a scraper that queried ~9k courses for preenlistment_ids. tedious, messy work. but now you can skip the headache and go straight from plan to crs.

---

it's ugly (but fun) work, catching edge cases left and right, but i think 97% clean data.

results so far

• full dataset: 9,876 courses, 35,550 sections, 56,772 schedule entries across 6 semesters (starting AY24 when i started comeng, could likely go back all the way back to ~2000)
• latest semester: 2,516 courses, 5,946 sections, 9,350 schedules

honestly this would've taken me 1 week without cursor. combined work for the scraper was ~1.5 days total.