Skip to content

Commit 5f7b358

Browse files
Ammaar-AlamAmmaar-Alam
committed
updating scraping readme as requested
1 parent 0fe4cf7 commit 5f7b358

1 file changed

Lines changed: 210 additions & 68 deletions

File tree

‎server/scraping/README.md‎

Lines changed: 210 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -1,68 +1,210 @@
1-
# Princeton Course Evaluation Scraper for TigerType
2-
*[AI DISCLAIMER: THIS DOCUMENTATION WAS CREATED WITH THE HELP OF AI]*
3-
4-
This directory contains the script and documentation for scraping course evaluations from the official Princeton University course evaluation website. The goal is to extract engaging text snippets for use in the TigerType application.
5-
6-
## Process Overview
7-
8-
1. **Scraping:** The `scrape_evals.js` script (adapted from an existing tool) logs into the Princeton course evaluation system using a user-provided `PHPSESSID` cookie and scrapes evaluation data, including comments, for specified courses.
9-
2. **Data Extraction & Formatting:** The script needs modification to extract not just the comments but also associated metadata like course name, course ID, term, and potentially evaluation scores (though comments are the priority for snippets). The output format must be changed from database insertion to JSON output. Each JSON object should represent a single comment/review.
10-
3. **AI Filtering:** A separate AI processing step (to be implemented later) will analyze the raw scraped JSON data. This AI agent will:
11-
* Read the comments in batches.
12-
* Identify comments that are suitable as typing snippets (considering length, clarity, and entertainment value). Humorous or particularly insightful comments are preferred.
13-
* Filter out very short, nonsensical, or unhelpful comments (e.g., "good", "N/A", numerical scores mistaken for comments).
14-
* Add metadata to the selected snippets, including `word_count` and `character_count`.
15-
* Optionally, categorize snippets (e.g., `positive`, `negative`, `funny`, `long`).
16-
4. **Final Output:** The AI process will produce a final JSON file containing only the curated, high-quality snippets, ready to be imported into TigerType's text snippet database.
17-
18-
## `scrape_evals.js` Modifications Required
19-
20-
The provided `scrape_evals.js` script needs the following changes:
21-
22-
1. **Remove Database Logic:** All MongoDB-related code (`require('../models/...)`, `courseModel`, `evaluationModel`, database connection logic, `findOneAndUpdate`, `update`, `deleteMany`) must be removed. The script should not interact with any database.
23-
2. **JSON Output:** Instead of saving to a database, the script should collect the scraped data into a JavaScript array of objects. Once all courses are processed, this array should be written to a JSON file (e.g., `raw_evaluations.json`).
24-
3. **Data Structure:** Each object in the output JSON array should represent a single comment and include at least the following fields:
25-
* `course_id`: The course identifier (e.g., "COS333").
26-
* `course_name`: The full name of the course (needs to be fetched or joined from course data). *Initially, the script only has courseID. Modification needed to potentially fetch/map this.*
27-
* `term`: The semester ID (e.g., "1244" for Spring 2024).
28-
* `comment_text`: The raw text of the student comment.
29-
* `evaluation_url`: The URL from which the comment was scraped (`https://registrarapps.princeton.edu/course-evaluation?terminfo=[term]&courseinfo=[courseID]`).
30-
* *(Optional but helpful)* `scores`: The quantitative scores associated with the evaluation (if available and easily parsed).
31-
4. **Fetch Course Names:** The original script assumes course data (like names) is available via the `courseModel`. Since we are removing database interaction, the script might need modification to either:
32-
* Accept a pre-existing mapping of `courseID` to `course_name`.
33-
* Perform an additional scraping step or lookup if necessary (though this might complicate the script). *Simplest approach initially might be to just output `course_id` and handle name mapping later.*
34-
5. **Error Handling:** Improve error handling for network issues or changes in the registrar's website structure.
35-
6. **Dependencies:** Ensure `package.json` includes `cheerio`, `request`, `promptly`, `colors`, and `dotenv`. Remove database-related dependencies.
36-
37-
## Running the Scraper
38-
39-
1. **Install Dependencies:**
40-
```bash
41-
cd server/scraping
42-
# (You might need to create a package.json first: npm init -y)
43-
npm install cheerio request promptly colors dotenv
44-
```
45-
2. **Get Session Cookie:**
46-
* Log in to [https://registrarapps.princeton.edu/course-evaluation](https://registrarapps.princeton.edu/course-evaluation).
47-
* Open browser developer tools (Inspect Element).
48-
* Go to the "Application" (or "Storage") tab.
49-
* Find the Cookies section for `registrarapps.princeton.edu`.
50-
* Copy the value of the `PHPSESSID` cookie.
51-
3. **Run the Script (non-interactive examples):**
52-
- Full subject for a term:
53-
```bash
54-
TERM=1252 SUBJECT=COS PHPSESSID=YOUR_PHPSESSID PRINCETON_API_KEY=YOUR_OIT_BEARER \
55-
node scrape_evals.js
56-
```
57-
- Single course via combined code (term+course):
58-
```bash
59-
PHPSESSID=YOUR_PHPSESSID PRINCETON_API_KEY=YOUR_OIT_BEARER \
60-
node scrape_evals.js --course 1262002051
61-
```
62-
(The script derives `TERM=1262` and course `002051` automatically.)
63-
- Single course via 5–6 digit course id (requires term):
64-
```bash
65-
TERM=1262 PHPSESSID=YOUR_PHPSESSID PRINCETON_API_KEY=YOUR_OIT_BEARER \
66-
node scrape_evals.js --course 2051
67-
```
68-
4. **Output:** The script writes `raw_evaluations.json` in `server/scraping/data/`.
1+
# TigerType Snippet Scraping Pipeline
2+
*[AI DISCLAIMER: THIS DOCUMENTATION WAS CREATED WITH THE HELP OF AI; as such, some information might not be accurate or relevant for users looking to use this code for their own apps/COS333 projects]*
3+
4+
This folder holds the tooling that turns Princeton course-evaluation comments into curated TigerType snippets. The pipeline has three stages:
5+
6+
```
7+
registrar + Student-App APIs --(scrape_evals.js)--> data/raw_evaluations.json
8+
|
9+
v
10+
(process_evals.py + OpenAI)
11+
|
12+
v
13+
data/processed_snippets.json
14+
|
15+
v
16+
(import_snippets.py -> Postgres)
17+
```
18+
19+
A GitHub Actions workflow (`.github/workflows/import-snippets.yml`) wraps these steps so maintainers can run the whole process from the Actions tab without setting up a local environment.
20+
21+
---
22+
23+
## 1. Requirements & Secrets
24+
25+
| Purpose | Variable / Input | Where it is used | Notes |
26+
|---------|------------------|------------------|-------|
27+
| Student-App API bearer token | `PRINCETON_API_KEY` or `--apikey` | `scrape_evals.js` | Request via OIT. Same token works for API and workflow. |
28+
| Registrar session cookie | `PHPSESSID` or `--sessid` | `scrape_evals.js` | Copy from browser after logging into the registrar course evaluation site. |
29+
| Registrar term code | `TERM` or `--term` | `scrape_evals.js` | Four digits (e.g. `1252`). |
30+
| Subject filter | `SUBJECT` or `--subject` | `scrape_evals.js` | Optional. Leave blank to scrape every subject in the term. |
31+
| Course selector | `COURSE` or `--course` | `scrape_evals.js` | Optional single-course scrape (either 10-digit `term+course` or 5–6 digit course id). |
32+
| OpenAI key | `OPENAI_API_KEY` | `process_evals.py` | Needed to run the AI snippet selector. |
33+
| Local Postgres creds | `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASSWORD` | `import_snippets.py --production` **not** set | Optional; use when importing into a local database. |
34+
| Production Postgres URL | `DATABASE_URL` | `import_snippets.py --production` | Provided by Heroku config or GitHub environment secret. |
35+
36+
> Tip: All scripts read the project root `.env` if present, otherwise they fall back to environment variables.
37+
38+
Dependencies:
39+
40+
- Node 18+ with `axios`, `cheerio`, `request`, `promptly`, `colors`, `dotenv` (installed via `npm install` at repo root).
41+
- Python 3.10+ with `openai`, `python-dotenv`, `psycopg2-binary` (`pip install -r requirements` style, see workflow for exact installs).
42+
43+
The scripts write all intermediate files to `server/scraping/data/` so you can safely keep everything under version control (outputs are already ignored).
44+
45+
---
46+
47+
## 2. Stage-by-Stage Guide
48+
49+
### 2.1 Scrape Course Evaluations (`scrape_evals.js`)
50+
51+
Purpose: Download every course comment and quantitative score available for a term (optionally filtered by subject or course) straight into `data/raw_evaluations.json`.
52+
53+
Run locally:
54+
55+
```bash
56+
# Example: scrape all COS evaluations for term 1252
57+
TERM=1252 SUBJECT=COS PHPSESSID=copy_from_browser \
58+
PRINCETON_API_KEY=BearerTokenFromOIT \
59+
node server/scraping/scrape_evals.js
60+
```
61+
62+
You can also pass everything as flags (helpful for automation):
63+
64+
```bash
65+
node server/scraping/scrape_evals.js \
66+
--term 1252 \
67+
--subject COS \
68+
--sessid YOUR_PHPSESSID \
69+
--apikey YOUR_OIT_BEARER
70+
```
71+
72+
A single course can be targeted with `--course`:
73+
74+
- Ten-digit combined `term+course` (e.g. `1262002051`).
75+
- Five/six digit course id (e.g. `2051`) when `--term` is provided.
76+
77+
What the script does:
78+
79+
1. Calls `https://api.princeton.edu/student-app/courses/courses?term=<TERM>` (optionally `&subject=`) to build a course list containing ids, titles, and catalog numbers.
80+
2. Visits each registrar evaluation page using `PHPSESSID` to stay logged in.
81+
3. Extracts the course name, data-bar scores, and every comment block.
82+
4. Writes an array of objects like:
83+
```json
84+
{
85+
"course_id": "002051",
86+
"term": "1252",
87+
"course_name": "COS 126: General Computer Science",
88+
"comment_text": "This class definitely put me in my lowest lows...",
89+
"evaluation_url": "https://registrarapps.princeton.edu/course-evaluation?terminfo=1252&courseinfo=002051",
90+
"scores": {
91+
"Overall Quality of Course": 4.4,
92+
"Quality of Medium": 3.9
93+
}
94+
}
95+
```
96+
97+
The script will prompt for any missing inputs and stops if the cookie expires or the API call fails (look for the friendly red errors).
98+
99+
### 2.2 Curate Snippets with AI (`process_evals.py`)
100+
101+
Purpose: Read `raw_evaluations.json`, call OpenAI to pick the funniest / most interesting snippets, and store results in `processed_snippets.json`. The script is restartable—processed items are written after every comment.
102+
103+
Run locally:
104+
105+
```bash
106+
OPENAI_API_KEY=sk-... \ # or stored in .env
107+
python3 server/scraping/process_evals.py
108+
```
109+
110+
What happens inside:
111+
112+
- Loads or creates the following files in `server/scraping/data/`:
113+
- `raw_evaluations.json` – comments waiting to be processed (the script removes items as it goes).
114+
- `processed_snippets.json` – cumulative list of curated snippets (safe to commit or inspect).
115+
- Skips obvious junk (short strings, pure numbers, "N/A").
116+
- Sends each review to `gpt-5-mini` with a strict prompt that demands high-quality, entertaining snippets and assigns an appropriate difficulty rating.
117+
- Normalizes grammar/typos lightly for readability while preserving the student's voice.
118+
- Adds metadata (`source`, `category`, `word_count`, `character_count`, and the original evaluation URL) so the importer can map back to PrincetonCourses.
119+
120+
You can interrupt (`Ctrl+C`) at any time—the script writes progress atomically using `.tmp` files and continues where it left off next run.
121+
122+
### 2.3 Import into Postgres (`import_snippets.py`)
123+
124+
Purpose: Upsert curated snippets into the `public.snippets` table used by the live TigerType app.
125+
126+
Run against your local database:
127+
128+
```bash
129+
DB_HOST=localhost DB_PORT=5432 DB_NAME=tigertype DB_USER=postgres \
130+
python3 server/scraping/import_snippets.py
131+
```
132+
133+
Run against production (uses `DATABASE_URL`, e.g. from Heroku config vars):
134+
135+
```bash
136+
DATABASE_URL=postgres://... \
137+
python3 server/scraping/import_snippets.py --production
138+
```
139+
140+
How it works:
141+
142+
1. Loads `processed_snippets.json` from `server/scraping/data/`.
143+
2. Normalizes punctuation (curly quotes → straight, em dashes → hyphen, ellipsis → `...`, removes zero-width spaces) so typing races stay ASCII.
144+
3. Recomputes `word_count`, `character_count`, and difficulty on the final text (difficulty tiers: `<100 chars = 1`, `100–185 = 2`, `>185 = 3`).
145+
4. Extracts `term_code` and `course_id` from the stored registrar URL, generating a PrincetonCourses link when possible.
146+
5. Performs a bulk `INSERT ... ON CONFLICT (text) DO NOTHING` so duplicates are skipped gracefully.
147+
148+
The script prints how many rows were prepared, inserted, or skipped because of invalid text.
149+
150+
---
151+
152+
## 3. GitHub Actions Automation
153+
154+
The workflow `.github/workflows/import-snippets.yml` lets maintainers run the full scrape → process → import sequence from the GitHub UI.
155+
156+
Trigger it via **Actions → Scrape & Import Snippets → Run workflow** and supply:
157+
158+
- `term` (required, four digits)
159+
- `subject` (optional, default `""` for all)
160+
- `course` (optional) – same semantics as the CLI `--course`
161+
- `phpsessid` (required) – paste the registrar cookie; masked in logs
162+
- `oit_api_key` (optional) – if blank, the workflow uses the `PRINCETON_API_KEY` repo/environment secret
163+
- `import_to_db` (boolean) – set to `true` to run the Python stages and import into the production DB
164+
- `environment` – choose between `tigertype` (production) and `staging`
165+
166+
Workflow stages:
167+
168+
1. Checkout repository
169+
2. Install Node dependencies (respects `package-lock.json`)
170+
3. Mask sensitive inputs in logs
171+
4. Run `scrape_evals.js` → uploads `raw_evaluations.json` as an artifact
172+
5. Install Python 3.11 + dependencies (`openai`, `python-dotenv`, `psycopg2-binary`)
173+
6. Run `process_evals.py` → uploads `processed_snippets.json`
174+
7. If `import_to_db` is `true`, the workflow:
175+
- Imports snippets into the target Postgres via `import_snippets.py --production`
176+
- Runs `server/scripts/fix_snippet_trailing_newlines.js --apply`
177+
- Validates for duplicates (`server/scripts/verify_snippet_duplicates.js`)
178+
179+
Use this workflow when you want a reproducible audit trail and artifact archive. For experimentation, it is still fine to run scripts locally with personal tokens.
180+
181+
---
182+
183+
## 4. Troubleshooting
184+
185+
| Symptom | Likely Cause | Fix |
186+
|---------|--------------|-----|
187+
| `scrape_evals.js` prints `CAS login page (cookie expired?)` | PHPSESSID expired or incorrect | Log into the registrar site again and copy the new cookie. |
188+
| `Failed to fetch course list` | OIT token invalid/expired | Request a fresh Student-App bearer token from OIT. |
189+
| `process_evals.py` exits with `OPENAI_API_KEY not set` | Missing API key | Set the key in `.env` or export it before running. |
190+
| OpenAI rate-limit / API errors | Too many rapid requests | Script auto-retries up to 5 times. If failures persist, rerun later. |
191+
| `import_snippets.py` DB error about SSL | Production URL requires SSL | Use the Heroku-provided `DATABASE_URL` (already SSL-enabled) or add `?sslmode=require`. |
192+
| Snippets still contain smart quotes or blank trailing lines | Import script not run after manual edits | Re-run `import_snippets.py` so normalization applies, or run `node server/scripts/fix_snippet_trailing_newlines.js --apply`. |
193+
194+
---
195+
196+
## 5. FAQ
197+
198+
**Can I skip the AI step and hand-pick snippets?** Yes. Manually edit `processed_snippets.json` to include objects matching the same shape (`text`, `source`, `category`, `original_url`, etc.) and run the import script. The importer recomputes counts automatically.
199+
200+
**How do I regenerate snippets for a single course?** Run `scrape_evals.js --course ...`, remove any existing entries for that course from `processed_snippets.json`, then run `process_evals.py` again. The script only processes items remaining in `raw_evaluations.json`.
201+
202+
**Where should secrets live in CI?** Use GitHub environment secrets (`PRINCETON_API_KEY`, `DATABASE_URL`, optional `OPENAI_API_KEY`) scoped per environment (`tigertype`, `staging`). `phpsessid` is passed as a manual input and masked.
203+
204+
**Does the workflow overwrite old artifacts?** Each run uploads `raw_evaluations` and `processed_snippets` artifacts; GitHub retains them per retention policy so you have a historical snapshot of what was imported.
205+
206+
---
207+
208+
The scripts here are production-ready: no manual preprocessing is required. Follow the steps above or trigger the workflow, and new snippets will flow straight into TigerType with a full audit trail.
209+
210+
Note: For any questions (future COS 333 groups), clarifications, or corrections, please email the original team or TigerApps.

0 commit comments

Comments
 (0)