Class Scraper

Hierarchy

Scraper

Index

Constructors

Properties

URL _lastRequest _rateLimit _userAgent

Accessors

rateLimit userAgent

Methods

all courses extractCourses fetchAndParse latestTerm parseHTML terms wait

Constructors

constructor

new Scraper(userAgent: string, rateLimit?: number): Scraper
Initialize a new Scraper
Parameters
- userAgent: string
  
  User agent must be a valid W&M email address
- Optional rateLimit: number
Returns Scraper
- Defined in src/classes/scraper.ts:25

Properties

`Private` URL

URL: URL = ...

`Private` _lastRequest

_lastRequest: number = 0

`Private` _rateLimit

_rateLimit: number = 500

`Private` _userAgent

_userAgent: string

Accessors

rateLimit

get rateLimit(): number
Returns number
- Defined in src/classes/scraper.ts:58
set rateLimit(rateLimit: number): void
Sets the rate limit in milliseconds.

Example
```
Scraper.rateLimit = 5000 // 5 seconds
```
Parameters
- rateLimit: number
  
  Must be an integer greater than 0
Returns void
- Defined in src/classes/scraper.ts:52

userAgent

get userAgent(): string
Returns string
- Defined in src/classes/scraper.ts:43
set userAgent(userAgent: string): void
Sets the user agent.
Parameters
- userAgent: string
  
  Valid W&M email address (regex: ^[a-zA-Z0-9_.+-]+@wm.edu$)
Returns void
- Defined in src/classes/scraper.ts:35

Methods

all

all(term?: Pick<Filter, "term">, concurrency?: number): Promise<Readonly<Course>[]>
Fetches all courses from the Open Course List.

Example
```
Scraper.all({ term: 202320, concurrency: 10 }) // 74 subjects/10 batches = 7 requests => 7 requests * 500ms rate limit = 3.5 seconds
```
Parameters
- Optional term: Pick<Filter, "term">
  
  Term code to fetch courses from (default: top term in the Open Course List dropdown)
- concurrency: number = 1
  
  Number of concurrent requests to make in batches
Returns Promise<Readonly<Course>[]>
- Defined in src/classes/scraper.ts:102

courses

courses(filters?: Filter): Promise<Readonly<Course>[]>
Fetches courses from the Open Course List based on the specified filter.
Parameters
- Optional filters: Filter
  
  If no filters are specified, all courses will be returned.
Returns Promise<Readonly<Course>[]>
- Defined in src/classes/scraper.ts:80

`Private` extractCourses

extractCourses(dom: any): Readonly<Course>[]
Parameters
- dom: any
Returns Readonly<Course>[]
- Defined in src/classes/scraper.ts:118

`Private` fetchAndParse

fetchAndParse(req: Promise<Response>): Promise<any>
Parameters
- req: Promise<Response>
Returns Promise<any>
- Defined in src/classes/scraper.ts:149

`Private` latestTerm

latestTerm(): Promise<number>
Returns Promise<number>
- Defined in src/classes/scraper.ts:144

`Private` parseHTML

parseHTML(html: string): any
Parameters
- html: string
Returns any
- Defined in src/classes/scraper.ts:173

terms

terms(): Promise<number[]>
Extracts all available terms from the Open Course List under the 'term_code' dropdown.

Returns Promise<number[]>
- Defined in src/classes/scraper.ts:63

`Private` wait

wait(): Promise<void>
Returns Promise<void>
- Defined in src/classes/scraper.ts:161

Generated using TypeDoc