How Zoron securely communicates with Aspen

Author: haelp
Published: 31/11/2024
8 minute read

I recently started developing an app called Zoron, which pulls student data from a system called Aspen and then takes that data and displays in in a much more friendly manner. However, doing so requires the server to use the user's username and password to Aspen to access their account. (The client can't directly communicate with Aspen because of CORS: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). This article explains how Zoron stores this information in a secure way that prevents anyone except the user from being able to access this private and very sensitive information.

Prerequisites

To understand how Zoron stores this information, you must understand three key concepts: encryptionhashing, and cookies.

Encryption

Encryption is the process of converting plaintext into ciphertext using an algorithm and a key, making it unreadable to unauthorized users. It is reversible, meaning the original data can be retrieved (decrypted) using the correct key. Encryption is used to secure data during transmission or storage, ensuring privacy.

Hashing

Hashing, on the other hand, transforms data into a fixed-length string (hash) using a hash function. It is a one-way process, meaning you can't reverse it to get the original data. Hashing is used for verifying data integrity and storing passwords securely.

Cookies

Cookies are small pieces of data stored on a user's browser by websites. Cookies can be read by a server, and they help websites remember information about the user, like login sessions, preferences, or items in a shopping cart.

Part 1 - How Zoron Communicates with Aspen

To communicate with Aspen, Zoron uses a similar approach to how a person would interact with the website. First, it takes the login information for a given user and essentially "fills out" the login form, sending the user data to Aspen's servers. After the Aspen server receives the information, it checks that the login information is correct and returns a cookie with a session token back to Zoron. This session token identifies the user for all subsequent requests without the user needing to send their password over the network every time.

Once Zoron has the session token, it starts to pull data about the user from Aspen, sending the session token along with each request. It pulls some basic information when the page loads (which is why there is a loading screen), and then pulls more detailed information (such as a user's grades) as the user requests more data. Sometimes, if the user does not interact with the site for several minutes, Zoron's stored session token will expire and Zoron will be unable to retrieve more data. When this happens, Zoron logs in again before requesting more data from Aspen. This allows users to use Zoron seamlessly without having to log in repeatedly.

Part 2 - How Zoron securely stores user login information

This part is where the ideas of hashing and encryption become important. The process occurs in 4 key steps:

1 - Adding a password to your account

When you add a password to your account, Zoron doesn't store the password directly in the database. Instead, it stores a hashed version of your password in the database, which means even if someone gained access to the database, they would not be able to acquire any user passwords.

2 - Logging in to your account

When you enter your email and password on the login page, two key steps happen.

 First, the server verifies that your information is correct by hashing your submitted password and checking if that hash matches your stored password hash in the database. If they don't match, your login is rejected, otherwise, the process continues. 

Then, your browser hashes your password with a different algorithm and stores it in your cookies. Because the server stores a hash of your password, it is impossible for the server to get this alternate hash stored on your browser.

3 - Adding your Aspen information

After you log in and your browser has its own unique hash of your password, you enter your Aspen login information. When the browser sends this information to the server,  the server can also see your secret hash in your cookies. After confirming that your login information is correct, it then encrypts your credentials twice: once with the key being your secret hash, and another time with a secret key hidden on the server. This means that the resulting encrypted data can only be decrypted when your browser and the server work together, because each device has one of the keys needed to decrypt your credentials. As a result, even if someone were to gain access to the database and the servers, they would still be unable to decrypt your Aspen credentials.

4 - Loading Aspen Data

Once you open your dashboard, Zoron uses your secret key in your cookies and the server secret to decrypt your credentials on your account. Then, it uses those credentials to load your data from Aspen, as described in part 1.

By combining encryption, hashing, and secure cookie storage, Zoron ensures that sensitive user information, like Aspen credentials, is protected at every step. This layered approach to security minimizes the risk of unauthorized access and ensures that even if part of the system is compromised, the private information remains safe.

Developing Zoron with these measures wasn't just about solving technical challenges—it was about building trust. Users deserve to feel confident that their data is handled responsibly, and this security model is Zoron's way of making that a reality. As the app evolves, its commitment to protecting user data will remain at the forefront, ensuring a secure and seamless experience for everyone.

If you have any questions or feedback about how Zoron handles security, feel free to reach out!

Extra information

The following code block is the module of the Zoron codebase that handles direct connections to Aspen. For security reasons, we can not share the parts of the codebase described in part 2 of the article.

import { JSDOM } from "jsdom"; import Parser, { type Page, type Text } from "pdf2json"; import { parseStringPromise } from "xml2js"; import { decrypt as _decrypt, encrypt as _encrypt } from "./crypt"; import type { Assignment, Attendance, AuthResponse, PeriodAttendance, PostedGrade, RecentActivityList } from "./types"; export namespace aspen { export namespace Types { export type ProgressCallback = (step: number, total: number) => void; export interface Grade { number: number; letter: string; } export interface Name { first: string; last: string; } export interface AssignmentScore { percentage: number; scored: number; total: number; } export interface ClassOptions { year: "current" | "previous"; term: 0 | 1 | 2 | 3 | 4; } export interface Class { id: string; name: string; course: string; term: string; teachers: Name[]; email: string; room?: string; grade?: Grade; attendance: { absent: number; tardy: number; dismissed: number; }; } export interface ClassDetailCategory { name: string; terms: { weight?: number; grade?: Grade; }[]; } export interface Assignment { id: string; name: string; assigned: string; due: string; weight?: number; score?: AssignmentScore; } export interface ClassDetail { grades?: { categories: ClassDetailCategory[]; averages: (Grade | undefined)[]; posted: (Grade | undefined)[]; final?: Grade; }; assignments: Assignment[]; } export namespace Schedule { export type Semester = 1 | 2; export type Lunch = 3 | 2 | 1; export interface PDFCourse { course: string; level?: "Hon" | "AP" | "CP"; description: string; room: string; teacher: string; term: "ALL" | "S 1" | "S 2"; schedule?: string; credit: number; } export interface Course extends PDFCourse { $: boolean; block: string; } export interface Schedule { semester: Semester; lunches: Lunch[]; schedule: (Course | null)[]; } } } export const encrypt = _encrypt; export const decrypt = _decrypt; const getCookies = (res: Response) => [...res.headers.entries()] .filter(([name]) => name === "set-cookie") .map(([_, value]) => value) .map((value) => processCookie(value)); const processCookie = (cookie: string) => { return cookie.split(";")[0]; }; const progressTicker = (steps: number, cb?: Types.ProgressCallback) => { let step = 0; return () => { step++; cb && cb(step, steps); }; }; export namespace constants { export namespace steps { export const authenticate = 5; export const classDetail = 4; export const assignment = 6; export namespace schedule { export const pdf = 5; } } } export const authenticate = async ( username: string, password: string, onProgress?: Types.ProgressCallback ) => { const tick = progressTicker(constants.steps.authenticate, onProgress); let cookie = "deploymentId=ma-lexington; locale=en_US"; const sessionRes = await fetch("https://ma-lexington.myfollett.com/app/rest/i18n/locales", { headers: { accept: "application/json", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", deploymentid: "ma-lexington", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin", cookie, Referer: "https://ma-lexington.myfollett.com/aspen-login/?deploymentId=ma-lexington", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" }); if (sessionRes.status !== 200) { throw new Error( `Failed to initialize authorization sequence: ${sessionRes.status} (${sessionRes.statusText})` ); } cookie = `${cookie}; ${getCookies(sessionRes).join("; ")}`; const authRes = await fetch("https://ma-lexington.myfollett.com/app/rest/auth", { headers: { accept: "application/json", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", "content-type": "application/x-www-form-urlencoded", deploymentid: "ma-lexington", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin", cookie, Referer: "https://ma-lexington.myfollett.com/aspen-login/?deploymentId=ma-lexington", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: `username=${username}&password=${password}`, method: "POST" }); // check auth if (authRes.status !== 200) { throw new Error("Invalid credentials: " + (await authRes.json()).message); } tick(); const auth: AuthResponse = await authRes.json(); cookie = `${cookie}; user=${encodeURIComponent(JSON.stringify(auth))}`; const authTokenRes = await fetch(auth.aspenUrl, { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie, Referer: "https://ma-lexington.myfollett.com/aspen-login/?deploymentId=ma-lexington", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" }); if (authTokenRes.status !== 200) { throw new Error(`Failed to get auth token: ${await authTokenRes.text()}`); } tick(); const homeRes = await fetch("https://ma-lexington.myfollett.com/aspen/home.do", { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/portalAssignmentDetail.do?navkey=academics.classes.list.gcd.detail&oid=GCD0000017PfFO", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" }); if (homeRes.status !== 200) { throw new Error(`Failed to get home: ${await homeRes.text()}`); } tick(); const homeText = await homeRes.text(); const tokenSearch = homeText.match( /<input type="hidden" name="org.apache.struts.taglib.html.TOKEN" value="(.+?)"/ ); if (!tokenSearch) { throw new Error("Failed to find token"); } const token = tokenSearch[1]; const nameSearch = homeText.match( /<div id="userPreferenceMenu" class="toolbarText pointer toolbarItem" tabindex="0">\s*([\w\s,]+)\s*/ ); if (!nameSearch) { throw new Error("Failed to find name"); } const name = nameSearch[1].trim(); const [last, first] = name.split(", "); tick(); return { cookie, token, name: { first, last } }; }; export const email = async (cookie: string) => { const mainPageRes = await fetch( "https://ma-lexington.myfollett.com/aspen/portalStudentDetail.do?navkey=myInfo.details.detail", { headers: { accept: "application/json", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", deploymentid: "ma-lexington", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin", cookie, Referer: "https://ma-lexington.myfollett.com/aspen-login/?deploymentId=ma-lexington", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" } ); if (mainPageRes.status !== 200) { throw new Error(`Failed to get main info page`); } const mainPageHTML = await mainPageRes.text(); const mainPageDom = new JSDOM(mainPageHTML); const form = mainPageDom.window.document.forms["genericDetailForm" as any]; const formData = new mainPageDom.window.FormData(form); formData.set("userParam", "3"); formData.set("userEvent", "2030"); const body = new mainPageDom.window.URLSearchParams(formData as any).toString(); const pageRes = await fetch(`https://ma-lexington.myfollett.com/aspen/portalStudentDetail.do`, { headers: { accept: "*/*", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", "content-type": "application/x-www-form-urlencoded; charset=UTF-8", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin", "x-requested-with": "XMLHttpRequest", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body, method: "POST" }); const email = [ ...new JSDOM(await pageRes.text()).window.document.querySelectorAll("input") ].filter((i) => i.value.includes("@lexingtonma.org"))[0].value; return email; }; export const activity = async (cookie: string) => { const activityRes = await fetch( "https://ma-lexington.myfollett.com/aspen/studentRecentActivityWidget.do?preferences=%3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22UTF-8%22%3F%3E%3Cpreference-set%3E%0A%20%20%3Cpref%20id%3D%22dateRange%22%20type%3D%22int%22%3E4%3C%2Fpref%3E%0A%3C%2Fpreference-set%3E&rand=1728335376000", { headers: { accept: "application/xml, text/xml, */*; q=0.01", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin", "x-requested-with": "XMLHttpRequest", cookie: cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" } ); if (activityRes.status !== 200) { throw new Error(`Failed to get activity: ${await activityRes.text()}`); } const activityXml = await activityRes.text(); const activity: RecentActivityList = await parseStringPromise(activityXml); const attendance = activity["recent-activity-list"]["recent-activity"][0].attendance?.map((period) => { return { type: "attendance", date: period.$.date, code: period.$.code, absent: period.$.absent === "true", dismissed: period.$.dismissed === "true", tardy: period.$.tardy === "true", excused: period.$.excused === "true", portionabsent: parseFloat(period.$.portionabsent), id: period.$.oid } satisfies Attendance; }) || []; const periodAttendance = activity["recent-activity-list"]["recent-activity"][0].periodAttendance?.map((period) => { return { type: "period-attendance", date: period.$.date, period: period.$.period, code: period.$.code, class: period.$.classname, id: period.$.oid, sscid: period.$.sscoid } satisfies PeriodAttendance; }) || []; const grades = activity["recent-activity-list"]["recent-activity"][0].gradebookScore?.map((score) => { return { type: "grade", date: score.$.date, assignment: score.$.assignmentname, class: score.$.classname, grade: score.$.grade, id: score.$.assignmentoid, sscid: score.$.sscoid, gtmid: score.$.gtmoid } satisfies Assignment; }) || []; const postedGrades = activity["recent-activity-list"]["recent-activity"][0].gradePost?.map((score) => { return { type: "posted-grade", date: score.$.date, classname: score.$.classname, oid: score.$.oid, teacher: { first: score.$.teacherfirst, last: score.$.teacherlast }, postType: parseInt(score.$.type), sscid: score.$.sscoid } satisfies PostedGrade; }) || []; const computeMilliseconds = (date: string): number => { const d = new Date( parseInt(date.substring(0, 4)), parseInt(date.substring(5, 7)) - 1, parseInt(date.substring(8, 10)) ); return d.getTime(); }; const mergedActivity: (Assignment | PeriodAttendance | Attendance | PostedGrade)[] = [ ...attendance, ...grades, ...periodAttendance, ...postedGrades ].sort((a, b) => computeMilliseconds(b.date) - computeMilliseconds(a.date)); return { merged: mergedActivity, attendance: attendance.sort( (a, b) => computeMilliseconds(b.date) - computeMilliseconds(a.date) ), grades: grades.sort((a, b) => computeMilliseconds(b.date) - computeMilliseconds(a.date)), raw: activity }; }; export const classes = async ( cookie: string, options?: Types.ClassOptions | { type: "dom"; dom: JSDOM } ): Promise<{ classes: Types.Class[] }> => { const dom = options && "type" in options && options.type === "dom" ? options.dom : await (async () => { const res = await fetch( "https://ma-lexington.myfollett.com/aspen/portalClassList.do?navkey=academics.classes.list", { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie: cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" } ); if (res.status !== 200) { throw new Error(`Failed to get classes: ${res.status}`); } const text = await res.text(); return new JSDOM(text); })(); if (options && !("type" in options)) { if (options.year === "previous" && options.term === 0) { throw new Error("Invalid options: cannot get current term of previous year"); } const formData = new dom.window.FormData(dom.window.document.forms["classListForm" as any]); formData.set("userEvent", "950"); formData.set("yearFilter", options.year); formData.set( "termFilter", ( [...dom.window.document.querySelectorAll("select#termFilter option")].find((option) => option .textContent!.toLowerCase() .includes(options.term === 0 ? "current" : options.term.toString()) ) as HTMLOptionElement ).value ); const body = new dom.window.URLSearchParams(formData as any).toString(); const res = await fetch("https://ma-lexington.myfollett.com/aspen/portalClassList.do", { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", "content-type": "application/x-www-form-urlencoded", pragma: "no-cache", "sec-ch-ua": '"Not A(Brand";v="8", "Chromium";v="132", "Google Chrome";v="132"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/portalClassList.do?navkey=academics.classes.list&maximized=false", "Referrer-Policy": "strict-origin-when-cross-origin" }, body, method: "POST" }); if (res.status !== 200) { throw new Error(`Failed to get classes: ${res.status}`); } return await classes(cookie, { type: "dom", dom: new JSDOM(await res.text()) }); } const body = dom.window.document.querySelector("#dataGrid table tbody"); if (!body) throw new Error("Failed to find data"); const rows = [...body.children].slice(1); const data: Types.Class[] = []; for (const row of rows) { const items = [...row.children].slice(1) as HTMLTableCellElement[]; if (items.length === 0) continue; const getItem = (index: number) => [...items[index].children][0]?.innerHTML?.trim() || items[index].innerHTML.trim(); data.push({ id: items[0].id, name: getItem(0), course: getItem(1), term: getItem(2), teachers: getItem(3) .split("; ") .map( (item) => ({ first: item.split(", ")[1], last: item.split(", ")[0] }) satisfies Types.Name ), email: getItem(4), room: getItem(5), grade: { number: Math.round(parseFloat(getItem(6).split(" ")[0]) * 100) / 100, letter: getItem(6).split(" ")[1] }, attendance: { absent: parseInt(getItem(7)), tardy: parseInt(getItem(8)), dismissed: parseInt(getItem(9)) } }); } return { classes: data }; }; const rewriteUrl = (url: string) => { url = url.replaceAll("\\", "%5C"); url = url.replaceAll("^", "%5E"); url = url.replaceAll("`", "%60"); url = url.replaceAll("{", "%7B"); url = url.replaceAll("|", "%7C"); url = url.replaceAll("}", "%7D"); var paramsExist = url.lastIndexOf("?"); if (paramsExist > 0) { var parts = url.split("?"); if (parts.length == 2) url = parts[0] + "?" + parts[1].replaceAll(":", "%3A").replaceAll("[", "%5B").replaceAll("]", "%5D"); } return url; }; export const classDetail = async ({ cookie, classID, onProgress, assignments = { category: "All", term: 0 } }: { cookie: string; classID: string; onProgress?: Types.ProgressCallback; assignments?: { category?: string; term?: number; }; }) => { const tick = progressTicker(constants.steps.classDetail, onProgress); const prefetch = await fetch( "https://ma-lexington.myfollett.com/aspen/portalClassList.do?navkey=academics.classes.list&maximized=false", { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Not A(Brand";v="8", "Chromium";v="132", "Google Chrome";v="132"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie: cookie, Referer: "https://ma-lexington.myfollett.com/aspen/portalClassDetail.do?navkey=academics.classes.list.detail", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" } ); if (prefetch.status !== 200) { throw new Error(`Failed to prefetch classes: ${prefetch.status}`); } tick(); const prefetchDom = new JSDOM(await prefetch.text()); const prefetchForm = prefetchDom.window.document.forms["classListForm" as any]; const formData = new prefetchDom.window.FormData(prefetchForm); formData.set("userParam", classID); formData.set("userEvent", "2100"); const body = new prefetchDom.window.URLSearchParams(formData as any).toString(); const res = await fetch("https://ma-lexington.myfollett.com/aspen/portalClassList.do", { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", "content-type": "application/x-www-form-urlencoded", pragma: "no-cache", "sec-ch-ua": '"Not A(Brand";v="8", "Chromium";v="132", "Google Chrome";v="132"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/portalClassList.do?navkey=academics.classes.list&maximized=false", "Referrer-Policy": "strict-origin-when-cross-origin" }, body, method: "POST" }); if (res.status !== 200) { throw new Error(`Failed to get class detail: ${res.status}`); } tick(); const [grades, ass] = await Promise.all([ (async () => { const text = await res.text(); const dom = new JSDOM(text); const table = [...dom.window.document.querySelectorAll("table")] .filter((item) => item.textContent?.includes("Average Summary")) .at(-1); if (!table) throw new Error("Failed to find data table"); const rows = [...table.querySelectorAll("tr.listCell")]; const categories = [...table.querySelectorAll('td[rowspan="2"]')] .map((item) => item.textContent?.trim()!) .filter((i) => i); const result: Types.ClassDetail["grades"] = (categories.length > 0 && { categories: [], averages: [], posted: [] }) || undefined; if (categories.length > 0) { for (let i = 0; i < categories.length * 2; i += 2) { const weights = [...rows[i].children] .slice(2) .map((item) => item.textContent?.trim()) .map((item) => item === "N/A" ? undefined : Math.round(100 * parseFloat(item?.slice(0, -1)!)) / 100 ); const grades = [...rows[i + 1].children] .slice(1) .map((item) => item.textContent?.trim()) .map( (item) => (item && item.length > 0 && ([ Math.round(100 * parseFloat(item.split(" ")[0])) / 100, item.split(" ")[1] ] as const)) || undefined ); const r: Types.ClassDetailCategory = { name: categories[i / 2], terms: weights.map( (weight, idx) => ({ weight, grade: grades[idx] ? { number: grades[idx][0], letter: grades[idx][1] } : undefined }) satisfies Types.ClassDetailCategory["terms"][number] ) }; result!.categories.push(r); } result!.averages = [ ...(table.querySelector("tr.listCellHighlight") as HTMLTableRowElement).children ] .slice(1) .map((item) => item.textContent?.trim()) .map((item) => item && item.length > 0 ? ({ number: Math.round(100 * parseFloat(item.split(" ")[0])) / 100, letter: item.split(" ")[1] ?? item } satisfies Types.Grade) : undefined ); result!.posted = [...rows.at(-1)!.children] .slice(1) .map((item) => item.textContent?.trim()) .map((item) => item && item.length > 0 ? ({ number: Math.round(100 * parseFloat(item.split(" ")[0])) / 100, letter: item.split(" ")[1] ?? item } satisfies Types.Grade) : undefined ); const finalText = [ ...[ ...dom.window.document.querySelectorAll( "div.detailContainer table tbody tr td.detailProperty.headerLabelBackground" ) ].at(-1)!.parentNode!.children ].at(-1)!.textContent; if (!finalText) throw new Error("Failed to find final grade"); if (finalText.trim().length > 1) { result!.final = { number: Math.round(parseFloat(finalText.split(" ")[0]) * 100) / 100, letter: finalText.split(" ")[1].trim() }; } } return result; })(), (async () => { const initialRes = await fetch( "https://ma-lexington.myfollett.com/aspen/portalAssignmentList.do?navkey=academics.classes.list.gcd", { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Not A(Brand";v="8", "Chromium";v="132", "Google Chrome";v="132"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "none", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie }, referrerPolicy: "strict-origin-when-cross-origin", body: null, method: "GET" } ); if (initialRes.status !== 200) { throw new Error(`Failed to get initial assignments: ${initialRes.status}`); } const parseAssignements = (document: JSDOM["window"]["document"]): Types.Assignment[] => { const rows = [...document.querySelectorAll("#dataGrid > table > tbody > tr.listCell")]; if (rows[0].textContent?.trim() === "No matching records") return []; return rows.map( (row) => ({ id: row.children[1].id, name: row.children[1].textContent!.trim(), assigned: row.children[2].textContent!.trim(), due: row.children[3].textContent!.trim(), weight: row.children.length === 7 ? Math.round(parseFloat(row.children[4].textContent!.trim()) * 100) / 100 : undefined, score: row.children[4].textContent!.trim() === "Ungraded" ? undefined : ((): Types.Assignment["score"] => { const items = row.children[row.children.length === 7 ? 5 : 4].querySelectorAll( "table > tbody > tr > td" ); if (items.length === 1) return; const str = items[items.length - 2].textContent!.trim().split(" / "); const scored = Math.round(parseFloat(str[0]) * 100) / 100; const total = Math.round(parseFloat(str[1]) * 100) / 100; return { scored, total, percentage: Math.round((scored / total) * 100 * 100) / 100 }; })() }) satisfies Types.Assignment ); }; const window = new JSDOM(await initialRes.text()).window; const document = window.document; if ( !assignments || (assignments.category === undefined && assignments.term === undefined) || (assignments.category === "All" && assignments.term === undefined) ) { return parseAssignements(document); } else { const form = document.forms["portalAssignmentListForm" as any]; const formData = new window.FormData(form); const termString = assignments.term === 0 || assignments.term === undefined ? "All" : `T${assignments.term}`; formData.set("userEvent", "2210"); const termID = ( [...(document.getElementById("gradeTermOid")?.children || [])].find( (item) => item.textContent === termString )! as HTMLOptionElement )?.value; formData.set("gradeTermOid", termID || ""); const categoryString = assignments.category || "All"; const categoryID = ( [...(document.getElementById("categoryOid")?.children || [])].find( (item) => item.textContent === categoryString )! as HTMLOptionElement )?.value; formData.set("categoryOid", categoryID || ""); const body = new window.URLSearchParams(formData as any).toString(); const newAssignmentsRes = await fetch( "https://ma-lexington.myfollett.com/aspen/portalAssignmentList.do", { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", "content-type": "application/x-www-form-urlencoded", pragma: "no-cache", "sec-ch-ua": '"Not A(Brand";v="8", "Chromium";v="132", "Google Chrome";v="132"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/portalAssignmentList.do?navkey=academics.classes.list.gcd", "Referrer-Policy": "strict-origin-when-cross-origin" }, body, method: "POST" } ); if (newAssignmentsRes.status !== 200) { throw new Error( `Failed to get new assignments based on query (${categoryString} and ${termString}): ${newAssignmentsRes.status}` ); } const newAssignmentsText = await newAssignmentsRes.text(); const newAssignmentsDom = new JSDOM(newAssignmentsText); return parseAssignements(newAssignmentsDom.window.document); } })() ] as const); return { grades, assignments: ass } satisfies Types.ClassDetail; }; export const assignment = async ({ cookie, assignment, studentID, token, onProgress }: { cookie: string; token: string; assignment: Assignment; studentID: string; onProgress?: Types.ProgressCallback; }): Promise<Types.AssignmentScore> => { const tick = progressTicker(constants.steps.assignment, onProgress); const preload = rewriteUrl("portalClassList.do"); const resource = rewriteUrl(`${preload}?navkey=academics.classes.list`); const resourceRes = await fetch(`https://ma-lexington.myfollett.com/aspen/${resource}`, { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie: cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" }); if (resourceRes.status !== 200) { throw new Error(`Failed to get resource: ${resourceRes.status}`); } tick(); const preloadRes = await fetch(`https://ma-lexington.myfollett.com/aspen/${preload}`, { headers: { accept: "*/*", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", "content-type": "application/x-www-form-urlencoded; charset=UTF-8", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin", "x-requested-with": "XMLHttpRequest", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: `selectedStudentOid=${studentID}&userEvent=2210&org.apache.struts.taglib.html.TOKEN=${token}`, method: "POST" }); if (preloadRes.status !== 200) { throw new Error(`Failed to get preload: ${preloadRes.status}`); } tick(); const filterRes = await fetch(`https://ma-lexington.myfollett.com/aspen/${resource}`, { headers: { accept: "*/*", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", "content-type": "application/x-www-form-urlencoded; charset=UTF-8", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin", "x-requested-with": "XMLHttpRequest", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: `filterDefinitionId=%23%23%23all&userEvent=2060&org.apache.struts.taglib.html.TOKEN=${token}`, method: "POST" }); if (filterRes.status !== 200) { throw new Error(`Failed to get filter: ${filterRes.status}`); } tick(); const preloadRes2 = await fetch( `https://ma-lexington.myfollett.com/aspen/portalAssignmentList.do?navkey=academics.classes.list.gcd&oid=${assignment.sscid}&gtmoid=${assignment.gtmid}`, { headers: { accept: "*/*", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin", "x-requested-with": "XMLHttpRequest", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" } ); if (preloadRes2.status !== 200) { throw new Error(`Failed to get preload2: ${preloadRes2.status}`); } tick(); const assignmentRes = await fetch( `https://ma-lexington.myfollett.com/aspen/portalAssignmentDetail.do?navkey=academics.classes.list.gcd.detail&oid=${assignment.id}`, { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie: cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" } ); if (assignmentRes.status !== 200) { throw new Error(`Failed to get assignment: ${assignmentRes.status}`); } tick(); const assignmentHtml = await assignmentRes.text(); const dom = new JSDOM(assignmentHtml); let percentage = parseFloat( dom.window.document.querySelector(".percentFieldInlineLabel")?.textContent?.slice(0, -1) || "NaN" ); const [rawPoints]: [string] = [ ...dom.window.document.querySelectorAll("td.detailValue table tbody tr td") ] .map((td) => td.textContent) .filter((item) => item?.includes(" / ")) as any; if (!rawPoints) { throw new Error("Failed to parse points"); } const [points, maxPoints] = rawPoints.split(" / ").map((item) => parseFloat(item.trim())); if (Number.isNaN(percentage)) percentage = Math.round((points / maxPoints) * 100); tick(); return { percentage, scored: Math.round(points * 100) / 100, total: maxPoints }; }; export namespace schedule { export const pdf = async ( cookie: string, semester: Types.Schedule.Semester = 1, onProgress?: Types.ProgressCallback ): Promise<Types.Schedule.Schedule> => { const tick = progressTicker(constants.steps.schedule.pdf, onProgress); const toolRes = await fetch( `https://ma-lexington.myfollett.com/aspen/runTool.do?maximized=false&oid=RPT0000010rMZR&toolClass=com.follett.fsc.core.k12.beans.Report&deploymentId=ma-lexington`, { headers: { accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" } ); if (toolRes.status !== 200) { throw new Error(`Failed to get tool page: ${toolRes.status} (${toolRes.statusText})`); } tick(); const { window: toolWindow } = new JSDOM(await toolRes.text()); const { document: toolDoc } = toolWindow; const toolForm = new toolWindow.FormData(toolDoc.forms["toolInputForm" as any]); // toolForm.set("formatStr", "0"); // we don't want csv anymore :(( toolForm.set("userEvent", "960"); const body = new FormData(); for (const [key, value] of toolForm.entries()) { body.append(key, value); } const res = await fetch(`https://ma-lexington.myfollett.com/aspen/runTool.do`, { headers: { accept: "*/*", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin", "x-requested-with": "XMLHttpRequest", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body, method: "POST" }); if (res.status !== 200) { throw new Error(`Failed to run schedule job: ${res.status} (${res.status})`); } tick(); const text = await res.text(); const urlStart = "doNamedPopup('"; const idx = text.indexOf(urlStart); if (idx === -1) { throw new Error("Failed to find schedule download URL"); } const url = text.substring( idx + urlStart.length, text.indexOf("'", idx + urlStart.length + 1) ); const prePdfRes = await fetch(`https://ma-lexington.myfollett.com/aspen/${url}`, { headers: { accept: "*/*", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, body: null, method: "GET" }); if (prePdfRes.status !== 200) { throw new Error( `Failed to prefetch schedule: ${prePdfRes.status} (${prePdfRes.statusText})` ); } tick(); const pdfURLStart = "rewriteUrl('"; const prePdfText = await prePdfRes.text(); const pdfURL = prePdfText.substring( prePdfText.indexOf(pdfURLStart) + pdfURLStart.length, prePdfText.indexOf("')", prePdfText.indexOf(pdfURLStart) + pdfURLStart.length) ); const pdfRes = await fetch(pdfURL, { headers: { accept: "*/*", "accept-language": "en-US,en;q=0.9,und;q=0.8,es;q=0.7", "cache-control": "no-cache", pragma: "no-cache", "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "same-origin", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", cookie, Referer: "https://ma-lexington.myfollett.com/aspen/home.do", "Referrer-Policy": "strict-origin-when-cross-origin" }, method: "GET" }); if (pdfRes.status !== 200) { throw new Error(`Failed to download schedule: ${pdfRes.status} (${pdfRes.statusText})`); } tick(); const pdfBuffer = Buffer.from(await pdfRes.arrayBuffer()); const parsed = await parser.extract(pdfBuffer, semester); tick(); return parsed; }; export namespace parser { export const pdfPromise = async (data: Buffer): Promise<import("pdf2json").Page[]> => new Promise<Page[]>((res, rej) => { const parser = new Parser(); parser.on("pdfParser_dataError", (errData) => rej(errData.parserError)); parser.on("pdfParser_dataReady", (pdfData) => { res(pdfData.Pages); }); parser.parseBuffer(data); }); export const parse = async (d: Buffer) => { const et = (text: Text) => decodeURIComponent(text.R[0].T); const data = await pdfPromise(d); const schedulePage = data[0]; const text = schedulePage.Texts; const name = et(text[0]); const columns = { 1.688: "course", 5.625: "level", 9.313: "description", 18.563: "room", 21.438: "teacher", 26.813: "term", 29.375: "schedule", 34.125: "credit" }; const res: Types.Schedule.Course[] = []; const cols = Object.keys(columns).map((key) => parseFloat(key)); const idCol = cols[0]; const rows = text.filter((t) => t.x === idCol).map((t) => t.y); rows.forEach((y) => { const r: Record<string, string | number> = {}; cols.forEach((col) => { const i = text.find((t) => t.x === col && t.y === y); if (i) r[columns[col as keyof typeof columns]] = et(i).trim(); }); if ("credit" in r) r.credit = parseInt(r.credit as string); res.push(r as any); }); return { name, courses: res.filter((r) => r.course !== "Course") }; }; export const generateSchedule = ( data: Awaited<ReturnType<typeof parse>>, semester: 1 | 2 ): Types.Schedule.Schedule => { const schedule = [ ["A1", "B1", "C1", "D1", "E1", "F1"], ["E2", "F2", "G1", "H1", "R", "D2"], ["B2", "A2", "G2", "H2", "I", "C2"], ["A3", "B3", "C3", "D3", "E3", "F3"], ["E4", "F4", "G3", "H3", "I", "D4"], ["B4", "A4", "G4", "H4", "I", "C4"] ].flat(); /** @type {({...(typeof data.courses[number]), $: boolean} | null)[]} */ const res: (null | Types.Schedule.Course)[] = Array(schedule.length).fill(null); const lunches: Types.Schedule.Lunch[] = Array(6).fill(3); data.courses.forEach((course) => { if ( !course.schedule || course.schedule === "I" || !(course.term === "ALL" || course.term === `S ${semester}`) || course.schedule.length === 0 ) return; if (course.schedule === "HR") { res[schedule.findIndex((b) => b === "R")] = JSON.parse(JSON.stringify(course)); return; } const blocks: [string, boolean, ...number[]][] = []; for (const char of course.schedule.split("")) { if (char === "$") { blocks.at(-1)![1] = true; } else if ("1234".includes(char)) blocks.at(-1)!.push(parseInt(char)); else blocks.push([char, false]); } blocks.forEach((block) => { if (block.length === 2) block.push(1, 2, 3, 4); }); schedule.forEach((block, i) => { blocks.forEach((b) => { const code = block[0], day = parseInt(block[1]); if (b[0] === code && b.slice(2).includes(day)) res[i] = { ...course, $: b[1], block: block[0] + (b[1] ? "$" : "") + day.toString() }; }); }); }); for (let i = 0; i < 6; i++) { const $s = res .slice(i * 6, (i + 1) * 6) .filter((item, idx) => item?.$ && (idx === 2 || idx === 3)); lunches[i] = (3 - $s.length) as any; } return { semester, lunches, schedule: res }; }; export const extract = (data: Buffer, semester: 1 | 2): Promise<Types.Schedule.Schedule> => parse(data).then((data) => generateSchedule(data, semester)); } } }