Chapter 5: Adding LLM Feature and Deployment.
We keep our book up to date with recent libraries and packages.
The section below is a preview of Browser Extension Book. To read the full text, you will need to purchase the book.
In Chapter 5, you will start with the codebase in the 5-begin folder of our browser extension repo and end up with the codebase in the 5-end folder.
In this section, we'll integrate large language model (LLM) feature into our browser extension using the OpenAI API.
This will enable our extension to generate summaries and provide insights.
We will cover the following topics in this chapter:
- Setting up an OpenAI account
- Creating an AWS Account and setting up S3 and Textract Services
- Server
- Installing new dependencies
- Creating and understanding new files
- Modifying files and understanding the changes
- Configuring new environment variables
- Extension
- Installing new dependencies
- Modifying files and understanding the changes
- Visual exploration of the LLM feature process
- Deploying a Node.js server on AWS Elastic Beanstalk
- Submitting the extension to the Chrome Web Store
Setting up an OpenAI account
To use OpenAI's API, you first need to create an account and obtain an API key. Here’s how you can do it:
- Visit the OpenAI website and click on 'Sign Up'.
- Follow the instructions to create an account.
- Once your account is set up, navigate to the API section in your dashboard.
- Generate a new API key; this key will be used to authenticate your API requests.
Remember to keep your API key confidential and not to expose it in your client-side code.
Creating an AWS Account and setting up S3 and Textract Services link
We need to set up an AWS account, as we'll be using Amazon S3 for storing resume files and Amazon Textract for extracting text from PDF files. Here are detailed instructions on how to create an AWS account and configure the necessary services.
Step 1: Create an AWS Account
- Go to the AWS homepage: AWS Home
- Click on "Create an AWS Account".
- Follow the on-screen instructions to set up your account, including providing your email address, a password, and an account name.
- Enter your billing information. AWS requires a credit card for setting up an account, but you will not be charged for the free tier services.
- Complete the identity verification by phone.
- Select a support plan that suits your needs. The free plan should suffice for development and initial testing.
Step 2: Create an S3 Bucket
Amazon S3 will be used to store resume files securely:
- Sign into the AWS Management Console.
- Navigate to the S3 service by searching for "S3" in the "Services" menu.
- Click "Create bucket".
- Provide a unique name for your bucket in the "Bucket name" field. Note that this name must be globally unique across all existing bucket names in Amazon S3.
- Choose the region closest to you or your users to reduce latency and costs.
- Ensure that "Block Public Access" settings are enabled to keep your bucket private.
- Click "Create bucket" to complete the setup.
Step 3: Set Up Amazon Textract
Amazon Textract will be used for extracting text from uploaded PDF files:
- Navigate to the Textract service page within the AWS Console.
- Ensure that the service is available in your region and enable it if it is not already active.
Express server related changes link
Installing the openai package
To communicate with OpenAI’s API from our server, we need to install their official Node.js package. Run the following command in your project directory:
yarn add openai
Adding new files
We need to create several new files in our server's src
directory to handle different aspects of our LLM features. Here are the files and their intended purposes:
src/models/jobs.ts: This file will define a data model for jobs that we need to process using the LLM.
import mongoose from 'mongoose'; export interface IJobDocument extends mongoose.Document { userId: string; jobId: string; status: string; statusError?: string; jobDetailText: string; jobTitle: string; educationLevel: string; jobSkills: string; responseText?: string; } interface IJobModel extends mongoose.Model<IJobDocument> { createJob(doc): Promise<IJobDocument>; } class JobClass extends mongoose.Model { public static async createJob(doc) { return this.create({ ...doc }); } } export const JobSchema = new mongoose.Schema<IJobDocument, IJobModel>({ userId: { type: String }, jobId: { type: String }, status: { type: String }, statusError: { type: String }, jobDetailText: { type: String }, jobTitle: { type: String }, educationLevel: { type: String }, jobSkills: { type: String }, responseText: { type: String }, }); JobSchema.loadClass(JobClass); const Job = mongoose.model<IJobDocument, IJobModel>('jobs', JobSchema); export default Job;
src/models/summaries.ts: This will store summaries generated from the text processed by the LLM.
import mongoose from 'mongoose'; export interface ISummaryDocument extends mongoose.Document { userId: string; createdAt: Date; jobId: string; jobDbId: string; applicantId: string; applicantFullName: string, applicantContactInfo: { email: string, phone: string, }, resumeId: string; wholeContent: string; truncatedContent: string; responseText: string; error?: string; status: string; } interface ISummaryModel extends mongoose.Model<ISummaryDocument> { createSummary(doc): Promise<ISummaryDocument>; } class summaryClass extends mongoose.Model { public static async createSummary(doc) { return this.create({ createdAt: new Date(), ...doc }); } } export const summarieschema = new mongoose.Schema<ISummaryDocument, ISummaryModel>({ userId: { type: String }, createdAt: { type: Date }, jobId: { type: String }, jobDbId: { type: String }, applicantId: { type: String }, applicantFullName: { type: String }, applicantContactInfo: { type: Object }, resumeId: { type: String }, wholeContent: { type: String }, truncatedContent: { type: String }, responseText: { type: String }, error: { type: String }, status: { type: String }, }); summarieschema.loadClass(summaryClass); const summary = mongoose.model<ISummaryDocument, ISummaryModel>('summaries', summarieschema); export default summary;
src/models/prompts.ts: This file will manage the prompts that we submit to the LLM.
export const prompts = { 'summary-default': { promptName: 'summary-default', chatGptModel: 'gpt-3.5-turbo', system: ` You are an experienced recruiter in the biotech industry responsible for hiring the best job applicant for the job by summarizing the job applicant's resume. Your task is to compare the job applicant's resume to the job description on the following parameters: {{ prompts }} ------ Respond in a helpful tone, with very concise answers. `, user: ` Review the job applicant's resume. Find and mention all of the above parameters. See the job applicant's resume below. <!--start-resume--> <%= resumeContent %> <!--end-resume--> Use the below example response, your response should contain similar structure. <!--start-example-response--> {{ prompts }} <!--end-example-response--> ` } } export const templatePrompts = [ { name: 'Education level', promptText: `Education level. Find and mention the most senior education level of the job applicant from the job applicant's resume. Example values for educational level: "Postdoctoral Fellow", "M.D.", "Ph.D.", "Pharm.D.", "M.S.", "B.S.", "A.S.", "M.S. in Bioengineering", "PhD in Bioinformatics" and other values. If field of study is mentioned, mention field of study together with education level. Examples of field of study: "Chemistry", "Biology", "Computer Biology", "Biochemistry", "Bioinformatics", "Immunology", "Pharmacology", "Microbiology" and other values. Mention the graduation date for this found education level and university's name.`, exampleResponse: 'Education level: The job applicant has an Ph.D. degree in Virology from Caltech (Pasadena, CA). Completion date is May 2020.' }, { name: 'Job title', promptText: `Job title. Mention the most senior job title of the job applicant from the job applicant's resume. Example values for job title: "Intern", "Co-op", "Technicain", "Research Associate", "Scientist", "Engineer", "Developer", "Analyst", "Manager", "Supervisor", "Director", "Head", "Lead", "President", "Officer", "Executive" and other values. Mentioned the date when job applicant received this job title. Mention employer's name.`, exampleResponse: `Job title: The job applicant's most recent job title is "Senior Scientist" at Moderna (Cambridge, MA). The job applicant held this job title since July 2021.` }, { name: 'Skills', promptText: `Skills. Skills inside the job applicant's resume that are related to research, laboratory and technology. Example values for skills: "Flow cytometry", "PCR", "CHIP-seq", "Brain dissection", "NGS-based assays", "target discovery", "PreSeq software", "Max Reads assay", "cell culturing", "Python". Mention hard skills and don't mention soft skills. Indicate one or two skills that are mentioned most frequently in the job applicant's resume.`, exampleResponse: `Skills: The job applicant has experience with Next-Generation Sequencing (NGS)-based genetic tests, PCR-based methods, RNA-Seq, FACS and Flow cytometry. The job applicant has extensive experience in RNA sequencing.` }, { name: 'Physical location', promptText: `Physical location. Indicate the current physical location of the job applicant or physical location of their most recent employer or physical location of their university if the job applicant is the recent graduate. Mention this location (city, state) and if this location is outside of United States then indicate it.`, exampleResponse: `Skills: The job applicant has experience with Next-Generation Sequencing (NGS)-based genetic tests, PCR-based methods, RNA-Seq, FACS and Flow cytometry. The job applicant has extensive experience in RNA sequencing.` }, ]
src/textract.ts: A utility file to extract text from pdf file using AWS textract service.
import * as aws from 'aws-sdk'; import * as dotenv from 'dotenv'; dotenv.config(); aws.config.update({ region: process.env.AWS_REGION_ZONE, accessKeyId: process.env.AWS_ACCESSKEYID, secretAccessKey: process.env.AWS_SECRETACCESSKEY, }); export const parsePdf = async (url: string): Promise<string> => { const objKey = url.split('amazonaws.com/')[1]; let blocks = await analyzeDocument(objKey); return getText(blocks); }; export const extractPdf = async (url: string) => { const content = await parsePdf(url); return { wholeContent: content, truncatedContent: content.slice(0, 11000) }; } const textract = new aws.Textract({ apiVersion: 'latest' }); async function startDocumentAnalysis(fileName: string) { console.log('startDocumentAnalysis fileName', fileName); const params = { DocumentLocation: { S3Object: { Bucket: process.env.BUCKET_FOR_CANDIDATES, Name: fileName, }, }, FeatureTypes: ['TABLES'], }; return new Promise<aws.Textract.StartDocumentAnalysisResponse>((resolve, reject) => { textract.startDocumentAnalysis(params, (err, data) => { if (err) { reject(err); } else { resolve(data); } }); }); } async function getDocumentAnalysis(jobId: string) { const res = await textract.getDocumentAnalysis({ JobId: jobId }).promise(); const responses: aws.Textract.GetDocumentAnalysisResponse[] = [res]; let nextToken = res.NextToken; while (nextToken) { const res2 = await textract .getDocumentAnalysis({ JobId: jobId, NextToken: nextToken }) .promise(); responses.push(res2); nextToken = res2.NextToken; } return responses; } async function analyzeDocument(fileName: string, verbose = false) { if (verbose) { console.log('Starting analysis'); } const jobId = (await startDocumentAnalysis(fileName)).JobId; if (!jobId) { throw new Error('no job id returned'); } let count = 0; while (true) { if (verbose) { console.info('waiting 1 second'); } await new Promise((resolve) => setTimeout(resolve, 1000)); count++; const jobResponses = await getDocumentAnalysis(jobId); const jobStatus = jobResponses[0].JobStatus; if (verbose) { console.info('job status:', jobStatus); } if (jobStatus === 'SUCCEEDED') { const blocks: aws.Textract.Block[] = []; for (const jobRes of jobResponses) { if (jobRes.JobStatus === 'SUCCEEDED' && jobRes.Blocks) { blocks.push(...jobRes.Blocks); } } return blocks; } else if (jobStatus !== 'IN_PROGRESS') { throw new Error(`Job failed; id: ${jobId}, status: ${jobStatus}`); } if (count > 180) { throw new Error(`Job timed out; id: ${jobId}. Waited for ${count} seconds`); } } } async function getText(blocks: aws.Textract.Block[]) { const lines: string[] = []; for (const block of blocks || []) { if (block.BlockType === 'LINE' && block.Text) { lines.push(block.Text); } } return lines.join('\n'); }
parsePdf Parses a PDF document by its URL to extract text
- Extracts the object key from the URL.
- Calls
analyzeDocument
with the extracted key to get document analysis data. - Uses
getText
to convert the analyzed data into a text format.
extractPdf Extracts text from a PDF and provides both the full and a truncated version
- Uses
parsePdf
to get the complete textual content of the document. - Returns an object containing both the complete text and a truncated version (first 11,000 characters).
- Uses
startDocumentAnalysis Starts the document analysis process
- Sets up parameters specifying the S3 bucket and file name.
- Initiates analysis with options for detecting table data.
- Returns a promise that resolves or rejects based on the success of the operation.
getDocumentAnalysis Retrieves the results of the document analysis.
- Initiates and handles multiple requests if the analysis data is paginated using
NextToken
.
- Initiates and handles multiple requests if the analysis data is paginated using
analyzeDocument Orchestrates the complete process of submitting a document for analysis and retrieving the results.
- Starts the document analysis and checks for its completion.
- Periodically polls for the analysis status and compiles results once completed.
getText Converts the blocks of text received from the AWS Textract service into a single string.
- Iterates over the blocks, filtering for lines of text, and combines them into a single string separated by new lines.
src/llm-routes.ts: It contains routes that handle API requests related to LLM operations.
import { Configuration, OpenAIApi } from 'openai'; import * as dotenv from 'dotenv'; import Jobs, { IJobDocument } from './models/jobs'; import Summaries from './models/summaries'; import Users, { IUserDocument } from './models/users'; import { requireLogin, routeErrorHandling } from "./utils" import { extractPdf } from './textract'; import template from 'lodash/template'; import { prompts, templatePrompts } from './models/prompts'; import * as aws from 'aws-sdk'; import * as pathModule from 'path'; dotenv.config(); const { CHATGPT_MODEL, OPENAI_API_KEY, } = process.env; aws.config.update({ region: process.env.AWS_REGION_ZONE, accessKeyId: process.env.AWS_ACCESSKEYID, secretAccessKey: process.env.AWS_SECRETACCESSKEY, }); const s3 = new aws.S3({ apiVersion: 'latest' }); async function signRequestForUpload({ fileName, fileType, userId, applicantId, resumeId, jobId }) { const fileExt = pathModule.extname(fileName); const fileNameWithoutExtension = pathModule.basename(fileName, fileExt); const randomString20 = Math.random().toString(36).substring(2, 12) + Math.random().toString(36).substring(2, 12); const key = `user-${userId}/job-${jobId}/applicant-${applicantId}/resume-${resumeId}/${randomString20}/${fileNameWithoutExtension}${fileExt}` // summary ACL const params: any = { Bucket: process.env.BUCKET_FOR_CANDIDATES, Key: key, Expires: 300, // 5 minutes, sufficient for uploading ContentType: fileType, }; return new Promise((resolve, reject) => { s3.getSignedUrl('putObject', params, (err, data) => { const returnedDataFromS3 = { signedRequest: data, path: key as string, url: `https://${process.env.BUCKET_FOR_CANDIDATES}.s3.amazonaws.com/${key}`, }; if (err) { reject(err); } else { resolve(returnedDataFromS3); } }); }); } interface IPrompt { promptName: string; chatGptModel: string; system: string; user: string; } const fetchGPT = async ( prompt: IPrompt, templateVariables: any, ): Promise<string> => { const configuration = new Configuration({ apiKey: OPENAI_API_KEY, }); const chatGPTClient: any = new OpenAIApi(configuration); const userPrompt = template(prompt.user)(templateVariables); const systemPrompt = template(prompt.system)(templateVariables); const messages = [ { role: 'system', content: systemPrompt, }, { role: 'user', content: userPrompt, }, ]; let model = prompt.chatGptModel || CHATGPT_MODEL || 'gpt-4'; const temperature = 0; const completion = await chatGPTClient.createChatCompletion({ model, temperature, messages, }); return completion.data.choices[0].message?.content || ''; }; const evaluate = async ({ resumeContent, jobDetailText, jobTitle, educationLevel, jobSkills, }: { resumeContent: string; jobDetailText: string; jobTitle: string; educationLevel: string; jobSkills: string; }): Promise<string> => { const prompt = prompts['summary-default']; let systemPrompt = prompt.system; let userPrompt = prompt.user; let prompTexts = ''; let exampleResponses = ''; let index = 0; for (const bp of templatePrompts) { index++; prompTexts = `${prompTexts} ${index}) ${bp.promptText || ''} `; exampleResponses = `${exampleResponses} ${index}) ${bp.exampleResponse || ''} `; } systemPrompt = systemPrompt.replace('{{ prompts }}', prompTexts); userPrompt = userPrompt.replace('{{ prompts }}', exampleResponses); try { const text = await fetchGPT( { ...prompt, system: systemPrompt, user: userPrompt, }, { jobDetailText, jobTitle, resumeContent, educationLevel, jobSkills, } ); return text; } catch (e) { console.error( 'Error during evaluate ==========', e.response ? e.response.data.error.message : e ); return 'Summary was not generated due to error: ' + `${e.response ? e.response.data.error.message : e}` } }; type SummaryArgs = { user: IUserDocument, job: IJobDocument, req: any, res: any, } const generateSummary = async ({ user, job, req, res }: SummaryArgs) => { const params = req.body; const userId = user._id; const { s3Url, jobId, applicantId, applicantFullName, applicantContactInfo, resumeId, } = params; if ( (await Summaries.find({ userId, jobId, status: 'startedProcessing', }).countDocuments()) > 0 ) { return res.json({ message: 'Not finished previous evaluation' }); } const selector = { userId, jobId, applicantId }; const doc = { userId, jobId, jobDbId: job._id, applicantId, applicantFullName, applicantContactInfo, resumeId, status: 'startedProcessing', }; if ((await Summaries.find(selector).countDocuments()) === 0) { await Summaries.createSummary(doc); } else { await Summaries.updateOne(selector, { $set: doc }); } const summary = await Summaries.findOne(selector); const { wholeContent, truncatedContent } = await extractPdf(s3Url); await Summaries.updateOne( { _id: summary._id }, { $set: { wholeContent, truncatedContent } } ); const updatedSummary = await Summaries.findOne({ _id: summary._id }); const resumeContent = updatedSummary.truncatedContent; const jobDetailText = job.jobDetailText; const text = await evaluate({ resumeContent, jobDetailText, jobTitle: job.jobTitle, educationLevel: job.educationLevel, jobSkills: job.jobSkills, }); const array = text.split(/\r?\n|\r|\n/g); const reasoning = array[array.length - 1]; if (reasoning) { const setDoc: any = { status: 'evaluated', error: '', }; await Summaries.updateOne( { _id: summary._id }, { $set: { ...setDoc, responseText: text } } ); } }; const save = async (req, res, next) => { await requireLogin(req, res, next); const params = req.body; const userId = req.user ? req.user._id : ''; const user = await Users.findOne({ _id: userId }); const { action, jobId, applicantId, error, } = params; let returnResult: any = { message: 'success' }; let job: IJobDocument = await Jobs.findOne({ userId, jobId }).lean(); if (!job) { job = await Jobs.createJob({ userId, jobId, status: 'parsed', statusError: '', }); } if (action === 'summary-error') { const selector = { userId, jobId, applicantId }; const summaryErrorDoc = { userId, jobId, applicantId, error, status: '' }; if ((await Summaries.find(selector).countDocuments()) === 0) { await Summaries.createSummary(summaryErrorDoc); } else { await Summaries.updateOne(selector, { $set: summaryErrorDoc }); } } if (action === 'summary') { const selector = { userId, jobId, applicantId }; try { await generateSummary({ user, job, req, res }); } catch (e) { await Summaries.updateOne( selector, { $set: { status: 'evaluated', responseText: e.message } } ); } } return res.json(returnResult); } export default (app) => { app.post( '/get-summary-status', routeErrorHandling(async (req, res, next) => { await requireLogin(req, res, next); const params = req.body; const { jobId, applicantId } = params; const response = await Summaries.findOne({ jobId, applicantId, userId: req.user._id }); return res.json(response); }) ); app.post('/save', routeErrorHandling(save)); app.post( '/signurl', routeErrorHandling(async (req, res, next) => { await requireLogin(req, res, next); const params = req.body; const { applicantId, resumeId, jobId } = params; const response = await signRequestForUpload({ fileName: 'resume.pdf', fileType: 'application/pdf', userId: req.user._id, applicantId, resumeId, jobId: jobId, }); return res.json(response); }) ); app.get( '/prev-results', routeErrorHandling(async (req, res, next) => { await requireLogin(req, res, next); const { jobId } = req.query; const summaries: any[] = await Summaries.find( { userId: req.user._id, jobId }, ).lean(); return res.json(summaries); }) ); app.post( '/save-job-detail', routeErrorHandling(async (req, res, next) => { await requireLogin(req, res, next); const userId = req.user ? req.user._id : ''; const { ats, jobId, jobDetailText, jobDetailLink, jobLocation, jobTitle } = req.body; const selector = { userId, jobId }; let job: IJobDocument = await Jobs.findOne(selector).lean(); const doc = { jobDetailText, jobDetailLink, jobLocation, jobTitle } if (job) { await Jobs.updateOne(selector, { $set: doc }); } else { job = await Jobs.createJob({ ...selector, ...doc, ats }); } return res.json({ status: 'ok' }); }) ); app.post('/save', routeErrorHandling(save)); }
signRequestForUpload Generates a signed URL for file uploads to an AWS S3 bucket.
- Extracts the file extension and the base file name from
fileName
using thepathModule
. - Generates a random string of characters to ensure the file path in the bucket is unique.
- Constructs a storage key using identifiers for the user, job, applicant, and resume, along with the random string and the original file name.
- Defines parameters for the S3
putObject
operation, including the bucket name, the file key, expiration time of the signed URL (set to 5 minutes), and the content type of the file.
- Extracts the file extension and the base file name from
fetchGPT Interacts with OpenAI's API to generate responses from a model, based on user inputs and predefined system messages.
- Initializes a new configuration for the OpenAI API client using an API key stored in
OPENAI_API_KEY
. - Creates a new instance of
OpenAIApi
using the specified configuration, which allows interaction with the API. - Processes templates for user and system prompts using the
templateVariables
. This step involves replacing placeholders in the template strings with actual values fromtemplateVariables
. - Constructs an array of messages, which includes a system message followed by a user message. These messages serve as the context for the model to understand and generate appropriate responses.
- Determines the model to use (defaulting to 'gpt-4' if no specific model is provided in the prompt configuration or the default environment). It also sets the
temperature
to 0, which influences the randomness of the response (with 0 resulting in deterministic responses). - Sends a request to the OpenAI API to generate a chat completion. This request includes the chosen model, the temperature setting, and the array of messages as context.
- Extracts and returns the generated message from the API's response. If no message is returned, it defaults to an empty string.
- Initializes a new configuration for the OpenAI API client using an API key stored in
generateSummary Processes job applicant data to generate a summary evaluation, using data from multiple sources including a user-provided resume. It performs several key operations, interfacing with a MongoDB database and handling various asynchronous operations.
- Initializes or updates an entry in the Summary collection with the applicant's name, contact information, and resume ID.
- Extracts content from the resume PDF and updates the summary record with this extracted content.
- Sends a request to OpenAI with the resume content and job details for summarization.
- Updates the summary record with the response received from OpenAI.
save generateSummary wrapper function.
- Checks for an existing entry in the Jobs collection and creates one if it's absent.
- Manages error scenarios related to summaries as reported by the browser extension.
- Executes the generateSummary function to start the summarization.
evaluate Prepares and sends request to OpenAI, processes the response, and handles any potential errors.
get-summary-status Returns MongoDB summary object by jobId and userId.
prev-results Returns MongoDB summary objects by jobId and userId.
save-job-detail Creates or updates the MongoDB Jobs collection entry with jobTitle, jobLocation, and jobDetailText.
Configuring new environment variables
Add the following environment variables to your server's configuration to manage sensitive data and configurations:
- OPENAI_API_KEY: Your secret API key obtained from OpenAI.
- BUCKET_FOR_CANDIDATES: The bucket name that you created previously.
Add these 2 lines to .env
file
OPENAI_API_KEY=your_openai_api_key_here
BUCKET_FOR_CANDIDATES=your_bucket_name_here
Changing files
- src/server: We just included
llm-routes.ts
in it
Here is the updated version of the server.ts
file
https://github.com/async-labs/browser-extension/blob/main/chapters/5-end/server/src/server.ts
You've reached the end of the Chapter 5 preview. To continue reading, you will need to purchase the book.
We keep our book up to date with recent libraries and packages.