Builder Book logo

Book: Builder Book

  1. Introduction
  2. Project structure. ESLint. Next.js. HOC. Material-UI. Server-side rendering. Styles.
  3. Server. Database. Session. Header and MenuDrop components.
  4. Authentication HOC. Promise. Async/await. Static method for User model. Google OAuth.
  5. Testing with Jest. Transactional emails with AWS SES API. In-app notifications.
  6. Book and Chapter data models. Internal API infrastructure, API methods and Express routes. ReadChapter page.
  7. Set up Github API infrastructure. Sync content API infrastructure. Missing UI infrastructure for Admin user. Redirects for Admin and Customer users. Testing.
  8. Table of Contents. Highlight for section. Hide Header. Mobile browser.
  9. BuyButton component. Buy book API infrastructure. Setup at Stripe dashboard and environmental variables. isPurchased and ReadChapter page. Redirect. My books API and MyBooks page. Mailchimp API.
  10. Prepare project for deployment. Environmental variables, production/development. Logger. SEO, robots.txt, sitemap.xml. Google Analytics. Compression and security. Heroku. Testing deployed project. AWS Elastic Beanstalk.

Chapter 6: Set up Github API infrastructure. Sync content API infrastructure. Missing UI infrastructure for Admin user. Redirects for Admin and Customer users. Testing.

The section below is a preview of Builder Book. To read the full text, you will need to purchase the book.

We are currently refactoring and updating the code in Builder Book. To reduce confusion, we have temporarily disabled the buy button. You can follow our progress here: https://github.com/builderbook/builderbook/issues/359

You can still buy our second book, SaaS Boilerplate: https://builderbook.org/books/saas-boilerplate/introduction-project-structure


In Chapter 6, you'll start with the codebase in the 6-begin folder of our builderbook repo and end up with the codebase in the 6-end folder. We'll cover the following topics in this chapter:

  • Set up Github API infrastructure
    - setupGithub method
    - Github-related Express routes

  • Sync content API infrastructure
    - Github server-side methods
    - Book.syncContent and Chapter.syncContent
    - markdownToHtml and getSections methods
    - Express routes for Admin user
    - API methods for Admin user

  • Missing UI infrastructure for Admin user

  • Two improvements
    - routesWithSlug
    - Redirect UX for Admin and Customer users

  • Testing
    - Connecting Github
    - Adding new book
    - Editing existing book
    - Syncing content


In the previous chapter (Chapter 5), we built a complete internal API infrastructure and corresponding pages twice:
- We rendered a list of books on the main Admin page (pages/admin/index.jsx).
- We rendered chapter content on the ReadChapter page (public/read-chapter.jsx).

In this chapter, we will build two new API infrastructures so that our web application's server can communicate with Github's server to retrieve data from a repository. Our goal is to access a repository's content on Github's server and save it to our MongoDB database using our server.

These two new API infrastructures are:
- Set up Github API
- Sync content API

Besides building server-only infrastructure, we need to add corresponding UI code as well, for example a Connect Github button and a Sync with Github button. Once implemented, an Admin user is able to create a new book and pull a book's content from a user's Github repository (repo). An Admin user should be able to get updated content from a Github repo and save this updated content to a MongoDB database.

link Set up Github API infrastructure

This is the section where we implement server-side only code for our Github API integration. Let's first discuss why we chose Github as our content management system. First, Github's markdown is familiar to most web developers, and we built our Builder Book web application specifically for web developers (and, more broadly, for software engineers). Our project's Admin user is a web developer who writes, edits, and hosts free or paywalled content using markdown. We prefer Visual Studio code editor (VS editor) for code and content editing. VS editor generally has better scrolling and faster navigation, and it lets you save your progress offline.

The second reason for using Github for content hosting is cloud storage for media files, such as images. Without Github, we would have to integrate with AWS S3 or another storage solution. You can simply upload, say, an image file to a Github issue and then copy-paste the link for the uploaded image to your content.

Finally, pushing updates to content is easy with Github. You can see all commits and quickly access a summary of all changes per commit.

In this section, our goal is to implement the server-only code of our Github API infrastructure. Later in this chapter, we will work on user interface (UI) code. For example, we will implement Connect Github and Sync with Github buttons that an Admin user clicks to authorize our web application on Github and to get the latest content from a Github's repository.

OAuth flow for Github is similar to the one for Google (see Chapter 3). A summary of all request-response cycles for Set up Github API infrastructure can be displayed in the same way as for Google OAuth API:
Builder Book

On the browser, an Admin user who did not connect Github to our web application, will see a Connect Github button inside the Header component. When an Admin user clicks on this button, the browser sends a request to our server at the Express route /auth/github. Our server sends a response back (res.redirect) and tells the browser to redirect the user from our web application to a Github OAuth page hosted by Github. The Admin user then needs to log in to Github and give permission for our web application to access some data on Github. After permission is given, the Github server sends a request to our server. Our server, if no error is caught, updates the corresponding User MongoDB document with the following new fields:
- isGithubConnected with value true
- githubAccessToken with a value provided by the request from the Github server
- githubId with a value provided by the Github server
- githubUsername with a value provided by the Github server

Finally, our server redirects a user to either the /admin page (in the case of success) or to /admin?error=${error.toString()} (in the case of error). We show any caught error to the Admin user using the notify method we defined earlier, in Chapter 4.


link setupGithub method

In order for our Set up Github API infrastructure to work, we need to mount Github-related Express routes on our Express server. Without the Express routes /auth/github and /auth/github/callback, the API infrastructure that we discussed in the above section simply does not work.

How do we mount new Express routes on our Express server? For an internal API, it is straightforward. You find the appropriate file inside the server/api/* folder and add any new Express route there. Because we call api(server); on our Express route, all Express routes inside the server/api/* folder are successfully mounted when the server starts.

But what about Express routes for external APIs? Recall that for our external Google OAuth API, we created a new method, setupGoogle, that takes server as an argument. Open server/google.js:

function setupGoogle({ server, ROOT_URL }) {
    const verify = async (accessToken, refreshToken, profile, verified) => {
        // some code
    };

    // some passport methods

    server.get(
        '/auth/google',
        // some code
    );

    server.get(
        '/oauth2callback',
        // some code
    );

    server.get('/logout', (req, res) => {
        req.logout();
        res.redirect('/login');
    });
}

Then we imported and called setupGoogle inside our server/server.js file like this:

setupGoogle({ server, ROOT_URL });

Once called, we have three Google-related Express routes mounted on our Express server: /auth/google, /oauth2callback, and /logout.

This is exactly how we will add two Github-related Express routes - we will define a setupGithub method that takes server as an argument and contains the definition of the two new Express routes. Create a new file, server/github.js, with the following content:

const { Octokit } = require('@octokit/rest');
const fetch = require('node-fetch');
const { oauthLoginUrl } = require('@octokit/oauth-authorization-url');
const _ = require('lodash');

const User = require('./models/User');

require('dotenv').config();

const dev = process.env.NODE_ENV !== 'production';
const CLIENT_ID = dev ? process.env.GITHUB_TEST_CLIENTID : process.env.GITHUB_LIVE_CLIENTID;
const API_KEY = dev ? process.env.GITHUB_TEST_SECRETKEY : process.env.GITHUB_LIVE_SECRETKEY;

function setupGithub({ server, ROOT_URL }) {
    const verify = async ({ user, accessToken, profile }) => {
        // defined later
    };

    server.get('/auth/github', (req, res) => {
        // define later
    });

    server.get('/auth/github/callback', async (req, res) => {
        // defined later
    });
}

exports.setupGithub = setupGithub;

So far, we haven't defined the two Github-related Express routes /auth/github and /auth/github/callback. We will do so in the next subsection. We will also define a verify method that gets called inside the Express route /auth/github/callback when an Admin user successfully grants permission on the Github OAuth page and no error was caught in the process.


link Github-related Express routes

In this subsection, we discuss and define the new Express routes /auth/github and /auth/github/callback. We haven't written any of the client-side only code yet. We decided to start building our new API infrastructure from server-only code. Later in this chapter, we will create a Connect Github button inside the Header component. For now, you just need to know that when an Admin user clicks the Connect Github button, the browser sends a request to the /auth/github route and initiates a request-response cycle in our API infrastructure.

Both of the new Express routes will receive a request with the method GET. And in both of Express routes, we need to check if a user is logged-in and if a user is an Admin. This is a typical check for Express routes that are not public. To check if a user is logged-in, we check for truthiness of req.user. To check if a user is an Admin user, we check for truthiness of req.user.isAdmin:

if (!req.user || !req.user.isAdmin) {
    res.redirect(`${ROOT_URL}/login`);
    return;
}

In the first Express route, /auth/github, we want to achieve a few things:
- To generate url to which our web application will redirect an Admin user after he/she click the Connect Github button. url is the URL of the Github OAuth page, where an Admin user logs in using his/her Github account and gives permission for our web application to access data on Github.
- To generate state and save its value to req.session.githubAuthState. Later, when our server receives a request from Github's server, we will compare the value saved as req.session.githubAuthState to the req.query.state value provided by Github's server. If the values match, then we indeed redirected an Admin user to the Github OAuth page and received a request from Github's server.
- Finally, we want to redirect an Admin user from our web application to Github's. This is straightforward: res.redirect(url);

To generate url and state values, we use the oauthLoginUrl method from the @octokit/oauth-authorization-url package. Here is an example of usage:
https://github.com/octokit/oauth-authorization-url.js#usage

In our case, we use it like this:

const { url, state } = oauthLoginUrl({
    clientId: CLIENT_ID,
    redirectUrl: `${ROOT_URL}/auth/github/callback`,
    scopes: ['repo', 'user:email'],
    log: { warn: (message) => console.log(message) },
});

Put all of the above information together and you get our first Express route, /auth/github:

server.get('/auth/github', (req, res) => {
    if (!req.user || !req.user.isAdmin) {
        res.redirect(`${ROOT_URL}/login`);
        return;
    }

    const { url, state } = oauthLoginUrl({
        clientId: CLIENT_ID,
        redirectUrl: `${ROOT_URL}/auth/github/callback`,
        scopes: ['repo', 'user:email'],
        log: { warn: (message) => console.log(message) },
    });

    req.session.githubAuthState = state;
    if (req.query && req.query.redirectUrl && req.query.redirectUrl.startsWith('/')) {
        req.session.next_url = req.query.redirectUrl;
    } else {
        req.session.next_url = null;
    }

    res.redirect(url);
});

The first Express route, /auth/github, receives a GET request from the browser after an Admin user clicks the Connect Github button. The second Express route, /auth/github/callback, receives a GET request from the browser as well, but Github's server sends a redirect to the /auth/github/callback command after successful authorization of an Admin user on Github.

Inside the second Express route, /auth/github/callback, we want to:
- Check if req.session.githubAuthState and req.query.state values match. If thye don't, we redirect a user to the Admin page using a URL that contains an error query
- Send a request to Github's server using the fetch method from the node-fetch package and get a response with a so called-access token. - Access an Admin user's profile on Github using the access token.
- Call the verify method and pass the access token and profile data to it as arguments.
- If all successful, redirect an Admin user to /admin page. If an error is caught, redirect to /admin?error=${error.toString().
- Define a verify method that finds and updates a corresponding User MongoDB document.

By this point of the book, you can write code for most of the above steps. However, let's discuss two steps in more detail.

The fetch method from node-fetch works similarly to fetch from isomorphic-unfetch. With the latter, we send a request from the browser to the server. With the former, in this specific case, we send a server-to-server request. But the method's arguments are the same. Check up the lib/api/sendRequest.js file to see how fetch works. In this case, we send a POST request from our server to Github's server. Our goal is to get a response that contains an access token:

try {
    const response = await fetch('https://github.com/login/oauth/access_token', {
        method: 'POST',
        headers: { 'Content-type': 'application/json;', Accept: 'application/json' },
        body: JSON.stringify({
            client_id: CLIENT_ID,
            client_secret: API_KEY,
            code: req.query.code,
            state: req.query.state,
            redirect_uri: `${ROOT_URL}/auth/github/callback`,
        }),
    });

const resData = await response.json();

Getting profile is not an obvious task as well. First, we create an access token for authentication using Octokit from the @octokit/rest package:
https://octokit.github.io/rest.js/v18#authentication

Like this:

const github = new Octokit({
    auth: resData.access_token,
    request: { timeout: 10000 },
});

Then we use the octokit.users.getAuthenticated Github API method to access a user's profile at Github:
https://octokit.github.io/rest.js/v18#users-get-authenticated

Like this:

const profile = await github.users.getAuthenticated();

Now we are ready to put our second Express route together using the above discussion:

server.get('/auth/github/callback', async (req, res) => {
    if (!req.user || !req.user.isAdmin) {
        res.redirect(`${ROOT_URL}/login`);
        return;
    }

    const { next_url, githubAuthState } = req.session;

    let redirectUrl = ROOT_URL;

    if (next_url && next_url.startsWith('/')) {
        req.session.next_url = null;         redirectUrl = `${ROOT_URL}${next_url}`;
    }

    if (githubAuthState !== req.query.state) {
        res.redirect(`${redirectUrl}/admin?error=Wrong request`);
    }

    try {
        const response = await fetch('https://github.com/login/oauth/access_token', {
            method: 'POST',
            headers: { 'Content-type': 'application/json;', Accept: 'application/json' },
            body: JSON.stringify({
                client_id: CLIENT_ID,
                client_secret: API_KEY,
                code: req.query.code,
                state: req.query.state,
                redirect_uri: `${ROOT_URL}/auth/github/callback`,
            }),
        });

        const resData = await response.json();

        const github = new Octokit({
            auth: resData.access_token,
            request: { timeout: 10000 },
        });

        const profile = await github.users.getAuthenticated();

        await verify({
            user: req.user,
            accessToken: resData.access_token,
            profile: profile.data,
        });
    } catch (error) {
        console.error(error.toString());

        res.redirect(`${redirectUrl}/admin?error=${error.toString()}`);
    }

    res.redirect(`${redirectUrl}/admin`);
});

Finally, we need to find the User MongoDB document of our Admin user and update it by adding Github-related fields. We do so inside the verify method:

const verify = async ({ user, accessToken, profile }) => {
    const modifier = {
        githubId: profile.id,
        githubAccessToken: accessToken,
        githubUsername: profile.login,
        isGithubConnected: true,
    };

    if (!user.displayName) {
        modifier.displayName = profile.name || profile.login;
    }

    if (!user.avatarUrl && profile.avatar_url) {
        modifier.avatarUrl = profile.avatar_url;
    }

    await User.updateOne({ _id: user._id }, modifier);
};

We used Mongoose's API method updateOne to find a document and update all relevant fields in it.

We've completed our Set up Github API infrastructure. This allows us to access a user's data on Github. However, we did not write any code related to accessing a repo's content. Our goal is to access a repo's content and save the content to our MongoDB database. When a Public or Customer user loads ReadChapter, we will serve the content from our database.


link Sync content API infrastructure

In the previous section, we created server-side only code that ultimately allows our web application to access a user's data on Github. We haven't written any code that actually accesses a repository's content and saves it to a MongoDB database. Also, we haven't created any user interfaces (for example, buttons).

In this section, we will work on our Sync content API infrastructure. At the end of this chapter, we will work on the user interfaces.

The Sync content API infrastructure can be summarized with the following diagram:
Builder Book

As you can see, there are some similarities to our Set up Github API infrastructure. The main similarity is that the Admin user initiates both APIs with clicking buttons. Set up Github API gets initiated by clicking the Connect Github button and Sync content API by clicking the Sync with Github button. Another similarity is that we will use the Github API method. Inside Set up Github API, we used the octokit.users.getAuthenticated Github API method. In Sync content API, we will use:
- octokit.repos.listForAuthenticatedUser
- octokit.repos.getContent
- octokit.repos.listCommits

But the rest of the API infrastructure is different. We need to create two new static methods - one for our Book data model and one for our Chapter data model. Inside the Book.syncContent and Chapter.syncContent methods, we need to write logic that takes content from .md files of a repository and compares it to the content saved in our database. If the content is outdated, we need to save content from the repository to the database.

In the next subsection, we will discuss Github server-side methods and corresponding Github API methods. Then we will discuss and define Book.syncContent and Chapter.syncContent.


You've reached the end of the Chapter 6 preview. To continue reading, you will need to purchase the book.

We are currently refactoring and updating the code in Builder Book. To reduce confusion, we have temporarily disabled the buy button. You can follow our progress here: https://github.com/builderbook/builderbook/issues/359

You can still buy our second book, SaaS Boilerplate: https://builderbook.org/books/saas-boilerplate/introduction-project-structure

format_list_bulleted
help_outline
lens