1. Project Overview & Quickstart (epicweb-dev/epic-stack)
File: README.md
npx epicliWatch Kent's Introduction to The Epic Stack
"The Epic Stack" by Kent C. Dodds
Docs
Read the docs
(please 🙏).
Support
- 🆘 Join the
discussion on GitHub
and the KCD Community on Discord. - 💡 Create an
idea discussion
for suggestions. - 🐛 Open a GitHub issue to
report a bug.
Branding
Want to talk about the Epic Stack in a blog post or talk? Great! Here are some
assets you can use in your material:
EpicWeb.dev/brand
Thanks
You rock 🪨
File: app/components/ui/README.md
shadcn/ui
Some components in this directory are downloaded via the
shadcn/ui CLI. Feel
free to customize them to your needs. It's important to know that shadcn/ui is
not a library of components you install, but instead it's a registry of prebuilt
components which you can download and customize.
File: docs/decisions/000-template.md
Title
Date: YYYY-MM-DD
Status: proposed | rejected | accepted | deprecated | … | superseded by
0005
Context
Decision
Consequences
File: docs/decisions/001-typescript-only.md
TypeScript Only
Date: 2023-05-08
Status: accepted
Context
The create-remix CLI allows users to select whether they want to use
JavaScript instead of TypeScript. This will auto-convert everything to
JavaScript.
There is (currently) no way to control this behavior.
Teams and individuals building modern web applications have many great reasons
to build them with TypeScript.
One of the challenges with TypeScript is getting it configured properly. This is
not an issue with a stack which starts you off on the right foot without needing
to configure anything.
Another challenge with TypeScript is handling dependencies that are not written
in TypeScript. This is increasingly becoming less of an issue with more and more
dependencies being written in TypeScript.
Decision
We strongly advise the use of TypeScript even for simple projects and those
worked on by single developers. So instead of working on making this project
work with the JavaScript option of the create-remix CLI, we've decided to
throw an error informing the user to try again and select the TypeScript option.
We've also made the example script in the README.md provide a selected option
of --typescript so folks shouldn't even be asked unless they leave off that
flag in which case our error will be thrown.
Consequences
This makes the initial experience not great for folks using JavaScript.
Hopefully the Remix CLI will eventually allow us to have more control over
whether that question is asked.
This also may anger some folks who really don't like TypeScript. For those
folks, feel free to fork the starter.
File: docs/decisions/002-email-service.md
Email Service
Date: 2023-05-08
Status: superseded by 017
Context
When you're building a web application, you almost always need to send emails
for various reasons. Packages like nodemailer make it quite easy to send your
own emails through your own mailserver or a third party's SMTP server as well.
Unfortunately,
deliverability will suffer if you're not using a service.
The TL;DR is you either dedicate your company's complete resources to "play the
game" of email deliverability, or you use a service that does. Otherwise, your
emails won't reliably make it through spam filters (and in some cases it can
just get deleted altogether).
The guiding principles
discourage services and encourage quick setup.
Decision
We will use a service for sending email. If emails don't get delivered then it
defeats the whole purpose of sending email.
We selected Mailgun because it has a generous free
tier and has proven itself in production. However, to help with quick setup, we
will allow deploying to production without the Mailgun environment variables set
and will instead log the email to the console so during the experimentation
phase, developers can still read the emails that would have been sent.
During local development, the Mailgun APIs are mocked and logged in the terminal
as well as saved to the fixtures directory for tests to reference.
Consequences
Developers will need to either sign up for Mailgun or update the email code to
use another service if they prefer. Emails will actually reach their
destination.
File: docs/decisions/003-sqlite.md
SQLite
Date: 2023-05-13
Status: accepted
Context
SQLite is a sql-based database with a particularly unique feature: the entire
database is in a single file. Largely for this reason, I've historically seen it
as a simple database for simple use cases.
However, in recent years SQLite has received a great deal of development and
attention making it a simple database for even more advanced use cases. The fact
that SQLite is a single file on disk comes with a major benefit of 0 latency
which
drastically reduces the "n+1 problem".
Another issue is database size, however,
SQLite is capable of handling databases that are an Exabyte in size
(that's one million Terabytes, or one billion Gigabytes 🤯).
SQLite does not support subscriptions which can be a limitation on certain
real-time use cases. However, there are plenty of reasons to recommend against
using database subscriptions for real-time use cases anyway, and that is the
case in the Epic Stack (as of today, we don't have real-time examples or docs,
but when we do it's likely we wouldn't use database subscriptions anyway).
SQLite being a file on disk does make connecting from external clients
effectively impossible. This makes it more difficult to connect to it using
database admin tools. However, it is possible to run tools like prisma studio
on the machine where the sqlite volume is mounted if necessary. And theDockerfile is already configured to allow easy SSH connections into the sqlite
CLI. This is nowhere near as good as a proper admin tool, but there's likely
more that could be done here to improve the experience. On my own website
(kentcdodds.com), I am able to have prisma studio run in production protected by
authentication and I expect we'll be able to add something like that to the Epic
Stack in the future.
SQLite does not support plugins like
TimescaleDB for Postgres. While
time-series data is possible with SQLite, I do not have experience with this use
case and can't speak to the challenges there. My intuition says it's not
advisable to use SQLite for that use case.
SQLite does not support enums which means you're forced to use strings. I have
mixed feelings about this, but I mostly don't like enums anyway. The main
drawback to this is when it comes to the typings for the client which doesn't
allow you to ensure all values of a column are only within a set of specific
possible values for the string. However, with Prisma client extensions, handling
this kind of enforcement at the client (and typing) level should be possible.
This would need to be documented in the future
#29.
As a file on disk, you cannot "distribute" SQLite directly. However, with tools
like Turso or LiteFS, you
can make that work. And both of these tools even have built-in solutions to the
"Read Replica Consistency" challenge. So if you need your app to run in multiple
instances, you need to use one of these tools.
Using SQLite is an enormous simplification both during development and
production. With one less service to worry about (especially something as
critical as your database), you're less likely to experience an outage.
Additionally, all you need is a persisted volume for your application (which you
would need for a database service anyway), so it's less costly as well.
Decision
We'll use SQLite because it satisfies the use cases of our target audience.
Consequences
This means we need to have a way to connect to our SQLite database in
production. We'll also need to have a way to easily seed the database
(documentation will need to be written and perhaps some scripts). We want to
support multi-region so LiteFS needs to be configured.
This also means real-time use cases will need to find a solution that doesn't
rely on database subscriptions.
Ultimately, this decision drastically simplifies development, deployment,
maintenance, and services for the web application and reduces running costs. A
huge win for the majority of web applications.
File: docs/decisions/004-github-actions.md
GitHub Actions
Date: 2023-05-15
Status: accepted
Context
Deploying serious web applications to production on a regular basis requires
automation. Running testing, linting, and a build before deployment is a
accepted practice to ensure a quality product is deployed.
Alongside this, we want to run this automation any time code is merged (or will
soon be merged) from contributors to the project. This is called "Continuous
Integration" and is necessary for teams to move confidently and focus on their
value proposition. Make the robots do the boring stuff so we can focus on the
creative work.
We can run this automation on our own machines during development, but it can be
easy to forget to do this. It's even harder to be confident the automation was
run when you are trying to combine the work of multiple people (who's
responsible to run the deploy script?). So it's best to have this automation run
on a separate machine that's dedicated to this task.
I don't want to have that machine running in my closet, so instead we need to
look to outside services for managing this for us. There are many such services.
Most of the target audience of the Epic Stack are familiar with GitHub and many
use it already for other projects. The vast majority already have accounts on
GitHub as well.
GitHub has a CI service called
GitHub Actions which satisfies all the
necessary use cases for the Epic Stack. It does not require an additional
account, though it does necessitate you use GitHub for hosting the code. Also,
it is free for open source projects, but paid for private projects.
Decision
We've decided to use GitHub Actions for Continuous Integration in the Epic
Stack. We have a single action that handles running linting, type checking,
tests, and deployment to both staging (dev branch) and production (main
branch).
Consequences
This means users of the Epic Stack must host their repositories within GitHub
and pay for private repositories. This reduces the number of extra services
users of the Epic Stack need to sign up for since the vast majority already have
accounts with GitHub anyway.
File: docs/decisions/005-client-pref-cookies.md
Client Preference Cookies
Date: 2023-05-16
Status: accepted
Context
Server rendering is preferred for most things to avoid content layout shift
(which causes a poor user experience). Unfortunately, there are a number of
limitations with this because the browser doesn't give the server enough
information about the user's preferences. For example:
prefers-color-scheme(light/dark mode)prefers-reduced-data- time zone offset
- locale
And much more.
The problem is that if what you display to the user relies on these values, then
what the server renders could be wrong and the user will see the incorrect UI
until client-side JavaScript can take over and correct it which causes a "Flash
of Incorrect UI." This is a terrible user experience.
This is such an issue that the web platform will (hopefully soon) be adding new
user preferences headers to each request for us to know these values on the
server. Read,
User preference media features client hints headers
and
User Locale Preferences.
However, there's no telling when these features will become standard and
implemented in all browsers Epic Stack apps target so we cannot rely on this or
wait for it.
One solution I've used on my personal website is to include a tiny bit of inline
JavaScript in the client that runs before the user has a chance to see anything
to correct any flaws in the UI. This is normally enough, but on extremely slow
connections the user will still see a flash of incorrect UI. It's also quite
complex and pretty hacky.
Another solution I've seen from
Jacob Paris is to simply
use cookies to solve this problem. You simply set a cookie in the browser from
the client for every property your application needs and then the server knows
the user preference during the document request and server render. There are two
limitations here:
- First time users won't have the cookie set
- The cookie will be stale if the user changes their preference
To solve the first problem, we can simply check that the cookies are set and if
they are not, then we instead send a minimal document that includes a tiny bit
of JavaScript that sets the cookies and then reloads the page. This is not
ideal, however it's effectively as harmful to the user as a redirect which many
websites do anyway (for example, go to https://youtube.com and you instantly
get redirected to https://www.youtube.com). Additionally, this is effectively
how
the draft specification
for the web platform's solution works as well.
To solve the second problem, we can simply keep a tiny bit of JS in the head of
the document that does a quick check of the cookie values and if they are stale,
then it sets them again and triggers a reload. Still not ideal, but again, it's
better than a content layout shift. And hopefully this solution isn't permanent
and we can remove it once the web platform offers a better solution.
To take things further, we can future proof this solution a bit by trying to
adhere to the web platform's proposed solution as closely as possible, so that
when it does become available, we can simply switch from the cookies to headers
and remove the JS, leaving us with few changes to make.
Decision
Even though the web platform is working on a solution for this, we cannot wait
for it. Despite page reloads being a sub-optimal user experience, it's better
than the content layout shift (flash of incorrect UI) alternative. Therefore, we
will use cookies and reloads to solve this problem.
Consequences
The user's first page load will be a bit slower than normal (as will any page
load after their preferences change) because we have to do a page reload to set
the cookies. However, this is a one-time cost and the user will not experience
this again until they change their preferences.
The user will not experience content layout shift for any user preferences our
app depends on for the server render. This is a huge win for user experience.
If a user has cookies disabled, then we need to detect that and fallback to
default values. Users in this situation will experience content layout shift,
but there's nothing else we can do about that. Additionally, users who have
cookies disabled will not be able to authenticate which is a separate concern
that should be addressed in a different decision document.
File: docs/decisions/006-native-esm.md
Native ESM
Date: 2023-05-18
Status: accepted
Context
Oh boy, where do I start? The history of JavaScript modules is long and
complicated. I discuss this a bit in my talk
More than you want to know about ES6 Modules.
Many modern packages on npm are now publishing esm-only versions of their
packages. This is fine, but it does mean that using them from a CommonJS module
system requires dynamic imports which is limiting.
In Remix v2, ESM will be the default behavior. Everywhere you look, ESM is
becoming more and more the standard module option. CommonJS modules aren't going
anywhere, but it's a good idea to stay on top of the latest.
Sadly, this is a bit of a "who moved my cheese" situation. Developers who are
familiar with CommonJS modules will be annoyed by things they were used to doing
in CJS that they can't do the same way in ESM. The biggest is dynamic (and
synchronous) requires. Another is the way that module resolution changes. There
are some packages which aren't quite prepared for ESM and therefore you end up
having to import their exports directly from the files (like radix for example).
This is hopefully a temporary problem.
Decision
We're adopting ESM as the default module system for the Epic Stack.
Consequences
Experienced developers will hit a couple bumps along the way as they change
their mental model for modules. But it's time to do this.
Some tools aren't very ergonomic with ESM. This will hopefully improve over
time.
File: docs/decisions/007-sessions.md
Sessions
Date: 2023-05-25
Status: accepted
Context
There are a lot of ways to manage authentication in applications on the web.
We've chosen to use cookies to identify a user. However, you can just stick the
user's ID in that cookie and then sign it with a secret and then you'll always
know that the user ID in the cookie is legitimate and you won't need to go to
the database at all to determine who is making the request (or, at least to know
their ID).
The limitation here is that once a client has a signed cookie with the user ID,
it cannot be revoked. You can set a maxAge or a expires, but you can't
proactively revoke it. There's also not a great way to know how many sessions
are currently active for a given user.
For many applications, being able to proactively invalidate sessions is
necessary. My own site has this capability. Many sites will even associate some
identifiable information about each session as well as the last time that
session was used to display that information to the user and allow them to
revoke specific sessions.
Decision
We will add a sessions table to the built-in template's schema allowing a
capability for a simple "revoke all" feature. Adding more information to the
session model would be simple for anyone needing a more sophisticated session
management strategy. This simple implementation gets people on the right foot
with regards to session management.
Consequences
This requires doing a database lookup for every authenticated request to
determine whether there is an active session in the database for the user's
request. This is not a problem for us since we're using SQLite and there's 0
latency, so the query is extremely fast.
The sessions table does not currently have the capability to proactively delete
expired sessions which means it could fill up indefinitely. This would be a good
problem for built-in cron job support to solve eventually.
2. Official Technical Reference & Guides (epicweb-dev/epic-stack-docs)
File: README.md
npx create-epic-app@latestWatch Kent's Introduction to The Epic Stack
"The Epic Stack" by Kent C. Dodds
Docs
Read the docs
(please 🙏).
Support
- 🆘 Join the
discussion on GitHub
and the KCD Community on Discord. - 💡 Create an
idea discussion
for suggestions. - 🐛 Open a GitHub issue to
report a bug.
Branding
Want to talk about the Epic Stack in a blog post or talk? Great! Here are some
assets you can use in your material:
EpicWeb.dev/brand
Thanks
You rock 🪨
File: app/components/ui/icons/README.md
Icons
This directory contains SVG icons that are used by the app.
Everything in this directory is generated by npm run build:icons.
File: app/components/ui/README.md
shadcn/ui
Some components in this directory are downloaded via the
shadcn/ui CLI. Feel
free to customize them to your needs. It's important to know that shadcn/ui is
not a library of components you install, but instead it's a registry of prebuilt
components which you can download and customize.
File: content/docs/caching/index.mdx
title: Caching
date: 2023-11-02
categories:
- performance
- caching
meta:
keywords:- caching
- lru
- sqlite
- performance
The Epic Stack comes with caching utilities and a management dashboard that allows you to view and clear your cache. There are two caches built into the Epic Stack:
SQLite: This is a separate database from the main application database. It's managed by LiteFS so the data is replicated across all instances of your app. This can be used for long-lived cached values.
LRU: This is an in-memory cache that is used to store the results of expensive queries or help deduplicate requests for data. It's not replicated across instances and as it's in-memory it will be cleared when your app is restarted. So this should be used for short-lived cached values.
Caching is intended to be used for data that is expensive and/or slow to compute or retrieve. It can help you avoid costs or rate limits associated with making requests to third parties.
It's important to note that caching should not be the first solution to slowness issues. If you've got a slow query, look into optimizing it with database indexes before caching the results.
Using the cache
You won't typically interact directly with the caches. Instead, you will use cachified which is a nice abstraction for cache management.
We have a small abstraction on top of it which allows you to pass timings to work seamlessly with the server timing utility.
Let's say we're making a request to tito to get a list of events. Tito's API is kinda slow and our event details don't change much so we're ok speeding things up by caching them and utilizing the stale-while-revalidate features in cachified.
Here's how you would use cachified to do this:
import { cachified, cache } from '#app/utils/cache.server.ts'
import { type Timings } from '#app/utils/timing.server.ts'
const eventSchema = z.object({
/* the schema for events */
})
export async function getScheduledEvents({ timings, }: { timings?: Timings } = {}) {
const scheduledEvents = await cachified({
key: 'tito:scheduled-events',
cache,
timings,
getFreshValue: () => {
// do a fetch request to the tito API and stuff here
return [
/* the events you got from tito */
]
},
checkValue: eventSchema.array(),
// Time To Live (ttl) in milliseconds:
// the cached value is considered valid for 24 hours
ttl: 1000 * 60 * 60 * 24,
// Stale While Revalidate (swr) in milliseconds:
// if the cached value is less than 30 days expired,
// return it while fetching a fresh value in the background
staleWhileRevalidate: 1000 * 60 * 60 * 24 * 30,
})
return scheduledEvents
}With this setup, the first time you call getScheduledEvents it will make a
request to the tito API and return the results. It will also cache the results
in the cache (which is the SQLite cache). The next time you callgetScheduledEvents it will return the cached value if the cached value is less
than 30 days old. If the cached value is older than 24 hours, it will also make
a request to the tito API. If the cache value is more than 30 days old, it will
wait until the tito request is complete and then return the fresh value.
Bottom line: You make the request much less often and users are never waiting
for it. Every situation will require you think through the implications of
caching and acceptable stale-ness, but the point is you have those levers to
pull.
A lot more needs to be said on this subject (an entire workshop full!), but this
should be enough to get you going!
File: content/docs/authentication/index.mdx
title: Authentication
date: 2023-11-02
categories:
- authentication
- security
meta:
keywords:- authentication
- oauth
- openid
- sso
- passwords
- bcrypt
- totp
- two-factor-authentication
The Epic Stack manages its own authentication using web standards and established libraries and tools. By default, the Epic Stack offers you two mechanisms for authentication:
- Username and password authentication
- Provider authentication
Username and password authentication
When a user wishes to sign up for an account, they are asked for their email address. The Epic Stack will send them an email with a code as well as a link. The user can then enter the code or click the link to verify their email address which takes them through the onboarding flow which will allow them to set their username and password. The password is stored using the bcrypt algorithm.
Provider authentication
The Epic Stack ships with a system for third party authentication allowing you to easily add SSO (Single Sign On) to your application. The Epic Stack ships with support for GitHub OAuth2 authentication out of the box. But you can easily remove that and/or add other providers. It's all built using remix-auth, so any provider supported there, can be added, including web-oidc which handles OpenID Connect authentication and exports a remix-auth compatible auth strategy. You can check this example which shows using OpenID Connect to add Google authentication to the Epic Stack. You can expand beyond this to add any other provider you'd like, and if you need to support SAML, you may look into @boxyhq/remix-auth-sso.
TOTP and Two-Factor Authentication
Two factor authentication is built-into the Epic Stack. It's managed using a the @epic-web/totp (Time-based One Time Passwords) utility. You can read more about the decision to use TOTP in the totp decision document. The secret and other pertinent information is stored in a verification model (check the Prisma schema). This verification model is used as the basis for all TOTP secrets. This is used for non-expiring Two-Factor Authentication secrets as well as temporary TOTP codes which are emailed to verify a user's ownership of an email/account. So it's used for onboarding, forgot password, and change email flows. When a user has 2FA enabled on their account, they also are required to enter their 2FA code within 2 hours of performing destructive actions like changing their email or disabling 2FA. This time is controlled by the shouldRequestTwoFA utility in the login full stack component in the resource routes.
File: content/docs/apis/index.mdx
title: APIs
date: 2023-11-02
categories:
- api
meta:
keywords:- api
- rest
- graphql
- endpoints
- resource-routes
Remix routes have the ability to handle both backend code and UI code in the same file. Remix loaders and actions are backend code that's tightly coupled to the UI code for that route.
Additionally, you can define routes that don't have any UI at all. These are called resource routes. This allows you to create REST endpoints or a GraphQL endpoint to make your app data and logic consumable by third parties or additional clients (like a mobile app).
You can also use this to generate PDFs, images, stream multi-media and more.
The Epic Stack has a few resource routes in place for managing images, the cache, and even has a few "full stack components" for components that manage the connection with their associated backend code. Watch the talk.
So, yes, you can absolutely use the Epic Stack to build APIs for consumption by third party clients.
File: content/docs/client-hints/index.mdx
title: Client Hints
date: 2023-11-02
categories:
- performance
- rendering
meta:
keywords:- client hints
- cookies
- server render
- ssr
- hydration
NOTE: Find background on this concept in the decision document:
0005-client-pref-cookies.md.
The Problem
Sometimes your server render code needs to know something about the client that the browser doesn't send. For example, the server might need to know the user's preferred language, or whether the user prefers light or dark mode.
For some of this you should have user preferences which can be persisted in a cookie or a database, but you can't do this for first-time visitors. All you can do is guess. Unfortunately, if you guess wrong, you end up with a bad experience for the user.
And what often happens is we render HTML that's wrong and then hydrate the application to be interactive with client-side JavaScript that now knows the user preferences and now we know the right thing to render. This is great, except we've already render the wrong thing so by hydrating we cause a shift from the wrong thing to the right thing which is jarring and can be even a worse user experience than leaving the wrong thing in place (I call this a "flash of incorrect content"). You'll get an error in the console from React when this happens.
The Solution
Client hints are a way to avoid this problem. The standards for this are still a work in progress and there is uncertainty when they will land in all major browsers we are concerned with supporting.
So the Epic Stack comes with built-in support for a "ponyfill" of sorts of a similar feature to the client hints headers proposed to the standard.
The idea behind the standard is when the browser makes a request, instead of responding to the request immediately, the server instead responds to the client informing it there's a need for certain headers. The client will then repeat the request with those headers added. The server can then respond with the correct content.
Our solution is inspired by this, but instead of headers we use cookies. In app/utils/client-hints.tsx we have a component called ClientHintCheck which is rendered in the <head> of our document before anything else.
That component renders a small and fast inline script which checks the user's cookies for the expected client hints. If they are not present or if they're outdated, it sets a cookie and triggers a reload of the page. Effectively doing the same thing the browser would do with the client hints headers.
This allows us to server render the right thing for first time visitors without triggering a content layout shift or a flash of incorrect content. After that first render, the client will have the correct cookies and the server will render the right thing every time thereafter.
The built-in template currently only has support for the user's color scheme preference and the user's timezone, but you can add more for your application's needs.
File: content/docs/community/index.mdx
title: Epic Stack Community
date: 2023-11-02
categories:
- community
meta:
keywords:- community
- videos
- examples
- libraries
Here you can find useful learning resources and tools built and maintained by
the community, such as libraries, examples, articles, and videos.
Learning resources
The primary learning resources for the Epic Stack is
EpicWeb.dev, EpicReact.dev,
and TestingJavaScript.com. On these you will
find free and paid premium content that will help you build epic web
applications (with or without the Epic Stack).
The community has put together some additional learning resources that you may
enjoy!
Videos
- Dark Mode Toggling using Client-preference cookies by
@rajeshdavidbabu - Youtube
link
File: content/docs/database/index.mdx
title: Database
date: 2023-11-02
categories:
- database
meta:
keywords:- database
- sqlite
- litefs
- fly.io
Primary Instance
The way that LiteFS works is that it has a "primary instance" that is the only
instance that can write to the database. All other instances are "replicas" that
can only read from the database. This is a pretty common pattern for databases
and it's how Fly's Postgres service works as well.
The primary instance is determined by Fly's consul
service and can change over time. By default, all instances are allowed to be
primary instances. However, writes will be slower for people who are not
geographically close to the primary instance, so the Epic Stack has configured
consul to only allow instances in the primary region and it's recommended that
you choose a primary region that's closest to most of your users. This
configuration can be found in other/litefs.yml. The PRIMARY_REGION is
determined from the primary_region property in the fly.toml file.
When you initialize an Epic Stack app, it will ask you which region you wish to
deploy to and this will set the primary_region in the fly.toml file. It is
recommended that you deploy two instances in this region so you have zero
downtime deploys.
You can determine which instance is the primary instance by running the
following command:
fly status --app [YOUR_APP_NAME]This will show you a table of all your instances and the primary instance will
have "ROLE" set to "primary".
To deploy more regions, you can run fly scale count. For example, if I wanted
to have two instances in sjc (my primary region) and one in ams, I would
run:
fly scale count 2 --region sjc
fly scale count 1 --region amsConnecting to your production database
The location of the sqlite database is kinda funny. The real location is in/data/litefs/dbs/sqlite.db. However, during development you connect to it via
the fake filesystem managed by LiteFS so it can propagate any changes to your
database to all replicas.
So to connect to your database, you'll want to connect to it at/litefs/data/sqlite.db in the deployed application. Because it is SQLite, you
cannot connect to it unless you're running a command-line session on the
machine. You can do this using fly ssh console. The Dockerfile simplifies this
further by adding a database-cli command. You can connect to the live database
by running fly ssh console -C database-cli.
To connect to the deployed database from your local machine using Prisma Studio,
you can utilize Fly's ssh and proxy commands.
- Run in one terminal the command to start Prisma Studio on your desired Fly appsh
fly ssh console -C "npm run prisma:studio" --app [YOUR_APP_NAME] - Run in a second terminal the command to proxy your local port 5556 to Prisma
Studioshfly proxy 5556:5555 --app [YOUR_APP_NAME]
To work with Prisma Studio and your deployed app's database, simply openhttp://localhost:5556 in your browser.
Note: You may want to add
--selectto thefly ssh consolecommand to
select the instance you want to connect to if you have multiple instances
running. Otherwise you could connect to a non-primary instance. The easiest
way to determine the primary instance (because it can change) is to runfly statusand the row that has "ROLE" set to "primary" is the primary
instance. Alternatively, you can open the deployed application and check the
request headers. One of them will beFly-Primary-Instancewhich will tell
you the instance ID of the primary instance.
Migrations
Thanks to Prisma, we've got a great mechanism for handling database migrations.
Any migrations necessary are run (by the primary instance only) as part of the
deploy process. You can find this in the other/litefs.yml file.
We deploy to multiple instances at once and the way we deploy means we don't
have any downtime during deploys. However, to make this work, you do need to
make sure you can run two versions of your app at once. Specifically, you should
not deploy "breaking schema changes" to your app in a way that will break any
running instances. This is a pretty typical requirement for production
applications.
The basic idea is that you maintain support for any client that is currently
running. In Epic Stack apps that's normally just the currently running version
of the app. So you really only need to worry about "breaking schema changes"
whenever you deploy a new version, but you don't have to worry about avoiding
breaking schema changes for old versions of the app that are no longer running.
In practice, this means you should adopt a "widen then narrow" strategy for
schema migrations. This is a pretty common practice, but here's a simple example
of how this works (each step here is an individual deploy):
- Widen app to consume A or B
- Widen db to provide A and B and the app to write to both A and B
- Narrow app to consume B and only write to B
- Narrow db to provide B
So, let's say that today your app allows users to provide a "name" and you want
to change that to firstName and lastName instead. Here's how you'd do that
(again, each of these steps end in a deploy):
- Widen app to consume
firstNameandlastNameorname. So all new code
that references thefirstNameandlastNamefields should fallback to thenamefield and not error if thefirstNameandlastNamefields don't
exist yet, which it won't at this point. - Widen db to provide
firstNameandlastNameandname. So thename
field should be populated with thefirstNameandlastNamefields. You can
do this as part of the migration SQL script that you run. The easiest way to
do this is to generate the migration script to add the fields usingprisma migrateand then modify the script to copy the existing data in thenamefield to thefirstNamefield (maybe with the help of VSCode Copilot
😅). - Narrow app to consume
firstNameandlastNameby only writing to those
fields and removing the fallback to thenamefield. - Narrow db to provide
firstNameandlastNameby removing thenamefield.
So now you can remove thenamefield from the db schema.
By following this strategy, you can ensure zero downtime deploys and schema
migrations.
Seeding Production
In this application we have Role-based Access Control implemented. We initialize
the database with admin and user roles with appropriate permissions.
This is done in the migration.sql file that's included in the template. If you
need to seed the production database, modifying migration files manually is the
recommended approach to ensure it's reproducible.
The trick is not all of us are really excited about writing raw SQL (especially
if what you need to seed is a lot of data), so here's an easy way to help out:
- Create a script very similar to our
prisma/seed.tsfile which creates all
the data you want to seed. - Run the script locally to generate the data you want to seed.
- Create a "dump" of the seed database using the
sqlite3command line tool.sh nonumbersqlite3 seed.db .dump > seed.sql - Copy the relevant bits from the
seed.sqlfile into yourmigration.sql
file. Theseed.sqlwill include create table/index lines etc. which should
already be in yourmigration.sql. You probably just wantINSERTcommands. - Deploy your app and verify that the data was seeded correctly.
If your app has already applied all migrations, then the changes to themigration.sql won't be applied (because prisma's already applied it). So then
you can run the following command to apply the migration:
fly ssh console -C "npx prisma migrate reset --skip-seed --force" --app [YOUR_APP_NAME]WARNING: This will reset your database and apply all migrations. Continue
reading if you want to avoid this.
If you have existing data in your production database and you'd like to seed it
with more data without performing a migration, then it's a bit more involved.
- Backup your production database.
- Create a new database file (locally) with the data you want to seed.
- Create a "dump" of the seed database using the
sqlite3command line tool.sh nonumbersqlite3 seed.db .dump > seed.sql - Copy the
seed.sqlfile to your production volume next to your database (viafly sftp) - SSH into your production server and run the following command:sh nonumber
sqlite3 data.db < seed.sql - Verify that your production database has been seeded correctly. If it hasn't,
then restore your backup (asap).
Backups
LiteFS Cloud Backups
LifeFS Cloud is a service offered by Fly.io for managing backup and restore
functionality.
This is the simplest method for backing up your database.
It offers the ability to restore your database to any point in time in the last
30 days, with 5 minute granularity.
Fly.io has some great documentation on how to set this up:
The following is a summary of the steps to set up LiteFS Cloud:
- Create a LiteFS Cloud cluster in your Fly.io dashboard:
https://fly.io/dashboard/personal/litefs- Take note of the auth token, you'll need it in the next step
- Set the
LITEFS_CLOUD_TOKENto the token from your dashboard:shfly secrets set LITEFS_CLOUD_TOKEN="LITEFS_CLOUD_TOKEN_HERE" --app [YOUR_APP_NAME] - You should now be able to restore backups from the LiteFS dashboard.
Manual DB Backups
Manual DB backups can be taken/restored using litefs commands:
litefs export: https://fly.io/docs/litefs/export/litefs import: https://fly.io/docs/litefs/import/
Make sure to keep the backup in a secure location. Your DB backup will contain
user information and password hashes!
You can manually create a backup for your database using the following steps:
- SSH into your fly instance:sh
fly ssh console --app [YOUR_APP_NAME] - Create a
backupsfolder:shmkdir /backups - Create a backup file using
litefs export, and exit the console (it is
recommended to name the exported file with the current date):shlifefs export -name sqlite.db /backups/backup-2023-10-10.db exit - Use sftp to download the backup file:sh
fly ssh sftp get /backups/backup-2023-10-10.db --app [YOUR_APP_NAME]
You can now store this backup file wherever you like, such as an S3 bucket
(again, make sure it's a secure location!).
See the fly docs for more info: https://fly.io/docs/litefs/backup/
Manual DB restoration
WARNING - THIS OVERWRITES YOUR DATABASE, YOU CAN LOSE DATA!! TAKE ANOTHER
BACKUP OF THE CURRENT DATABASE BEFORE DOING THIS!!
- Establish an sftp session with the fly instance and upload the backup file to
the server usingput:shfly ssh sftp shell --app [YOUR_APP_NAME] put backup-2023-10-10.db - Quit the sftp session with CTRL+C
- SSH into the fly instance:sh
fly ssh console --app [YOUR_APP_NAME] - Restore the database from the backup file using
litefs importshlitefs import -name sqlite.db /backup-2023-10-10.db - Exit the ssh sessionsh
exit
File: content/docs/deployment/index.mdx
title: Deployment
date: 2023-11-02
categories:
- deployment
- fly.io
meta:
keywords:- deployment
- fly.io
- docker
- github-actions
When you first create an Epic Stack repo, it should take you through a series of
questions to get your app setup and deployed. However, we'll document the steps
here in case things don't go well for you or you decide to do it manually later.
Here they are!
Deploying to Fly.io
Prior to your first deployment, you'll need to do a few things:
-
Note: Try
flyctlinstead offlyif the commands below won't work. Sign up and log in to Fly:
shfly auth signupNote: If you have more than one Fly account, ensure that you are signed
into the same account in the Fly CLI as you are in the browser. In your
terminal, runfly auth whoamiand ensure the email matches the Fly
account signed into the browser.Create two apps on Fly, one for staging and one for production:
shfly apps create [YOUR_APP_NAME] fly apps create [YOUR_APP_NAME]-stagingNote: Make sure this name matches the
appset in yourfly.toml
file. Otherwise, you will not be able to deploy.Initialize Git.
shgit init
Create a new GitHub Repository, and then add it as the
remote for your project. Do not push your app yet!shgit remote add origin <ORIGIN_URL>
- Add secrets:
Add a
FLY_API_TOKENto your GitHub repo. To do this, go to your user
settings on Fly and create a new
token, then add it to
your repo secrets
with the nameFLY_API_TOKEN.Add a
SESSION_SECRET,INTERNAL_COMMAND_TOKEN, andHONEYPOT_SECRETto
your fly app secrets, to do this you can run the following commands:shfly secrets set SESSION_SECRET=$(openssl rand -hex 32) INTERNAL_COMMAND_TOKEN=$(openssl rand -hex 32) HONEYPOT_SECRET=$(openssl rand -hex 32) --app [YOUR_APP_NAME] fly secrets set SESSION_SECRET=$(openssl rand -hex 32) INTERNAL_COMMAND_TOKEN=$(openssl rand -hex 32) HONEYPOT_SECRET=$(openssl rand -hex 32) --app [YOUR_APP_NAME]-stagingNote: If you don't have openssl installed, you can also use
1Password to generate a random
secret, just replace$(openssl rand -hex 32)with the generated secret.
Create production database:
Create a persistent volume for the sqlite database for both your staging and
production environments. Run the following (feel free to change the GB size
based on your needs and the region of your choice
(https://fly.io/docs/reference/regions/). If you do change the region, make
sure you change theprimary_regionin fly.toml as well):shfly volumes create data --region sjc --size 1 --app [YOUR_APP_NAME] fly volumes create data --region sjc --size 1 --app [YOUR_APP_NAME]-stagingAttach Consul:
Consul is a fly-managed service that manages your primary instance for data
replication
(learn more about configuring consul).shfly consul attach --app [YOUR_APP_NAME] fly consul attach --app [YOUR_APP_NAME]-staging
Commit!
The Epic Stack comes with a GitHub Action that handles automatically
deploying your app to production and staging environments.Now that everything is set up you can commit and push your changes to your
repo. Every commit to yourmainbranch will trigger a deployment to your
production environment, and every commit to yourdevbranch will trigger a
deployment to your staging environment.
Optional: Email service setup
Find instructions for this optional step in the email docs.
Optional: Error monitoring setup
Find instructions for this optional step in
the error tracking docs.
Optional: Connecting to your production database
Find instructions for this optional step in the database docs.
Optional: Seeding Production
Find instructions for this optional step in the database docs.
Deploying locally
If you'd like to deploy locally you definitely can. You need to (temporarily)
move the Dockerfile and the .dockerignore to the root of the project first.
Then you can run the deploy command:
mv ./other/Dockerfile Dockerfile
mv ./other/.dockerignore .dockerignore
fly deployOnce it's done, move the files back:
mv Dockerfile ./other/Dockerfile
mv .dockerignore ./other/.dockerignoreYou can keep the Dockerfile and .dockerignore in the root if you prefer,
just make sure to remove the move step from the .github/workflows/deploy.yml.