## 1. Project Overview & Quickstart (gruntwork-io/LICENSE) # LICENSE Open-source repository gruntwork-io/LICENSE ### Repository Details - **Repository:** [gruntwork-io/LICENSE](https://github.com/gruntwork-io/LICENSE) - **Primary Language:** Code *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (gruntwork-io/docs) ## File: README.md # Docs Site This website is built using [Docusaurus 3](https://docusaurus.io/), a modern static website generator. ## Overview ### Docs Sourcer The vast majority of content in the `docs` folder is maintained by developers and not automation. There are, however, some types of docs that are maintained by a tool called `docs-sourcer`. You can identify those files by the `##DOCS-SOURCER-START` tag at the end of the file. In general this is content that is pulled from external sources, such as change logs, release notes and GitHub discussions. Docs Sourcer is generally run in CI or via webhooks and doesn't have to be run locally. If, however, you do need to run it locally you will need access to various secrets. These secrets live in 1password under a secure note called "docs sourcer .env file". ## Installing dependencies ```sh yarn ``` ## Local development ```sh yarn start ``` This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server. ### Running a production build locally ```sh yarn build && yarn serve ``` This will create and serve a production build. This can be used to verify the expected behaviors in a production environment and view the Google analytics. # Linking to other pages We only allow absolute links to other internal docs pages, e.g. `(modules)[/2.0/docs/library/concepts/modules]`. This is to ensure that links work when the site is deployed to s3 which always appends a trailing slash. # Customizable Values It's often useful to present the user with the ability to edit a variable inline to documentation. For example, you're about to present a block of code that references an S3 bucket name. Rather than put in a fixed placeholder like which the user has to manually edit, you can use our CustomizableValue component. This component will render as a clickable text field which, when clicked, switches to an input. Any inputted characters will be auto-propegated to all other instances of the component that share the same ID. You can use this component both as a React component and inline to MDX code blocks. We have swizzled the CodeBlock components to make this possible. Example usage: As a component: ```js ``` Using the \$\$ID\$\$ syntax inside a code block ```md This is inside a code block, $$BUCKET_NAME$$ ``` # Spell Check The codebase (in `docs/2.0/`) is spell checked via CI using `cspell`. We have a custom dictionary in `./custom-dictionary.txt`. You can test spelling locally with `yarn spellcheck` (your IDE likely also automatically does this). Feel free to add new terms to the dictionary that are bespoke to Gruntwork. You can [disable spell check](https://cspell.org/configuration/document-settings/) for a block using the following pattern: ``` MY POURLEY SPELLED WURDS Content that is once again spell checked. ``` # Automated Pull Request Flows There are two main categories of the Automated PRs that get opened in the Docs repostory: * PRs for Pull requests in module repositories - have "Automated Preview" title * PRs for Releases in module repositories - have "Automated Update" title ## Automated Preview PRs The "Automated Preview" PRs are meant for the developers of different modules to be responsible for their docs while developing so they can ensure correctness. However, they are not intended to be merged since Docs are tied to versions of the modules When the Pull request in a module repository is closed or merged, the associated "Automated Preview" docs PR is automatically closed. ## Automated Updated PRs When a module repository gets a new release, the "Automated Update" pull request is opened on the docs site and that is what is intended to be merged and eventually released to the Docs site. ## Automation Setup The automation of the PRs is accomplished via docs-sourcer ([docs](https://github.com/gruntwork-io-team/dogfood-infrastructure-live/blob/main/OPERATOR.md#docs-sourcer) - [repo](https://github.com/gruntwork-io/docs-sourcer)), ultimately its a lambda function receiving GitHub webhooks. --- ## File: docs/2.0/way/intro/how-to-use-this.md --- title: How To Use This sidebar_position: 3 --- To get the most out of the Gruntwork Way, start by making sure you understand the big picture. ## The Gruntwork Way Big Picture First, a **developer platform** is the product your platform team builds to balance three essential concerns: enabling developers to move fast ([velocity](/2.0/way/why/velocity)), meeting business needs like security, cost, and compliance ([governance](/2.0/way/why/governance)), and avoiding unsustainable technical debt ([maintainability](/2.0/way/why/maintainability)). To build a successful platform, you need to combine three types of building blocks: [patterns](/2.0/way/platform/patterns/overview) (pre-built solutions to common infrastructure problems), [platform components](/2.0/way/platform/components/overview) (the machinery for deploying and managing infrastructure), and [interfaces](/2.0/way/platform/interfaces/overview) (how developers interact with your platform). Finally, everything in this framework is based on the [guiding principles](/2.0/way/principles/overview) we've discovered through years of building and refining developer platforms. ## Using the Gruntwork Way Once you understand these foundational concepts, there are two primary ways to use the Gruntwork Way. ### Option 1: Read it comprehensively You can work through this framework from start to finish to build a complete mental model of developer platforms. This approach works well if you're just getting started or want to strengthen your overall understanding. ### Option 2: Use it as a reference Alternatively, you can jump directly to the sections most relevant to your current challenges. This approach works well once you understand the fundamentals (which you just read above!) and need guidance on a specific aspect of your platform. ## Next Alright, let's dive into the framework that helps you build platforms developers actually want to use! --- ## File: docs/2.0/way/intro/share-your-feedback.md --- title: Share your feedback sidebar_position: 4 --- - This framework is useful because it's opinionated. - Our opinions are based on nearly a decade of experience across hundreds of customers and thousand of customer conversations - But maybe your mileage varies, or maybe we got it wrong. If so, tell us! - To give feedback, go to https://github.com/gruntwork-io/docs and create a GitHub Issue or Pull Request. Alternatively click on the "Edit this page" link at the bottom of each page. ### Next Alright, let's dive into the framework that helps you build platforms developers actually want to use! --- ## File: docs/2.0/way/intro/welcome.md --- title: Welcome! sidebar_position: 1 --- # Welcome! The **Gruntwork Way** is Gruntwork's official framework for how to build a world-class developer platform. ## What's a developer platform? Nearly every company on the planet needs software, and when companies write their own software with more than a few developers, the same common needs arise: 1. Enable developers to move fast ([velocity](/2.0/way/why/velocity)) 2. ...while meeting the other needs of the business like security, cost, and compliance ([governance](/2.0/way/why/governance)) 3. ...without accumulating an unsustainable level of debt ([maintainability](/2.0/way/why/maintainability)) A **developer platform** is the product your platform team builds to balance these three concerns. It consists of three essential building blocks: - [Patterns](/2.0/way/platform/patterns/overview) - pre-built solutions to common infrastructure problems - [Platform components](/2.0/way/platform/components/overview) - the machinery for deploying and managing infrastructure - [Interfaces](/2.0/way/platform/interfaces/overview) - how developers interact with your platform. Together, these enable developers to move fast while meeting the needs of the business, and to do so in a way that incurs manageable levels of debt. ## Why read this framework Making sense of developer platforms can be confusing. We've found bits and pieces of insights scattered across blog posts, vendor solutions, and industry talks, but none of these presents a holistic way to think about developer platforms. This framework will help you build comprehensive clarity. Ultimately, that clarity will empower you to build a successful developer platform. ## Why trust Gruntwork We've spent years building expertise on how to deliver software effectively by serving as: * **Authors** - We've written leading books and guides, including [Terraform Up & Running](https://www.terraformupandrunning.com/), [Fundamentals of DevOps and Software Delivery](https://www.fundamentals-of-devops.com/?ref=blog-post-series), ,[The Startup CTO's Handbook](https://github.com/ZachGoldberg/Startup-CTO-Handbook), and [A Comprehensive Guide to Scaling Apps on AWS](https://joshpadnick.com/posts/a-comprehensive-guide-to-scaling-apps-on-aws-part-1/). * **Makers** - We've created leading open source DevOps tooling like [OpenTofu](https://opentofu.org/) (co-founder), [Terragrunt](https://terragrunt.gruntwork.io/), [Terratest](https://terratest.gruntwork.io/), [Runbooks](https://runbooks.gruntwork.io), [Boilerplate](https://github.com/gruntwork-io/boilerplate), and [Cloud Nuke](https://github.com/gruntwork-io/cloud-nuke). * **Experts** - As a vendor, we've equipped over 500 companies and thousands of engineers with AWS, DevOps, and Platform Engineering best practices. ## Next Now let's learn more about who we wrote the Gruntwork Way for. --- ## File: docs/2.0/way/intro/who-this-is-for.md --- title: Who This Is For sidebar_position: 2 --- We wrote The Gruntwork Way for anyone trying to balance velocity, governance, and maintainability in their quest to better deliver software. ## Roles In practice, we expect users in these roles will find this useful: - **Platform engineers** building or improving their developer platforms - **DevOps engineers** looking for best practices and patterns - **Engineering leaders** evaluating their developer platform and/or cloud strategy ## Maturity levels As you build your developer platform, you will go through a series of maturity stages. Actually, to be more precise, you will build maturity separately in your [patterns](/2.0/way/platform/patterns/overview), [platform components](/2.0/way/platform/components/overview), and [interfaces](/2.0/way/platform/interfaces/overview). Your collective maturity level for the "developer platform" is really just the collection of maturity levels for its component parts. In general, the component parts track these maturity levels: - **Ad hoc** - There is no standard, so your team does things manually with minimal automation. - **Building** - You've built something, but it's limited. - **Established** - This component does add value, but with lots of room for improvement. - **Mature** - This component performs its functions comprehensively and effectively. - **Self-improving** - This component has a systematic feedback loop that leads to ongoing improvements. The Gruntwork Way is especially useful for organizations with many building blocks in the **ad hoc,** **building,** or **established** stages. ## Next Let's close out the introduction by giving you some pointers on how to use this framework! --- ## File: docs/2.0/way/platform/components/deploy/catalog.md --- sidebar_position: 1 title: Catalog --- # Catalog **Where you store your organization's patterns** ## What is the catalog component? The catalog is the system of record for all your organization's [infrastructure patterns](/2.0/way/platform/patterns/overview). When a member of your organization goes to ask "how do we approach infrastructure problem X around here?", you need a system of record to point them to. Or when you update a pattern, your pattern authors need to know which pattern version is canonical and where to place their updates. That system of record is the **catalog.** ## Good catalogs ### Minimum requirements To qualify as a catalog, you need only meet one requirement. The catalog is: - **A system of record.** The latest available information about your patterns lives in the catalog. ### Effective catalogs To be an _effective_ catalog, we need a few more requirements. Effective catalogs are: - **Browsable.** Users can browse the available patterns using their preferred [interface](/2.0/way/platform/interfaces/overview). - **Searchable.** Users can search for exactly the pattern they want. - **Detailed.** Users can browse detailed information about each pattern. - **Extensible.** As the catalog maintainer, you can add support for new patterns. - **Reports pattern usage.** Users can see how often patterns are consumed. - **Educational.** Users can learn background information necessary to be a savvy consumer of a pattern. - **Connect authors and consumers.** Users can connect easily with pattern authors, creating a virtuous feedback loop. - **Connect to self-service.** When user identifies a pattern they want to instantiate, the catalog directs them right to the applicable [runbook](/2.0/way/platform/components/deploy/runbooks). ## Catalog options You have several options when implementing a catalog. In order from least favorable to most favorable: ### No system of record You simply do not have a system of record. If a user wants to know whether a pattern exists, they verbally ask, but there is no written resource to consult. ### Documentation site You have a documentation site that users visit to learn about what patterns are available, but the patterns themselves are scattered across the organization. ### Git repository All your patterns are collected in one or more git repositories. ### Catalog UI You have a Terminal UI and/or Web UI that enables users to browse the catalog. The underlying system of record may still be a git repository. :::info In our open source tool, Terragrunt, we expose a [terragrunt catalog](https://terragrunt.gruntwork.io/docs/features/catalog/) command that presents a Terminal UI to select the desired module from a specified set of git repos. ::: --- ## File: docs/2.0/way/platform/components/deploy/pipelines.md --- sidebar_position: 3 title: Pipelines --- # Pipelines **How you deploy infrastructure changes** Pipelines are the automated workflows that take infrastructure code and deploy it to your cloud environment. They include the checks, tests, and policies that ensure infrastructure meets your standards before deployment. ## Why it matters Centralized pipelines enforce governance, provide audit trails, and ensure consistent deployment processes across all infrastructure changes. ## What it enables - Automated deployment of infrastructure code - Policy enforcement before deployment - Testing and validation in pipeline - Rollback capabilities when issues arise - Audit trail of all infrastructure changes --- ## File: docs/2.0/way/platform/components/deploy/runbooks.md --- sidebar_position: 2 title: Runbooks --- # Runbooks **How developers configure an infrastructure module instance** ## What is the Runbooks component? ### The challenge When a developer needs to deploy new infrastructure, per the principle of [define all live infrastructure as pattern instances](/2.0/way/principles/technical-foundations/define-all-live-infrastructure-as-pattern-instances), the developer will need a way to create a new [infrastructure module instance](/2.0/way/platform/patterns/types#infrastructure-module-instances). But creating a module instance can be complex! The developer may need to do any or all of the following: - Confirm that they meet all pre-requisites to use a given infrastructure module - Perform other actions needed to meet the pre-requisites (e.g. obtain a new internal API key) - Write new code that calls the desired [infrastructure module](/2.0/way/platform/patterns/types#infrastructure-modules) (e.g. a Terragrunt unit) - Compose many infrastructure modules together to achieve a more complex use case - "Apply" the code they generated - Validate that a pattern instance was deployed successfully The breadth and depth of these requirements impose a heavy burden on the humble developer who wishes only to deploy a pattern so they can get back to their real job of building their app. ### The solution The solution is to make meeting the requirements above easy for the developer by encapsulating all required expertise into an intuitive, streamlined format. One format for accomplishing this is the **runbook.** We call this format a runbook because users will go through a sequence of individual steps that, once finished, achieve a configured instance of the desired infrastructure module. A first-class Runbook should be able to do all the following: - Run pre-flight checks to ensure the user is "ready" to instantiate the module - Perform "side effects" like requesting an API key, or installing new local tools - Enable the user to configure the module with custom parameter values - Generate the relevant code - Validate that the infrastructure module instance deployed correctly by running post-flight checks Runbooks pack a wide collection of functionality into a single format, so we explore some real-world options below. ### Developer self-service A common industry term is "developer self-service." We can now assert a definition of developer self-service using the terms we've introduced so far. Specifically, developer self-service is the combination of a **[catalog](/2.0/way/platform/components/deploy/catalog)** of infrastructure modules, plus the ability to use **runbooks,** plus a way to deploy the generated code using a **[pipeline](/2.0/way/platform/components/deploy/pipelines)**. ## Effective runbooks ### Minimum requirements To qualify as a runbook, you need to at least meet the following requirements. A runbook must: - **Expose parameter values.** The user can see a list of the parameter values available. - **Configure parameter values.** The user can configure the parameter values. - **Generate code.** The user can instantiate an [infrastructure module](/2.0/way/platform/patterns/types#infrastructure-modules), either by generating code or other means. ### Effective runbooks To be an _effective_ runbook, we need a few more requirements. Effective runbooks are/do: #### General - **Easily authored.** It must be easy for runbook authors to capture their expertise. - **Easily launched.** It must be easy for the runbook consumer to begin using the runbook. - **Documented.** The runbook must teach the user about this particular area of subject matter expertise, intermixing documentation and interactivity. - **Testable.** A runbook must be programmatically testable so that you can continually validate that it functions as expected. - **Capture feedback.** The runbook must expose a way for runbook consumers to share feedback about their experience and request improvements. #### Code execution - **Run arbitrary code.** To check pre-requisites, perform "side effects" (like requesting an API key), and validate post-apply correctness, users need to run arbitrary code written in standard programming languages. - **Real-time.** When users execute arbitrary code to run pre-flight checks, post-flight checks, or achieve side effects, they should see what is happening in real-time. - **Secure.** The ability to run arbitrary code needs to be paired with a strong security posture that ensures only trusted code is executed. #### Configuration - **Read external data.** An infrastructure module instance often needs data from other module instances or other sources, so the runbook must be able to "import" these values somehow. - **Adaptive.** As users fill in configuration values, some configuration options are disabled and new options are revealed. ## Runbook options You have several options when implementing a runbooks. In order from least favorable to most favorable: ### Static documentation You can describe in writing how a user would create the code necessary to instantiate an infrastructure module, how they would identify the parameter values, and how to configure those values. You _can_ do all of this, but as you'll see shortly, there are better options. ### Code templates You can define a template that generates the code necessary to instantiate an infrastructure module, and then give some mechanism for users to specify parameters. For example, the command [terragrunt scaffold](https://terragrunt.gruntwork.io/docs/features/scaffold/) expects a parameter for an OpenTofu/Terraform module URL and will then generate a template that looks like this: ```hcl # This is a Terragrunt unit generated by Gruntwork Boilerplate. terraform { source = "git::https://github.com/gruntwork-io/terragrunt-infrastructure-modules-example.git//modules/mysql?ref=v0.8.1" } inputs = { # -------------------------------------------------------------------------------------------------------------------- # Required input variables # -------------------------------------------------------------------------------------------------------------------- # Type: string # Description: The AWS region to deploy to (e.g. us-east-1) aws_region = "" # TODO: fill in value # Type: string # Description: The name of the DB name = "" # TODO: fill in value # Type: string # Description: The instance class of the DB (e.g. db.t2.micro) instance_class = "" # TODO: fill in value # (... full list of inputs omitted for brevity ...) } ``` This improves on using static documentation because we offer the user a pre-set opinion on how to generate inputs. Rather than telling the user how to write this code, we can now tell them how to generate a template that writes the code. Better yet, either the process of generating the code includes a step to configure variables, or the template itself can leave placeholders for users to enter variables (as it does above). :::info Gruntwork maintains an open-source template generator built specially for DevOps and platform engineering called **Gruntwork Boilerplate.** Learn more at [https://github.com/gruntwork-io/boilerplate](https://github.com/gruntwork-io/boilerplate). ::: ### Infrastructure module UI You can build (or buy) a solution that renders a UI for a given infrastructure module. The UI lists the available parameter values, and users fill in those values directly through the UI. Ideally, the UI can allow users to fetch value from external data sources such as infrastructure-as-code state, a secrets manager, or a third-party tool. The primary limitation of the infrastructure module UI is that, by definition, it is scoped to a single infrastructure module. ### Infrastructure template UI An infrastructure template UI renders a UI not around an infrastructure module, but around a [code template](#code-templates). This makes it more powerful because a code template could generate a single infrastructure module instance, or a collection of infrastructure module instances, or really anything. Beyond that, the functionality is the same. The UI lists the available parameters, and users fill in those values directly through the UI. The template then renders based on those input parameters. ### First-class runbooks It's not a common solution, but the best option is to combine an [infrastructure template UI](#infrastructure-template-ui) with the ability to execute arbitrary code, all in a format that's easy to document, test, and author. :::info Gruntwork maintains an open source tool called [Gruntwork Runbooks](https://runbooks.gruntwork.io/) that is based on all the insights and philosophy that we capture here. Check it out! ::: --- ## File: docs/2.0/way/platform/components/maintain/drift-detector.md --- sidebar_position: 1 title: Drift Detector --- # Drift Detector **Detect drift between code and cloud** Drift detection continuously monitors your infrastructure to identify when deployed resources no longer match their Infrastructure as Code definitions. This helps you maintain the integrity of your infrastructure as code practice. ## Why it matters Drift happens inevitably as teams make manual changes, resources get modified outside IaC, or automation scripts run. Undetected drift means your IaC no longer represents reality, breaking reproducibility and creating hidden risks. ## What it enables - Automated detection of infrastructure drift - Alerts when drift occurs - Reports showing what drifted and how - Ability to reconcile drift automatically or manually --- ## File: docs/2.0/way/platform/components/maintain/iac-updater.md --- sidebar_position: 2 title: IaC Updater --- # IaC Updater **Update out-of-date IaC or tooling versions** The IaC updater identifies infrastructure using outdated patterns or tool versions and helps you modernize it. This prevents technical debt from accumulating as your patterns and tools evolve. ## Why it matters Infrastructure ages quickly. What was modern six months ago may now be outdated. Without systematic updates, your infrastructure estate becomes increasingly difficult to maintain. ## What it enables - Identification of outdated infrastructure - Automated or guided update processes - Migration paths from old to new patterns - Tracking update progress across your estate