## 1. Project Overview & Quickstart (turbot/steampipe) ## File: README.md [](https://steampipe.io) [](https://hub.steampipe.io/)   [](https://turbot.com/community/join?utm_id=gspreadme&utm_source=github&utm_medium=repo&utm_campaign=github&utm_content=readme)   [](https://turbot.com?utm_id=gspreadme&utm_source=github&utm_medium=repo&utm_campaign=github&utm_content=readme) ## select * from cloud; [Steampipe](https://steampipe.io) is **the zero-ETL way** to query APIs and services. Use it to expose data sources to SQL. **SQL**. It's been the data access standard for decades. **Live data**. Query APIs in real-time. **Speed**. Query APIs faster than you ever thought possible. **Concurrency**. Query many data sources in parallel. **Single binary**. Use it locally, deploy it in CI/CD pipelines. ## Demo time! ## Documentation See the [documentation](https://steampipe.io/docs) for: - [Running queries](https://steampipe.io/docs/query/overview) - [Managing Steampipe](https://steampipe.io/docs/managing/overview) - [CLI commands](https://steampipe.io/docs/reference/cli/overview) - [Integrations](https://steampipe.io/docs/integrations/overview) - [Developing plugins](https://steampipe.io/docs/develop/overview) ## Install Steampipe Install Steampipe from the [downloads](https://steampipe.io/downloads) page: ```sh # MacOS brew install turbot/tap/steampipe ``` ``` # Linux or Windows (WSL2) sudo /bin/sh -c "$(curl -fsSL https://steampipe.io/install/steampipe.sh)" ``` Install a plugin for your favorite service (e.g. [AWS](https://hub.steampipe.io/plugins/turbot/aws), [Azure](https://hub.steampipe.io/plugins/turbot/azure), [GCP](https://hub.steampipe.io/plugins/turbot/gcp), [GitHub](https://hub.steampipe.io/plugins/turbot/github), [Kubernetes](https://hub.steampipe.io/plugins/turbot/kubernetes), [Hacker News](https://hub.steampipe.io/plugins/turbot/hackernews), etc): ```sh steampipe plugin install hackernews ``` Query! ```sh steampipe query > select * from hackernews_new limit 10 ``` ## Steampipe plugins The Steampipe community has grown a suite of [plugins](https://hub.steampipe.io/plugins) that map APIs to database tables. Plugins are available for [AWS](https://hub.steampipe.io/plugins/turbot/aws), [Azure](https://hub.steampipe.io/plugins/turbot/azure), [GCP](https://hub.steampipe.io/plugins/turbot/gcp), [Kubernetes](https://hub.steampipe.io/plugins/turbot/kubernetes), [GitHub](https://hub.steampipe.io/plugins/turbot/github), [Microsoft 365](https://hub.steampipe.io/plugins/turbot/microsoft365), [Salesforce](https://hub.steampipe.io/plugins/turbot/salesforce), and many more. There are more than 2000 tables in all, each clearly documented with copy/paste/run examples. ## Steampipe distributions Plugins are available in these distributions. **Steampipe CLI**. Run [queries](https://steampipe.io/docs/query/overview) that translate APIs to tables in the Postgres instance that's bundled with Steampipe. **Steampipe Postgres FDWs**. Use [native Postgres Foreign Data Wrappers](https://steampipe.io/docs/steampipe_postgres/overview) to translate APIs to foreign tables. **Steampipe SQLite extensions**. Use [SQLite extensions](https://steampipe.io/docs/steampipe_sqlite/overview) to translate APIS to SQLite virtual tables. **Steampipe export tools**. Use [standalone binaries](https://steampipe.io/docs/steampipe_export/overview) that export data from APIs, no database required. **Turbot Pipes**. Use [Turbot Pipes](https://turbot.com/pipes) to run Steampipe in the cloud. ## Developing If you want to help develop the core Steampipe binary, these are the steps to build it. Clone ```sh git clone git@github.com:turbot/steampipe ``` Build ``` cd steampipe make ``` The Steampipe binary lands in `/usr/local/bin/steampipe` directory unless you specify an alternate `OUTPUT_DIR`. Check the version ``` $ steampipe --version steampipe version 0.22.0 ``` Install a plugin ``` $ steampipe plugin install steampipe ``` Run your first query Try it! ``` steampipe query > .inspect steampipe +-----------------------------------+-----------------------------------+ | TABLE | DESCRIPTION | +-----------------------------------+-----------------------------------+ | steampipe_registry_plugin | Steampipe Registry Plugins | | steampipe_registry_plugin_version | Steampipe Registry Plugin Version | +-----------------------------------+-----------------------------------+ > select * from steampipe_registry_plugin; ``` If you're interested in developing [Steampipe plugins](https://hub.steampipe.io), see our [documentation for plugin developers](https://steampipe.io/docs/develop/overview). ## Turbot Pipes Bring your team to [Turbot Pipes](https://turbot.com/pipes) to use Steampipe together in the cloud. In a Pipes workspace you can use Steampipe for data access, [Powerpipe](https://github.com/turbot/powerpipe) to visualize query results, and [Flowpipe](https://github.com/turbot/flowpipe) to automate workflow. ## Open source and contributing This repository is published under the [AGPL 3.0](https://www.gnu.org/licenses/agpl-3.0.html) license. Please see our [code of conduct](https://github.com/turbot/.github/blob/main/CODE_OF_CONDUCT.md). Contributors must sign our [Contributor License Agreement](https://turbot.com/open-source#cla) as part of their first pull request. We look forward to collaborating with you! [Steampipe](https://steampipe.io) is a product produced from this open source software, exclusively by [Turbot HQ, Inc](https://turbot.com). It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our [Open Source FAQ](https://turbot.com/open-source). ## Get involved **[Join #steampipe on Slack →](https://turbot.com/community/join)** --- ## File: .ai/docs/bug-fix-prs.md # Bug Fix PR Guide ## Two-Commit Pattern Every bug fix PR must have **exactly 2 commits**: 1. **Commit 1**: Demonstrate the bug (test fails) 2. **Commit 2**: Fix the bug (test passes) This pattern provides: - Clear demonstration that the bug exists - Proof that the fix resolves the issue - Easy code review (reviewers can see the test fail, then pass) - Test-driven development (TDD) workflow ## Commit 1: Unskip/Add Test ### Purpose Demonstrate that the bug exists by having a failing test. ### Changes - If test exists in test suite: Remove `t.Skip()` line - If test doesn't exist: Add the test - **NO OTHER CHANGES** ### Commit Message Format ``` Unskip test demonstrating bug #: ``` or ``` Add test for #: ``` ### Examples ``` Unskip test demonstrating bug #4767: GetDbClient error handling ``` ``` Add test for #4717: Target.Export() should handle nil exporter gracefully ``` ### Verification ```bash # Test should FAIL go test -v -run TestName ./pkg/path # Exit code: 1 ``` ## Commit 2: Implement Fix ### Purpose Fix the bug with minimal changes. ### Changes - Implement the fix in production code - **NO changes to test code** - Keep changes minimal and focused ### Commit Message Format ``` Fix #: ``` ### Examples ``` Fix #4767: GetDbClient returns (nil, error) on failure ``` ``` Fix #4717: Add nil check to Target.Export() ``` ### Verification ```bash # Test should PASS go test -v -run TestName ./pkg/path # Exit code: 0 ``` ## Creating the Two Commits ### Method 1: Interactive Rebase (Recommended) If you have more commits, squash them: ```bash # View commit history git log --oneline -5 # Interactive rebase to squash git rebase -i HEAD~3 # Mark commits: # pick Unskip test... # squash Additional test changes # pick Fix bug # squash Address review comments ``` ### Method 2: Cherry-Pick If rebasing from another branch: ```bash # In your fix branch based on develop git cherry-pick git cherry-pick ``` ### Method 3: Build Commits Correctly ```bash # Start from develop git checkout -b fix/1234-description develop # Commit 1: Unskip test # Edit test file to remove t.Skip() git add pkg/path/file_test.go git commit -m "Unskip test demonstrating bug #1234: Description" # Verify it fails go test -v -run TestName ./pkg/path # Commit 2: Fix bug # Edit production code git add pkg/path/file.go git commit -m "Fix #1234: Description of fix" # Verify it passes go test -v -run TestName ./pkg/path ``` ## Pushing to GitHub: Two-Phase Push **IMPORTANT**: Push commits separately to trigger CI runs for each commit. This provides clear visual evidence in the PR that the test fails before the fix and passes after. ### Phase 1: Push Test Commit (Should Fail CI) ```bash # Create and switch to your branch git checkout -b fix/1234-description develop # Make commit 1 (unskip test) git add pkg/path/file_test.go git commit -m "Unskip test demonstrating bug #1234: Description" # Verify test fails locally go test -v -run TestName ./pkg/path # Push ONLY the first commit git push -u origin fix/1234-description ``` At this point: - GitHub Actions will run tests - CI should **FAIL** on the test you unskipped - This proves the test catches the bug ### Phase 2: Push Fix Commit (Should Pass CI) ```bash # Make commit 2 (fix bug) git add pkg/path/file.go git commit -m "Fix #1234: Description of fix" # Verify test passes locally go test -v -run TestName ./pkg/path # Push the second commit git push ``` At this point: - GitHub Actions will run tests again - CI should **PASS** with the fix - This proves the fix works ### Creating the PR Create the PR after the first push (before the fix): ```bash # After phase 1 push gh pr create --base develop \ --title "Brief description closes #1234" \ --body "## Summary [Description] ## Changes - Commit 1: Unskipped test demonstrating the bug - Commit 2: Implemented fix (coming in next push) ## Test Results Will be visible in CI runs: - First CI run should FAIL (demonstrating bug) - Second CI run should PASS (proving fix works) " ``` Or create it after both commits are pushed - either way works. ### Why This Matters for Reviewers This two-phase push gives reviewers: 1. **Visual proof** the test fails without the fix (failed CI run) 2. **Visual proof** the test passes with the fix (passed CI run) 3. **No manual verification needed** - just look at the CI history in the PR 4. **Clear diff** between what fails and what fixes it ### Example PR Timeline ``` ✅ PR opened ❌ CI run #1: Test failure (commit 1) "FAIL: TestName - expected nil, got non-nil client" ⏱️ Commit 2 pushed ✅ CI run #2: All tests pass (commit 2) "PASS: TestName" ``` Reviewers can click through the CI runs to see the exact failure and success. ## PR Structure ### Branch Naming ``` fix/-brief-kebab-case-description ``` Examples: - `fix/4767-getdbclient-error-handling` - `fix/4743-status-spinner-visible-race` - `fix/4717-nil-exporter-check` ### PR Title ``` Brief description closes # ``` Examples: - `GetDbClient error handling closes #4767` - `Race condition on StatusSpinner.visible field closes #4743` ### PR Description ```markdown ## Summary [Brief description of the bug and fix] ## Changes - Commit 1: Unskipped test demonstrating the bug - Commit 2: Implemented fix by [description] ## Test Results - Before fix: [Describe failure - panic, wrong result, etc.] - After fix: Test passes ## Verification \`\`\`bash # Commit 1 (test only) go test -v -run TestName ./pkg/path # FAIL: [error message] # Commit 2 (with fix) go test -v -run TestName ./pkg/path # PASS \`\`\` ``` ### Labels Add appropriate labels: - `bug` - Severity: `critical`, `high-priority` (if available) - Type: `security`, `race-condition`, `nil-pointer`, etc. ## What NOT to Include ### ❌ Don't Add to Commits - Unrelated formatting changes - Refactoring not directly related to the bug - go.mod changes (unless required by new imports) - Documentation updates (separate PR) - Multiple bug fixes in one PR ### ❌ Don't Combine Commits - Keep test and fix as separate commits - Don't squash them together - Don't add "fix review comments" commits (amend instead) ## Handling Review Feedback ### If Test Needs Changes ```bash # Amend commit 1 git checkout HEAD~1 # Make test changes git add file_test.go git commit --amend git rebase --continue ``` ### If Fix Needs Changes ```bash # Amend commit 2 # Make fix changes git add file.go git commit --amend ``` ### Force Push After Amendments ```bash git push --force-with-lease ``` ## Multiple Related Bugs If fixing multiple related bugs: - Create separate issues for each - Create separate PRs for each - Don't combine into one PR - Each PR: 2 commits ## Test Suite PRs (Different Pattern) Test suite PRs follow a different pattern: - **Single commit** with all tests - Branch: `feature/tests-for-` - Base: `develop` - Include bug-demonstrating tests (marked as skipped) See [templates/test-pr-template.md](../templates/test-pr-template.md) ## Verifying Commit Structure Before pushing: ```bash # Check commit count git log --oneline origin/develop..HEAD # Should show exactly 2 commits # Check first commit (test only) git show HEAD~1 --stat # Should only modify test file(s) # Check second commit (fix only) git show HEAD --stat # Should only modify production code file(s) # Verify test behavior git checkout HEAD~1 && go test -v -run TestName ./pkg/path # Should FAIL git checkout HEAD && go test -v -run TestName ./pkg/path # Should PASS ``` ## Common Mistakes ### ❌ Mistake 1: Combined Commit ``` Fix #1234: Add test and fix bug ``` **Problem**: Can't verify test catches the bug **Solution**: Split into 2 commits ### ❌ Mistake 2: Modified Test in Fix Commit ``` Commit 1: Add test Commit 2: Fix bug and adjust test ``` **Problem**: Test changes hide whether original test would pass **Solution**: Only modify test in commit 1 ### ❌ Mistake 3: Multiple Bugs in One PR ``` Fix #1234 and #1235: Multiple fixes ``` **Problem**: Hard to review, test, and merge independently **Solution**: Create separate PRs ### ❌ Mistake 4: Extra Commits ``` Commit 1: Add test Commit 2: Fix bug Commit 3: Address review Commit 4: Fix typo ``` **Problem**: Cluttered history **Solution**: Squash into 2 commits ## Examples Real examples from our codebase: - PR #4769: [Fix #4750: Nil pointer panic in RegisterExporters](https://github.com/turbot/steampipe/pull/4769) - PR #4773: [Fix #4748: SQL injection vulnerability](https://github.com/turbot/steampipe/pull/4773) ## Next Steps - [GitHub Issues](bug-workflow.md) - Creating bug reports - [Parallel Coordination](parallel-coordination.md) - Working on multiple bugs in parallel - [Templates](../templates/) - PR templates --- ## File: .ai/docs/bug-workflow.md # GitHub Issue Guidelines Guidelines for creating bug reports and issues. ## Bug Issue Format **Title:** ``` BUG: Brief description of the problem ``` For security issues, use `[SECURITY]` prefix. **Labels:** Add `bug` label **Body Template:** ```markdown ## Description [Clear description of the bug] ## Severity **[HIGH/MEDIUM/LOW]** - [Impact statement] ## Reproduction 1. [Step 1] 2. [Step 2] 3. [Observed result] ## Expected Behavior [What should happen] ## Current Behavior [What actually happens] ## Test Reference See `TestName` in `path/file_test.go:line` (currently skipped) ## Suggested Fix [Optional: proposed solution] ## Related Code - `path/file.go:line` - [description] ``` ## Example ```markdown ## Description The `GetDbClient` function returns a non-nil client even when an error occurs during connection, causing nil pointer panics when callers attempt to call `Close()` on the returned client. ## Severity **HIGH** - Nil pointer panic crashes the application ## Reproduction 1. Call `GetDbClient()` with an invalid connection string 2. Function returns both an error AND a non-nil client 3. Caller attempts to defer `client.Close()` which panics ## Expected Behavior When an error occurs, `GetDbClient` should return `(nil, error)` following Go conventions. ## Current Behavior Returns `(non-nil-but-invalid-client, error)` leading to panics. ## Test Reference See `TestGetDbClient_WithConnectionString` in `pkg/initialisation/init_data_test.go:322` (currently skipped) ## Suggested Fix Ensure all error paths return `nil` for the client value. ## Related Code - `pkg/initialisation/init_data.go:45-60` - GetDbClient function ``` ## When You Find a Bug 1. **Create the GitHub issue** using the template above 2. **Skip the test** with reference to the issue: ```go t.Skip("Demonstrates bug #XXXX - description. Remove skip in bug fix PR.") ``` 3. **Continue your work** - don't stop to fix immediately ## Bug Fix Workflow See [bug-fix-prs.md](bug-fix-prs.md) for the bug fix PR workflow (2-commit pattern). ## Best Practices - Include specific reproduction steps - Reference exact code locations with line numbers - Explain the impact clearly - Link to the test that demonstrates the bug - For security issues: assess severity carefully and consider private disclosure --- ## File: .ai/docs/parallel-coordination.md # Parallel Agent Coordination Simple patterns for coordinating multiple AI agents working in parallel. ## Basic Pattern When working on multiple related tasks in parallel: 1. **Create a work directory** in `wip/`: ```bash mkdir -p .ai/wip/ ``` Example: `.ai/wip/bug-fixes-wave-1/` or `.ai/wip/test-snapshot-pkg/` 2. **Coordinator creates task files**: ```bash # In .ai/wip// task-1-fix-bug-4767.md task-2-fix-bug-4768.md task-3-fix-bug-4769.md plan.md # Overall coordination plan ``` 3. **Parallel agents read and execute**: ``` Agent 1: "See plan in .ai/wip/bug-fixes-wave-1/ and run task-1" Agent 2: "See plan in .ai/wip/bug-fixes-wave-1/ and run task-2" Agent 3: "See plan in .ai/wip/bug-fixes-wave-1/ and run task-3" ``` ## Task File Format Keep task files simple: ```markdown # Task: Fix bug #4767 ## Goal Fix GetDbClient error handling bug ## Steps 1. Create worktree: /tmp/fix-4767 2. Branch: fix/4767-getdbclient 3. Unskip test in pkg/initialisation/init_data_test.go 4. Verify test fails 5. Implement fix 6. Verify test passes 7. Push (two-phase) 8. Create PR with title: "GetDbClient error handling (closes #4767)" ## Context See issue #4767 for details Test is already written and skipped ``` ## Work Directory Structure Example for a bug fixing session: ``` .ai/wip/bug-fixes-wave-1/ ├── plan.md # Coordinator's overall plan ├── task-1-fix-4767.md # Task for agent 1 ├── task-2-fix-4768.md # Task for agent 2 ├── task-3-fix-4769.md # Task for agent 3 └── status.md # Optional: track completion ``` Example for test generation: ``` .ai/wip/test-snapshot-pkg/ ├── plan.md # What to test, approach ├── findings.md # Bugs found during testing └── test-checklist.md # Coverage checklist ``` ## Benefits - **Isolated**: Each focus area has its own directory - **Clean**: Old work directories can be deleted when done - **Reusable**: Pattern works for any parallel work - **Simple**: Just files and directories, no complex coordination ## Cleanup When work is complete: ```bash # Archive or delete the work directory rm -rf .ai/wip// ``` The `.ai/wip/` directory is gitignored, so these temporary files won't clutter the repo. ## Examples **Parallel bug fixes:** ``` Coordinator: Creates .ai/wip/bug-fixes-wave-1/ with 10 task files Agents 1-10: Each picks a task file and works independently ``` **Test generation with bug discovery:** ``` Coordinator: Creates .ai/wip/test-generation-phase-2/plan.md Agent: Writes tests, documents bugs in findings.md ``` **Feature development:** ``` Coordinator: Creates .ai/wip/feature-auth/ - task-1-backend.md - task-2-frontend.md - task-3-tests.md Agents: Work in parallel on each component ``` --- ## File: .ai/README.md # AI Development Guide for Steampipe This directory contains documentation, templates, and conventions for AI-assisted development on the Steampipe project. ## Guides - **[Bug Fix PRs](docs/bug-fix-prs.md)** - Two-commit pattern, branch naming, PR format for bug fixes - **[GitHub Issues](docs/bug-workflow.md)** - Reporting bugs and issues - **[Test Generation](docs/test-generation-guide.md)** - Writing effective tests - **[Parallel Coordination](docs/parallel-coordination.md)** - Working with multiple agents in parallel ## Directory Structure ``` .ai/ ├── docs/ # Permanent documentation and guides ├── templates/ # Issue and PR templates └── wip/ # Temporary workspace (gitignored) ``` ## Key Conventions - **Base branch**: `develop` for all work - **Bug fixes**: 2-commit pattern (demonstrate → fix) - **Small PRs**: One logical change per PR - **Issue linking**: PR title ends with `closes #XXXX` ## For AI Agents - Reference the relevant guide in `docs/` for your task - Use templates in `templates/` for PR descriptions - Use `wip//` for coordinated parallel work (gitignored) - Follow project conventions for branches, commits, and PRs **Parallel work pattern**: Create `.ai/wip//` with task files, then agents can work independently. See [parallel-coordination.md](docs/parallel-coordination.md). --- ## File: pkg/otel/README.md # OpenTelemetry Collector This collector is provided for local testing purposes. It uses `docker-compose` and by default runs against the `otel/opentelemetry-collector-contrib-dev:latest` image. To run the collector, switch to the `otel` folder and run: ```shell docker-compose up -d ``` The demo exposes the following backends: - Jaeger at http://0.0.0.0:16686 - Prometheus at http://0.0.0.0:9090 Notes: - It may take some time for the application metrics to appear on the Prometheus dashboard; To clean up any docker container from the demo run `docker-compose down` from the `examples/demo` folder. ## 2. Official Technical Reference & Guides (turbot/steampipe-docs) # Steampipe docs format & structure Docs are written in Markdown format and are located in the `docs` folder. The entry-point document will contain front-matter with `slug: /`. Each document requires the following frontmatter, adjust the values as per your requirement: ```yaml id: learn title: Learn Steampipe sidebar_label: Learn Steampipe ``` We support up to 2 levels of docs, e.g.: - `docs/foo` - `docs/foo/bar` For your docs to appear in the sidebar, you need to edit `docs/sidebar.json`. This is an array of sidebar entries, which are either stings matching the path of the required document, or a category to nest the docs down 1 level. Any images required by docs must be placed in `/images/docs/...` and must be referenced by the tag ``. # Guidelines for contribution Thank you for your interest in contributing to Steampipe documentation! We greatly value feedback and contributions from our community. Please read through this document before you submit any pull requests or issues. It will help us to collaborate more effectively. ## What to expect when you contribute When you submit a pull request, our team is notified and will respond as quickly as we can. We'll do our best to work with you to ensure that your pull request adheres to our style and standards. We look forward to receiving your pull requests for: * Inaccuracies in the content * Information gaps in the content that need more detail to be complete * Grammatical errors or typos * Suggested rewrites that improve clarity and reduce confusion ## How to contribute To contribute, send us a pull request. 1. [Fork the repository](https://help.github.com/articles/fork-a-repo/). 2. In your fork, make your change in a branch that's based on this repo's **main** branch. 3. Commit the change to your fork, using a clear and descriptive commit message. 4. [Create a pull request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) Before you send us a pull request, please be sure that: 1. You're working from the latest source on the **main** branch. 2. You check [existing open](https://github.com/turbot/steampipe-docs/pulls) pull requests to be sure that someone else hasn't already addressed the problem. 3. You [create an issue](https://github.com/turbot/steampipe-docs/issues/new) before working on a contribution that will take a significant amount of your time. For contributions that will take a significant amount of time, [open a new issue](https://github.com/turbot/steampipe-docs/issues/new) to pitch your idea before you get started. Explain the problem and describe the content you want to see added to the documentation. We don't want you to spend a lot of time on a contribution that might be outside the scope of the documentation or that's already in progress. ## Finding contributions to work on If you'd like to contribute, but don't have a project in mind, look at the [open issues](https://github.com/turbot/steampipe-docs/issues/news) in this repository for some ideas. ## Open Source & Contributing This repository is published under the [CC BY-NC-ND](https://creativecommons.org/licenses/by-nc-nd/4.0/) license. Please see our [code of conduct](https://github.com/turbot/.github/blob/main/CODE_OF_CONDUCT.md). Contributors must sign our [Contributor License Agreement](https://turbot.com/open-source#cla) as part of their first pull request. We look forward to collaborating with you! [Steampipe](https://steampipe.io) is a product produced from this open source software, exclusively by [Turbot HQ, Inc](https://turbot.com). It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but they cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our [Open Source FAQ](https://turbot.com/open-source).