## 1. Project Overview & Quickstart (PrefectHQ/prefect) ## File: README.md # Prefect Prefect is a workflow orchestration framework for building data pipelines in Python. It's the simplest way to elevate a script into a production workflow. With Prefect, you can build resilient, dynamic data pipelines that react to the world around them and recover from unexpected changes. With just a few lines of code, data teams can confidently automate any data process with features such as scheduling, caching, retries, and event-based automations. Workflow activity is tracked and can be monitored with a self-hosted [Prefect server](https://docs.prefect.io/latest/manage/self-host/?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) instance or managed [Prefect Cloud](https://www.prefect.io/cloud-vs-oss?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) dashboard. > [!TIP] > Prefect flows can handle retries, dependencies, and even complex branching logic > > [Check our docs](https://docs.prefect.io/v3/get-started/index?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) or see the example below to learn more! ## Getting started Prefect requires Python 3.10+. To [install the latest version of Prefect](https://docs.prefect.io/v3/get-started/install), run one of the following commands: ```bash pip install -U prefect ``` ```bash uv add prefect ``` Then create and run a Python file that uses Prefect `flow` and `task` decorators to orchestrate and observe your workflow - in this case, a simple script that fetches the number of GitHub stars from a repository: ```python from prefect import flow, task import httpx @task(log_prints=True) def get_stars(repo: str): url = f"https://api.github.com/repos/{repo}" count = httpx.get(url).json()["stargazers_count"] print(f"{repo} has {count} stars!") @flow(name="GitHub Stars") def github_stars(repos: list[str]): for repo in repos: get_stars(repo) # run the flow! if __name__ == "__main__": github_stars(["PrefectHQ/prefect"]) ``` Fire up a Prefect server and open the UI at http://localhost:4200 to see what happened: ```bash prefect server start ``` To run your workflow on a schedule, turn it into a deployment and schedule it to run every minute by changing the last line of your script to the following: ```python if __name__ == "__main__": github_stars.serve( name="first-deployment", cron="* * * * *", parameters={"repos": ["PrefectHQ/prefect"]} ) ``` You now have a process running locally that is looking for scheduled deployments! Additionally you can run your workflow manually from the UI or CLI. You can even run deployments in response to [events](https://docs.prefect.io/latest/automate/?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none). > [!TIP] > Where to go next - check out our [documentation](https://docs.prefect.io/v3/get-started/index?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) to learn more about: > - [Deploying flows to production environments](https://docs.prefect.io/v3/deploy?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) > - [Adding error handling and retries](https://docs.prefect.io/v3/develop/write-tasks#retries?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) > - [Integrating with your existing tools](https://docs.prefect.io/integrations/integrations?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) > - [Setting up team collaboration features](https://docs.prefect.io/v3/manage/cloud/manage-users/manage-teams#manage-teams?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) ## Prefect Cloud Prefect Cloud provides workflow orchestration for the modern data enterprise. By automating over 200 million data tasks monthly, Prefect empowers diverse organizations — from Fortune 50 leaders such as Progressive Insurance to innovative disruptors such as Cash App — to increase engineering productivity, reduce pipeline errors, and cut data workflow compute costs. Read more about Prefect Cloud [here](https://www.prefect.io/cloud-vs-oss?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) or sign up to [try it for yourself](https://app.prefect.cloud?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none). ## prefect-client If your use case is geared towards communicating with Prefect Cloud or a remote Prefect server, check out our [prefect-client](https://pypi.org/project/prefect-client/). It is a lighter-weight option for accessing client-side functionality in the Prefect SDK and is ideal for use in ephemeral execution environments. ## Connect & Contribute Join a thriving community of over 25,000 practitioners who solve data challenges with Prefect. Prefect's community is built on collaboration, technical innovation, and continuous improvement. ### Community Resources 🌐 **[Explore the Documentation](https://docs.prefect.io)** - Comprehensive guides and API references 💬 **[Join the Slack Community](https://prefect.io/slack)** - Connect with thousands of practitioners 🤝 **[Contribute to Prefect](https://docs.prefect.io/contribute/)** - Help shape the future of the project 🔌 **[Support or create a new Prefect integration](https://docs.prefect.io/contribute/contribute-integrations)** - Extend Prefect's capabilities 📋 **[Tail the Dev Log](https://dev-log.prefect.io/)** - Prefect's open source development blog ### Stay Informed 📥 **[Subscribe to our Newsletter](https://prefect.io/newsletter)** - Get the latest Prefect news and updates 📣 **[X](https://x.com/PrefectIO)** and **[Bluesky](https://bsky.app/profile/prefect.io)** - Latest updates and announcements 📺 **[YouTube](https://www.youtube.com/@PrefectIO)** - Video tutorials and webinars 📱 **[LinkedIn](https://www.linkedin.com/company/prefect)** - Professional networking and company news Your contributions, questions, and ideas make Prefect better every day. Whether you're reporting bugs, suggesting features, or improving documentation, your input is invaluable to the Prefect community. --- ## File: docs/index.md --- home: true heroText: ' ' tagline: ' ' footer: Copyright © 2018-present Prefect Technologies, Inc. --- Don't Panic. Core Workflow Engine Orchestration & API Looking for the latest [Prefect 2](https://docs.prefect.io/) release? Prefect 2 and [Prefect Cloud 2](https://app.prefect.cloud) have been released for General Availability. See [https://docs.prefect.io/](https://docs.prefect.io/) for details. Prefect 1 Core, Server, and Cloud are our first-generation workflow and orchestration tools. You can continue to use them and we'll continue to support them while users migrate to Prefect 2. If you're ready to start migrating your workflows to Prefect 2, see our [migration guide](https://docs.prefect.io/migration-guide/). We recommend starting new projects with [Prefect 2](https://docs.prefect.io/). If you have questions about specific use cases, see our [Prefect Discourse](https://discourse.prefect.io/) discussions for more information. ## Automate all the things If you can do it with Python, you can automate it with Prefect. ## Test local, deploy global Workflows are developed and tested locally, then deployed for execution at scale. ## Simple but powerful Prefect Cloud is powered by GraphQL, Dask, and Kubernetes, so it's ready for anything. --- ### Prefect We've rebuilt data engineering for the data science era. Prefect is a new workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine. Users organize `Tasks` into `Flows`, and Prefect takes care of the rest. Read the [docs](/core/); get the [code](https://github.com/PrefectHQ/prefect); ask us [anything](https://www.prefect.io/slack); chat with the community via [Prefect Discourse](https://discourse.prefect.io/)! ### Hello, world! 👋 ```python from prefect import task, Flow, Parameter @task(log_stdout=True) def say_hello(name): print("Hello, {}!".format(name)) with Flow("My First Flow") as flow: name = Parameter('name') say_hello(name) flow.run(name='world') # "Hello, world!" flow.run(name='Marvin') # "Hello, Marvin!" ``` --- ## File: docs/README.md Read the docs Get the code ## Automate all the things If you can do it with Python, you can automate it with Prefect. ## Test local, deploy global Workflows are developed and tested locally, then deployed for execution at scale. ## Simple but powerful Prefect Cloud is powered by GraphQL, Dask, and Kubernetes, so it's ready for anything. --- ### Prefect We've rebuilt data engineering for the data science era. Prefect is a new workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine. Users organize `Tasks` into `Flows`, and Prefect takes care of the rest. Read the [docs](/core/); get the [code](https://github.com/PrefectHQ/prefect); ask us [anything](https://www.prefect.io/slack); chat with the community via [Prefect Discourse](https://discourse.prefect.io/)! ### Hello, world! 👋 ```python from prefect import task, Flow, Parameter @task(log_stdout=True) def say_hello(name): print("Hello, {}!".format(name)) with Flow("My First Flow") as flow: name = Parameter('name') say_hello(name) flow.run(name='world') # "Hello, world!" flow.run(name='Marvin') # "Hello, Marvin!" ``` ## 2. Official Technical Reference & Guides (PrefectHQ/docs) # Prefect Versioned Documentation This repository contains Prefect's versioned documentation. Development of the docs happens in [the `docs` directory](https://github.com/PrefectHQ/prefect/tree/main/docs) of the main Prefect repository, so any edits or additions should be done there. The `docs` repository serves two purposes: 1. Build versioned documentation via a GitHub action. 2. Make versioned docs available for publishing via Netlify. ## How it works First, the docs build action is triggered, either manually or by an automatic repository dispatch call from the `PrefectHQ/prefect` repository. Next, the build happens in the [build workflow action](https://github.com/PrefectHQ/docs/blob/main/.github/workflows/build-docs.yaml). To keep things simple, the workflow primarily consists of Bash commands that clone the a branch or release tag of [the Prefect repository](https://github.com/PrefectHQ/prefect), prepare it for the `mike` build utility, sort the versions list `mike` generates, and then create a pull request to merge the changes into `main`. The workflow calls several small Python scripts in the `utilities` directory to handle scripts handle preparation steps that would be difficult or verbose in bash. ## Manual docs builds Manually triggered builds accept two parameters: - A version - An optional display name that's displayed in the version selector box. ### Choosing version and display name If you don't provide a version name, the build action uses the version as the display name. In most cases, this is what you want. If you're building the docs for `2.10.4`, for example, you probably want them published under the name `2.10.4`. In that scenario, entering `2.10.4` as the version and leaving the display name blank will build the docs for Prefect `2.10.4` and publish them as `2.10.4` - exactly what you want. However, you may occasionally want to publish a docs version under an alias. For example, it might be better to combine the docs for older release series under a single name to keep the length of the version selector box manageable. So, if you know that the docs for `2.6.12` are useful for the entire series of `2.6` releases, you might decide to build the docs for `2.6.12` but publish them as `2.6.x`. ### Merging and deploying manual builds When a manual docs build completes, the build action will create a pull request that must be reviewed by a human and merged into the `main` branch. An updated copy of the website is deployed to Netlify every time a merge to `main` occurs and the updated docs usually appear at https://docs.prefect.io in under a minute. ## Automated docs builds Automated docs builds run in response specific changes to in the Prefect repository. The events in the Prefect repo that trigger a versioned docs build in _this_ repository are: - A new release is published - A change to the `docs` directory is merged to main - A change to `netlify.toml` is merged to main ### Merging and deploying automated builds Automated builds create auto-merge pull requests, so no human intervention should be required unless something goes wrong. If the docs site isn't updating in response to changes that _should_ trigger a new build, check the [list of recent build workflow runs](https://github.com/PrefectHQ/docs/actions/workflows/build-docs.yaml) for build failures. ## `unreleased` docs version The versioned docs currently show an `unreleased` version in the version selector. This version includes the latest docs changes from the main branch of the `prefect` repository. To hide the unreleased docs, do the following: - Change `HIDE_UNRELEASED_VERSION` to `True` in `sort-docs-versions.py`. - Delete the `unreleased` version from `versions/versions.json`. - Push the changes to the `main` branch of this repository.