1. Project Overview & Quickstart (PrefectHQ/prefect)
File: README.md
Prefect
Prefect is a workflow orchestration framework for building data pipelines in Python.
It's the simplest way to elevate a script into a production workflow.
With Prefect, you can build resilient, dynamic data pipelines that react to the world around them and recover from unexpected changes.
With just a few lines of code, data teams can confidently automate any data process with features such as scheduling, caching, retries, and event-based automations.
Workflow activity is tracked and can be monitored with a self-hosted Prefect server instance or managed Prefect Cloud dashboard.
[!TIP]
Prefect flows can handle retries, dependencies, and even complex branching logicCheck our docs or see the example below to learn more!
Getting started
Prefect requires Python 3.10+. To install the latest version of Prefect, run one of the following commands:
pip install -U prefectuv add prefectThen create and run a Python file that uses Prefect flow and task decorators to orchestrate and observe your workflow - in this case, a simple script that fetches the number of GitHub stars from a repository:
from prefect import flow, task
import httpx
@task(log_prints=True)
def get_stars(repo: str):
url = f"https://api.github.com/repos/{repo}"
count = httpx.get(url).json()["stargazers_count"]
print(f"{repo} has {count} stars!")
@flow(name="GitHub Stars")
def github_stars(repos: list[str]):
for repo in repos:
get_stars(repo)
# run the flow!
if __name__ == "__main__":
github_stars(["PrefectHQ/prefect"])Fire up a Prefect server and open the UI at http://localhost:4200 to see what happened:
prefect server startTo run your workflow on a schedule, turn it into a deployment and schedule it to run every minute by changing the last line of your script to the following:
if __name__ == "__main__":
github_stars.serve(
name="first-deployment",
cron="* * * * *",
parameters={"repos": ["PrefectHQ/prefect"]}
)You now have a process running locally that is looking for scheduled deployments!
Additionally you can run your workflow manually from the UI or CLI. You can even run deployments in response to events.
[!TIP]
Where to go next - check out our documentation to learn more about:
Prefect Cloud
Prefect Cloud provides workflow orchestration for the modern data enterprise. By automating over 200 million data tasks monthly, Prefect empowers diverse organizations — from Fortune 50 leaders such as Progressive Insurance to innovative disruptors such as Cash App — to increase engineering productivity, reduce pipeline errors, and cut data workflow compute costs.
Read more about Prefect Cloud here or sign up to try it for yourself.
prefect-client
If your use case is geared towards communicating with Prefect Cloud or a remote Prefect server, check out our
prefect-client. It is a lighter-weight option for accessing client-side functionality in the Prefect SDK and is ideal for use in ephemeral execution environments.
Connect & Contribute
Join a thriving community of over 25,000 practitioners who solve data challenges with Prefect. Prefect's community is built on collaboration, technical innovation, and continuous improvement.
Community Resources
🌐 Explore the Documentation - Comprehensive guides and API references
💬 Join the Slack Community - Connect with thousands of practitioners
🤝 Contribute to Prefect - Help shape the future of the project
🔌 Support or create a new Prefect integration - Extend Prefect's capabilities
📋 Tail the Dev Log - Prefect's open source development blog
Stay Informed
📥 Subscribe to our Newsletter - Get the latest Prefect news and updates
📣 X and Bluesky - Latest updates and announcements
📺 YouTube - Video tutorials and webinars
📱 LinkedIn - Professional networking and company news
Your contributions, questions, and ideas make Prefect better every day. Whether you're reporting bugs, suggesting features, or improving documentation, your input is invaluable to the Prefect community.
File: docs/index.md
home: true heroText: ' ' tagline: ' ' footer: Copyright © 2018-present Prefect Technologies, Inc.
Don't Panic.
<router-link to="core/">
Core Workflow Engine
</router-link>
<router-link to="orchestration/">
Orchestration & API
</router-link>Looking for the latest Prefect 2 release? Prefect 2 and Prefect Cloud 2 have been released for General Availability. See https://docs.prefect.io/ for details.
Prefect 1 Core, Server, and Cloud are our first-generation workflow and orchestration tools. You can continue to use them and we'll continue to support them while users migrate to Prefect 2.
If you're ready to start migrating your workflows to Prefect 2, see our migration guide.
We recommend starting new projects with Prefect 2. If you have questions about specific use cases, see our Prefect Discourse discussions for more information.
Automate all the things
If you can do it with Python, you can automate it with Prefect.
Test local, deploy global
Workflows are developed and tested locally, then deployed for execution at scale.
Simple but powerful
Prefect Cloud is powered by GraphQL, Dask, and Kubernetes, so it's ready for anything.
Prefect
We've rebuilt data engineering for the data science era.
Prefect is a new workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine. Users organize Tasks into Flows, and Prefect takes care of the rest.
Read the docs; get the code; ask us anything; chat with the community via Prefect Discourse!
Hello, world! 👋
from prefect import task, Flow, Parameter
@task(log_stdout=True)
def say_hello(name):
print("Hello, {}!".format(name))
with Flow("My First Flow") as flow:
name = Parameter('name')
say_hello(name)
flow.run(name='world') # "Hello, world!"
flow.run(name='Marvin') # "Hello, Marvin!"File: docs/README.md
<a href="https://github.com/PrefectHQ/prefect">
Get the code
</a>Automate all the things
If you can do it with Python, you can automate it with Prefect.
Test local, deploy global
Workflows are developed and tested locally, then deployed for execution at scale.
Simple but powerful
Prefect Cloud is powered by GraphQL, Dask, and Kubernetes, so it's ready for anything.
Prefect
We've rebuilt data engineering for the data science era.
Prefect is a new workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine. Users organize Tasks into Flows, and Prefect takes care of the rest.
Read the docs; get the code; ask us anything; chat with the community via Prefect Discourse!
Hello, world! 👋
from prefect import task, Flow, Parameter
@task(log_stdout=True)
def say_hello(name):
print("Hello, {}!".format(name))
with Flow("My First Flow") as flow:
name = Parameter('name')
say_hello(name)
flow.run(name='world') # "Hello, world!"
flow.run(name='Marvin') # "Hello, Marvin!"2. Official Technical Reference & Guides (PrefectHQ/docs)
Prefect Versioned Documentation
This repository contains Prefect's versioned documentation. Development of the docs
happens in the docs directory
of the main Prefect repository, so any edits or additions should be done there.
The docs repository serves two purposes:
- Build versioned documentation via a GitHub action.
- Make versioned docs available for publishing via Netlify.
How it works
First, the docs build action is triggered, either manually or by an automatic
repository dispatch call from the PrefectHQ/prefect repository.
Next, the build happens in the build workflow action.
To keep things simple, the workflow primarily consists of Bash commands that clone the a
branch or release tag of the Prefect repository,
prepare it for the mike build utility, sort the versions list mike generates, and
then create a pull request to merge the changes into main.
The workflow calls several small Python scripts in the utilities directory to handle
scripts handle preparation steps that would be difficult or verbose in bash.
Manual docs builds
Manually triggered builds accept two parameters:
- A version
- An optional display name that's displayed in the version selector box.
Choosing version and display name
If you don't provide a version name, the build action uses the version as the display
name. In most cases, this is what you want. If you're building the docs for 2.10.4,
for example, you probably want them published under the name 2.10.4.
In that scenario, entering 2.10.4 as the version and leaving the display name blank
will build the docs for Prefect 2.10.4 and publish them as 2.10.4 - exactly what
you want.
However, you may occasionally want to publish a docs version under an alias.
For example, it might be better to combine the docs for older release series under a
single name to keep the length of the version selector box manageable.
So, if you know that the docs for 2.6.12 are useful for the entire series of 2.6
releases, you might decide to build the docs for 2.6.12 but publish them as 2.6.x.
Merging and deploying manual builds
When a manual docs build completes, the build action will create a pull request that
must be reviewed by a human and merged into the main branch.
An updated copy of the website is deployed to Netlify every time a merge to main
occurs and the updated docs usually appear at https://docs.prefect.io in under a minute.
Automated docs builds
Automated docs builds run in response specific changes to in the Prefect repository.
The events in the Prefect repo that trigger a versioned docs build in this repository
are:
- A new release is published
- A change to the
docsdirectory is merged to main - A change to
netlify.tomlis merged to main
Merging and deploying automated builds
Automated builds create auto-merge pull requests, so no human intervention should be
required unless something goes wrong.
If the docs site isn't updating in response to
changes that should trigger a new build, check the list of recent build workflow
runs for build failures.
`unreleased` docs version
The versioned docs currently show an unreleased version in the version selector.
This version includes the latest docs changes from the main branch of the prefect
repository.
To hide the unreleased docs, do the following:
- Change
HIDE_UNRELEASED_VERSIONtoTrueinsort-docs-versions.py. - Delete the
unreleasedversion fromversions/versions.json. - Push the changes to the
mainbranch of this repository.