## 1. Project Overview & Quickstart (prestodb/presto)
# Presto
[](https://insights.linuxfoundation.org/project/presto)
Presto is a distributed SQL query engine for big data.
See the [Presto installation documentation](https://prestodb.io/docs/current/installation.html) for deployment instructions.
See the [Presto documentation](https://prestodb.io/docs/current/) for general documentation.
## Mission and Architecture
See [PrestoDB: Mission and Architecture](ARCHITECTURE.md).
## Requirements
* Mac OS X or Linux
* Java 17 64-bit. Both Oracle JDK and OpenJDK are supported.
* Maven 3.6.3+ (for building)
* Python 2.4+ (for running with the launcher script)
Building Presto
### Overview (Java)
Presto is a standard Maven project. Simply run the following command from the project root directory:
./mvnw clean install
On the first build, Maven will download all the dependencies from the internet and cache them in the local repository (`~/.m2/repository`), which can take a considerable amount of time. Subsequent builds will be faster.
When building multiple Presto projects locally, each project may write updates to the user's global M2 cache, which could cause build issues. You can configure your local `.mvn/maven.config` to support a local cache specific to that project via `-Dmaven.repo.local=./.m2/repository`.
Presto has a comprehensive set of unit tests that can take several minutes to run. You can disable the tests when building:
./mvnw clean install -DskipTests
After building Presto for the first time, you can load the project into your IDE and run the server. We recommend using [IntelliJ IDEA](http://www.jetbrains.com/idea/). Because Presto is a standard Maven project, you can import it into your IDE using the root `pom.xml` file. In IntelliJ, choose Open Project from the Quick Start box or choose Open from the File menu and select the root `pom.xml` file.
After opening the project in IntelliJ, double check that the Java SDK is properly configured for the project:
* Open the File menu and select Project Structure
* In the SDKs section, ensure that a distribution of JDK 17 is selected (create one if none exist)
* In the Project section, ensure the Project language level is set to at least 8.0.
* When using JDK 17, an [IntelliJ bug](https://youtrack.jetbrains.com/issue/IDEA-201168) requires you
to disable the `Use '--release' option for cross-compilation (Java 9 and later)` setting in
`Settings > Build, Execution, Deployment > Compiler > Java Compiler`. If this option remains enabled,
you may encounter errors such as: `package sun.misc does not exist` because IntelliJ fails to resolve
certain internal JDK classes.
Presto comes with sample configuration that should work out-of-the-box for development. Use the following options to create a run configuration:
* Main Class: `com.facebook.presto.server.PrestoServer`
* VM Options: `-ea -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -Xmx2G -Dconfig=etc/config.properties -Dlog.levels-file=etc/log.properties -Djdk.attach.allowAttachSelf=true`
* Working directory: `$MODULE_WORKING_DIR$` or `$MODULE_DIR$`(Depends your version of IntelliJ)
* Use classpath of module: `presto-main`
The working directory should be the `presto-main` subdirectory. In IntelliJ, using `$MODULE_DIR$` accomplishes this automatically.
Additionally, the Hive plugin must be configured with location of your Hive metastore Thrift service. Add the following to the list of VM options, replacing `localhost:9083` with the correct host and port (or use the below value if you do not have a Hive metastore):
-Dhive.metastore.uri=thrift://localhost:9083
To modify the loaded plugins in IntelliJ, modify the `config.properties` located in `presto-main/etc`. You can modify `plugin.bundles` with the location of the plugin pom.xml
### Additional configuration for Java 17
When running with Java 17, additional `--add-opens` flags are required to allow reflective access used by certain catalogs based on which catalogs are configured.
For the default set of catalogs loaded when starting the Presto server in IntelliJ without changes, add the following flags to the **VM Options**:
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.ref=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.security=ALL-UNNAMED
--add-opens=java.base/javax.security.auth=ALL-UNNAMED
--add-opens=java.base/javax.security.auth.login=ALL-UNNAMED
--add-opens=java.base/java.text=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/java.util.regex=ALL-UNNAMED
--add-opens=java.base/jdk.internal.loader=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED
These flags ensure that internal JDK modules are accessible at runtime for components used by Presto’s default configuration.
It is not a comprehensive list. Additional flags may need to be added, depending on the catalogs configured on the server.
### Using SOCKS for Hive or HDFS
If your Hive metastore or HDFS cluster is not directly accessible to your local machine, you can use SSH port forwarding to access it. Setup a dynamic SOCKS proxy with SSH listening on local port 1080:
ssh -v -N -D 1080 server
Then add the following to the list of VM options:
-Dhive.metastore.thrift.client.socks-proxy=localhost:1080
### Running the CLI
Start the CLI to connect to the server and run SQL queries:
presto-cli/target/presto-cli-*-executable.jar
Run a query to see the nodes in the cluster:
SELECT * FROM system.runtime.nodes;
In the sample configuration, the Hive connector is mounted in the `hive` catalog, so you can run the following queries to show the tables in the Hive database `default`:
SHOW TABLES FROM hive.default;
### Building the Documentation
To build the Presto docs, see the [docs README](presto-docs/README.md).
### Building the Presto Console
The Presto Console is composed of several React components and is written in JSX and ES6. This
source code is stored in the `presto-ui/` module. The compilation process generates
browser-compatible javascript which is added as JAR resources during the maven build. When the
resource JAR is included on the classpath of Presto coordinator, it will be able to serve the
resources.
None of the Java code relies on the Presto UI project being compiled, so it is possible to exclude
this UI when building Presto. Add the property `-DskipUI` to the maven command to disable building
the `ui` maven module.
./mvnw clean install -DskipUI
You must have [Node.js](https://nodejs.org/en/download/) and [Yarn](https://yarnpkg.com/en/) installed to build the UI. When using Maven to build
the project, Node and yarn are installed in the `presto-ui/target` folder. Add the node and yarn
executables to the `PATH` environment variable.
To update Presto Console after making changes, run:
yarn --cwd presto-ui/src install
To simplify iteration, you can also run in `watch` mode, which automatically re-compiles when
changes to source files are detected:
yarn --cwd presto-ui/src run watch
To iterate quickly, simply re-build the project in IntelliJ after packaging is complete. Project
resources will be hot-reloaded and changes are reflected on browser refresh.
## Presto native and Velox
[Presto native](https://github.com/prestodb/presto/tree/master/presto-native-execution) is a C++ rewrite of Presto worker. [Presto native](https://github.com/prestodb/presto/tree/master/presto-native-execution) uses [Velox](https://github.com/facebookincubator/velox) as its primary engine to run presto workloads.
[Velox](https://github.com/facebookincubator/velox) is a C++ database library which provides reusable, extensible, and high-performance data processing components.
Check out [building instructions](https://github.com/prestodb/presto/tree/master/presto-native-execution#build-from-source) to get started.
## Using development containers
The PrestoDB project provides support for development containers in its own repository.
Please visit the [presto-dev README](https://github.com/prestodb/presto-dev/blob/main/README.md) for details.
## Contributing!
Please refer to the [contribution guidelines](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md) to get started.
## Questions?
[Please join our Slack channel and ask in `#dev`](https://communityinviter.com/apps/prestodb/prestodb).
## License
By contributing to Presto, you agree that your contributions will be licensed under the [Apache License Version 2.0 (APLv2)](LICENSE).
## 2. Official Technical Reference & Guides (prestodb/presto-dev)
# Presto/Prestissimo Dev Container
With this dev container, you can:
* Build Presto and Prestissimo from source
* Run Presto and Prestissimo with different configurations
* Debugging with VS Code or any IDE with ssh support
* Working on UI development
* Develop unit tests with test containers
## Prerequisites
* Docker or Podman must be installed and running.
## Quick start
This environment can be run on both Linux and macOS.
Run the following command to clone the `presto-dev` repository into your local `presto` directory, start the dev container and enter the container shell.
```sh
# Inside your local presto directory
git clone https://github.com/prestodb/presto-dev.git
# Start the dev container and enter container shell
cd presto-dev
make
```
### Optional: Create aliases for convenience
If you want to start the dev container from anywhere without navigating to the `presto-dev` directory, you can create a shell alias. For example, if your presto workspace is in `~/presto` and you cloned `presto-dev` to `~/presto/presto-dev`, add the following alias to your shell configuration file (e.g., `~/.bashrc`, `~/.zshrc`, or `~/.config/fish/config.fish`):
```sh
alias presto-dev-shell='pushd ~/presto/presto-dev; make; popd'
alias presto-dev-update='pushd ~/presto/presto-dev; make pull; git pull; popd'
alias presto-dev-stop='pushd ~/presto/presto-dev; make stop; popd'
```
After reloading your shell configuration, you can simply run `presto-dev-shell` from any directory to start the container and enter the shell.
### Work with source code
In the dev container shell, you can run the following commands to work with source code.
```sh
# Build presto (.m2 already cached)
cd /presto
./mvnw clean install -DskipTests
# Run unit tests with test container
start-podman
cd /presto
./mvnw test -pl presto-main -Dtest="com/facebook/presto/server/security/oauth2/**"
# Build prestissimo(make debug or make release)
cd /presto/presto-native-execution
make
# Start presto-ui development server, open http://localhost:8081 in your browser
# Run "start-cluster " first to start a presto cluster
cd /presto/presto-ui/src
yarn install
yarn serve
```
### Update presto-dev image
When a new `presto-dev` image is published, run the following command to pull the latest version.
```sh
# Under presto-dev, run this to pull the latest presto-dev image
make pull
# Or use alias
presto-dev-pull
```
## Start Presto & Prestissimo server
Run `make` or `make shell` in the presto-dev directory to open a new container shell.
All the commands below must be executed inside the container shell.
```sh
ls -la /root
# Start presto coordinator and prestissimo worker
start-cluster default
# Start presto single node and prestissimo sidecar
start-cluster sidecar
# Stop cluster
stop-cluster
```
Use http://localhost:8080 to open the Presto console.
## Update Presto or Prestissimo configuration
The configuration files are organized within cluster profiles in the `clusters/` directory. Each cluster profile contains its own configuration files for Presto coordinator, Presto worker, and/or Prestissimo worker.
Available cluster profiles:
- `presto`: Presto coordinator and worker
- `prestissimo`: Presto coordinator and Prestissimo worker
- `presto-single`: Single node Presto server
- `sidecar`: Presto coordinator with worker and Prestissimo worker with sidecar configuration
You can modify the existing configuration files in these profiles or create a new cluster profile based on your specific requirements. To create a new cluster profile:
1. Create a new directory under `clusters/` with your profile name
2. Copy the configuration files from an existing profile
3. Modify the configuration files as needed
4. Create a `start-cluster` script for your profile
The [presto/data](./presto/data) is mounted as `/opt/presto/data` to persist data between container restarts.
This approach allows you to maintain multiple configuration profiles and easily switch between them using the `start-cluster` script.
## Dev with dev container(VSCode)
1. Install [Remote Development](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack) extension.
2. Open `Remote Explorer` panel, switch to `Dev Containers` from the drop down widget on the top.
3. Open the `/presto` directory in the composed container `presto-dev` (If importing Maven projects takes a long time, run `./mvnw clean` in the `/presto` directory and `make clean` in the `/presto/presto-native-execution` directory inside the container.).
4. If `presto/.vscode/launch.json` is not already configured, the file [launch.json](./launch.json) is copied there automatically. If it is already configured, append the contents of [launch.json](./launch.json) from this repository to your existing `presto/.vscode/launch.json`.
5. Install Microsoft [Extension Pack for Java](https://marketplace.visualstudio.com/items?itemName=vscjava.vscode-java-pack) and [C/C++ Extension Pack](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools-extension-pack) in the container.
6. Start Presto & Prestissimo in the container.
7. After the Java projects are imported, open the Debug panel and select either `Attach to Prestissimo` or `Attach to Presto` to start debugging.
## Dev with ssh
VSCodium based IDEs cannot use the Remote Development extension. If your IDE supports remote ssh extension, you can try this:
1. Make sure you have generated ssh keys and updated the `authorized_keys` file in your local machine.
2. Use `make start` to start the container. This will copy your SSH keys from `~/.ssh/authorized_keys` to the `root/.ssh` directory inside `presto-dev`.
3. Add the following entry to your local `~/.ssh/config` file:
```
Host presto-dev
HostName localhost
Port 2222
User root
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
```
4. Test the ssh connection with `ssh presto-dev`.
5. Connect to the `presto-dev` SSH server using your IDE.
## Keep your data
By default, the `root` directory in this repo will be mounted to `/root` in the container. Use this `root` directory to share your data between your local machine and the container.
By default, the `.ccache`, `.m2`, and `.cache` directories under `/root` directory are symbolic links to the image's `/opt/cache` directories. These symbolic links will disappear when the container is shut down. To preserve your cache data between container restarts, you need to replace these symbolic links with local directories in the mounted `/root` directory.
Run the following command in the shell of dev container:
```sh
use-local-cache
```
This script replaces the symbolic links with actual directories in your local `/root` folder, copying the cache data from `/opt/cache`. Since the `root` directory is mounted into the container, these cache files will be preserved even when the container is removed, significantly improving build performance on subsequent runs.
## Development Scripts
The development environment includes scripts to help with building and installing Presto and Prestissimo.
### 1. Building and installing script
#### Available Scripts
- **build-and-install**: Builds Presto and Prestissimo from source and installs them into `/opt/presto` and `/opt/prestissimo` respectively.
#### Usage
```sh
build-and-install
```
This script:
1. Builds Presto from source in `/presto`
2. Builds Prestissimo from source in `/presto/presto-native-execution`
3. Installs the built binaries to `/opt/presto` and `/opt/prestissimo`
4. Makes the newly built versions available for use
After running this script, you can start the servers with the newly built versions using the cluster management scripts.
### 2. Podman Container Scripts
The development environment includes scripts to manage Podman containers for testing. Currently it only supports centos based dev container.
#### Available Scripts
- **start-podman**: Starts rootful podman with sock file `unix:///var/run/docker.sock` to enable running test containers inside the dev container.
- **stop-podman**: Stops the podman service.
#### Usage
```sh
# Start podman service
start-podman
# Run your tests that use test containers
cd /presto
./mvnw test -pl presto-main -Dtest="com/facebook/presto/server/security/oauth2/**"
# Stop podman service when done
stop-podman
```
The `start-podman` script enables running test containers inside the dev container by providing a Docker-compatible socket at `unix:///var/run/docker.sock`. This allows test frameworks like TestContainers to work properly within the development environment.
### 3. Cluster Management Scripts
The development environment includes scripts to easily start and stop Presto and Prestissimo clusters. These scripts are located in the `/root/bin` directory.
#### Available Scripts
- **start-cluster**: Starts a specific cluster profile (presto, prestissimo, presto-single, or sidecar)
- **stop-cluster**: Stops all running Presto and Prestissimo servers across all cluster profiles
#### Usage
##### Starting a Cluster
```sh
start-cluster
```
Where `` is one of the available cluster profiles in the `/opt/clusters` directory (e.g., presto, prestissimo, presto-single, sidecar).
Example:
```sh
# Start the presto-single cluster
start-cluster presto-single
# Start the prestissimo cluster
start-cluster prestissimo
```
##### Stopping All Clusters
```sh
stop-cluster
```
This command will stop all running Presto and Prestissimo servers across all cluster profiles.
#### Cluster Profiles
Each cluster profile has its own configuration and behavior:
- **presto**: Starts a Presto coordinator and a Presto worker
- **prestissimo**: Starts a Presto coordinator and a Prestissimo worker
- **presto-single**: Starts a single node Presto server (coordinator with worker)
- **sidecar**: Starts a single node Presto coordinator with worker and a Prestissimo worker with sidecar configuration
- **router**: Starts a Presto router and a single node Presto server
The logs for Presto servers are available at `/opt/presto/data/var/log/server.log`, the logs for Prestissimo servers are available at `/opt/prestissimo/logs/server.log`, and the logs for Presto router are available at `/opt/router/logs/server.log`.
## Build your own images
You can build your own images by forking this repository and update [.env](./.env) file to customize your build.
For example, if you want to push the images to your own docker registry, you can add the following lines to [.env](./.env) file:
```
ORG =
DOCKERHUB =
```
If you want to customize the compiling options like cmake flags or parallel build options, you can also update the [.env](./.env) file:
```
NUM_THREADS=
COMMON_CMAKE_FLAGS=
CENTOS_CMAKE_FLAGS=
UBUNTU_CMAKE_FLAGS=
```
In this way, you can build your own images and customize them according to your needs.
To build and push the images, you can use the following commands under directory `presto-dev`:
```sh
# Build centos images
make centos-cpp-dev && make centos-java-dev && make centos-dev
# Build ubuntu images
make ubuntu-cpp-dev && make ubuntu-java-dev && make ubuntu-dev
# Re-tag images for publishing
make release-prepare
# Publish images
make release-publish
```