## File: README.md # Mountpoint for Amazon S3 [](https://github.com/awslabs/mountpoint-s3/actions/workflows/tests.yml) [](https://github.com/awslabs/mountpoint-s3/actions/workflows/integration_main.yml) Mountpoint for Amazon S3 is a simple, high-throughput file client for [mounting an Amazon S3 bucket as a local file system](https://aws.amazon.com/blogs/storage/the-inside-story-on-mountpoint-for-amazon-s3-a-high-performance-open-source-file-client/). With Mountpoint for Amazon S3, your applications can access objects stored in Amazon S3 through file operations like `open` and `read`. Mountpoint for Amazon S3 automatically translates these operations into S3 object API calls, giving your applications access to the elastic storage and throughput of Amazon S3 through a file interface. Mountpoint for Amazon S3 is optimized for applications that need high read throughput to large objects, potentially from many clients at once, and to write new objects sequentially from a single client at a time. This means it's a great fit for applications that use a file interface to: * read large objects from S3, potentially from many instances concurrently, without downloading them to local storage first * access only some S3 objects out of a larger data set, but can't predict which objects in advance * upload their output to S3 directly, or upload files from local storage with tools like `cp` but probably not the right fit for applications that: * use file operations that S3 doesn't natively support, like directory renaming or symlinks * make edits to existing files (don't work on your Git repository or run `vim` in Mountpoint 😄) Mountpoint for Amazon S3 does not implement all the features of a POSIX file system, and there are some differences that may affect compatibility with your application. See [Mountpoint file system behavior](doc/SEMANTICS.md) for a detailed description of Mountpoint for Amazon S3's behavior and POSIX support and how they could affect your application. To troubleshoot file operations that may not be supported by Mountpoint, see the [troubleshooting documentation](doc/TROUBLESHOOTING.md). ## Current status Mountpoint for Amazon S3 is generally available! We're tracking future feature development on the [Mountpoint for Amazon S3 public roadmap](https://github.com/orgs/awslabs/projects/84). We're always interested in feedback on features, performance, and compatibility. Please send feedback by [opening a new GitHub issue](https://github.com/awslabs/mountpoint-s3/issues/new/choose) or adding your input to an existing roadmap issue. ### Notices > [!IMPORTANT] > Mountpoint for Amazon S3 v1.4.0, released on January 26, 2024, contains an [issue](https://github.com/awslabs/mountpoint-s3/pull/751) that causes intermittent read failures. We recommend that customers using v1.4.0 upgrade to v1.4.1 or later. ## Getting started On Amazon Linux 2023, you can install Mountpoint for Amazon S3 directly from the repository: sudo dnf install mount-s3 On other RPM-based distributions (Amazon Linux 2, Fedora, CentOS, RHEL; excluding SUSE), run these two commands to install Mountpoint for Amazon S3 on your EC2 instance (for Graviton instances, replace `x86_64` with `arm64` in the URL): wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm sudo yum install -y ./mount-s3.rpm On Ubuntu, use these commands instead (for Graviton instances, replace `x86_64` with `arm64` in the URL): wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb sudo apt-get install -y ./mount-s3.deb > [!NOTE] > See [Installing Mountpoint for Amazon S3](https://github.com/awslabs/mountpoint-s3/blob/main/doc/INSTALL.md) for detailed instructions and other installation options, including the [Mountpoint for Amazon S3 CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/s3-csi.html) for Kubernetes applications, or building from source. Once you've got Mountpoint for Amazon S3 installed, you can mount your Amazon S3 bucket. You'll need valid AWS credentials to access your bucket. Mountpoint will automatically use credentials from [an IAM role associated with your EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html), or the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables, or from [several other sources](https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#aws-credentials). To mount your bucket, run this command, replacing `amzn-s3-demo-bucket` with the name of your bucket and `/path/to/mount` with the directory you want to mount the bucket to: mount-s3 amzn-s3-demo-bucket /path/to/mount Now you can work with your bucket contents as if they were a local file system: ls /path/to/mount echo "Hello World!" > /path/to/mount/Data.txt cat /path/to/mount/Data.txt When you're finished accessing your bucket, you can unmount it (you might need `sudo`): umount /path/to/mount > [!NOTE] > See [Configuring Mountpoint for Amazon S3](https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md) for more details on how to configure and use Mountpoint. ### Compatibility with other storage services Mountpoint for Amazon S3 is designed for high-performance access to the Amazon S3 service. While it may be functional against other storage services that use S3-like APIs, we aren't able to provide support for those use cases, and they may inadvertently break when we make changes to better support Amazon S3. We welcome contributions of minor compatibility fixes or performance improvements for these services if the changes can be tested against Amazon S3. ## Contributing We welcome contributions to Mountpoint for Amazon S3! Please see [CONTRIBUTING.md](doc/CONTRIBUTING.md) for more information on how to report bugs or submit pull requests. We especially welcome contributions to issues tagged as [good first issues to work on](https://github.com/awslabs/mountpoint-s3/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22). For potential bug reports, see [LOGGING.md](doc/LOGGING.md) for details on how to capture logging data. For performance issues, see [BENCHMARKING.md](doc/BENCHMARKING.md) for documentation about performance regression testing. ### Security If you discover a potential security issue in this project we ask that you notify AWS Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public GitHub issue. ### Code of conduct This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). See [CODE_OF_CONDUCT.md](doc/CODE_OF_CONDUCT.md) for more details. ## License This project is licensed under the Apache-2.0 License. It builds on a number of other awesome projects with open source licenses, primarily: * The [fuser](https://github.com/cberner/fuser) bindings for FUSE (MIT license) * The [AWS Common Runtime](https://docs.aws.amazon.com/sdkref/latest/guide/common-runtime.html) for interacting with S3 (Apache 2.0 license) See [deny.toml](deny.toml) for a list of licenses used by our dependencies. --- ## File: benchmark/README.md # Benchmark experiment runner This project allows to perform some Mountpoint benchmarks with different variables, such that a number of experiments can be run with ease and the logs and results be collected in a directory for each experiment run. The Python script `benchmark.py` handles the setup and teardown for each experiment. The experiment configuration space is managed using [Hydra](https://hydra.cc/). Configurations in `conf/` describe which values to configure to run experiments over a set of parameters such as the maximum count of Mountpoint FUSE workers, number of application workers reading from unique file handles, etc.. The benchmark script currently supports FIO jobs. The list is defined in `conf/config.yaml` under the `fio_benchmarks` config entry. The FIO jobs define what workload they run, and also use environment variables in the job definition to allow this script to vary parameters. ## Before you start You should have the environment setup where you want to run the benchmarking experiments. For instance, this might be an EC2 instance. You also need an S3 bucket to run the workload against. You should clone this repository to the environment. This tool will build Mountpoint for you. This project uses [uv](https://github.com/astral-sh/uv) to manage Python environments and dependencies. Think of `uv` as a close analog of Rust's _cargo_ but for Python. It will automatically configure a Python virtual environment for you and install the project dependencies. Assuming `uv` is installed, getting started is (almost) as easy as running the `benchmark.py` script from this directory! ```sh uv run benchmark.py -- ``` It should tell you that you forgot some arguments for the Python script itself. ## Running the experiment There are a few variables that are required, such as the S3 bucket used for testing. You must set this in order to be able to use the benchmark script. Additionally, you should configure the AWS credentials for Mountpoint. You might use AWS profiles or set some credentials in the environment. To run the experiment, you can execute a command like this: ``` uv run benchmark.py -- s3_bucket=amzn-s3-demo-bucket ``` This will run the default experiment, including many different configuration combinations. The default benchmark type is fio. You can modify the benchmark type to run prefetch, client or crt benchmarks. To run prefetch benchmarks, you can execute a command like this: ``` uv run benchmark.py benchmark_type=fio -- s3_bucket=amzn-s3-demo-bucket ``` Output is written to `multirun/` within directories for the date, time, and experiment number run. The output directory includes a few different files from an individual experiment run, including the individual benchmark output `benchmark.log` and benchmark specific log files. For fio benchmarks, this will include FIO output, and Mountpoint logs. To run crt benchmarks, download [aws-crt-s3-benchmarks](https://github.com/awslabs/aws-crt-s3-benchmarks) and provide the path as input ``` uv run benchmark.py benchmark_type=crt benchmarks.crt.crt_benchmarks_path=aws-crt-s3-benchmarks-path -- s3_bucket=amzn-s3-demo-bucket ``` ## Advanced configuration ### Configuring multiple network interfaces When investigating performance with multiple network cards, we need to tell Mountpoint about what network interfaces are available and even configure it with things like a 'target throughput' such that it allocates enough resources to maximize its utilisation of the available network bandwidth. Below shows how to configure two network cards: ```sh uv run benchmark.py -- s3_bucket=amzn-s3-demo-bucket \ "network.interface_names=['eth0', 'eth1']" network.maximum_throughput_gbps=200 ``` If you want to run experiments varying the interfaces provided to Mountpoint, you can vary it like below: ```sh uv run benchmark.py -- s3_bucket=amzn-s3-demo-bucket \ "network.interface_names=['eth0'], ['eth0', 'eth1']" network.maximum_throughput_gbps=200 ``` If you want to do specific combinations, you will need to create full dictionaries to vary values over. Below is an example of varying both network interfaces alongside the target network throughput. ```sh uv run benchmark.py -- s3_bucket=amzn-s3-demo-bucket \ "network={interface_names:['eth0'],maximum_throughput_gbps:100},{interface_names:['eth0','eth1'],maximum_throughput_gbps:200}" ``` --- ## File: doc/BENCHMARKING.md ## Benchmarking Mountpoint for Amazon S3 is a simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system. To avoid new changes introducing performance regressions, we run a performance benchmark on every commit using [fio](https://github.com/axboe/fio), an awesome open-source application for file system benchmarking. ### Workloads ***read workload*** - we measure two aspects of the read operation, throughput (with and without caching enabled) and latency. For the first part, we use fio to simulate IO workloads for sequential read or random read for a specific duration then measure their throughput. On the latency side, we are using time to first byte as data points by running workloads that read one byte off of existing files on Mountpoint and measure the time it takes to complete the operation. Each of the test is defined in a separate .fio file, and the file name indicates what is the test case for that file, for example `seq_read.fio` is the benchmark for sequential read. All of fio configuration files can be found at path [mountpoint-s3/scripts/fio/read/](../mountpoint-s3/scripts/fio/read) and [mountpoint-s3/scripts/fio/read_latency/](../mountpoint-s3/scripts/fio/read_latency). In general, we run each IO operation for 30 seconds against a 100 GiB file. But there are some variants in configuration where we also want to test to see how Mountpoint would perform with these configurations. Here is the list of all the variants we have tested. * **four_threads**: running the workload concurrently by spawning four fio threads to do the same job. * **direct_io**: bypassing kernel page cache by opening the files with `O_DIRECT` option. This option is only available on Linux. * **small_file**: run the IO operation against smaller files (5 MiB instead of 100 GiB). ***readdir workload*** - we measure how long it takes to run `ls` command against directories with different size. Each directory has no subdirectory and contains a specific number of files, range from 100 to 100000 files, which we have to create manually using fio then upload them to S3 bucket before running the benchmark. The fio configuration files for creating them can be found at path [mountpoint-s3/scripts/fio/create/](../mountpoint-s3/scripts/fio/create). ***write workload*** - we measure write throughput by using fio to simulate sequential write workloads. The fio configuration files for write workloads can be found at path [mountpoint-s3/scripts/fio/write/](../mountpoint-s3/scripts/fio/write). ### Regression Testing Our CI runs the benchmark automatically for any new commits to the main branch or specific pull requests that we have reviewed and tagged with **performance** label. Every benchmark from the CI workflow will be running on `m5dn.24xlarge` EC2 instances (100 Gbps network speed) with Ubuntu 22.04 in us-east-1 against a bucket in us-east-1. We keep the records of benchmarking results in `gh-pages` branch and the performance charts are available for viewing: - [throughput chart](https://awslabs.github.io/mountpoint-s3/dev/bench/) - [throughput (with cache) chart](https://awslabs.github.io/mountpoint-s3/dev/cache_bench/) - [throughput (s3-express) chart](https://awslabs.github.io/mountpoint-s3/dev/s3-express/bench) - [latency chart](https://awslabs.github.io/mountpoint-s3/dev/latency_bench/) - [latency (s3-express) chart](https://awslabs.github.io/mountpoint-s3/dev/s3-express/latency_bench/) - [throughput chart (memory-limited)](https://awslabs.github.io/mountpoint-s3/dev/bench/mem_limited/) - [throughput (with cache) chart (memory-limited)](https://awslabs.github.io/mountpoint-s3/dev/cache_bench/mem_limited/) - [throughput (s3-express) chart (memory-limited)](https://awslabs.github.io/mountpoint-s3/dev/s3-express/bench/mem_limited/) - [throughput (s3-express, incremental upload) chart](https://awslabs.github.io/mountpoint-s3/dev/s3-express/bench/incremental_upload/) - [throughput (s3-express, incremental upload, memory-limited) chart](https://awslabs.github.io/mountpoint-s3/dev/s3-express/bench/incremental_upload/mem_limited/) ### Running the benchmark While our benchmark script is written for CI testing only, it is possible to run manually. You can use the following steps. 1. Install dependencies and configure FUSE by running the following script in the `mountpoint-s3` repository: bash .github/actions/install-dependencies/install.sh \ --fuse-version 2 \ --with-fio --with-libunwind 2. Set environment variables related to the benchmark. There are four required environment variables you need to set in order to run the benchmark. export S3_BUCKET_NAME=bucket_name export S3_BUCKET_TEST_PREFIX=prefix_path/ export S3_ENDPOINT_URL=endpoint_url # optional # these filenames only needed by cache benchmark export S3_BUCKET_BENCH_FILE=bench_file_name export S3_BUCKET_SMALL_BENCH_FILE=small_bench_file_name # to filter by job name; e.g. to only run small jobs export S3_JOB_NAME_FILTER=small # when running locally, we skip setting up local EC2 storage by default; to override that you can pass the following (value of the variable does not matter) export S3_MOUNT_LOCAL_STORAGE=yes 3. Run the benchmark script for [throughput](../mountpoint-s3/scripts/fs_bench.sh), [throughput with caching](../mountpoint-s3/scripts/fs_cache_bench.sh) or [latency](../mountpoint-s3/scripts/fs_latency_bench.sh). # to run throughput benchmarks ./mountpoint-s3/scripts/fs_bench.sh # to run throughput benchmarks with caching enabled ./mountpoint-s3/scripts/fs_cache_bench.sh # to run latency benchmarks ./mountpoint-s3/scripts/fs_latency_bench.sh 4. You should see the benchmark logs in `bench.out` file in the project root directory. The combined results will be saved into a JSON file at `results/output.json`. --- ## File: doc/CODE_OF_CONDUCT.md ## Code of Conduct This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact opensource-codeofconduct@amazon.com with any additional questions or comments. --- ## File: doc/CONFIGURATION.md # Configuring Mountpoint for Amazon S3 In most scenarios, you can use Mountpoint by running the following command, where you should replace `amzn-s3-demo-bucket` with the name of your Amazon S3 bucket, and `/path/to/mount` with the directory you want to mount your bucket into: mount-s3 amzn-s3-demo-bucket /path/to/mount We've tried hard to make this simple command adopt good defaults for most scenarios. However, some scenarios may need additional configuration. This document shows how to configure these elements of Mountpoint: * [AWS credentials](#aws-credentials) * [S3 bucket configuration](#s3-bucket-configuration), including mounting a bucket prefix or changing the endpoint to which Mountpoint sends S3 requests * [File system configuration](#file-system-configuration), including making a bucket read-only or allowing file deletion * [Caching configuration](#caching-configuration), where metadata and object data can be served from a cache * [Logging](#logging) for troubleshooting Mountpoint * [Metrics](#metrics) for monitoring Mountpoint ## AWS credentials Mountpoint uses the same [credentials configuration options](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) as the AWS CLI, and will automatically discover credentials from multiple sources. If you are able to run AWS CLI commands like `aws s3 ls` against your bucket, you should generally also be able to use Mountpoint against that bucket. > [!NOTE] > Mountpoint does not currently support authenticating with IAM Identity Center (SSO or Legacy SSO). This issue is tracked in [#433](https://github.com/awslabs/mountpoint-s3/issues/433). We recommend you use short-term AWS credentials whenever possible. Mountpoint supports several options for short-term AWS credentials: * When running Mountpoint on an Amazon EC2 instance, you can [associate an IAM role with your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) using an instance profile, and Mountpoint will automatically assume that IAM role and manage refreshing the credentials. * When running Mountpoint in an Amazon ECS task, you can similarly [associate an IAM role with the task](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) for Mountpoint to automatically assume and manage refreshing the credentials. * You can configure Mountpoint to [automatically assume a specific IAM role](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-role-overview) using the `role_arn` field of the `~/.aws/config` file. This configuration can be useful for cross-account access, where the target IAM role is in a different AWS account. You will need to specify how to obtain credentials that have permission to assume the role with either the `source_profile` or `credential_source` fields. For example, if you want Mountpoint to assume the IAM role `arn:aws:iam::123456789012:role/marketingadminrole`, you can associate an instance profile with your EC2 instance that has permission to assume that role, and then configure a profile in your `~/.aws/config` file: ``` [profile marketingadmin] role_arn = arn:aws:iam::123456789012:role/marketingadminrole credential_source = Ec2InstanceMetadata ``` With this configuration, running Mountpoint with the `--profile marketingadmin` command-line argument will automatically assume the specified IAM role and manage refreshing the credentials. * Otherwise, you can [acquire temporary AWS credentials for an IAM role](https://docs.aws.amazon.com/cli/latest/userguide/cli-authentication-short-term.html) from the AWS Console or with the `aws sts assume-role` AWS CLI command, and store them in the `~/.aws/credentials` file. If you need to use long-term AWS credentials, you can [store them in the configuration and credentials files](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) in `~/.aws`, or [specify them with environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`). To manage multiple AWS credentials, you can use the `--profile` command-line argument or `AWS_PROFILE` environment variable to select a profile from the configuration and credentials files. For public buckets that do not require AWS credentials, you can use the `--no-sign-request` command-line flag to disable AWS credentials. ### IAM permissions Amazon S3 offers both resource-based access policies attached to your S3 buckets (*bucket policies*) and user policies attached to IAM users (*user policies*). You can use either or both of these access policy options to control access to your S3 objects with Mountpoint. The permissions required to successfully mount your bucket and perform file system operations via Mountpoint differ between general purpose and directory buckets. Additionally, depending on the [file system configuration flags](#file-system-configuration) passed at mount time, some permissions may or may not be necessary. #### General purpose buckets On general purpose buckets, the IAM credentials you use with Mountpoint must have permission for the `s3:ListBucket` action for the S3 bucket you mount. To be able to read files with Mountpoint, you also need permission for the `s3:GetObject` action for the objects you read. By default, Mountpoint allows writing new files to your S3 bucket, and does not allow deleting existing files. You can disable writing new files, or enable deleting existing files, with [file system configuration flags](#file-modifications-and-deletions). Writing files requires permission for the `s3:PutObject` and `s3:AbortMultipartUpload` actions. Deleting existing files requires permission for the `s3:DeleteObject` action. If you only [mount a prefix of your S3 bucket](#mounting-a-bucket-prefix) rather than the entire bucket, you need these IAM permissions only for the prefix you mount. You can scope down your IAM permissions to a prefix using the `Resource` element of the policy statement for most of these permissions, but for `s3:ListBucket` you must use the `s3:prefix` condition key instead. If you are using server-side encryption with KMS (SSE-KMS), you will need additional permissions for KMS operations when reading or writing to objects. To read objects that are server-side encrypted with SSE-KMS, you will need permission for the `kms:Decrypt` action for the keys used to encrypt the objects. To upload new objects that will be encrypted with SSE-KMS, you will need permission for both the `kms:Decrypt` and `kms:GenerateDataKey` actions on the key used to encrypt the object. More details on permissions required when using SSE-KMS can be found in the [SSE-KMS section of the S3 User Guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). Here is an example least-privilege policy document to add to an IAM user or role that allows full access to your S3 bucket for Mountpoint. Replace `amzn-s3-demo-bucket` with the name of your bucket. Alternatively, you can use the [`AmazonS3FullAccess`](https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-iam-awsmanpol.html) managed policy, but the managed policy grants more permissions than needed for Mountpoint. ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "MountpointFullBucketAccess", "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket" ] }, { "Sid": "MountpointFullObjectAccess", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:AbortMultipartUpload", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket/*" ] } ] } ``` Mountpoint also respects [access control lists (ACLs) applied to objects](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html) in your S3 bucket, but does not allow you to automatically attach ACLs to objects created with Mountpoint. A majority of modern use cases in Amazon S3 no longer require the use of ACLs. We recommend that you keep ACLs disabled for your S3 bucket, and instead use bucket policies to control access to your objects. #### Directory buckets Directory buckets, introduced with the S3 Express One Zone storage class, use a different authentication mechanism from general purpose buckets. Instead of using `s3:*` actions, you should allow the `s3express:CreateSession` action. This will permit the principal to create an S3 Express session, allowing Mountpoint to perform any supported operation against the bucket. To further restrict what operations the principal can perform, you can use Mountpoint with access points for directory buckets. With access points for directory buckets, you can use the access point scope to restrict access to specific prefixes or API operations. Please refer to the Amazon S3 User Guide for more information, specifically [Security for directory buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-directory-buckets-policies.html) and [Working with access points for directory buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-directory-buckets.html). Below is an example of a least-privilege bucket policy granting read-write, session-based access to the directory bucket. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3express:CreateSession", "Resource": "arn:aws:s3express:REGION:ACCOUNT-ID:bucket/amzn-s3-demo-bucket--az_id--x-s3" } ] } ``` ## S3 bucket configuration By default, Mountpoint automatically mounts your S3 bucket based on the bucket name and selects a default S3 HTTPS endpoint. You can override the default if you want additional control on how Mountpoint connects to your bucket. ### S3 bucket types Mountpoint for Amazon S3 supports three bucket types: general purpose, directory buckets (in Availability Zones for S3 Express One Zone, and in AWS Dedicated Local Zones), and outposts buckets. To mount a bucket, provide the bucket name and following command: For a general purpose bucket use the full bucket name: ``` mount-s3 amzn-s3-demo-bucket /path/to/mount ``` For a directory bucket use the full bucket name, including the suffix: ``` mount-s3 amzn-s3-demo-bucket--az_id--x-s3 /path/to/mount ``` For more information on directory buckets see [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html). For more information on directory buckets in Dedicated Local Zones see [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-bucket-data-residency.html). For an Outposts bucket use the access point ARN or alias: ``` mount-s3 arn:aws:s3-outposts:region:123456789012:outpost/op-01ac5d28a6a232904/bucket/example-outposts-bucket /path/to/mount ``` Mountpoint for Amazon S3 supports [access points](#access-points) and [object lambda endpoints](#s3-object-lambda). ### Mounting a bucket prefix You can use Mountpoint to access only a prefix of your S3 bucket rather than the entire bucket. This allows you to isolate multiple users, applications, or workloads from each other within a single bucket. Use the `--prefix` command-line argument to specify a prefix of your S3 bucket, which must end with the `/` character. With this argument, only objects in your bucket that begin with the given prefix will be visible with Mountpoint. Alternatively, you can mount using an S3 URI including a prefix. Note that using an S3 URI prevents usage of ARNs to identify the bucket name. ``` mount-s3 s3://amzn-s3-demo-bucket/my/prefix/here/ /path/to/mount ``` When constructing the directory structure for your mount, Mountpoint removes any prefix you specify from object keys. For example, if your bucket has a key `2023/Files/data.json`, and you specify the `--prefix 2023/` command-line argument, the mounted directory will contain a single sub-directory `Files` with a file `data.json` inside it. If you specify the `--prefix 2023/Files/` command-line argument, the mounted directory will contain only a file `data.json` at its root. ### Region detection Amazon S3 buckets are associated with a single AWS Region. Mountpoint attempts to automatically detect the region for your S3 bucket at startup time and directs all S3 requests to that region. However, in some scenarios like cross-region mount with a directory bucket, this region detection may fail, preventing your bucket from being mounted and displaying Access Denied or No Such Bucket errors. You can override Mountpoint's automatic bucket region detection with the `--region` command-line argument or `AWS_REGION` environment variable. Mountpoint uses [instance metadata (IMDS)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) to help detect the region for an S3 bucket. If you want to disable IMDS, set the `AWS_EC2_METADATA_DISABLED` environment variable to `true`. ### Access points [Amazon S3 access points](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points.html) are network endpoints attached to buckets that you can use to perform S3 object operations. Each access point has distinct permissions and network controls that S3 applies for any request that is made through that access point. You can use an access point to S3 Standard bucket with Mountpoint by specifying either the [access point ARN](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-access-points.html) or the [access point bucket-style alias](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-points-usage-examples.html) as the bucket argument to `mount-s3`. For example, if your access point has the following ARN and alias: * ARN: `arn:aws:s3:region:account-id:accesspoint/my-access-point` * Access point alias: `my-access-point-hrzrlukc5m36ft7okagglf3gmwluquse1b-s3alias` then you can mount your S3 bucket to the `/path/to/mount` directory with either of the following commands: * `mount-s3 arn:aws:s3:region:account-id:accesspoint/my-access-point /path/to/mount` * `mount-s3 my-access-point-hrzrlukc5m36ft7okagglf3gmwluquse1b-s3alias /path/to/mount` You can use an access point to S3 directory bucket with Mountpoint by specifying name of the access point as the bucket argument to `mount-s3`. #### Multi-Region Access Points [Amazon S3 Multi-Region Access Points](https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPoints.html) provide a global endpoint that applications can use to fulfill requests to S3 buckets that are located in multiple AWS Regions. You can use a Multi-Region Access Point with Mountpoint by specifying its ARN as the bucket argument to `mount-s3`. For example, if your Multi-Region Access Point ARN is `arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap`, then you can mount your S3 bucket to the `/path/to/mount` directory with the command `mount-s3 arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap /path/to/mount`. ### S3 Object Lambda > [!IMPORTANT] > Not all Object Lambda functions will work with Mountpoint. Your Lambda function must satisfy some additional properties (listed below) for it to be usable with Mountpoint. [Amazon S3 Object Lambda](https://docs.aws.amazon.com/AmazonS3/latest/userguide/transforming-objects.html) allows you to add your own code to Amazon S3 `GET`, `LIST`, and `HEAD` requests to modify and process data as it is returned to an application. S3 Object Lambda uses AWS Lambda functions to automatically process the output of standard S3 `GET`, `LIST`, or `HEAD` requests. You can use S3 Object Lambda with Mountpoint by mounting an [Object Lambda Access Point](https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-use.html). Mounting an Object Lambda Access Point works the same way as [mounting an access point](#access-points), by specifying either the ARN or the bucket-style alias of the Object Lambda Access Point as the bucket argument to `mount-s3`. To use S3 Object Lambda with Mountpoint (or any other client), your IAM identity needs [additional permissions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-policies.html). To use S3 Object Lambda with Mountpoint, your Lambda function must satisfy three additional properties that may not be required by other applications: 1. Mountpoint uses the `Range` HTTP header for all `GetObject` requests to S3. To use S3 Object Lambda with Mountpoint, your Lambda function must be configured to enable the `Range` header, and must map the provided `Range` header to the transformed object. See [Working with Range and partNumber headers](https://docs.aws.amazon.com/AmazonS3/latest/userguide/range-get-olap.html) in the Amazon S3 User Guide for more details. 2. When looking up files and directories in your S3 bucket, Mountpoint sends concurrent `HeadObject` and `ListObjectV2` requests. The `HeadObject` request is expected to fail with a 404 Not Found HTTP status code when a file does not exist. For example, if your bucket contains a key `Files/data.json` and you run a command like `ls Files` on your mount, Mountpoint sends a `HeadObject` request for the key `Files` to discover if a file exists with that name, and will receive a 404 Not Found response from S3. Your Lambda function must correctly generate a 404 Not Found response for these requests. 3. When [working with `ListObjectV2` requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-writing-lambda.html#olap-listobjectsv2), your Lambda function's response can either include a JSON-formatted `listBucketResult` result that S3 Object Lambda automatically converts to a valid `ListObjectsV2` XML response, or include an XML-formatted `listResultXML` result that S3 Object Lambda does not validate further. If your Lambda function's response includes `listResultXML`, it must precisely match the XML schema for `ListObjectV2` responses, or Mountpoint may fail to parse it. ### Endpoints and AWS PrivateLink In most scenarios, Mountpoint automatically infers the appropriate Amazon S3 endpoint to send requests to based on the bucket name and region. This includes automatically using [gateway endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) you have created in your VPC to access S3 without internet access. However, you may need to provide additional command-line arguments to change the endpoint Mountpoint uses in some situations: * To [make requests to S3 over IPv6](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ipv6-access.html), use the `--dual-stack` command-line flag. * To use [Amazon S3 Transfer Acceleration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html) to optimize transfer speeds when accessing your S3 bucket over the internet, use the `--transfer-acceleration` command-line flag. Transfer Acceleration must be [enabled](https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html) on your S3 bucket to use this option. * To use interface VPC endpoints provisioned with [AWS PrivateLink for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html), specify the interface endpoint's DNS name with the `--endpoint-url` command-line argument. You must replace the `*` part of the DNS name displayed in the console with `bucket`. For example, if the console shows your interface endpoint's DNS name as `*.vpce-0e25b8cdd720f900e-argc85vg.s3.us-east-1.vpce.amazonaws.com`, specify the following endpoint URL argument to Mountpoint: ``` --endpoint-url https://bucket.vpce-0e25b8cdd720f900e-argc85vg.s3.us-east-1.vpce.amazonaws.com ``` Alternatively, if you enable [private DNS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html#private-dns) for your interface endpoint, you do not need to provide the `--endpoint-url` command-line argument. If necessary, you can use the `--endpoint-url` command-line argument to fully override Mountpoint's endpoint detection. For example, the argument `--endpoint-url https://example.com` will force Mountpoint to send S3 requests to `example.com`. You may need to also use the `--region` flag to correctly specify the region to use for signing requests. By default, Mountpoint will use [virtual-hosted-style addressing](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) for the configured endpoint, and so will send requests to `https://docexamplebucket.example.com` if configured with `--endpoint-url https://example.com` and the bucket name `docexamplebucket`. To disable virtual-hosted-style addressing, use the `--force-path-style` command-line flag to instead send requests to `https://example.com/docexamplebucket/`. We also support the `AWS_ENDPOINT_URL` environment variable. The endpoint determination follows this order: - Use the CLI parameter `endpoint-url` if provided. - Use `AWS_ENDPOINT_URL` if provided. - Fallback to automatically inferring the endpoint. ### Custom CA trust bundle To verify HTTPS connections against a private certificate authority, pass a PEM file containing the CA certificate chain with `--ca-bundle`: ``` mount-s3 amzn-s3-demo-bucket /path/to/mount --ca-bundle /path/to/ca.pem ``` When set, this bundle is used in place of the operating-system default trust store for all HTTPS connections Mountpoint makes. Mountpoint also honours the `AWS_CA_BUNDLE` environment variable as a fallback when `--ca-bundle` is not provided. ### Data encryption Amazon S3 supports a number of [server-side encryption types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingEncryption.html). Mountpoint supports reading and writing to buckets that are configured with Amazon S3 managed keys (SSE-S3), with AWS KMS keys (SSE-KMS), or with dual-layer encryption with AWS KMS keys (DSSE-KMS) as the default encryption method. It does not currently support reading objects encrypted with customer-provided keys (SSE-C). By default, Amazon S3 encrypts all objects with Amazon S3 managed keys (SSE-S3) and you can elect to use SSE-KMS with a customer managed key to meet compliance requirements. You can specify the AWS KMS key with Mountpoint when mounting a bucket or prefix. New objects can be uploaded using different server-side encryption (SSE) settings than the bucket's default. The CLI argument `--sse ` can be used to specify a different SSE encryption type. When either `aws:kms` or `aws:kms:dsse` is used as a type, `--sse-kms-key-id ` may be used to optionally specify a KMS key ARN. When a KMS key ARN is not specified, S3 will use an [AWS managed KMS key](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-mgmt), which is created automatically. Please note that these command-line arguments only configure server-side encryption for *new* objects created with Mountpoint, all *existing* objects will remain unchanged. > [!IMPORTANT] > Mountpoint currently accepts only **KMS key ARN** as the value for `--sse-kms-key-id` argument. AWS KMS [defines](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id) several other key identifiers, including key ID, key alias name and key alias ARN, which are not supported by Mountpoint. > [!IMPORTANT] > When Mountpoint is configured to use a [shared cache](#shared-cache) in [Amazon S3 Express One Zone](https://aws.amazon.com/s3/storage-classes/express-one-zone/), the CLI arguments `--sse` and `--sse-kms-key-id` also apply to the objects uploaded to the shared cache bucket and can be used to enforce or override their encryption settings. However, we recommend configuring encryption settings to match the default for the shared cache bucket, since some overriding configurations are not supported in Amazon S3 Express One Zone and may result in degraded performance when Mountpoint fails to read from the cache. For more information on overriding SSE settings for objects in Amazon S3 Express One Zone see [documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-specifying-kms-encryption.html). Mountpoint does not support client-side encryption using the Amazon S3 Encryption Client. ### Other S3 bucket configuration If the bucket you are mounting is a [Requester Pays bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html), you must acknowledge that you will be charged for the request and the data transferred, rather than the bucket owner. You provide this acknowledgement by using the `--requester-pays` command-line flag. If you try to mount a Requester Pays bucket without using this flag, mounting will fail with an Access Denied error. If you want to verify that the S3 bucket you are mounting, or an S3 directory bucket used for a shared cache is [owned by the expected AWS account](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-owner-condition.html), use the `--expected-bucket-owner` command-line argument. For example, if you expect the bucket to be owned by the AWS account `111122223333`, specify the argument `--expected-bucket-owner 111122223333`. If the argument doesn't match the bucket owner's account ID, mounting will fail with an Access Denied error. The same expected bucket owner is used for both the mounted S3 bucket and the shared cache bucket if configured. There are certain situations where Mountpoint receives a response from Amazon S3 indicating that a retry is necessary. For example, if an application generates high request rates (typically sustained rates of over 5,000 requests per second to a small number of objects), Mountpoint might receive HTTP 503 slowdown responses from S3. Mountpoint automatically retries these requests up to a total of 10 attempts, using jittered exponential backoff between attempts. If these attempts are exhausted, Mountpoint will return an error to your application (usually `EIO`). If you need to modify the maximum number of attempts, set the `AWS_MAX_ATTEMPTS` environment variable. ## File system configuration Mountpoint automatically configures reasonable defaults for file system settings such as permissions and for performance. You can adjust these settings if you need finer control over how the Mountpoint file system behaves. ### File modifications and deletions By default, Mountpoint allows creating new files but does not allow deleting or overwriting existing objects. If you want to allow file deletion, use the `--allow-delete` flag at mount time. Delete operations are immediately actioned against the object in S3, even if the file is being read from. File rename is supported for objects stored in the S3 Express One Zone storage class. Renames that would replace the destination file are only enabled when the `--allow-overwrite` flag is set. Rename operations are performed atomically and immediately actioned against the objects in S3, even where the source or any destination file is being read from. If you want to allow overwriting existing files, use the `--allow-overwrite` flag at mount time. The file must be opened with the `O_TRUNC` flag which will truncate the existing file. All writes must start from the beginning of the file and must be made sequentially. You can also allow appending to existing files for objects stored in the S3 Express One Zone storage class, by setting the `--incremental-upload` flag at mount time. In this mode, writes to existing files opened without the `O_TRUNC` flag are allowed, provided they start at the end of the file and are made sequentially. For more details, see [Reading and writing files](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md#reading-and-writing-files). If you want to forbid all mutating actions on your S3 bucket via Mountpoint, use the `--read-only` command-line flag. For more details on the behavior of file operations with Mountpoint, see the [file operations section](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md#file-operations) of the semantics documentation for more information. ### Content type detection By default, the `Content-Type` of new objects is set to `binary/octet-stream`. If you want Mountpoint to automatically infer the Content-Type from the file extension, use the `--infer-content-type` flag at mount time. Content-Type is not updated on rename. ### S3 storage classes Amazon S3 offers a [range of storage classes](https://aws.amazon.com/s3/storage-classes/) that you can choose from based on the data access, resiliency, and cost requirements of your workloads. When creating new files with Mountpoint, you can control which storage class the corresponding objects are stored in. Mountpoint respects the default storage class from S3 unless otherwise configured, which is appropriate for a wide variety of use cases. To store new objects in a different storage class, use the `--storage-class` command-line flag. Possible values for this argument include: * `STANDARD` for S3 Standard * `STANDARD_IA` for S3 Standard-Infrequent Access * `INTELLIGENT_TIERING` for [S3 Intelligent-Tiering](https://aws.amazon.com/s3/storage-classes/intelligent-tiering/), which automatically moves your data to the most cost-effective access tier when access patterns change * `GLACIER_IR` for [S3 Glacier Instant Retrieval](https://aws.amazon.com/s3/storage-classes/glacier/instant-retrieval/) * `GLACIER` for [S3 Glacier Flexible Retrieval](https://aws.amazon.com/s3/storage-classes/glacier/) * `DEEP_ARCHIVE` for [S3 Glacier Deep Archive](https://aws.amazon.com/s3/storage-classes/glacier/) > [!IMPORTANT] > `EXPRESS_ONEZONE` is a distinct storage class for directory buckets. You can neither use other storage classes in directory buckets nor use `EXPRESS_ONEZONE` in general purpose buckets. If you want to use [S3 Express One Zone](https://aws.amazon.com/s3/storage-classes/express-one-zone/) storage class, just specify a directory bucket name when mounting. For the full list of possible storage classes, see the [PutObject documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#AmazonS3-PutObject-request-header-StorageClass) in the Amazon S3 User Guide. Mountpoint supports reading existing objects from your S3 bucket when they are stored in any instant-retrieval storage class. You cannot use Mountpoint to read objects stored in the S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage classes, or the Archive Access or Deep Archive Access tiers of S3 Intelligent-Tiering, unless they've been [restored](https://docs.aws.amazon.com/AmazonS3/latest/userguide/restoring-objects.html). You can use Mountpoint to write new objects into these storage classes or S3 Intelligent-Tiering. ### File and directory permissions Mountpoint applies default permissions that allow all files in your mounted directory to be read and written by the local user who ran the `mount-s3` command. You can override these defaults in several ways: * To apply a different permission mode to files or directories, use the `--file-mode` and `--dir-mode` command-line arguments. * To change the ownership (user and group) of all files and directories, use the `--uid` and `--gid` command-line arguments. These arguments take user and group identifiers rather than names. You can find your user and group identifiers with the `id` command on Linux. By default, users other than the user who ran the `mount-s3` command cannot access your mounted directory, even if the permissions and ownership settings above would allow it. This is true even for the `root` user, and is a limitation of the FUSE system Mountpoint uses to create a file system. To allow other non-root users to access your mounted directory, use the `--allow-other` command-line flag. To allow the root user to access your mounted directory if you ran `mount-s3` as a different user, use the `--allow-root` command-line flag. To use these flags, you may need to first [configure FUSE](https://manpages.debian.org/testing/fuse/mount.fuse.8.en.html#CONFIGURATION) by adding the line `user_allow_other` to the `/etc/fuse.conf` file. Even with these flags enabled, Mountpoint still respects the permissions and ownership configured with the other flags above. Despite these configurations, [IAM permissions](#iam-permissions) still always apply to accessing the files and directories in your S3 bucket. ### Configuring Mountpoint performance At mount time, Mountpoint automatically selects appropriate defaults to provide high-performance access to Amazon S3. These defaults include [Amazon S3 performance best practices](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html) such as scaling requests across multiple S3 connections, using range `GET` requests to parallelize sequential reads, and using request timeouts and retries. Most applications should not need to adjust these defaults, but if necessary, you can change them in several ways: * Mountpoint scales the number and rate of parallel requests to meet a targeted maximum network throughput. This maximum is shared across all file and directory accesses made by a single Mountpoint process. By default, Mountpoint sets this maximum network throughput to the [available network bandwidth](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html) when running on an EC2 instance or to 10 Gbps elsewhere. To change this default, use the `--maximum-throughput-gbps` command-line argument, providing a value in gigabits-per-second (Gbps). For example, if you have multiple Mountpoint processes on the same instance, you can adjust this argument to partition the available network bandwidth between them. * By default, Mountpoint can serve up to 16 concurrent file or directory operations, and automatically scales up to reach this limit. If your application makes more than this many concurrent reads and writes (including to the same or different files), you can improve performance by increasing this limit with the `--max-threads` command-line argument. Higher values of this flag might cause Mountpoint to use more of your instance's resources. * When reading or writing files to S3, Mountpoint divides them into parts and uses parallel requests to improve throughput. You can change the part size Mountpoint uses for these parallel requests using the `--read-part-size` and `--write-part-size` command-line arguments, providing a maximum number of bytes per part for reading or writing respectively. For Mountpoint v1.7.2 or earlier, use `--part-size` instead. The default value for these arguments is 8 MiB (8,306,688 bytes), which in our testing is the largest value that achieves maximum throughput. Larger values can reduce the number of billed requests Mountpoint makes, but also reduce the throughput of object reads and writes to S3. ### Maximum object size In its default configuration, there is no maximum on the size of objects Mountpoint can read. However, Mountpoint uses [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) when writing new objects, and multipart upload allows a maximum of 10,000 parts for an object. This means Mountpoint can only upload objects up to 80,000 MiB (78.1 GiB) in size. If your application tries to write objects larger than this limit, writes will fail with an out of space error. To increase the maximum object size for writes, use the `--write-part-size` command-line argument to specify a maximum number of bytes per part, which defaults to 8 MiB. The maximum object size will be 10,000 multiplied by the value you provide for this argument. S3 allows a maximum part size of 5 GiB and a maximum object size of 48.8 TiB. For more information, see the [Amazon S3 multipart upload limits](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html). ### Automatically mounting an S3 bucket at boot Since v1.18, Mountpoint supports automatically mounting an S3 bucket as a local filesystem when your EC2 instance boots up or restarts using the filesystem table file (`/etc/fstab`). Once you modify the fstab file to add a new entry for Mountpoint, your compute instance will read the configuration from the fstab file whenever it restarts to automatically mount the S3 bucket. #### Example fstab entry You can use the following example to edit the fstab file to configure automatic mounting. This example uses the first field to specify the S3 location to mount (`amzn-s3-demo-bucket/example-prefix/`) and the second field to specify the local path (`/mnt/mountpoint`). The third field is always set to `mount-s3`, specifying that Mountpoint manages this mount. ``` s3://amzn-s3-demo-bucket/example-prefix/ /mnt/mountpoint mount-s3 _netdev,nosuid,nodev,nofail,rw 0 0 ``` Where: * `_netdev` specifies that the filesystem requires networking to mount. * `nosuid` specifies that the filesystem cannot contain set userid files. * `nodev` specifies that the filesystem cannot contain special devices. * `nofail` specifies that failure to mount the filesystem should still allow the system to boot. * `rw` specifies that the mount point be created with read and write permissions. Alternatively, use `ro` for read only. > [!WARNING] > The `_netdev`, `nosuid`, and `nodev` options are required when using Mountpoint from fstab. If you do not include these options, Mountpoint will fail to start. We recommend you also use the `nofail` option to allow the operating system to start up in case of problems. > [!IMPORTANT] > When running using fstab, Mountpoint will run as root and will use credentials as normal when launched from fstab, but must be available at instance startup. We recommend using IMDS as a credential provider when using Mountpoint with fstab. Given Mountpoint runs as root when using fstab, it will look in root’s home directory for any AWS profile (typically `/root/.aws/config` and `/root/.aws/credentials`). Unless you use the `allow-other` flag, only root will be able to access the filesystem. The options field takes regular Mountpoint CLI arguments as comma separated values in the form `key=value`. For example if you previously used `--allow-delete --uid 7 --region us-east-1 --allow-other` as CLI arguments, you would use `allow-delete,uid=7,region=us-east-1,allow-other` in your fstab file. ``` s3://amzn-s3-demo-bucket /mnt/mountpoint mount-s3 _netdev,nosuid,nodev,nofail,rw,allow-delete,uid=7,region=us-east-1,allow-other 0 0 ``` If you need to include commas in your CLI argument, for example `--user-agent-prefix foo,bar`, you would escape the commas in the argument with backslashes like so: `user-agent-prefix=foo\,bar`. Characters that need backslash escaping include backslashes (`\`), commas (`,`), and double quotes (`"`). ``` s3://amzn-s3-demo-bucket /mnt/mountpoint mount-s3 _netdev,nosuid,nodev,nofail,rw,user-agent-prefix=foo\,bar 0 0 ``` To stop Mountpoint from running at boot, remove the corresponding line from your fstab file. The filesystem will no longer be automatically mounted on subsequent reboots. #### Validating changes to the fstab file If your fstab file is invalid, your operating system may fail to boot. We recommend making a backup of your fstab file, and also using the `nofail` option to ensure invalid Mountpoint configuration or startup failures do not prevent the instance from booting. To validate changes, you can use the following snippet, replacing `/mnt/mountpoint` with the local path specified as the mount location for Mountpoint in your fstab file: ``` MNT_PATH=/mnt/mountpoint sudo systemctl daemon-reload sudo systemctl restart "$(systemd-escape --suffix=mount --path $MNT_PATH)" sudo systemctl status "$(systemd-escape --suffix=mount --path $MNT_PATH)" ``` Example output for a successful mount: `/etc/fstab` content: ``` s3://amzn-s3-demo-bucket/ /mnt/mountpoint mount-s3 _netdev,nosuid,nodev,rw,nofail,allow-other ``` systemd status: ``` ● mnt-mountpoint.mount - /mnt/mountpoint Loaded: loaded (/etc/fstab; generated) Active: active (mounted) since Tue 2025-05-20 10:53:36 UTC; 16s ago Where: /mnt/mountpoint What: mountpoint-s3 Docs: man:fstab(5) man:systemd-fstab-generator(8) Tasks: 17 (limit: 9347) Memory: 22.8M CPU: 72ms CGroup: /system.slice/mnt-mountpoint.mount └─476 /sbin/mount.mount-s3 s3://amzn-s3-demo-bucket/ /mnt/mountpoint -o rw,nosuid,nodev,allow-other,_netdev May 20 10:53:35 ip-172-31-40-171 systemd[1]: Mounting mnt-mountpoint.mount - /mnt/mountpoint... May 20 10:53:35 ip-172-31-40-171 mount[466]: Using 'fstab' style options as detected use of `-o` argument. May 20 10:53:36 ip-172-31-40-171 mount[466]: bucket amzn-s3-demo-bucket is mounted at /mnt/mountpoint May 20 10:53:36 ip-172-31-40-171 systemd[1]: Mounted mnt-mountpoint.mount - /mnt/mountpoint. ``` Example output for a failed mount: `/etc/fstab` content: ``` s3://amzn-s3-demo-bucket/ /mnt/mountpoint mount-s3 _netdev,nosuid,nodev,rw,nofail,invalid-argument ``` systemd status: ``` × mnt-mountpoint.mount - /mnt/mountpoint Loaded: loaded (/etc/fstab; generated) Active: failed (Result: exit-code) since Mon 2025-06-02 10:12:29 UTC; 125ms ago Duration: 6min 34.936s Where: /mnt/mountpoint What: s3://amzn-s3-demo-bucket/ Docs: man:fstab(5) man:systemd-fstab-generator(8) CPU: 5ms Jun 02 10:12:29 ip-172-31-35-235.eu-north-1.compute.internal systemd[1]: Mounting mnt-mountpoint.mount - /mnt/mountpoint... Jun 02 10:12:29 ip-172-31-35-235.eu-north-1.compute.internal mount[2616]: Using 'fstab' style options as detected use of `-o` argument. Jun 02 10:12:29 ip-172-31-35-235.eu-north-1.compute.internal mount[2616]: error: unexpected argument '--invalid-argument' found Jun 02 10:12:29 ip-172-31-35-235.eu-north-1.compute.internal mount[2616]: tip: to pass '--invalid-argument' as a value, use '-- --invalid-argument' Jun 02 10:12:29 ip-172-31-35-235.eu-north-1.compute.internal mount[2616]: Usage: mount.mount-s3 |--region |--endpoint-url Jun 02 10:12:29 ip-172-31-35-235.eu-north-1.compute.internal mount[2616]: For more information, try '--help'. Jun 02 10:12:29 ip-172-31-35-235.eu-north-1.compute.internal systemd[1]: mnt-mountpoint.mount: Mount process exited, code=exited, status=2/INVALIDARGUMENT Jun 02 10:12:29 ip-172-31-35-235.eu-north-1.compute.internal systemd[1]: mnt-mountpoint.mount: Failed with result 'exit-code'. Jun 02 10:12:29 ip-172-31-35-235.eu-north-1.compute.internal systemd[1]: Failed to mount mnt-mountpoint.mount - /mnt/mountpoint. ``` After successfully validating the fstab file, Mountpoint will be automatically mounted after subsequent reboots #### Using EC2 User Data to install and configure Mountpoint You can set up Mountpoint as part of new instance launches by including a user data script to install Mountpoint and configure the fstab file. Example user data script for x86 AL2023; you can adapt it for other architectures or operating systems: ``` #!/bin/bash -e # Install Mountpoint MP_RPM=$(mktemp --suffix=.rpm) curl https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm > $MP_RPM # cloud-init installs conflict with SSM agent: https://github.com/amazonlinux/amazon-linux-2023/issues/397 attempt=0 max_attempts=5 until yum install -y $MP_RPM; do attempt=$((attempt + 1)) if [ $attempt -ge $max_attempts ]; then echo "Failed to install mount-s3 after $max_attempts attempts. Exiting." exit 1 fi echo "yum install mount-s3 failed (attempt $attempt/$max_attempts), retrying in 3 seconds..." sleep 3 done rm $MP_RPM # Setup the fstab file and create the mount MNT_PATH=/mnt/mountpoint echo "s3://amzn-s3-demo-bucket/ ${MNT_PATH} mount-s3 _netdev,nosuid,nodev,rw,allow-other,nofail" >> /etc/fstab mkdir --parents $MNT_PATH # Mount all filesystems systemctl daemon-reload mount -a ``` ### Providing a FUSE file descriptor for mounting Mountpoint supports mounting S3 buckets at a given path, or using a provided FUSE file descriptor (only on Linux). For directory mount points, the passed path must be an existing directory. For FUSE file descriptors on Linux, you can specify an open FUSE file descriptor as a mount point with `/dev/fd/N` syntax. This is useful in container environments to achieve unprivileged mounts. In this case, the caller is responsible for the following: 1. Opening the FUSE device (`/dev/fuse`) in read-write mode to obtain a file descriptor. 2. Performing the `mount` syscall with the desired mount point, the file descriptor, and any mount options. Mountpoint by default uses and recommends enabling `nodev`, `nosuid`, `default_permissions`, and `noatime` mount options. See the [Linux kernel documentation](https://man7.org/linux/man-pages/man8/mount.fuse3.8.html#OPTIONS) for more details on mount options. 3. Spawning Mountpoint with the file descriptor using `/dev/fd/N` syntax as the mount point argument. 4. Closing the file descriptor in the parent process. 5. Performing the `unmount` syscall on the mount point when unmounting is desired or when the Mountpoint process terminates. See [mounthelper.go](https://github.com/awslabs/mountpoint-s3/tree/main/examples/fuse-fd-mount-point/mounthelper.go) as an example usage of this feature. ## Caching configuration Mountpoint can optionally cache file system metadata and object content to reduce cost and improve performance for repeated reads to the same file. Mountpoint can serve [supported file system requests](./SEMANTICS.md) from the cache, excluding listing of directory contents. ### Metadata Cache The command-line flag `--metadata-ttl ` controls the time-to-live (TTL) for cached metadata entries. It can be set to a positive numerical value in seconds, or to one of the pre-configured values of `minimal` (default configuration when not using `--cache` or `--cache-xz`) or `indefinite` (metadata entries never expire). > [!WARNING] > Caching of metadata entries relaxes the strong read-after-write consistency offered by Amazon S3 and Mountpoint in its default configuration. > See the [consistency and concurrency section of the semantics documentation](./SEMANTICS.md#consistency-and-concurrency) for more details. The `--metadata-ttl` flag is used to control how long Mountpoint considers it's file system metadata (file existence, size, object etag, etc) accurate before re-fetching from S3. When configured, on its own or in conjunction with local cache or shared cache, Mountpoint will typically perform fewer requests to the mounted S3 bucket, but will not guarantee that the information it reports is up to date with the content of the mounted S3 bucket. When configured with a local cache or shared cache, the stored data is considered accurate until the metadata TTL expires. After this period, Mountpoint revalidates if the cached data is still accurate by verifying the object's etag hasn't changed. Mountpoint provides two presets which trade off consistency (`--metadata-ttl minimal`) and performance/cost optimization (`--metadata-ttl indefinite`), and should be used depending on the requirements of your workload. In scenarios where the content of the mounted S3 bucket is modified by another client, and you require Mountpoint to return recently up-to-date information, setting `--metadata-ttl minimal` is most appropriate. A setting of `--metadata-ttl 300` would instead allow Mountpoint to perform fewer requests to the mounted S3 bucket by delaying updates for up to 300 seconds. If your workload does not require consistency, for example because the content of the mounted S3 bucket does not change, you should use `--metadata-ttl indefinite`. ### Data Cache Mountpoint for Amazon S3 supports different types of data caching that you can opt in to accelerate repeated read requests. First, you can use local data caching on your Amazon EC2 instance storage or an Amazon EBS volume. You should use a local cache if you repeatedly read the same data from the same compute instance and if you have unused space in your local instance storage for the repeatedly read dataset. Second, you can cache object content on S3 Express One Zone, shared across many instances. You should use a shared cache if you repeatedly read small objects from multiple compute instances or if you do not know the size of your dataset that you repeatedly read and want to benefit from elasticity of cache size. Once you opt in, Mountpoint retains objects with sizes up to one mebibyte (MiB) in an S3 directory bucket that uses S3 Express One Zone. Finally, you can use a local cache and shared cache together if you have unused space in your local cache, but also want a shared cache across multiple instances. Configuring a local or shared cache will also enable caching of metadata in memory using a default time-to-live (TTL) of 1 minute (60 seconds), which can be configured with the `--metadata-ttl` argument. ### Local Cache You can opt in to a local cache in your Amazon EC2 instance storage, instance memory, or an Amazon EBS volume. The main command-line flag to enable local caching is `--cache `, which specifies the directory in which to store cached object content. Mountpoint will create a new subdirectory within the path that you specify, and will remove any existing files or directories within that subdirectory at mount time and at exit. You should use the local cache if you repeatedly read the same data from the same compute instance, and if you have enough space in local storage to cache your dataset. This avoids redundant requests to your mounted S3 bucket when you read the same data repeatedly from the same instance. > [!WARNING] > If you enable local caching, Mountpoint will persist unencrypted object content from your mounted S3 bucket at the local cache location provided at mount. > In order to protect your data, you should restrict access to the data cache location by using file system access control mechanisms. #### Disk Cache Size By default, Mountpoint will limit the maximum size of the local cache such that the free space on the file system does not fall below 5%, and will automatically evict the least recently used content from the local cache when caching new content. You can instead manually configure the maximum size of the local cache with the `--max-cache-size ` command-line argument. > [!WARNING] > If you enable caching, Mountpoint will persist unencrypted object content from your S3 bucket at the location provided at mount. > In order to protect your data, we recommend you restrict access to the data cache location. #### Caching object content to local storage You should use local storage, such as Amazon EC2 instance storage or an Amazon EBS volume, as the target of the Mountpoint local cache. When caching to EBS, you can use your instance's root EBS volume, or create and attach a new volume just for caching. There are several factors that can affect the performance of EBS volumes. See the [EBS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html) for more details about EBS volume types and their performance characteristics. If you create a new EBS volume or use EC2 instance storage, you will first need to [create a file system](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/add-instance-store-volumes.html#making-instance-stores-available-on-your-instances) on that storage and mount it at a path such as `/mnt/mp-cache`. The user running Mountpoint needs write access to the mounted file system, and you should set the permissions on the file system to not allow reads by any other users (e.g., `chmod 0700 /mnt/mp-cache`). You can then start Mountpoint using the cache directory you mounted: ``` mount-s3 amzn-s3-demo-bucket /path/to/mount --cache /mnt/mp-cache ``` #### Caching object content to memory Rather than caching to local storage, you can configure Mountpoint to cache to instance memory by using a RAM disk. To create a RAM disk on Linux, you can use [tmpfs](https://www.kernel.org/doc/html/latest/filesystems/tmpfs.html) to mount a temporary file system at a path such as `/mnt/mp-cache-tmpfs`: ``` sudo mkdir /mnt/mp-cache-tmpfs sudo mount -o uid=$(id --user),mode=700 -t tmpfs none /mnt/mp-cache-tmpfs ``` This will create a RAM disk mounted at `/mnt/mp-cache-tmpfs` with access restricted to the current user. By default, Linux will set the size of the RAM disk to 50% of the physical memory available on the system. The size is configurable using the `size` option. You can then start Mountpoint using the directory where the RAM disk was mounted. ``` mount-s3 amzn-s3-demo-bucket /path/to/mount --cache /mnt/mp-cache-tmpfs ``` ### Shared Cache When mounting an S3 bucket, you can opt in to a shared cache in [Amazon S3 Express One Zone](https://aws.amazon.com/s3/storage-classes/express-one-zone/). You should use the shared cache if you repeatedly read small objects (up to 1 MB) from multiple compute instances, or the size of the dataset that you repeatedly read often exceeds the size of your local cache. This improves latency when reading the same data repeatedly from multiple instances by avoiding redundant requests to your mounted S3 bucket. > [!WARNING] > If you enable shared caching, Mountpoint will copy object content from your mounted S3 bucket in to the S3 directory bucket that you provide as your shared cache location, making it accessible to any caller with access to the S3 directory bucket. > To protect your cached data, you should follow the [security best practices for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-best-practices.html), and ensure that your Amazon S3 buckets use the correct policies and are not publicly accessible. > You should use a directory bucket dedicated to Mountpoint shared caching and grant access ONLY to Mountpoint clients. #### Getting started with caching object content to directory buckets that support S3 Express One Zone storage class To start using shared caching with S3 Express One Zone, use the `--cache-xz ` flag specifying an S3 directory bucket as your cache location when mounting. ``` mount-s3 amzn-s3-demo-bucket /path/to/mount --cache-xz amzn-s3-demo-bucket--usw2-az1--x-s3 ``` Please note the following key considerations while opting in to the shared cache: * To manage your storage cost, you should set up [Lifecycle configuration on your S3 directory bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-objects-lifecycle.html) so that Amazon S3 expires the cached data in S3 Express One Zone after a period of time you specify. Once you opt in to the shared cache in S3 Express One Zone, you pay for the data cached in your directory bucket in S3 Express One Zone. You also pay for requests made against your data in the directory bucket in S3 Express One Zone. Visit the [Amazon S3 pricing](https://aws.amazon.com/s3/pricing/) page to learn more. Mountpoint for Amazon S3 never deletes cached objects from S3 directory buckets. * Mountpoint requires sufficient read and write permissions in the S3 directory bucket, including `s3express:CreateSession`. See the [Mountpoint IAM configuration](./CONFIGURATION.md#iam-permissions) section for details. Note that the S3 directory bucket may not have the same permissions as your mounted S3 bucket. This means that users who have permissions on your S3 directory bucket can read data stored in the shared cache, even if they do not have permissions on the mounted S3 bucket. Write access to the shared cache could lead to cache poisoning. You should use a directory bucket dedicated to Mountpoint shared caching and grant access ONLY to Mountpoint clients. Mountpoint does not manage the permissions you set on your S3 buckets. You should also ensure that the mounted S3 bucket and the S3 directory bucket used as the shared cache belong to the same AWS account. * Mountpoint stores data in the shared cache using the encryption configuration set on the S3 directory bucket. SSE-S3 is the default encryption configuration for every S3 directory bucket. You can also specify [SSE-KMS as the S3 directory bucket's default encryption configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-specifying-kms-encryption.html) with a KMS key (specifically, a [customer managed key](https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#customer-cmk)). Note that when you choose the [SSE-KMS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html) option, Mountpoint will require `kms:Decrypt` and `kms:GenerateDataKey` permissions for the specified AWS KMS key to utilize the cache. * For optimal performance, you should consider keeping your compute instances in the same availability zone as the S3 directory bucket that you use for shared caching. * The S3 directory bucket that you use for shared caching and the mounted bucket must reside in the same region. You may need to specify `--region` flag in some cases, for more information see the [region detection](https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#region-detection) section. ### Combined Local and Shared Cache You can opt in to a local cache and shared cache together if you have unused space on your instance, but also want to share the cache across multiple instances. This avoids redundant read requests from the same instance to the shared cache in S3 directory bucket when the required data is cached in local storage, reducing request cost as well as improving performance. To opt in to local and shared cache together, you can specify both the cache locations by using `--cache` and `--cache-xz` flags when mounting an S3 bucket, as shown below: ``` mount-s3 amzn-s3-demo-bucket /path/to/mount --cache /path/to/mountpoint/cache --cache-xz amzn-s3-demo-bucket--usw2-az1--x-s3 ``` ### Using multiple Mountpoint processes on a host The cache directory is not reusable by other Mountpoint processes and will be cleaned at mount time and exit. When running multiple Mountpoint processes concurrently on the same host, you should use unique cache directories to avoid different processes interfering with the others' cache content. ## Using multiple network cards By default, Mountpoint will use the network interface associated with the default route on the host (such as that specified by `ip route list`). You can use the `--bind ` command-line argument to specify a different network interface for Mountpoint network requests to be routed over. This argument can be specified multiple times to allow requests to be fanned-out over multiple network interfaces. As an example, this command binds to two network interfaces and Mountpoint traffic will be distributed over them both: ``` mount-s3 amzn-s3-demo-bucket /path/to/mount --bind ens0 --bind ens1 ``` This feature is a work-in-progress. We welcome feedback on how this works for your applications and workloads. > [!NOTE] > This feature requires either Linux 5.7+ or running Mountpoint as the root user. > We recommend using a newer Linux kernel version to make use of this feature. ## Logging By default, Mountpoint emits high-severity log information to [syslog](https://datatracker.ietf.org/doc/html/rfc5424) if available on your system. You can change what level of information is logged, and to where it is logged. See [LOGGING.md](LOGGING.md) for more details on configuring logging. ## Metrics Mountpoint supports exporting metrics using OpenTelemetry protocol (OTLP) to provide insights into operations such as FUSE requests, S3 requests, and throughput. Use the `--otlp-endpoint` command-line argument to export metrics to CloudWatch Agent or other OTLP-compatible collectors. See [METRICS.md](METRICS.md) for more details. ## Unstable configurations Configurations in this section are experimental and may be removed or superseded by other configurations in any future release. ### Maximum prefetch window size Mountpoint downloads object contents into memory ahead of read requests in order to optimize sequential-read throughput. The prefetch window size determines how much data Mountpoint would prefetch data into memory and the value can be different for each file handle. These window sizes are adjusted automatically based on available system memory, scaling up to a maximum of 2 GiB per file handle by default. This should work well for most use cases but if you need to use a different value, you can set the maximum prefetch window size to an arbitrary number with the `UNSTABLE_MOUNTPOINT_MAX_PREFETCH_WINDOW_SIZE` environment variable at mount time, providing a value in bytes. > [!WARNING] > Overriding the default maximum prefetch window size may reduce overall read throughput. --- ## File: doc/CONTRIBUTING.md # Contributing Guidelines Thank you for your interest in contributing to Mountpoint for Amazon S3. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution. ## Reporting Bugs/Feature Requests Please use the GitHub issue tracker to [report bugs](https://github.com/awslabs/mountpoint-s3/issues/new?assignees=&labels=bug&template=bug-report.yml) or [suggest features](https://github.com/awslabs/mountpoint-s3/issues/new?assignees=&labels=enhancement&template=feature-request.yml). When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: * A reproducible test case or series of steps * The version of our code being used * Any modifications you've made relevant to the bug * Anything unusual about your environment or deployment Logs are also valuable for bug reports. Please read the [logging documentation](LOGGING.md) for details on how to capture verbose logs. ## Contributing via Pull Requests Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 1. You open an issue to discuss any significant work before starting implementation -- we would hate for your time to be wasted. 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 3. You are working against the latest source on the *main* branch. To send us a pull request, please: 1. Fork the repository. 2. Modify the source. Please focus on the specific change you are contributing; we value multiple small pull requests that are easier to review. 3. Run `make pre-pr-check` to ensure local tests pass and formatting is correct. 4. Commit to your fork using clear commit messages. * We require all contributors to agree to the terms of the [Developer Certificate of Origin (DCO)](https://developercertificate.org/), and to indicate this agreement by signing their commits using [Git's `Signed-off-by` support](https://git-scm.com/docs/git-commit#Documentation/git-commit.txt---signoff). * To add this trailer to your commits, use the `-s, --sign-off` option to `git commit`. 5. Send us a pull request, answering the default questions in the pull request interface. 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). ### Pull request title and description The pull request title and description is there to describe the change - both what is changing and why. It is important to address each section of the pull request template, and additionally to keep it maintained. The title and description will be used at merge time to create the squash commit message on the base branch (usually `main`). Keeping them both up-to-date as the code change evolves ensures that the commit message is useful for future readers. ### Changelog updates New features, bug fixes, or changes in behavior often justify a note in the `CHANGELOG.md` file for the relevant crate. The main Mountpoint changelog lives in the `mountpoint-s3` crate/directory, but we also maintain changelog entries for each of the crates that are published to crates.io. Here is the full list of changelog files we maintain for reference. - https://github.com/awslabs/mountpoint-s3/blob/main/mountpoint-s3/CHANGELOG.md - https://github.com/awslabs/mountpoint-s3/blob/main/mountpoint-s3-fs/CHANGELOG.md - https://github.com/awslabs/mountpoint-s3/blob/main/mountpoint-s3-client/CHANGELOG.md - https://github.com/awslabs/mountpoint-s3/blob/main/mountpoint-s3-crt/CHANGELOG.md - https://github.com/awslabs/mountpoint-s3/blob/main/mountpoint-s3-crt-sys/CHANGELOG.md Note that we do not maintain a change log for `mountpoint-s3-fuser`. ### Version numbers Mountpoint and the other `mountpoint-s3-*` crates in this repository follow [semantic versioning](https://semver.org/) rules. `mountpoint-s3` is the only crate with a stable release version (1.x), while the internal crates are unstable (0.x). All but the `mountpoint-s3` crate are published to crates.io. For `mountpoint-s3` (i.e. Mountpoint itself), new features and very minor breaking changes result in a minor version update while bug fixes result in a patch version update. For the crates, both we and Cargo treat the middle number (0.Y.z) as breaking changes and all new additions or bug fixes should update the latter number (0.y.Z). When a pull request is submitted, we consider whether the changes it contains should trigger a version number increment by comparing the latest published version with the one on `main`. Ultimately, we want both the crate version and the changelog to reflect all changes since the last published release. We should ensure that the dependent packages (for example, `mountpoint-s3-client` is a dependent of `mountpoint-s3-crt`) declare a dependency on the latest version number. In other words, when updating `mountpoint-s3-crt` from `0.1.0` to `0.1.1`, we should update `mountpoint-s3-client`'s dependency on that package to `0.1.1` also. Mountpoint (`mountpoint-s3`) change example: * A pull request introduces a new feature to Mountpoint such as a CLI flag and adds an entry to the changelog. * The latest release, as reported on the changelog, is `1.13.0`. * The minor version number must be incremented as a new feature is introduced when compared to the published (i.e. released) version, so the new version should be `1.14.0`. `mountpoint-s3-client` change example: * A pull request adds a new configurable option to the client and adds an entry to the changelog. * The latest release, as reported on the changelog, is `0.14.0`. * The patch version number should be incremented compared to the published version as this is new behavior for an unstable crate, so the new version should be `0.14.1`. * The declared dependencies of the workspace should be updated within the root `Cargo.toml`. In this case, the dependency for the client should be updated to be `0.14.1`. For any versioning change, we should ensure that the workspace lock file is updated and committed. This can be done by running a build, or something like `cargo check`. ## Finding contributions to work on Looking at the existing issues is a great way to find something to contribute on. We use issue labels to identify [good first issues](https://github.com/awslabs/mountpoint-s3/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) as a great place to start. ## Code of Conduct This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact opensource-codeofconduct@amazon.com with any additional questions or comments. ## Security issue notifications If you discover a potential security issue in this project we ask that you notify AWS Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public GitHub issue. ## Licensing Mountpoint for Amazon S3 is licensed under the Apache-2.0 license. The pull request template will ask you to confirm the licensing of your contribution and to agree to the [Developer Certificate of Origin (DCO)](https://developercertificate.org/). --- ## File: doc/INSTALL.md # Installing Mountpoint for Amazon S3 We recommend installing Mountpoint for Amazon S3 by [downloading and installing prebuilt packages using the command line](#download-and-install-mountpoint-for-amazon-s3-from-the-command-line), or [using the Mountpoint for Amazon S3 CSI driver for Kubernetes](#install-in-a-kubernetes-cluster-with-the-mountpoint-for-amazon-s3-csi-driver). Other installation options are also available. Mountpoint for Amazon S3 is only available for Linux operating systems. ## Download and install Mountpoint for Amazon S3 from the command line The instructions for downloading and installing Mountpoint for Amazon S3 depend on which Linux operating system you are using. ### Amazon Linux 2023 (AL2023) Mountpoint is available directly in the Amazon Linux 2023 repository since AL2023 version 2023.9.20251110. 1. Install it by entering the following command: ``` sudo dnf install mount-s3 ``` 2. Verify that Mountpoint for Amazon S3 is successfully installed: ``` mount-s3 --version ``` You should see output similar to the following: ``` mount-s3 1.21.0+1.amzn2023 ``` ### Other RPM-based distributions (Amazon Linux 2, Fedora, CentOS, RHEL; excluding SUSE) To download and install Mountpoint for Amazon S3 on RPM-based distributions, including Amazon Linux, follow these steps: 1. Download the Mountpoint for Amazon S3 package using the appropriate command for your architecture: * x86_64: ``` wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm ``` * ARM64 (Graviton): ``` wget https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.rpm ``` 2. Optionally, you can verify authenticity and integrity of the downloaded file. Identify the appropriate signature link depending on your architecture: * x86_64 architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm.asc` * ARM64 (Graviton) architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.rpm.asc` Then see [Verifying the signature of the Mountpoint for Amazon S3 package](#optional-verifying-the-signature-of-the-mountpoint-for-amazon-s3-package) below. 3. Install the package by entering the following command: ``` sudo yum install ./mount-s3.rpm ``` 4. Verify that Mountpoint for Amazon S3 is successfully installed by entering the following command: ``` mount-s3 --version ``` You should see output similar to the following: ``` mount-s3 1.0.0 ``` ### DEB-based distributions (Debian, Ubuntu) To download and install Mountpoint for Amazon S3 on DEB-based distributions, follow these steps: 1. Download the Mountpoint for Amazon S3 package using the appropriate command for your architecture: * x86_64: ``` wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb ``` * ARM64 (Graviton): ``` wget https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.deb ``` 2. Optionally, you can verify authenticity and integrity of the downloaded file. Identify the appropriate signature link depending on your architecture: * x86_64 architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb.asc` * ARM64 (Graviton) architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.deb.asc` Then see [Verifying the signature of the Mountpoint for Amazon S3 package](#optional-verifying-the-signature-of-the-mountpoint-for-amazon-s3-package) below. 3. Ensure that package index files are up to date by running the following command: ``` sudo apt-get update ``` 4. Install the package by entering the following command: ``` sudo apt-get install ./mount-s3.deb ``` 5. Verify that Mountpoint for Amazon S3 is successfully installed by entering the following command: ``` mount-s3 --version ``` You should see output similar to the following: ``` mount-s3 1.0.0 ``` ### On SUSE Linux Enterprise Server (SLES) To download and install Mountpoint for Amazon S3 on SLES, follow these steps: 1. Download the Mountpoint for Amazon S3 package using the appropriate command for your architecture: * x86_64: ``` wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.suse.rpm ``` * ARM64 (Graviton): ``` wget https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.suse.rpm ``` 2. Optionally, you can verify authenticity and integrity of the downloaded file. Identify the appropriate signature link depending on your architecture: * x86_64 architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.suse.rpm.asc` * ARM64 (Graviton) architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.suse.rpm.asc` Then see [Verifying the signature of the Mountpoint for Amazon S3 package](#optional-verifying-the-signature-of-the-mountpoint-for-amazon-s3-package) below. 3. Install the package by entering the following command: ``` sudo zypper refresh && sudo zypper --no-gpg-checks install -y ./mount-s3.suse.rpm ``` 4. Verify that Mountpoint for Amazon S3 is successfully installed by entering the following command: ``` mount-s3 --version ``` You should see output similar to the following: ``` mount-s3 1.0.0 ``` ### Other Linux distributions To download and install Mountpoint for Amazon S3 on other Linux distributions, follow these steps: 1. Install the necessary dependencies. Consult your operating system documentation to install the FUSE and libfuse (v2) packages. 2. Download the Mountpoint for Amazon S3 package using the appropriate command for your architecture: * x86_64: ``` wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.tar.gz ``` * ARM64 (Graviton): ``` wget https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.tar.gz ``` 3. Optionally, you can verify authenticity and integrity of the downloaded file. Identify the appropriate signature link depending on your architecture: * x86_64 architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.tar.gz.asc` * ARM64 (Graviton) architecture: `https://s3.amazonaws.com/mountpoint-s3-release/latest/arm64/mount-s3.tar.gz.asc` Then see [Verifying the signature of the Mountpoint for Amazon S3 package](#optional-verifying-the-signature-of-the-mountpoint-for-amazon-s3-package) below. 4. Install the package by entering the following command: ``` sudo mkdir -p /opt/aws/mountpoint-s3 && sudo tar -C /opt/aws/mountpoint-s3 -xzf ./mount-s3.tar.gz ``` 5. Add the `mount-s3` binary to your PATH environment variable. To do so, add the following line to your `$HOME/.profile` file: ``` export PATH=$PATH:/opt/aws/mountpoint-s3/bin ``` After adding this line, enter the following command: ``` source $HOME/.profile ``` 6. Verify that Mountpoint for Amazon S3 is successfully installed by entering the following command: ``` mount-s3 --version ``` You should see output similar to the following: ``` mount-s3 1.0.0 ``` ### Optional: Verifying the signature of the Mountpoint for Amazon S3 package To verify the authenticity and integrity of a downloaded Mountpoint for Amazon S3 package, follow these steps: 1. Install GnuPG (the `gpg` command) on your host if it is not already available. GnuPG is installed by default on Amazon Linux AMIs. 2. Download the Mountpoint for Amazon S3 public key by entering the following command: ``` wget https://s3.amazonaws.com/mountpoint-s3-release/public_keys/KEYS ``` 3. Import the Mountpoint for Amazon S3 public key into your keyring by entering the following command: ``` gpg --import KEYS ``` 4. Verify the fingerprint of the Mountpoint for Amazon S3 public key by entering the following command: ``` gpg --fingerprint mountpoint-s3@amazon.com ``` Confirm that the displayed fingerprint string matches one of the following: ``` 8AEF E705 EBE3 29C0 948C 75A6 6F1C 3B3A EF4B 030B 673F E406 1506 BB46 9A0E F857 BE39 7A52 B086 DA5A (older key) ``` If the fingerprint string does not match, do not finish installing Mountpoint for Amazon S3. 5. Download the package signature file by entering the following command. For `signature-link`, use the appropriate signature link from the sections above. ``` wget signature-link ``` 6. Verify the signature of the downloaded package by entering the following command. Use the appropriate `signature-filename` from the previous step. ``` gpg --verify signature-filename ``` For example, on RPM-based distributions, including Amazon Linux, enter the following command. ``` gpg --verify mount-s3.rpm.asc ``` The output should report a `Good signature`. If the output includes the phrase `BAD signature`, re-download the Mountpoint for Amazon S3 package file and repeat these steps. If the issue persists, do not finish installing Mountpoint for Amazon S3. The output may include a warning about a trusted signature. This does not indicate a problem, only that you have not independently verified the Mountpoint for Amazon S3 public key. ## Install in a Kubernetes cluster with the Mountpoint for Amazon S3 CSI driver To use Mountpoint for Amazon S3 with applications running in a Kubernetes cluster, we recommend installing and using the [Mountpoint for Amazon S3 CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/s3-csi.html). You can install the CSI driver on a self-managed Kubernetes cluster by following its [installation instructions](https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/docs/install.md#installation), or if you use Amazon EKS, it is also available as an [EKS managed add-on](https://docs.aws.amazon.com/eks/latest/userguide/s3-csi.html). ## Running Mountpoint for Amazon S3 in a Docker container You can run Mountpoint for Amazon S3 in a Docker container using our provided Dockerfiles. We don't recommend this option, as running a FUSE file system like Mountpoint inside a container requires giving the container broad root-level privileges to your host system. See [Running Mountpoint for Amazon S3 in a Docker container](../docker/README.md) for instructions on using Mountpoint with Docker. ## Installing previous Mountpoint for Amazon S3 releases We recommend always installing the latest Mountpoint for Amazon S3 release, but if you need to install a previous version, you can find links to them on the [GitHub Releases](https://github.com/awslabs/mountpoint-s3/releases) page. ## Building Mountpoint for Amazon S3 from source You can build Mountpoint for Amazon S3 from source. However, binaries built in this way are not officially supported by AWS. 1. Install the necessary dependencies. * For RPM-based distributions (Amazon Linux, Fedora, CentOS, RHEL), run the following command: ``` sudo yum install -y fuse fuse-devel cmake3 clang git pkgconfig ``` * For DEB-based distributions (Debian, Ubuntu), run the following command: ``` sudo apt-get install -y fuse libfuse-dev cmake clang git pkg-config ``` 2. Install the Rust compiler using [rustup](https://rustup.rs/): ``` curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && source "$HOME/.cargo/env" ``` 3. Clone the Mountpoint for Amazon S3 source code from GitHub: ``` git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git ``` 4. Build Mountpoint for Amazon S3: ``` cd mountpoint-s3 cargo build --release ``` 5. The final binary will be at `target/release/mount-s3`. Optionally, you can install this binary by copying it to the `/usr/bin` directory: ``` sudo cp target/release/mount-s3 /usr/bin/ ``` --- ## File: doc/LOGGING.md # Logging By default, Mountpoint for Amazon S3 emits INFO level and higher severity log information to [syslog](https://datatracker.ietf.org/doc/html/rfc5424) if available on your system (Note that for versions up to 1.20, only WARN level and higher severity events are logged by default). To view these logs on systems using `journald` (most modern Linux distributions, including Amazon Linux), run: journalctl -e SYSLOG_IDENTIFIER=mount-s3 On other systems, syslog entries are likely written to a file such as `/var/log/syslog`. When running `mount-s3` in foreground mode (the `-f, --foreground` command-line argument), Mountpoint will emit logs to stdout in addition to syslog or any configured log directory (see below). > [!NOTE] > If you are using [Mountpoint for Amazon S3 CSI Driver](https://github.com/awslabs/mountpoint-s3-csi-driver/), Mountpoint logs are available in the Mountpoint Pod logs as standard Kubernetes logs. > Follow [Mountpoint CSI Driver's LOGGING.md guide](https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/docs/LOGGING.md) to learn about how to get logs for both Mountpoint and the driver itself. ## Logging to a file You can direct logs to a file instead of syslog by providing a destination directory using the `-l, --log-directory` command-line argument with `mount-s3`. mount-s3 --log-directory The directory will be created if it doesn't exist. A new log file will be created for each execution of `mount-s3`. Log file names are not considered stable and may change in the future. Both the directory and log files are created with read/write access for the process owner and read access for the process owner's group. Log files are not automatically rotated or cleaned up. ## Disabling logging If you do not want to record any logs, use the `--no-log` command-line argument. This argument cannot be combined with other logging-related command-line arguments. The `--no-log` argument has no effect on messages sent to the standard output. If no output is desired, consider redirecting it. ## Verbose logging By default, Mountpoint logs INFO level and higher severity events (WARN level and higher up to version 1.20). For reporting issues or debugging application problems, it can be helpful to increase this verbosity. You can enable more verbose logging with the `--debug` command-line argument. We recommend logging to a file (the `-l, --log-directory` argument above) when using this option. ### Advanced logging verbosity options To enable more verbose logging for the AWS Common Runtime that Mountpoint uses to communicate with S3, use the `--debug-crt` command-line argument. These logs are very verbose, and should be combined with the `-l, --log-directory` and `--debug` arguments described above. For finer-grained control over log verbosity, Mountpoint uses the `MOUNTPOINT_LOG` environment variable, which overrides the verbosity options above. The `MOUNTPOINT_LOG` environment variable uses the [`tracing-subscriber` directive syntax](https://docs.rs/tracing-subscriber/0.3.17/tracing_subscriber/filter/struct.EnvFilter.html), and can be used to control log verbosity on a per-subject basis. For example, setting `MOUNTPOINT_LOG` to `trace` enables all trace-level logs, while `trace,awscrt=warn` enables trace-level logs for all log subjects except `awscrt`, which has only warning-level logging enabled. #### Changing logging verbosity at runtime > [!WARNING] > This is an unstable interface and might be subject to change in the future. > > The default action of `SIGUSR2` POSIX signal is to terminate the process. > Ensure Mountpoint version supports `SIGUSR2` signal before sending it, as otherwise it might terminate the process. Mountpoint v1.17.0 onward supports `SIGUSR2` signal as long as `--no-log` options is not passed. Mountpoint allows changing logging verbosity dynamically at runtime using `SIGUSR2` POSIX signal, e.g., `kill -USR2 `. Mountpoint toggles between the following verbosity levels each time it receives a `SIGUSR2` signal: 1. Default logging verbosity (i.e., the one configured using `--debug`, `--debug-crt`, or `MOUNTPOINT_LOG` environment variable) 2. Debug logging for all except CRT (i.e., `debug,awscrt=off`) 3. Debug logging for all (i.e., `debug,awscrt=debug`) 4. Trace logging for all except CRT (i.e., `trace,awscrt=off`) 5. Trace logging for all (i.e., `trace,awscrt=trace`) Note that increasing logging verbosity might affect runtime performance and cause more more log entries to be generated. Caution must be taken before using this feature in a production workload. ## Metrics Mountpoint optionally collects metrics measuring various values across different components. For example, Mountpoint records the durations of FUSE operations and the number of S3 responses grouped by HTTP status code. Metrics logging is controlled by the '--log-metrics' flag. When neither '--log-metrics' nor '--debug' is set, metrics logging will be turned off. In debug mode or when '--log-metrics' is set, metrics will be logged at the INFO level. Metrics will be collected by Mountpoint and flushed to the logs every five seconds. See below an example of what the emitted metrics may look like in the logs. [INFO] mountpoint_s3_fs::metrics: fuse.io_size(B)[fuse_request=read]: n=4: min=3184 p10=3199 p50=16511 avg=26494.00 p90=70143 p99=70143 p99.9=70143 max=70143 [INFO] mountpoint_s3_fs::metrics: fuse.request_latency(μs)[fuse_request=lookup]: n=8: min=22912 p10=23039 p50=65023 avg=62632.00 p90=95231 p99=95231 p99.9=95231 max=95231 [INFO] mountpoint_s3_fs::metrics: fuse.request_latency(μs)[fuse_request=open]: n=3: min=24448 p10=24575 p50=64255 avg=54037.33 p90=73727 p99=73727 p99.9=73727 max=73727 We recommend using the metrics only for debugging at this time. Metrics are currently output in an unstructured format and are subject to change in future releases. --- ## File: doc/PUBLISHING_CRATES.md # Publishing new releases of the `mountpoint-s3-*` crates This document guides maintainers in releasing new versions of the [`mountpoint-s3-*` crates on crates.io](https://crates.io/search?q=mountpoint-s3). We currently publish 5 crates: * A set of 3 crates for Mountpoint S3 Client (typically released together): * [`mountpoint-s3-client`](https://crates.io/crates/mountpoint-s3-client) * [`mountpoint-s3-crt`](https://crates.io/crates/mountpoint-s3-crt) * [`mountpoint-s3-crt-sys`](https://crates.io/crates/mountpoint-s3-crt-sys) * Mountpoint fork of the [fuser](https://crates.io/crates/fuser) crate: * [`mountpoint-s3-fuser`](https://crates.io/crates/mountpoint-s3-fuser) * Core Mountpoint file system crate: * [`mountpoint-s3-fs`](https://crates.io/crates/mountpoint-s3-fs) The remaining package in the repository, `mountpoint-s3` is used for the binary release of Mountpoint and is not currently published (except as a [placeholder](https://crates.io/crates/mountpoint-s3)). ## When to release Releases of Mountpoint, or of the `mountpoint-s3-client` crates, are not on a fixed schedule, but rather triggered by ad-hoc decisions based on the set of new features or bug fixes that have completed development. However, when releasing a new version of Mountpoint, or any of its crates, we must also release all its dependencies, **as long as they contain any changes**. Relevant crates must be published in reverse dependency order. For reference, this is the dependency tree of the crates in the repository: * `mountpoint-s3` (binary release) * `mountpoint-s3-fs` * `mountpoint-s3-client` * `mountpoint-s3-crt` * `mountpoint-s3-crt-sys` * `mountpoint-s3-fuser` * `mountpoint-s3-client` * [..] Before releasing a new Mountpoint version, consider all dependencies in the following order: * `mountpoint-s3-crt-sys` * `mountpoint-s3-crt` * `mountpoint-s3-client` * `mountpoint-s3-fuser` * `mountpoint-s3-fs` If only releasing `mountpoint-s3-client`, for example, just consider its dependencies (in the same order): * `mountpoint-s3-crt-sys` * `mountpoint-s3-crt` In any case, follow the instructions below for each crate in order to determine if it needs a new release and to publish it to [crates.io](https://crates.io). ## Preparing the release First, confirm that for each crate: * `CHANGELOG.md` lists the relevant changes since the latest released version, * `Cargo.toml` sets the version that you intend to publish. If `CHANGELOG.md` does not report any entries since the latest release, and the latter matches the version in `Cargo.toml`, **the crate should not contain any changes and there should be no need to publish a new version**. Second, prepare a new commit to update each `CHANGELOG.md` file by adding a header for the new release with the current date under the `## Unreleased` header. E.g.: ``` ## v0.10.0 (October 17, 2024) ``` Once ready, check everything still compiles and publish a pull request. After that is merged, the next step is to publish the new crates. ## Publishing First, ensure you have the latest commit on main which was just merged. ``` git switch main && git pull upstream main ``` Next, verify that publishing in `--dry-run` mode succeeds (there should be no warnings in logs and versions match expectations): ``` cargo publish --workspace --dry-run ``` If you haven't already, you should login to crates.io. Maintainers' GitHub accounts are authorized to publish new crate versions. ``` cargo login ``` The next steps should be completed for each crate that is being updated in the following order (see discussion above): * `mountpoint-s3-crt-sys` * `mountpoint-s3-crt` * `mountpoint-s3-client` * `mountpoint-s3-fuser` * `mountpoint-s3-fs` For each crate replacing the crate and version number where applicable: 1. Create a new Git tag for this crate and version. ``` git tag mountpoint-s3-crt-sys-9.9.9 ``` 2. Publish the new crate version. ``` cargo publish -p mountpoint-s3-crt-sys ``` 3. If successful, push the tag. ``` git push upstream mountpoint-s3-crt-sys-9.9.9 ``` Once these steps have been completed for all crates that need to be updated, you're done. You can check the new versions on crates.io: * [mountpoint-s3-crt-sys](https://crates.io/crates/mountpoint-s3-crt-sys) * [mountpoint-s3-crt](https://crates.io/crates/mountpoint-s3-crt) * [mountpoint-s3-client](https://crates.io/crates/mountpoint-s3-client) * [mountpoint-s3-fuser](https://crates.io/crates/mountpoint-s3-fuser) * [mountpoint-s3-fs](https://crates.io/crates/mountpoint-s3-fs) --- ## File: doc/SEMANTICS.md # Mountpoint for Amazon S3 file system behavior Mountpoint for Amazon S3 allows your applications to access objects stored in Amazon S3 through file operations like `open` and `read`. This file access is optimized for applications that need high read throughput to large objects, potentially from many clients at once, and to write new objects sequentially from a single client at a time. While this model suits a wide range of applications, Mountpoint does not implement all the features of a POSIX file system, and there are some differences that may affect compatibility with your application. If you need support for richer file system semantics that Mountpoint does not provide, you should consider other AWS file services such as [Amazon Elastic File System](https://aws.amazon.com/efs/) or [Amazon FSx](https://aws.amazon.com/fsx/). ## Behavior tenets While the rest of this document gives details on specific file system behaviors, we can summarize the Mountpoint approach in three high-level tenets: 1. Mountpoint does not support file behaviors that cannot be implemented efficiently against S3's object APIs. It does not emulate operations like `rename` on S3 general purpose buckets, which would require many API calls to S3 to perform. 2. Mountpoint presents a common view of S3 object data through both file and object APIs. It does not emulate POSIX file features that have no close analog in S3's object APIs, such as mutable ownership and permissions. 3. When these tenets conflict with POSIX requirements, Mountpoint fails early and explicitly. We would rather cause applications to fail with IO errors than silently accept operations that Mountpoint will never successfully persist, such as extended attributes. ## Reading and writing files Mountpoint supports opening and reading existing objects from your S3 bucket. It is optimized for reading large files sequentially, and will automatically make multiple concurrent requests to S3 to improve throughput when reads are sequential. Mountpoint also supports random reads from an existing object, including seeking in an open file. Mountpoint supports creating new objects in your S3 bucket by allowing writes to new files. If the `--allow-overwrite` flag is set at startup time, Mountpoint also supports replacing existing objects by allowing writes to existing files, but only when the `O_TRUNC` flag is used at open time to truncate the existing file. In both cases, writes must always start from the beginning of the file and must be made sequentially. Mountpoint uploads new files to S3 asynchronously, and optimizes for high write throughput using multiple concurrent upload requests. If your application needs to guarantee that a new file has been uploaded to S3, it should call `fsync` on the file before closing it. You cannot continue writing to the file after calling `fsync`. The new (or overwritten) object will be visible to other S3 clients only after successfully closing it (or on `fsync`). For objects stored in S3 Express One Zone, Mountpoint supports appending to files. If the `--incremental-upload` flag is set at startup time, Mountpoint allows opening existing files for writing without specifying the `O_TRUNC` flag. All writes must still be sequential and start from the end of the file. In this mode, Mountpoint will always upload data to S3 in sequential increments and offer the same throughput of a single PUT API call on S3. Moreover, partial writes will be visible to other S3 clients before the file is closed. Applications can call `fsync` to guarantee that the data written so far is uploaded to S3 and are then allowed to continue writing to the file. Mountpoint supports atomic file rename for objects stored in the S3 Express One Zone storage class. Attempting to rename files where unsupported by S3 will result in the operation being rejected. Mountpoint distinguishes between rename operations that move to an empty destination (non-replacing) and those that replace an existing object at the destination (replacing). While non-replacing renames do not require further flags to be set, replacing rename require passing the `--allow-overwrite` flag to Mountpoint at startup time. Rename operations immediately rename the object in S3. Existing readers (to source or destination) may eventually fail to read more data from the object after it has been renamed. You cannot rename a file while it or the destination of a rename is being written by the same Mountpoint instance. Append and rename are not supported for directory buckets that reside in Local Zones. You can only append data to or rename existing objects in directory buckets that reside in Availability Zones. You should not pass the `--incremental-upload` flag to Mountpoint in this case, as writes to files will fail. Attempting to rename files in this case will lead to the operation being rejected. By default, Mountpoint does not allow deleting existing objects with commands like `rm`. To enable deletion, pass the `--allow-delete` flag to Mountpoint at startup time. Delete operations immediately delete the object from S3, even if the file is being read from. We recommend that you enable [Bucket Versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html) to help protect against unintentionally deleting objects. You cannot delete a file while it is being written or renamed. Objects in the S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive storage classes, and the Archive Access and Deep Archive Access tiers of S3 Intelligent-Tiering, are only accessible with Mountpoint if they have been restored. To access these objects with Mountpoint, [restore](https://docs.aws.amazon.com/AmazonS3/latest/userguide/restoring-objects.html) them first. ## Directories The S3 data model is a flat structure, with no hierarchy of subdirectories. However, Mountpoint automatically infers a directory structure for your bucket by treating the `/` separator in your object keys as a delimiter between directories. For example, if your bucket contains the following object keys: * `colors/blue/cat.jpg` * `colors/red/dog.jpg` * `colors/list.txt` then mounting your bucket with Mountpoint gives the following file system structure: * `colors` (directory) * `blue` (directory) * `cat.jpg` (file) * `red` (directory) * `dog.jpg` (file) * `list.txt` (file) Not all S3 object keys correspond to valid file names, and these objects will not be accessible with Mountpoint. For example, a file system directory cannot contain both a file and a directory of the same name. If your bucket's directory structure would result in this state, only the directory will be accessible. This means that if your bucket contains the following object keys: * `blue` * `blue/image.jpg` then mounting your bucket with Mountpoint will show only the `blue` directory, containing the file `image.jpg`. The `blue` object will not be accessible. See the [detailed semantics](#mapping-s3-object-keys-to-files-and-directories) below for more information about invalid object keys. ### Modifying directories Mountpoint allows creating new directories with commands like `mkdir`. Creating a new directory is a local operation and no changes are made to your S3 bucket. A new directory will only be visible to other clients once a file has been written and uploaded inside it. If you restart Mountpoint or your instance before writing any files into the new directory, it will not be preserved. You cannot remove or rename an existing directory with Mountpoint. However, you can remove a new directory created locally if no files have been written inside it. Mountpoint does not support hard or symbolic links. ## Permissions and metadata By default, files and directories in your bucket will be readable only by the local user that mounted the bucket. If you want to allow other users on the system to read or write the bucket, pass the `--allow-other` flag to Mountpoint at startup time. Mountpoint assigns default permissions (modes) and owners to all files and directories, and these cannot be changed with commands like `chmod` and `chown` once the bucket is mounted. You can use the `--uid`, `--gid`, `--file-mode`, and `--dir-mode` flags at startup time to override these defaults. Mountpoint respects all Amazon S3 [identity and access management options](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-access-control.html), including bucket policies and access control lists (ACLs). At startup time, you provide IAM credentials for Mountpoint to use. Files and directories will only be accessible with Mountpoint if these credentials have the required access. If your credentials only have access to a prefix (a subdirectory) of an S3 bucket, you can use the `--prefix` argument at startup time to mount only that prefix instead of the entire bucket. Mountpoint has limited support for other file and directory metadata, including file modification times and sizes, and you cannot modify this metadata. ## Consistency and concurrency Amazon S3 provides [strong read-after-write consistency](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel) for PUT and DELETE requests of objects in your S3 bucket. By default, Mountpoint provides strong read-after-write consistency for file writes, directory listing operations, and new object creation. For example, if you create a new object using another S3 client, it will be immediately accessible with Mountpoint. If you modify an existing object in your bucket with another client while also reading that object through Mountpoint, the reads will return either the old data or the new data, but never partial or corrupt data. To guarantee your reads see the newest object data, you can re-open the file after modifying the object. "Another client" can be a separate software, such as the AWS SDK/CLI or the S3 Console, or another Mountpoint instance running on the same or a different machine. If you modify or delete an existing object in your S3 bucket with another client, however, Mountpoint may return stale metadata for that object for up to 1 second, by default. This occurs only if the object had already been accessed through Mountpoint immediately before being modified or deleted in your S3 bucket. The stale metadata will only be visible through metadata operations such as `stat` on individual files. Directory listings will never be stale and always reflect the current metadata. These cases do not apply to newly created objects, which are always immediately visible through Mountpoint. Stale metadata can be refreshed by either opening the file or listing its parent directory. Mountpoint allows multiple readers to access the same object at the same time. However, files can only be written to sequentially and by one writer at a time. Files that are being written to are not available for reading until the writing application closes the file, regardless of upload mode. If you have multiple Mountpoint mounts for the same bucket, on the same or different hosts, there is no coordination between writes to the same object. Your application should not write to the same object from multiple instances at the same time as this may have unexpected results in S3. By default, Mountpoint ensures that new file uploads to a single key are atomic. As soon as an upload completes, other clients are able to see the new key and the entire content of the object. If the `--incremental-upload` flag is set, however, Mountpoint may issue multiple separate uploads during file writes to append data to the object. After each upload, the appended object in your S3 bucket will be visible to other clients. ### Optional metadata and object content caching Mountpoint also offers optional metadata and object content caching. See the [caching section of the configuration documentation](./CONFIGURATION.md#caching) for more information. When opting into caching, the strong read-after-write consistency model is relaxed, and you may see stale file system metadata or object data for up to the cache's metadata time-to-live (TTL), which defaults to 1 minute but can be configured using the `--metadata-ttl` flag. For example, with local and/or shared caching enabled, you can successfully open and read a file that has been deleted from the mounted S3 bucket if it is already cached. Reads to that file will either return the cached data or an error for data that is not cached, but will never return corrupt data or combine data from two versions of the file. To force an up-to-date view of a file, use the `O_DIRECT` flag when opening the file for reading. When this option is provided, Mountpoint will check S3 to ensure the object exists, and return the latest object content. Unlike other file systems, Mountpoint does not support setting the `O_DIRECT` flag via `fcntl` after the file has been opened. When caching is enabled, Mountpoint also remembers when objects do *not* exist. Once you try to access a file that does not exist in your mounted S3 bucket, subsequent attempts (within the configured TTL) may still fail, even if it was later added to the mounted S3 bucket, until the TTL expires. Caching does not affect the behavior of writing to files. Files that are being written to remain unavailable for reading until the file is closed, consistent with behavior without caching. After the file is closed, it is possible to open it for reading. Parts of the file that are read from S3 will then be cached and available for subsequent repeated reads. ## Durability Mountpoint translates file operations like `read` and `write` into API calls to Amazon S3, which uses a combination of Content-MD5 checksums, secure hash algorithms (SHAs), and cyclic redundancy checks (CRCs) to verify data integrity. S3 performs these checksums on data at rest and repairs any disparity using redundant data. In addition, S3 calculates checksums on all internal network traffic to detect alterations of data packets when storing or retrieving data. However, POSIX file operations like `read` and `write` do not offer a built-in integrity mechanism. Like any file system operation, it is possible for data integrity to be lost in transit between your application and Mountpoint. If your application needs to verify data integrity, we recommend you use an AWS SDK instead of Mountpoint, and use [end-to-end checksums](https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/) for all object read and write operations. ## Error handling Unlike local file systems, operations against files and directories with Mountpoint can experience transient failures such as network timeouts or temporary unavailability. Mountpoint uses [best practices for S3 requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html#optimizing-performance-timeouts-retries), including retries, exponential backoff, and horizontal scaling. When a file operation fails despite these efforts, it might return a timeout or input/output error to your application. If your application needs to ensure that newly written files have been successfully uploaded to S3, use the `fsync` operation before closing the file. If the `fsync` operation returns an error, the file may not have been uploaded. ## Detailed semantics This section gives a detailed description of Mountpoint's semantics for individual operations. ### Mapping S3 object keys to files and directories Mountpoint interprets keys in your S3 bucket as file system paths by splitting them on the `/` character. For example, if your bucket contains the following object keys: * `colors/blue/image.jpg` * `colors/red/image.jpg` * `colors/list.txt` then mounting your bucket would give the following file system structure: * `colors` (directory) * `blue` (directory) * `image.jpg` (file) * `red` (directory) * `image.jpg` (file) * `list.txt` (file) S3 places fewer restrictions on [valid object keys](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) than POSIX does for valid file and directory names. As a result, some object keys in your S3 bucket may not be visible when mounting the bucket using Mountpoint: * Object keys that contain null bytes (`\0`) will not be accessible. * Object keys that would result in files or directories named `.` or `..` will not be accessible. This includes the object keys `.` or `..`, any key that ends in `/.` or `/..`, and any key that contains `/./` or `/../`. The `.` and `..` names are instead reserved for use by the usual relative directories (`.` for the current directory, and `..` for the parent). * Object keys that end in the path delimiter (`/`) will not be accessible. Instead, a directory of the same name will be visible. For example, if your bucket has the following object keys: * `blue/` * `blue/image.jpg` * `red/` then mounting your bucket would give a file system with a `blue` directory containing an `image.jpg` file, and an empty `red` directory. The `blue/` and `red/` objects will not be accessible. Note that the S3 Console creates zero-byte objects like `blue/` and `red/` when creating directories in a bucket, and so these directories will work as expected. * Files will be shadowed by directories with the same name. For example, if your bucket has the following object keys: * `blue` * `blue/image.jpg` then mounting your bucket would give a file system with a `blue` directory, containing the file `image.jpg`. The `blue` object will not be accessible. Deleting the key `blue/image.jpg` will remove the `blue` directory, and cause the `blue` file to become visible. Additionally, remote directories will always shadow local directories or files. Thus, Mountpoint shadows directory entries in the following order, where the first takes precedence: remote directories, any local state, remote files. For example, if you create a directory i.e. `blue/` and a conflicting object with key `blue` appears in the bucket, the local directory will still be accessible. We test Mountpoint against these restrictions using a [reference model](https://github.com/awslabs/mountpoint-s3/blob/main/mountpoint-s3-fs/tests/reftests/reference.rs) that programmatically encodes the expected mapping between S3 objects and file system structure. Windows-style path delimiters (`\`) are not supported. ### File operations #### Open `open` creates a file handle and returns it back to the kernel. A file handle can only be used for one type of operation, either read or write, for its lifetime. You can open a file in read-write mode (`O_RDWR`), but you cannot both read and write to the same file descriptor even in this mode. The default is to open for reads; unless in the below cases, where it is opened for writes: * The file being opened is new * Mountpoint has the `--allow-overwrite` flag set and the file is opened in truncate mode (with the `O_TRUNC` flag) * Mountpoint has the `--incremental-upload` flag set and the file is opened in append mode (with the `O_APPEND` flag) A file can not have a reader and a writer file handle open at the same time, and attempting to do so will result in an `EPERM` error for the second `open` request. However, a file can have multiple reader file handles open concurrently for it. Both `open` and `openat` operations are supported. `close` is also supported to conclude use of the file handle. #### Reads Basic read-only operations are fully supported, including both sequential and random reads: * `read`, `readv`, `pread`, `preadv` * `lseek` Multiple readers are allowed to access the same file concurrently. #### Writes Mountpoint supports sequential write operations (through `write`, `writev`, `pwrite`, `pwritev`), but with some limitations: * All writes must be sequential: writes after seeking to any offset other than the end of the previous write will fail. * Writes to new files are supported and must start at the beginning of the file. * If the `--allow-overwrite` flag is set, replacing an existing file is also allowed: * The existing file must be opened in truncate mode (`O_TRUNC`). * You cannot overwrite files that are currently being read or renamed. * The upload to S3 starts as soon as Mountpoint receives the first `write` request and cannot be cancelled. The data is uploaded to S3 in fixed-size parts (controlled by `--write-part-size`). * Both for new files and overwrites: * Synchronization operations (`fsync`, `fdatasync`) complete the upload of the object to S3 and disallow further writes. * The data written to the file will be visible to other S3 clients only once the upload completes. * If the `--incremental-upload` flag is set, and only when mounting directory buckets in S3 XOZ, appending to existing files is allowed: * The existing file must be opened without the `O_TRUNC` flag or any existing content will be truncated. * Only sequential writes at the end of the file are allowed. Setting the `O_APPEND` flag on open will enforce this behavior, but is not required by Mountpoint. * You cannot append to files that are currently being read, renamed or overwritten. * The data is uploaded incrementally to S3 in fixed-size parts (controlled by `--write-part-size`). * Synchronization operations (`fsync`, `fdatasync`) trigger the upload of the appended parts and do allow to continue writing. * Parts successfully appended to an object are visible as the whole (appended) object to other S3 clients. `close` also generally completes the upload of the object and reports an error if not successful. However, if the file is empty or if `close` is invoked by a different process than the one that originally opened it, `close` returns immediately and the upload is only completed asynchronously _after_ the last reference to the file is closed or when a new handle is opened to read/write to the file. These exceptions allow Mountpoint to support common usage patterns seen in tools like `dd`, `touch`, or in shell redirection, that hold multiple references to an open file and keep writing to one after closing another. Space allocation operations (`fallocate`, `posix_fallocate`) are not supported. Changing last access and modification times (`utime`) is supported only on files that are being written. #### Close and re-open Mountpoint allows a file to have multiple concurrent readers _or_ a single writer. The reference to a file reader/writer can be duplicated by the user (e.g. using `dup()`, `fork()`, as seen in tools like `dd` and `touch`, or in shell redirection), resulting in multiple references pointing to the same file handle in Mountpoint. A `close` request for a writer generally completes the upload of the object and reports an error if not successful. If a new `open` request is made for the file afterwards, it will succeed following semantics mentioned [here](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md#open). However, if the file is empty, or if `close` is invoked by a different process than the one that originally opened it, `close` returns immediately and the upload is only completed asynchronously _after_ the last reference to the file is closed. This is done to support common usage patterns that hold multiple references to an open file and keep writing to one after closing another. In cases where `close` deferred the upload, a subsequent new call to `open` (for reading or writing) on the same file will ensure that the upload to S3 is completed before proceeding. Duplicate references to the original writer will be invalidated by a new `open`, and any subsequent writes to them will start to fail (with the `EBADF: file handle has been invalidated by a newer handle opened` error). However, a `write` to a duplicate reference after the original `close`, but __before__ a new `open`, will succeed, marking the writer as still active. `open` requests for a reader/writer will return an `EPERM` error, since the file is already being written to. An `open` for writing request made for a file once _all_ its existing readers are closed will invalidate the duplicate references to the previous readers, and any subsequent reads to them will start to fail (with the `EBADF: file handle has been invalidated by a newer handle opened` error). However, a `read` to a duplicate reference after the original `close`, but __before__ a new `open`, will succeed, marking the reader as still active. `open` requests for another reader will succeed, but `open` requests for a writer will return an `EPERM` error since the file is already being read. > [!WARNING] > In releases up to `v1.21.0`, an `open` for writing request made immediately after a `close` could occasionally fail unexpectedly, because Mountpoint would try to process the `open` before completing an asynchronous upload for the writer or recording that the last reference to a reader had been closed. > See GitHub issues [#1344](https://github.com/awslabs/mountpoint-s3/issues/1344) and [#1327](https://github.com/awslabs/mountpoint-s3/issues/1327). #### Deletes File deletion (`unlink`) can be enabled by setting the `--allow-delete` option and is implemented with the following behavior: * For files not yet committed to S3 or those being renamed, the client does not permit `unlink` operations. * The file should be closed and thus committed to S3, at which point an `unlink` can be performed on the remote file. * For files already committed to S3, the client _immediately_ deletes the corresponding object from S3, and removes the file from its directory. * If there are still open file handles to the file, future reads to them may eventually fail. * Because the object is immediately deleted from S3, future reads from other hosts will also fail. ### Directory operations Basic read-only directory operations (`opendir`, `readdir`, `closedir`, `rewinddir`) are supported. However, seeking (`lseek`) on directory handles is not supported. Sorting order of `readdir` results: * For general purpose buckets, `readdir` returns results in lexicographical order. * For directory buckets, `readdir` does not return results in lexicographical order. Creating directories (`mkdir`) is supported, with the following behavior: * `mkdir` will create a new empty directory in the file system, but not affect the S3 bucket. * Note that this is different from e.g. the S3 Console, which creates "directory markers" (i.e. zero-byte objects with `/` key) in the bucket. * If a file is created under the new (or a nested) directory and committed to S3, Mountpoint will revert to using the default mapping of S3 object keys. This implies that the directory will be visible as long as there are keys which contain it as a prefix. Rename (`rename`, `renameat`, `renameat2`) semantics are described in the [File and directory rename](#file-and-directory-rename) section. File deletion (`unlink`) semantics are described in the [Deletes](#deletes) section above. Empty directory removal (`rmdir`) is supported, with the following semantics: * `rmdir` will only delete empty directories created by `mkdir`. * `rmdir` will fail on directories backed on S3 by a directory marker (i.e. zero-byte object with `/` key). * As soon as a file is committed to the S3 bucket by Mountpoint, the directory will be considered to exist implicitly. If Mountpoint later observes that there are no files existing for that directory in S3, Mountpoint will implicitly consider the directory to have been deleted. * On success, the directory will be deleted immediately. Subsequent reads or writes to the directory (e.g. creating a file or subdirectory) will fail. Synchronization operations (`fsync`) on directories are not supported. ### File and directory rename File and directory renames refer to the `rename` family of system calls, where a file is moved from one part of the file system tree to another. On Amazon S3 directory buckets in S3 Express One Zone, renaming individual files within the same bucket is supported with the following semantics: * Non-replacing file rename will work without any further Mountpoint configuration. * If a file already exists at the new destination path (replacing rename), the rename will fail, unless the `--allow-overwrite` flag was set at mount time and the system call does not specify the `RENAME_NOREPLACE` flag. * For files currently open for writing or not yet committed to S3, the client does not permit `rename` or `renameat2` operations. Similarly, if the destination file of a rename operation is open for writing or not yet committed to S3, the rename will be rejected. * For files not open for writing, the client _immediately_ renames the corresponding S3 object, and moves the file in the local file system representation. * Mountpoint does not support `rename` or `renameat2` where `RENAME_EXCHANGE` is specified, as exchanging two objects is not supported by Amazon S3. * Mountpoint does not support `rename` or `renameat2` where `RENAME_WHITEOUT` is specified, as Mountpoint is not an overlay/union file system. * If there are still open read file handles to the file being renamed or the destination file, future reads to them may eventually fail. * Because the object is immediately renamed in S3, future reads with file handles from other hosts will also fail. * Concurrent rename to a destination and uploads to the same key may result in the renamed object being overwritten. We do not recommend concurrent mutations to the same key. * Note, that renaming the last file in a directory has a similar effect to that directory as removing the last file. Thus, moving the last file out of a directory may lead to that directory being inaccessible, as there will no longer be an object in S3 under that directory. Renaming individual files is not supported by Amazon S3 general purpose buckets, nor objects not in the S3 Express One Zone storage class. Directory rename is not supported on any Amazon S3 bucket type. ### File and directory metadata and permissions Reading file metadata (`stat`, `fstatat`) is supported, but with some limitations: * File mode will be a default value (`0644` for files, `0755` for directories) unless you manually configure them with the `--file-mode` and `--dir-mode` command-line arguments. * File owner and group will default to the user/group that mounted the bucket unless you manually configure them with the `--uid` and `--gid` command-line arguments. * Last access time and last status change time will be the same as the last modified time. * Inode numbers are not stable and can change. Modifying file metadata (`chmod`, `chown`, `chgrp`) is not supported. Extended attributes (`getxattr`, `setxattr`, `listxattr`, `removexattr`) are not supported. POSIX file locks (`lockf`) are not supported. ### Links Hard links and symbolic links are both unsupported. ### Consistency Mountpoint provides strong read-after-write consistency for new object creation and writes of existing objects. However, it can return stale metadata for up to 1 second when an existing object is modified concurrently by another client. The [consistency and concurrency](#consistency-and-concurrency) section above describes this behavior, but here are some examples: * A process replaces an existing object in your S3 bucket using another client, and then opens the same object with Mountpoint and reads from it. The process will read the new data. * A process opens a file with Mountpoint, then replaces the object in your S3 bucket using another client, and then reads from the open file. The process will either read the old data or the read will fail. The process can see the new data by opening the file again. * A process replaces an existing object in your S3 bucket using another client, and then queries the object’s metadata with Mountpoint using the `stat` system call. The returned metadata could reflect either the old or new object for up to 1 second after the PutObject request. * A process writes a new object to your S3 bucket, using either Mountpoint or another client, and then lists the directory the object is in with Mountpoint. The new object will appear in the list. * A process deletes an existing object from your S3 bucket using another client, and then tries to open the object with Mountpoint and read from it. The open operation will fail. * A process deletes an existing object from your S3 bucket, using either Mountpoint or another client, and then lists the directory the object was previously in with Mountpoint. The object will not appear in the list. * A process deletes an existing object from your S3 bucket using another client, and then queries the object’s metadata with Mountpoint using the `stat` system call. The returned metadata could reflect the old object for up to 1 second after the DeleteObject request.