## File: README.md **JuiceFS** is a high-performance [POSIX](https://en.wikipedia.org/wiki/POSIX) file system released under Apache License 2.0, particularly designed for the cloud-native environment. The data, stored via JuiceFS, will be persisted in Object Storage _(e.g. Amazon S3)_, and the corresponding metadata can be persisted in various compatible database engines such as Redis, MySQL, and TiKV based on the scenarios and requirements. With JuiceFS, massive cloud storage can be directly connected to big data, machine learning, artificial intelligence, and various application platforms in production environments. Without modifying code, the massive cloud storage can be used as efficiently as local storage. 📖 **Document**: [Quick Start Guide](https://juicefs.com/docs/community/quick_start_guide) ## Highlighted Features 1. **Fully POSIX-compatible**: Use as a local file system, seamlessly docking with existing applications without breaking business workflow. 2. **Fully Hadoop-compatible**: JuiceFS' [Hadoop Java SDK](https://juicefs.com/docs/community/hadoop_java_sdk) is compatible with Hadoop 2.x and Hadoop 3.x as well as a variety of components in the Hadoop ecosystems. 3. **S3-compatible**: JuiceFS' [S3 Gateway](https://juicefs.com/docs/community/s3_gateway) provides an S3-compatible interface. 4. **Cloud Native**: A [Kubernetes CSI Driver](https://juicefs.com/docs/community/how_to_use_on_kubernetes) is provided for easily using JuiceFS in Kubernetes. 5. **Shareable**: JuiceFS is a shared file storage that can be read and written by thousands of clients. 6. **Strong Consistency**: The confirmed modification will be immediately visible on all the servers mounted with the same file system. 7. **Outstanding Performance**: The latency can be as low as a few milliseconds, and the throughput can be expanded nearly unlimitedly _(depending on the size of the Object Storage)_. [Test results](https://juicefs.com/docs/community/benchmark) 8. **Data Encryption**: Supports data encryption in transit and at rest (please refer to [the guide](https://juicefs.com/docs/community/security/encrypt) for more information). 9. **Global File Locks**: JuiceFS supports both BSD locks (flock) and POSIX record locks (fcntl). 10. **Data Compression**: JuiceFS supports [LZ4](https://lz4.github.io/lz4) or [Zstandard](https://facebook.github.io/zstd) to compress all your data. --- [Architecture](#architecture) | [Getting Started](#getting-started) | [Advanced Topics](#advanced-topics) | [POSIX Compatibility](#posix-compatibility) | [Performance Benchmark](#performance-benchmark) | [Supported Object Storage](#supported-object-storage) | [Who is using](#who-is-using) | [Roadmap](#roadmap) | [Reporting Issues](#reporting-issues) | [Contributing](#contributing) | [Community](#community) | [Usage Tracking](#usage-tracking) | [License](#license) | [Credits](#credits) | [FAQ](#faq) --- ## Architecture JuiceFS consists of three parts: 1. **JuiceFS Client**: Coordinates Object Storage and metadata storage engine as well as implementation of file system interfaces such as POSIX, Hadoop, Kubernetes, and S3 gateway. 2. **Data Storage**: Stores data, with supports of a variety of data storage media, e.g., local disk, public or private cloud Object Storage, and HDFS. 3. **Metadata Engine**: Stores the corresponding metadata that contains information of file name, file size, permission group, creation and modification time and directory structure, etc., with supports of different metadata engines, e.g., Redis, MySQL, SQLite and TiKV. JuiceFS can store the metadata of file system on different metadata engines, like Redis, which is a fast, open-source, in-memory key-value data storage, particularly suitable for storing metadata; meanwhile, all the data will be stored in Object Storage through JuiceFS client. [Learn more](https://juicefs.com/docs/community/architecture) Each file stored in JuiceFS is split into **"Chunk"** s at a fixed size with the default upper limit of 64 MiB. Each Chunk is composed of one or more **"Slice"**(s), and the length of the slice varies depending on how the file is written. Each slice is composed of size-fixed **"Block"** s, which are 4 MiB by default. These blocks will be stored in Object Storage in the end; at the same time, the metadata information of the file and its Chunks, Slices, and Blocks will be stored in metadata engines via JuiceFS. [Learn more](https://juicefs.com/docs/community/architecture/#how-juicefs-store-files) When using JuiceFS, files will eventually be split into Chunks, Slices and Blocks and stored in Object Storage. Therefore, the source files stored in JuiceFS cannot be found in the file browser of the Object Storage platform; instead, there are only a chunks directory and a bunch of digitally numbered directories and files in the bucket. Don't panic! This is just the secret of the high-performance operation of JuiceFS! ## Getting Started Before you begin, make sure you have: 1. One supported metadata engine, see [How to Set Up Metadata Engine](https://juicefs.com/docs/community/databases_for_metadata) 2. One supported Object Storage for storing data blocks, see [Supported Object Storage](https://juicefs.com/docs/community/how_to_setup_object_storage) 3. [JuiceFS Client](https://juicefs.com/docs/community/installation) downloaded and installed Please refer to [Quick Start Guide](https://juicefs.com/docs/community/quick_start_guide) to start using JuiceFS right away! ### Command Reference Check out all the command line options in [command reference](https://juicefs.com/docs/community/command_reference). ### Containers JuiceFS can be used as a persistent volume for Docker and Podman, please check [here](https://juicefs.com/docs/community/juicefs_on_docker) for details. ### Kubernetes It is also very easy to use JuiceFS on Kubernetes. Please find more information [here](https://juicefs.com/docs/community/how_to_use_on_kubernetes). ### Hadoop Java SDK If you wanna use JuiceFS in Hadoop, check [Hadoop Java SDK](https://juicefs.com/docs/community/hadoop_java_sdk). ## Advanced Topics - [Redis Best Practices](https://juicefs.com/docs/community/redis_best_practices) - [How to Setup Object Storage](https://juicefs.com/docs/community/how_to_setup_object_storage) - [Cache](https://juicefs.com/docs/community/cache) - [Fault Diagnosis and Analysis](https://juicefs.com/docs/community/fault_diagnosis_and_analysis) - [FUSE Mount Options](https://juicefs.com/docs/community/fuse_mount_options) - [Using JuiceFS on Windows](https://juicefs.com/docs/community/installation#windows) - [S3 Gateway](https://juicefs.com/docs/community/s3_gateway) Please refer to [JuiceFS Document Center](https://juicefs.com/docs/community/introduction) for more information. ## POSIX Compatibility JuiceFS has passed all of the compatibility tests (8813 in total) in the latest [pjdfstest](https://github.com/pjd/pjdfstest) . ``` All tests successful. Test Summary Report ------------------- /root/soft/pjdfstest/tests/chown/00.t (Wstat: 0 Tests: 1323 Failed: 0) TODO passed: 693, 697, 708-709, 714-715, 729, 733 Files=235, Tests=8813, 233 wallclock secs ( 2.77 usr 0.38 sys + 2.57 cusr 3.93 csys = 9.65 CPU) Result: PASS ``` Aside from the POSIX features covered by pjdfstest, JuiceFS also provides: - **Close-to-open consistency**. Once a file is written _and_ closed, it is guaranteed to view the written data in the following opens and reads from any client. Within the same mount point, all the written data can be read immediately. - Rename and all other metadata operations are atomic, which are guaranteed by supported metadata engine transaction. - Opened files remain accessible after unlink from same mount point. - Mmap (tested with FSx). - Fallocate with punch hole support. - Extended attributes (xattr). - BSD locks (flock). - POSIX record locks (fcntl). ## Performance Benchmark ### Basic benchmark JuiceFS provides a subcommand that can run a few basic benchmarks to help you understand how it works in your environment: ### Throughput A sequential read/write benchmark has also been performed on JuiceFS, [EFS](https://aws.amazon.com/efs) and [S3FS](https://github.com/s3fs-fuse/s3fs-fuse) by [fio](https://github.com/axboe/fio). Above result figure shows that JuiceFS can provide 10X more throughput than the other two (see [more details](https://juicefs.com/docs/community/fio)). ### Metadata IOPS A simple mdtest benchmark has been performed on JuiceFS, [EFS](https://aws.amazon.com/efs) and [S3FS](https://github.com/s3fs-fuse/s3fs-fuse) by [mdtest](https://github.com/hpc/ior). The result shows that JuiceFS can provide significantly more metadata IOPS than the other two (see [more details](https://juicefs.com/docs/community/mdtest)). ### Analyze performance See [Real-Time Performance Monitoring](https://juicefs.com/docs/community/fault_diagnosis_and_analysis#performance-monitor) if you encountered performance issues. ## Supported Object Storage - Amazon S3 _(and other S3 compatible Object Storage services)_ - Google Cloud Storage - Azure Blob Storage - Alibaba Cloud Object Storage Service (OSS) - Tencent Cloud Object Storage (COS) - Qiniu Cloud Object Storage (Kodo) - QingStor Object Storage - Ceph RGW - MinIO - Local disk - Redis - ... JuiceFS supports numerous Object Storage services. [Learn more](https://juicefs.com/docs/community/how_to_setup_object_storage#supported-object-storage). ## Who is using JuiceFS is production ready and used by thousands of machines in production. A list of users has been assembled and documented [here](https://juicefs.com/docs/community/adopters). In addition JuiceFS has several collaborative projects that integrate with other open source projects, which we have documented [here](https://juicefs.com/docs/community/integrations). If you are also using JuiceFS, please feel free to let us know, and you are welcome to share your specific experience with everyone. The storage format is stable, and will be supported by all future releases. ## Roadmap - Gateway Optimization - Resumable Sync - Read-ahead Optimization - Optimization for Large-scale Scenarios - Snapshots ## Reporting Issues We use [GitHub Issues](https://github.com/juicedata/juicefs/issues) to track community reported issues. You can also [contact](#community) the community for any questions. ## Contributing Thank you for your contribution! Please refer to the [JuiceFS Contributing Guide](https://juicefs.com/docs/community/development/contributing_guide) for more information. ## Community Welcome to join the [Discussions](https://github.com/juicedata/juicefs/discussions) and the [Discord channel](https://go.juicefs.com/discord) to connect with JuiceFS team members and other users. ## Usage Tracking JuiceFS collects **anonymous** usage data by default to help us better understand how the community is using JuiceFS. Only core metrics (e.g. version number) will be reported, and user data and any other sensitive data will not be included. The related code can be viewed [here](pkg/usage/usage.go). You could also disable reporting easily by command line option `--no-usage-report`: ```bash juicefs mount --no-usage-report ``` ## License JuiceFS is open-sourced under Apache License 2.0, see [LICENSE](LICENSE). ## Credits The design of JuiceFS was inspired by [Google File System](https://research.google/pubs/pub51), [HDFS](https://hadoop.apache.org) and [MooseFS](https://moosefs.com). Thanks for their great work! ## FAQ ### Why doesn't JuiceFS support XXX Object Storage? JuiceFS supports many Object Storage services. Please check out [this list](https://juicefs.com/docs/community/how_to_setup_object_storage#supported-object-storage) first. If the Object Storage you want to use is compatible with S3, you could treat it as S3. Otherwise, try reporting any issue. ### Can I use Redis Cluster as metadata engine? Yes. Since [v1.0.0 Beta3](https://github.com/juicedata/juicefs/releases/tag/v1.0.0-beta3) JuiceFS supports the use of [Redis Cluster](https://redis.io/docs/manual/scaling) as the metadata engine, but it should be noted that Redis Cluster requires that the keys of all operations in a transaction must be in the same hash slot, so a JuiceFS file system can only use one hash slot. See ["Redis Best Practices"](https://juicefs.com/docs/community/redis_best_practices) for more information. ### What's the difference between JuiceFS and XXX? See ["Comparison with Others"](https://juicefs.com/docs/community/comparison/juicefs_vs_alluxio) for more information. For more FAQs, please see the [full list](https://juicefs.com/docs/community/faq). ## Stargazers over time [](https://star-history.com/#juicedata/juicefs&Date) --- ## File: docs/en/development/contributing_guide.md --- title: Contributing Guide sidebar_position: 1 --- ## Guidelines - Before starting work on a feature or bug fix, search GitHub or reach out to us via GitHub or Discord, make sure no one else is already working on it and we'll ask you to open a GitHub issue if necessary. - Before contributing, use the GitHub issue to discuss the feature and reach an agreement with the core developers. - For major feature updates, write a design document to help the community understand your motivation and solution. - Find issues with the label ["kind/good-first-issue"](https://github.com/juicedata/juicefs/labels/kind%2Fgood-first-issue) or ["kind/help-wanted"](https://github.com/juicedata/juicefs/labels/kind%2Fhelp-wanted). Read [internals](./internals.md) for important data structure references. ## Coding style - We're following ["Effective Go"](https://go.dev/doc/effective_go) and ["Go Code Review Comments"](https://github.com/golang/go/wiki/CodeReviewComments). - Use `go fmt` to format your code before committing. You can find information in editor support for Go tools in ["IDEs and Plugins for Go"](https://github.com/golang/go/wiki/IDEsAndTextEditorPlugins). - Every new source file must begin with a license header. - Install [pre-commit](https://pre-commit.com) and use it to set up a pre-commit hook for static analysis. Just run `pre-commit install` in the root of the repo. ## Sign the CLA Before you can contribute to JuiceFS, you will need to sign the [Contributor License Agreement](https://cla-assistant.io/juicedata/juicefs). There're a CLA assistant to guide you when you first time submit a pull request. ## What is a good PR - Presence of unit tests - Adherence to the coding style - Adequate in-line comments - Explanatory commit message ## Contribution flow 1. Create a topic branch from where to base the contribution. This is usually `main`. 1. Make commits of logical units. 1. Make sure commit messages are in the proper format. 1. Push changes in a topic branch to a personal fork of the repository. 1. Submit a pull request to [`juicedata/juicefs`](https://github.com/juicedata/juicefs/compare). The PR should link to one issue which either created by you or others. 1. The PR must receive approval from at least one maintainer before it be merged. --- ## File: docs/en/development/internals.md --- title: Internals sidebar_position: 4 slug: /internals --- This article introduces implementation details of JuiceFS, use this as a reference if you'd like to contribute. The content below is based on JuiceFS v1.0.0, metadata version v1. Before digging into source code, you should read [Data Processing Workflow](../introduction/io_processing.md). ## Keyword Definition High level concepts: - File system: i.e. JuiceFS Volume, represents a separate namespace. Files can be moved freely within the same file system, while data copies are required between different file systems. - Metadata engine: A supported database instance of your choice, that stores and manages file system metadata. There are three categories of metadata engines currently supported by JuiceFS. - Redis: Redis and various protocol-compatible services - SQL: MySQL, PostgreSQL, SQLite, etc. - TKV: TiKV, BadgerDB, etc. - Datastore: Object storage service that stores and manages file system data, such as Amazon S3, Aliyun OSS, etc. It can also be served by other storage systems that are compatible with object storage semantics, such as local file systems, Ceph RADOS, TiKV, etc. - Client: can be in various forms, such as mount process, S3 gateway, WebDAV server, Java SDK, etc. - File: refers to all types of files in general in this documentation, including regular files, directory files, link files, device files, etc. - Directory: is a special kind of file used to organize the tree structure, and its contents are an index to a set of other files. Low level concepts (learn more at [Data Processing Workflow](../introduction/io_processing.md)): - Chunk: Logical concept, file is split into 64MiB chunks, allowing fast lookups during file reads; - Slice: Logical concept, basic unit for file writes. Block's purpose is to improve read speed, and slice exists to improve file edits and random writes. All file writes are assigned a new or existing slice, and when file is read, what application sees is the consolidated view of all slices. - Block: A chunk contains one or more blocks (4MiB by default), block is the basic storage unit in object storage. JuiceFS Client reads multiple blocks concurrently which greatly improves read performance. Apart from this, block is also the basic storage unit on disk cache, so this design improves cache eviction efficiency. Apart from this, block is immutable, all file edits is achieved through new blocks: after file edit, new blocks are uploaded to object storage, and new slices are appended to the slice list in the corresponding file metadata; ## Learn source code {#source-code-structure} Assuming you're already familiar with Go, as well as [JuiceFS architecture](https://juicefs.com/docs/community/architecture), this is the overall code structure: * [`cmd`](https://github.com/juicedata/juicefs/tree/main/cmd) is the top-level entrance, all JuiceFS functionalities is rooted here, e.g. the `juicefs format` command resides in `cmd/format.go`； * [`pkg`](https://github.com/juicedata/juicefs/tree/main/pkg) is actual implementation: * `pkg/fuse/fuse.go` provides abstract FUSE API; * `pkg/vfs` contains actual FUSE implementation, Metadata requests are handled in `pkg/meta`, read requests are handled in `pkg/vfs/reader.go` and write requests are handled by `pkg/vfs/writer.go`; * `pkg/meta` directory is the implementation of all metadata engines, where: * `pkg/meta/interface.go` is the interface definition for all types of metadata engines * `pkg/meta/redis.go` is the interface implementation of Redis database * `pkg/meta/sql.go` is the interface definition and general interface implementation of relational database, and the implementation of specific databases is in a separate file (for example, the implementation of MySQL is in `pkg/meta/sql_mysql.go`) * `pkg/meta/tkv.go` is the interface definition and general interface implementation of the KV database, and the implementation of a specific database is in a separate file (for example, the implementation of TiKV is in `pkg/meta/tkv_tikv.go`) * `pkg/object` contains all object storage integration code; * [`sdk/java`](https://github.com/juicedata/juicefs/tree/main/sdk/java) is the Hadoop Java SDK, it uses `sdk/java/libjfs` through JNI. ## FUSE interface implementation {#fuse-interface-implementation} JuiceFS implements a userspace file system based on [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) (Filesystem in Userspace), and the implementation library [`libfuse`](https://github.com/libfuse/libfuse) provides two APIs: high-level API and low-level API, where the high-level API is based on file name and path, and the low-level API is based on inode. JuiceFS is implemented based on low-level API (in fact JuiceFS does not depend on `libfuse`, but [`go-fuse`](https://github.com/hanwen/go-fuse)), because this is the same set of APIs used by kernel VFS when interacting with FUSE. If JuiceFS were to use high level API, it'll have to implement the VFS tree within `libfuse`, and then expose path based API. This method works better for systems that already expose path based APIs (e.g. HDFS, S3). If metadata itself implements file / directory tree based on inode, the inode → path → inode conversions will have an impact on performance (this is the reason why FUSE API for HDFS doesn't perform well). JuiceFS Metadata directly implements file tree and API based on inode, so naturally it uses FUSE low level API. ## Metadata Structure File systems are usually organized in a tree structure, where nodes represent files and edges represent directory containment relationships. There are more than ten metadata structures in JuiceFS. Most of them are used to maintain the organization of file tree and properties of individual nodes, while the rest are used to manage system configuration, client sessions, asynchronous tasks, etc. All metadata structures are described below. ### General Structure #### Setting It is created when the `juicefs format` command is executed, and some of its fields can be modified later by the `juicefs config` command. The structure is specified as follows. ```go type Format struct { Name string UUID string Storage string Bucket string AccessKey string `json:",omitempty"` SecretKey string `json:",omitempty"` SessionToken string `json:",omitempty"` BlockSize int Compression string `json:",omitempty"` Shards int `json:",omitempty"` HashPrefix bool `json:",omitempty"` Capacity uint64 `json:",omitempty"` Inodes uint64 `json:",omitempty"` EncryptKey string `json:",omitempty"` KeyEncrypted bool `json:",omitempty"` TrashDays int `json:",omitempty"` MetaVersion int `json:",omitempty"` MinClientVersion string `json:",omitempty"` MaxClientVersion string `json:",omitempty"` EnableACL bool } ``` - Name: name of the file system, specified by the user when formatting - UUID: unique ID of the file system, automatically generated by the system when formatting - Storage: short name of the object storage used to store data, such as `s3`, `oss`, etc. - Bucket: the bucket path of the object storage - AccessKey: access key used to access the object storage - SecretKey: secret key used to access the object storage - SessionToken: session token used to access the object storage, as some object storage supports the use of temporary token to obtain permission for a limited time - BlockSize: size of the data block when splitting the file (the default is 4 MiB) - Compression: compression algorithm that is executed before uploading data blocks to the object storage (the default is no compression) - Shards: number of buckets in the object storage, only one bucket by default; when Shards > 1, data objects will be randomly hashed into Shards buckets - HashPrefix: whether to set a hash prefix for the object name, false by default - Capacity: quota limit for the total capacity of the file system - Inodes: quota limit for the total number of files in the file system - EncryptKey: the encrypted private key of the data object, which can be used only if the data encryption function is enabled - KeyEncrypted: whether the saved key is encrypted or not, by default the SecretKey, EncryptKey and SessionToken will be encrypted - TrashDays: number of days the deleted files are kept in trash, the default is 1 day - MetaVersion: the version of the metadata structure, currently V1 (V0 and V1 are the same) - MinClientVersion: the minimum client version allowed to connect, clients earlier than this version will be denied - MaxClientVersion: the maximum client version allowed to connect - EnableACL: enable ACL or not This structure is serialized into JSON format and stored in the metadata engine. #### Counter Maintains the value of each counter in the system and the start timestamps of some background tasks, specifically - usedSpace: used capacity of the file system - totalInodes: number of used files in the file system - nextInode: the next available inode number (in Redis, the maximum inode number currently in use) - nextChunk: the next available sliceId (in Redis, the largest sliceId currently in use) - nextSession: the maximum SID (sessionID) currently in use - nextTrash: the maximum trash inode number currently in use - nextCleanupSlices: timestamp of the last check on the cleanup of residual slices - lastCleanupSessions: timestamp of the last check on the cleanup of residual stale sessions - lastCleanupFiles: timestamp of the last check on the cleanup of residual files - lastCleanupTrash: timestamp of the last check on the cleanup of trash #### Session Records the session IDs of clients connected to this file system and their timeouts. Each client sends a heartbeat message to update the timeout, and those who have not updated for a long time will be automatically cleaned up by other clients. :::tip Read-only clients cannot write to the metadata engine, so their sessions **will not** be recorded. ::: #### SessionInfo Records specific metadata of the client session so that it can be viewed with the `juicefs status` command. This is specified as ```go type SessionInfo struct { Version string // JuiceFS version HostName string // Host name MountPoint string // path to mount point. S3 gateway and WebDAV server are "s3gateway" and "webdav" respectively ProcessID int // Process ID } ``` This structure is serialized into JSON format and stored in the metadata engine. #### Node Records attribute information of each file, as follows ```go type Attr struct { Flags uint8 // reserved flags Typ uint8 // type of a node Mode uint16 // permission mode Uid uint32 // owner id Gid uint32 // group id of owner Rdev uint32 // device number Atime int64 // last access time Mtime int64 // last modified time Ctime int64 // last change time for meta Atimensec uint32 // nanosecond part of atime Mtimensec uint32 // nanosecond part of mtime Ctimensec uint32 // nanosecond part of ctime Nlink uint32 // number of links (sub-directories or hardlinks) Length uint64 // length of regular file Parent Ino // inode of parent; 0 means tracked by parentKey (for hardlinks) Full bool // the attributes are completed or not KeepCache bool // whether to keep the cached page or not AccessACL uint32 // access ACL id (identical ACL rules share the same access ACL ID.) DefaultACL uint32 // default ACL id (default ACL and the access ACL share the same cache and store) } ``` There are a few fields that need clarification. - Atime/Atimensec: See [`--atime-mode`](../reference/command_reference.mdx#mount-metadata-options) - Nlink - Directory file: initial value is 2 ('.' and '..'), add 1 for each subdirectory - Other files: initial value is 1, add 1 for each hard link created - Length - Directory file: fixed at 4096 - Soft link (symbolic link) file: the string length of the path to which the link points - Other files: the length of the actual content of the file This structure is usually encoded in binary format and stored in the metadata engine. #### Edges Records information on each edge in the file tree, as follows ``` parentInode, name -> type, inode ``` where parentInode is the inode number of the parent directory, and the others are the name, type, and inode number of the child files, respectively. #### LinkParent Records the parent directory of some files. The parent directory of most files is recorded in the Parent field of the attribute; however, for files that have been created with hard links, there may be more than one parent directory, so the Parent field is set to 0, and all parent inodes are recorded independently, as follows ``` inode -> parentInode, links ``` where links is the count of the parentInode, because multiple hard links can be created in the same directory, and these hard links share one inode. #### Chunk Records information on each Chunk, as follows ``` inode, index -> []Slices ``` where inode is the inode number of the file to which the Chunk belongs, and index is the number of all Chunks in the file, starting from 0. The Chunk value is an array of Slices. Each Slice represents a piece of data written by the client, and is appended to this array in the order of writing time. When there is an overlap between different Slices, the later Slice is used. ```go type Slice struct { Pos uint32 // offset of the Slice in the Chunk ID uint64 // ID of the Slice, globally unique Size uint32 // size of the Slice Off uint32 // offset of valid data in this Slice Len uint32 // size of valid data in this Slice } ``` This structure is encoded and saved in binary format, taking up 24 bytes. #### SliceRef {#sliceref} Records the reference count of a Slice, as follows ``` sliceId, size -> refs ``` Since the reference count of most Slices is 1, to reduce the number of related entries in the database, the actual value minus 1 is used as the stored count value in Redis and TKV. In this way, most of the Slices have a refs value of 0, and there is no need to create related entries in the database. #### Symlink Records the location of the softlink file, as follows ``` inode -> target ``` #### Xattr Records extended attributes (Key-Value pairs) of a file, as follows ``` inode, key -> value ``` #### Flock Records BSD locks (flock) of a file, specifically. ``` inode, sid, owner -> ltype ``` where `sid` is the client session ID, `owner` is a string of numbers, usually associated with a process, and `ltype` is the lock type, which can be 'R' or 'W'. #### Plock Record POSIX record locks (fcntl) of a file, specifically ``` inode, sid, owner -> []plockRecord ``` Here plock is a more fine-grained lock that can only lock a certain segment of the file. ```go type plockRecord struct { ltype uint32 // lock type pid uint32 // process ID start uint64 // start position of the lock end uint64 // end position of the lock } ``` This structure is encoded and stored in binary format, taking up 24 bytes. #### DelFiles Records the list of files to be cleaned. It is needed as data cleanup of files is an asynchronous and potentially time-consuming operation that can be interrupted by other factors. ``` inode, length -> expire ``` where length is the length of the file and expire is the time when the file was deleted. #### DelSlices Records delayed deleted Slices. When the Trash feature is enabled, old Slices deleted by the Slice Compaction will be kept for the same amount of time as the Trash configuration, to be available for data recovery if necessary. ``` sliceId, deleted -> []slice ``` where sliceId is the ID of the new slice after compaction, deleted is the timestamp of the compaction, and the mapped value is the list of all old slices that were compacted. Each slice only encodes its ID and size. ```go type slice struct { ID uint64 Size uint32 } ``` This structure is encoded and stored in binary format, taking up 12 bytes. #### Sustained Records the list of files that need to be kept temporarily during the session. If a file is still open when it is deleted, the data cannot be cleaned up immediately, but needs to be held temporarily until the file is closed. ``` sid -> []inode ``` where `sid` is the session ID and the mapped value is the list of temporarily undeleted file inodes. ### Redis The common format of keys in Redis is `${prefix}${JFSKey}`, where - In standalone mode the prefix is an empty string, while in cluster mode it is a database number enclosed in curly braces, e.g. "{10}" - JFSKey is the Key of different data structures in JuiceFS, which are listed in the subsequent subsections In Redis Keys, integers (including inode numbers) are represented as decimal strings if not otherwise specified. #### Setting {#redis-setting} - Key: `setting` - Value Type: String - Value: file system formatting information in JSON format #### Counter - Key: counter name - Value Type: String - Value: value of the counter, which is actually an integer #### Session - Key: `allSessions` - Value Type: Sorted Set - Value: all non-read-only sessions connected to this file system. In Set, - Member: session ID - Score: timeout point of this session #### SessionInfo - Key: `sessionInfos` - Value Type: Hash - Value: basic meta-information on all non-read-only sessions. In Hash, - Key: session ID - Value: session information in JSON format #### Node {#redis-node} - Key: `i${inode}` - Value Type: String - Value: binary encoded file attribute #### Edge {#redis-edge} - Key: `d${inode}` - Value Type: Hash - Value: all directory entries in this directory. In Hash, - Key: file name - Value: binary encoded file type and inode number #### LinkParent - Key: `p${inode}` - Value Type: Hash - Value: all parent inodes of this file. in Hash. - Key: parent inode - Value: count of this parent inode #### Chunk {#redis-chunk} - Key: `c${inode}_${index}` - Value Type: List - Value: list of Slices, each Slice is binary encoded with 24 bytes #### SliceRef - Key: `sliceRef` - Value Type: Hash - Value: the count value of all Slices to be recorded. In Hash, - Key: `k${sliceId}_${size}` - Value: reference count of this Slice minus 1 (if the reference count is 1, the corresponding entry is generally not created) #### Symlink - Key: `s${inode}` - Value Type: String - Value: path that the symbolic link points to #### Xattr - Key: `x${inode}` - Value Type: Hash - Value: all extended attributes of this file. In Hash, - Key: name of the extended attribute - Value: value of the extended attribute #### Flock - Key: `lockf${inode}` - Value Type: Hash - Value: all flocks of this file. In Hash, - Key: `${sid}_${owner}`, owner in hexadecimal - Value: lock type, can be 'R' or 'W' #### Plock {#redis-plock} - Key: `lockp${inode}` - Value Type: Hash - Value: all plocks of this file. In Hash, - Key: `${sid}_${owner}`, owner in hexadecimal - Value: array of bytes, where every 24 bytes corresponds to a [plockRecord](#plock) #### DelFiles - Key：`delfiles` - Value Type: Sorted Set - Value: list of all files to be cleaned. In Set, - Member: `${inode}:${length}` - Score: the timestamp when this file was added to the set #### DelSlices {#redis-delslices} - Key: `delSlices` - Value Type: Hash - Value: all Slices to be cleaned. In Hash, - Key: `${sliceId}_${deleted}` - Value: array of bytes, where every 12 bytes corresponds to a [slice](#delslices) #### Sustained - Key: `session${sid}` - Value Type: List - Value: list of files temporarily reserved in this session. In List, - Member: inode number of the file ### SQL Metadata is stored in different tables by type, and each table is named with `jfs_` followed by its specific structure name to form the table name, e.g. `jfs_node`. Some tables use `Id` with the `bigserial` type as primary keys to ensure that each table has a primary key, and the `Id` columns do not contain actual information. #### Setting {#sql-setting} ```go type setting struct { Name string `xorm:"pk"` Value string `xorm:"varchar(4096) notnull"` } ``` There is only one entry in this table with "format" as Name and file system formatting information in JSON as Value. #### Counter ```go type counter struct { Name string `xorm:"pk"` Value int64 `xorm:"notnull"` } ``` #### Session ```go type session2 struct { Sid uint64 `xorm:"pk"` Expire int64 `xorm:"notnull"` Info []byte `xorm:"blob"` } ``` #### SessionInfo There is no separate table for this, but it is recorded in the `Info` column of `session2`. #### Node {#sql-node} ```go type node struct { Inode Ino `xorm:"pk"` Type uint8 `xorm:"notnull"` Flags uint8 `xorm:"notnull"` Mode uint16 `xorm:"notnull"` Uid uint32 `xorm:"notnull"` Gid uint32 `xorm:"notnull"` Atime int64 `xorm:"notnull"` Mtime int64 `xorm:"notnull"` Ctime int64 `xorm:"notnull"` Nlink uint32 `xorm:"notnull"` Length uint64 `xorm:"notnull"` Rdev uint32 Parent Ino AccessACLId uint32 `xorm:"'access_acl_id'"` DefaultACLId uint32 `xorm:"'default_acl_id'"` } ``` Most of the fields are the same as [Attr](#node), but the timestamp precision is lower, i.e., Atime/Mtime/Ctime are in microseconds. #### Edge {#sql-edge} ```go type edge struct { Id int64 `xorm:"pk bigserial"` Parent Ino `xorm:"unique(edge) notnull"` Name []byte `xorm:"unique(edge) varbinary(255) notnull"` Inode Ino `xorm:"index notnull"` Type uint8 `xorm:"notnull"` } ``` #### LinkParent There is no separate table for this. All `Parent`s are found based on the `Inode` index in `edge`. #### Chunk {#sql-chunk} ```go type chunk struct { Id int64 `xorm:"pk bigserial"` Inode Ino `xorm:"unique(chunk) notnull"` Indx uint32 `xorm:"unique(chunk) notnull"` Slices []byte `xorm:"blob notnull"` } ``` Slices are an array of bytes, and each [Slice](#chunk) corresponds to 24 bytes. #### SliceRef ```go type sliceRef struct { Id uint64 `xorm:"pk chunkid"` Size uint32 `xorm:"notnull"` Refs int `xorm:"notnull"` } ``` #### Symlink ```go type symlink struct { Inode Ino `xorm:"pk"` Target []byte `xorm:"varbinary(4096) notnull"` } ``` #### Xattr ```go type xattr struct { Id int64 `xorm:"pk bigserial"` Inode Ino `xorm:"unique(name) notnull"` Name string `xorm:"unique(name) notnull"` Value []byte `xorm:"blob notnull"` } ``` #### Flock ```go type flock struct { Id int64 `xorm:"pk bigserial"` Inode Ino `xorm:"notnull unique(flock)"` Sid uint64 `xorm:"notnull unique(flock)"` Owner int64 `xorm:"notnull unique(flock)"` Ltype byte `xorm:"notnull"` } ``` #### Plock {#sql-plock} ```go type plock struct { Id int64 `xorm:"pk bigserial"` Inode Ino `xorm:"notnull unique(plock)"` Sid uint64 `xorm:"notnull unique(plock)"` Owner int64 `xorm:"notnull unique(plock)"` Records []byte `xorm:"blob notnull"` } ``` Records is an array of bytes, and each [plockRecord](#plock) corresponds to 24 bytes. #### DelFiles ```go type delfile struct { Inode Ino `xorm:"pk notnull"` Length uint64 `xorm:"notnull"` Expire int64 `xorm:"notnull"` } ``` #### DelSlices {#sql-delslices} ```go type delslices struct { Id uint64 `xorm:"pk chunkid"` Deleted int64 `xorm:"notnull"` Slices []byte `xorm:"blob notnull"` } ``` Slices is an array of bytes, and each [slice](#delslices) corresponds to 12 bytes. #### Sustained ```go type sustained struct { Id int64 `xorm:"pk bigserial"` Sid uint64 `xorm:"unique(sustained) notnull"` Inode Ino `xorm:"unique(sustained) notnull"` } ``` ### TKV The common format of keys in TKV (Transactional Key-Value Database) is `${prefix}${JFSKey}`, where - prefix is used to distinguish between different file systems, usually `${VolumeName}0xFD`, where `0xFD` is used as a special byte to handle cases when there is an inclusion relationship between different file system names. In addition, for databases that are not shareable (e.g. BadgerDB), the empty string is used as prefix. - JFSKey is the JuiceFS Key for different data types, which is listed in the following subsections. In TKV's Keys, all integers are stored in encoded binary form. - inode and counter value occupy 8 bytes and are encoded with **small endian**. - SID, sliceId and timestamp occupy 8 bytes and are encoded with **big endian**. #### Setting {#tkv-setting} ``` setting -> file system formatting information in JSON format ``` #### Counter ``` C${name} -> counter value ``` #### Session ``` SE${sid} -> timestamp ``` #### SessionInfo ``` SI${sid} -> session information in JSON format ``` #### Node {#tkv-node} ``` A${inode}I -> encoded Attr ``` #### Edge {#tkv-edge} ``` A${inode}D${name} -> encoded {type, inode} ``` #### LinkParent ``` A${inode}P${parentInode} -> counter value ``` #### Chunk {#tkv-chunk} ``` A${inode}C${index} -> Slices ``` where index takes up 4 bytes and is encoded with **big endian**. Slices is an array of bytes, one [Slice](#chunk) per 24 bytes. #### SliceRef ``` K${sliceId}${size} -> counter value ``` where size takes up 4 bytes and is encoded with **big endian**. #### Symlink ``` A${inode}S -> target ``` #### Xattr ``` A${inode}X${name} -> xattr value ``` #### Flock ``` F${inode} -> flocks ``` where flocks is an array of bytes, one flock per 17 bytes. ```go type flock struct { sid uint64 owner uint64 ltype uint8 } ``` #### Plock {#tkv-plock} ``` P${inode} -> plocks ``` where plocks is an array of bytes and the corresponding plock is variable-length. ```go type plock struct { sid uint64 owner uint64 size uint32 records []byte } ``` where size is the length of the records array and every 24 bytes in records corresponds to one [plockRecord](#plock). #### DelFiles ``` D${inode}${length} -> timestamp ``` where length takes up 8 bytes and is encoded with **big endian**. #### DelSlices {#tkv-delslices} ``` L${timestamp}${sliceId} -> slices ``` where slices is an array of bytes, and one [slice](#delslices) corresponds to 12 bytes. #### Sustained ``` SS${sid}${inode} -> 1 ``` Here the Value value is only used as a placeholder. ## File Data Format ### Finding files by path According to the design of [Edge](#edges), only the direct children of each directory are recorded in the metadata engine. When an application provides a path to access a file, JuiceFS needs to look it up level by level. Now suppose the application wants to open the file `/dir1/dir2/testfile`, then it needs to 1. search for the entry with name "dir1" in the Edge structure of the root directory (inode number is fixed to 1) and get its inode number N1 2. search for the entry with the name "dir2" in the Edge structure of N1 and get its inode number N2 3. search for the entry with the name "testfile" in the Edge structure of N2, and get its inode number N3 4. search for the [Node](#node) structure corresponding to N3 to get the attributes of the file Failure in any of the above steps will result in the file pointed to by that path not being found. ### File data splitting From the previous section, we know how to find the file based on its path and get its attributes. The metadata related to the contents of the file can be found based on the inode and size fields in the file properties. Now suppose a file has an inode of 100 and a size of 160 MiB, then the file has `(size-1) / 64 MiB + 1 = 3` Chunks, as follows. ``` File: |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _| Chunk: |<--- Chunk 0 --->|<--- Chunk 1 --->|<-- Chunk 2 -->| ``` In standalone Redis, this means that there are 3 [Chunk Keys](#chunk), i.e.,`c100_0`, `c100_1` and `c100_2`, each corresponding to a list of Slices. These Slices are mainly generated when the data is written and may overwrite each other or may not fill the Chunk completely, so you need to traverse this list of Slices sequentially and reconstruct the latest version of the data distribution before using it, so that 1. the part covered by more than one Slice is based on the last added Slice 2. the part that is not covered by Slice is automatically zeroed, and is represented by sliceId = 0 3. truncate Chunk according to file size Now suppose there are 3 Slices in Chunk 0 ```go Slice{pos: 10M, id: 10, size: 30M, off: 0, len: 30M} Slice{pos: 20M, id: 11, size: 16M, off: 0, len: 16M} Slice{pos: 16M, id: 12, size: 10M, off: 0, len: 10M} ``` It can be illustrated as follows (each '_' denotes 2 MiB) ``` Chunk: |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _| Slice 10: |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _| Slice 11: |_ _ _ _ _ _ _ _| Slice 12: |_ _ _ _ _| New List: |_ _ _ _ _|_ _ _|_ _ _ _ _|_ _ _ _ _|_ _|_ _ _ _ _ _ _ _ _ _ _ _| 0 10 12 11 10 0 ``` The reconstructed new list contains and only contains the latest data distribution for this Chunk as follows ```go Slice{pos: 0, id: 0, size: 10M, off: 0, len: 10M} Slice{pos: 10M, id: 10, size: 30M, off: 0, len: 6M} Slice{pos: 16M, id: 12, size: 10M, off: 0, len: 10M} Slice{pos: 26M, id: 11, size: 16M, off: 6M, len: 10M} Slice{pos: 36M, id: 10, size: 30M, off: 26M, len: 4M} Slice{pos: 40M, id: 0, size: 24M, off: 0, len: 24M} // can be omitted ``` ### Data objects #### Object naming {#object-storage-naming-format} Block is the basic unit for JuiceFS to manage data. Its size is 4 MiB by default, and can be changed only when formatting a file system, within the interval [64 KiB, 16 MiB]. Each Block is an object in the object storage after upload, and is named in the format `${fsname}/chunks/${hash}/${basename}`, where - fsname is the file system name - "chunks" is a fixed string representing the data object of JuiceFS - hash is the hash value calculated from basename, which plays a role in isolation management - basename is the valid name of the object in the format of `${sliceId}_${index}_${size}`, where - sliceId is the ID of the Slice to which the object belongs, and each Slice in JuiceFS has a globally unique ID - index is the index of the object in the Slice it belongs to, by default a Slice can be split into at most 16 Blocks, so its value range is [0, 16) - size is the size of the Block, and by default it takes the value of (0, 4 MiB] Currently there are two hash algorithms, and both use the sliceId in basename as the parameter. Which algorithm will be chosen to use follows the [HashPrefix](#setting) of the file system. ```go func hash(sliceId int) string { if HashPrefix { return fmt.Sprintf("%02X/%d", sliceId%256, sliceId/1000/1000) } return fmt.Sprintf("%d/%d", sliceId/1000/1000, sliceId/1000) } ``` Suppose a file system named `jfstest` is written with a continuous 10 MiB of data and internally given a SliceID of 1 with HashPrefix disabled, then the following three objects will be generated in the object storage. ``` jfstest/chunks/0/0/1_0_4194304 jfstest/chunks/0/0/1_1_4194304 jfstest/chunks/0/0/1_2_2097152 ``` Similarly, now taking the 64 MiB chunk in the previous section as an example, its actual data distribution is as follows ``` 0 ~ 10M: Zero 10 ~ 16M: 10_0_4194304, 10_1_4194304(0 ~ 2M) 16 ~ 26M: 12_0_4194304, 12_1_4194304, 12_2_2097152 26 ~ 36M: 11_1_4194304(2 ~ 4M), 11_2_4194304, 11_3_4194304 36 ~ 40M: 10_6_4194304(2 ~ 4M), 10_7_2097152 40 ~ 64M: Zero ``` According to this, the client can quickly find the data needed for the application. For example, reading 8 MiB data at offset 10 MiB location will involve 3 objects, as follows - Read the entire object from `10_0_4194304`, corresponding to 0 to 4 MiB of the read data - Read 0 to 2 MiB from `10_1_4194304`, corresponding to 4 to 6 MiB of the read data - Read 0 to 2 MiB from `12_0_4194304`, corresponding to 6 to 8 MiB of the read data To facilitate obtaining the list of objects of a certain file, JuiceFS provides the `info` command, e.g. `juicefs info /mnt/jfs/test.tmp`. ```bash objects: +------------+---------------------------------+----------+---------+----------+ | chunkIndex | objectName | size | offset | length | +------------+---------------------------------+----------+---------+----------+ | 0 | | 10485760 | 0 | 10485760 | | 0 | jfstest/chunks/0/0/10_0_4194304 | 4194304 | 0 | 4194304 | | 0 | jfstest/chunks/0/0/10_1_4194304 | 4194304 | 0 | 2097152 | | 0 | jfstest/chunks/0/0/12_0_4194304 | 4194304 | 0 | 4194304 | | 0 | jfstest/chunks/0/0/12_1_4194304 | 4194304 | 0 | 4194304 | | 0 | jfstest/chunks/0/0/12_2_2097152 | 2097152 | 0 | 2097152 | | 0 | jfstest/chunks/0/0/11_1_4194304 | 4194304 | 2097152 | 2097152 | | 0 | jfstest/chunks/0/0/11_2_4194304 | 4194304 | 0 | 4194304 | | 0 | jfstest/chunks/0/0/11_3_4194304 | 4194304 | 0 | 4194304 | | 0 | jfstest/chunks/0/0/10_6_4194304 | 4194304 | 2097152 | 2097152 | | 0 | jfstest/chunks/0/0/10_7_2097152 | 2097152 | 0 | 2097152 | | ... | ... | ... | ... | ... | +------------+---------------------------------+----------+---------+----------+ ``` The empty objectName in the table means a file hole and is read as 0. As you can see, the output is consistent with the previous analysis. It is worth mentioning that the 'size' here is size of the original data in the Block, rather than that of the actual object in object storage. The original data is written directly to object storage by default, so the 'size' is equal to object size. However, when data compression or data encryption is enabled, the size of the actual object will change and may no longer be the same as the 'size'. #### Data compression You can configure the compression algorithm (supporting `lz4` and `zstd`) with the `--compress ` parameter when formatting a file system, so that all data blocks of this file system will be compressed before uploading to object storage. The object name remains the same as default, and the content is the result of the compression algorithm, without any other meta information. Therefore, the compression algorithm in the [file system formatting Information](#setting) is not allowed to be modified, otherwise it will cause the failure of reading existing data. #### Data encryption The RSA private key can be configured to enable [static data encryption](../security/encryption.md) when formatting a file system with the `--encrypt-rsa-key ` parameter, which allows all data blocks of this file system to be encrypted before uploading to the object storage. The object name is still the same as default, while its content becomes a header plus the result of the data encryption algorithm. The header contains a random seed and the symmetric key used for decryption, and the symmetric key itself is encrypted with the RSA private key. Therefore, it is not allowed to modify the RSA private key in the [file system formatting Information](#setting), otherwise reading existing data will fail. :::note If both compression and encryption are enabled, the original data will be compressed and then encrypted before uploading to the object storage. ::: --- ## File: docs/en/deployment/automation.md --- title: Automated Deployment sidebar_position: 7 --- Automated deployment is recommended when JuiceFS Client is to be installed on a large number of hosts. Below examples only demonstrate the mount process, you should [Create a file system](../getting-started/standalone.md#juicefs-format) before getting started. ## Ansible Below is the [Ansible](https://ansible.com) example to install and mount JuiceFS in localhost: ```yaml - hosts: localhost tasks: - set_fact: # Change accordingly meta_url: sqlite3:///tmp/myjfs.db jfs_path: /jfs jfs_pkg: /tmp/juicefs-ce.tar.gz jfs_bin_dir: /usr/local/bin - get_url: # Change download URL accordingly url: https://d.juicefs.com/juicefs/releases/download/v1.0.2/juicefs-1.0.2-linux-amd64.tar.gz dest: "{{jfs_pkg}}" - ansible.builtin.unarchive: src: "{{jfs_pkg}}" dest: "{{jfs_bin_dir}}" include: - juicefs - name: Create symbolic for fstab ansible.builtin.file: src: "{{jfs_bin_dir}}/juicefs" dest: "/sbin/mount.juicefs" state: link - name: Mount JuiceFS and create fstab entry mount: path: "{{jfs_path}}" src: "{{meta_url}}" fstype: juicefs opts: _netdev state: mounted ``` --- ## File: docs/en/deployment/hadoop_java_sdk.md --- title: Use JuiceFS on Hadoop Ecosystem sidebar_position: 3 slug: /hadoop_java_sdk --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; JuiceFS provides [Hadoop-compatible File System](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/introduction.html) by Hadoop Java SDK. Various applications in the Hadoop ecosystem can smoothly use JuiceFS to store data without changing the code. ## Requirements ### 1. Hadoop and related components JuiceFS Hadoop Java SDK is compatible with Hadoop 2.x and Hadoop 3.x. As well as variety of components in Hadoop ecosystem. ### 2. User permissions JuiceFS uses local "User/UID" and "Group/GID" mappings by default, and when used in a distributed environment, to avoid permission issues, please refer to [documentation](../administration/sync_accounts_between_multiple_hosts.md) synchronizes the "User/UID" and "Group/GID" that needs to be used to all Hadoop nodes. It is also possible to define a global user and group file to make all nodes in the cluster share the permission configuration. Please see [here](#other-configurations) for related configurations. ### 3. File system You should first create at least one JuiceFS file system to provide storage for components related to the Hadoop ecosystem through the JuiceFS Java SDK. When deploying the Java SDK, specify the metadata engine address of the created file system in the configuration file. To create a file system, please refer to [our quick start](../getting-started/standalone.md). :::note If you want to use JuiceFS in a distributed environment, when creating a file system, please plan the object storage and database to be used reasonably to ensure that they can be accessed by each node in the cluster. ::: ### 4. Memory Depending on the read and write load of computing tasks (such as Spark executor), JuiceFS Hadoop Java SDK may require an additional 4 * [`juicefs.memory-size`](#io-configurations) off-heap memory to speed up read and write performance. By default, it is recommended to configure at least 1.2GB of off-heap memory for compute tasks. ### 5. Java runtime version JuiceFS Hadoop Java SDK is compiled with JDK 8 by default. If it needs to be used in a higher version of Java runtime (such as Java 17), the following options need to be added to the JVM parameters to allow the use of reflection API: ```shell --add-exports=java.base/sun.nio.ch=ALL-UNNAMED ``` For more information on the above option, please refer to [official documentation](https://docs.oracle.com/en/java/javase/17/migrate/migrating-jdk-8-later-jdk-releases.html#GUID-7BB28E4D-99B3-4078-BDC4-FC24180CE82B). ## Install and compile the client ### Install the pre-compiled client Please refer to the ["Installation"](../getting-started/installation.md#install-the-pre-compiled-client) document to learn how to download the precompiled JuiceFS Hadoop Java SDK. ### Compile the client manually :::note No matter which system environment the client is compiled for, the compiled JAR file has the same name and can only be deployed in the matching system environment. For example, when compiled in Linux, it can only be used in the Linux environment. In addition, since the compiled package depends on glibc, it is recommended to compile with a lower version system to ensure better compatibility. ::: Compilation depends on the following tools: - [Go](https://golang.org) 1.20+ - JDK 8+ - [Maven](https://maven.apache.org) 3.3+ - Git - make - GCC 5.4+ #### Linux and macOS Clone the repository: ```shell git clone https://github.com/juicedata/juicefs.git ``` Enter the directory and compile: ```shell cd juicefs/sdk/java make ``` :::note If Ceph RADOS is used to store data, you need to install `librados-dev` first and [build `libjfs.so`]`. ::: ```shell cd juicefs/sdk/java make ceph ``` After the compilation, you can find the compiled `JAR` file in the `sdk/java/target` directory, including two versions: - Contains third-party dependent packages: `juicefs-hadoop-X.Y.Z.jar` - Does not include third-party dependent packages: `original-juicefs-hadoop-X.Y.Z.jar` It is recommended to use a version that includes third-party dependencies. #### Windows The client used in the Windows environment needs to be obtained through cross-compilation on Linux or macOS. The compilation depends on [mingw-w64](https://www.mingw-w64.org), which needs to be installed first. The steps are the same as compiling on Linux or macOS. For example, on the Ubuntu system, install the `mingw-w64` package first to solve the dependency problem: ```shell sudo apt install mingw-w64 ``` Clone and enter the JuiceFS source code directory, execute the following code to compile: ```shell cd juicefs/sdk/java ``` ```shell make win ``` ## Deploy the client To enable each component of the Hadoop ecosystem to correctly identify JuiceFS, the following configurations are required: 1. Place the compiled JAR file and `$JAVA_HOME/lib/tools.jar` into the `classpath` of the component. The installation paths of common big data platforms and components are shown in the table below. 2. Put JuiceFS configurations into the configuration file of each Hadoop ecosystem component (usually `core-site.xml`), see [Client Configurations](#client-configurations) for details. It is recommended to place the JAR file in a fixed location, and the other locations are called it through symbolic links. ### Big Data Platforms | Name | Installing Paths | |-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | CDH | `/opt/cloudera/parcels/CDH/lib/hadoop/lib``/opt/cloudera/parcels/CDH/spark/jars``/var/lib/impala` | | HDP | `/usr/hdp/current/hadoop-client/lib``/usr/hdp/current/hive-client/auxlib``/usr/hdp/current/spark2-client/jars` | | Amazon EMR | `/usr/lib/hadoop/lib``/usr/lib/spark/jars``/usr/lib/hive/auxlib` | | Alibaba Cloud EMR | `/opt/apps/ecm/service/hadoop/*/package/hadoop*/share/hadoop/common/lib``/opt/apps/ecm/service/spark/*/package/spark*/jars``/opt/apps/ecm/service/presto/*/package/presto*/plugin/hive-hadoop2``/opt/apps/ecm/service/hive/*/package/apache-hive*/lib``/opt/apps/ecm/service/impala/*/package/impala*/lib` | | Tencent Cloud EMR | `/usr/local/service/hadoop/share/hadoop/common/lib``/usr/local/service/presto/plugin/hive-hadoop2``/usr/local/service/spark/jars``/usr/local/service/hive/auxlib` | | UCloud UHadoop | `/home/hadoop/share/hadoop/common/lib``/home/hadoop/hive/auxlib``/home/hadoop/spark/jars``/home/hadoop/presto/plugin/hive-hadoop2` | | Baidu Cloud EMR | `/opt/bmr/hadoop/share/hadoop/common/lib``/opt/bmr/hive/auxlib``/opt/bmr/spark2/jars` | ### Community Components | Name | Installing Paths | |-----------|-----------------------------------------------------------------------------------------| | Hadoop | `${HADOOP_HOME}/share/hadoop/common/lib/`, `${HADOOP_HOME}/share/hadoop/mapreduce/lib/` | | Spark | `${SPARK_HOME}/jars` | | Presto | `${PRESTO_HOME}/plugin/hive-hadoop2` | | Trino | `${TRINO_HOME}/plugin/hive` | | Flink | `${FLINK_HOME}/lib` | | StarRocks | `${StarRocks_HOME}/fe/lib/`, `${StarRocks_HOME}/be/lib/hadoop/common/lib` | ### Client Configurations Please refer to the following table to set the relevant parameters of the JuiceFS file system and write it into the configuration file, which is generally `core-site.xml`. #### Core Configurations | Configuration | Default Value | Description | |----------------------------------|------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `fs.jfs.impl` | `io.juicefs.JuiceFileSystem` | Specify the storage implementation to be used. By default, `jfs://` scheme is used. If you want to use different scheme (e.g. `cfs://`), just modify it to `fs.cfs.impl`. No matter what scheme you use, it is always access the data in JuiceFS. | | `fs.AbstractFileSystem.jfs.impl` | `io.juicefs.JuiceFS` | Specify the storage implementation to be used. By default, `jfs://` scheme is used. If you want to use different scheme (e.g. `cfs://`), just modify it to `fs.AbstractFileSystem.cfs.impl`. No matter what scheme you use, it is always access the data in JuiceFS. | | `juicefs.meta` | | Specify the metadata engine address of the pre-created JuiceFS file system. You can configure multiple file systems for the client at the same time through the format of `juicefs.{vol_name}.meta`. Refer to ["Multiple file systems configuration"](#multiple-file-systems-configuration). | #### Cache Configurations | Configuration | Default Value | Description | |------------------------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `juicefs.cache-dir` | `memory` | Directory paths of local cache. Use colon to separate multiple paths. Also support wildcard in path. **It's recommended create these directories manually and set `0777` permission so that different applications could share the cache data.** If not specified, default to process memory. | | `juicefs.cache-size` | 100 | Maximum size of local cache in MiB. Default size is small because Hadoop SDK uses memory as default cache location. It's the total size when set multiple cache directories. | | `juicefs.cache-full-block` | `true` | Whether cache every read blocks, `false` means only cache random/small read blocks. | | `juicefs.free-space` | 0.1 | Min free space ratio of cache directory | | `juicefs.open-cache` | 0 | Open files cache timeout in seconds (0 means disable this feature) | | `juicefs.attr-cache` | 0 | Expire of attributes cache in seconds | | `juicefs.entry-cache` | 0 | Expire of file entry cache in seconds | | `juicefs.dir-entry-cache` | 0 | Expire of directory entry cache in seconds | | `juicefs.discover-nodes-url` | | Specify the node discovery API, the node list will be refreshed every 10 minutes.
  • YARN: `yarn`
  • Spark Standalone: `http://spark-master:web-ui-port/json/`
  • Spark ThriftServer: `http://thrift-server:4040/api/v1/applications/`
  • Presto: `http://coordinator:discovery-uri-port/v1/service/presto/`
  • File system: `jfs://{VOLUME}/etc/nodes`, you need to create this file manually, and write the hostname of the node into this file line by line
    • | #### I/O Configurations | Configuration | Default Value | Description | |--------------------------|---------------|-------------------------------------------------| | `juicefs.max-uploads` | 20 | The max number of connections to upload | | `juicefs.max-downloads` | 200 | The max number of connections to download | | `juicefs.max-deletes` | 10 | The max number of connections to delete | | `juicefs.get-timeout` | 5 | The max number of seconds to download an object | | `juicefs.put-timeout` | 60 | The max number of seconds to upload an object | | `juicefs.memory-size` | 300 | Total read/write buffering in MiB | | `juicefs.prefetch` | 1 | Prefetch N blocks in parallel | | `juicefs.upload-limit` | 0 | Bandwidth limit for upload in Mbps | | `juicefs.download-limit` | 0 | Bandwidth limit for download in Mbps | | `juicefs.io-retries` | 10 | Number of retries after network failure | | `juicefs.writeback` | `false` | Upload objects in background | #### Other Configurations | Configuration | Default Value | Description | |-------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `juicefs.bucket` | | Specify a different endpoint for object storage | | `juicefs.debug` | `false` | Whether enable debug log | | `juicefs.access-log` | | Access log path. Ensure Hadoop application has write permission, e.g. `/tmp/juicefs.access.log`. The log file will rotate automatically to keep at most 7 files. | | `juicefs.superuser` | `hdfs` | The super user | | `juicefs.supergroup` | `supergroup` | The super user group | | `juicefs.users` | `null` | The path of username and UID list file, e.g. `jfs://name/etc/users`. The file format is `:`, one user per line. | | `juicefs.groups` | `null` | The path of group name, GID and group members list file, e.g. `jfs://name/etc/groups`. The file format is `::,`, one group per line. | | `juicefs.umask` | `null` | The umask used when creating files and directories (e.g. `0022`), default value is `fs.permissions.umask-mode`. | | `juicefs.push-gateway` | | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) address, format is `:`. | | `juicefs.push-auth` | | [Prometheus basic auth](https://prometheus.io/docs/guides/basic-auth) information, format is `:`. | | `juicefs.push-graphite` | | [Graphite](https://graphiteapp.org) address, format is `:`. | | `juicefs.push-remote-write` | | [Prometheus remote write](https://prometheus.io/docs/specs/prw/remote_write_spec) endpoint, format is `http://:`. | | `juicefs.push-remote-write-auth` | | Authentication for remote write endpoint, format is `:`. | | `juicefs.push-interval` | 10 | Metric push interval (in seconds) | | `juicefs.push-labels` | | Metric labels, format is `key1:value1;key2:value2`. | | `juicefs.fast-resolve` | `true` | Whether enable faster metadata lookup using Redis Lua script | | `juicefs.no-usage-report` | `false` | Whether disable usage reporting. JuiceFS only collects anonymous usage data (e.g. version number), no user or any sensitive data will be collected. | | `juicefs.no-bgjob` | `false` | Disable background jobs (clean-up, backup, etc.) | | `juicefs.backup-meta` | 3600 | Interval (in seconds) to automatically backup metadata in the object storage (0 means disable backup) | | `juicefs.backup-skip-trash` | `false` | Skip files and directories in trash when backup metadata. | | `juicefs.heartbeat` | 12 | Heartbeat interval (in seconds) between client and metadata engine. It's recommended that all clients use the same value. | | `juicefs.skip-dir-mtime` | 100ms | Minimal duration to modify parent dir mtime. | | `juicefs.subdir` | | Allow access only to the subpaths of this directory. Multiple paths can be specified, separated by commas. All other paths, including the root or sibling directories, will be denied access. | #### Multiple file systems configuration When multiple JuiceFS file systems need to be used at the same time, all the above configuration items can be specified for a specific file system. You only need to put the file system name in the middle of the configuration item, such as `jfs1` and `jfs2` in the following example: ```xml juicefs.jfs1.meta redis://jfs1.host:port/1 juicefs.jfs2.meta redis://jfs2.host:port/1 ``` #### Configuration Example The following is a commonly used configuration example. Please replace the `{HOST}`, `{PORT}` and `{DB}` variables in the `juicefs.meta` configuration with actual values. ```xml fs.jfs.impl io.juicefs.JuiceFileSystem fs.AbstractFileSystem.jfs.impl io.juicefs.JuiceFS juicefs.meta redis://{HOST}:{PORT}/{DB} juicefs.cache-dir /data*/jfs juicefs.cache-size 1024 juicefs.access-log /tmp/juicefs.access.log ``` ## Configuration in Hadoop Please refer to the aforementioned configuration tables and add configuration parameters to the Hadoop configuration file `core-site.xml`. ### CDH6 If you are using CDH 6, in addition to modifying `core-site`, you also need to modify `mapreduce.application.classpath` through the YARN service interface, adding: ```shell $HADOOP_COMMON_HOME/lib/juicefs-hadoop.jar ``` ### HDP In addition to modifying `core-site`, you also need to modify the configuration `mapreduce.application.classpath` through the MapReduce2 service interface and add it at the end (variables do not need to be replaced): ```shell /usr/hdp/${hdp.version}/hadoop/lib/juicefs-hadoop.jar ``` ### Flink Add configuration parameters to `conf/flink-conf.yaml`. If you only use JuiceFS in Flink, you don't need to configure JuiceFS in the Hadoop environment, you only need to configure the Flink client. ### Hudi :::note Hudi supports JuiceFS since v0.10.0, please make sure you are using the correct version. ::: Please refer to ["Hudi Official Documentation"](https://hudi.apache.org/docs/jfs_hoodie) to learn how to configure JuiceFS. ### Kafka Connect It is possible to use Kafka Connect and HDFS Sink Connector（[HDFS 2](https://docs.confluent.io/kafka-connect-hdfs/current/overview.html) and [HDFS 3](https://docs.confluent.io/kafka-connect-hdfs3-sink/current/overview.html)）to store data on JuiceFS. First you need to add JuiceFS SDK to `classpath` in Kafka Connect, e.g., `/usr/share/java/confluentinc-kafka-connect-hdfs/lib`. While creating a Connect Sink task, configuration needs to be set up as follows: - Specify `hadoop.conf.dir` as the directory that contains the configuration file `core-site.xml`. If it is not running in Hadoop environment, you can create a separate directory such as `/usr/local/juicefs/hadoop`, and then add the JuiceFS related configurations to `core-site.xml`. - Specify `store.url` as a path starting with `jfs://`. For example: ```ini # Other configuration items are omitted. hadoop.conf.dir=/path/to/hadoop-conf store.url=jfs://path/to/store ``` ### HBase JuiceFS can be used by HBase for HFile, but is not fast (low latency) enough for Write Ahead Log (WAL), because it take much longer time to persist data into object storage than memory of DataNode. It is recommended to deploy a small HDFS cluster to store WAL and HFile files to be stored on JuiceFS. #### Create a new HBase cluster Modify `hbase-site.xml`: ```xml title="hbase-site.xml" hbase.rootdir jfs://{vol_name}/hbase hbase.wal.dir hdfs://{ns}/hbase-wal ``` #### Modify existing HBase cluster In addition to modifying the above configurations, since the HBase cluster has already stored some data in ZooKeeper, in order to avoid conflicts, there are two solutions: 1. Delete the old cluster Delete the znode (default `/hbase`) configured by `zookeeper.znode.parent` via the ZooKeeper client. :::note This operation will delete all data on this HBase cluster. ::: 2. Use a new znode Keep the znode of the original HBase cluster so that it can be recovered later. Then configure a new value for `zookeeper.znode.parent`: ```xml title="hbase-site.xml" zookeeper.znode.parent /hbase-jfs ``` ### Restart Services When the following components need to access JuiceFS, they should be restarted. :::note Before restart, you need to confirm JuiceFS related configuration has been written to the configuration file of each component, usually you can find them in `core-site.xml` on the machine where the service of the component was deployed. ::: | Components | Services | | ---------- | -------------------------- | | Hive | HiveServerMetastore | | Spark | ThriftServer | | Presto | CoordinatorWorker | | Impala | Catalog ServerDaemon | | HBase | MasterRegionServer | HDFS, Hue, ZooKeeper and other services don't need to be restarted. When `Class io.juicefs.JuiceFileSystem not found` or `No FilesSystem for scheme: jfs` exceptions was occurred after restart, reference [FAQ](#faq). ### Trash JuiceFS Hadoop Java SDK also has the same trash function as HDFS, which needs to be enabled by setting `fs.trash.interval` and `fs.trash.checkpoint.interval`, please refer to [HDFS documentation](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#File_Deletes_and_Undeletes) for more information. ## Environmental Verification After the deployment of the JuiceFS Java SDK, the following methods can be used to verify the success of the deployment. ### Hadoop CLI ```bash hadoop fs -ls jfs://{JFS_NAME}/ ``` :::info The `JFS_NAME` is the volume name when you format JuiceFS file system. ::: ### Hive ```sql CREATE TABLE IF NOT EXISTS person ( name STRING, age INT ) LOCATION 'jfs://{JFS_NAME}/tmp/person'; ``` ### Java/Scala project 1. Add Maven or Gradle dependencies: ```xml org.apache.hadoop hadoop-common {HADOOP_VERSION} provided io.juicefs juicefs-hadoop {JUICEFS_HADOOP_VERSION} provided ``` ```groovy dependencies { implementation 'org.apache.hadoop:hadoop-common:${hadoopVersion}' implementation 'io.juicefs:juicefs-hadoop:${juicefsHadoopVersion}' } ``` 2. Use the following sample code to verify: ```java package demo; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; public class JuiceFSDemo { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); conf.set("fs.jfs.impl", "io.juicefs.JuiceFileSystem"); conf.set("juicefs.meta", "redis://127.0.0.1:6379/0"); // JuiceFS metadata engine URL Path p = new Path("jfs://{JFS_NAME}/"); // Please replace "{JFS_NAME}" with the correct value FileSystem jfs = p.getFileSystem(conf); FileStatus[] fileStatuses = jfs.listStatus(p); // Traverse JuiceFS file system and print file paths for (FileStatus status : fileStatuses) { System.out.println(status.getPath()); } } } ``` ## Monitoring metrics collection Please see the ["Monitoring"](../administration/monitoring.md) documentation to learn how to collect and display JuiceFS monitoring metrics. ## Benchmark Here are a series of methods to use the built-in stress testing tool of the JuiceFS client to test the performance of the client environment that has been successfully deployed. ### 1. Local Benchmark #### Metadata - **create** ```shell hadoop jar juicefs-hadoop.jar nnbench create -files 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench -local ``` This command will create 10000 empty files - **open** ```shell hadoop jar juicefs-hadoop.jar nnbench open -files 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench -local ``` This command will open 10000 files without reading data - **rename** ```shell hadoop jar juicefs-hadoop.jar nnbench rename -files 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench -local ``` - **delete** ```shell hadoop jar juicefs-hadoop.jar nnbench delete -files 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench -local ``` - **For reference** | Operation | TPS | Latency (ms) | | --------- | ---- | ------------ | | create | 644 | 1.55 | | open | 3467 | 0.29 | | rename | 483 | 2.07 | | delete | 506 | 1.97 | #### I/O Performance - **sequential write** ```shell hadoop jar juicefs-hadoop.jar dfsio -write -size 20000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/DFSIO -local ``` - **sequential read** ```shell hadoop jar juicefs-hadoop.jar dfsio -read -size 20000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/DFSIO -local ``` When run the cmd for the second time, the result may be much better than the first run. It's because the data was cached in memory, just clean the local disk cache. - **For reference** | Operation | Throughput (MB/s) | | --------- | ----------------- | | write | 647 | | read | 111 | If the network bandwidth of the machine is relatively low, it can generally reach the network bandwidth bottleneck. ### 2. Distributed Benchmark The following command will start the MapReduce distributed task to test the metadata and IO performance. During the test, it is necessary to ensure that the cluster has sufficient resources to start the required map tasks. Computing resources used in this test: - **Server**: 4 cores and 32 GB memory, burst bandwidth 5Gbit/s x 3 - **Database**: Alibaba Cloud Redis 5.0 Community 4G Master-Slave Edition #### Metadata - **create** ```shell hadoop jar juicefs-hadoop.jar nnbench create -maps 10 -threads 10 -files 1000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench ``` 10 map task, each has 10 threads, each thread create 1000 empty file. 100000 files in total - **open** ```shell hadoop jar juicefs-hadoop.jar nnbench open -maps 10 -threads 10 -files 1000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench ``` 10 map task, each has 10 threads, each thread open 1000 file. 100000 files in total - **rename** ```shell hadoop jar juicefs-hadoop.jar nnbench rename -maps 10 -threads 10 -files 1000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench ``` 10 map task, each has 10 threads, each thread rename 1000 file. 100000 files in total - **delete** ```shell hadoop jar juicefs-hadoop.jar nnbench delete -maps 10 -threads 10 -files 1000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench ``` 10 map task, each has 10 threads, each thread delete 1000 file. 100000 files in total - **For reference** - 10 threads | Operation | IOPS | Latency (ms) | | --------- | ---- | ------------ | | create | 4178 | 2.2 | | open | 9407 | 0.8 | | rename | 3197 | 2.9 | | delete | 3060 | 3.0 | - 100 threads | Operation | IOPS | Latency (ms) | | --------- | ---- | ------------ | | create | 11773 | 7.9 | | open | 34083 | 2.4 | | rename | 8995 | 10.8 | | delete | 7191 | 13.6 | #### I/O Performance - **sequential write** ```shell hadoop jar juicefs-hadoop.jar dfsio -write -maps 10 -size 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/DFSIO ``` 10 map task, each task write 10000MB random data sequentially - **sequential read** ```shell hadoop jar juicefs-hadoop.jar dfsio -read -maps 10 -size 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/DFSIO ``` 10 map task, each task read 10000MB random data sequentially - **For reference** | Operation | Average throughput (MB/s) | Total Throughput (MB/s) | | --------- | ------------------------- | ----------------------- | | write | 198 | 1835 | | read | 124 | 1234 | ### 3. TPC-DS The test dataset is 100GB in size, and both Parquet and ORC file formats are tested. This test only tests the first 10 queries. Spark Thrift JDBC/ODBC Server is used to start the Spark resident process and then submit the task via Beeline connection. #### Test Hardware | Node Category | Instance Type | CPU | Memory | Disk | Number | | ------------- | ------------- | --- | ------ | ---- | ------ | | Master | Alibaba Cloud ecs.r6.xlarge | 4 | 32GiB | System Disk: 100GiB | 1 | | Core | Alibaba Cloud ecs.r6.xlarge | 4 | 32GiB | System Disk: 100GiBData Disk: 500GiB Ultra Disk x 2 | 3 | #### Software Configuration ##### Spark Thrift JDBC/ODBC Server ```shell ${SPARK_HOME}/sbin/start-thriftserver.sh \ --master yarn \ --driver-memory 8g \ --executor-memory 10g \ --executor-cores 3 \ --num-executors 3 \ --conf spark.locality.wait=100 \ --conf spark.sql.crossJoin.enabled=true \ --hiveconf hive.server2.thrift.port=10001 ``` ##### JuiceFS Cache Configurations The 2 data disks of Core node are mounted in the `/data01` and `/data02` directories, and `core-site.xml` is configured as follows: ```xml juicefs.cache-size 200000 juicefs.cache-dir /data*/jfscache juicefs.cache-full-block false juicefs.discover-nodes-url yarn juicefs.attr-cache 3 juicefs.entry-cache 3 juicefs.dir-entry-cache 3 ``` #### Test The task submission command is as follows: ```shell ${SPARK_HOME}/bin/beeline -u jdbc:hive2://localhost:10001/${DATABASE} \ -n hadoop \ -f query{i}.sql ``` #### Results JuiceFS can use local disk as a cache to accelerate data access, the following data is the result (in seconds) after 4 runs using Redis and TiKV as the metadata engine of JuiceFS respectively. ##### ORC | Queries | JuiceFS (Redis) | JuiceFS (TiKV) | HDFS | | ------- | --------------- | -------------- | ---- | | q1 | 20 | 20 | 20 | | q2 | 28 | 33 | 26 | | q3 | 24 | 27 | 28 | | q4 | 300 | 309 | 290 | | q5 | 116 | 117 | 91 | | q6 | 37 | 42 | 41 | | q7 | 24 | 28 | 23 | | q8 | 13 | 15 | 16 | | q9 | 87 | 112 | 89 | | q10 | 23 | 24 | 22 | ##### Parquet | Queries | JuiceFS (Redis) | JuiceFS (TiKV) | HDFS | | ------- | --------------- | -------------- | ---- | | q1 | 33 | 35 | 39 | | q2 | 28 | 32 | 31 | | q3 | 23 | 25 | 24 | | q4 | 273 | 284 | 266 | | q5 | 96 | 107 | 94 | | q6 | 36 | 35 | 42 | | q7 | 28 | 30 | 24 | | q8 | 11 | 12 | 14 | | q9 | 85 | 97 | 77 | | q10 | 24 | 28 | 38 | ## Permission control by Apache Ranger(from v1.3) JuiceFS currently supports path permission control by integrating with Apache Ranger's HDFS module. Only supported in Hadoop Java SDK. ### 1. Configurations The config for Apache Ranger is sotred in the metadata database. You can enable Ranger permission control by the following methods: ```shell # configure with format juicefs format META-URL NAME --ranger-rest-url http://localhost:6080 --ranger-service jfs # or configure with config juicefs config META-URL --ranger-rest-url http://localhost:6080 --ranger-service jfs # disable ranger juicefs config META-URL --ranger-rest-url "" --ranger-service jfs "" ``` ### 2. Dependencies Considering the convenience of use, JuiceFS packages all Ranger dependencies into the JuiceFS SDK. If you encounter version conflicts with Apache Ranger, you may need to modify the version and recompile. ### 3. Configure Kerberos When Kerberos is enabled, the JuiceFS client used by business applications cannot download Ranger security policies. You need to configure `policy.download.auth.users` and `tag.download.auth.users` in the Ranger Admin Web UI under the HDFS Service to grant specified users the permission to download security policies. Multiple users should be separated by commas (,). After completing the configuration, you must use that user to update the policies stored in JuiceFS. The command to update the security policy file is as follows (it is recommended to configure it as a scheduled task for automatic updates). Replace `{PRINCIPAL}` with the user configured in `policy.download.auth.users`: ```shell hadoop jar juicefs-hadoop-{version}.jar ranger \ --fs jfs://{VOL_NAME}/ \ --keytab /path/to/keytab \ --principal {PRINCIPAL} ``` ### 4. Tips #### 4.1 Ranger version The code is tested on `Ranger2.3` and `Ranger2.4`. As no other features are used except for `HDFS` module authentication, theoretically all other versions are applicable. #### 4.2 Ranger Audit Currently, only support authentication function, and the `Ranger Audit` is disabled. #### 4.3 Ranger's other parameters To improve usage efficiency, currently only support some **CORE** parameters of Ranger. #### 4.4 Security tips Due to the complete open source of the project, it is unavoidable for users to disrupt permission control by replacing parameters such as `ranger-rest-url`. If stricter control is required, it is recommended to compile the code independently and solve the problem by encrypting relevant security parameters. ## Kerberos support (from v1.4) JuiceFS supports Kerberos authentication, but only for the Hadoop Java SDK. ### 1. Configurations Kerberos configuration is stored in the metadata database and can be managed via the JuiceFS CLI. ```shell # Enable Kerberos when formatting juicefs format META-URL --kerberos-config-file kerb.cfg # Or enable Kerberos on an existing filesystem ./juicefs config META-URL --kerberos-config-file kerb.cfg # Disable Kerberos ./juicefs config META-URL --kerberos-config-file "" ``` The Kerberos configuration file: ``` # Kerberos keytab, encoded with BASE64 # base64 -w 0 meta.keytab {VOL_NAME}.keytab={BASE64 KEYTAB} # Delegation token {VOL_NAME}.token.life=604800 {VOL_NAME}.token.renew=86400 # Superuser and supergroup {VOL_NAME}.superuser=hadoop {VOL_NAME}.supergroup=supergroup # Mapping from Kerberos principals to OS user accounts # https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Mapping_from_Kerberos_principals_to_OS_user_accounts {VOL_NAME}.mechanism=hadoop {VOL_NAME}.rule=DEFAULT # Proxy user settings # https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Proxy_user # Users: user1,user2 or * {VOL_NAME}.proxy.client.users=foo # Groups: group1,group2 or * {VOL_NAME}.proxy.client.groups=foogrp # Hosts: host1,host2 or 192.168.1.1,192.168.1.2 or 192.168.1.1/32 or * {VOL_NAME}.proxy.client.hosts=* ``` `core-site.xml` configuration: ```xml juicefs.server-principal {YOUR_SPN} ``` ## FAQ ### 1. `Class io.juicefs.JuiceFileSystem not found` exception It means JAR file was not loaded, you can verify it by `lsof -p {pid} | grep juicefs`. You should check whether the JAR file was located properly, or other users have the read permission. Some Hadoop distribution also need to modify `mapred-site.xml` and put the JAR file location path to the end of the parameter `mapreduce.application.classpath`. ### 2. `No FilesSystem for scheme: jfs` exception It means JuiceFS Hadoop Java SDK was not configured properly, you need to check whether there is JuiceFS related configuration in the `core-site.xml` of the component configuration. ### 3. What are the similarities and differences between user permission management in JuiceFS and HDFS? JuiceFS also uses the "User/Group" method to manage file permissions, using local users and groups by default. In order to ensure the unified permissions of different nodes during distributed computing, you can configure global "User/UID" and "Group/GID" mappings through `juicefs.users` and `juicefs.groups` configurations. ### 4. After the data is deleted, it is directly stored in the `.trash` directory of JuiceFS. Although the files are all there, it is difficult to restore the data through the `mv` command as easily as HDFS. Is there any way to achieve a similar effect of HDFS trash? In the Hadoop application scenario, the functions similar to the HDFS trash are still retained. It needs to be explicitly enabled by `fs.trash.interval` and `fs.trash.checkpoint.interval` configurations, please refer to [document](#trash) for more information. ### 5. What are the benefits of setting the `juicefs.discover-nodes-url` configuration? In HDFS, each data block will have [`BlockLocation`](https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/BlockLocation.html) information, which the computing engine uses to schedule the computing tasks as much as possible to the nodes where the data is stored. JuiceFS will calculate the corresponding `BlockLocation` for each data block through the consistent hashing algorithm, so that when the same data is read for the second time, the computing engine may schedule the computing task to the same node, and the data cached on the local disk during the first computing can be used to accelerate data access. This algorithm needs to know all the computing node information in advance. The `juicefs.discover-nodes-url` configuration is used to obtain these computing node information. ### 6. Does the community version of JuiceFS currently support a Kerberos-authenticated CDH cluster? Not supported. JuiceFS does not verify the validity of Kerberos users, but can use Kerberos-authenticated username. --- ## File: docs/en/deployment/how_to_use_on_kubernetes.md --- title: Use JuiceFS on Kubernetes sidebar_position: 2 slug: /how_to_use_on_kubernetes --- JuiceFS is an ideal storage layer for Kubernetes, read this chapter to learn how to use JuiceFS in Kubernetes. ## Use JuiceFS via `hostPath` If you simply need to use JuiceFS inside Kubernetes pods, without any special requirements (e.g. isolation, permission control), then [`hostPath`](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) can be a good practice, which is also really easy to setup: 1. Install and mount JuiceFS on all Kubernetes worker nodes, [Automated Deployment](./automation.md) is recommended for this type of work. 1. Use `hostPath` volume inside pod definition, and mount a JuiceFS sub-directory to container: ```yaml {8-16} apiVersion: v1 kind: Pod metadata: name: juicefs-app spec: containers: - ... volumeMounts: - name: jfs-data mountPath: /opt/app-data volumes: - name: jfs-data hostPath: # Assuming JuiceFS is mounted on /jfs path: "/jfs/myapp/" type: Directory ``` In comparison to using JuiceFS CSI Driver, `hostPath` is a much more simple practice, and easier to debug when things go wrong, but notice that: * For ease of management, generally all pods use the same host mount point. Lack of isolation may lead to data security issues, and obviously, you won't be able to adjust JuiceFS mount parameters separately for each application. Please evaluate carefully. * All worker nodes should mount JuiceFS in advance, so when adding a new node to the cluster, JuiceFS needs to be installed and mounted during the initialization process, otherwise the new node does not have a JuiceFS mount point, and the container will not be created. * The system resources (such as CPU, memory, etc.) occupied by the JuiceFS mounting process on the host are not controlled by Kubernetes, and may occupy too many host resources. You can consider using [`system-reserved`](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#system-reserved) to properly adjust the system resource reservation of Kubernetes, to reserve more resources for the JuiceFS mount process. * If the JuiceFS mount process on the host exits unexpectedly, the application pod will not be able to access the mount point normally. In this case, the JuiceFS file system needs to be remounted and the application pod must be rebuilt. However, JuiceFS CSI Driver solves this problem well by providing the [Automatic Mount Point Recovery](https://juicefs.com/docs/csi/recover-failed-mountpoint) mechanism. * If you're using Docker as Kubernetes container runtime, it's best to start JuiceFS mount prior to Docker in startup order, to avoid containers being created before JuiceFS is properly mounted. For systemd, you can use below unit file to manually control startup order: ```systemd title="/etc/systemd/system/docker.service.d/override.conf" [Unit] # Use below command to obtain JuiceFS mount service name # systemctl list-units | grep "\.mount" After=network-online.target firewalld.service containerd.service jfs.mount ``` ## JuiceFS CSI Driver To use JuiceFS in Kubernetes, refer to [JuiceFS CSI Driver Documentation](https://juicefs.com/docs/csi/introduction). ## Mount JuiceFS in the container In some cases, you may need to mount JuiceFS volume directly in the container, which requires the use of the JuiceFS client in the container. You can refer to the following `Dockerfile` example to integrate the JuiceFS client into your application image: ```dockerfile title="Dockerfile" FROM alpine:latest LABEL maintainer="Juicedata " # Install JuiceFS client RUN apk add --no-cache curl && \ JFS_LATEST_TAG=$(curl -s https://api.github.com/repos/juicedata/juicefs/releases/latest | grep 'tag_name' | cut -d '"' -f 4 | tr -d 'v') && \ wget "https://github.com/juicedata/juicefs/releases/download/v${JFS_LATEST_TAG}/juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \ tar -zxf "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \ install juicefs /usr/bin && \ rm juicefs "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \ rm -rf /var/cache/apk/* && \ apk del curl ENTRYPOINT ["/usr/bin/juicefs", "mount"] ``` Since JuiceFS needs to use the FUSE device to mount the file system, it is necessary to allow the container to run in privileged mode when creating a Pod: ```yaml {19-20} apiVersion: apps/v1 kind: Deployment metadata: name: nginx-run spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: linuxserver/nginx ports: - containerPort: 80 securityContext: privileged: true ``` :::caution With the privileged mode being enabled by `privileged: true`, the container has access to all devices of the host, that is, it has full control of the host's kernel. Improper uses will bring serious safety hazards. Please conduct a thorough safety assessment before using it. ::: --- ## File: docs/en/deployment/juicefs_on_docker.md --- title: Using JuiceFS in Docker sidebar_position: 6 slug: /juicefs_on_docker --- You can use the JuiceFS file system in Docker by running the client directly in the container or using a volume plugin. ## Using a volume plugin {#volume-plugin} If you have specific requirements for mount management, such as managing mount points through Docker to facilitate different application containers using different JuiceFS file systems, you can use a [Docker volume plugin](https://github.com/juicedata/docker-volume-juicefs). Docker plugins are usually provided in the form of images. The [JuiceFS volume plugin image](https://hub.docker.com/r/juicedata/juicefs) contains the [JuiceFS Community Edition](../introduction/README.md) and [JuiceFS Cloud Service](https://juicefs.com/docs/cloud) clients. After installation, you can run the volume plugin to create JuiceFS volumes in Docker. Install the plugin using the following command and provide the necessary permissions for FUSE as prompted: ```shell docker plugin install juicedata/juicefs ``` You can use the following commands to manage the volume plugin: ```shell # Disable the plugin docker plugin disable juicedata/juicefs # Upgrade the plugin (must be disabled first) docker plugin upgrade juicedata/juicefs docker plugin enable juicedata/juicefs # Remove the plugin docker plugin rm juicedata/juicefs ``` ### Create a storage volume {#create-volume} Replace ``, ``, ``, ``, ``, and `` in the following command with your own file system configuration: ```shell docker volume create -d juicedata/juicefs \ -o name= \ -o metaurl= \ -o storage= \ -o bucket= \ -o access-key= \ -o secret-key= \ jfsvolume ``` For pre-created file systems, you only need to specify the file system name and database address when creating the volume plugin, for example: ```shell docker volume create -d juicedata/juicefs \ -o name= \ -o metaurl= \ jfsvolume ``` If you need to pass additional environment variables when mounting the file system, such as in [Google Cloud](../reference/how_to_set_up_object_storage.md#google-cloud), you can append parameters similar to `-o env=FOO=bar,SPAM=egg` to the above command. ### Usage and management {#usage-and-management} ```shell # Mount the volume when creating a container docker run -it -v jfsvolume:/opt busybox ls /opt # After unmounting, you can delete the storage volume. Note that this only deletes the corresponding resources in Docker and does not affect the data stored in JuiceFS. docker volume rm jfsvolume ``` ### Using the plugin in Docker Compose {#using-plugin-in-docker-compose} Here is an example of using the JuiceFS volume plugin in `docker-compose`: ```yaml version: '3' services: busybox: image: busybox command: "ls /jfs" volumes: - jfsvolume:/jfs volumes: jfsvolume: driver: juicedata/juicefs driver_opts: name: ${VOL_NAME} # SQLite creates the database file in the plugin container's local path, # and sqlite:// will fail when the service is restarted. # (See https://github.com/juicedata/docker-volume-juicefs/issues/37 for details) metaurl: ${META_URL} storage: ${STORAGE_TYPE} bucket: ${BUCKET} access-key: ${ACCESS_KEY} secret-key: ${SECRET_KEY} # If necessary, you can pass additional environment variables using env # env: FOO=bar,SPAM=egg ``` Usage and management: ```shell # Start the service docker-compose up # Stop the service and unmount the JuiceFS file system from Docker docker-compose down --volumes ``` ### Troubleshooting the volume plugin {#troubleshooting} If it is not working properly, it is recommended to first [upgrade the volume plugin](#volume-plugin), and then check the logs based on the problem. * Collect JuiceFS client logs. The logs are located inside the Docker volume plugin container and need to be accessed by entering the container: ```shell # Confirm the docker plugins runtime directory, which may be different from the example below depending on the actual situation # The directory printed by ls is the container directory, and the name is the container ID ls /run/docker/plugins/runtime-root/plugins.moby # Print plugin container information # If the printed container list is empty, it means that the plugin container failed to be created # Read the plugin startup log below to continue troubleshooting runc --root /run/docker/plugins/runtime-root/plugins.moby list # Enter the container and print the log runc --root /run/docker/plugins/runtime-root/plugins.moby exec 452d2c0cf3fd45e73a93a2f2b00d03ed28dd2bc0c58669cca9d4039e8866f99f cat /var/log/juicefs.log ``` If the container does not exist (`ls` finds an empty directory) or the `juicefs.log` does not exist in the final log printing stage, it is likely that the mount itself failed. Continue to check the plugin's own logs to find the cause. * Collect plugin logs, using systemd as an example: ```shell journalctl -f -u docker | grep "plugin=" ``` If there is an error when the plugin calls `juicefs` or if the plugin itself reports an error, it will be reflected in the logs. ## Using the JuiceFS client in containers {#mount-juicefs-in-docker} Compared to the volume plugin, using the JuiceFS client directly in the container is more flexible. You can directly mount the JuiceFS file system in the container or access it through S3 Gateway or WebDAV. ### Method 1: Build your own image The JuiceFS client is a standalone binary program that provides versions for both AMD64 and ARM64 architectures. You can define the command to download and install the JuiceFS client in the Dockerfile, for example: ```Dockerfile FROM ubuntu:22.04 ... # Use the official one-click installation script RUN curl -sSL https://d.juicefs.com/install | sh - ``` For more information, see [Customizing Container Images](https://juicefs.com/docs/csi/guide/custom-image). ### Method 2: Use the officially maintained image The JuiceFS officially maintained image [`juicedata/mount`](https://hub.docker.com/r/juicedata/mount) is tagged to specify the desired version. **The community edition tags include `latest` and `ce`**, such as `ce-v1.1.2` and `ce-nightly`. The `latest` tag represents the latest community edition, and the `nightly` tag points to the latest development version. For details, see the [tags page](https://hub.docker.com/r/juicedata/mount/tags) on Docker Hub. Before you start, you need to prepare [object storage](../reference/how_to_set_up_object_storage.md) and [metadata engine](../reference/how_to_set_up_metadata_engine.md). #### Create a file system Create a file system through a temporary container, for example: ```sh docker run --rm \ juicedata/mount:ce-v1.1.2 juicefs format \ --storage s3 \ --bucket https://xxx.your-s3-endpoint.com \ --access-key=ACCESSKEY \ --secret-key=SECRETKEY \ rediss://user:password@xxx.your-redis-server.com:6379/1 myjfs ``` Replace `--storage`, `--bucket`, `--access-key`, `--secret-key`, and the metadata engine URL with your own configuration. #### Mount the file system directly in the container Create a container and mount the JuiceFS file system in the container, for example: ```sh docker run --privileged --name myjfs \ juicedata/mount:ce-v1.1.2 juicefs mount \ rediss://user:password@xxx.your-redis-server.com:6379/1 /mnt ``` Replace the metadata engine URL with your own configuration. `/mnt` is the mount point and can be modified as needed. Since FUSE is used, `--privileged` permission is also required. #### Mount the file system through Docker Compose Here is an example using Docker Compose. Replace the metadata engine URL and mount point with your own configuration. ```yaml version: "3" services: busybox: image: busybox command: "ls /jfs" volumes: - ./mnt:/jfs depends_on: juicefs: condition: service_healthy juicefs: image: juicedata/mount:ce-v1.1.2 container_name: myjfs volumes: - ./mnt:/mnt:rw,rshared cap_add: - SYS_ADMIN devices: - /dev/fuse security_opt: - apparmor:unconfined command: ["juicefs", "mount", "rediss://user:password@xxx.your-redis-server.com:6379/1", "/mnt"] restart: unless-stopped healthcheck: test: ["CMD-SHELL", "cat /mnt/.control"] interval: 60s retries: 5 start_period: 30s timeout: 10s ``` In the container, the JuiceFS file system is mounted to the `/mnt` directory, and the volumes section in the configuration file maps the `/mnt` in the container to the `./mnt` directory on the host, allowing direct access to the JuiceFS file system mounted in the container from the host. At the same time, by combining `depends_on` and `volumes`, the directory mapped to the host machine can be remounted into the container for use. #### Access the file system through S3 Gateway Here is an example of exposing JuiceFS for access through S3 Gateway. Replace `MINIO_ROOT_USER`, `MINIO_ROOT_PASSWORD`, the metadata engine URL, and the address and port number to listen on with your own configuration. ```yaml version: "3" services: s3-gateway: image: juicedata/mount:ce-v1.1.2 container_name: juicefs-s3-gateway environment: - MINIO_ROOT_USER=your-username - MINIO_ROOT_PASSWORD=your-password ports: - "9090:9090" command: ["juicefs", "gateway", "rediss://user:password@xxx.your-redis-server.com:6379/1", "0.0.0.0:9090"] restart: unless-stopped ``` Use port `9090` on the host to access the S3 Gateway console, and use the same address to read and write the JuiceFS file system through the S3 client or SDK. --- ## File: docs/en/deployment/nfs.md --- title: Create NFS Shares sidebar_position: 9 --- NFS (Network File System) is a network file-sharing protocol that allows different computers to share files and directories over a network. It was originally developed by Sun Microsystems and is a standard way of file sharing between Unix and Unix-like systems. The NFS protocol enables clients to access remote file systems as if they were local, achieving transparent remote file access. When you need to share directories from the JuiceFS file system through NFS, you can simply use the `juicefs mount` command to mount the file system. Then, you can create NFS shares with the JuiceFS mount point or subdirectories. :::note `juicefs mount` mounts the file system as a local user-space file system through the FUSE interface, making it identical to the local file system in terms of appearance and usage. Hence, it can be directly used to create NFS shares. ::: ## Step 1. Install NFS To configure NFS shares, you need to install the relevant software packages on both the server and client sides. Let's take Ubuntu/Debian systems as an example: ### 1. Server-side installation Create a host for NFS sharing (with the JuiceFS file system also mounted on this server). ```shell sudo apt install nfs-kernel-server ``` ### 2. Client-side installation All Linux hosts that need to access NFS shares should install the client software. ```shell sudo apt install nfs-common ``` ## Step 2. Create shares Assuming the JuiceFS is mounted on the server system at the path `/mnt/myjfs`, if you want to set the `media` subdirectory as an NFS share, you can add the following configuration to the `/etc/exports` file on the server system: ``` "/mnt/myjfs/media" *(rw,sync,no_subtree_check,fsid=1) ``` The syntax for NFS share configuration is as follows: ``` (options) ``` For example, if you want to restrict the mounting of this share to hosts in the `192.168.1.0/24` IP range and avoid squashing root privileges, you can modify it as follows: ``` "/mnt/myjfs/media" 192.168.1.0/24(rw,async,no_subtree_check,no_root_squash,fsid=1) ``` ### Share option description **Explanation of the share options:** - `rw`: Represents read and write permissions. If read-only access is desired, use `ro`. - `sync` and `async`: `sync` enables synchronous writes, meaning that when writing to the NFS share, the client waits for the server's confirmation of successful data write before proceeding with subsequent operations. `async`, on the other hand, allows asynchronous writes. In this mode, the client does not wait for the server's confirmation of successful write before proceeding with subsequent operations. - `no_subtree_check`: Disables subtree checking, allowing clients to mount both the parent and child directories of the NFS share. This can reduce some security but improve NFS compatibility. Setting it to `subtree_check` enables subtree checking, allowing clients to only mount the NFS share and its subdirectories. - `no_root_squash`: Controls the mapping behavior of the client's root user when accessing the NFS share. By default, when the client mounts the NFS share as root, the server maps it to a non-privileged user (usually nobody or nfsnobody), which is known as root squashing. Enabling this option cancels the root squashing, giving the client the same root user privileges as the server. This option comes with certain security risks and should be used with caution. - `fsid`: A file system identifier used to identify different file systems on NFS. In NFSv4, the root directory of NFS is defined as fsid=0, and other file systems need to be numbered uniquely under it. Here, JuiceFS is an externally mounted FUSE file system, so it needs to be assigned a unique identifier. ### Choosing between async and sync modes For NFS shares, the sync (synchronous writes) mode can improve data reliability but always requires waiting for the server's confirmation before proceeding with the next operation. This may result in lower write performance. For JuiceFS, which is a cloud-based distributed file system, network latency also needs to be considered. Using the sync mode can often lead to lower write performance due to network latency. In most cases, when creating NFS shares with JuiceFS, it is recommended to set the write mode to async (asynchronous writes) to avoid sacrificing write performance. If data reliability must be prioritized and sync mode is necessary, it is recommended to configure JuiceFS with a high-performance SSD as a local cache with sufficient capacity and enable the writeback cache mode.