juicefs Documentation

File: README.md Copy Chapter

JuiceFS is a high-performance POSIX file system released under Apache License 2.0, particularly designed for the cloud-native environment. The data, stored via JuiceFS, will be persisted in Object Storage (e.g. Amazon S3), and the corresponding metadata can be persisted in various compatible database engines such as Redis, MySQL, and TiKV based on the scenarios and requirements.

With JuiceFS, massive cloud storage can be directly connected to big data, machine learning, artificial intelligence, and various application platforms in production environments. Without modifying code, the massive cloud storage can be used as efficiently as local storage.

📖 Document: Quick Start Guide

Highlighted Features Copy Chapter

1. Fully POSIX-compatible: Use as a local file system, seamlessly docking with existing applications without breaking business workflow.
2. Fully Hadoop-compatible: JuiceFS' Hadoop Java SDK is compatible with Hadoop 2.x and Hadoop 3.x as well as a variety of components in the Hadoop ecosystems.
3. S3-compatible: JuiceFS' S3 Gateway provides an S3-compatible interface.
4. Cloud Native: A Kubernetes CSI Driver is provided for easily using JuiceFS in Kubernetes.
5. Shareable: JuiceFS is a shared file storage that can be read and written by thousands of clients.
6. Strong Consistency: The confirmed modification will be immediately visible on all the servers mounted with the same file system.
7. Outstanding Performance: The latency can be as low as a few milliseconds, and the throughput can be expanded nearly unlimitedly (depending on the size of the Object Storage). Test results
8. Data Encryption: Supports data encryption in transit and at rest (please refer to the guide for more information).
9. Global File Locks: JuiceFS supports both BSD locks (flock) and POSIX record locks (fcntl).
10. Data Compression: JuiceFS supports LZ4 or Zstandard to compress all your data.

Architecture | Getting Started | Advanced Topics | POSIX Compatibility | Performance Benchmark | Supported Object Storage | Who is using | Roadmap | Reporting Issues | Contributing | Community | Usage Tracking | License | Credits | FAQ

Architecture Copy Chapter

JuiceFS consists of three parts:

1. JuiceFS Client: Coordinates Object Storage and metadata storage engine as well as implementation of file system interfaces such as POSIX, Hadoop, Kubernetes, and S3 gateway.
2. Data Storage: Stores data, with supports of a variety of data storage media, e.g., local disk, public or private cloud Object Storage, and HDFS.
3. Metadata Engine: Stores the corresponding metadata that contains information of file name, file size, permission group, creation and modification time and directory structure, etc., with supports of different metadata engines, e.g., Redis, MySQL, SQLite and TiKV.

JuiceFS can store the metadata of file system on different metadata engines, like Redis, which is a fast, open-source, in-memory key-value data storage, particularly suitable for storing metadata; meanwhile, all the data will be stored in Object Storage through JuiceFS client. Learn more

Each file stored in JuiceFS is split into "Chunk" s at a fixed size with the default upper limit of 64 MiB. Each Chunk is composed of one or more "Slice"(s), and the length of the slice varies depending on how the file is written. Each slice is composed of size-fixed "Block" s, which are 4 MiB by default. These blocks will be stored in Object Storage in the end; at the same time, the metadata information of the file and its Chunks, Slices, and Blocks will be stored in metadata engines via JuiceFS. Learn more

When using JuiceFS, files will eventually be split into Chunks, Slices and Blocks and stored in Object Storage. Therefore, the source files stored in JuiceFS cannot be found in the file browser of the Object Storage platform; instead, there are only a chunks directory and a bunch of digitally numbered directories and files in the bucket. Don't panic! This is just the secret of the high-performance operation of JuiceFS!

Getting Started Copy Chapter

Before you begin, make sure you have:

1. One supported metadata engine, see How to Set Up Metadata Engine
2. One supported Object Storage for storing data blocks, see Supported Object Storage
3. JuiceFS Client downloaded and installed

Please refer to Quick Start Guide to start using JuiceFS right away!

Command Reference Copy Section

Check out all the command line options in command reference.

Containers Copy Section

JuiceFS can be used as a persistent volume for Docker and Podman, please check here for details.

Kubernetes Copy Section

It is also very easy to use JuiceFS on Kubernetes. Please find more information here.

Hadoop Java SDK Copy Section

If you wanna use JuiceFS in Hadoop, check Hadoop Java SDK.

Advanced Topics Copy Chapter

• Redis Best Practices
• How to Setup Object Storage
• Cache
• Fault Diagnosis and Analysis
• FUSE Mount Options
• Using JuiceFS on Windows
• S3 Gateway

Please refer to JuiceFS Document Center for more information.

POSIX Compatibility Copy Chapter

JuiceFS has passed all of the compatibility tests (8813 in total) in the latest pjdfstest .

All tests successful.

Test Summary Report
-------------------
/root/soft/pjdfstest/tests/chown/00.t          (Wstat: 0 Tests: 1323 Failed: 0)
  TODO passed:   693, 697, 708-709, 714-715, 729, 733
Files=235, Tests=8813, 233 wallclock secs ( 2.77 usr  0.38 sys +  2.57 cusr  3.93 csys =  9.65 CPU)
Result: PASS

Aside from the POSIX features covered by pjdfstest, JuiceFS also provides:

• Close-to-open consistency. Once a file is written and closed, it is guaranteed to view the written data in the following opens and reads from any client. Within the same mount point, all the written data can be read immediately.
• Rename and all other metadata operations are atomic, which are guaranteed by supported metadata engine transaction.
• Opened files remain accessible after unlink from same mount point.
• Mmap (tested with FSx).
• Fallocate with punch hole support.
• Extended attributes (xattr).
• BSD locks (flock).
• POSIX record locks (fcntl).

Performance Benchmark Copy Chapter

Basic benchmark Copy Section

JuiceFS provides a subcommand that can run a few basic benchmarks to help you understand how it works in your environment:

Throughput Copy Section

A sequential read/write benchmark has also been performed on JuiceFS, EFS and S3FS by fio.

Above result figure shows that JuiceFS can provide 10X more throughput than the other two (see more details).

Metadata IOPS Copy Section

A simple mdtest benchmark has been performed on JuiceFS, EFS and S3FS by mdtest.

The result shows that JuiceFS can provide significantly more metadata IOPS than the other two (see more details).

Analyze performance Copy Section

See Real-Time Performance Monitoring if you encountered performance issues.

Supported Object Storage Copy Chapter

• Amazon S3 (and other S3 compatible Object Storage services)
• Google Cloud Storage
• Azure Blob Storage
• Alibaba Cloud Object Storage Service (OSS)
• Tencent Cloud Object Storage (COS)
• Qiniu Cloud Object Storage (Kodo)
• QingStor Object Storage
• Ceph RGW
• MinIO
• Local disk
• Redis
• ...

JuiceFS supports numerous Object Storage services. Learn more.

Who is using Copy Chapter

JuiceFS is production ready and used by thousands of machines in production. A list of users has been assembled and documented here. In addition JuiceFS has several collaborative projects that integrate with other open source projects, which we have documented here. If you are also using JuiceFS, please feel free to let us know, and you are welcome to share your specific experience with everyone.

The storage format is stable, and will be supported by all future releases.

Roadmap Copy Chapter

• Gateway Optimization
• Resumable Sync
• Read-ahead Optimization
• Optimization for Large-scale Scenarios
• Snapshots

Reporting Issues Copy Chapter

We use GitHub Issues to track community reported issues. You can also contact the community for any questions.

Contributing Copy Chapter

Thank you for your contribution! Please refer to the JuiceFS Contributing Guide for more information.

Community Copy Chapter

Welcome to join the Discussions and the Discord channel to connect with JuiceFS team members and other users.

Usage Tracking Copy Chapter

JuiceFS collects anonymous usage data by default to help us better understand how the community is using JuiceFS. Only core metrics (e.g. version number) will be reported, and user data and any other sensitive data will not be included. The related code can be viewed here.

You could also disable reporting easily by command line option --no-usage-report:

juicefs mount --no-usage-report

License Copy Chapter

JuiceFS is open-sourced under Apache License 2.0, see LICENSE.

Credits Copy Chapter

The design of JuiceFS was inspired by Google File System, HDFS and MooseFS. Thanks for their great work!

FAQ Copy Chapter

Why doesn't JuiceFS support XXX Object Storage? Copy Section

JuiceFS supports many Object Storage services. Please check out this list first. If the Object Storage you want to use is compatible with S3, you could treat it as S3. Otherwise, try reporting any issue.

Can I use Redis Cluster as metadata engine? Copy Section

Yes. Since v1.0.0 Beta3 JuiceFS supports the use of Redis Cluster as the metadata engine, but it should be noted that Redis Cluster requires that the keys of all operations in a transaction must be in the same hash slot, so a JuiceFS file system can only use one hash slot.

See "Redis Best Practices" for more information.

What's the difference between JuiceFS and XXX? Copy Section

See "Comparison with Others" for more information.

For more FAQs, please see the full list.

Stargazers over time Copy Chapter

File: docs/en/development/contributing_guide.md Copy Chapter

title: Contributing Guide sidebar_position: 1 Copy Chapter

Guidelines Copy Chapter

• Before starting work on a feature or bug fix, search GitHub or reach out to us via GitHub or Discord, make sure no one else is already working on it and we'll ask you to open a GitHub issue if necessary.
• Before contributing, use the GitHub issue to discuss the feature and reach an agreement with the core developers.
• For major feature updates, write a design document to help the community understand your motivation and solution.
• Find issues with the label "kind/good-first-issue" or "kind/help-wanted".

Read internals for important data structure references.

Coding style Copy Chapter

• We're following "Effective Go" and "Go Code Review Comments".
• Use go fmt to format your code before committing. You can find information in editor support for Go tools in "IDEs and Plugins for Go".
• Every new source file must begin with a license header.
• Install pre-commit and use it to set up a pre-commit hook for static analysis. Just run pre-commit install in the root of the repo.

Sign the CLA Copy Chapter

Before you can contribute to JuiceFS, you will need to sign the Contributor License Agreement. There're a CLA assistant to guide you when you first time submit a pull request.

What is a good PR Copy Chapter

• Presence of unit tests
• Adherence to the coding style
• Adequate in-line comments
• Explanatory commit message

Contribution flow Copy Chapter

1. Create a topic branch from where to base the contribution. This is usually main.
2. Make commits of logical units.
3. Make sure commit messages are in the proper format.
4. Push changes in a topic branch to a personal fork of the repository.
5. Submit a pull request to juicedata/juicefs. The PR should link to one issue which either created by you or others.
6. The PR must receive approval from at least one maintainer before it be merged.

File: docs/en/development/internals.md Copy Chapter

title: Internals sidebar_position: 4 slug: /internals Copy Chapter

This article introduces implementation details of JuiceFS, use this as a reference if you'd like to contribute. The content below is based on JuiceFS v1.0.0, metadata version v1.

Before digging into source code, you should read Data Processing Workflow.

Keyword Definition Copy Chapter

High level concepts:

• File system: i.e. JuiceFS Volume, represents a separate namespace. Files can be moved freely within the same file system, while data copies are required between different file systems.
• Metadata engine: A supported database instance of your choice, that stores and manages file system metadata. There are three categories of metadata engines currently supported by JuiceFS.
  • Redis: Redis and various protocol-compatible services
  • SQL: MySQL, PostgreSQL, SQLite, etc.
  • TKV: TiKV, BadgerDB, etc.
• Datastore: Object storage service that stores and manages file system data, such as Amazon S3, Aliyun OSS, etc. It can also be served by other storage systems that are compatible with object storage semantics, such as local file systems, Ceph RADOS, TiKV, etc.
• Client: can be in various forms, such as mount process, S3 gateway, WebDAV server, Java SDK, etc.
• File: refers to all types of files in general in this documentation, including regular files, directory files, link files, device files, etc.
• Directory: is a special kind of file used to organize the tree structure, and its contents are an index to a set of other files.

Low level concepts (learn more at Data Processing Workflow):

• Chunk: Logical concept, file is split into 64MiB chunks, allowing fast lookups during file reads;
• Slice: Logical concept, basic unit for file writes. Block's purpose is to improve read speed, and slice exists to improve file edits and random writes. All file writes are assigned a new or existing slice, and when file is read, what application sees is the consolidated view of all slices.
• Block: A chunk contains one or more blocks (4MiB by default), block is the basic storage unit in object storage. JuiceFS Client reads multiple blocks concurrently which greatly improves read performance. Apart from this, block is also the basic storage unit on disk cache, so this design improves cache eviction efficiency. Apart from this, block is immutable, all file edits is achieved through new blocks: after file edit, new blocks are uploaded to object storage, and new slices are appended to the slice list in the corresponding file metadata;

Learn source code {#source-code-structure} Copy Chapter

Assuming you're already familiar with Go, as well as JuiceFS architecture, this is the overall code structure:

• cmd is the top-level entrance, all JuiceFS functionalities is rooted here, e.g. the juicefs format command resides in cmd/format.go；
• pkg is actual implementation:
  • pkg/fuse/fuse.go provides abstract FUSE API;
  • pkg/vfs contains actual FUSE implementation, Metadata requests are handled in pkg/meta, read requests are handled in pkg/vfs/reader.go and write requests are handled by pkg/vfs/writer.go;
  • pkg/meta directory is the implementation of all metadata engines, where:
    • pkg/meta/interface.go is the interface definition for all types of metadata engines
    • pkg/meta/redis.go is the interface implementation of Redis database
    • pkg/meta/sql.go is the interface definition and general interface implementation of relational database, and the implementation of specific databases is in a separate file (for example, the implementation of MySQL is in pkg/meta/sql_mysql.go)
    • pkg/meta/tkv.go is the interface definition and general interface implementation of the KV database, and the implementation of a specific database is in a separate file (for example, the implementation of TiKV is in pkg/meta/tkv_tikv.go)
  • pkg/object contains all object storage integration code;
• sdk/java is the Hadoop Java SDK, it uses sdk/java/libjfs through JNI.

FUSE interface implementation {#fuse-interface-implementation} Copy Chapter

JuiceFS implements a userspace file system based on FUSE (Filesystem in Userspace), and the implementation library libfuse provides two APIs: high-level API and low-level API, where the high-level API is based on file name and path, and the low-level API is based on inode.

JuiceFS is implemented based on low-level API (in fact JuiceFS does not depend on libfuse, but go-fuse), because this is the same set of APIs used by kernel VFS when interacting with FUSE. If JuiceFS were to use high level API, it'll have to implement the VFS tree within libfuse, and then expose path based API. This method works better for systems that already expose path based APIs (e.g. HDFS, S3). If metadata itself implements file / directory tree based on inode, the inode → path → inode conversions will have an impact on performance (this is the reason why FUSE API for HDFS doesn't perform well). JuiceFS Metadata directly implements file tree and API based on inode, so naturally it uses FUSE low level API.

Metadata Structure Copy Chapter

File systems are usually organized in a tree structure, where nodes represent files and edges represent directory containment relationships. There are more than ten metadata structures in JuiceFS. Most of them are used to maintain the organization of file tree and properties of individual nodes, while the rest are used to manage system configuration, client sessions, asynchronous tasks, etc. All metadata structures are described below.

General Structure Copy Section

Setting

It is created when the juicefs format command is executed, and some of its fields can be modified later by the juicefs config command. The structure is specified as follows.

type Format struct {
    Name             string
    UUID             string
    Storage          string
    Bucket           string
    AccessKey        string `json:",omitempty"`
    SecretKey        string `json:",omitempty"`
    SessionToken     string `json:",omitempty"`
    BlockSize        int
    Compression      string `json:",omitempty"`
    Shards           int    `json:",omitempty"`
    HashPrefix       bool   `json:",omitempty"`
    Capacity         uint64 `json:",omitempty"`
    Inodes           uint64 `json:",omitempty"`
    EncryptKey       string `json:",omitempty"`
    KeyEncrypted     bool   `json:",omitempty"`
    TrashDays        int    `json:",omitempty"`
    MetaVersion      int    `json:",omitempty"`
    MinClientVersion string `json:",omitempty"`
    MaxClientVersion string `json:",omitempty"`
    EnableACL        bool
}

• Name: name of the file system, specified by the user when formatting
• UUID: unique ID of the file system, automatically generated by the system when formatting
• Storage: short name of the object storage used to store data, such as s3, oss, etc.
• Bucket: the bucket path of the object storage
• AccessKey: access key used to access the object storage
• SecretKey: secret key used to access the object storage
• SessionToken: session token used to access the object storage, as some object storage supports the use of temporary token to obtain permission for a limited time
• BlockSize: size of the data block when splitting the file (the default is 4 MiB)
• Compression: compression algorithm that is executed before uploading data blocks to the object storage (the default is no compression)
• Shards: number of buckets in the object storage, only one bucket by default; when Shards > 1, data objects will be randomly hashed into Shards buckets
• HashPrefix: whether to set a hash prefix for the object name, false by default
• Capacity: quota limit for the total capacity of the file system
• Inodes: quota limit for the total number of files in the file system
• EncryptKey: the encrypted private key of the data object, which can be used only if the data encryption function is enabled
• KeyEncrypted: whether the saved key is encrypted or not, by default the SecretKey, EncryptKey and SessionToken will be encrypted
• TrashDays: number of days the deleted files are kept in trash, the default is 1 day
• MetaVersion: the version of the metadata structure, currently V1 (V0 and V1 are the same)
• MinClientVersion: the minimum client version allowed to connect, clients earlier than this version will be denied
• MaxClientVersion: the maximum client version allowed to connect
• EnableACL: enable ACL or not

This structure is serialized into JSON format and stored in the metadata engine.

Counter

Maintains the value of each counter in the system and the start timestamps of some background tasks, specifically

• usedSpace: used capacity of the file system
• totalInodes: number of used files in the file system
• nextInode: the next available inode number (in Redis, the maximum inode number currently in use)
• nextChunk: the next available sliceId (in Redis, the largest sliceId currently in use)
• nextSession: the maximum SID (sessionID) currently in use
• nextTrash: the maximum trash inode number currently in use
• nextCleanupSlices: timestamp of the last check on the cleanup of residual slices
• lastCleanupSessions: timestamp of the last check on the cleanup of residual stale sessions
• lastCleanupFiles: timestamp of the last check on the cleanup of residual files
• lastCleanupTrash: timestamp of the last check on the cleanup of trash

Session

Records the session IDs of clients connected to this file system and their timeouts. Each client sends a heartbeat message to update the timeout, and those who have not updated for a long time will be automatically cleaned up by other clients.

:::tip
Read-only clients cannot write to the metadata engine, so their sessions will not be recorded.
:::

SessionInfo

Records specific metadata of the client session so that it can be viewed with the juicefs status command. This is specified as

type SessionInfo struct {
    Version    string // JuiceFS version
    HostName   string // Host name
    MountPoint string // path to mount point. S3 gateway and WebDAV server are "s3gateway" and "webdav" respectively
    ProcessID  int    // Process ID
}

This structure is serialized into JSON format and stored in the metadata engine.

Node

Records attribute information of each file, as follows

type Attr struct {
    Flags     uint8  // reserved flags
    Typ       uint8  // type of a node
    Mode      uint16 // permission mode
    Uid       uint32 // owner id
    Gid       uint32 // group id of owner
    Rdev      uint32 // device number
    Atime     int64  // last access time
    Mtime     int64  // last modified time
    Ctime     int64  // last change time for meta
    Atimensec uint32 // nanosecond part of atime
    Mtimensec uint32 // nanosecond part of mtime
    Ctimensec uint32 // nanosecond part of ctime
    Nlink     uint32 // number of links (sub-directories or hardlinks)
    Length    uint64 // length of regular file

    Parent    Ino  // inode of parent; 0 means tracked by parentKey (for hardlinks)
    Full      bool // the attributes are completed or not
    KeepCache bool // whether to keep the cached page or not

    AccessACL  uint32 // access ACL id (identical ACL rules share the same access ACL ID.)
    DefaultACL uint32 // default ACL id (default ACL and the access ACL share the same cache and store)
}

There are a few fields that need clarification.

• Atime/Atimensec: See --atime-mode
• Nlink
  • Directory file: initial value is 2 ('.' and '..'), add 1 for each subdirectory
  • Other files: initial value is 1, add 1 for each hard link created
• Length
  • Directory file: fixed at 4096
  • Soft link (symbolic link) file: the string length of the path to which the link points
  • Other files: the length of the actual content of the file

This structure is usually encoded in binary format and stored in the metadata engine.

Edges

Records information on each edge in the file tree, as follows

parentInode, name -> type, inode

where parentInode is the inode number of the parent directory, and the others are the name, type, and inode number of the child files, respectively.

LinkParent

Records the parent directory of some files. The parent directory of most files is recorded in the Parent field of the attribute; however, for files that have been created with hard links, there may be more than one parent directory, so the Parent field is set to 0, and all parent inodes are recorded independently, as follows

inode -> parentInode, links

where links is the count of the parentInode, because multiple hard links can be created in the same directory, and these hard links share one inode.

Chunk

Records information on each Chunk, as follows

inode, index -> []Slices

where inode is the inode number of the file to which the Chunk belongs, and index is the number of all Chunks in the file, starting from 0. The Chunk value is an array of Slices. Each Slice represents a piece of data written by the client, and is appended to this array in the order of writing time. When there is an overlap between different Slices, the later Slice is used.

type Slice struct {
    Pos  uint32 // offset of the Slice in the Chunk
    ID   uint64 // ID of the Slice, globally unique
    Size uint32 // size of the Slice
    Off  uint32 // offset of valid data in this Slice
    Len  uint32 // size of valid data in this Slice
}

This structure is encoded and saved in binary format, taking up 24 bytes.

SliceRef {#sliceref}

Records the reference count of a Slice, as follows

sliceId, size -> refs

Since the reference count of most Slices is 1, to reduce the number of related entries in the database, the actual value minus 1 is used as the stored count value in Redis and TKV. In this way, most of the Slices have a refs value of 0, and there is no need to create related entries in the database.

Symlink

Records the location of the softlink file, as follows

inode -> target

Xattr

Records extended attributes (Key-Value pairs) of a file, as follows

inode, key -> value

Flock

Records BSD locks (flock) of a file, specifically.

inode, sid, owner -> ltype

where sid is the client session ID, owner is a string of numbers, usually associated with a process, and ltype is the lock type, which can be 'R' or 'W'.

Plock

Record POSIX record locks (fcntl) of a file, specifically

inode, sid, owner -> []plockRecord

Here plock is a more fine-grained lock that can only lock a certain segment of the file.

type plockRecord struct {
    ltype uint32 // lock type
    pid   uint32 // process ID
    start uint64 // start position of the lock
    end   uint64 // end position of the lock
}

This structure is encoded and stored in binary format, taking up 24 bytes.

DelFiles

Records the list of files to be cleaned. It is needed as data cleanup of files is an asynchronous and potentially time-consuming operation that can be interrupted by other factors.

inode, length -> expire

where length is the length of the file and expire is the time when the file was deleted.

DelSlices

Records delayed deleted Slices. When the Trash feature is enabled, old Slices deleted by the Slice Compaction will be kept for the same amount of time as the Trash configuration, to be available for data recovery if necessary.

sliceId, deleted -> []slice

where sliceId is the ID of the new slice after compaction, deleted is the timestamp of the compaction, and the mapped value is the list of all old slices that were compacted. Each slice only encodes its ID and size.

type slice struct {
    ID   uint64
    Size uint32
}

This structure is encoded and stored in binary format, taking up 12 bytes.

Sustained

Records the list of files that need to be kept temporarily during the session. If a file is still open when it is deleted, the data cannot be cleaned up immediately, but needs to be held temporarily until the file is closed.

sid -> []inode

where sid is the session ID and the mapped value is the list of temporarily undeleted file inodes.

Redis Copy Section

The common format of keys in Redis is ${prefix}${JFSKey}, where

• In standalone mode the prefix is an empty string, while in cluster mode it is a database number enclosed in curly braces, e.g. "{10}"
• JFSKey is the Key of different data structures in JuiceFS, which are listed in the subsequent subsections

In Redis Keys, integers (including inode numbers) are represented as decimal strings if not otherwise specified.

Setting {#redis-setting}

• Key: setting
• Value Type: String
• Value: file system formatting information in JSON format

Counter

• Key: counter name
• Value Type: String
• Value: value of the counter, which is actually an integer

Session

• Key: allSessions
• Value Type: Sorted Set
• Value: all non-read-only sessions connected to this file system. In Set,
  • Member: session ID
  • Score: timeout point of this session

SessionInfo

• Key: sessionInfos
• Value Type: Hash
• Value: basic meta-information on all non-read-only sessions. In Hash,
  • Key: session ID
  • Value: session information in JSON format

Node {#redis-node}

• Key: i${inode}
• Value Type: String
• Value: binary encoded file attribute

Edge {#redis-edge}

• Key: d${inode}
• Value Type: Hash
• Value: all directory entries in this directory. In Hash,
  • Key: file name
  • Value: binary encoded file type and inode number

LinkParent

• Key: p${inode}
• Value Type: Hash
• Value: all parent inodes of this file. in Hash.
  • Key: parent inode
  • Value: count of this parent inode

Chunk {#redis-chunk}

• Key: c${inode}_${index}
• Value Type: List
• Value: list of Slices, each Slice is binary encoded with 24 bytes

SliceRef

• Key: sliceRef
• Value Type: Hash
• Value: the count value of all Slices to be recorded. In Hash,
  • Key: k${sliceId}_${size}
  • Value: reference count of this Slice minus 1 (if the reference count is 1, the corresponding entry is generally not created)

Symlink

• Key: s${inode}
• Value Type: String
• Value: path that the symbolic link points to

Xattr

• Key: x${inode}
• Value Type: Hash
• Value: all extended attributes of this file. In Hash,
  • Key: name of the extended attribute
  • Value: value of the extended attribute

Flock

• Key: lockf${inode}
• Value Type: Hash
• Value: all flocks of this file. In Hash,
  • Key: ${sid}_${owner}, owner in hexadecimal
  • Value: lock type, can be 'R' or 'W'

Plock {#redis-plock}

• Key: lockp${inode}
• Value Type: Hash
• Value: all plocks of this file. In Hash,
  • Key: ${sid}_${owner}, owner in hexadecimal
  • Value: array of bytes, where every 24 bytes corresponds to a plockRecord

DelFiles

• Key：delfiles
• Value Type: Sorted Set
• Value: list of all files to be cleaned. In Set,
  • Member: ${inode}:${length}
  • Score: the timestamp when this file was added to the set

DelSlices {#redis-delslices}

• Key: delSlices
• Value Type: Hash
• Value: all Slices to be cleaned. In Hash,
  • Key: ${sliceId}_${deleted}
  • Value: array of bytes, where every 12 bytes corresponds to a slice

Sustained

• Key: session${sid}
• Value Type: List
• Value: list of files temporarily reserved in this session. In List,
  • Member: inode number of the file

SQL Copy Section

Metadata is stored in different tables by type, and each table is named with jfs_ followed by its specific structure name to form the table name, e.g. jfs_node. Some tables use Id with the bigserial type as primary keys to ensure that each table has a primary key, and the Id columns do not contain actual information.

Setting {#sql-setting}

type setting struct {
    Name  string `xorm:"pk"`
    Value string `xorm:"varchar(4096) notnull"`
}

There is only one entry in this table with "format" as Name and file system formatting information in JSON as Value.

Counter

type counter struct {
    Name  string `xorm:"pk"`
    Value int64  `xorm:"notnull"`
}

Session

type session2 struct {
    Sid    uint64 `xorm:"pk"`
    Expire int64  `xorm:"notnull"`
    Info   []byte `xorm:"blob"`
}

SessionInfo

There is no separate table for this, but it is recorded in the Info column of session2.

Node {#sql-node}

type node struct {
    Inode  Ino    `xorm:"pk"`
    Type   uint8  `xorm:"notnull"`
    Flags  uint8  `xorm:"notnull"`
    Mode   uint16 `xorm:"notnull"`
    Uid    uint32 `xorm:"notnull"`
    Gid    uint32 `xorm:"notnull"`
    Atime  int64  `xorm:"notnull"`
    Mtime  int64  `xorm:"notnull"`
    Ctime  int64  `xorm:"notnull"`
    Nlink  uint32 `xorm:"notnull"`
    Length uint64 `xorm:"notnull"`
    Rdev   uint32
    Parent Ino
    AccessACLId  uint32 `xorm:"'access_acl_id'"`
    DefaultACLId uint32 `xorm:"'default_acl_id'"`
}

Most of the fields are the same as Attr, but the timestamp precision is lower, i.e., Atime/Mtime/Ctime are in microseconds.

Edge {#sql-edge}

type edge struct {
    Id     int64  `xorm:"pk bigserial"`
    Parent Ino    `xorm:"unique(edge) notnull"`
    Name   []byte `xorm:"unique(edge) varbinary(255) notnull"`
    Inode  Ino    `xorm:"index notnull"`
    Type   uint8  `xorm:"notnull"`
}

LinkParent

There is no separate table for this. All Parents are found based on the Inode index in edge.

Chunk {#sql-chunk}

type chunk struct {
    Id     int64  `xorm:"pk bigserial"`
    Inode  Ino    `xorm:"unique(chunk) notnull"`
    Indx   uint32 `xorm:"unique(chunk) notnull"`
    Slices []byte `xorm:"blob notnull"`
}

Slices are an array of bytes, and each Slice corresponds to 24 bytes.

SliceRef

type sliceRef struct {
    Id   uint64 `xorm:"pk chunkid"`
    Size uint32 `xorm:"notnull"`
    Refs int    `xorm:"notnull"`
}

Symlink

type symlink struct {
    Inode  Ino    `xorm:"pk"`
    Target []byte `xorm:"varbinary(4096) notnull"`
}

Xattr

type xattr struct {
    Id    int64  `xorm:"pk bigserial"`
    Inode Ino    `xorm:"unique(name) notnull"`
    Name  string `xorm:"unique(name) notnull"`
    Value []byte `xorm:"blob notnull"`
}

Flock

type flock struct {
    Id    int64  `xorm:"pk bigserial"`
    Inode Ino    `xorm:"notnull unique(flock)"`
    Sid   uint64 `xorm:"notnull unique(flock)"`
    Owner int64  `xorm:"notnull unique(flock)"`
    Ltype byte   `xorm:"notnull"`
}

Plock {#sql-plock}

type plock struct {
    Id      int64  `xorm:"pk bigserial"`
    Inode   Ino    `xorm:"notnull unique(plock)"`
    Sid     uint64 `xorm:"notnull unique(plock)"`
    Owner   int64  `xorm:"notnull unique(plock)"`
    Records []byte `xorm:"blob notnull"`
}

Records is an array of bytes, and each plockRecord corresponds to 24 bytes.

DelFiles

type delfile struct {
    Inode  Ino    `xorm:"pk notnull"`
    Length uint64 `xorm:"notnull"`
    Expire int64  `xorm:"notnull"`
}

DelSlices {#sql-delslices}

type delslices struct {
    Id      uint64 `xorm:"pk chunkid"`
    Deleted int64  `xorm:"notnull"`
    Slices  []byte `xorm:"blob notnull"`
}

Slices is an array of bytes, and each slice corresponds to 12 bytes.

Sustained

type sustained struct {
    Id    int64  `xorm:"pk bigserial"`
    Sid   uint64 `xorm:"unique(sustained) notnull"`
    Inode Ino    `xorm:"unique(sustained) notnull"`
}

TKV Copy Section

The common format of keys in TKV (Transactional Key-Value Database) is ${prefix}${JFSKey}, where

• prefix is used to distinguish between different file systems, usually ${VolumeName}0xFD, where 0xFD is used as a special byte to handle cases when there is an inclusion relationship between different file system names. In addition, for databases that are not shareable (e.g. BadgerDB), the empty string is used as prefix.
• JFSKey is the JuiceFS Key for different data types, which is listed in the following subsections.

In TKV's Keys, all integers are stored in encoded binary form.

• inode and counter value occupy 8 bytes and are encoded with small endian.
• SID, sliceId and timestamp occupy 8 bytes and are encoded with big endian.

Setting {#tkv-setting}

setting -> file system formatting information in JSON format

Counter

C${name} -> counter value

Session

SE${sid} -> timestamp

SessionInfo

SI${sid} -> session information in JSON format

Node {#tkv-node}

A${inode}I -> encoded Attr

Edge {#tkv-edge}

A${inode}D${name} -> encoded {type, inode}

LinkParent

A${inode}P${parentInode} -> counter value

Chunk {#tkv-chunk}

A${inode}C${index} -> Slices

where index takes up 4 bytes and is encoded with big endian. Slices is an array of bytes, one Slice per 24 bytes.

SliceRef

K${sliceId}${size} -> counter value

where size takes up 4 bytes and is encoded with big endian.

Symlink

A${inode}S -> target

Xattr

A${inode}X${name} -> xattr value

Flock

F${inode} -> flocks

where flocks is an array of bytes, one flock per 17 bytes.

type flock struct {
    sid   uint64
    owner uint64
    ltype uint8
}

Plock {#tkv-plock}

P${inode} -> plocks

where plocks is an array of bytes and the corresponding plock is variable-length.

type plock struct {
    sid     uint64
    owner     uint64
    size     uint32
    records []byte
}

where size is the length of the records array and every 24 bytes in records corresponds to one plockRecord.

DelFiles

D${inode}${length} -> timestamp

where length takes up 8 bytes and is encoded with big endian.

DelSlices {#tkv-delslices}

L${timestamp}${sliceId} -> slices

where slices is an array of bytes, and one slice corresponds to 12 bytes.

Sustained

SS${sid}${inode} -> 1

Here the Value value is only used as a placeholder.

File Data Format Copy Chapter

Finding files by path Copy Section

According to the design of Edge, only the direct children of each directory are recorded in the metadata engine. When an application provides a path to access a file, JuiceFS needs to look it up level by level. Now suppose the application wants to open the file /dir1/dir2/testfile, then it needs to

1. search for the entry with name "dir1" in the Edge structure of the root directory (inode number is fixed to 1) and get its inode number N1
2. search for the entry with the name "dir2" in the Edge structure of N1 and get its inode number N2
3. search for the entry with the name "testfile" in the Edge structure of N2, and get its inode number N3
4. search for the Node structure corresponding to N3 to get the attributes of the file

Failure in any of the above steps will result in the file pointed to by that path not being found.

File data splitting Copy Section

From the previous section, we know how to find the file based on its path and get its attributes. The metadata related to the contents of the file can be found based on the inode and size fields in the file properties. Now suppose a file has an inode of 100 and a size of 160 MiB, then the file has (size-1) / 64 MiB + 1 = 3 Chunks, as follows.

File: |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _|
Chunk: |<---        Chunk 0        --->|<---        Chunk 1        --->|<-- Chunk 2 -->|

In standalone Redis, this means that there are 3 Chunk Keys, i.e.,c100_0, c100_1 and c100_2, each corresponding to a list of Slices. These Slices are mainly generated when the data is written and may overwrite each other or may not fill the Chunk completely, so you need to traverse this list of Slices sequentially and reconstruct the latest version of the data distribution before using it, so that

1. the part covered by more than one Slice is based on the last added Slice
2. the part that is not covered by Slice is automatically zeroed, and is represented by sliceId = 0
3. truncate Chunk according to file size

Now suppose there are 3 Slices in Chunk 0

Slice{pos: 10M, id: 10, size: 30M, off: 0, len: 30M}
Slice{pos: 20M, id: 11, size: 16M, off: 0, len: 16M}
Slice{pos: 16M, id: 12, size: 10M, off: 0, len: 10M}

It can be illustrated as follows (each '_' denotes 2 MiB)

Chunk: |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Slice 10:           |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Slice 11:                     |_ _ _ _ _ _ _ _|
Slice 12:                 |_ _ _ _ _|

New List: |_ _ _ _ _|_ _ _|_ _ _ _ _|_ _ _ _ _|_ _|_ _ _ _ _ _ _ _ _ _ _ _|
               0      10      12         11    10             0

The reconstructed new list contains and only contains the latest data distribution for this Chunk as follows

Slice{pos:   0, id:  0, size: 10M, off:   0, len: 10M}
Slice{pos: 10M, id: 10, size: 30M, off:   0, len:  6M}
Slice{pos: 16M, id: 12, size: 10M, off:   0, len: 10M}
Slice{pos: 26M, id: 11, size: 16M, off:  6M, len: 10M}
Slice{pos: 36M, id: 10, size: 30M, off: 26M, len:  4M}
Slice{pos: 40M, id:  0, size: 24M, off:   0, len: 24M} // can be omitted

Data objects Copy Section

Object naming {#object-storage-naming-format}

Block is the basic unit for JuiceFS to manage data. Its size is 4 MiB by default, and can be changed only when formatting a file system, within the interval [64 KiB, 16 MiB]. Each Block is an object in the object storage after upload, and is named in the format ${fsname}/chunks/${hash}/${basename}, where

• fsname is the file system name
• "chunks" is a fixed string representing the data object of JuiceFS
• hash is the hash value calculated from basename, which plays a role in isolation management
• basename is the valid name of the object in the format of ${sliceId}_${index}_${size}, where
  • sliceId is the ID of the Slice to which the object belongs, and each Slice in JuiceFS has a globally unique ID
  • index is the index of the object in the Slice it belongs to, by default a Slice can be split into at most 16 Blocks, so its value range is [0, 16)
  • size is the size of the Block, and by default it takes the value of (0, 4 MiB]

Currently there are two hash algorithms, and both use the sliceId in basename as the parameter. Which algorithm will be chosen to use follows the HashPrefix of the file system.

func hash(sliceId int) string {
    if HashPrefix {
        return fmt.Sprintf("%02X/%d", sliceId%256, sliceId/1000/1000)
    }
    return fmt.Sprintf("%d/%d", sliceId/1000/1000, sliceId/1000)
}

Suppose a file system named jfstest is written with a continuous 10 MiB of data and internally given a SliceID of 1 with HashPrefix disabled, then the following three objects will be generated in the object storage.

jfstest/chunks/0/0/1_0_4194304
jfstest/chunks/0/0/1_1_4194304
jfstest/chunks/0/0/1_2_2097152

Similarly, now taking the 64 MiB chunk in the previous section as an example, its actual data distribution is as follows

0 ~ 10M: Zero
10 ~ 16M: 10_0_4194304, 10_1_4194304(0 ~ 2M)
16 ~ 26M: 12_0_4194304, 12_1_4194304, 12_2_2097152
26 ~ 36M: 11_1_4194304(2 ~ 4M), 11_2_4194304, 11_3_4194304
36 ~ 40M: 10_6_4194304(2 ~ 4M), 10_7_2097152
40 ~ 64M: Zero

According to this, the client can quickly find the data needed for the application. For example, reading 8 MiB data at offset 10 MiB location will involve 3 objects, as follows

• Read the entire object from 10_0_4194304, corresponding to 0 to 4 MiB of the read data
• Read 0 to 2 MiB from 10_1_4194304, corresponding to 4 to 6 MiB of the read data
• Read 0 to 2 MiB from 12_0_4194304, corresponding to 6 to 8 MiB of the read data

To facilitate obtaining the list of objects of a certain file, JuiceFS provides the info command, e.g. juicefs info /mnt/jfs/test.tmp.

objects:
+------------+---------------------------------+----------+---------+----------+
| chunkIndex |            objectName           |   size   |  offset |  length  |
+------------+---------------------------------+----------+---------+----------+
|          0 |                                 | 10485760 |       0 | 10485760 |
|          0 | jfstest/chunks/0/0/10_0_4194304 |  4194304 |       0 |  4194304 |
|          0 | jfstest/chunks/0/0/10_1_4194304 |  4194304 |       0 |  2097152 |
|          0 | jfstest/chunks/0/0/12_0_4194304 |  4194304 |       0 |  4194304 |
|          0 | jfstest/chunks/0/0/12_1_4194304 |  4194304 |       0 |  4194304 |
|          0 | jfstest/chunks/0/0/12_2_2097152 |  2097152 |       0 |  2097152 |
|          0 | jfstest/chunks/0/0/11_1_4194304 |  4194304 | 2097152 |  2097152 |
|          0 | jfstest/chunks/0/0/11_2_4194304 |  4194304 |       0 |  4194304 |
|          0 | jfstest/chunks/0/0/11_3_4194304 |  4194304 |       0 |  4194304 |
|          0 | jfstest/chunks/0/0/10_6_4194304 |  4194304 | 2097152 |  2097152 |
|          0 | jfstest/chunks/0/0/10_7_2097152 |  2097152 |       0 |  2097152 |
|        ... |                             ... |      ... |     ... |      ... |
+------------+---------------------------------+----------+---------+----------+

The empty objectName in the table means a file hole and is read as 0. As you can see, the output is consistent with the previous analysis.

It is worth mentioning that the 'size' here is size of the original data in the Block, rather than that of the actual object in object storage. The original data is written directly to object storage by default, so the 'size' is equal to object size. However, when data compression or data encryption is enabled, the size of the actual object will change and may no longer be the same as the 'size'.

Data compression

You can configure the compression algorithm (supporting lz4 and zstd) with the --compress <value> parameter when formatting a file system, so that all data blocks of this file system will be compressed before uploading to object storage. The object name remains the same as default, and the content is the result of the compression algorithm, without any other meta information. Therefore, the compression algorithm in the file system formatting Information is not allowed to be modified, otherwise it will cause the failure of reading existing data.

Data encryption

The RSA private key can be configured to enable static data encryption when formatting a file system with the --encrypt-rsa-key <value> parameter, which allows all data blocks of this file system to be encrypted before uploading to the object storage. The object name is still the same as default, while its content becomes a header plus the result of the data encryption algorithm. The header contains a random seed and the symmetric key used for decryption, and the symmetric key itself is encrypted with the RSA private key. Therefore, it is not allowed to modify the RSA private key in the file system formatting Information, otherwise reading existing data will fail.

:::note
If both compression and encryption are enabled, the original data will be compressed and then encrypted before uploading to the object storage.
:::

File: docs/en/deployment/automation.md Copy Chapter

title: Automated Deployment sidebar_position: 7 Copy Chapter

Automated deployment is recommended when JuiceFS Client is to be installed on a large number of hosts.

Below examples only demonstrate the mount process, you should Create a file system before getting started.

Ansible Copy Chapter

Below is the Ansible example to install and mount JuiceFS in localhost:

- hosts: localhost
  tasks:
    - set_fact:
        # Change accordingly
        meta_url: sqlite3:///tmp/myjfs.db
        jfs_path: /jfs
        jfs_pkg: /tmp/juicefs-ce.tar.gz
        jfs_bin_dir: /usr/local/bin

    - get_url:
        # Change download URL accordingly
        url: https://d.juicefs.com/juicefs/releases/download/v1.0.2/juicefs-1.0.2-linux-amd64.tar.gz
        dest: "{{jfs_pkg}}"

    - ansible.builtin.unarchive:
        src: "{{jfs_pkg}}"
        dest: "{{jfs_bin_dir}}"
        include:
          - juicefs

    - name: Create symbolic for fstab
      ansible.builtin.file:
        src: "{{jfs_bin_dir}}/juicefs"
        dest: "/sbin/mount.juicefs"
        state: link

    - name: Mount JuiceFS and create fstab entry
      mount:
        path: "{{jfs_path}}"
        src: "{{meta_url}}"
        fstype: juicefs
        opts: _netdev
        state: mounted

File: docs/en/deployment/hadoop_java_sdk.md Copy Chapter

title: Use JuiceFS on Hadoop Ecosystem sidebar_position: 3 slug: /hadoop_java_sdk Copy Chapter

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

JuiceFS provides Hadoop-compatible File System by Hadoop Java SDK. Various applications in the Hadoop ecosystem can smoothly use JuiceFS to store data without changing the code.

Requirements Copy Chapter

1. Hadoop and related components Copy Section

JuiceFS Hadoop Java SDK is compatible with Hadoop 2.x and Hadoop 3.x. As well as variety of components in Hadoop ecosystem.

2. User permissions Copy Section

JuiceFS uses local "User/UID" and "Group/GID" mappings by default, and when used in a distributed environment, to avoid permission issues, please refer to documentation synchronizes the "User/UID" and "Group/GID" that needs to be used to all Hadoop nodes. It is also possible to define a global user and group file to make all nodes in the cluster share the permission configuration. Please see here for related configurations.

3. File system Copy Section

You should first create at least one JuiceFS file system to provide storage for components related to the Hadoop ecosystem through the JuiceFS Java SDK. When deploying the Java SDK, specify the metadata engine address of the created file system in the configuration file.

To create a file system, please refer to our quick start.

:::note
If you want to use JuiceFS in a distributed environment, when creating a file system, please plan the object storage and database to be used reasonably to ensure that they can be accessed by each node in the cluster.
:::

4. Memory Copy Section

Depending on the read and write load of computing tasks (such as Spark executor), JuiceFS Hadoop Java SDK may require an additional 4 * juicefs.memory-size off-heap memory to speed up read and write performance. By default, it is recommended to configure at least 1.2GB of off-heap memory for compute tasks.

5. Java runtime version Copy Section

JuiceFS Hadoop Java SDK is compiled with JDK 8 by default. If it needs to be used in a higher version of Java runtime (such as Java 17), the following options need to be added to the JVM parameters to allow the use of reflection API:

--add-exports=java.base/sun.nio.ch=ALL-UNNAMED

For more information on the above option, please refer to official documentation.

Install and compile the client Copy Chapter

Install the pre-compiled client Copy Section

Please refer to the "Installation" document to learn how to download the precompiled JuiceFS Hadoop Java SDK.

Compile the client manually Copy Section

:::note
No matter which system environment the client is compiled for, the compiled JAR file has the same name and can only be deployed in the matching system environment. For example, when compiled in Linux, it can only be used in the Linux environment. In addition, since the compiled package depends on glibc, it is recommended to compile with a lower version system to ensure better compatibility.
:::

Compilation depends on the following tools:

• Go 1.20+
• JDK 8+
• Maven 3.3+
• Git
• make
• GCC 5.4+

Linux and macOS

Clone the repository:

git clone https://github.com/juicedata/juicefs.git

Enter the directory and compile:

cd juicefs/sdk/java
make

:::note
If Ceph RADOS is used to store data, you need to install librados-dev first and [build libjfs.so]`.
:::

cd juicefs/sdk/java
make ceph

After the compilation, you can find the compiled JAR file in the sdk/java/target directory, including two versions:

• Contains third-party dependent packages: juicefs-hadoop-X.Y.Z.jar
• Does not include third-party dependent packages: original-juicefs-hadoop-X.Y.Z.jar

It is recommended to use a version that includes third-party dependencies.

Windows

The client used in the Windows environment needs to be obtained through cross-compilation on Linux or macOS. The compilation depends on mingw-w64, which needs to be installed first.

The steps are the same as compiling on Linux or macOS. For example, on the Ubuntu system, install the mingw-w64 package first to solve the dependency problem:

sudo apt install mingw-w64

Clone and enter the JuiceFS source code directory, execute the following code to compile:

cd juicefs/sdk/java

make win

Deploy the client Copy Chapter

To enable each component of the Hadoop ecosystem to correctly identify JuiceFS, the following configurations are required:

1. Place the compiled JAR file and $JAVA_HOME/lib/tools.jar into the classpath of the component. The installation paths of common big data platforms and components are shown in the table below.
2. Put JuiceFS configurations into the configuration file of each Hadoop ecosystem component (usually core-site.xml), see Client Configurations for details.

It is recommended to place the JAR file in a fixed location, and the other locations are called it through symbolic links.

Big Data Platforms Copy Section

Community Components Copy Section

Client Configurations Copy Section

Please refer to the following table to set the relevant parameters of the JuiceFS file system and write it into the configuration file, which is generally core-site.xml.

Core Configurations

Cache Configurations

I/O Configurations

Other Configurations

Multiple file systems configuration

When multiple JuiceFS file systems need to be used at the same time, all the above configuration items can be specified for a specific file system. You only need to put the file system name in the middle of the configuration item, such as jfs1 and jfs2 in the following example:

<name>juicefs.jfs1.meta</name>
  <value>redis://jfs1.host:port/1</value>


  <name>juicefs.jfs2.meta</name>
  <value>redis://jfs2.host:port/1</value>

Configuration Example

The following is a commonly used configuration example. Please replace the {HOST}, {PORT} and {DB} variables in the juicefs.meta configuration with actual values.

<name>fs.jfs.impl</name>
  <value>io.juicefs.JuiceFileSystem</value>


  <name>fs.AbstractFileSystem.jfs.impl</name>
  <value>io.juicefs.JuiceFS</value>


  <name>juicefs.meta</name>
  <value>redis://{HOST}:{PORT}/{DB}</value>


  <name>juicefs.cache-dir</name>
  <value>/data*/jfs</value>


  <name>juicefs.cache-size</name>
  <value>1024</value>


  <name>juicefs.access-log</name>
  <value>/tmp/juicefs.access.log</value>

Configuration in Hadoop Copy Chapter

Please refer to the aforementioned configuration tables and add configuration parameters to the Hadoop configuration file core-site.xml.

CDH6 Copy Section

If you are using CDH 6, in addition to modifying core-site, you also need to modify mapreduce.application.classpath through the YARN service interface, adding:

$HADOOP_COMMON_HOME/lib/juicefs-hadoop.jar

HDP Copy Section

In addition to modifying core-site, you also need to modify the configuration mapreduce.application.classpath through the MapReduce2 service interface and add it at the end (variables do not need to be replaced):

/usr/hdp/${hdp.version}/hadoop/lib/juicefs-hadoop.jar

Flink Copy Section

Add configuration parameters to conf/flink-conf.yaml. If you only use JuiceFS in Flink, you don't need to configure JuiceFS in the Hadoop environment, you only need to configure the Flink client.

Hudi Copy Section

:::note
Hudi supports JuiceFS since v0.10.0, please make sure you are using the correct version.
:::

Please refer to "Hudi Official Documentation" to learn how to configure JuiceFS.

Kafka Connect Copy Section

It is possible to use Kafka Connect and HDFS Sink Connector（HDFS 2 and HDFS 3）to store data on JuiceFS.

First you need to add JuiceFS SDK to classpath in Kafka Connect, e.g., /usr/share/java/confluentinc-kafka-connect-hdfs/lib.

While creating a Connect Sink task, configuration needs to be set up as follows:

• Specify hadoop.conf.dir as the directory that contains the configuration file core-site.xml. If it is not running in Hadoop environment, you can create a separate directory such as /usr/local/juicefs/hadoop, and then add the JuiceFS related configurations to core-site.xml.
• Specify store.url as a path starting with jfs://.

For example:

# Other configuration items are omitted.
hadoop.conf.dir=/path/to/hadoop-conf
store.url=jfs://path/to/store

HBase Copy Section

JuiceFS can be used by HBase for HFile, but is not fast (low latency) enough for Write Ahead Log (WAL), because it take much longer time to persist data into object storage than memory of DataNode.

It is recommended to deploy a small HDFS cluster to store WAL and HFile files to be stored on JuiceFS.

Create a new HBase cluster

Modify hbase-site.xml:

<name>hbase.rootdir</name>
  <value>jfs://{vol_name}/hbase</value>


  <name>hbase.wal.dir</name>
  <value>hdfs://{ns}/hbase-wal</value>

Modify existing HBase cluster

In addition to modifying the above configurations, since the HBase cluster has already stored some data in ZooKeeper, in order to avoid conflicts, there are two solutions:

1. Delete the old cluster

   Delete the znode (default /hbase) configured by zookeeper.znode.parent via the ZooKeeper client.

   :::note
   This operation will delete all data on this HBase cluster.
   :::

2. Use a new znode

   Keep the znode of the original HBase cluster so that it can be recovered later. Then configure a new value for zookeeper.znode.parent:

   xml title="hbase-site.xml" Copy

   <name>zookeeper.znode.parent</name>
     <value>/hbase-jfs</value>

Restart Services Copy Section

When the following components need to access JuiceFS, they should be restarted.

:::note
Before restart, you need to confirm JuiceFS related configuration has been written to the configuration file of each component, usually you can find them in core-site.xml on the machine where the service of the component was deployed.
:::

HDFS, Hue, ZooKeeper and other services don't need to be restarted.

When Class io.juicefs.JuiceFileSystem not found or No FilesSystem for scheme: jfs exceptions was occurred after restart, reference FAQ.

Trash Copy Section

JuiceFS Hadoop Java SDK also has the same trash function as HDFS, which needs to be enabled by setting fs.trash.interval and fs.trash.checkpoint.interval, please refer to HDFS documentation for more information.

Environmental Verification Copy Chapter

After the deployment of the JuiceFS Java SDK, the following methods can be used to verify the success of the deployment.

Hadoop CLI Copy Section

hadoop fs -ls jfs://{JFS_NAME}/

:::info
The JFS_NAME is the volume name when you format JuiceFS file system.
:::

Hive Copy Section

CREATE TABLE IF NOT EXISTS person
(
  name STRING,
  age INT
) LOCATION 'jfs://{JFS_NAME}/tmp/person';

Java/Scala project Copy Section

1. Add Maven or Gradle dependencies:

   xml Copy

   <dependency>
       <groupId>org.apache.hadoop</groupId>
       <artifactId>hadoop-common</artifactId>
       <version>{HADOOP_VERSION}</version>
       <scope>provided</scope>
   </dependency>
   <dependency>
       <groupId>io.juicefs</groupId>
       <artifactId>juicefs-hadoop</artifactId>
       <version>{JUICEFS_HADOOP_VERSION}</version>
       <scope>provided</scope>
   </dependency>

   groovy Copy

   dependencies {
     implementation 'org.apache.hadoop:hadoop-common:${hadoopVersion}'
     implementation 'io.juicefs:juicefs-hadoop:${juicefsHadoopVersion}'
   }

2. Use the following sample code to verify:

   java Copy

   package demo;
   
   import org.apache.hadoop.conf.Configuration;
   import org.apache.hadoop.fs.FileStatus;
   import org.apache.hadoop.fs.FileSystem;
   import org.apache.hadoop.fs.Path;
   
   public class JuiceFSDemo {
       public static void main(String[] args) throws Exception {
           Configuration conf = new Configuration();
           conf.set("fs.jfs.impl", "io.juicefs.JuiceFileSystem");
           conf.set("juicefs.meta", "redis://127.0.0.1:6379/0");  // JuiceFS metadata engine URL
           Path p = new Path("jfs://{JFS_NAME}/");  // Please replace "{JFS_NAME}" with the correct value
           FileSystem jfs = p.getFileSystem(conf);
           FileStatus[] fileStatuses = jfs.listStatus(p);
           // Traverse JuiceFS file system and print file paths
           for (FileStatus status : fileStatuses) {
               System.out.println(status.getPath());
           }
       }
   }

Monitoring metrics collection Copy Chapter

Please see the "Monitoring" documentation to learn how to collect and display JuiceFS monitoring metrics.

Benchmark Copy Chapter

Here are a series of methods to use the built-in stress testing tool of the JuiceFS client to test the performance of the client environment that has been successfully deployed.

1. Local Benchmark Copy Section

Metadata

• create

  shell Copy

  hadoop jar juicefs-hadoop.jar nnbench create -files 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench -local

  This command will create 10000 empty files

• open

  shell Copy

  hadoop jar juicefs-hadoop.jar nnbench open -files 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench -local

  This command will open 10000 files without reading data

• rename

  shell Copy

  hadoop jar juicefs-hadoop.jar nnbench rename -files 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench -local

• delete

  shell Copy

  hadoop jar juicefs-hadoop.jar nnbench delete -files 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench -local

• For reference

  Operation	TPS	Latency (ms)
  create	644	1.55
  open	3467	0.29
  rename	483	2.07
  delete	506	1.97

I/O Performance

• sequential write

  shell Copy

  hadoop jar juicefs-hadoop.jar dfsio -write -size 20000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/DFSIO -local

• sequential read

  shell Copy

  hadoop jar juicefs-hadoop.jar dfsio -read -size 20000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/DFSIO -local

  When run the cmd for the second time, the result may be much better than the first run. It's because the data was cached in memory, just clean the local disk cache.

• For reference

  Operation	Throughput (MB/s)
  write	647
  read	111

If the network bandwidth of the machine is relatively low, it can generally reach the network bandwidth bottleneck.

2. Distributed Benchmark Copy Section

The following command will start the MapReduce distributed task to test the metadata and IO performance. During the test, it is necessary to ensure that the cluster has sufficient resources to start the required map tasks.

Computing resources used in this test:

• Server: 4 cores and 32 GB memory, burst bandwidth 5Gbit/s x 3
• Database: Alibaba Cloud Redis 5.0 Community 4G Master-Slave Edition

Metadata

• create

  shell Copy

  hadoop jar juicefs-hadoop.jar nnbench create -maps 10 -threads 10 -files 1000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench

  10 map task, each has 10 threads, each thread create 1000 empty file. 100000 files in total

• open

  shell Copy

  hadoop jar juicefs-hadoop.jar nnbench open -maps 10 -threads 10 -files 1000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench

  10 map task, each has 10 threads, each thread open 1000 file. 100000 files in total

• rename

  shell Copy

  hadoop jar juicefs-hadoop.jar nnbench rename -maps 10 -threads 10 -files 1000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench

  10 map task, each has 10 threads, each thread rename 1000 file. 100000 files in total

• delete

  shell Copy

  hadoop jar juicefs-hadoop.jar nnbench delete -maps 10 -threads 10 -files 1000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/NNBench

  10 map task, each has 10 threads, each thread delete 1000 file. 100000 files in total

• For reference

  • 10 threads

    Operation	IOPS	Latency (ms)
    create	4178	2.2
    open	9407	0.8
    rename	3197	2.9
    delete	3060	3.0

  • 100 threads

    Operation	IOPS	Latency (ms)
    create	11773	7.9
    open	34083	2.4
    rename	8995	10.8
    delete	7191	13.6

I/O Performance

• sequential write

  shell Copy

  hadoop jar juicefs-hadoop.jar dfsio -write -maps 10 -size 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/DFSIO

  10 map task, each task write 10000MB random data sequentially

• sequential read

  shell Copy

  hadoop jar juicefs-hadoop.jar dfsio -read -maps 10 -size 10000 -baseDir jfs://{JFS_NAME}/tmp/benchmarks/DFSIO

  10 map task, each task read 10000MB random data sequentially

• For reference

  Operation	Average throughput (MB/s)	Total Throughput (MB/s)
  write	198	1835
  read	124	1234

3. TPC-DS Copy Section

The test dataset is 100GB in size, and both Parquet and ORC file formats are tested.

This test only tests the first 10 queries.

Spark Thrift JDBC/ODBC Server is used to start the Spark resident process and then submit the task via Beeline connection.

Test Hardware

Software Configuration

Spark Thrift JDBC/ODBC Server

${SPARK_HOME}/sbin/start-thriftserver.sh \
  --master yarn \
  --driver-memory 8g \
  --executor-memory 10g \
  --executor-cores 3 \
  --num-executors 3 \
  --conf spark.locality.wait=100 \
  --conf spark.sql.crossJoin.enabled=true \
  --hiveconf hive.server2.thrift.port=10001

JuiceFS Cache Configurations

The 2 data disks of Core node are mounted in the /data01 and /data02 directories, and core-site.xml is configured as follows:

<name>juicefs.cache-size</name>
  <value>200000</value>


  <name>juicefs.cache-dir</name>
  <value>/data*/jfscache</value>


  <name>juicefs.cache-full-block</name>
  <value>false</value>


  <name>juicefs.discover-nodes-url</name>
  <value>yarn</value>


  <name>juicefs.attr-cache</name>
  <value>3</value>


  <name>juicefs.entry-cache</name>
  <value>3</value>


  <name>juicefs.dir-entry-cache</name>
  <value>3</value>

Test

The task submission command is as follows:

${SPARK_HOME}/bin/beeline -u jdbc:hive2://localhost:10001/${DATABASE} \
  -n hadoop \
  -f query{i}.sql

Results

JuiceFS can use local disk as a cache to accelerate data access, the following data is the result (in seconds) after 4 runs using Redis and TiKV as the metadata engine of JuiceFS respectively.

ORC

Parquet

Permission control by Apache Ranger(from v1.3) Copy Chapter

JuiceFS currently supports path permission control by integrating with Apache Ranger's HDFS module. Only supported in Hadoop Java SDK.

1. Configurations Copy Section

The config for Apache Ranger is sotred in the metadata database. You can enable Ranger permission control by the following methods:

# configure with format
juicefs format META-URL NAME --ranger-rest-url http://localhost:6080 --ranger-service jfs

# or configure with config
juicefs config META-URL --ranger-rest-url http://localhost:6080 --ranger-service jfs

# disable ranger
juicefs config META-URL --ranger-rest-url "" --ranger-service jfs ""

2. Dependencies Copy Section

Considering the convenience of use, JuiceFS packages all Ranger dependencies into the JuiceFS SDK. If you encounter version conflicts with Apache Ranger, you may need to modify the version and recompile.

3. Configure Kerberos Copy Section

When Kerberos is enabled, the JuiceFS client used by business applications cannot download Ranger security policies. You need to configure policy.download.auth.users and tag.download.auth.users in the Ranger Admin Web UI under the HDFS Service to grant specified users the permission to download security policies. Multiple users should be separated by commas (,).

After completing the configuration, you must use that user to update the policies stored in JuiceFS.

The command to update the security policy file is as follows (it is recommended to configure it as a scheduled task for automatic updates). Replace {PRINCIPAL} with the user configured in policy.download.auth.users:

hadoop jar juicefs-hadoop-{version}.jar ranger \
  --fs jfs://{VOL_NAME}/ \
  --keytab /path/to/keytab \
  --principal {PRINCIPAL}

4. Tips Copy Section

4.1 Ranger version

The code is tested on Ranger2.3 and Ranger2.4. As no other features are used except for HDFS module authentication, theoretically all other versions are applicable.

4.2 Ranger Audit

Currently, only support authentication function, and the Ranger Audit is disabled.

4.3 Ranger's other parameters

To improve usage efficiency, currently only support some CORE parameters of Ranger.

4.4 Security tips

Due to the complete open source of the project, it is unavoidable for users to disrupt permission control by replacing parameters such as ranger-rest-url. If stricter control is required, it is recommended to compile the code independently and solve the problem by encrypting relevant security parameters.

Kerberos support (from v1.4) Copy Chapter

JuiceFS supports Kerberos authentication, but only for the Hadoop Java SDK.

1. Configurations Copy Section

Kerberos configuration is stored in the metadata database and can be managed via the JuiceFS CLI.

# Enable Kerberos when formatting
juicefs format META-URL --kerberos-config-file kerb.cfg
# Or enable Kerberos on an existing filesystem
./juicefs config META-URL --kerberos-config-file kerb.cfg

# Disable Kerberos
./juicefs config META-URL --kerberos-config-file ""

The Kerberos configuration file:

# Kerberos keytab, encoded with BASE64
# base64 -w 0 meta.keytab
{VOL_NAME}.keytab={BASE64 KEYTAB}
# Delegation token
{VOL_NAME}.token.life=604800
{VOL_NAME}.token.renew=86400

# Superuser and supergroup
{VOL_NAME}.superuser=hadoop
{VOL_NAME}.supergroup=supergroup

# Mapping from Kerberos principals to OS user accounts
# https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Mapping_from_Kerberos_principals_to_OS_user_accounts
{VOL_NAME}.mechanism=hadoop
{VOL_NAME}.rule=DEFAULT

# Proxy user settings
# https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Proxy_user
# Users: user1,user2 or *
{VOL_NAME}.proxy.client.users=foo
# Groups: group1,group2 or *
{VOL_NAME}.proxy.client.groups=foogrp
# Hosts: host1,host2 or 192.168.1.1,192.168.1.2 or 192.168.1.1/32 or *
{VOL_NAME}.proxy.client.hosts=*

core-site.xml configuration:

<name>juicefs.server-principal</name>
  <value>{YOUR_SPN}</value>

FAQ Copy Chapter

1. `Class io.juicefs.JuiceFileSystem not found` exception Copy Section

It means JAR file was not loaded, you can verify it by lsof -p {pid} | grep juicefs.

You should check whether the JAR file was located properly, or other users have the read permission.

Some Hadoop distribution also need to modify mapred-site.xml and put the JAR file location path to the end of the parameter mapreduce.application.classpath.

2. `No FilesSystem for scheme: jfs` exception Copy Section

It means JuiceFS Hadoop Java SDK was not configured properly, you need to check whether there is JuiceFS related configuration in the core-site.xml of the component configuration.

3. What are the similarities and differences between user permission management in JuiceFS and HDFS? Copy Section

JuiceFS also uses the "User/Group" method to manage file permissions, using local users and groups by default. In order to ensure the unified permissions of different nodes during distributed computing, you can configure global "User/UID" and "Group/GID" mappings through juicefs.users and juicefs.groups configurations.

4. After the data is deleted, it is directly stored in the `.trash` directory of JuiceFS. Although the files are all there, it is difficult to restore the data through the `mv` command as easily as HDFS. Is there any way to achieve a similar effect of HDFS trash? Copy Section

In the Hadoop application scenario, the functions similar to the HDFS trash are still retained. It needs to be explicitly enabled by fs.trash.interval and fs.trash.checkpoint.interval configurations, please refer to document for more information.

5. What are the benefits of setting the `juicefs.discover-nodes-url` configuration? Copy Section

In HDFS, each data block will have BlockLocation information, which the computing engine uses to schedule the computing tasks as much as possible to the nodes where the data is stored. JuiceFS will calculate the corresponding BlockLocation for each data block through the consistent hashing algorithm, so that when the same data is read for the second time, the computing engine may schedule the computing task to the same node, and the data cached on the local disk during the first computing can be used to accelerate data access.

This algorithm needs to know all the computing node information in advance. The juicefs.discover-nodes-url configuration is used to obtain these computing node information.

6. Does the community version of JuiceFS currently support a Kerberos-authenticated CDH cluster? Copy Section

Not supported. JuiceFS does not verify the validity of Kerberos users, but can use Kerberos-authenticated username.

File: docs/en/deployment/how_to_use_on_kubernetes.md Copy Chapter

title: Use JuiceFS on Kubernetes sidebar_position: 2 slug: /how_to_use_on_kubernetes Copy Chapter

JuiceFS is an ideal storage layer for Kubernetes, read this chapter to learn how to use JuiceFS in Kubernetes.

Use JuiceFS via `hostPath` Copy Chapter

If you simply need to use JuiceFS inside Kubernetes pods, without any special requirements (e.g. isolation, permission control), then hostPath can be a good practice, which is also really easy to setup:

1. Install and mount JuiceFS on all Kubernetes worker nodes, Automated Deployment is recommended for this type of work.

2. Use hostPath volume inside pod definition, and mount a JuiceFS sub-directory to container:

   yaml {8-16} Copy

   apiVersion: v1
   kind: Pod
   metadata:
     name: juicefs-app
   spec:
     containers:
       - ...
         volumeMounts:
           - name: jfs-data
             mountPath: /opt/app-data
     volumes:
       - name: jfs-data
         hostPath:
           # Assuming JuiceFS is mounted on /jfs
           path: "/jfs/myapp/"
           type: Directory

In comparison to using JuiceFS CSI Driver, hostPath is a much more simple practice, and easier to debug when things go wrong, but notice that:

• For ease of management, generally all pods use the same host mount point. Lack of isolation may lead to data security issues, and obviously, you won't be able to adjust JuiceFS mount parameters separately for each application. Please evaluate carefully.

• All worker nodes should mount JuiceFS in advance, so when adding a new node to the cluster, JuiceFS needs to be installed and mounted during the initialization process, otherwise the new node does not have a JuiceFS mount point, and the container will not be created.

• The system resources (such as CPU, memory, etc.) occupied by the JuiceFS mounting process on the host are not controlled by Kubernetes, and may occupy too many host resources. You can consider using system-reserved to properly adjust the system resource reservation of Kubernetes, to reserve more resources for the JuiceFS mount process.

• If the JuiceFS mount process on the host exits unexpectedly, the application pod will not be able to access the mount point normally. In this case, the JuiceFS file system needs to be remounted and the application pod must be rebuilt. However, JuiceFS CSI Driver solves this problem well by providing the Automatic Mount Point Recovery mechanism.

• If you're using Docker as Kubernetes container runtime, it's best to start JuiceFS mount prior to Docker in startup order, to avoid containers being created before JuiceFS is properly mounted. For systemd, you can use below unit file to manually control startup order:

  systemd title="/etc/systemd/system/docker.service.d/override.conf" Copy

  [Unit]
  # Use below command to obtain JuiceFS mount service name
  # systemctl list-units | grep "\.mount"
  After=network-online.target firewalld.service containerd.service jfs.mount

JuiceFS CSI Driver Copy Chapter

To use JuiceFS in Kubernetes, refer to JuiceFS CSI Driver Documentation.

Mount JuiceFS in the container Copy Chapter

In some cases, you may need to mount JuiceFS volume directly in the container, which requires the use of the JuiceFS client in the container. You can refer to the following Dockerfile example to integrate the JuiceFS client into your application image:

FROM alpine:latest
LABEL maintainer="Juicedata <https://juicefs.com>"

# Install JuiceFS client
RUN apk add --no-cache curl && \
  JFS_LATEST_TAG=$(curl -s https://api.github.com/repos/juicedata/juicefs/releases/latest | grep 'tag_name' | cut -d '"' -f 4 | tr -d 'v') && \
  wget "https://github.com/juicedata/juicefs/releases/download/v${JFS_LATEST_TAG}/juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \
  tar -zxf "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \
  install juicefs /usr/bin && \
  rm juicefs "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \
  rm -rf /var/cache/apk/* && \
  apk del curl

ENTRYPOINT ["/usr/bin/juicefs", "mount"]

Since JuiceFS needs to use the FUSE device to mount the file system, it is necessary to allow the container to run in privileged mode when creating a Pod:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-run
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: linuxserver/nginx
          ports:
            - containerPort: 80
          securityContext:
            privileged: true

:::caution
With the privileged mode being enabled by privileged: true, the container has access to all devices of the host, that is, it has full control of the host's kernel. Improper uses will bring serious safety hazards. Please conduct a thorough safety assessment before using it.
:::

File: docs/en/deployment/juicefs_on_docker.md Copy Chapter

title: Using JuiceFS in Docker sidebar_position: 6 slug: /juicefs_on_docker Copy Chapter

You can use the JuiceFS file system in Docker by running the client directly in the container or using a volume plugin.

Using a volume plugin {#volume-plugin} Copy Chapter

If you have specific requirements for mount management, such as managing mount points through Docker to facilitate different application containers using different JuiceFS file systems, you can use a Docker volume plugin.

Docker plugins are usually provided in the form of images. The JuiceFS volume plugin image contains the JuiceFS Community Edition and JuiceFS Cloud Service clients. After installation, you can run the volume plugin to create JuiceFS volumes in Docker.

Install the plugin using the following command and provide the necessary permissions for FUSE as prompted:

docker plugin install juicedata/juicefs

You can use the following commands to manage the volume plugin:

# Disable the plugin
docker plugin disable juicedata/juicefs

# Upgrade the plugin (must be disabled first)
docker plugin upgrade juicedata/juicefs
docker plugin enable juicedata/juicefs

# Remove the plugin
docker plugin rm juicedata/juicefs

Create a storage volume {#create-volume} Copy Section

Replace <VOLUME_NAME>, <META_URL>, <STORAGE_TYPE>, ``, <ACCESS_KEY>, and <SECRET_KEY> in the following command with your own file system configuration:

docker volume create -d juicedata/juicefs \
  -o name=<VOLUME_NAME> \
  -o metaurl=<META_URL> \
  -o storage=<STORAGE_TYPE> \
  -o bucket= \
  -o access-key=<ACCESS_KEY> \
  -o secret-key=<SECRET_KEY> \
  jfsvolume

For pre-created file systems, you only need to specify the file system name and database address when creating the volume plugin, for example:

docker volume create -d juicedata/juicefs \
  -o name=<VOLUME_NAME> \
  -o metaurl=<META_URL> \
  jfsvolume

If you need to pass additional environment variables when mounting the file system, such as in Google Cloud, you can append parameters similar to -o env=FOO=bar,SPAM=egg to the above command.

Usage and management {#usage-and-management} Copy Section

# Mount the volume when creating a container
docker run -it -v jfsvolume:/opt busybox ls /opt

# After unmounting, you can delete the storage volume. Note that this only deletes the corresponding resources in Docker and does not affect the data stored in JuiceFS.
docker volume rm jfsvolume

Using the plugin in Docker Compose {#using-plugin-in-docker-compose} Copy Section

Here is an example of using the JuiceFS volume plugin in docker-compose:

version: '3'
services:
  busybox:
    image: busybox
    command: "ls /jfs"
    volumes:
      - jfsvolume:/jfs
      
volumes:
  jfsvolume:
    driver: juicedata/juicefs
    driver_opts:
      name: ${VOL_NAME}
      # SQLite creates the database file in the plugin container's local path,
      # and sqlite:// will fail when the service is restarted.
      # (See https://github.com/juicedata/docker-volume-juicefs/issues/37 for details)
      metaurl: ${META_URL}
      storage: ${STORAGE_TYPE}
      bucket: ${BUCKET}
      access-key: ${ACCESS_KEY}
      secret-key: ${SECRET_KEY}
      # If necessary, you can pass additional environment variables using env
      # env: FOO=bar,SPAM=egg

Usage and management:

# Start the service
docker-compose up

# Stop the service and unmount the JuiceFS file system from Docker
docker-compose down --volumes

Troubleshooting the volume plugin {#troubleshooting} Copy Section

If it is not working properly, it is recommended to first upgrade the volume plugin, and then check the logs based on the problem.

• Collect JuiceFS client logs. The logs are located inside the Docker volume plugin container and need to be accessed by entering the container:

  shell Copy

  # Confirm the docker plugins runtime directory, which may be different from the example below depending on the actual situation
  # The directory printed by ls is the container directory, and the name is the container ID
  ls /run/docker/plugins/runtime-root/plugins.moby
  
  # Print plugin container information
  # If the printed container list is empty, it means that the plugin container failed to be created
  # Read the plugin startup log below to continue troubleshooting
  runc --root /run/docker/plugins/runtime-root/plugins.moby list
  
  # Enter the container and print the log
  runc --root /run/docker/plugins/runtime-root/plugins.moby exec 452d2c0cf3fd45e73a93a2f2b00d03ed28dd2bc0c58669cca9d4039e8866f99f cat /var/log/juicefs.log

  If the container does not exist (ls finds an empty directory) or the juicefs.log does not exist in the final log printing stage, it is likely that the mount itself failed. Continue to check the plugin's own logs to find the cause.

• Collect plugin logs, using systemd as an example:

  shell Copy

  journalctl -f -u docker | grep "plugin="

  If there is an error when the plugin calls juicefs or if the plugin itself reports an error, it will be reflected in the logs.

Using the JuiceFS client in containers {#mount-juicefs-in-docker} Copy Chapter

Compared to the volume plugin, using the JuiceFS client directly in the container is more flexible. You can directly mount the JuiceFS file system in the container or access it through S3 Gateway or WebDAV.

Method 1: Build your own image Copy Section

The JuiceFS client is a standalone binary program that provides versions for both AMD64 and ARM64 architectures. You can define the command to download and install the JuiceFS client in the Dockerfile, for example:

FROM ubuntu:22.04
...
# Use the official one-click installation script
RUN curl -sSL https://d.juicefs.com/install | sh -

For more information, see Customizing Container Images.

Method 2: Use the officially maintained image Copy Section

The JuiceFS officially maintained image juicedata/mount is tagged to specify the desired version. The community edition tags include latest and ce, such as ce-v1.1.2 and ce-nightly. The latest tag represents the latest community edition, and the nightly tag points to the latest development version. For details, see the tags page on Docker Hub.

Before you start, you need to prepare object storage and metadata engine.

Create a file system

Create a file system through a temporary container, for example:

docker run --rm \
    juicedata/mount:ce-v1.1.2 juicefs format \
    --storage s3 \
    --bucket https://xxx.your-s3-endpoint.com \
    --access-key=ACCESSKEY \
    --secret-key=SECRETKEY \
    rediss://user:[email protected]:6379/1 myjfs

Replace --storage, --bucket, --access-key, --secret-key, and the metadata engine URL with your own configuration.

Mount the file system directly in the container

Create a container and mount the JuiceFS file system in the container, for example:

docker run --privileged --name myjfs \
    juicedata/mount:ce-v1.1.2 juicefs mount \
    rediss://user:[email protected]:6379/1 /mnt

Replace the metadata engine URL with your own configuration. /mnt is the mount point and can be modified as needed. Since FUSE is used, --privileged permission is also required.

Mount the file system through Docker Compose

Here is an example using Docker Compose. Replace the metadata engine URL and mount point with your own configuration.

version: "3"
services:
    busybox:
      image: busybox
      command: "ls /jfs"
      volumes:
        - ./mnt:/jfs
      depends_on:
        juicefs:
          condition: service_healthy

    juicefs:
      image: juicedata/mount:ce-v1.1.2
      container_name: myjfs
      volumes:
        - ./mnt:/mnt:rw,rshared
      cap_add:
        - SYS_ADMIN
      devices:
        - /dev/fuse
      security_opt: 
        - apparmor:unconfined
      command: ["juicefs", "mount", "rediss://user:[email protected]:6379/1", "/mnt"]
      restart: unless-stopped
      healthcheck:
        test: ["CMD-SHELL", "cat /mnt/.control"]
        interval: 60s
        retries: 5
        start_period: 30s
        timeout: 10s

In the container, the JuiceFS file system is mounted to the /mnt directory, and the volumes section in the configuration file maps the /mnt in the container to the ./mnt directory on the host, allowing direct access to the JuiceFS file system mounted in the container from the host. At the same time, by combining depends_on and volumes, the directory mapped to the host machine can be remounted into the container for use.

Access the file system through S3 Gateway

Here is an example of exposing JuiceFS for access through S3 Gateway. Replace MINIO_ROOT_USER, MINIO_ROOT_PASSWORD, the metadata engine URL, and the address and port number to listen on with your own configuration.

version: "3"
services:
    s3-gateway:
      image: juicedata/mount:ce-v1.1.2
      container_name: juicefs-s3-gateway
      environment:
        - MINIO_ROOT_USER=your-username
        - MINIO_ROOT_PASSWORD=your-password
      ports:
        - "9090:9090"
      command: ["juicefs", "gateway", "rediss://user:[email protected]:6379/1", "0.0.0.0:9090"]
      restart: unless-stopped

Use port 9090 on the host to access the S3 Gateway console, and use the same address to read and write the JuiceFS file system through the S3 client or SDK.

File: docs/en/deployment/nfs.md Copy Chapter

title: Create NFS Shares sidebar_position: 9 Copy Chapter

NFS (Network File System) is a network file-sharing protocol that allows different computers to share files and directories over a network. It was originally developed by Sun Microsystems and is a standard way of file sharing between Unix and Unix-like systems. The NFS protocol enables clients to access remote file systems as if they were local, achieving transparent remote file access.

When you need to share directories from the JuiceFS file system through NFS, you can simply use the juicefs mount command to mount the file system. Then, you can create NFS shares with the JuiceFS mount point or subdirectories.

:::note
juicefs mount mounts the file system as a local user-space file system through the FUSE interface, making it identical to the local file system in terms of appearance and usage. Hence, it can be directly used to create NFS shares.
:::

Step 1. Install NFS Copy Chapter

To configure NFS shares, you need to install the relevant software packages on both the server and client sides. Let's take Ubuntu/Debian systems as an example:

1. Server-side installation Copy Section

Create a host for NFS sharing (with the JuiceFS file system also mounted on this server).

sudo apt install nfs-kernel-server

2. Client-side installation Copy Section

All Linux hosts that need to access NFS shares should install the client software.

sudo apt install nfs-common

Step 2. Create shares Copy Chapter

Assuming the JuiceFS is mounted on the server system at the path /mnt/myjfs, if you want to set the media subdirectory as an NFS share, you can add the following configuration to the /etc/exports file on the server system:

"/mnt/myjfs/media" *(rw,sync,no_subtree_check,fsid=1)

The syntax for NFS share configuration is as follows:

<Share Path> <Allowed IPs>(options)

For example, if you want to restrict the mounting of this share to hosts in the 192.168.1.0/24 IP range and avoid squashing root privileges, you can modify it as follows:

"/mnt/myjfs/media" 192.168.1.0/24(rw,async,no_subtree_check,no_root_squash,fsid=1)

Share option description Copy Section

Explanation of the share options:

• rw: Represents read and write permissions. If read-only access is desired, use ro.
• sync and async: sync enables synchronous writes, meaning that when writing to the NFS share, the client waits for the server's confirmation of successful data write before proceeding with subsequent operations. async, on the other hand, allows asynchronous writes. In this mode, the client does not wait for the server's confirmation of successful write before proceeding with subsequent operations.
• no_subtree_check: Disables subtree checking, allowing clients to mount both the parent and child directories of the NFS share. This can reduce some security but improve NFS compatibility. Setting it to subtree_check enables subtree checking, allowing clients to only mount the NFS share and its subdirectories.
• no_root_squash: Controls the mapping behavior of the client's root user when accessing the NFS share. By default, when the client mounts the NFS share as root, the server maps it to a non-privileged user (usually nobody or nfsnobody), which is known as root squashing. Enabling this option cancels the root squashing, giving the client the same root user privileges as the server. This option comes with certain security risks and should be used with caution.
• fsid: A file system identifier used to identify different file systems on NFS. In NFSv4, the root directory of NFS is defined as fsid=0, and other file systems need to be numbered uniquely under it. Here, JuiceFS is an externally mounted FUSE file system, so it needs to be assigned a unique identifier.

Choosing between async and sync modes Copy Section

For NFS shares, the sync (synchronous writes) mode can improve data reliability but always requires waiting for the server's confirmation before proceeding with the next operation. This may result in lower write performance. For JuiceFS, which is a cloud-based distributed file system, network latency also needs to be considered. Using the sync mode can often lead to lower write performance due to network latency.

In most cases, when creating NFS shares with JuiceFS, it is recommended to set the write mode to async (asynchronous writes) to avoid sacrificing write performance. If data reliability must be prioritized and sync mode is necessary, it is recommended to configure JuiceFS with a high-performance SSD as a local cache with sufficient capacity and enable the writeback cache mode.