foundationdb Documentation

File: README.md Copy Chapter

FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value store and employs ACID transactions for all operations. It is especially well-suited for read/write workloads, but also has excellent performance for write-intensive workloads. Users interact with the database using API language binding.

To learn more about FoundationDB, visit foundationdb.org

Documentation Copy Chapter

Documentation can be found online at https://apple.github.io/foundationdb/. The documentation covers details of API usage, background information on design philosophy, and extensive usage examples. Docs are built from the source in this repo.

Forums Copy Chapter

The FoundationDB Forums are the home for most of the discussion and communication about the FoundationDB project. We welcome your participation! We want FoundationDB to be a great project to be a part of, and as part of that, we have established a Code of Conduct to define what constitutes permissible modes of interaction.

Contributing Copy Chapter

Contributing to FoundationDB can be in contributions to the codebase, sharing your experience and insights in the community on the Forums, or contributing to projects that make use of FoundationDB. Please see the contributing guide for more specifics.

Getting Started Copy Chapter

Latest Stable Releases Copy Section

The latest stable releases are (were) versions that are recommended for production use, which have been extensively validated via simulation and real cluster tests and used in our production environment.

• Supported branches are those we actively maintain and will publish new patch releases.
• Bug fixes are branches where we still accept bug fixes, but may not publish newer patch releases. The community can build the latest release binaries if needed and is encouraged to upgrade to the Supported branches.
• Experimental branches are those used for internal feature testing. They are not recommended for production use.
• Unsupported branches are those that will no longer receive any updates.

If you are running on old production releases, we recommend always upgrading to the next major release's latest version, and then continuing to the next major version, e.g., 6.2.X -> 6.3.25 -> 7.1.57 -> 7.3.69. These upgrade paths have been well tested in production (skipping a major release, not marked as Experimental, for an upgrade is only tested in simulation).

Binary Downloads Copy Section

Developers interested in using FoundationDB can get started by downloading and installing a binary package. Please see the downloads page for a list of available packages.

Compiling from source Copy Section

Developers on an OS for which there is no binary package, or who would like to start hacking on the code, can get started by compiling from source.

NOTE: FoundationDB has a lot of dependencies. The Docker container
listed below tracks them and is what we use internally and is the
recommended method of building FDB.

Build Using the Official Docker Image

The official Docker image for building is foundationdb/build, which includes all necessary dependencies. The Docker image definitions used by FoundationDB team members can be found in the dedicated repository.

To build FoundationDB with the clang toolchain,

mkdir /some/build_output_dir
cd /some/build_output_dir
CC=clang CXX=clang++ LD=lld cmake -D USE_LD=LLD -D USE_LIBCXX=1 -G Ninja /some/fdb/source_dir
ninja

To use GCC, a non-default version is necessary. The following modifies environment
variables ($PATH, $LD_LIBRARY_PATH, etc) to pick up the right GCC version:

source /opt/rh/gcc-toolset-13/enable
gcc --version  # should say 13
mkdir /some/build_output_dir
cd /some/build_output_dir
cmake -G Ninja /some/fdb/source_dir
ninja

Slightly more elaborate compile commands can be found in the shell aliases
defined in /root/.bashrc in the container image.

Build Locally

To build outside of the official Docker image, you'll need at least these dependencies:

1. CMake version 3.24.2 or higher
2. Mono
3. ninja

This list is likely to be incomplete. Refer to the rockylinux9
Dockerfile in the fdb-build-support repo linked above for reference
material on specific packages and versions that are likely to be
required.

If compiling for local development, please set -DUSE_WERROR=ON in CMake. Our CI compiles with -Werror on, so this way you'll find out about compiler warnings that break the build earlier.

Once you have your dependencies, you can run cmake and then build:

1. Check out this repository.
2. Create a build directory (you can place it anywhere you like).
3. cd <FDB_BUILD_DIR>
4. cmake -G Ninja <FDB_SOURCE_DIR>
5. ninja

Building FoundationDB requires at least 8GB of memory. More memory is needed when building in parallel. If the computer freezes or crashes, consider disabling parallelized build using ninja -j1.

FreeBSD

1. Check out this repo on your server.

2. Install compile-time dependencies from ports.

3. (Optional) Use tmpfs & ccache for significantly faster repeat builds

4. (Optional) Install a JDK for Java Bindings. FoundationDB currently builds with Java 8.

5. Navigate to the directory where you checked out the FoundationDB repository.

6. Build from source.

   shell Copy

   sudo pkg install -r FreeBSD \
       shells/bash devel/cmake devel/ninja devel/ccache  \
       lang/mono lang/python3 \
       devel/boost-libs devel/libeio \
       security/openssl
   mkdir .build && cd .build
   cmake -G Ninja \
       -DUSE_CCACHE=on \
       -DUSE_DTRACE=off \
       ..
   ninja -j 10
   # run fast tests
   ctest -L fast
   # run all tests
   ctest --output-on-failure -v

macOS Copy Section

The build under macOS will work the same way as on Linux. Homebrew can be used to install the boost library and the ninja build tool. Be careful, current main branch uses boost 1.86, do install this version or just let cmake download one. Also, if the Swift binding is not of interest, use -DBUILD_SWIFT_BINDING=OFF.

cmake -G Ninja <FDB_SOURCE_DIR> -B 
cd 
ninja

To generate an installable package,

<FDB_SOURCE_DIR>/packaging/osx/buildpkg.sh  <FDB_SOURCE_DIR>

Windows Copy Section

Under Windows, only Visual Studio with ClangCl is supported

1. Install Visual Studio 2019 (IDE or Build Tools), and enable LLVM support
2. Install CMake 3.24.2 or higher
3. Download Boost 1.86.0
4. Unpack boost to C:\boost, or use -DBOOST_ROOT= with cmake if unpacked elsewhere
5. Install Python if it is not already installed by Visual Studio
6. (Optional) Install OpenJDK 11 to build Java bindings
7. (Optional) Install OpenSSL 3.x to build with TLS support
8. (Optional) Install WIX Toolset to build the Windows installer
9. mkdir build && cd build
10. cmake -G "Visual Studio 16 2019" -A x64 -T ClangCl <FDB_SOURCE_DIR>
11. msbuild /p:Configuration=Release foundationdb.sln
12. To increase build performance, use /p:UseMultiToolTask=true and /p:CL_MPCount=<NUMBER_OF_PARALLEL_JOBS>

Language Bindings Copy Section

The language bindings that CMake supports will have a corresponding README.md file in the bindings/lang directory corresponding to each language.

Generally, CMake will build all language bindings for which it can find all necessary dependencies. After each successful CMake run, CMake will tell you which language bindings it is going to build.

Generating `compile_commands.json` Copy Section

CMake can build a compilation database for you. However, the default generated one is not too useful as it operates on the generated files. When running ninja, the build system creates another compile_commands.json file in the source directory. This can then be used for tools such as CCLS and CQuery, among others. This way, you can get code completion and code navigation in flow. It is not yet perfect (it will show a few errors), but we are continually working to improve the development experience.

CMake will not produce a compile_commands.json by default; you must pass -DCMAKE_EXPORT_COMPILE_COMMANDS=ON. This also enables the target processed_compile_commands, which rewrites compile_commands.json to describe the actor compiler source file, not the post-processed output files, and places the output file in the source directory. This file should then be picked up automatically by any tooling.

Note that if the building is done inside the foundationdb/build Docker image, the resulting paths will still be incorrect and require manual fixing. One will wish to re-run cmake with -DCMAKE_EXPORT_COMPILE_COMMANDS=OFF to prevent it from reverting the manual changes.

Code Formatting and Static Analysis Copy Section

clang-format and clang-tidy run as part of CI on every pull request. See the clang-format and clang-tidy guides for how to run them locally before pushing.

Using IDEs Copy Section

CMake provides built-in support for several popular IDEs. However, most FoundationDB files are written in the flow language, which is an extension of the C++ programming language, for coroutine support (Note that when FoundationDB was being developed, C++20 was not available). The flow language will be transpiled into C++ code using actorcompiler, while preventing most IDEs from recognizing flow-specific syntax.

It is possible to generate project files for editing flow with a supported IDE. There is a CMake option called OPEN_FOR_IDE, which creates a project that can be opened in an IDE for editing. This project cannot be built, but you will be able to edit the files and utilize most of the editing and navigation features that your IDE supports.

For example, if you want to use Xcode to make changes to FoundationDB, you can create an Xcode project with the following command:

cmake -G Xcode -DOPEN_FOR_IDE=ON <FDB_SOURCE_DIRECTORY>

A second build directory with the OPEN_FOR_IDE flag off can be created for building and debugging purposes.

File: bindings/go/README.md Copy Chapter

fdb-go

Go language bindings for FoundationDB, a distributed key-value store with ACID transactions.

This package requires:

• Go 1.22+ with CGO enabled
• FoundationDB client package (can be installed from the release). It's recommended to install the matching client package for the FDB version you want to use.

Use of this package requires the selection of a FoundationDB API version at runtime.
This package currently supports FoundationDB API versions 200-800.

Documentation Copy Chapter

• API documentation
• Tutorial

Modules Copy Chapter

In your go project with modules run the following command to get/update the required FDB version:

# Example for FDB 7.3.63 bindings
go get github.com/apple/foundationdb/bindings/[email protected]

This will add or update the bindings entry in your go.mod file like this:

github.com/apple/foundationdb/bindings/go v0.0.0-20250221231555-5140696da2df

Note: @version should match the major and minor of the FDB version you want to use, e.g. all 7.3 bindings will work for an FDB cluster in the 7.3 version.

Developing Copy Chapter

You can build this package, in the top level of this repository run:

make fdb_go

This will create binary packages for the appropriate platform within the "build" subdirectory of this folder.

Generating options file Copy Chapter

The generated.go is generated based on the fdb.options file.
If you change the fdb.options file you can update the generated.go by running the following command from the root of this reporsitory:

go run bindings/go/src/_util/translate_fdb_options.go < fdbclient/vexillographer/fdb.options > bindings/go/src/fdb/generated.go

File: bindings/java/src/README.md Copy Chapter

Adding JUnit tests

For java development, it's often useful to use JUnit for testing due to the excellent tooling support.

To add a new unit test, do the following:

1. Write your test
2. Add the test path to tests.cmake, using the relative path starting at src(for example, src/junit/com/apple/foundationdb/tuple/ArrayUtilTests.java will add the ArrayUtilTests test file).
3. re-run the build (both cmake and make/xcode/ninja)

To add a new integration test:

1. Write the test, using JUnit.
2. Add the test path to tests.cmake, using the relative path starting at src (i.e. src/integration/com/apple/foundationdb/DirectoryTest.java).
3. re-run the build (both cmake and make/xcode/ninja)

To run all unit and integration tests, execute ctest . from ${BUILD_DIR}/bindings/java.

To skip integration tests, execute ctest -E integration from ${BUILD_DIR}/bindings/java.

To run only integration tests, run ctest -R integration from ${BUILD_DIR}/bindings/java.

There are lots of other useful ctest commands, which we don't need to get into here. For more information,
see the [https://cmake.org/cmake/help/v3.19/manual/ctest.1.html](ctest documentation).

Multi-Client tests Copy Section

Multi-Client tests are integration tests that can only be executed when multiple clusters are running. To write a multi-client
test, do the following:

1. Tag all tests that require multiple clients with @Tag("MultiClient")
2. Ensure that your tests have the MultiClientHelper extension present, and Registered as an extension
3. Ensure that your test class is in the the JAVA_INTEGRATION_TESTS list in test.cmake

( see BasicMultiClientIntegrationTest for a good reference example)

It is important to note that it requires significant time to start and stop 3 separate clusters; if the underlying test takes a long time to run,
ctest will time out and kill the test. When that happens, there is no guarantee that the FDB clusters will be properly stopped! It is thus
in your best interest to ensure that all tests run in a relatively small amount of time, or have a longer timeout attached.

File: bindings/java/README.md Copy Chapter

FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value store and employs ACID transactions for all operations. It is especially well-suited for read/write workloads but also has excellent performance for write-intensive workloads. Users interact with the database using API language binding.

To learn more about FoundationDB, visit foundationdb.org

FoundationDB Java Bindings Copy Chapter

In order to build the java bindings,
JDK >= 8
has to be installed. CMake will try to find a JDK installation, if it can find
one it will automatically build the java bindings.

If you have Java installed but cmake fails to find them, set the
JAVA_HOMEenvironment variable.

Fat Jar Copy Section

By default, the generated jar file will depend on an installed libfdb_java
(provided with the generated RPM/DEB file on Linux). However, users usually find
a Jar-file that contains this library more convenient. This is also what you
will get if you download the jar file from Maven.

This file can be generated by compiling the packages target. For example with
make, you can run:

make packages

Multi-Platform Jar-File

If you want to create a jar file that can run on more than one supported
architecture (the official one supports MacOS, Linux, and Windows), you can do
that by executing the following steps:

1. Create a directory called lib somewhere on your file system.
2. Create a subdirectory for each additional platform you want to support
   (windows for windows, osx for MacOS, and linux for Linux).
3. Under each of those create a subdirectory with the name of the architecture
   (currently only amd64 is supported - on MacOS this has to be called
   x86_64 - amd64 on all others).
4. Set the cmake variable FAT_JAR_BINARIES to this lib directory. For
   example, if you created this directory structure under /foo/bar, the
   corresponding cmake command would be:

cmake -DFAT_JAR_BINARIES=/foo/bar/lib

After executing building the packages (with make packages or the packages
target in Visual Studio) you will find a jar-file in the packages
directory in your build directory.

File: contrib/debug_determinism/README.md Copy Chapter

Utilities for debugging unseed mismatches for foundationdb simulation tests.

99/100 times the source of the nondeterminism is use of uninitialized memory and
what you want to do is build with -DUSE_VALGRIND=ON and run simulations under
valgrind.

Common sources of nondeterminism and specialized tools to find them.

1. Use of uninitialized memory (use valgrind!)
2. Memory errors (use valgrind and/or asan)
3. Undefined behavior (use ubsan. You can also try _GLIBCXX_DEBUG)

If it's not any of these then now it's time to try this technique. Look for

1. Call to some kind of "get current time" function that's not in INetwork
2. Depending on the relative ordering of allocated memory. E.g. Using heap-allocated pointers as keys in a std::map.
3. Inspecting something about the current state of the system (e.g. free disk space)
4. Depending on iteration order of an unordered map

Quickstart

Set these cmake flags

-DTRACE_PC_GUARD_INSTRUMENTATION_LIB=$BUILDDIR/lib/libdebug_determinism.a

and change #define DEBUG_DETERMINISM 0 to #define DEBUG_DETERMINISM 1 in
flow/Platform.h. This disables several known sources of nondeterminism that
don't affect unseeds.

For reasons I don't fully understand, it appears that sqlite exhibits some
nondeterminism if you don't add #define SQLITE_OMIT_LOOKASIDE to the top of
fdbserver/sqlite/sqlite3.amalgamation.c, so you probably want to do that too.

Now when you run an fdbserver simulation, it will write a file out.bin in the
current directory which contains the sequence of edges in the control flow graph
that were encountered during the simulation. If you rename out.bin to in.bin
and then re-run, the simulation will validate that the sequence of edges is the
same as the last run. If it's not, then the simulation will enter an infinite
loop at the first difference and print a message. Then you probably want to
attach gdb to the process and investigate from there.

You'll need to make sure you delete the simfdb folder before each run, because
otherwise you'll take a different codepath for deleting the simfdb folder at
the beginning of simulation.

File: contrib/fdbcstat/README.md Copy Chapter

fdbcstat

fdbcstat is a FoundationDB client monitoring tool which collects and displays transaction operation statistics inside the C API library (libfdb_c.so).

How it works Copy Chapter

fdbcstat utilizes eBPF/bcc to attach to libfdb_c.so shared library and insert special instructions to collect statistics in several common fdb_transaction_* calls, then it periodically displays the aggregated statistics.

How to use Copy Chapter

Syntax Copy Section

fdbcstat <full path to libfdb_c.so> <options...>

Options Copy Section

• -p or --pid : Only capture statistics for the functions called by the specified process
• -i or --interval : Specify the time interval in seconds between 2 outputs (Default: 1)
• -d or --duration : Specify the total duration in seconds fdbcstats will run (Default: Unset / Forever)
• -f or --functions : Specify the comma-separated list of functions to monitor (Default: Unset / All supported functions)

Supported Functions Copy Section

• get
• get_range
• get_read_version
• set
• clear
• clear_range
• commit

Examples Copy Section

Collect all statistics and display every second

fdbcstat /usr/lib64/libfdb_c.so

Collect all statistics for PID 12345 for 60 seconds with 10 second interval

fdbcstat /usr/lib64/libfdb_c.so -p 12345 -d 60 -i 10

Collect statistics only for get and commit

fdbcstat /usr/lib64/libfdb_c.so -f get,commit

Output Format Copy Chapter

Each line contains multiple fields. The first field is the timestamp. Other fields are the statistics for each operation. Each operation field contains the following statistics in a slash (/) separated format.

• Function
• Number of calls per second
• Average latency in microseconds (us)
• Maximum latency in microseconds (us)

Note: The latency is computed as the time difference between the start time and the end time of the fdb_transaction_* function call except for get, get_range, get_read_version and commit. For those 4 functions, the latency is the time difference between the start time of the function and the end time of the following fdb_future_block_until_ready call.

Sample Output Copy Chapter

...
15:05:31 clear/22426/2/34 commit/18290/859/15977 get/56230/1110/12748 get_range/14141/23/75 set/6276/3/19 
15:05:41 clear/24147/2/38 commit/18259/894/44259 get/57978/1098/15636 get_range/13171/23/90 set/6564/3/15 
15:05:51 clear/21287/2/34 commit/18386/876/17824 get/58318/1106/30539 get_range/13018/23/68 set/6559/3/13 
...

File: contrib/Implib.so/arch/README.md Copy Chapter

This folder contains target-specific config files and code snippets.

Basically to add a new target one needs to provide an .ini file with basic platform info
(like pointer sizes) and code templates for

• shim code which checks that real function address is available (and either jumps there or calls the slow path)
• the "slow path" code which
  • saves function arguments (to avoid trashing them in next steps)
  • calls code which loads the target library and locates function addresses
  • restores saved arguments and returns

File: contrib/Implib.so/README.md Copy Chapter

Motivation

In a nutshell, Implib.so is a simple equivalent of Windows DLL import libraries for POSIX shared libraries.

On Linux/Android, if you link against shared library you normally use -lxyz compiler option which makes your application depend on libxyz.so. This would cause libxyz.so to be forcedly loaded at program startup (and its constructors to be executed) even if you never call any of its functions.

If you instead want to delay loading of libxyz.so (e.g. its unlikely to be used and you don't want to waste resources on it or slow down startup time or you want to select best platform-specific implementation at runtime), you can remove dependency from LDFLAGS and issue dlopen call manually. But this would cause ld to err because it won't be able to statically resolve symbols which are supposed to come from this shared library. At this point you have only two choices:

• emit normal calls to library functions and suppress link errors from ld via -Wl,-z,nodefs; this is undesired because you loose ability to detect link errors for other libraries statically
• load necessary function addresses at runtime via dlsym and call them via function pointers; this isn't very convenient because you have to keep track which symbols your program uses, properly cast function types and also somehow manage global function pointers

Implib.so provides an easy solution - link your program with a wrapper which

• provides all necessary symbols to make linker happy
• loads wrapped library on first call to any of its functions
• redirects calls to library symbols

Generated wrapper code (often also called "shim" code or "shim" library) is analogous to Windows import libraries which achieve the same functionality for DLLs.

Implib.so can also be used to reduce API provided by existing shared library or rename it's exported symbols.

Implib.so was originally inspired by Stackoverflow question Is there an elegant way to avoid dlsym when using dlopen in C?.

Usage

A typical use-case would look like this:

$ implib-gen.py libxyz.so

This will generate code for host platform (presumably x86_64). For other targets do

$ implib-gen.py --target $TARGET libxyz.so

where TARGET can be any of

• x86_64-linux-gnu, x86_64-none-linux-android
• i686-linux-gnu, i686-none-linux-android
• arm-linux-gnueabi, armel-linux-gnueabi, armv7-none-linux-androideabi
• arm-linux-gnueabihf (ARM hardfp ABI)
• aarch64-linux-gnu, aarch64-none-linux-android
• e2k-linux-gnu

Script generates two files: libxyz.so.tramp.S and libxyz.so.init.cpp which need to be linked to your application (instead of -lxyz):

$ gcc myfile1.c myfile2.c ... libxyz.so.tramp.S libxyz.so.init.cpp ... -ldl

Note that you need to link against libdl.so. On ARM in case your app is compiled to Thumb code (which e.g. Ubuntu's arm-linux-gnueabihf-gcc does by default) you'll also need to add -mthumb-interwork.

Application can then freely call functions from libxyz.so without linking to it. Library will be loaded (via dlopen) on first call to any of its functions. If you want to forcedly resolve all symbols (e.g. if you want to avoid delays further on) you can call void libxyz_init_all().

Above command would perform a lazy load i.e. load library on first call to one of it's symbols.

If you do want to load library via dlopen but would prefer to call it yourself (e.g. with custom parameters or with modified library name), run script as

$ cat mycallback.c
#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

#ifdef __cplusplus
extern "C"
#endif

// Callback that tries different library names
void *mycallback(const char *lib_name) {
  lib_name = lib_name;  // Please the compiler
  void *h;
  h = dlopen("libxyz.so", RTLD_LAZY);
  if (h)
    return h;
  h = dlopen("libxyz-stub.so", RTLD_LAZY);
  if (h)
    return h;
  fprintf(stderr, "dlopen failed: %s\n", dlerror());
  exit(1);
}

$ implib-gen.py --dlopen-callback=mycallback libxyz.so

(callback must have signature void *(*)(const char *lib_name) and return handle of loaded library).

Wrapping vtables

By default the tool does not try to wrap vtables exported from the library. This can be enabled via --vtables flag:

$ implib-gen.py --vtables ...

Reducing external interface of closed-source library

Sometimes you may want to reduce public interface of existing shared library (e.g. if it's a third-party lib which erroneously exports too many unrelated symbols).

To achieve this you can generate a wrapper with limited number of symbols and override the callback which loads the library to use dlmopen instead of dlopen (and thus does not pollute the global namespace):

$ cat mysymbols.txt
foo
bar
$ cat mycallback.c
#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

#ifdef __cplusplus
extern "C"
#endif

// Dlopen callback that loads library to dedicated namespace
void *mycallback(const char *lib_name) {
  void *h = dlmopen(LM_ID_NEWLM, lib_name, RTLD_LAZY | RTLD_DEEPBIND);
  if (h)
    return h;
  fprintf(stderr, "dlmopen failed: %s\n", dlerror());
  exit(1);
}

$ implib-gen.py --dlopen-callback=mycallback --symbol-list=mysymbols.txt libxyz.so
$ ... # Link your app with libxyz.tramp.S, libxyz.init.cpp and mycallback.c

Similar approach can be used if you want to provide a common interface for several libraries with partially intersecting interfaces (see this example for more details).

Renaming exported interface of closed-source library

Sometimes you may need to rename API of existing shared library to avoid name clashes.

To achieve this you can generate a wrapper with renamed symbols which call to old, non-renamed symbols in original library loaded via dlmopen instead of dlopen (to avoid polluting global namespace):

$ cat mycallback.c
... Same as before ...
$ implib-gen.py --dlopen-callback=mycallback --symbol_prefix=MYPREFIX_ libxyz.so
$ ... # Link your app with libxyz.tramp.S, libxyz.init.cpp and mycallback.c

Linker wrapper

Generation of wrappers may be automated via linker wrapper scripts/ld.
Adding it to PATH (in front of normal ld) would by default result
in all dynamic libs (besides system ones) to be replaced with wrappers.
Explicit list of libraries can be specified by exporting
IMPLIBSO_LD_OPTIONS environment variable:

export IMPLIBSO_LD_OPTIONS='--wrap-libs attr,acl'

For more details run with

export IMPLIBSO_LD_OPTIONS=--help

Atm linker wrapper is only meant for testing.

Overhead

Implib.so overhead on a fast path boils down to

• predictable direct jump to wrapper
• predictable untaken direct branch to initialization code
• load from trampoline table
• predictable indirect jump to real function

This is very similar to normal shlib call:

• predictable direct jump to PLT stub
• load from GOT
• predictable indirect jump to real function

so it should have equivalent performance.

Limitations

The tool does not transparently support all features of POSIX shared libraries. In particular

• it can not provide wrappers for data symbols (except C++ virtual/RTTI tables)
• it makes first call to wrapped functions asynch signal unsafe (as it will call dlopen and library constructors)
• it may change semantics if there are multiple definitions of same symbol in different loaded shared objects (runtime symbol interposition is considered a bad practice though)
• it may change semantics because shared library constructors are delayed until when library is loaded

The tool also lacks the following very important features:

• proper support for multi-threading
• symbol versions are not handled at all
• support OSX
  (none should be hard to add so let me know if you need it).

Finally, Implib.so is only lightly tested and there are some minor TODOs in code.

Related work

As mentioned in introduction import libraries are first class citizens on Windows platform:

• Wikipedia on Windows Import Libraries
• MSDN on Linker Support for Delay-Loaded DLLs

Delay-loaded libraries were once present on OSX (via -lazy_lXXX and -lazy_library options).

Lazy loading is supported by Solaris shared libraries but was never implemented in Linux. There have been some discussions in libc-alpha but no patches were posted.

Implib.so-like functionality is used in OpenGL loading libraries e.g. GLEW via custom project-specific scripts.

File: contrib/Joshua/README.md Copy Chapter

Overview

This directory provides the files needed to create a Joshua correctness bundle for testing FoundationDB.

Rigorous testing is central to our engineering process. The features of our core are challenging, requiring us to meet exacting standards of correctness and performance. Data guarantees and transactional integrity must be maintained not only during normal operations but over a broad range of failure scenarios. At the same time, we aim to achieve performance goals such as low latencies and near-linear scalability. To meet these challenges, we use a combined regime of robust simulation, live performance testing, and hardware-based failure testing.

Joshua

Joshua is a powerful tool for testing system correctness. Our simulation technology, called Joshua, is enabled by and tightly integrated with flow, our programming language for actor-based concurrency. In addition to generating efficient production code, Flow works with Joshua for simulated execution.

The major goal of Joshua is to make sure that we find and diagnose issues in simulation rather than the real world. Joshua runs tens of thousands of simulations every night, each one simulating large numbers of component failures. Based on the volume of tests that we run and the increased intensity of the failures in our scenarios, we estimate that we have run the equivalent of roughly one trillion CPU-hours of simulation on FoundationDB.

Joshua is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-threaded process. Determinism is crucial in that it allows perfect repeatability of a simulated run, facilitating controlled experiments to home in on issues. The simulation steps through time, synchronized across the system, representing a larger amount of real time in a smaller amount of simulated time. In practice, our simulations usually have about a 10-1 factor of real-to-simulated time, which is advantageous for the efficiency of testing.

We run a broad range of simulations testing various aspects of the system. For example, we run a cycle test that uses key-values pairs arranged in a ring that executes transactions to change the values in a manner designed to maintain the ring's integrity, allowing a clear test of transactional isolation.

Joshua simulates all physical components of a FoundationDB system, beginning with the number and type of machines in the cluster. For example, Joshua models drive performance on each machine, including drive space and the possibility of the drive filling up. Joshua also models the network, allowing a small amount of code to specify delivery of packets.

We use Joshua to simulate failures modes at the network, machine, and datacenter levels, including connection failures, degradation of machine performance, machine shutdowns or reboots, machines coming back from the dead, etc. We stress-test all of these failure modes, failing machines at very short intervals, inducing unusually severe loads, and delaying communications channels.

For a while, there was an informal competition within the engineering team to design failures that found the toughest bugs and issues the most easily. After a period of one-upsmanship, the reigning champion is called "swizzle-clogging". To swizzle-clog, you first pick a random subset of nodes in the cluster. Then, you "clog" (stop) each of their network connections one by one over a few seconds. Finally, you unclog them in a random order, again one by one, until they are all up. This pattern seems to be particularly good at finding deep issues that only happen in the rarest real-world cases.

Joshua's success has surpassed our expectation and has been vital to our engineering team. It seems unlikely that we would have been able to build FoundationDB without this technology.

• scripts/: This directory contains shell scripts that serve as entry points for running tests. Joshua invokes these scripts, which then set up the environment and execute the test runner.
  • correctnessTest.sh: This is the primary script for running correctness tests (In the ensemble tarball, it is renamed joshua_test). It is responsible for invoking the Python-based TestHarness2 and passing it the necessary configuration. It also handles the creation and cleanup of temporary output directories.
  • Other scripts like bindingTest.sh and valgrindTest.sh are used for different, specialized test runs.

For detailed information on the operation of the Python test runner itself, including its configuration options and output structure, please see the TestHarness2 README.

File: contrib/local_cluster/README.md Copy Chapter

`local_cluster` library

local_cluster library provides a way of spawning local FoundationDB processes.