## File: README.md # UniFFI - a multi-language bindings generator for Rust UniFFI is a toolkit for building cross-platform software components in Rust. For the impatient, see [**the UniFFI user guide**](https://mozilla.github.io/uniffi-rs/) or [**the UniFFI examples**](https://github.com/mozilla/uniffi-rs/tree/main/examples#example-uniffi-components). By writing your core business logic in Rust and describing its interface in an "object model", you can use UniFFI to help you: * Compile your Rust code into a shared library for use on different target platforms. * Generate bindings to load and use the library from different target languages. You can describe your object model in an [interface definition file](https://mozilla.github.io/uniffi-rs/udl_file_spec.html) or [by using proc-macros](https://mozilla.github.io/uniffi-rs/proc_macro/index.html). UniFFI is currently used extensively by Mozilla in Firefox mobile and desktop browsers; written once in Rust, auto-generated bindings allow that functionality to be called from both Kotlin (for Android apps) and Swift (for iOS apps). It also has a growing community of users shipping various cool things to many users. UniFFI comes with support for **Kotlin**, **Swift**, **Python** and **Ruby** with 3rd party bindings available for **C#** and **Golang**. Additional foreign language bindings can be developed externally and we welcome contributions to list them here. See [Third-party foreign language bindings](#third-party-foreign-language-bindings). ## User Guide You can read more about using the tool in [**the UniFFI user guide**](https://mozilla.github.io/uniffi-rs/). We consider it ready for production use, but UniFFI is a long way from a 1.0 release with lots of internal work still going on. We try hard to avoid breaking simple consumers, but more advanced things might break as you upgrade over time. ### Etymology and Pronunciation ˈjuːnɪfaɪ. Pronounced to rhyme with "unify". A portmanteau word that also puns with "unify", to signify the joining of one codebase accessed from many languages. uni - [Latin ūni-, from ūnus, one] FFI - [Abbreviation, Foreign Function Interface] ## Alternative tools Other tools we know of which try and solve a similarly shaped problem are: * [Diplomat](https://github.com/rust-diplomat/diplomat/), which is focused more on C/C++ interop. * [Interoptopus](https://github.com/ralfbiedert/interoptopus/) (Please open a PR if you think other tools should be listed!) ## Third-party foreign language bindings * [Javascript bindings](https://github.com/jhugman/uniffi-bindgen-react-native): running in a web page, targeting WASM; and React Native targeting Android, iOS. The repository contains tooling to generate bindings for [Hermes](https://github.com/facebook/hermes), [creating Turbo Modules](https://reactnative.dev/blog/2024/10/23/the-new-architecture-is-here#new-native-modules), and for creating a [`wasm-bindgen` bindings crate](https://rustwasm.github.io/wasm-bindgen). * [Kotlin Multiplatform support (Gobley)](https://github.com/gobley/gobley). The repository contains Kotlin Multiplatform bindings generation for UniFFI, letting you target both JVM and Native. * [Go bindings](https://github.com/NordSecurity/uniffi-bindgen-go) * [C# bindings](https://github.com/NordSecurity/uniffi-bindgen-cs) * [Dart bindings](https://github.com/NiallBunting/uniffi-rs-dart) * [Java bindings](https://github.com/IronCoreLabs/uniffi-bindgen-java) * [Node bindings](https://github.com/livekit/uniffi-bindgen-node) (early development) * [Node bindings](https://github.com/criccomini/uniffi-bindgen-node-js) ### External resources There are a few third-party resources that make it easier to work with UniFFI: * [Plugin support for `.udl` files](https://github.com/Lonami/uniffi-dl) for the IDEA platform ([*uniffi-dl* in the JetBrains marketplace](https://plugins.jetbrains.com/plugin/20527-uniffi-dl)). It provides syntax highlighting, code folding, code completion, reference resolution and navigation (among others features) for the [UniFFI Definition Language (UDL)](https://mozilla.github.io/uniffi-rs/). * [cargo swift](https://github.com/antoniusnaumann/cargo-swift), a cargo plugin to build a Swift Package from Rust code. It provides an init command for setting up a UniFFI crate and a package command for building a Swift package from Rust code - without the need for additional configuration or build scripts. * [Cargo NDK Gradle Plugin](https://github.com/willir/cargo-ndk-android-gradle) allows you to build Rust code using [`cargo-ndk`](https://github.com/bbqsrc/cargo-ndk), which generally makes Android library builds less painful. * [`uniffi-starter`](https://github.com/ianthetechie/uniffi-starter) is a minimal project demonstrates a wide range of UniFFI in a complete project in a compact manner. It includes a full Android library build process, an XCFramework generation script, and example Swift package structure. (Please open a PR if you think other resources should be listed!) ## Contributing If this tool sounds interesting to you, please help us develop it! You can: * View the [contributor guidelines](./docs/contributing.md). * File or work on [issues](https://github.com/mozilla/uniffi-rs/issues) here in GitHub. * Join discussions in the [#uniffi:mozilla.org](https://matrix.to/#/#uniffi:mozilla.org) room on Matrix. ## Code of Conduct This project is governed by Mozilla's [Community Participation Guidelines](./CODE_OF_CONDUCT.md). --- ## File: docker/README.md This directory contains a Dockerfile for building the `uniffi-ci` docker image that we use for running tests in CI. To build a new version of this docker image, run the following from the root of the repository: ``` docker build -t uniffi-ci -f docker/Dockerfile-build . ``` --- ## File: docs/adr/0000-whats-the-big-idea.md # Write and maintain a custom tool for generating foreign-language bindings to rust code. * Status: accepted * Deciders: rfkelly, linacambridge, eoger, thomcc * Date: 2020-07-01 (or thereabouts) ## Context and Problem Statement On the Application Services team, we have successfully built several re-useable components for sync- and storage-related browser functionality by following what we've dubbed the "rust-components" approach: write the bulk of the code in rust so we can cross-compile it for different target platforms, have it expose a C-compatible FFI layer, then write a small amount of FFI bindings code to expose the functionality to each of several different target languages (e.g. Swift and Kotlin). The FFI layer and foreign-language bindings code is currently written by hand, a tedious and potentially error-prone process. Given that we expect to build additional components in this style in the future, and expect more teams at Mozilla to do the same, can we increase the efficiency and reliability of this work by auto-generating some of this code? ## Decision Drivers * Reduce time taken to launch a new rust component. * Improve maintainability of existing rust components. * Reduce possibility of errors in hand-written foreign language bindings code. * Continue shipping components on a regular cadence. ## Considered Options * Option A: Continue writing the FFI and foreign-language parts of rust components by hand * Option B: Move to using WebAssembly and wasm-bindgen * Option C: Use SWIG, Djinni, or another existing bindings-generator tool * Option D: Write and maintain a custom tool that automates our current best practices ## Decision Outcome Chosen option: * **Option D: Write and maintain a custom tool that automates our current best practices** On balance, this option provides us with the best tradeoff of potential upside and the ability to limit downside. If the approach succeeds then we expect to realize significant improvement in maintenance costs of rust-components code by reducing boilerplate and human error. Building our own will involve the least up-front investment before we can start to show results, because we did not identify any existing tools that were a close-enough fit for our needs. The first versions of the tool don't have to be perfect, or even particularly *good* - they just have to have a better value-proposition than writing the generated code by hand. We accept the risk that writing our own tool for this may turn out to be much more complex than expected, and will mitigate it by aggressively time-boxing initial prototypes, by developing it in parallel with a real shipping consumer with real deadlines, and by regularly asking the hard questions about whether the approach is working out. ## Pros and Cons of the Options ### Option A: Continue writing the FFI and foreign-language parts of rust components by hand We could decide that hand-writing some `pub extern "C"` function wrappers and a bit of custom Swift, Kotlin, etc isn't all that bad, and that the cost of doing so is unlikely to be offset by an investment in more automated tooling. * Good, because we can dedicate more people to building new component functionality, rather than working on tooling. * Good, because each component can use whatever bespoke FFI details work best for its use-case, rather than taking a one-size-fits-all approach. * Good, because we don't have to learn a new tool or maintain an existing one. * Bad, because the time commitment for maintaining bindings will only grow as we build more components. * Bad, because it's easy to make mistakes when writing the bindings by hand, and it has proven hard to avoid making similar mistakes multiple times. * Bad, because hand-writing bindings is low-engagement work that risks feeling like a chore, and we have plenty of other chores already. Ultimately, it feels like the potential long-term cost savings of automation will be significant, but we are glad to continue having this option as a Plan B. ### Option B: Move to using WebAssembly and wasm-bindgen The approach we've taken with rust components has many similarities to WebAssembly, particularly the [WebAssembly Interface Types](https://hacks.mozilla.org/2019/08/webassembly-interface-types/) proposal. We could try to use [`wasm-bindgen`](https://github.com/rustwasm/wasm-bindgen) to automatically generating bindings to our rust code, and rely on the portable nature of WebAssembly to run on multiple platforms. * Good, because the tooling around `wasm-bindgen` seems quite sophisticated and mature. * Good, because this toolchain is maintained by folks for whom it is a full-time job. * Good, because it meshes well with technical projects that are strategically important for Mozilla. * Bad, because `wasm-bindgen` currently only supports JavaScript as a target language. * Bad, because we have not been able to identify a mature solution for running WebAssembly on Android or iOS, which are important target platforms. * Bad, because this is a significant departure from how we've written components in the past, which adds timeline risk to shipping this solution. Ultimately, we can imagine a world in which the WebAssembly ecosystem is sufficiently advanced to make this the most compelling option, but that world seems far enough away that it doesn't make sense for us to pursue this right now. ### Option C: Use SWIG, Djinni, or another existing bindings-generator tool Writing code in a systems language and generating bindings for high-level languages is not a new idea. Among existing tools in this general space are [SWIG](http://www.swig.org/) and [Djinni](https://github.com/dropbox/djinni). We could adopt one of these existing tools instead of inventing our own thing. * Good, because these tools already exist and are mature, saving us development and maintenance effort. * Good, because it would realize the goal of avoiding hand-written boilerplate. * SWIG: * Bad, because it doesn't appear to have support for generating Kotlin or Swift bindings, which are key languages for our use-case. * Bad, because it is designed for C/C++ rather than Rust, meaning an unknown amount of exploratory work required to integrate it with our approach before we can ship anything. * Djinni: * Good, because it targets several of our key languages/platforms. * Bad, because it is designed using C++ as the implementation language rather than Rust, meaning an unknown amount of exploratory work required to integrate it with our approach before we can ship anything. * Bad, because it has explicitly been put into "maintenance mode" by its authors. Ultimately, while we could probably make one of these tools work, we could not find one that was a close enough fit for our needs to avoid the "unknown amount of exploratory integration work" problem. We're not willing to put off shipping incremental progress for long enough to be confident of making the integration work. ### Option D: Write and maintain a custom tool that automates our current best practices We can take the patterns we've established for writing the FFI layer and foreign language bindings by hand, and encapsulate them in a custom tool to automatically generate similar code. * Good, because it would realize the goal of avoiding hand-written boilerplate. * Good, because the first version of the tool only has to be good enough for our limited needs, meaning we can defer some complexity until after we've proven out the idea. * Good, because it can be designed up-front to meet some of our unusual needs (integrating with Firefox Desktop code, being built as part of a larger shared library, etc). * Good, because we should be able to determine whether the approach is working within a fairly strict timebox, and fall back to hand-written bindings if required. * Bad, because we take on all the development and maintenance burden of the tool, reducing time that can be spent on product features. * Bad, because we risk isolating knowledge of how to tool works in a small number of people. * Bad, because we might spend more time on developing and maintaining the tool than we'd ever hope to save from the generated code. * Bad, because the bindings will be limited to a one-size-fits-all, lowest-common-denominator feature set. * Bad, because the generated code risks being much harder to debug than hand-written bindings, especially while the tool itself is under heavy development. Ultimately, while there are risks with this approach, they seem sufficiently well-understood and well-bounded that we can try out the approach, and fall back to hand-written bindings if it doesn't seem to be working out. ## Links * [Engineering Program Review: Sync & Storage Components](https://docs.google.com/document/d/10I8MD_narf3D7w1F0rciye-cRpwKO6ItE_UdCtFKTmA); an earlier technical review of the "rust components" approach, including some discussion of the pain-points around manually writing FFI bindings. * [The (not so) hidden cost of sharing code between iOS and Android](https://dropbox.tech/mobile/the-not-so-hidden-cost-of-sharing-code-between-ios-and-android); a kind of technical post-mortem from Dropbox exploring why they abandoned a code-sharing approach that is similarly shaped to the one we're pursuing with rust components. Many of the risks highlighted in this post also apply to our chosen solution. --- ## File: docs/adr/0001-mvp-webidl.md # Build an MVP based on WebIDL and a manual workflow * Status: accepted * Deciders: rfkelly, linacambridge, eoger, jhugman, tarikeshaq * Date: 2020-07-01 (or thereabouts) ## Context and Problem Statement When [deciding to build this tool](./0000-whats-the-big-idea.md), the main risk identified was that we'd spend too much time on an ultimately unworkable or unmaintainable idea. What early design decisions can we make to mitigate this risk? What things are an existential risk to the success of this project that must be included in the first version, and what things can we safely defer to future work? In other words: how do we build an MVP of this tool that is both *minimal* and *viable*? ## Decision Drivers * Strictly timebox our efforts to "prove out" the approach. * Establish whether we can effectively maintain this kind of tool as a team. * Support initial development of a new rust component with externally-imposed, near-term deadlines. ## Considered Options This ADR encompasses several several related design questions, all of which feed together into an overall approach to building the MVP of the tool: * How will developers specify the API of their component? * Option A: Use an external interface definition file based on WebIDL. * Option B: Use an external interface definition file based on a custom language. * Option C: Infer the API directly from the rust code using annotations and macros. * How will developers integrate the tool into their workflow? * Option D: Use build-scripts and macros to deeply integrate with the rust crate's build system. * Option E: Provide a tool that developers need to run by hand. * How will we prioritize work on the capabilities offered by the tool itself? * Option F: Go broad, implementing many data types and API capabilities, even if they're slow or incomplete. * Option G: Go deep, implementing a few core data types and API capabilities, and make sure they're done well. ## Decision Outcome Chosen options: * **Option A: Use an external interface definition file based on WebIDL.** * **Option E: Provide a tool that developers need to run by hand.** * **Option F: Go broad, implementing many data types and API capabilities, even if they're slow or incomplete.** The set of options chosen here makes an explicit tradeoff, preferring to get something up and running quickly and accepting a certain amount of jank in the developer experience. We don't have to build the perfect tool right away, we only have to build something that's better than doing this work by hand. If we like the result we can polish it from there. The MVP tool will read API definitions from an external WebIDL file. This will be a bit weird and inconvenient for consumers because WebIDL is not a precise fit for our needs, but it avoids us bikeshedding the perfect API-definition experience during this first phase. The MVP developer experience will involve `cargo install`ing the tool onto your system and manually running it or integrating it into your build process. This risks being mildly inconvenient for consumers, but gives us lots of flexibility while we learn about what a better workflow might look like. The MVP tool may support more features than turn out to be strictly necessary, in the interests of ensuring multiple team members can be involved in its development at this early stage. As a tradeoff, the MVP generated code will be allowed to contain inefficiencies and limitations that hand-written code might not, on the premise that our first consumers are not very performance-sensitive, and that there is a lot of scope for improving these implementation details over time. We are likely to ***revisit every single one of these choices*** if the MVP of the tool proves successful, and will attempt to build it in such a what that they're easy to revisit. ## Pros and Cons of the Options ### Option A: Use an external interface definition file based on WebIDL. We can require developers to specify their component API in an external definition file, using the syntax of WebIDL to provide something that's familiar and has an existing spec. * Good, because WebIDL exists and has the features we need for our first consumer. * Good, because the `weedle` crate provides a ready-made parser for WebIDL. * Good, because WebIDL has some base level of familiarity around Mozilla. * Bad, because developers will need to duplicate their API, once in the Rust code and once in the IDL. * Bad, because WebIDL is designed for a different use-case, so it's likely to be an awkward fit. * Bad, because `weedle` doesn't generate particularly helpful error messages (it seems designed for parsing known-good WebIDL definitions rather than helping you develop new ones). Ultimately, this seems like the lowest-cost way to get started, while deferring the important-but-not-existentially-risky work of making an IDL experience that fits really well with Rust code. ### Option B: Use an external interface definition file based on a custom language. We can require developers to specify their component API in an external definition file, using our own custom variant of an IDL syntax. * Good, because the syntax can be custom designed to fit well with the developer's mental model of the generated code. * Good, because we already have several different IDL variants in use at Mozilla (WebIDL, XPIDL), so our chances of building something that feels familiar are high. * Bad, because developers will need to duplicate their API, once in the Rust code and once in the IDL. * Bad, because we have to make up and document a whole syntax. * Bad, because rust parsing crates such as `nom` do not seem to generate particularly helpful error messages by default, adding friction for the developer. Ultimately, while a custom syntax would probably "feel" better from the consumer's perspective than WebIDL, the costs involved are not worth that tradeoff for the MVP. Beyond the MVP we expect that direct annotation of the Rust code will provide a better developer experience, leaving this option as an unnecessary middle-ground. ### Option C: Infer the API directly from the rust code using annotations and macros. We can allow developers to sprinkle some macro annotations directly on their rust code in order to declare the component API, similar to the approach taken by `wasm-bindgen`. * Good, the Rust code is a single source of truth for the API definition. * Good, because developers are familiar with this approach from other tools. * Bad, because our team doesn't have much experience working with macros at scale. * Bad, because from a poke around in the `wasm-bindgen` code, they seem to need to do some pretty scary things in order to make the macros Just Work in various edge-cases. Ultimately, this feels like a good approach longer-term, but risks being too much of a time-sink for the MVP. ### Option D: Use build-scripts and macros to deeply integrate with the rust crate's build system. We can encourage consumers to structure their rust component as a crate, take a build-time dependency on our tool, and magic things into existence as part of `cargo build`. * Good, because it's a slick developer experience if we can make it work. * Bad, because it assumes many details of how the consuming component is being built and deployed, and we don't know exactly how that will work yet. * Bad, because it could be hard to integrate with e.g. a gradle-based build system for android packages. * Bad, because build scripts aren't supposed to create files outside of the rust target directory, but it doesn't really make sense to generate foreign language bindings into that directory. Ultimately, this approach does not provide enough flexibility for initial consumers, risking them declaring it a bad fit based on non-essential details of the tool itself. ### Option E: Provide a tool that developers need to run by hand. We can provide consumers with a `uniffi-bindgen` command-line tool that they manually run on their component code in order to generate foreign language bindings. * Good, because gives the consumer lots of flexibility and when and where to generate the different bits of code. * Bad, because consumers have to install an external tool. Ultimately, this approach wins based on flexibility. We may also provide a light wrapper around the tool that integrates it with `cargo build` for convenience. ### Option F: Go broad, implementing many data types and API capabilities, even if they're slow or incomplete. We can focus our initial efforts on fleshing out a broad suite of data types and API capabilities, spending less time focused on performance or edge-cases in the generated code. * Good, because consumers can iterate their API with less chance of being limited by the tool. * Good, because it makes more opportunities for team members to get involved in implementing features of the tool itself, helping us understand what it will be like to maintain it over time. * Bad, because sub-optimal performance may be offputting for consumers. * Bad, because we might accidentally entrench design decisions that limit our ability to improve the generated code in future. Ultimately, this option wins based on the known needs of our first target consumer (which favours iteration over performance) and the of team itself (which wants to ensure multiple developers are familiar with the tool's codebase as it gets off the ground). ### Option G: Go deep, implementing a few core data types and API capabilities, and make sure they're done well. We can focus our initial efforts on identifying just the data types and API capabilities required by our first target consumer and implementing them really well, spending less time on features that are unlikely to be required by the consumer. * Good, because it shows the resulting generated code in the best possible light. * Bad, because we might not identify the correct set of features. * Bad, because it's harder to parallelize this kind of work among multiple team members. Ultimately, the needs of our first target consumer make the "performance" argument fairly weak, so this option was not selected. ## Links * [The WebIDL specification](https://heycam.github.io/webidl/), for reference. --- ## File: docs/adr/0002-serialize-complex-datatypes.md # The MVP will pass complex datatypes over the FFI using simple, explicit serialization * Status: accepted * Deciders: rfkelly * Date: 2020-07-01 (or thereabouts) ## Context and Problem Statement Passing complex data-types from Rust to foreign-language code and back again can be, well, *complex*. Given a Rust struct with named fields, and a corresponding autogenerated data class in the foreign language bindings, how does one turn into the other? ## Decision Drivers * Ensuring safety of the generated code. * Fitting with our [MVP goal](./0001-mvp-webidl.md) of favouring initial-time-to-ship over performance. ## Considered Options * Option A: Declare complex datatypes using Protocol Buffers, pass them as serialized bytes. * Option B: Declare complex datatypes `[#repr(C)]` structs and pass them directly. * Option C: Implement a simple direct serialization scheme, pass them as serialized bytes. ## Decision Outcome Chosen option: * **Option C: Implement a simple direct serialization scheme, pass them as serialized bytes.** The choice here comes down to simplicity and safety for the MVP. Serializing complex datatypes into a bytebuffer makes it easier for us to pass the data safely across the FFI, because it strictly controls shared access to memory on each side of the boundary. Using our own simple serialization scheme reducing the number of moving parts relative to third-party serialization libraries. This choice comes with non-trivial performance costs, but that's acceptable for MVP. We are likely to ***revisit this choice*** if the MVP of the tool proves successful and expect that we can do so incrementally without changing the consumer-facing experience. ## Pros and Cons of the Options ### Option A: Declare complex datatypes using Protocol Buffers, pass them as serialized bytes. Following the approach currently taken by hand-written component bindings in application-services, we could generate a Protocol Buffers schema for each complex data type, and use generated serialization code to send it across the FFI as a bytebuffer. * Good, because we're familiar with this approach from our existing components. * Good, because Protocol Buffers exist and have decent tooling for generating serialization code in our various target languages. * Good, because it's hard to mishandle the data in a way that introduces memory-safety issues. * Bad, because we'd be calling the protobuf code-generator from inside our own code-generator, which seems like a very complex setup. * Bad, because there's likely to be a noticeable serialization-related performance overhead. * Bad, because Protocol Buffers contain some complexity that isn't useful for our use-case, such as affordances for backwards-compatibility. A similar set of considerations apply to other third-party serialization schemes such as flatbuffers, with the added disadvantage of unfamiliarity. Ultimately, the additional build complexity of integrating a code-generator inside our own code generator makes this option unattractive. ### Option B: Declare complex datatypes `[#repr(C)]` structs and pass them directly. Following the approach currently taken by hand-written component bindings in the glean project, we could generate a `#[repr(C)]` struct for each complex data type and a corresponding struct in the foreign language bindings, then pass structs directly across the FFI boundary either by value or as pointers. * Good, it has minimal serialization overhead. * Good, because it can avoid copying data from rust-memory to foreign-language-memory. * Bad, because using raw pointers gives more opportunities for memory-safety issues. * Bad, because it's not obvious how to handle nested values without using raw pointers. * Bad, because the codegen seems likely to be more complex than other options. Ultimately, given that our MVP explicitly prioritizes features over performance, this option is not a good choice for the MVP. It seems worthwhile revisiting as an option for post-MVP when performance becomes more important. ### Option C: Implement a simple direct serialization scheme, pass them as serialized bytes. We could invent a very simple serialization scheme without all the bells-and-whistles of Protocol Buffers, and serialize complex data types to send them across the FFI as a bytebuffer. * Good, because this is similar to the familiar protocol-buffers approach. * Good, because we can directly codegen the serialization logic as part of generating other code for the data structure. * Good, because it's hard to mishandle the data in a way that introduces memory-safety issues. * Bad, because we might introduce bugs in a new codebase that have already been shaken out in a more mature third-party serialization library. * Bad, because there's likely to be a noticeable serialization-related performance overhead. Ultimately, the simplicity and safety of this approach make it the best choice for our MVP. ## Links * [Crossing the Rust FFI frontier with Protocol Buffers](https://hacks.mozilla.org/2019/04/crossing-the-rust-ffi-frontier-with-protocol-buffers/); a discussion of how Application Services components pass complex data types by serializing them with protocol buffers, and deserializing on the other side. * [This Week in Glean: Bytes in Memory (on Android)](https://fnordig.de/2020/05/04/this-week-in-glean/); a discussion of how the Glean library passes complex data types by mapping them into `repr(c)` structs. --- ## File: docs/adr/0003-threadsafe-interfaces.md # Let consumers opt out of HandleMap locking if their interface is threadsafe * Status: accepted * Deciders: rfkelly, jhugman, mhammond, dmosedale * Date: 2021-01-13 ## Context and Problem Statement Uniffi currently uses a very coarse locking strategy for managing concurrent access to object instances, which has caused us to accidentally ship code in a product that [blocked the main thread on network I/O](https://jira.mozilla.com/browse/SDK-157). We need to enable finer-grained concurrency control in order to provide the desired API for a key consumer. Currently, every interface has a corresponding [ffi_support::ConcurrentHandleMap](https://docs.rs/ffi-support/0.4.0/ffi_support/handle_map/struct.ConcurrentHandleMap.html) that is responsible for owning all instances of that interface and for handing out references to them in a mutability-safe and threadsafe manner. This ensures that the generated code is safe in the face of concurrent operations, but has a substantial runtime cost: only one method call can be executed on an instance at any time. Any attempt to call an object method while a concurrent method is already executing, will block until the previous call has completed. The desired API for Project Nimbus includes methods that will be called synchronously from the main thread, and hence must not block on network or disk I/O. Such an API cannot be built with uniffi as currently implemented. ## Decision Drivers * Enabling consumers to control the potential blocking behaviour of their generated APIs. * Ensure safety of the generated code. * Ship a solution in a timely manner to unblock Project Nimbus. ## Considered Options * Option A: Do nothing, require consumers to assume that all method calls might block. * Option B: Let consumers mark interface definitions as threadsafe to opt in to a less-locking handlemap. * Option C: Insist that all interfaces be threadsafe and replace the handlemap with raw pointers. * Option D: Use a less-locking handlemap and rely on calling code to behave safely. ## Decision Outcome Chosen option: * **Option B: Let consumers mark interface definitions as threadsafe to opt in to a less-locking handlemap.** The choice here comes down to safety and simplicity. By making a more-concurrency-friendly handlemap we can maintain the current strict enforcement of Rust's mutability-safety and thread-safety guarantees, even in the face of errors in the generated bindings. It seems to be a relatively small change, and by making it opt-in we avoid creating busywork for other consumers who are not urgently facing this problem. One downside is that consumers need to opt-in to the fix, meaning that the default behavior may still be surprising to new consumers. We'll mitigate this with docs and will consider revisiting the default behaviour if the majority of consumers adopt the new approach. This choice does also punt some potential performance improvements to future work, but that seems in keeping with where we are in the project's lifecycle. ## Pros and Cons of the Options ### Option A: Do nothing, require consumers to assume that all method calls might block. Make no changes to uniffi, and instead accept the fact that method calls are executed serially. Document this limitation and work with consumers to update their API definitions to account for it. * Good, because we can ship this quickly. * Good, because we don't give up any of the safety guarantees of the current approach. * Bad, because it makes it basically impossible to meet the needs of one of our key early consumers, and would force us into awkward compromises around a suboptimal API. * Bad, because it makes uniffi less attractive to potential future consumers. * Bad, because it keeps all the run-time overhead of the handlemap. * Bad, because the default behaviour still has a hidden mutex, which might be a nasty surprise for future consumers in the same way that it was for Nimbus. Ultimately, we want uniffi to be a tool that helps consumers deliver value, not something that foist limitations upon them, which makes this option unattractive. ### Option B: Let consumers mark interface definitions as threadsafe to opt in to a less-locking handlemap. Implement a variant of `ConcurrentHandleMap` that does not protect its members with a Mutex, which is the main source of constraints on concurrent execution in the current setup. Instead, this handlemap would only give out immutable references to its members, and would insist that its members are `Send` and `Sync` so they can be safely accessed from multiple threads. Introduce a new annotation to the UDL so that `interface` definitions can be declared as threadsafe. In the generated Rust scaffolding, use the new handlemap for threadsafe interfaces but keep using the existing `ConcurrentHandleMap` by default. * Good, because this is a relatively small change from the current behaviour, which should be fairly quick to ship. * Good, because this is a non-breaking change for consumers who don't opt in to it. * Good, because the `Sync` and `Send` bounds on the underlying struct will help consumers to implement their own fine-grained concurrency control while being supported by Rust's compile-time guarantees. * Good, because consumers don't have to care about this until they discover that they need fine-grained locking. * Good, because it leaves the door open to further improvements (like "Option C") in the future if we decide that makes sense later. * Bad, because it keeps all the run-time overhead of the handlemap. * Bad, because now we need to maintain two handlemap variants. * Bad, because the default behaviour still has a hidden mutex, which might be a nasty surprise for future consumers in the same way that it was for Nimbus. This is the solution we have ultimately selected. ### Option C: Insist that all interfaces be threadsafe and replace the handlemap with raw pointers. Stop using `ConcurrentHandleMap` to intermediate object access. Instead, put each object instance in a `Box` and use `Box::into_raw` to transfer ownership of the box to the foreign language code as a raw pointer. Maintain safety by insisting that any struct implementing a UDL `interface` must be `Sync` and `Send`, and by refusing to hand out mutable references to the boxed instance. * Good, because it removes the issue as a potential footgun for all consumers. * Good, because the `Sync` and `Send` bounds on the underlying struct will help consumers to implement their own fine-grained concurrency control while being supported by Rust's compile-time guarantees. * Good, because it likely reduces runtime overhead and gives a small performance boost. * Bad, because it's a breaking change for all consumers, even if they don't care about this issue. * Bad, because it increases the amount of unsafe rust that we need to emit in the generated scaffolding. * Bad, because we lose some additional runtime checks provided by the handlemap, such as guarding against passing a handle that belongs to a different datatype. * Bad, because it's a non-trivial departure from the current approach which will take more time to QA. This seems like a promising longer-term option, especially if we find that the majority of consumers are opting in to the fix proposed in this ADR. But the additional complexity weights heavily against trying to ship this as an initial fix for Project Nimbus. ### Option D: Use a less-locking handlemap and rely on calling code to behave safely. Implement a variant of `ConcurrentHandleMap` that does not protect its members with a Mutex, which is the main source of constraints on concurrent execution in the current setup. Instead, this handlemap would hand out references without any runtime checks, on the assumption that the calling code is behaving in a safe manner. Replace all currently uses of `ConcurrentHandleMap` with this new less-locking variant. * Good, because this is a relatively small change from the current behaviour, which should be fairly quick to ship. * Bad, because we lost Rust's compile-time safety guarantees. * Bad, because it's hard to communicate to consumers what "behave safely" really means (and we might not even understand it ourselves). This seems to throw away some of the key safety benefits of using Rust, which makes it a very unattractive option. ## Implementation Sketch ### Add a `[Threadsafe]` attribute to UDL `interface` definitions. We would advise Nimbus SDK to update their UDL definition for `NimbusClient` like so: ```idl [Threadsafe] interface NimbusClient { // existing method definitions remain unchanged }; ``` The [`uniffi_bindgen::interface::Object`](https://github.com/mozilla/uniffi-rs/blob/803bb3d79daa8ea088fb2d8f05c08ada09821986/uniffi_bindgen/src/interface/mod.rs#L722) struct would grow a corresponding boolean `threadsafe` field and corresponding public accessor. When [building an instance of this struct from the UDL](https://github.com/mozilla/uniffi-rs/blob/803bb3d79daa8ea088fb2d8f05c08ada09821986/uniffi_bindgen/src/interface/mod.rs#L786), we would inspect the list of attributes to look for one named "Threadsafe" and set the field if present. The way that we [handle the `[ByRef]` annotation on method arguments](https://github.com/mozilla/uniffi-rs/blob/803bb3d79daa8ea088fb2d8f05c08ada09821986/uniffi_bindgen/src/interface/mod.rs#L658) will likely serve as a good example to follow. ### Implement a less-locking HandleMap variant The `ffi_support` crate provides a basic non-locking [`HandleMap`](https://docs.rs/ffi-support/0.4.0/ffi_support/handle_map/struct.HandleMap.html) struct, and it implements `ConcurrentHandleMap` as a thin wrapper around a `RwLock>>`. We can make our own variant that removes the inner mutex, as a thin wrapper around a`RwLock>>`. (The outer `RwLock` is still needed in order to guard mutations of the handlemap itself, while the `Arc` is needed so that we can quickly clone a reference and release the lock before calling potentially-long-running methods on that reference). Bikeshed name: `LessLockingHandleMap`, implemented under the [`uniffi::ffi` module](https://github.com/mozilla/uniffi-rs/tree/main/uniffi/src/ffi). It needs to live in the `uniffi` crate so that it can be used by the generated Rust scaffolding at runtime. We don't need to support the entire `ConcurrentHandleMap` interface, only: * `insert_with_output` * `insert_with_result` * `call_with_output` * `call_with_result` * `delete_u64` The `LessLockingHandleMap` struct should be capable of handing out `&T` references but should never hand out a `&mut T`. It should also require that `T: Send + Sync`. ### Use `LessLockingHandleMap` in the Rust scaffolding for `[Threadsafe]` instances In [ObjectTemplate.rs](https://github.com/mozilla/uniffi-rs/blob/main/uniffi_bindgen/src/templates/ObjectTemplate.rs), check the `threadsafe` property of the object definition. If it's true, use `LessLockingHandleMap` instead of `ConcurrentHandleMap` in the [lazy static that declares the handlemap for that interface](https://github.com/mozilla/uniffi-rs/blob/803bb3d79daa8ea088fb2d8f05c08ada09821986/uniffi_bindgen/src/templates/ObjectTemplate.rs#L11). By ensuring that `LessLockingHandleMap` and `ConcurrentHandleMap` expose a similar API, this will hopefully be a fairly minimal if-then-else that just chooses the correct name of the struct to use. ### ~~Optional: Add a `[Blocking]` annotation to methods and functions~~ (This was moved into a follow-up issue, ref [#378](https://github.com/mozilla/uniffi-rs/issues/378)). ## Links * [Fenix Bug: Large regression in MAIN/VIEW start up](https://jira.mozilla.com/browse/SDK-157). * [An proof-of-concept experiment in using raw pointers rather than a handlemap](https://github.com/mozilla/uniffi-rs/pull/366). * [The final implementation of this ADR: #372](https://github.com/mozilla/uniffi-rs/pull/372) --- ## File: docs/adr/0004-only-threadsafe-interfaces.md # Remove support for non-`Send+Sync` interfaces * Status: accepted * Deciders: mhammond, rfkelly, travis, jhugman, dmose * Date: 2021-04-19 Discussion and approval: [PR 421](https://github.com/mozilla/uniffi-rs/pull/421) Technical Story: [Issue 419](https://github.com/mozilla/uniffi-rs/issues/419) ## Context and Problem Statement [ADR-0003](0003-threadsafe-interfaces.md) introduced support for "thread-safe interfaces" - possibly leading to the impression that there is such a thing as non-threadsafe interfaces and confusion about exactly what the attribute means. However, the entire concept of non-threadsafe interfaces is a misconception - the Rust compiler insists that everything wrapped by uniffi is thread-safe - the only question is who manages this thread-safety. Interfaces which are not marked as thread-safe cause uniffi to wrap the interface in a mutex which is hidden in the generated code and therefore not obvious to the casual reader. The `[Threadsafe]` marker acts as a way for the component author to opt out of the overhead and blocking behaviour of this mutex, at the cost of opting in to managing their own locking internally. This ADR proposes that uniffi forces component authors to explicitly manage that locking in all cases - or to put this in Rust terms, that all structs supported by uniffi must already be `Send+Sync` Note that this ADR will hence-forth use the term `Send+Sync` instead of "Threadsafe" because it more accurately describes the actual intent and avoids any misunderstandings that might be caused by using the somewhat broad and generic "Threadsafe". ## Decision Drivers * Supporting non-`Send+Sync` structs means uniffi must add hidden locking to make them `Send+Sync`. We consider this a "foot-gun" as it may lead to accidentally having method calls unexpectedly block for long periods, such as [this Fenix bug](https://github.com/mozilla-mobile/fenix/issues/17086) (with more details available in [this JIRA ticket](https://jira.mozilla.com/browse/SDK-157)). * Supporting such structs will hinder uniffi growing in directions that we've found are desired in practice, such as allowing structs to use [alternative method receivers](https://github.com/mozilla/uniffi-rs/issues/417) or to [pass interface references over the FFI](https://github.com/mozilla/uniffi-rs/issues/419). ## Considered Options * [Option 1] Continue supporting non-`Send+Sync` interfaces while also working on the enhancements listed above, but exclude non-`Send+Sync` interfaces from such enhancements. * [Option 2] Immediately deprecate, then remove entirely, support for non-`Send+Sync` interfaces. ## Decision Outcome Chosen option: * **[Option 2] Immediately deprecate, then remove entirely, support for non-`Send+Sync` interfaces.** This decision was taken because our real world experience tells us that non-`Send+Sync` interfaces are only useful in toy or example applications (eg, the nimbus and autofill projects didn't get very far before needing these capabilities), so the extra ongoing work in supporting these interfaces cannot be justified. ### Positive Consequences * The locking in all uniffi supported components will be more easily discoverable - it will be in hand-written rust code and not hidden inside generated code. This is a benefit to the developers of the uniffi supported component rather than to the consumers of it; while we are considering other features to help communicate the lock semantics to such consumers, that is beyond the scope of this ADR. * Opens the door to enhancements that would be impossible for non-`Send+Sync` interfaces, and simpler to implement for `Send+Sync` interfaces if support for non-`Send+Sync` interfaces did not exist. * Simpler implementation and documentation. ### Negative Consequences * All consumers (both inside Mozilla and external) will need to change their interfaces to be `Send+Sync`. As an example of what this entails, see [this commit](https://github.com/mozilla/uniffi-rs/commit/454dfff6aa560dffad980a9258853108a44d5985) which converts the `todolist` example. * Simple, toy applications may be more difficult to wrap - consumers will not be able to defer decisions about `Send+Sync` support and will instead need to implement simple locking as demonstrated in [this commit]( https://github.com/mozilla/uniffi-rs/commit/454dfff6aa560dffad980a9258853108a44d5985). * Existing applications that are yet to consider how to make their implementations `Send+Sync` cannot be wrapped until they have. * The examples which aren't currently marked with the `[Threadsafe]` attribute will become more complex as they will all need to implement and explain how they achieve being `Send+Sync`. * The perception that its more difficult to wrap interfaces will lead to less adoption of the tool. ## Pros and Cons of the Options ### [Option 1] * Good, because we don't break anyone. * Bad, because we believe non-`Send+Sync` interfaces aren't useful in the real-world, but we would pay the maintenance cost as though they were. * Bad, because locking remains hidden and leaves the door open to the same gun we have already shot ourselves in the foot with. ### [Option 2] * Good, because it makes the implementation of desired features easier. * Good, because it removes a foot-gun and makes locking both explicit and visible to the developers of the uniffi-wrapped component. * Bad, because it breaks existing external consumers - it also breaks a couple of internal consumers (for example, [fxa-client]( https://github.com/mozilla/application-services/blob/f3f0cf6e3386bf3036b074dad3950389cbd05746/components/fxa-client/src/fxa_client.udl#L97)), but we believe fixing them is easy and low cost. ## Implications We know there are external consumers and we know that this will break them. Therefore, we will commit to the following actions: * Communicating both this decision and how consumers can work around it as soon as possible. * Noisily deprecate non-`Send+Sync` interfaces in at least 1 release, so existing consumers are likely to see warnings and a link to our documentation when they upgrade. * Upgrade all internal mozilla consumers as soon as possible so they do not issue deprecation warnings. As an example of what this entails, see [this PR](https://github.com/mozilla/uniffi-rs/commit/454dfff6aa560dffad980a9258853108a44d5985) which converts the `todolist` example to be `Send+Sync`. * Perform the actual removal as late as possible (ie, until support for non `Send+Sync` interfaces actually inhibits our ability to add new features). Concretely, the actual removal involves: * Making `[Threadsafe]` the default. The attribute will not be immediately removed as that would break existing `Send+Sync` components, although we will mark it as deprecated and remove it on an aggressive timeline as the attribute may be confusing given `Send+Sync` would now be the default. * Remove support for generating the `Send+Sync` support in generated rust. This will cause rust objects that don't support `Send + Sync` to fail to compile. ## Links * Logical extension of [ADR-0003](0003-threadsafe-interfaces.md) --- ## File: docs/adr/0005-arc-pointers.md # Use raw `Arc` pointers to pass objects across the FFI * Status: proposed * Deciders: mhammond, rfkelly * Consulted: travis, jhugman, dmose * Date: 2021-04-19 Discussion and approval: [PR 430](https://github.com/mozilla/uniffi-rs/pull/430). Technical Story: [Issue 419](https://github.com/mozilla/uniffi-rs/issues/419). Prototype: [PR 420](https://github.com/mozilla/uniffi-rs/pull/420). ## Context and Problem Statement UniFFI currently manages object instances using the `HandleMap` struct in the ffi-support crate. This means that external consumers of UniFFI-wrapped interfaces never see any pointers to structs - instead, they get what is (roughly) an index into an array, with the struct being stored in (and owned by) that array. This has a number of safety characteristics which are particularly important for hand-written FFI interfaces, but it does cause some issues in evolving UniFFI in directions we consider important. In addition to the slight performance overhead, the use of `HandleMap`s makes it difficult to support: * Passing object instances as arguments ([#40](https://github.com/mozilla/uniffi-rs/issues/40)). Getting objects out of a `HandleMap` involves a closure, so accepting multiple object-typed arguments would involve code-generating nested closures. * Returning object instances from functions ([#197](https://github.com/mozilla/uniffi-rs/issues/197)). Does the returned object already exist in the handlemap? If so, what is its handle? How will we manage the lifetime of multiple references to the object? These restrictions mean that UniFFI's `Object` type is currently only suitable as the `self` argument for method calls, and is forbidden in argument position, as record fields, etc. This ADR considers ways to evolve the handling of object instances and their lifetimes, so that references to structs can be used more widely than currently allowed. ## Decision Drivers * We desire the ability to have more flexible lifetimes for object interfaces, so they can be stored in dictionaries or other interfaces, and be returned by functions or methods other than constructors. * We would like to keep the UniFFI implementation as simple as possible while providing a suitable degree of safety - in particular, a promise that it should be impossible to misuse the generated bindings in a way that triggers Rust's "undefined behavior" or otherwise defeats Rust's safety characteristics and ownership model (and in particular, avoiding things like use-after-free issues). * We would like to keep the overhead of UniFFI as small as possible so that it is a viable solution to more use-cases. ## Considered Options * **[Option 1] We extend the `HandleMap` abstraction to track lifetimes and support easier codegen** This would involve deciding on how we want to track lifetimes (eg, via a reference counting or garbage collection) and actually building it. * **[Option 2] We replace `HandleMap` with raw pointers to Rust's builtin `Arc`** We replace the use of HandleMaps with Rust `Arc<>`, using `Arc::into_raw` to pass values to the foreign-language code and `Arc::from_raw` to receive them back in Rust. * **[Option 3] We replace `HandleMap` with raw pointers to a special-purpose reference container** We replace the use of HandleMaps with something like [triomphe::Arc](https://docs.rs/triomphe/0.1.2/triomphe/), that is specifically intended for use in FFI code, using `Arc::into_raw` to pass values to the foreign-language code and `Arc::from_raw` to receive them back in Rust. ## Decision Outcome Chosen option: * **[Option 2] We replace `HandleMap` with raw pointers to Rust's builtin `Arc`** This decision is taken because: * We believe the additional safety offered by `HandleMap`s is far less important for this use-case, because the code using these pointers is generated instead of hand-written. * Correctly implementing better lifetime management in a thread-safe way is not trivial and subtle errors there would defeat all the safety mechanisms the `HandleMap`s offer. Ultimately we'd just end up reimplementing `Arc<>` anyway, and the one in the stdlib is far more likely to be correct. * There are usability and familiarity benefits to using the stdlib `Arc<>` rather than a special-purpose container like `triomphe::Arc`, and the way we currently do codegen means we're unlikely to notice any potential performance improvements from using a more specialized type. ### Positive Consequences * There will be less overhead in our generated code - both performance overhead and cognitive overload - it will be much easier to rationalize about how the generated code actually works and performs. ### Negative Consequences * Errors in our generated code might cause pointer misuse and lead to "use after free" type issues. * Misuse of generated APIs may be able to create reference cycles between Rust objects that cannot be deallocated, and consumers coming from a garbage-collected language may assume that such cycles will be collected. ## Pros and Cons of the Options ### [Option 1] We extend the `HandleMap` abstraction to track lifetimes and support easier codegen. * Good, because raw pointers aren't handed out anywhere. * Bad, because we need to reimplement safe reference counting or garbage collection. * Bad, because code generation is likely to remain somewhat complex. Overall, this option is dispreferred because it will involve writing significant new and complex code, the safety benefits of which will be quite marginal in practice. ### [Option 2] We replace `HandleMap` with raw pointers to Rust's builtin `Arc` * Good, because the code generated by UniFFI is clearer and easier to understand. * Good, because we can reuse the Rust standard library and have confidence in its implementation. * Bad, because handing raw pointers around means bugs in the generated code or intentional misuse of the bindings might cause vulnerabilities. Overall, this option is preferred because it achieves the goals while reducing both the performance overheads of the generated code, and the cognitive overheads of maintaining the tool. ### [Option 3] We replace `HandleMap` with raw pointers to a special-purpose reference container * Good, because the code generated by UniFFI is clearer and easier to understand. * Good, because we can reuse an existing well-tested container type like `triomphe:Arc`. * Good, because the special-purpose container type may be more performant than the default implementation in the Rust stdlib. * Bad, because handing raw pointers around means bugs in the generated code or intentional misuse of the bindings might cause vulnerabilities. * Bad, because the special-purpose container may "leak" into the implementation of the UniFFI-wrapped Rust code, adding cognitive overhead for consumers. Overall, this option is dispreferred due to additional cognitive overhead for consumers. The potential performance improvements seem likely to be lost in amongst the many other sources of overhead in our current generated code. We may reconsider this decision if future profiling shows the use of stdlib `Arc` to be a bottleneck. ## Links * Thom discusses this a bit in [this issue](https://github.com/mozilla/uniffi-rs/issues/244) and agrees with the assertion that raw pointer make sense when all the relevant code is generated. * Ryan discusses his general approval for this approach in [this issue](https://github.com/mozilla/uniffi-rs/issues/419) and the [PR for this ADR](https://github.com/mozilla/uniffi-rs/pull/430) --- ## File: docs/adr/0006-wrapping-types.md # Update the code or update the template wrappers? * Status: proposed * Deciders: bendk, rfkelly, mhammond * Consulted: jhugman, jan-erik, travis * Date: 2021-08-06 Discussion : [PR 1001](https://github.com/mozilla/uniffi-rs/pull/1001). ## Context and Problem Statement UniFFI was not able to support types from external crates because Rust's orphan rule prevents implementing the `ViaFfi` trait. In order to add support we needed to choose between updating the `uniffi` traits or updating the `lift_py` and `lower_py` scaffolding functions. The same general question comes up often. When adding new features we often need to choose between two paths: * Updating the code in the target language * Updating the template wrapping code This ADR discusses this particular decision and also the general pros and cons of each path. ## Decision Drivers * We wanted to support external crates that define new types by wrapping an UniFFI primitive type. For example supporting `serde_json::Value` that wraps `string` or a `Handle` that wraps `int32`. We wanted this kind of wrapping code to exist outside of `uniffi` to allow for more experimentation with wrapped types and to support types that were specific to particular libraries (for example the application-services `Guid` type). ## Considered Options * **[Option 1] Extend the template code to wrap the type** * In the Record/Enum/Object/Error code, we would use the newtype pattern to wrap the external type (`struct WrapperType(ExternalType)`) * In the filters functions we generate code to wrap lift/lower/read/write. For example the lower_rs filter could output `WrapperType(x).lower()` to lower `WrapperType`. * **[Option 2] Update the `uniffi` code and generalize the `ViaFfi` trait** * We define `FfiConverter` which works almost the same as `ViaFfi` except instead of always converting between `Self` and `FfiType`, we define a second associated type `RustType`. `FfiConverter` converts between any `RustType` and `FfiType`. * For each user-defined type (Record, Error, Object, and Enum), we create a new unit struct and set `RustType` to the type. This handles external types without issues since we're implementing `FfiConverter` on our on struct. The orphan rule doesn't apply to associated types. * This eliminated the `lower_rs`, `lift_rs`, `read_rs`, and `write_rs` filter functions. All FFI conversions were now handled by Rust code directly. ## Decision Outcome Chosen option: * **[Option 2] Update the `uniffi` code and generalize the `ViaFfi` trait** This decision is taken because: * It was relatively easy to implement wrapper types by allowing the external crates to add custom scaffolding code. This code could wrap primitive types because all lifting/lowering/reading/writing was handled by Rust code. If we had gone with option 1, then the wrapping code would need to hook into the template functions (`lift_rs`, `lower_rs`, etc.). We couldn't see a simple way to implement this. * Updating the code in the target language results in more readable generated code. The newtype pattern makes the generated code more difficult to read, especially when types are wrapped in `Option<>`, `Vec<>`, etc. * The same pattern could be used to implement wrapping on the bindings side. ### Positive Consequences * Paved the way for wrapper types. * Simplified the template code. ### Negative Consequences * Implementing wrapping with template functions can lead to more direct code. For example, lifting a integer value is a no-op, but we still generate a function call to do it. This is not an issue with Rust, since the compiler will optimize the call away, but it could be an issue for the bindings code. If we decide that it is an issue, we could go with a hybrid solution: generate the lifting/lowering code in the target language, but also have lift/lower filter functions that exist solely to optimize lifting/lowering simple types. --- ## File: docs/adr/0007-enable-implementing-bindings-separately.md # Enable implementing bindings in separate crates. * Status: Accepted * Deciders: teshaq, bdk, mhammond * Date: 2022-04-7 Technical Story: [Issue 299](https://github.com/mozilla/uniffi-rs/issues/299) Implementation: [PR 1201](https://github.com/mozilla/uniffi-rs/pull/1201) Testing Implementation: [PR 1206](https://github.com/mozilla/uniffi-rs/pull/1206) ## Context and Problem Statement All the binding generators currently live in the [`uniffi_bindgen`](../../uniffi_bindgen/src/bindings) crate. This creates the following difficulties: - All the bindings live in the `uniffi` repository, so the `uniffi` team has to maintain them (or at the very least review changes to them). This makes it difficult to support third-party developers writing bindings for languages the core team does not wish to maintain. - Any change to a specific binding generator requires a new `uniffi_bindgen` release for it to be accessible by consumers. Even if it doesn't impact any of the other bindings. - Some bindings require complex build systems to test. Including those build systems in `uniffi` would require developers to install those build systems, and CI to do the same. For example, any type of `gecko-js` bindings would require the mozilla-central build system to build and test. - We currently run all the tests for the bindings in our CI and through `cargo test`. This means that if one binding target gets outdated and fails, or if a developer doesn't have the needed libraries installed for one of the targets, the tests would fail. Before [PR 1201](https://github.com/mozilla/uniffi-rs/pull/1201), it was also impossible to write new bindings that did not live in the [`uniffi_bindgen`](../../uniffi_bindgen/src/bindings) crate. This ADR proposes enabling third-party crates to implement binding generators, and describes the necessary uniffi changes to enable this. ## Decision Drivers * Support Firefox Desktop JavaScript binding generation * Testability, it should be easy for developers to test the bindings they care about. Without having to navigate and install unfamiliar libraries and build systems. * Developer experience, it should be easier to write and maintain a new binding generator than it currently is. * Releases, cutting releases for changes in one binding generator shouldn't harm another. **NOTE**: Version compatibility is handled in a [separate ADR](https://github.com/mozilla/uniffi-rs/pull/1203) ## Considered Options * **[Option 1] Do nothing** This means keeping everything as-is, and deciding that all binding generators (at least for now) should live under the `uniffi_bindgen` crate. * **[Option 2] Create a public API for external crates to implement their own bindings.** Developers would have traits exposed they can leverage to implement binding generators that do not live in `uniffi_bindgen`. `uniffi_bindgen` would still handle generic tasks related to binding generation. ## Pros and Cons of the Options ### **[Option 1] Do nothing** * Good, because it makes it harder for users to accidentally use different versions of `uniffi` for scaffolding and bindings since they are all implemented together in the same crate. * Good, it makes it easier to make changes to multiple bindings at a time (in the case of a breaking change in `uniffi`, etc). * Bad, because maintainability can grow to be difficult - especially if more bindings are added which the core `uniffi` team is not familiar with. * Bad, because testability can also grow to be difficult - as more bindings are added the requirement to test all the bindings together in one repository is difficult to maintain. * Bad, because releases of all the binding generators are tied to the release of `uniffi_bindgen`. ### **[Option 2] Create a public API for external crates to implement their own bindings.** * Good, because ownership will be clear, and members of the community can opt to maintain their own binding generators. * Good, because our CI would only need to test the core bindings we maintain, and others can be tested by their own maintainers (for example, a `gecko-js` binding generator should be tested in `mozilla-central` and not here). * Good, because a release in external bindings wouldn't have an impact on any internal ones unless it changes internal `uniffi` behavior. * Bad, because it's easier to accidentally have a version mismatch. (see [this ADR](https://github.com/mozilla/uniffi-rs/pull/1203)) * Bad, because testability increases in complexity. We are required to publish fixtures and examples we have. (see [PR 1206](https://github.com/mozilla/uniffi-rs/pull/1206)) Overall this option is preferred because: - It's a requirement to implement bindings for gecko-js, which can't be tested end-to-end without a complex build system. - It creates the possibility of community contributors writing and maintaining their own binding generators in their own repositories. - The increased risk of version mismatch can be dealt with. (see [this ADR](https://github.com/mozilla/uniffi-rs/pull/1203)) ## Decision Outcome Chosen option: ### **[Option 2] Create a public API for external crates to implement their own bindings.** ## Changes ### Expose a trait `BindingGenerator` The trait would have the following: 1. An associated type that implements `BindingGeneratorConfig` - `BindingGeneratorConfig` would be another trait, that binding generators can implement on their own configuration types. The purpose of this type is to carry any binding specific configuration parsed from the `uniffi.toml` 1. A function `write_bindings` that takes in the ComponentInterface and Config and writes the bindings into directory `out_dir` ### Expose a generic function as entry point 1. The binding generator should call a generic function when generating bindings exposed by `uniffi_bindgen`. The generic function will: - Parse the UDL. - Parse the configuration from `uniffi.toml`, using the `BindingGeneratorConfig` trait the consumer implements. - Initialize a `BindingGenerator`, with the type a consumer provides. - Call `write_bindings` on the generic type. See [PR 1201](https://github.com/mozilla/uniffi-rs/pull/1201) for implementation of the above changes. ### Expose fixtures for testing To enable external binding generators to implement tests, we would publish our fixtures and a new `uniffi_testing` crate that is a helper for consumers to build and consume the fixture crates. See [PR 1206](https://github.com/mozilla/uniffi-rs/pull/1206) for implementation of the testing changes.