github.com/xlsynth/xlsynth release process

TL;DR: The xlsynth/xlsynth GitHub repository is a lightly-modified release fork of google/xls, mainly to provide timely, multi-platform C ABI library artifacts (like libxls.so and .dylib) for broader ecosystem use (e.g., in Rust). It tracks upstream closely, occasionally carrying patches for faster releases, and automates artifact delivery for dependent projects (like the xlsynth Rust crates). The process also benefits from increased visibility of Google's upstream pull requests (PRs), enabling more community feedback before changes land.

github.com/xlsynth/xlsynth is a GitHub repository that acts as a close-to-zero persistent changes "release fork" of the original google/xls repository.

"xlsynth" is used as the name for this repository as it's not as straightforward a fork as is typically made for contributions (it runs ahead a bit and has release branches and similar). 'xlsynth' is also arguably a bit easier to search for.

Impetus

There are a few things that are not in the natural objectives of the google/xls repository but are of interest for broadening the developer community and ecosystem reach — e.g., building macOS releases and .dylib artifacts. (As implied in the README material, the primary use for google/xls is a particular Linux distribution.)

Spot fixes and "temporarily running ahead" of google/xls upstream is sometimes necessary. Realistically, pull requests (PRs) can take time to land against the upstream repository (I set up tracking data at xlsynth/xls-pr-data to get an empirical understanding of this delay).

By contrast, there are times when I need a patch to be immediately available to the portion of the ecosystem that I help release. If all indicators suggest the patch is likely to be accepted upstream (from a functional perspective, even if minor edits are needed later), I prefer to land that change in xlsynth/xlsynth and make an immediate release.

Build artifacts

The key artifact we expose as the 'public API' of the XLS C++ code base is an extern "C" style C Application Binary Interface (ABI) library, libxls.so — https://github.com/xlsynth/xlsynth/blob/54d29b76e2f7e836f9d4bbffe99c86c272077b8c/xls/public/BUILD#L338 shows the BUILD target for this. The idea is that other languages (right now primarily Rust, though my understanding is that members of the community are developing Python bindings as well) can use this artifact.

In the github.com/xlsynth/xlsynth repository there is a branch we do releases from by convention — it is called cdleary/release-dylib-workflow — this branch contains a github action with a workflow that builds libxls artifacts for various platforms.

There are a few platform considerations:

Linux-based platforms: which versions of glibc and libstdc++ are assumed to be available from the distribution
OS X platforms: building x86-64 or arm64 artifacts, what the "minimum OS" requirements are, as these relate to toolchain assumptions

As a result of this, the current set of released artifacts includes:

libxls-ubuntu2004.so
libxls-ubuntu2204.so
libxls-rocky8.so
libxls-arm64.dylib
libxls-x64.dylib

Note that the upstream google/xls code base is developed assuming nearly top-of-tree C++ features (generally via Clang). The current guidance is to target C++20 per the Google C++ Style Guide.

However, while the C++20 specification was ratified some time ago, features are implemented in compiler and standard library implementations in a trailing fashion. Therefore, it's not possible to simply freeze on the earliest toolchain version offering some C++20 compatibility; newer features might require newer toolchains.

Because of this, the C++ compiler and standard library used to create these release artifacts must also be relatively modern. On platforms where the package-manager-given version of libstdc++ is modern enough to support the language features used we use that. For platforms where it is lacking capabilities in a way that causes breakage, we install the libc++ that corresponds to the Clang release being used. We currently do this for Ubuntu 20.04, for example: https://github.com/xlsynth/xlsynth/blob/cdleary/release-dylib-workflow/.github/workflows/build-and-release-dylib.yml#L361

Relationship to xlsynth-crate

As a new Rust crate release maintainer, I looked for inspiration on how existing crates shipped Rust code that wrapped native libraries exposing a C ABI. Ultimately, we need to know the location of the dynamic library so that we can offer up its path/name to the Rust binary linker so it can be appropriately loaded.

Originally I had done this via user-managed runtime linking facilities; i.e. making a little library shim that did dlopen/dlsym, but it turned out it was easier to collaborate with folks in other build environments by using standard dynamic linking/loading.

Some Rust crates that wrap native code require users to "get the native library on your system somehow and then point at it", often via environment variables.

However, I hoped for an easier out-of-the-box experience with fewer steps, similar to how native components often accompany Python packages via pip install. By analogy, JAX seamlessly downloads and leverages the equivalent of libxla.so.

To achieve a similar effect for xlsynth-crate, I implemented an artifact downloader in its build.rs script. This script downloads the pre-built Dynamic Shared Object (DSO) artifact corresponding to the target platform from the xlsynth/xlsynth release assets and provides its location to Cargo via a linker flag. This download behavior can be overridden or disabled via an environment variable if the user already has the DSO available locally.

One of the consequences of this setup is that for each xlsynth-crate release there is a corresponding DSO release it expects, which is noted in the build.rs source code.

xlsynth-crate is a DAG of crates

You can see the crates in the root directory of the repository: https://github.com/xlsynth/xlsynth-crate/tree/main

xlsynth-sys is the crate that "simply wraps the shared library" as that is the Cargo convention.
xlsynth is the crate that layers on top to provide the Rustic object model for the ideas/capabilities exposed by the DSO.
xlsynth-g8r is a gate-level facility that goes from optimized XLS IR to gate-level estimates.
xlsynth-driver is the crate that ties together the facilities in common case usage scenarios (sometimes called "flows" in hardware engineering parlance) into command line tools via subcommands.

We notably run many of our CI tests under Valgrind to ensure there wasn't anything mishandled in the ownership/lifetimes that are expected for the C API. The Rust crates also do not assume the same allocator is used by libxls.so and the Rust code, e.g., it can be common for the Rust binary to use mimalloc or similar, so all entities produced by libxls.so are assumed to be returned to the API's corresponding xls_free_* call.

Recap: the flow, in effect

google/xls repo makes progress "on its own"
we contribute to google/xls from branches on xlsynth/xlsynth through PRs
we directly mirror google/xls main to xlsynth/xlsynth main (i.e. mainline)
occasionally we'll rebase xlsynth/xlsynth cdleary/release-dylib on top of mainline
we try to do builds for a more exotic set of targets (e.g., including macOS) and if patches are necessary we make/carry patches at the tip of that release branch
a successful release results in published release artifacts
dependent projects like xlsynth-sys crate can then bump their revisions to grab the newer artifact releases and can depend on the newly released functionality

Note that, as of "recently", the Googler-contributing activity is also often visible as PRs, which is a huge help to the community in seeing what things are coming "down the pike" and being able to comment on them before they've landed. Previously we only had a "post hoc" ability to comment on things once they were already landed in the tree. Switching this to enable more community discussion before landing was a nontrivial lift for the Google team — I and other folks in the community are very appreciative for it!