Exploring Seamless Rust Interop for Newer Languages, Part 1

Languages like C++, Typescript, Kotlin, and Swift had a brilliant approach: they were created to harness an existing ecosystem of libraries from another pre-existing language.

But that's easier said than done! Especially for newer languages focusing on memory safety and speed. 0

Luckily, newer languages can tap into C's ecosystem by adding FFI 1 "bindings" (or "wrappers") to call into C, like the following Vale code:

extern func stdlib_launch_command(args []str, maybe_cwd str) i64;

...which calls into this C function:

int64_t stdlib_launch_command(stdlib_StrArray* chain, ValeStr* cwd_str) {
  ...
}

However, there are some downsides to these wrappers:

• It can be tricky to translate between C's and the newer language's concepts.
• They often fall out of date as the underlying C libraries evolve.
• It can be hard to find a good cross-platform C library for something.

This is, sadly, the state of affairs for a lot of new languages.

And alas, we dare not dream for more. 2 3

Some hope

Unless I'm mistaken, it might be possible to seamlessly call into Rust code, without writing bindings or wrappers or anything, similar to how C++, Typescript, Kotlin, and Swift all called into their predecessors.

If a new language could do this, it could tap into the over 100,000 crates on cargo. 4

This technique could even be used from C or C++, as our little proof-of-concept shows below. I use C++ a lot, so I'm personally pretty excited to call into Rust libraries more easily from C++. 5

Can't we already do that?

"But Evan," you say, "don't cbindgen and cxx already do this?"

Unfortunately, not quite. For example, if I want to use a third-party library like subprocess from my C code, I'll need to write the binding for cbindgen and cxx, and we're back at step one.

On top of that, they require us to use a limited whitelist of types that both sides understand; our C code can't receive a subprocess::Popen object.

It would be really nice if we could instead just call directly into Rust code.

Impossible challenges

Of course, calling directly into Rust code is widely believed to be impossible, because:

• Rust doesn't have a stable ABI; you can't link to a Rust library because rustc doesn't expose a predictable API for code to link to.
• Rust has generics. How would we expose a Vec<T> when Rust doesn't even know how big T is?
• How do we make Rust's mental model work with something like Vale that doesn't have a traditional borrow checker?
• How do we uphold Vale's other features, like determinism and perfect replayability if we can call directly into Rust?

These points are all true, but I think they're all solvable.

And if they are solvable, it paves the way to real "Rust++" style interop that lets us easily use a vast amount of existing Rust libraries. 6

A concrete example

Let's see an example of a language (C in this case) seamlessly calling into Rust, and then farther below we'll talk about how to make it work.

This approach is mainly designed with newer languages in mind, but we're exploring it with C for now. Later I'll talk about how we'll make it work with the Vale compiler.

We're going to make a C program that can use the Rust subprocess crate to launch a /bin/cat command to print out the contents of hello.txt.

If it were written in Rust, it would look like the below code (unidiomatically crafted to better match the C code below it).

use std::ffi::OsString;
use subprocess::Popen;
use subprocess::PopenConfig;

fn main() {
  let mut argv = Vec::new();
  argv.push(OsString::from("/bin/cat"));
  argv.push(OsString::from("hello.txt"));
  let create_result = Popen::create(&argv, PopenConfig::default());

  let mut process = 
    match create_result {
      Err(_) => panic!("Failed to open subprocess!"),
      Ok(process) => process
    };

  let wait_result = process.wait();
  if !wait_result.is_ok() {
    panic!("Failed to wait on subprocess!");
  }

  println!("Success!");
}

Below, we've rewritten main in C, and have it call into the Rust code. Some notes about it:

• All the noisy #pragma rsuse under the // Import methods comment shouldn't be needed for newer languages. I'll explain more about that later on.
• I've also included the Rust equivalent to each line, to better show the equivalences.
• Below the C code is an explanation of the basics.
• This is an actual working example!

That's a lot! Before we talk about how it works under the hood, here's an explanation of what's going on from the C programmer's perspective.

The #pragma rsuse OsString = std:ffi::OsString is setting up a name that our C code can use to refer to the Rust type.

C ignores unrecognized #pragmas like these, but our Makefile will grep for all #pragma rsuse lines and send them to our tool so it can generate rust_deps/rust_deps.h.

We #include "rust_deps/rust_deps.h" to bring in all those newly generated things, such as the VecOsString_new we later use.

In main, we call VecOsString_new, a generated wrapper that calls Rust's Vec<String>::new().

It returns an opaque type VecOsString. With a smart enough compiler we can access its fields directly, but C or C++ will have to use accessor functions.

We later hand a pointer to other functions, like VecOsString_push, VecOsString_as_slice, or VecOsString_drop.

Next, we call VR_StrFromCStr. This is a built-in function that takes a constant-data C string and returns an equivalent to Rust's &str. 7

The rest is pretty much the same.

Though, I do want to point out that since this is C, we need to manage our own memory: we need to call the right drop methods at the right times. Newer languages with RAII should be able to do this automatically, and wouldn't need all those calls to the drop methods.

C code also needs to be mindful of memory safety, and shouldn't move anything that has a live pointer pointing at it. More advanced languages will need to handle this carefully, and make their memory safety paradigms compatible with Rust's here.

I've got some plans for this in Vale. In grimoire-speak, the short-term plan is to do move-only programming with Rust objects, then shift to a certain regions/unique-references blend later on. 8

Okay, now that we know how to use it, let's talk about how it works under the hood!

The approach: passing opaque types by value

So what in the world is going on here?

Let's first talk about at that VecOsString type that our tool generated from Vec<OsString>.

A struct

It's an "opaque struct" of the right size: 24 bytes.

The important part is the [u8; 24] which is just a 24-byte array.

More details:

• We got the size (24) and alignment (8) from the original Vec<OsString>, we'll talk about that below too.
• The VALIGN(8) expands to __attribute__((aligned(n))) 9 to align it correctly. Thanks to Jeff Niu, snej, and kornel for noticing this was missing!
• The MaybeUninit avoids some undefined behavior, we'll talk about that below. Thanks to bjorn3 for this part!

So what goes in that 24 bytes? For the answer, let's look at the generated VecOsString_new() function.

A function returning some data by value

Here's the generated Rust code, paraphrased for clarity: 10

#[no_mangle]
pub extern "C" fn VecOsString_new() -> VecOsString {
  let result_rs: Vec<OsString> = Vec::new();
  let result_c: VecOsString = unsafe { mem::transmute(result_rs) };
  return result_c;
}

As you can see, we're first calling the underlying function, Vec::new(), and then we're casting it to the right type with transmute and returning it.

Previous versions memcpyd the Vec<OsString> into a new VecOsString's internal array, but transmuteing seemed simpler.

Also, this transmute is why our structs have MaybeUninit in them. bjorn3 offered a great explanation:

Thank you bjorn3!

To summarize everything so far, we're conceptually just casting things to opaque types and moving them into C. 11

A function taking arguments by value

We do the same thing for parameters as well. Here's the generated PopenResult_unwrap, which calls Result<Popen, Error>::unwrap:

#[no_mangle]
pub extern "C" fn PopenResult_unwrap(self_c: PopenResult) -> Popen {
  let self_rs: Result<Popen, PopenError> = unsafe { mem::transmute(self_c) };
  let result_rs: Popen = Result::unwrap(self_rs,);
  let result_c: Popen = unsafe { mem::transmute(result_rs) };
  return result_c;
}

Note we're taking the PopenResult argument by value, and transmuting it to the correct type. As you can see, we do the same fundamental approach for parameters and returns.

A function taking a pointer

Some functions take pointers, like PopenResult_is_ok, which calls Result<Popen, PopenError>::drop:

#[no_mangle]
pub extern "C" fn PopenResult_is_ok(
  self_c: *const PopenResult,
) -> bool {
  let self_rs: &Result<Popen, PopenError> = unsafe { mem::transmute(self_c) };
  let result_rs: bool = Result::is_ok(self_rs,);
  let result_c: bool = unsafe { mem::transmute(result_rs) };
  return result_c;
}

The actual generated code

The above examples were simplified. Here's the actual struct definition:

#[repr(C, align(8))]
pub struct VecOsString(std::mem::MaybeUninit<[u8; 24]>);
const_assert_eq!(
    std::mem::size_of::<alloc::vec::Vec<std::ffi::os_str::OsString>>(),
    24);

and the actual PopenResult_unwrap:

#[no_mangle]
pub extern "C" fn PopenResult_unwrap(
  self_c: PopenResult,
) -> Popen {
  const_assert_eq!(
      std::mem::size_of::<core::result::Result<subprocess::Popen, subprocess::PopenError>>(),
      std::mem::size_of::<PopenResult>());
  let self_rs: core::result::Result<subprocess::Popen, subprocess::PopenError> =
      unsafe { mem::transmute(self_c) };
  let result_rs: subprocess::Popen =
      core::result::Result::unwrap(self_rs,);
  const_assert_eq!(
      std::mem::size_of::<subprocess::Popen>(),
      std::mem::size_of::<Popen>());
  let result_c: Popen = unsafe { mem::transmute(result_rs) };
  return result_c;
}

Note that everything is fully qualified (e.g. core::Result::Result instead of Result), and the addition of the assertions which sanity check that the tool works properly.

The approach so far

If I had to summarize this all, I'd say we're sending things across the FFI boundary after casting them into opaque types, and then using generated functions to work with them.

The overall approach seems okay because all Rust types (except Pinned ones) can be moved around as long as they aren't currently borrowed.

Also, I haven't gotten any crashes from the handful of tests I've made: a test that uses subprocess to /bin/cat hello.txt, a test that uses a lifetime'd iterator to reverse a string, 12, a test that uses the regex crate to extract a string, a test that gets a string's length, 13, and a test that makes a directory.

Keep in mind, I can't say for certain that this works.

There's a good chance this transmuteing is causing some sort of undefined behavior, or perhaps even violating strict aliasing.

I'm also uncertain about when/whether we can pass things by value over the FFI boundary. Calling conventions, especially on Windows, have a lot of quirks. I hope that just putting extern "C" makes it okay, but I can't be sure yet.

I've written this article to get feedback on this mechanism, so let me know your thoughts in the comments on HN or Reddit!

The Tool

As you can see, the generated code was actually pretty simple: it just casts things to correctly-sized opaque types and sends them over to C.

This article is running long so I'll save the more detailed explanation for a follow-up article, but here's an overview of what it does:

First, parse the #pragma rsuse lines, such as the:

#pragma rsuse VecOsString = std::vec::Vec<OsString>

We need to parse std::vec::Vec<OsString> and resolve its actual type, which is alloc::vec::Vec<std::ffi::os_str::OsString>.

The tool comes out to 3,200 lines, and most of it is this step. Basically, we grab all the data from rustc, parse it, and then make an index of what owns what, taking into account imports and type aliases and function overloading. 14

Second, measure the sizes and alignments of all types. The tool makes a temporary Rust program that looks like this: 15

use std::mem;
extern crate alloc;
use core;
use core::ffi::c_char;

fn main() {
  println!("alloc::vec::Vec<std::ffi::os_str::OsString>={},{}", std::mem::size_of::<alloc::vec::Vec<std::ffi::os_str::OsString>>(), std::mem::align_of::<alloc::vec::Vec<std::ffi::os_str::OsString>>());
  println!("core::result::Result<subprocess::ExitStatus, subprocess::PopenError>={},{}", std::mem::size_of::<core::result::Result<subprocess::ExitStatus, subprocess::PopenError>>(), std::mem::align_of::<core::result::Result<subprocess::ExitStatus, subprocess::PopenError>>());
  println!("core::result::Result<subprocess::Popen, subprocess::PopenError>={},{}", std::mem::size_of::<core::result::Result<subprocess::Popen, subprocess::PopenError>>(), std::mem::align_of::<core::result::Result<subprocess::Popen, subprocess::PopenError>>());
  println!("std::ffi::os_str::OsString={},{}", std::mem::size_of::<std::ffi::os_str::OsString>(), std::mem::align_of::<std::ffi::os_str::OsString>());
  println!("[std::ffi::os_str::OsString]={},{}", std::mem::size_of::<&[std::ffi::os_str::OsString]>(), std::mem::align_of::<&[std::ffi::os_str::OsString]>());
  println!("str={},{}", std::mem::size_of::<&str>(), std::mem::align_of::<&str>());
  println!("subprocess::Popen={},{}", std::mem::size_of::<subprocess::Popen>(), std::mem::align_of::<subprocess::Popen>());
  println!("subprocess::PopenConfig={},{}", std::mem::size_of::<subprocess::PopenConfig>(), std::mem::align_of::<subprocess::PopenConfig>());
}

...and then runs it, and collects the output.

Third, actually generate the structs and functions. It's not too difficult once you have all the information from rustc and the sizer program.

Fourth, invoke cbindgen to generate a C header file and static library.

At that point, the tool's job is done! The Makefile then invokes clang with the right paths to the header file and the static library, and we get a program out of it!

Implications, Benefits, and Drawbacks

There are some nice benefits to this approach:

• Newer fast, memory-safe languages like Vale will have access to a lot of higher-quality fast and memory safe libraries.
• Newer scripting languages will be able to call into faster Rust languages to do the heavy lifting, similar to how Python calls into C.
• This could even speed up C++ to Rust migrations, as it'll be a little easier to call between the two.

The biggest drawback I've found is that the tool runs pretty slowly, since it invokes rustc twice (to get the sizes of types, then to do the final build). I'll be speeding it up, but it's hard to say by how much. 16

I thought at first there would be a large binary size penalty for bringing in the entire Rust standard library, but it's actually quite manageable with the right compiler options.

For example, the strlen.c test's binary size is 34,008 bytes, only 576 bytes bigger than the plain C binary size of 33,432 bytes. 17 18 Big thanks to bjorn3 for the tips here!

There's a few more things I want to explore too:

• I want to explore how link-time optimization works with these. I suspect it will do pretty well, especially since LLVM 16 switched to opaque pointers.
• I haven't decided how we'll handle async functions. Perhaps a built-in function could launch a thread with a given function and return us a future. Perhaps we could also give C some other helper functions to deal with those futures.
• Macros could be a problem. It's hard to generate a wrapper function for macros that have the correct types. Luckily, a lot of crates that lean on macros also offer a normal imperative API, so we'll see how much of a problem this is.

Hopefully all this works out! If it does, then I'll be pretty excited about where this is heading.

That's it for now!

Huge thanks to Jeff Niu, snej, kornel, and bjorn3 for fixes and advice!

Keep an eye out for the next articles, which will detail:

• How the tool parses and resolves the given Rust names.
• How we would integrate this into the Vale compiler.
• How we might do this and still uphold Vale's other features like determinism and perfect replayability.

If you have any questions, always feel free to reach out via email, twitter, the discord server, or the subreddit.

Donations and sponsorships are currently paused, but if you like these articles, please Donate to Kākāpō Recovery and let me know! I love those birds, let's save them!

Cheers,

- Evan Ovadia