ReFIX: a new FIX engine in the making
I’m building ReFIX, a new FIX engine in Rust, designed from day one for first-class Python bindings.
Despite its age, FIX continues to power much of the electronic trading world. Yet two ecosystems that I deeply care about are currently underserved when it comes to FIX.
One is Rust. I’ve always held the view that Rust is a uniquely strong fit for many areas within finance, in part due to its performance, but more importantly because its powerful type system is a godsend for modelling complex domains. Rust’s adoption in finance has been slow, which in large part comes down to gaps in the ecosystem. Some of these gaps aren’t specific to finance and are slowly getting filled. Other parts - like FIX - are very specific, and there seems to be little appetite to fill them, at least in the open-source world.
More surprisingly, the other one is Python. Python is used extensively in finance, but open-source options for FIX are outdated bindings for QuickFIX, libraries with limited scope (e.g. for message parsing), and immature pure Python engines. Now, I understand that Python isn’t the language of choice for high-frequency trading, but neither is FIX. And even outside actual trading, a lot of the supporting work around FIX gets done in Python.
I built HotFIX for my former employer, Validus Risk Management. HotFIX is a pure Rust engine with a narrow but complete scope: end-to-end, robust trading for a buy-side firm using FIX 4.4 with tag-value encoding. It does what it set out to do, but it’s not able to completely fill the ecosystem gap. And it wasn’t designed to, either.
The best way to write Python is to write Rust, but it takes some care to design crates that are PyO3 friendly. Luckily, with a little upfront planning and effort, a well-designed FIX engine in Rust can also fill the Python gap, and do so better than a pure Python library could (or bindings for C++, for that matter).
ReFIX is the FIX tooling I always wished existed. Correctness first, and no compromises on developer experience, performance or type safety. Not even in the Python layer.
Here are the key design decisions driving the new engine’s implementation.
Pure state machine in the session layer
Most FIX engines implement the session layer as a maze of conditional branches and scattered I/O actions, such as persisting sequence numbers, writing to logs, and sending outbound messages.
HotFIX went a step further with a clearer state machine, which made most invalid states unrepresentable through the type system. The handling of actions is offloaded to the states, which makes the conditional paths a lot easier to reason about. It aims to keep the places it performs I/O to a minimum, but I/O actions remain nested within the session logic.
Taking this direction a bit further gives an entirely pure session. Inbound messages, timer ticks, and application requests go in, and effects are returned for the caller to execute. This has two massive benefits. The obvious one is testability - the complex workflows and edge cases become simple unit tests without any need for setting up external services or mocking dependencies. The less obvious one, which matters more for an engine that supports Python, is that a pure session can leave the sync/async decision to the consumer. Defaulting to async Rust coupled to tokio is a sane choice for a Rust-only engine. PyO3, however, really doesn’t play nice with async Rust, and it’s better to either drive the session from Python’s asyncio or not use async at all.
So ReFIX’s session code stays pure. The one exception I’m still weighing is logging.
On performance
I have little interest in trying to squeeze every microsecond out of ReFIX, but most of the performance gains come from two easily addressable things in a FIX engine:
- persistence policy
- allocation-free parsing and encoding
The pure session layer hands full control over persistence to the caller. The caller can decide what persistence characteristics work best for their use case: fsyncing every message, batching the writes, or persisting nothing at all in a backtest. The session just returns its effects in the order they must be executed. This is a more flexible approach than the one many engines take, where an injected message store implements a fixed interface.
The QuickFIX lineage of engines parses each message into a heap-allocated map of tags to strings. This typically results in many allocations, making parsing slow. HotFIX took its main inspiration from QuickFIX and falls into the same category. I now think that an allocation-free design can be achieved without forfeiting developer experience or type safety, so ReFIX’s parser and encoder will be allocation-free from the start.
On type safety
A common way to let engine users customise message types for their dialect of FIX is to generate code from data dictionaries (typically XML).
This turns out to be extremely convenient for a Rust engine with Python bindings, as it sidesteps one of the great pain points of PyO3: type stubs for the Rust-exposed API. Automatic stub generation in PyO3 is in its infancy, and I believe it remains largely manual to this day. If the types are generated anyway, the Python code can be generated, too.
In fact, for Python users, only the Python code should be generated. Generating Rust would require an extra compilation step and create friction. With the right layering, they never have to. The raw message stays untyped and lives in pre-compiled Rust, unaffected by the dictionary. Code generation then happens per language - Rust users get typed messages at build time, Python users get typed accessors generated in pure Python. Both are thin layers over the same core.
The resulting engine is one where Python isn’t an afterthought, but a first-class citizen equal to Rust.
What’s next
I’m only just starting the development of ReFIX, and it will no doubt be a long journey.
With HotFIX, the objective was an engine working end-to-end as soon as possible. With ReFIX, I want to perfect the building blocks, going layer by layer.
The first piece will be the message layer with fully typed messages and codegen in both languages. This is a much smaller undertaking than the session layer and creates a lot of value by itself.
The session layer will follow immediately, built on the decisions above. You can follow the work at GitHub.