Erlang ARM32 JIT诞生

Erlang ARM32 JIT诞生
Erlang ARM32 JIT is born

原始链接: https://www.grisp.org/blog/posts/2025-10-07-jit-arm32.3

## BEAM JIT 在 ARM32 上的首次里程碑在 Erlang 生态系统基金会的支持下，BEAM JIT 编译器移植到 ARM32 架构的项目已取得重要里程碑：成功通过 JIT 编译的 ARM32 机器码执行 Erlang 函数。团队使用 QEMU 模拟 ARM32 环境，并运行了一个修改后的 BEAM VM，其中包含一个简单的 `hello.erl` 模块，设计为以退出代码 42 停止。最初的成功包括初始化 JIT 编译器（利用 AsmJit），发出共享代码片段，以及编译 `erts_beamasm` 模块——这对于进程执行至关重要。团队专注于到达 `halt` 内置函数所需的最小代码路径，并承认许多指令尚未实现（在汇编中标记为“NYI”）。重要的是，程序以预期的退出代码 42 终止，确认 JIT 编译器已成功生成并执行了 Erlang 代码的 ARM32 指令。团队发现值 42 在 ARM32 寄存器中表示为一个带标签的整数，这展示了 BEAM 的内部数据表示。下一步是扩展 JIT 实现以支持更广泛的 Erlang 指令，从而实现增量开发和测试。所有代码都可在 [GitHub](https://github.com/stritzinger/otp/tree/arm32-jit) 上获取。

## Erlang ARM32 即时编译 (JIT) 开发一个针对ARM32架构的Erlang即时编译 (JIT) 编译器已经创建，引发了Hacker News上的讨论。虽然有些人质疑其相关性，考虑到向64位ARM和更快速语言的转变，但另一些人强调了32位ARM在嵌入式系统和物联网设备中的持续重要性。开发者的动机源于利用现有的aarch32硬件。Erlang的适用性得益于像Nerves这样的项目，它促进了构建精简的嵌入式Erlang固件，以及Erlang在电信领域可靠分布式计算方面的历史渊源。评论者指出，ARM32并未停产——它仍然在许多ARM实现中得到支持，包括预计将在嵌入式应用中由于内存限制而使用数十年的Cortex-M芯片。虽然为ESP32（现在也包括RISC-V）开发JIT编译器意义重大，但该项目专门关注在现有ARM32硬件上启用Erlang。

原文

A blog series recounting our adventures in the quest to port the BEAM JIT to the ARM32-bit architecture.

This work is made possible thanks to funding from the Erlang Ecosystem Foundation and the ongoing support of its Embedded Working Group.

This week we finally achieved our first milestone in developing the ARM32 JIT. We executed our first Erlang function through JITted ARM32 machine code!

The BEAM successfully runs and terminates with error code 42! That 42 comes from an Erlang function, just-in-time compiled by our ARM32 JIT!

Announcement is done! All code is available at https://github.com/stritzinger/otp/tree/arm32-jit

Keep reading for a lot of interesting details!

The first piece of Erlang code

This is hello.erl that contains a start/2 function. The function head mimics the erl_init:start/2 function, which is the entry point of the first Erlang process. We replaced erl_init:start/2 with hello:start/2 in the erl_init.c module of the BEAM VM. This way, we forced the runtime to execute this Erlang function.

hello:start/2 is very simple as it just calls the erlang:halt/2. This function is a BIF (Built-in Function) that executes C code, part of the BEAM VM. This code executes an ordered shutdown of the BEAM and allows us to customize the error code, in this case: 42.

(Why {flush, false}? At the time I am writing this, letting it be true causes a segmentation fault EHEH)

Obviously, we need to compile this Erlang module, but I will also generate the BEAM assembly so we can have a look at what we will have to deal with.

You can spot the start function and the two standard module_info functions that all Erlang modules have. We do not care much about those right now as we discovered that they are not executed and are not required to work, for now.

We can see that the core of the start function is just two move operations and one call_ext_only. But bear in mind that the BEAM loader will transmute these Generic BEAM Operations into Specific operations. More complexity will pop up!

Execution

We are using qemu-arm to emulate Arm32 and we are directly using beam.smp to run the BEAM.

JIT initialization

At boot, the BEAM initializes the JIT if enabled. The JIT leverages the AsmJit library to emit all machine code instructions.

Emission of all global shared fragments

There are 90+ code snippets that are shared among all modules. The JIT loads them one single time and sets up jumps to them in every other module. It is like a global library for all modules.

We skipped most of these because just the shared fragments involved in the hello:start/2 execution were needed.

Emission of the erts_beamasm module

As part of the JIT initialization, erts_beamasm is emitted. This module is an internal hardcoded module that exists only when BEAM is using the JIT. It holds 7 fundamental instructions used to manage the Erlang process executions.

run_process - The main process execution entry point
normal_exit - Normal process termination
continue_exit - Continue after exit handling
exception_trace - Exception tracing functionality
return_trace - Return value tracing
return_to_trace - Return to tracing state
call_trace_return - Call tracing return handling

Preloaded modules

The hello.erl module has been compiled and put as first and single Erlang module in the list of preloaded modules. Preloaded modules are Erlang fundamental modules that are always loaded by the BEAM before the first Erlang process can start. They implement, in Erlang, the core features of the Erlang Runtime System (ERTS). The OTP build scripts group all ebin files into a single C header that is then linked into the executable. This makes the Erlang binaries available as a static C array in the BEAM source code. These are then loaded one by one after the BEAM VM is initialized.

Cool, let's nuke all these modules and leave just our hello.erl. It does not need many BEAM instructions and we can easily verify that it executes. To do the substitution we just need to change this build variable in otp/erts/emulator/Makefile.in

We are running BEAMASM with -JDdump true so asmjit will dump all ARM32 assembly for each module! This is incredibly useful if monitored while executing with a debugger, as we can see the assembler being printed line by line by our code.

Bear in mind, this assembler is not what hello should look like. We are missing a lot of things.

You can spot many sequences like:

This is a call to nyi (Not Yet Implemented) function and the argument loaded to R0 is the pointer to a string that contains the name of the BEAM instruction that should have been emitted instead. You can spot many of these since we are only emitting the code to reach halt. Everything after that is not important now as halt will never return!

There are many more comments we could make around all the details in this assembler dump, but let's move on.

Jumping into Jitted code!

Later in the BEAM initialization the first Erlang process will be allocated and started.

We swap the module and function with hello in erts/emulator/beam/erl_init.c

One BEAM scheduler thread will jump to the process_main function. You can find it here in the source code. This is emitted by our JIT and is the first emitted code that will run.

Here we need to handle the Erlang processes scheduling by calling BEAM routines that implement the algorithms of Erlang concurrency, like erts_schedule.

erts_schedule will return the pointer to the Process C structure that holds all information about the process that is going to execute. We then load all necessary data inside registers and then we branch to the exact point where the program execution stopped.

The first Erlang function call

In this case we are calling hello:start/2 so the first instruction to execute is apply_only that does a few things but ends up calling the C apply routine.

The routine processes the Module-Function-Arity information to get the address where the function code resides in memory.

What follows is the Erlang function prologue. You can see it in the assembler code section above. For example, all functions have these instructions in their prologue:

i_breakpoint_trampoline: handle breakpoints for the debugger app
i_test_yield: checks if the function should yield and go back to the scheduler

We have minimal or partial implementations of these since we do not really need them. We have to emit them though, as the C++ generated loader functions from the BEAM are expanding the Erlang function call Operation into a more specific and complex function prologue sequence.

After that, we added support for the call_light_bif operation that precedes the call to the halt_2 BIF routine. This implementation is also minimal.

Question for later: did you notice that we put a 42 as a number in the code? Numeric constants are printed as decimals in the dump, but we cannot spot any 42!?

After the call, we see two other operations:

These are just calls to NYI as we will never reach this code! So for now, we can skip them...

Let's roll the JIT!

Impressive, the program returns immediately without even saying "Hi" ... and without Segmentation Fault!!

But let's check the program return code!

~/arm32-jit$ echo $?
42

We can safely say that number is not there by accident! This is a great achievement as from now on we will be able to incrementally add Erlang instructions.

Every Erlang line we add will trigger new Opcodes. By emitting them and running the code we will have immediate feedback on everything.

The next goal now is to complete the hello module to host all possible beam instructions!

Hey where is 42???

One interesting thing I spotted looking at the assembly: You cannot find the number 42 in there. Or actually, you can, it is just hidden in plain sight. To understand you need to know how we are using ARM32 registers.

In particular the register r4, a callee-saved register. We are using it to store the pointer to the ErtsSchedulerRegisters struct. The ErtsSchedulerRegisters contains the X register array. When a function is called, X registers are used to store the arguments of the call.

This becomes more obvious if we compare the Erlang assembly to the Arm32 assembly.

42 is stored at r4+64.

r4: pointer to the ErtsSchedulerRegisters struct
64: base offset from the beginning of the struct to the beginning of the x_reg_array

The list is stored at r4+68.

68: is the base offset + the size of one Eterm (4 bytes on ARM32)

But why in assembly do we see 687 and not 42?

Converting both numbers to hex we get:

Yep, this is an example of a Tagged Value. If we consult the BEAM book we can learn about the Tagging Scheme:

00 11 Pid
01 11 Port
10 11 Immediate 2
11 11 Small integer

42 is tagged with 1111 at the low end. So the BEAM can quickly recognize during a pattern match that this Erlang Term is a Small Integer!