0x646b - xv6

RISC-V Startup Code

2026-06-26T00:00:00+00:00

Motivation</h2>
I work with different microcontroller boards day to day. My job is to write startup code in Rust for board bring-up, especially RISC-V cores and to write, generate, and maintain the Peripheral Access Crate (PAC) for each RISC-V SoC.</p>
Working this low in the stack, I wanted to write down what actually makes a RISC-V core come alive and run our code. Concretely:</p>

What is already true when the board powers on or resets?</li>
What does the hardware hand us for free, and what must we set up ourselves?</li>
How do we make the hardware do what we want, purely from software?</li>
What has to be in place before we can call our first Rust function?</li> </ul>
I’ll answer these using xv6</a>, a small teaching OS that targets a 64-bit RISC-V core and runs on QEMU. Its boot path is the same bare-metal bring-up problem I deal with every day - set up a stack, place code and data, pick a privilege level, jump to our code - just unusually well documented, which makes it a great thing to learn from.</p>
This post follows only the path from `power-on</code> to the main Rust function (main</code>); everything the kernel does after that is for a later post.</p>`
`I’m porting xv6 from C to Rust to learn real OS concepts by building one, reading the` `C source</a> alongside the xv6 book</a>. You can find my in-progress</strong> port on GitHub: xv6rs</a>. Every code snippet in this post is taken from that repo.</p>`
RISC-V EEI</h2> EEI stands for Execution Environment Interface</strong>, defined by the RISC-V Unprivileged spec.</p> A RISC-V core is defined to have its own Instruction Fetch Unit and each core might support multiple RISC-V-compatible hardware threads, or harts, through multithreading.</li> RISC-V defines up to three privilege levels</strong>: Machine (M), Supervisor (S), and User (U), most to least privileged. Only M is mandatory; S and U are optional. The xv6 target implements all three.</li> An EEI is the contract a layer offers to the software above it: it pins down the initial state, the harts, the memory and I/O regions, instruction behavior, and how traps and ecall</code> are handled</strong> - to have defined behavior for the running software.</li> Crucially, the EEI is relative</strong>: it depends on which level you run at. For M-mode software the EEI is the hardware platform itself (which begins at power-on reset); for S-mode software, M-mode sets up the EEI; and so on up the stack.</li> </ul> See the RISC-V Unprivileged spec</a> for the full definition.</p> </blockquote> Program’s Entry Point</h2> xv6 operating system runs on RISC-V multiprocessor under QEMU. So QEMU sets up EEI for us, such as:</p> Loads kernel at address 0x80000000</code>, where DRAM is present</li> Initial values of DRAM will be 0</code>.</li> Starting privilege level will be set to M</code>.</li> Hart numbers are incremental and start from 0</code>.</li> </ul> With the combination of linker script and assembly, we will set our program to start from address 0x80000000</code>. In theory we could have done this using Rust instead of assembly, but compiled Rust code, like C, assumes some things are already set up when any function is called like a valid stack pointer, a zeroed .bss</code> and an initialized .data</code> section. Here we are starting from garbage state w.r.t. QEMU, we will use assembly to set this up so that we could use Rust functions later.</p> As DRAM is filled with 0’s, we need not clear the .bss</code> section and we need not move contents from Flash to RAM for the .data</code> section as it will already be in DRAM. So the only thing missing is setting a valid stack pointer.</p> entry.rs</code></strong>:</p> use </span>core::arch::global_asm; </span> </span>global_asm!( </span> " </span> .section .text.entry </span> .global _entry </span> _entry: </span> la sp, stack0 </span> csrr a1, mhartid </span> addi a1, a1, 1 </span> slli a0, a1, 12 # `li a0, 4096`; `mul a0, a0, a1`; </span> add sp, sp, a0 </span> call start </span> </span>", </span>); </span></code></pre> kernel.x</code></strong>:</p> OUTPUT_ARCH</span>( </span>"riscv"</span> ) </span>ENTRY</span>( </span>_entry</span> ) </span> </span>SECTIONS </span>{ </span> </span>.</span> = </span>0x80000000</span>; </span> </span> </span>.text</span> : { </span> </span>KEEP</span>((</span>.text.entry</span>)) </span> (</span>.text .text.</span>) </span> } </span> </span> </span>.bss</span> : { </span> </span>.</span> = </span>ALIGN</span>(</span>16</span>); </span> (</span>.bss.stack0</span>) </span> } </span>} </span></code></pre> start.rs</code></strong>:</p> #[</span>unsafe</span>(no_mangle)] </span>#[</span>unsafe</span>(link_section = "</span>.bss.stack0</span>")] </span>static mut</span> stack0: [</span>u8</span>; </span>4096 </span>* </span>NCPU</span>] = [</span>0</span>; </span>4096 </span>* </span>NCPU</span>]; </span>// NCPU: No of CPU's </span></code></pre> In kernel.x</code>, we are setting start address to 0x80000000</code> and also we are keeping all symbols from .text.entry</code> section at this address. As we only have _entry</code> symbol in this section (see entry.rs</code>), the first instruction of our program is la sp, stack0</code>.</p> stack0</code> is declared in start.rs</code> file, which is placed in .bss</code> section. It should have 16-byte alignment according to RISC-V spec which is enforced by linker script.</li> Each hart will get 4 KiB of stack</code>, we are using mhartid</code> register (remember QEMU should set this as per EEI’s contract) to get the hart id.</li> Once the stack is set up, we call start</code>. The C version follows this with a spin: j spin</code> loop as a safety net; I leave it out because our start</code> is marked -> !</code> and never returns.</li> C version uses mul</code> instruction instead of shift instruction slli</code> that we are using to calculate sp = stack0 + ((hartid + 1) * 4096)</code>, the reason for this change on the Rust side is a known limitation in how target features reach the assembler for hand-written asm. If we use mul a0, a0, a1</code>, the assembler will throw the following error:</li> </ul> error: mul instruction requires the following: ‘Zmmul’ (Integer Multiplication)</p> </blockquote> Even though we are targeting riscv64gc-unknown-none-elf</code></strong> which has multiplication support, that feature isn’t automatically applied to the assembler for hand-written global_asm!</code>, so this will not compile. We could solve this issue by using .option arch, +m</code>, but I just showed you how we could avoid mul</code> altogether by just using a base instruction (i.e., shift). You can find more details about this here</a> and here</a>.</p> S-Mode Kernel</h2> xv6 kernel runs in S-mode whereas user programs run in U-mode. Now that we are in M-mode by default, we will use a function named start</code> which we are calling at the end of assembly code to set up relevant things in M-mode and then we transition to S-mode.</p> The following things have to be set in xv6 during M-S mode transition:</p> Set Previous privilege mode to S</code>, so that when we call mret</code>, it transitions to S-mode.</li> Set Exception Program counter to our main</code> function.</li> Disable paging (satp = 0</code>, Bare mode) so that S-mode starts out using physical addresses. Paging applies to S-mode too once enabled - the kernel will run on virtual addresses through a page table we set up in a later post. (Unrestricted access to physical</em> memory comes separately, from PMP)</li> Delegate all interrupts and exceptions to S-mode, so that traps will be handled by S-mode handlers instead of M-mode.</li> Configure PMP (Physical Memory Protection) to give S-mode access to all of physical memory, so that kernel has unrestricted access to all I/O and peripherals.</li> Enable/Configure timer interrupts for scheduling purposes.</li> Keep each hart’s id in tp</code> register, so that we can tell which hart we’re running on to index per-CPU data).</li> </ul> start.rs</code></strong>:</p> #[</span>unsafe</span>(no_mangle)] </span>pub extern </span>"</span>C</span>" </span>fn </span>start</span>() -> ! { </span> </span>let mut</span> x: </span>u64 </span>= </span>r_mstatus</span>(); </span> x &= !</span>MSTATUS_MPP_MASK</span>; </span> x |= </span>MSTATUS_MPP_S</span>; </span> </span>w_mstatus</span>(x); </span> </span> </span>w_mepc</span>(main as </span>const </span>() as </span>u64</span>); </span> </span> </span>w_satp</span>(</span>0</span>); </span> </span> </span>w_medeleg</span>(</span>0xffff</span>); </span> </span>w_mideleg</span>(</span>0xffff</span>); </span> </span>w_sie</span>(</span>r_sie</span>() | </span>SIE_SEIE </span>| </span>SIE_STIE</span>); </span> </span> </span>w_pmpaddr0</span>(</span>0x3fffffffffffff</span>); </span> </span>w_pmpcfg0</span>(</span>0xf</span>); </span> </span> </span>timerinit</span>(); </span> </span> </span>let</span> id: </span>u64 </span>= </span>r_mhartid</span>(); </span> </span>w_tp</span>(id); </span> </span> </span>unsafe </span>{ </span> asm!("</span>mret</span>", </span>options</span>(noreturn)); </span> }; </span>} </span> </span>// ask each hart to generate timer interrupts. </span>fn </span>timerinit</span>() { </span> </span>w_mie</span>(</span>r_mie</span>() | </span>MIE_STIE</span>); </span> </span>w_menvcfg</span>(</span>r_menvcfg</span>() | (</span>1 </span><< </span>63</span>)); </span> </span>w_mcounteren</span>(</span>r_mcounteren</span>() | </span>2</span>); </span> </span>w_stimecmp</span>(</span>r_time</span>() + </span>1000000</span>); </span>} </span></code></pre> main function</h2> Once mret</code> instruction at the end of start</code> function is executed, we will fall into S-mode and jump to main</code> function because mepc</code> is set to main</code> and MPP</code> is set to S</code>.</p> main.rs</code></strong>:</p> #[</span>unsafe</span>(no_mangle)] </span>pub extern </span>"</span>C</span>" </span>fn </span>main</span>() -> ! { </span> </span>loop </span>{ </span> </span>unsafe </span>{ core::arch::asm!("</span>wfi</span>") } </span> } </span>} </span></code></pre> For now main</code> just parks each hart in a low-power wfi</code> loop; we don’t handle traps yet, so it just loops. In further posts I will explain how we could add UART</code> to this and print some helpful messages from each core.</p> Running on QEMU</h2> Here comes the juicy part i.e., actually running our kernel and verifying that our kernel boots and runs main</code> function from all 3 cores.</p> Follow the below steps:</p> Checkout my repo at specified commit hash</li> </ol> ❯</span> git clone https://github.com/Karthik-d-k/xv6rs.git </span> </span>❯</span> cd xv6rs/ </span> </span>❯</span> git checkout 99b67fe8db4a7d50033a612a0902c9444caa32c3 </span></code></pre> In terminal 1, Run QEMU</li> </ol> ❯</span> just qemu-gdb </span></code></pre> In terminal 2, Run GDB</li> </ol> ❯</span> just gdb </span></code></pre> Verify our kernel in GDB terminal</li> </ol> (</span>gdb</span>) continue </span>Continuing. </span>^C </span># Press Ctrl+C to interrupt the program and then inspect </span>Thread</span> 3 received signal SIGINT, Interrupt. </span>[Switching</span> to Thread 1.3] </span>0x0000000080000020</span> in kernel::main () </span>at</span> kernel/src/main.rs:23 </span>23</span> unsafe { core::arch::asm!("</span>wfi</span>") } </span>(</span>gdb</span>) info threads </span> </span>Id</span> Target Id Frame </span> </span>1</span> Thread 1.1 (CPU#0 </span>[</span>running</span>]</span>) </span>0x0000000080000020</span> in kernel::main () </span> </span>at</span> kernel/src/main.rs:23 </span> </span>2</span> Thread 1.2 (CPU#1 </span>[</span>running</span>]</span>) </span>0x0000000080000020</span> in kernel::main () </span> </span>at</span> kernel/src/main.rs:23 </span></span> 3 Thread 1.3 (CPU#2 </span>[</span>running</span>]</span>) </span>0x0000000080000020</span> in kernel::main () </span> </span>at</span> kernel/src/main.rs:23 </span>(</span>gdb</span>) thread 1 </span>[Switching</span> to thread 1 (Thread 1.1)</span>] </span>#0 0x0000000080000020 in kernel::main () at kernel/src/main.rs:23 </span>23</span> unsafe { core::arch::asm!("</span>wfi</span>") } </span>(</span>gdb</span>) info reg pc tp </span>pc</span> 0x80000020 0x80000020 <kernel::main+4> </span>tp</span> 0x0 0x0 </span>(</span>gdb</span>) thread 2 </span>[Switching</span> to thread 2 (Thread 1.2)</span>] </span>#0 0x0000000080000020 in kernel::main () at kernel/src/main.rs:23 </span>23</span> unsafe { core::arch::asm!("</span>wfi</span>") } </span>(</span>gdb</span>) info reg pc tp </span>pc</span> 0x80000020 0x80000020 <kernel::main+4> </span>tp</span> 0x1 0x1 </span>(</span>gdb</span>) thread 3 </span>[Switching</span> to thread 3 (Thread 1.3)</span>] </span>#0 0x0000000080000020 in kernel::main () at kernel/src/main.rs:23 </span>23</span> unsafe { core::arch::asm!("</span>wfi</span>") } </span>(</span>gdb</span>) info reg pc tp </span>pc</span> 0x80000020 0x80000020 <kernel::main+4> </span>tp</span> 0x2 0x2 </span></code></pre> Each core/thread/hart is looping at main</code> and its respective tp</code> registers are set to 0/1/2</code>.</p> </blockquote> References</h2> RISC-V Unprivileged Spec</a></li> RISC-V Privileged Spec</a></li> xv6 RISC-V Book</a></li> xv6 RISC-V Source code</a></li> My xv6rs repo</a></li> </ul>