I’m a bit of a sucker for low level programming. I’m not an expert at it … but I understand enough to be pretty dangerous. All low level programming is interesting.. but I’ve found 64-bit Windows programming to be one of the most interesting. It is also one of the most confusing for many people.

This is the start of a series on 64-bit Windows assembly for beginners. This is a topic many find hard to get into due to the lack of/hard to find information and some simple explanations. I hope this series can alleviate some of the issues for people.

Prerequisites

There is a little bit of assumed Assembly knowledge here. Ideally you’ll be familiar with the syntax and be able to read and follow along with small snippets at a time. I will try to keep them as short as possible to aid with this. You’ll understand what the stack is and how it works, and that the stack grows downward in memory (high addresses to low addresses).

I will be using NASM as the assembler and GoLink as the linker. If you feel more comfortable with another assembler or linker, you should still be able to follow along. However, you should note that the NASM syntax may not directly match that of your own assembler and the linker may have different switches.

Debugging will make use of Visual Studio. There is a free community edition now which has quite a lot of features.

The end goal

The end goal will be a shunting yard implementation and a reverse polish notation calculator. Ideally, we’ll have written a command line application that can accept a mathematical expression and spit out a result; entirely in assembly. Let’s get started.

Setting up the project

I like to keep the build and run steps nice and quick to save on typing. Start by creating your new project folder and inside it, add a file called asmcalc.asm. This is where we will put our first set of code.

Now create a build folder inside your project folder.

The build and run batch files

Next, in your project folder, create a file called build.bat and add this:

    
nasm -f win64 asmcalc.asm -o build/asmcalc.obj
GoLink /console /entry main build/asmcalc.obj /fo build/asmcalc.exe

This batch file does two things. Firstly, it uses nasm to assemble our code into a 64-bit Windows object and it puts it into the build folder. Then GoLink links the object and any other resources we tell it to into an executable (there are none yet).

Finally, lets just create another batch file called run.bat. This batch file will simply run the build.bat batch file, before running our new executable:

    
build.bat
build\asmcalc.exe

All done. Now when we write some code, we can simply navigate to the project folder in a command prompt and type build or run and everything will build and run for us.

Let’s write some code

Alright, let’s get a basic setup happening. In asmcalc.asm, drop this code:

    
global main

section .text

main:

    xor rax,rax
    ret

Pretty straight forward stuff here. All we’re doing is declaring an entry point, emptying the rax register and returning. The equivalent C code is:

    
int main() {
    return 0;
}

The rax register is where our integer and pointer return values are stored when returning from any functions (this is different for floating point return values - which will need to be covered in a future post).

The 64-bit Windows ABI

The 64-bit Windows Application Binary Interface specifies exactly how code should run when running under 64-bit Windows. It covers many topics including how to pass arguments to functions, and even the byte boundaries/alignment requirements.

Stack alignment

It might sound stupid… but right now, this simple bit of code is not 64-bit Windows ABI compatible. Why not?

The stack must be aligned to a 16 byte boundary to satisfy the ABI. There are a couple of reasons for this - none we really have to worry about at this stage. However, let’s investigate to see what has happened.

Run build.bat and then fire up Visual Studio. Open up the executable as a solution and it should become the only item in the Solution Explorer.

asmcalc_project

Step into the code by pressing F11 (or Debug > Step Into). In your favourite viewing area, lets check the value of the rsp register. I use the Immediate Window for most of this stuff. You can get there via Debug > Windows > Immediate.

unaligned_stack

Viewed as a hexidecimal value, the value of the rsp register doesn’t end in a zero (NOTE: If your default settings aren’t to view values in hex, then you can use the ,h formatting suffix for Visual Studio. Instead of inspecting rsp, you can inspect rsp,h to show its value as hexidecimal). Basically, to align the stack on a 16 byte boundary, the hexidecimal representation of rsp must end in a zero (since that would make it a multiple of 16).

“But I haven’t done anything.. how can the stack be unaligned?”. Well, lets see.

When a function is called using the call instruction, two things happen. Firstly, the address of the very next instruction is placed on the stack. Secondly, a jmp is performed to jump to the function address you’re calling. When that function returns via a ret instruction, the stack is popped to find the address execution should return to.

Lets go back to the first part of that though.

“the address of the very next instruction is placed on the stack”.

Assuming the calling process had an aligned stack (it should.. since the operating system has called our code and the operating system should be following its own ABI), when it calls our process, it will automatically unalign the stack pointer. This is important, because literally every call we make in our own code will also cause this to happen.

To fix this, we can simply align the stack manually upon entering our code and returning it to what it was before exiting.

    
global main

section .text

main:

    sub rsp,0x08
    xor rax,rax
    add rsp,0x08 ; restore our stack pointer (thanks Michael Petch for noting this omission in the comments!)

    ret

The stack will now be aligned after entering our main function:

aligned_stack

Leaf vs non-leaf functions

It should be noted that this isn’t important for our very small program, but its important to understand as our program grows. This function is technically a “leaf function” and as such we don’t actually need to align the stack. This example is just for demonstration purposes and as we progress, we will talk about register volatility, leaf functions and non-leaf functions.

Wrapping up

Congratulations! We’ve just created a very simple 64-bit Windows executable and learned something about the Windows ABI along the way.

Next time, we’ll make it do something more interesting by displaying text on the screen.

If you’re following along at home, this post is tagged as part1 in my asmcalc repository on GitHub