I wrote a book to learn x86 programming and it was published in 1994 back before "real digital age", oriented to be the best didactical text possible (I had to learn the hard way, I wrote what I would have had to read 10 years earlier). Over 20 reprints, recommended in all Spanish-speaking universities (yeah, I know, if I had known then what I know now I'd have done it in English). It was discontinued 10 years ago or so, they asked me to revise it for the modern world, and it didn't really make sense: rewriting it for what assembly language is and is used for in today's world would be a ton of work because it's a qualitative difference, and if I added an extra chapter explaining 32-bit and 64-bit changes and letting the publisher stamp a "2010 edition" logo on the cover would just be scamming people, which I don't want to do. Here is the link to a scanned copy of the original: https://www.dropbox.com/s/sz6rinfhyc8sai6/Lenguaje%20Ensambl... . Could prove useful if you can speak Spanish.
Honestly: I learned Z80 assembly first, in the 80s, and then switched to x86 very easily. Learn whatever assembly language first, what's hard is learning about registers, flags, memory... and if you learn it will, then you can switch to another architecture quite easily.
IA-32e/AMD64 is basically an extension of IA-32/x86. What you're looking for is to understand the difference between real mode, 386 protected mode, PAE, paging and long mode because the registers, instructions, addressing and sizes of structures differ.
You can learn all of this by reading the Intel software manuals and digging into it yourself.
Tools you need (for non-Windows):
- gdb/lldb
- assemblers: yasm, nasm, binutils (contains "as")
Optional tools:
- IDA Pro
- Virtualization/emulation such as VMware Fusion/Workstation (because it supports being a gdbserver), VirtualBox or QEMU
- Intel's CPUID app
You also need good references like:
- the Intel manual set (a giant PDF or a collection of several) https://software.intel.com/en-us/articles/intel-sdm
- https://sandpile.org (fairly current)
- https://ref.x86asm.net (outdated by useful)
Another helpful exercise is writing a toy operating system in Rust or only assembly. https://osdev.org has many resources and guides.
Background:
--- Matt Pietrek's "Just Enough Assembly Language to Get By" - http://bytepointer.com/resources/pietrek_asm_pt1.htm
--- Hongjiu Lu "ELF: From The Programmer's Perspective" - http://beefchunk.com/documentation/sys-programming/binary_fo...
Books:
--- Computer Systems: A Programmer's Perspective (3rd Edition) by Bryant & O'Hallaron - https://www.amazon.com/Computer-Systems-Programmers-Perspect...
--- Modern X86 Assembly Language Programming by Daniel Kusswurm - https://www.amazon.com/Modern-X86-Assembly-Language-Programm...
--- Low-Level Programming by Igor Zhirkov - https://www.amazon.com/Low-Level-Programming-Assembly-Execut...
But the early CPUs were actually quite simple! At the time "CISC" was a good thing because it provided straightforward mechanisms for expressing things (e.g. "push", "call", load a struct field with an offset...) that real world programmers needed to do all the time.
If you know how to write low-level C (i.e. with direct Win32/POSIX API calls), that's a good start. If you don't know what that means, you need to master this first. So much of assembly is built to support higher-level programming, things like segments, indirect references, system calls, etc., so it pays to know the WHY of all of it (e.g. do you know what a frame pointer is? why it's useful and can sometimes be omitted)? Learn this stuff first.
Once you have a good handle on C, you need to start learning how operating systems and memory management work, at least the "client side" of them accessible from userspace.
Then you might want to dip your toe into how CPUs are built, with pipelines, registers, caches, all of that.
If you've mastered those things, gcc -S will provide everything else you need (another comment suggested this).
I learned this stuff in a traditional university computer engineering program. It helped a lot. But for context, this question feels a little like, "can someone explain quantum physics"? It's a huge topic but only like 5% is the actual thing you're asking about, the other 95% is the conceptual machinery (calculus, probability, mechanics) it's built on. Mastering all the other stuff is actually the hard part. For all intents and purposes, assembly is just a notational convenience for expressing program structure in the only way a CPU can understand it. At least 90% of the difficulty is understanding how CPUs work, and how to think at sufficiently low level that you can express human-useful work at such a low level of abstraction.
It would also be nice to know why you want to know this. Writing a device driver is going to be different from writing an operating system, which will be different from writing tight numerical loops in assembly. And in any case, dollars to donuts you won't be able to beat a modern optimizing compiler performance-wise.
Hope this helps.
The other useful step has been to write C functions and compile the code with a good compiler (good experiences with Intel, gcc and llvm) first at -O0 and later at -O2 or -O3 to understand how high level concepts can be translated to assembly.
This requires to have at least a basic understanding of registers and basic assembly concepts but at least has helped me apply the concepts to real code.
For me, a great way of learning the latter is to pick a task with some instant gratification (audio dsp, software texmapping or shaders?) and start writing some of that in inline assembly.
I suggest looking at your C compiler output, learning about SIMD instructions and using a pipeline simulator to see why and how it performs like it does. I used VTune and AMD Codeanalyst back in the day, don't know what's the current SoA.
https://software.intel.com/en-us/articles/introduction-to-x6...
I mean, you have different modes, and in 64-bit mode EAX becomes RAX for example.
https://software.intel.com/en-us/articles/intel-sdm
Try to reverse engineer something you don't have the debug symbols for. One of the big pain points was figuring out the calling convention i.e. what registers correspond to which function arguments on different platforms:
https://en.wikipedia.org/wiki/X86_calling_conventions
For reverse engineering in particular, this was a good resource:
1. https://www.amazon.com/Low-Level-Programming-Assembly-Execut...
2. http://www.egr.unlv.edu/~ed/assembly64.pdf
3. https://www.amazon.com/Introduction-Bit-Assembly-Programming...
The first one is somewhat written in "weird English" (Russian English?), but it is still readable. It really helped me with x64 assembly as a C programmer. I have used 2 and 3 as reference most of the time and the first was basically my "main x64 assembly book". I would also recommend getting more proficient in C programming either by studying books such as "Expert C" and the "The C Programming Language" or reading "advanced" stuff somewhere on the Internet, e.g.: http://www.pvv.org/~oma/DeepC_slides_oct2011.pdf
The main takeaway in all this for me was learning about the call stack and the different calling conventions which gives you a clue on how recursion works under the hood.
Also when you are done learning about "practical computer architecture", i.e. assembly language programming, learn stuff about operating systems as well:
http://pages.cs.wisc.edu/~remzi/OSTEP/
Fun fact: this is not really related to assembly programming, but functions such as setjmp() and longjmp() are used for implementing exception handling.
It starts with the very foundation and builds up, up to interfacing with C and implementing data structures, still in assembly.
Sadly, this book still references 32-bit ISA. My guess is that the author got old enough not to bother updating the book anymore (the first editions of the book were about assembly programming in DOS, then DOS and Linux, then Linux only).
Still a very valid book, as x86-64 is backwards-compatible with 32-bit x86. Also, as somebody else has written, once you understand the basics, you can mostly swap things like register names and/or look to the reference pages of your cpu/linker/assembler.
3rd edition has much info on modern x86.
I have since then ventured into Motorola 68000 assembly coding, by reading an old Amiga course in machine code from the late 80'ties. It's fascinating to learn from an old platform like the Amiga 500.
To keep myself motivated, I have written a lot of posts that chronicles my progress through the course.
https://www.markwrobel.dk/project/amigamachinecode/
EDIT: I can also recommend this game, that is inspired by NAND to Tetris. Where you build the hardware in your browser :) http://www.nandgame.com/
$ gcc foo.c -S
Stay with Intel syntax, nevermind about AT&T syntax, beyond learning on how to read it.
PC world is all about Intel syntax, AT&T is lower level on how instructions get expressed and Assemblers macro capabilities just suck compared with what TASM and MASM were already capable of on the MS-DOS days.
It looks like his x64 is just slides instead of videos which may be better, sitting through 16hrs of video when I was learning x86 was ... an experience, but a very useful one.
Anyway, whenever someone asks this question I don't hesitate to bring up his work and trainings I think they're really useful. I know this is for reverse engineering but if you can reverse engineer x64 you can certainly write it.
One of the best 32bit x86 assembly books out there.
The only really annoying thing is the stack alignment for calls. I usually just make a macro to check rsp and align if needed. Other than that I think it is similar enough to x86 just with different registers.
It's useful to at least look at such a textbook to make sure you don't skip any of the easy, sort-of prerequisite material.
video link -> https://www.youtube.com/watch?v=vtWKlgEi9js&list=PLXYpRL4SQp...
linker -> http://www.masm32.com/
some way to understand assembly lang
Plus, you now have the ymm and possibly zmm stuff going on :)