How to get started with real-time Linux?

I found this to be hard-ish to get started as well. However, it turns out you just need to write your program to be performant, and know a few tricks to tune the OS and the scheduler. After looking on the internet and failing to find material about this, I wrote a series of blog post for this topic recently, starting with this post: https://shuhaowu.com/blog/2022/01-linux-rt-appdev-part1.html

The series is mostly complete, except one to two posts on more advanced topics like validation and lockless programming.

It's not Linux but if you wanted to learn about real time, you could check out Ardupilot for drones. You'll get to learn about a variety of topics including:

- hardware (embedded hardware, brushless motors, propellers, batteries, radio, etc.)

- guidance (sensor fusion algorithms, kalman filtering, etc.)

- navigation (GPS/GNSS, magnetometers, barometers, accelerometers, gyroscopes, etc.)

- control (PWM control, PIDs, feed forward loops, etc.)

Your best starting point in LinuxCNC it may very well be the single greatest application for real time linux

Real-Time is a broad field and implemented in one of 3-ways a) On bare-metal, b) Using a RTOS (eg. FreeRTOS, QNX) c) On Linux/etc. As such you have to understand what is actually involved when you say something is "real-time" (Hard/Soft). As a first approximation "real-time" can be said to be about minimizing latency (within accepted bounds) of service requests (ISRs, task functions etc). Thus if you do it via (a) above you get the best control but have to do everything, (b) gives you some framework support via a RTOS kernel and (c) is always "soft real-time" unless you break out of the kernel. The difficulty increases as you go from (a)->(b)->(c) since there are more intermediate layers and abstractions. But this is the best way to approach the study of the subject. With that in mind i can suggest the following books;

1) Real-Time Embedded Systems: Design Principles and Engineering Practices by Xiaocong Fan. - Good text book.

2) Introduction to Embedded Systems: Using Microcontrollers and the MSP430 by Manuel Jiminez et.al. - Very good book teaching you bare-metal MCU programming.

3) Hands-On RTOS with Microcontrollers: Building real-time embedded systems using FreeRTOS, STM32 MCUs, and SEGGER debug tools by Brian Amos - Great book showing the use of all tools/ecosystem needed for RTOS programming.

4) POSIX.4 Programmers Guide: Programming for the Real World by Bill Gallmeister. - The classic indispensable work on "real-time programming" for Unix-like systems.

Real-time linux can be thought of taking vanilla linux - configuring the kernel in a certain way, booting your kernel with certain boot params and writing your real-time application in a certain way.

While there quite a few hiccups along the way - you will be able to bring down jitter or OS noise to around 1 us. Of course a microkernel based OS like QNX can take you down to around 1 ns - but then the point of using Linux is that you are able to utilize the ecosystem around it.

Look into isolation, real-time patch and high performance computing.

If it must not be linux, I would recommend going for something embedded, like FreeRTOS or similar. The advantages: - Linux + Real Time has become a buzz word, and you will find lots of bad information - If you want to learn HW, nothing better as embedded, where you interact with the hardware. - The FreeRTOS is so small you can really understand it pretty fast. You can see how it is made, much easier than Linux. - The essentials you learn with that, will work with Linux later.

You are aware that real time only means guaranteed right?

What is your goal as this is a broad field?

If you wanna just start with it, I would try to read up on cars, can Bus and how all of that works.

The best I've managed to achieve is by taking Linux off the real-time equation as much as possible : use isolcpu to tell Linux to not schedule processes on specific cores, put them there yourself with pthread API, nice etc. You can also control on which core interrupts happen fairly easily. This gives you almost microcontroller-like access to some of your CPU cores.

I think I have been on a similar path of thought: While there are many speed tricks and parallelized processing to try and to use in modern PC, there are surprisingly little available resources for realtime, as in synchronous-to-wall-clock, processing.

You can set up a timer interrupts in Arduino, There are web-JavaScript tutorials to set up callback function to execute every 3000ms, but you won't be doing RPC over network with timestamps and expiration, without going neck deep in something specialized and chaotic as ROS2.

I'm picturing some sort of distributed, networked, timestamped system that can schedule remote executions and construct time correlated observations, something obvious to designers/architects of original SNMP spec and NIST 4D/RCS project, and I'm predicting we'll have an industry standard framework just casually appearing by the end of this decade. It's weird we don't have one.

Basically:

- Traditional Unix optimises for sharing resources and using them efficiently - 'multiprocessing', 'time slicing', 'resource sharing', 'garbage collecting', etc.

- Realtime: optimises for deterministic low latency. SCHED_FIFO processes that use a logical CPU and do not yield during the trading day, use money to buy more memory or crash rather than allow the GC to destroy your latency during the trading day.

https://access.redhat.com/documentation/en-us/red_hat_enterp... is RHEL RT focused but the kernel optimisations discussed are all OSS licensed.

Don't.

Linux is too complex to provide any sort of guarantee; real-time Linux is just soft realtime.

Look at an actual RTOS instead, such as seL4[0].

0. https://sel4.systems/About/seL4-whitepaper.pdf

I seem to recall an ubuntu music variant distro with the RT patch baked in but cannot for the life of me dig it up.

This however may give you a nice point to dive in from:

https://ubuntu.com/engage/an-introduction-to-real-time-linux...

for more advanced projects consider getting preempt-rt going in a yocto install.

Others are offering good books and things, but my experience during my autonomous vehicle days, I found just seeing how rt-linux was being used by others during market research was very useful in understanding how it was being used and where it failed and needed a more tested RTOS. So my suggestion is start searching the startup space and see how its being used these days.

Not a book or a project, but I can't recommend enough using Xenomai4/EVL. Under the common use case of "some small kernel of code needs bounded latency" plus "I want to use normal Linux and not think about realtime shenanigans damnit," you'll have a substantially easier time developing in that environment than anything else I'm aware of.

Having only dealt with this at the level of writing code for a RISC microcontroller, another facet I don't see mentioned that may be of importance is the following CPU implementation detail.

Code compiles to assembly instructions which typically appear atomic to the author, however, such instructions may take variable numbers of cycles to execute.

1. install hwlatdetect

2. use hwlatdetect

3. give up when you realize you can't buy commodity hardware with documented NMIs or open-source firmware

I am not an expert on the matter but as far as I understand you need the appropriate hardware platform, too? Last time I did some research about it I came to the conclusion that for instance Raspberries make for a poor RT platform. Is that not the case?

https://rt.wiki.kernel.org https://en.m.wikipedia.org/wiki/RTLinux