sched-ext Tutorial

Extensible Scheduler Class, or better known as sched-ext is a Linux kernel feature which enables implementing kernel thread schedulers in BPF (Berkeley Package Filter) and dynamically loading them. Essentially this allows end-users to change their schedulers in userspace without the need to build another kernel just to have a different scheduler.

Methods for starting and managing schedulers

The schedulers can be found in the scx-scheds and scx-scheds-git package.

# Stable branch
sudo pacman -S scx-scheds

# Bleeding edge branch (This branch includes the latest changes
# and may contain a scheduler that has not yet been released.)
sudo pacman -S scx-scheds-git

Starting the Scheduler in the Terminal

To start the scheduler, open your terminal and enter the following command:
Example of starting rusty
```
sudo scx_rusty
```

This will launch the rusty scheduler and detach the default scheduler.

To stop the scheduler. Press CTRL + C and the scheduler will then be stopped and the default kernel scheduler will take over again.

Systemd Service

The scx package includes a systemd service that uses the configuration specified in the /etc/default/scx file.

In this configuration file, you can specify the scheduler that the service starts and optionally include custom flags for the desired scheduler.

If you want to change the scheduler started by the service simply modify the SCX_SCHEDULER= line to the scheduler you want to start by default.
- Example
```
SCX_SCHEDULER=scx_lavd
```
Adding flags
- Uncomment SCX_FLAGS and add the flags you desire.
  Example
```
SCX_FLAGS='--performance'
```

Now you can start/enable/stop the scheduler as if it was any other systemd service.

Check the brief guide on how to manage it below.

sudo systemctl enable --now scx

sudo systemctl start scx

sudo systemctl stop scx

For more information about this service: Sched-ext systemd service

scx_loader

As the name implies, it is a utility that functions as a loader and manager for the sched-ext framework using the D-Bus interface.

While it does not require systemd, it can still be utilized in conjunction with it. Check the transition guide for reference)

Has the ability to stop, start, restart, read information about a scx scheduler and more.
- You can use tools like dbus-send or gdbus to communicate with it.

This guide explains how to use scx_loader with the dbus-send command.

dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.scx.Loader.StartScheduler string:scx_rusty uint32:0

# This example starts scx_bpfland with the following flags: -k -c 0
dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.scx.Loader.StartSchedulerWithArgs string:scx_bpfland array:string:"-k","-c","0"

dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.scx.Loader.StopScheduler

dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.scx.Loader.SwitchScheduler string:scx_lavd uint32:2
# This switches to scx_lavd with the scheduler mode 2 meaning it starts LAVD in powersaving

dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.scx.Loader.SwitchSchedulerWithArgs string:scx_bpfland array:string:"-k","-c","0"

dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.freedesktop.DBus.Properties.Get string:org.scx.Loader string:CurrentScheduler

dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.freedesktop.DBus.Properties.Get string:org.scx.Loader string:SupportedSchedulers

CachyOS Kernel Manager

The scx schedulers can be accessed and configured using the brand new scx_loader.

Introduction to the main schedulers

Since there are many schedulers to choose from, we want to give a little introduction about the schedulers in hand.

Feel free to report any issue or feedback to their GitHub referenced below.

scx_bpfland

Developed by: Andrea Righi (arighi GitHub)

A vruntime-based sched_ext scheduler that prioritizes interactive workloads. Highly flexible and easy to adapt.

Bpfland when making decisions on which cores to use, it takes in consideration their cache layout and which cores share the same L2/L3 cache leading to fewer cache misses = more performance.

Use cases:

Gaming
Desktop usage
Multimedia/Audio production
Great interactivity under intensive workloads
Power saving
Server workloads

scx_flash

Developed by: Andrea Righi (arighi GitHub)

A scheduler that focuses on ensuring fairness among tasks and performance predictability. This scheduler is introduced as a replacement of the “lowlatency” mode in scx_bpfland.

Use cases:

Gaming
Latency sensitive workloads such as multimedia or real-time audio processing
Need for responsiveness under over-stressed situations
Consistency in performance

scx_lavd

Developed by: Changwoo Min (multics69 GitHub).

Brief introduction to LAVD from Changwoo:

LAVD is a new scheduling algorithm which is still under development. It is motivated by gaming workloads, which are latency-critical and communication-heavy. It aims to minimize latency spikes while maintaining overall good throughput and fair use of CPU time among tasks.

Use cases:

Gaming
Audio Production
Latency sensitive workloads
Desktop usage
Great interactivity under intensive workloads
Power saving

One of the main and awesome capabilities that LAVD includes is Core Compaction. which without going into technical details: When CPU usage < 50%, Currently active cores will run for longer and at a higher frequency. Meanwhile Idle Cores will stay in C-State (Sleep) for a much longer duration achieving less overall power usage.

scx_rusty

Developed by: David Vernet (Byte-Lab GitHub)

Rusty offers a wide range of features that enhance its capabilities, providing greater flexibility for various use cases. One of these features is tunability, allowing you to customize Rusty to suit your preferences and specific requirements.

Use cases:

Gaming
Latency sensitive workloads
Desktop usage
Multimedia/Audio production
Latency sensitive workloads
Great interactivity under intensive workloads
Power saving

For a more in depth look about what can be tuned for Rusty. Check out the help page

scx_rusty --help

General recommendations

LAVD Autopilot & Autopower

Quotes from Changwoo Min:

In autopilot mode, the scheduler adjusts its power mode Powersave, Balanced, or Performance based on the system’s load, specifically CPU utilization
Autopower: Automatically decide the scheduler’s power mode based on the system’s energy profile aka EPP (Energy Performance Preference).

# Autopower can be activated by the following flag:
--autopower
# e.g:
scx_lavd --autopower

Disable ananicy-cpp

In order to disable/stop ananicy-cpp, run the following command:

systemctl disable --now ananicy-cpp

Transitioning from scx.service to scx_loader: A Comprehensive Guide

First let’s start with a close-up comparison between the scx.service file structure against the scx_loader configuration file structure.

If you previously had LAVD running with the old scx.service like this example below:

# List of scx_schedulers: scx_bpfland scx_central scx_flash scx_lavd scx_layered scx_nest scx_qmap scx_rlfifo scx_rustland scx_rusty scx_simple scx_userland
SCX_SCHEDULER=scx_lavd

# Set custom flags for the scheduler
SCX_FLAGS='--performance'

Then the equivalent on the scx_loader configuration file will look like:

default_sched = "scx_lavd"
default_mode = "Auto"

[scheds.scx_lavd]
auto_mode = ["--performance"]

For more information on how to configure the scx_loader file

Follow the guide below for an easy transition from the scx systemd service to the new scx_loader utility.

 systemctl disable --now scx.service && systemctl enable --now scx_loader.service

# Micro editor is going to create a new file.
sudo micro /etc/scx_loader.toml
# Add the following lines:

default_sched = "scx_bpfland" # Edit this line to the scheduler you want scx_loader to start at boot
default_mode = "Auto" # Possible values: "Auto", "Gaming", "LowLatency", "PowerSave".

# Press CTRL + S to save changes and CTRL + Q to exit Micro.

Restarting the scx_loader
```
systemctl restart scx_loader.service
```
- You’re done, the scx_loader will now load and start the desired scheduler.

Debugging in the scx_loader

Checking the service status
```
systemctl status scx_loader.service
```
Viewing all the service log entries
```
journalctl -u scx_loader.service
```
Viewing only the logs of the current session.
```
journalctl -u scx_loader.service -b 0
```
Advanced logging

In order to get a more detailed log, follow these steps.

Edit the service file
```
sudo systemctl edit scx_loader.service
```
Add the following line under the [Service] section
```
Environment=RUST_LOG=trace
```

sudo systemctl restart scx_loader.service

Check the logs again for a more detailed debugging information.

FAQ

Why X scheduler performs worse than the other?

There are many variables to consider when comparing them. For example, how do they measure a task’s weight? Do they prioritize interactive tasks over non-interactive ones? Ultimately, it depends on their design choices.

Why everyone keeps saying this X scheduler is the best for X case but it does not perform as well for me?

Like the previous answer, the choice of CPU and its design such as the core layout, how they share cache across the cores and other related factors can lead to the scheduler operating less efficiently.
That’s why having choices is one of the highlights from the sched-ext framework, so don’t be scared to try one and see which one works best for your use case. Examples: fps stability, maximum performance, responsiveness under intensive workloads etc.

The use cases of these schedulers are quite similar… why is that?

Primarily because they are multipurpose schedulers, which means they can accommodate a variety of workloads, even if they may not excel in every area.
To determine which scheduler suits you best, there’s no better advice than to try it out for yourself.

I’m missing a scheduler that some users are mentioning or testing in the CachyOS Discord server

Make sure you’re using the bleeding edge version of the scx-scheds package named as scx-scheds-git

One of the reasons will be that this scheduler is very new and is currently being tested by the users, therefore it has not yet been added to the scx-scheds-git package.

Why did the scheduler suddenly crashed? Is it unstable?

There could be a few reasons on why this happened:
- One of the most common reason is that you were using ananicy-cpp alongside the scheduler. This why we added this warning
- Another reason could be that the workload you were running exceeded the limits and capacity of the scheduler causing it to stall.
  - Example of an unreasonable workload: hackbench
- Or the more obvious reason, you’ve found a bug in the scheduler, if so. Please report it as an issue in their GitHub or let them know about it in the CachyOS Discord channel sched-ext

I have previously used the scx_loader in the Kernel Manager GUI. Do I still need to follow the transition steps?

In this particular case, no, it is not necessary because the Kernel Manager already handles the transition process.
- Unless you have previously added custom flags in /etc/default/scx and still want to use them.

Learn More

If you want to learn more about the sched-ext framework. Take a look at the links below.