sched-ext Tutorial

Extensible Scheduler Class, or better known as sched-ext is a Linux kernel feature which enables implementing kernel thread schedulers in BPF (Berkeley Package Filter) and dynamically loading them. Essentially this allows end-users to change their schedulers in userspace without the need to build another kernel just to have a different scheduler.

Methods for starting and managing schedulers

• The schedulers can be found in the scx-scheds and scx-scheds-git package.

  # Stable branch
  sudo pacman -S scx-scheds
  
  # Bleeding edge branch (This branch includes the latest changes
  # and may contain a scheduler that has not yet been released.)
  sudo pacman -S scx-scheds-git

Starting the Scheduler in the Terminal

• To start the scheduler, open your terminal and enter the following command:

  Example of starting rusty

  sudo scx_rusty

This will launch the rusty scheduler and detach the default scheduler.

To stop the scheduler. Press CTRL + C and the scheduler will then be stopped and the default kernel scheduler will take over again.

Systemd Service

Note

This service is going to get deprecated and replaced by the scx_loader

The scx package includes a systemd service that uses the configuration specified in the /etc/default/scx file.

In this configuration file, you can specify the scheduler that the service starts and optionally include custom flags for the desired scheduler.

• If you want to change the scheduler started by the service simply modify the SCX_SCHEDULER= line to the scheduler you want to start by default.

  • Example

    SCX_SCHEDULER=scx_lavd

• Adding flags

  • Uncomment SCX_FLAGS and add the flags you desire.

    Example

    SCX_FLAGS='--performance'

Now you can start/enable/stop the scheduler as if it was any other systemd service.

Check the brief guide on how to manage it below.

Enabling and starting the systemd service

sudo systemctl enable --now scx

Running the scheduler a single time through the systemd service

sudo systemctl start scx

Stopping the SCX scheduler using the systemd service

sudo systemctl stop scx

For more information about this service: Sched-ext systemd service

scx_loader

As the name implies, it is a utility that functions as a loader and manager for the sched-ext framework using the D-Bus interface.

While it does not require systemd, it can still be utilized in conjunction with it. Check the transition guide for reference)

• Has the ability to stop, start, restart, read information about a scx scheduler and more.
  • You can use tools like dbus-send or gdbus to communicate with it.
• This guide explains how to use scx_loader with the dbus-send command.

  • Starting scx_rusty with its default arguments

    dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.scx.Loader.StartScheduler string:scx_rusty uint32:0

  • Starting a scheduler with arguments

    # This example starts scx_bpfland with the following flags: -k -c 0
    dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.scx.Loader.StartSchedulerWithArgs string:scx_bpfland array:string:"-k","-c","0"

  • Stopping the currently running scheduler

    dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.scx.Loader.StopScheduler

  • Switching to another scheduler in Mode 2

    dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.scx.Loader.SwitchScheduler string:scx_lavd uint32:2
    # This switches to scx_lavd with the scheduler mode 2 meaning it starts LAVD in powersaving

  • Switching to another scheduler with arguments

    dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.scx.Loader.SwitchSchedulerWithArgs string:scx_bpfland array:string:"-k","-c","0"

  • Getting the currently running scheduler

    dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.freedesktop.DBus.Properties.Get string:org.scx.Loader string:CurrentScheduler

  • Getting a list of the supported schedulers

    dbus-send --system --print-reply --dest=org.scx.Loader /org/scx/Loader org.freedesktop.DBus.Properties.Get string:org.scx.Loader string:SupportedSchedulers

Tip

Here is an explanation of what each mode means in scx_loader.

• Mode 0 = Default scheduler flags
• Mode 1 = Gaming
• Mode 2 = Power Saving
• Mode 3 = Low Latency
• Mode 4 = Server Currently only available for bpfland

Example: LAVD running in Mode 1 is the equivalent of scx_lavd --performance

TLDR: Each mode is a set of different flags meant to improve the intended use case.

For a more in depth look on what these Modes change on each scheduler

CachyOS Kernel Manager

The scx schedulers can be accessed and configured using the brand new scx_loader.

Introduction to the main schedulers

Since there are many schedulers to choose from, we want to give a little introduction about the schedulers in hand.

Note

These schedulers are in constant development while being tested, so expect some of its features/flags which are subject to change.

Feel free to report any issue or feedback to their GitHub referenced below.

scx_bpfland

Developed by: Andrea Righi (arighi GitHub)

A vruntime-based sched_ext scheduler that prioritizes interactive workloads. Highly flexible and easy to adapt.

Bpfland when making decisions on which cores to use, it takes in consideration their cache layout and which cores share the same L2/L3 cache leading to fewer cache misses = more performance.

Use cases:

• Gaming
• Desktop usage
• Multimedia/Audio production
• Great interactivity under intensive workloads
• Power saving
• Server workloads

scx_flash

Developed by: Andrea Righi (arighi GitHub)

A scheduler that focuses on ensuring fairness among tasks and performance predictability. This scheduler is introduced as a replacement of the “lowlatency” mode in scx_bpfland.

Use cases:

• Gaming
• Latency sensitive workloads such as multimedia or real-time audio processing
• Need for responsiveness under over-stressed situations
• Consistency in performance

scx_lavd

Developed by: Changwoo Min (multics69 GitHub).

Brief introduction to LAVD from Changwoo:

LAVD is a new scheduling algorithm which is still under development. It is motivated by gaming workloads, which are latency-critical and communication-heavy. It aims to minimize latency spikes while maintaining overall good throughput and fair use of CPU time among tasks.

Use cases:

• Gaming
• Audio Production
• Latency sensitive workloads
• Desktop usage
• Great interactivity under intensive workloads
• Power saving

One of the main and awesome capabilities that LAVD includes is Core Compaction. which without going into technical details: When CPU usage < 50%, Currently active cores will run for longer and at a higher frequency. Meanwhile Idle Cores will stay in C-State (Sleep) for a much longer duration achieving less overall power usage.

scx_rusty

Developed by: David Vernet (Byte-Lab GitHub)

Rusty offers a wide range of features that enhance its capabilities, providing greater flexibility for various use cases. One of these features is tunability, allowing you to customize Rusty to suit your preferences and specific requirements.

Use cases:

• Gaming
• Latency sensitive workloads
• Desktop usage
• Multimedia/Audio production
• Latency sensitive workloads
• Great interactivity under intensive workloads
• Power saving

For a more in depth look about what can be tuned for Rusty. Check out the help page

scx_rusty --help

General recommendations

LAVD Autopilot & Autopower

Quotes from Changwoo Min:

• In autopilot mode, the scheduler adjusts its power mode Powersave, Balanced, or Performance based on the system’s load, specifically CPU utilization

• Autopower: Automatically decide the scheduler’s power mode based on the system’s energy profile aka EPP (Energy Performance Preference).

Note

LAVD now enables autopilot by default.

# Autopower can be activated by the following flag:
--autopower
# e.g:
scx_lavd --autopower

Disable ananicy-cpp

Caution

When using any of the schedulers from the sched-ext framework, it’s strongly advised to disable and avoid using ananicy-cpp due to possible conflicts because it amplifies the priority gap that the scheduler is already creating/handling by boosting interactive tasks creating an excessive prioritization starving other tasks increasing the chance of hitting the sched_ext watchdog timeout kicking the running scheduler aka causing an stall.

In order to disable/stop ananicy-cpp, run the following command:

systemctl disable --now ananicy-cpp

Transitioning from scx.service to scx_loader: A Comprehensive Guide

Caution

Do not attempt to run the scx_loader.service alongside the scx.service otherwise, the loader service will start but do nothing.

This conflict arises because both services are unaware of each other, especially regarding which one manages the schedulers.

Tip

The CachyOS Kernel Manager already includes a GUI for managing the scx_loader.

First let’s start with a close-up comparison between the scx.service file structure against the scx_loader configuration file structure.

If you previously had LAVD running with the old scx.service like this example below:

scx.service file structure

# List of scx_schedulers: scx_bpfland scx_central scx_flash scx_lavd scx_layered scx_nest scx_qmap scx_rlfifo scx_rustland scx_rusty scx_simple scx_userland
SCX_SCHEDULER=scx_lavd

# Set custom flags for the scheduler
SCX_FLAGS='--performance'

Then the equivalent on the scx_loader configuration file will look like:

scx_loader file structure

default_sched = "scx_lavd"
default_mode = "Auto"

[scheds.scx_lavd]
auto_mode = ["--performance"]

For more information on how to configure the scx_loader file

Follow the guide below for an easy transition from the scx systemd service to the new scx_loader utility.

1. Disabling scx.service in favor of the scx_loader.service

    systemctl disable --now scx.service && systemctl enable --now scx_loader.service

2. Creating the configuration file for the scx_loader and adding the default structure

   # Micro editor is going to create a new file.
   sudo micro /etc/scx_loader.toml
   # Add the following lines:
   
   default_sched = "scx_bpfland" # Edit this line to the scheduler you want scx_loader to start at boot
   default_mode = "Auto" # Possible values: "Auto", "Gaming", "LowLatency", "PowerSave".
   
   # Press CTRL + S to save changes and CTRL + Q to exit Micro.

3. Restarting the scx_loader

   systemctl restart scx_loader.service

   • You’re done, the scx_loader will now load and start the desired scheduler.

Debugging in the scx_loader

• Checking the service status

  systemctl status scx_loader.service

• Viewing all the service log entries

  journalctl -u scx_loader.service

• Viewing only the logs of the current session.

  journalctl -u scx_loader.service -b 0

• Advanced logging

In order to get a more detailed log, follow these steps.

1. Edit the service file

   sudo systemctl edit scx_loader.service

2. Add the following line under the [Service] section

   Environment=RUST_LOG=trace

3. Restart the service

   sudo systemctl restart scx_loader.service

• Check the logs again for a more detailed debugging information.

FAQ

Why X scheduler performs worse than the other?

• There are many variables to consider when comparing them. For example, how do they measure a task’s weight? Do they prioritize interactive tasks over non-interactive ones? Ultimately, it depends on their design choices.

Why everyone keeps saying this X scheduler is the best for X case but it does not perform as well for me?

• Like the previous answer, the choice of CPU and its design such as the core layout, how they share cache across the cores and other related factors can lead to the scheduler operating less efficiently.
• That’s why having choices is one of the highlights from the sched-ext framework, so don’t be scared to try one and see which one works best for your use case. Examples: fps stability, maximum performance, responsiveness under intensive workloads etc.

The use cases of these schedulers are quite similar… why is that?

• Primarily because they are multipurpose schedulers, which means they can accommodate a variety of workloads, even if they may not excel in every area.

• To determine which scheduler suits you best, there’s no better advice than to try it out for yourself.

I’m missing a scheduler that some users are mentioning or testing in the CachyOS Discord server

Make sure you’re using the bleeding edge version of the scx-scheds package named as scx-scheds-git

• One of the reasons will be that this scheduler is very new and is currently being tested by the users, therefore it has not yet been added to the scx-scheds-git package.

Why did the scheduler suddenly crashed? Is it unstable?

• There could be a few reasons on why this happened:
  • One of the most common reason is that you were using ananicy-cpp alongside the scheduler. This why we added this warning
  • Another reason could be that the workload you were running exceeded the limits and capacity of the scheduler causing it to stall.
    • Example of an unreasonable workload: hackbench
  • Or the more obvious reason, you’ve found a bug in the scheduler, if so. Please report it as an issue in their GitHub or let them know about it in the CachyOS Discord channel sched-ext

I have previously used the scx_loader in the Kernel Manager GUI. Do I still need to follow the transition steps?

• In this particular case, no, it is not necessary because the Kernel Manager already handles the transition process.
  • Unless you have previously added custom flags in /etc/default/scx and still want to use them.

Learn More

If you want to learn more about the sched-ext framework. Take a look at the links below.

• Sched-ext Schedulers Source Code
• Sched-ext Official Wiki
• Changwoo Min: Introduction to sched-ext & CPU Scheduling Part 1
  • Part 2
• Andrea Righi: Re-implementing my Linux Rust scheduler in eBPF
• Andrea Righi: AI-generated Linux kernel schedulers in Rust