The control plane

Everything about a device’s existence, as opposed to its I/O, goes through /dev/ublk-control: creating the device, describing it, starting and stopping it, deleting it, and on newer kernels resizing, quiescing and recovering it. This chapter covers how a control command is built and then each command in turn.

Issuing a control command#

Control commands are io_uring passthrough commands. Open /dev/ublk-control read-write, create an io_uring with IORING_SETUP_SQE128 (the driver rejects control commands from a ring without 128-byte SQEs), and for each command fill one SQE:

The CQE’s res is 0 on success or a negative errno. UBLK_U_CMD_REG_BUF is the only command that returns a positive value (a buffer index).

Several commands block in the kernel until something else happens: START_DEV until every tag has been fetched, DEL_DEV until the device’s last reference is gone, END_USER_RECOVERY until the new server has fetched every tag, QUIESCE_DEV until its timeout. The driver never runs them inline from the submitting io_uring_enter (it returns -EAGAIN for non-blocking issue, so io_uring punts them to a worker thread), so the submitter is not stuck, but the CQE will not arrive until the condition is met. Wait with a timeout and expect EINTR-style interruptions. Do not issue a blocking control command from a thread whose progress it waits on: START_DEV sent from the only thread that was going to submit the fetches never completes.

struct ublksrv_ctrl_cmd#

The struct is 32 bytes. Any buffer you pass by addr must stay valid and unmoved until the CQE arrives, which matters in garbage-collected languages: pin it, and if you give up waiting, leak it rather than free it, because the kernel may still write to it.

Legacy and ioctl-encoded opcodes#

Linux 6.0 through 6.3 identified commands by small integers (UBLK_CMD_ADD_DEV is 0x04). Linux 6.4 introduced ioctl-style encodings built with _IOR/_IOWR('u', nr, struct ublksrv_ctrl_cmd), named UBLK_U_CMD_*, and every command added since exists only in that form. The driver accepts both encodings unless the kernel was built without CONFIG_BLKDEV_UBLK_LEGACY_OPCODES, in which case legacy numbers fail with EOPNOTSUPP. New servers should use the encoded form; supporting kernels older than 6.4 means also speaking the legacy numbers.

The kernel always reports UBLK_F_CMD_IOCTL_ENCODE in the negotiated flags of devices it creates, so a server that sees the bit knows the encoded form is understood (since 6.4).

Commands#

The table lists every command in the 7.3-rc5 header. “Blocks” means the CQE waits on another event.

The legacy opcodes of the 6.0 and 6.1 commands are their low byte (ADD_DEV 0x04, DEL_DEV 0x05, START_DEV 0x06, STOP_DEV 0x07, SET_PARAMS 0x08, GET_PARAMS 0x09, GET_QUEUE_AFFINITY 0x01, GET_DEV_INFO 0x02, recovery 0x10 and 0x11, GET_DEV_INFO2 0x12).

ADD_DEV#

Creates a device and its char device /dev/ublkcN. addr points at a 64-byte struct ublksrv_ctrl_dev_info that the server fills in and the kernel overwrites with the negotiated result:

Read every field back. The kernel can give you fewer queues than you asked for (one per CPU at most), a smaller buffer size, and a different feature set: it clears flags it does not know, clears UBLK_F_NEED_GET_DATA when a user-copy or zero-copy mode is set, and forces on flags that describe its own behavior (in 6.17: UBLK_F_CMD_IOCTL_ENCODE, UBLK_F_URING_CMD_COMP_IN_TASK, UBLK_F_PER_IO_DAEMON and UBLK_F_BUF_REG_OFF_DAEMON; 7.3-rc5 adds UBLK_F_SAFE_STOP_DEV and withholds UBLK_F_PER_IO_DAEMON from batch-I/O devices). Feature flags lists the negotiation rules in order. The flags it returns are the contract for the device’s lifetime. A server that sizes its queue threads or buffers from what it requested instead of what came back will map descriptor arrays that do not exist or overrun buffers.

ADD_DEV fails with:

• EINVAL for a depth or queue count of 0 or above 4096, a header queue_id other than -1, a header dev_id that differs from info.dev_id, an ID above the maximum, an invalid combination of recovery flags, UBLK_F_QUIESCE without UBLK_F_USER_RECOVERY, UBLK_F_ZONED without a user-copy or zero-copy mode, UBLK_F_INTEGRITY without UBLK_F_USER_COPY, an out-of-range io_desc_size with UBLK_F_IO_DESC_SIZE, or (for unprivileged devices) a copy mode that unprivileged servers may not use.
• EPERM without CAP_SYS_ADMIN, unless UBLK_F_UNPRIVILEGED_DEV is set. With the capability, that flag is silently cleared and the device is an ordinary privileged one.
• EEXIST if the requested ID is taken (an auto-assigned ID is the lowest free one), EACCES when the unprivileged-device limit is reached, ENOMEM.

SET_PARAMS and GET_PARAMS#

SET_PARAMS describes the block device that START_DEV will create: capacity, block sizes, maximum request size, attributes such as read-only and volatile cache, discard limits, zoned limits and so on. addr points at a struct ublk_params whose first two fields are len (bytes of the structure the server is passing) and types (a bitmask of UBLK_PARAM_TYPE_* saying which sub-structures are valid). The basic block is mandatory. Device parameters covers every field and validation rule.

Parameters can be set any number of times before the first START_DEV, and not after it: once the disk exists the command fails with EACCES until the disk is gone again. A failed validation returns EINVAL and clears all parameters, so a later START_DEV fails too.

The kernel copies at most its own sizeof(struct ublk_params) and masks types to the types it knows, without telling you. A server built against a newer header that sets, say, UBLK_PARAM_TYPE_SEGMENT on a 6.14 kernel gets a device without segment limits and a successful return. Read the result back with GET_PARAMS (set len in the buffer first; the kernel copies min(len, sizeof) bytes and always fills the read-only devt block) if a parameter matters.

START_DEV#

Exposes /dev/ublkbN. data[0] must be the PID (thread-group ID) of the process that opened /dev/ublkcN; anything else, or a missing basic parameter block, is EINVAL.

The order matters. START_DEV first waits, interruptibly, until every tag of every queue has a FETCH_REQ outstanding, then allocates the disk with the limits from SET_PARAMS, applies the attributes, and calls add_disk, which emits udev events. Before 6.19 (and in stable 6.18 before 6.18.4) add_disk also scans the partition table synchronously, so the server receives reads of the first sectors while START_DEV is still in flight. From 6.19 the driver suppresses that scan during add_disk and queues it on a work item instead, so START_DEV no longer waits for it; no scan happens at all for unprivileged servers or with UBLK_F_NO_AUTO_PART_SCAN. Either way the queue threads must be submitting their fetches before or while START_DEV is pending, and must be serving reads from that moment. A device already live returns EEXIST.

Partition scanning is suppressed for devices whose queues are served by unprivileged tasks, and from Linux 7.0 can be turned off with UBLK_F_NO_AUTO_PART_SCAN.

STOP_DEV#

Removes /dev/ublkbN and returns the device to UBLK_S_DEV_DEAD. Internally this is del_gendisk, which flushes and drains: a mounted filesystem is synced through the device, and every in-flight request has to complete before the command returns. The server must keep its queues running while STOP_DEV is in progress; the drain is served by the same threads. After the disk is gone the driver completes every pending fetch with UBLK_IO_RES_ABORT (-ENODEV), which is the signal for the queue threads to exit. The command itself returns 0.

The safe shutdown order for a server is therefore:

STOP_DEV            (queues still serving; returns after the drain)
stop queue threads  (their fetches complete with -ENODEV)
close io_urings, munmap descriptors, close /dev/ublkcN
DEL_DEV

TRY_STOP_DEV#

Linux 7.0+ Stops the device only if nothing has /dev/ublkbN open. It fails with EBUSY while the disk has openers and ENODEV if the device has no disk; otherwise it blocks new opens and performs a normal STOP_DEV. Belongs to UBLK_F_SAFE_STOP_DEV, which 7.3-rc5 reports on every device. Useful for “detach if idle” without pulling a disk out from under a mounted filesystem.

DEL_DEV and DEL_DEV_ASYNC#

DEL_DEV removes /dev/ublkcN and frees the device ID. If the device is still live it is stopped first. The command then waits until the device’s last reference is dropped, so that the ID can be reused as soon as it returns. Every open file description of /dev/ublkcN holds such a reference, including ones a server forgot it had (a duplicated fd, a descriptor array still mapped, an io_uring with the fd registered). Sending DEL_DEV from a process that still holds one is a self-deadlock; the wait is interruptible, so the command returns EINTR if the process is signalled.

DEL_DEV_ASYNC (Linux 6.9) does the same removal without waiting. The ID becomes reusable whenever the last reference goes away.

A device whose server died without deleting it stays registered in state DEAD (or, with recovery enabled, QUIESCED or FAIL_IO) and keeps the module pinned. Any process with CAP_SYS_ADMIN can delete it by ID.

GET_DEV_INFO and GET_DEV_INFO2#

Copy the device’s current ublksrv_ctrl_dev_info to addr (len at least 64). Fails with ENODEV for an ID that does not exist, which is also how a server enumerates devices: there is no list command, so tools probe IDs.

GET_DEV_INFO2 (Linux 6.3) is the same command with the char-device path prepended to the buffer, for the unprivileged permission check. The kernel requires the path for GET_DEV_INFO2 even on privileged devices, because the caller may not know which kind it is asking about.

GET_QUEUE_AFFINITY#

Returns the set of CPUs that blk-mq maps to queue data[0], as a CPU bitmask in the buffer at addr. len must be a multiple of sizeof(unsigned long) and large enough for every possible CPU ID. A server uses it to pin each queue thread to the CPUs whose I/O lands on that queue, so a request is served on the CPU that issued it.

GET_FEATURES#

Linux 6.5+ Writes the 64-bit set of UBLK_F_* features this kernel supports into an 8-byte buffer (len must be exactly UBLK_FEATURES_LEN, 8). It does not take a device, so it is the way to probe a kernel before creating one. On kernels before 6.5 it fails, and the fallback is to request the flags you want in ADD_DEV and inspect what comes back.

UPDATE_SIZE#

Linux 6.16+ Changes the capacity of a started device to data[0] 512-byte sectors and notifies the block layer, which emits a resize uevent. It belongs to UBLK_F_UPDATE_SIZE, though the driver does not check the flag. Only send it to a started device: 7.3-rc5 returns ENODEV otherwise, and the 6.17 driver does not check at all. The server must be ready to serve the new range before it grows the device and must stop relying on the old range only after it shrinks it.

QUIESCE_DEV, START_USER_RECOVERY, END_USER_RECOVERY#

The recovery commands move a device whose server has exited, or is about to be replaced, back to a live server without removing /dev/ublkbN. QUIESCE_DEV (6.16, needs UBLK_F_QUIESCE) takes a timeout in milliseconds in data[0] and parks a live device so that a new server can take over; START_USER_RECOVERY and END_USER_RECOVERY (6.1, need UBLK_F_USER_RECOVERY) bracket the takeover. They are covered with the states they produce in User recovery and quiesce.

REG_BUF and UNREG_BUF#

Linux 7.1+ Register and unregister a shared-memory buffer for UBLK_F_SHMEM_ZC. REG_BUF takes a struct ublk_shmem_buf_reg (address, length, flags) through addr and returns the buffer index; UNREG_BUF takes the index in data[0]. See Data copy modes.

Device states#

The state field of ublksrv_ctrl_dev_info takes four values:

A stopped device cannot be started again. STOP_DEV does not reset the queues: they still count as fully fetched, so new FETCH_REQs fail with -EBUSY and a different process cannot even map the descriptors, while a START_DEV sent anyway can bring up a disk with no fetch commands armed and crash the kernel on its first request. go-ublk’s kernel matrix saw all three outcomes, -EBUSY, a wedged control plane (6.10 to 6.12) and a NULL dereference in ublk_queue_rq (Arch 7.2.8); see known kernel bugs. Delete the device and add a new one.

Putting it together#

A minimal server’s control thread, error handling elided:

int ctrl = open("/dev/ublk-control", O_RDWR);
struct io_uring ring;
io_uring_queue_init(32, &ring, IORING_SETUP_SQE128);

struct ublksrv_ctrl_dev_info info = {
    .nr_hw_queues     = 4,
    .queue_depth      = 128,
    .max_io_buf_bytes = 1 << 20,
    .dev_id           = -1,                 /* let the kernel choose */
    .flags            = 0,                  /* plain copy mode */
};
ctrl_cmd(&ring, ctrl, UBLK_U_CMD_ADD_DEV,
         &(struct ublksrv_ctrl_cmd){ .dev_id = -1, .queue_id = -1,
                                     .addr = (__u64)&info, .len = sizeof(info) });
/* info now holds the negotiated queues, depth, buffer size, flags and ID */

struct ublk_params p = { .len = sizeof(p), .types = UBLK_PARAM_TYPE_BASIC,
    .basic = { .logical_bs_shift = 9, .physical_bs_shift = 12,
               .io_min_shift = 9, .io_opt_shift = 12,
               .max_sectors = info.max_io_buf_bytes >> 9,
               .dev_sectors = size_bytes >> 9,
               .attrs = UBLK_ATTR_VOLATILE_CACHE } };
ctrl_cmd(&ring, ctrl, UBLK_U_CMD_SET_PARAMS,
         &(struct ublksrv_ctrl_cmd){ .dev_id = info.dev_id, .queue_id = -1,
                                     .addr = (__u64)&p, .len = sizeof(p) });

start_queue_threads(&info);    /* open /dev/ublkcN, mmap, FETCH_REQ every tag */
ctrl_cmd(&ring, ctrl, UBLK_U_CMD_START_DEV,
         &(struct ublksrv_ctrl_cmd){ .dev_id = info.dev_id, .queue_id = -1,
                                     .data = { getpid() } });

wait_for_shutdown_signal();

ctrl_cmd(&ring, ctrl, UBLK_U_CMD_STOP_DEV, &(struct ublksrv_ctrl_cmd){ .dev_id = info.dev_id, .queue_id = -1 });
join_queue_threads();          /* they exit on -ENODEV; close every fd and mapping */
ctrl_cmd(&ring, ctrl, UBLK_U_CMD_DEL_DEV, &(struct ublksrv_ctrl_cmd){ .dev_id = info.dev_id, .queue_id = -1 });

ctrl_cmd here prepares one IORING_OP_URING_CMD SQE with the given cmd_op and header, submits it, and waits for its CQE, returning cqe->res. The data plane chapter fills in start_queue_threads.

Edit this page on GitHub