mClock Config Reference

QoS support in Ceph is implemented using a queuing scheduler based on the dmClock algorithm. See QoS Based on mClock section for more details.

备注

The mclock_scheduler is supported for BlueStore OSDs. For Filestore OSDs the osd_op_queue is set to wpq and is enforced even if you attempt to change it.

To make the usage of mclock more user-friendly and intuitive, mclock config profiles are introduced. The mclock profiles mask the low level details from users, making it easier to configure and use mclock.

The following input parameters are required for a mclock profile to configure the QoS related parameters:

Using the settings in the specified profile, an OSD determines and applies the lower-level mclock and Ceph parameters. The parameters applied by the mclock profile make it possible to tune the QoS between client I/O and background operations in the OSD.

mClock Client Types

The mclock scheduler handles requests from different types of Ceph services. Each service can be considered as a type of client from mclock's perspective. Depending on the type of requests handled, mclock clients are classified into the buckets as shown in the table below,

Client Type

Request Types

Client

I/O requests issued by external clients of Ceph

Background recovery

Internal recovery/backfill requests

Background best-effort

Internal scrub, snap trim and PG deletion requests

The mclock profiles allocate parameters like reservation, weight and limit (see QoS Based on mClock) differently for each client type. The next sections describe the mclock profiles in greater detail.

mClock Profiles - Definition and Purpose

A mclock profile is “a configuration setting that when applied on a running Ceph cluster enables the throttling of the operations(IOPS) belonging to different client classes (background recovery, scrub, snaptrim, client op, osd subop)”.

The mclock profile uses the capacity limits and the mclock profile type selected by the user to determine the low-level mclock resource control configuration parameters and apply them transparently. Additionally, other Ceph configuration parameters are also applied. Please see sections below for more information.

The low-level mclock resource control parameters are the reservation, limit, and weight that provide control of the resource shares, as described in the QoS Based on mClock section.

mClock Profile Types

mclock profiles can be broadly classified into built-in and custom profiles,

Built-in Profiles

Users can choose between the following built-in profile types:

备注

The values mentioned in the tables below represent the percentage of the total IOPS capacity of the OSD allocated for the service type.

high_client_ops (default)

This profile optimizes client performance over background activities by allocating more reservation and limit to client operations as compared to background operations in the OSD. This profile is enabled by default. The table shows the resource control parameters set by the profile:

Service Type

Reservation

Weight

Limit

client

50%

2

MAX

background recovery

25%

1

100%

background best-effort

25%

1

MAX

high_recovery_ops

This profile optimizes background recovery performance as compared to external clients and other background operations within the OSD. This profile, for example, may be enabled by an administrator temporarily to speed-up background recoveries during non-peak hours. The table shows the resource control parameters set by the profile:

Service Type

Reservation

Weight

Limit

client

30%

1

80%

background recovery

60%

2

200%

background best-effort

1 (MIN)

1

MAX

balanced

This profile allocates equal reservation to client I/O operations and background recovery operations. This means that equal I/O resources are allocated to both external and background recovery operations. This profile, for example, may be enabled by an administrator when external client performance requirement is not critical and there are other background operations that still need attention within the OSD.

Service Type

Reservation

Weight

Limit

client

40%

1

100%

background recovery

40%

1

150%

background best-effort

20%

1

MAX

备注

Across the built-in profiles, internal background best-effort clients of mclock ("scrub", "snap trim", and "pg deletion") are given lower reservations but no limits(MAX). This ensures that requests from such clients are able to complete quickly if there are no other competing operations.

Custom Profile

This profile gives users complete control over all the mclock configuration parameters. This profile should be used with caution and is meant for advanced users, who understand mclock and Ceph related configuration options.

mClock Built-in Profiles

When a built-in profile is enabled, the mClock scheduler calculates the low level mclock parameters [reservation, weight, limit] based on the profile enabled for each client type. The mclock parameters are calculated based on the max OSD capacity provided beforehand. As a result, the following mclock config parameters cannot be modified when using any of the built-in profiles:

The following Ceph options will not be modifiable by the user:

This is because the above options are internally modified by the mclock scheduler in order to maximize the impact of the set profile.

By default, the high_client_ops profile is enabled to ensure that a larger chunk of the bandwidth allocation goes to client ops. Background recovery ops are given lower allocation (and therefore take a longer time to complete). But there might be instances that necessitate giving higher allocations to either client ops or recovery ops. In order to deal with such a situation, the alternate built-in profiles may be enabled by following the steps mentioned in the next section.

If any mClock profile (including "custom") is active, the following Ceph config sleep options will be disabled,

The above sleep options are disabled to ensure that mclock scheduler is able to determine when to pick the next op from its operation queue and transfer it to the operation sequencer. This results in the desired QoS being provided across all its clients.

Steps to Enable mClock Profile

As already mentioned, the default mclock profile is set to high_client_ops. The other values for the built-in profiles include balanced and high_recovery_ops.

If there is a requirement to change the default profile, then the option osd_mclock_profile may be set during runtime by using the following command:

ceph config set osd.N osd_mclock_profile <value>

For example, to change the profile to allow faster recoveries on "osd.0", the following command can be used to switch to the high_recovery_ops profile:

ceph config set osd.0 osd_mclock_profile high_recovery_ops

备注

The custom profile is not recommended unless you are an advanced user.

And that's it! You are ready to run workloads on the cluster and check if the QoS requirements are being met.

OSD Capacity Determination (Automated)

The OSD capacity in terms of total IOPS is determined automatically during OSD initialization. This is achieved by running the OSD bench tool and overriding the default value of osd_mclock_max_capacity_iops_[hdd, ssd] option depending on the device type. No other action/input is expected from the user to set the OSD capacity.

备注

If you wish to manually benchmark OSD(s) or manually tune the Bluestore throttle parameters, see section Steps to Manually Benchmark an OSD (Optional).

You may verify the capacity of an OSD after the cluster is brought up by using the following command:

ceph config show osd.N osd_mclock_max_capacity_iops_[hdd, ssd]

For example, the following command shows the max capacity for "osd.0" on a Ceph node whose underlying device type is SSD:

ceph config show osd.0 osd_mclock_max_capacity_iops_ssd

Steps to Manually Benchmark an OSD (Optional)

备注

These steps are only necessary if you want to override the OSD capacity already determined automatically during OSD initialization. Otherwise, you may skip this section entirely.

小技巧

If you have already determined the benchmark data and wish to manually override the max osd capacity for an OSD, you may skip to section Specifying Max OSD Capacity.

Any existing benchmarking tool can be used for this purpose. In this case, the steps use the Ceph OSD Bench command described in the next section. Regardless of the tool/command used, the steps outlined further below remain the same.

As already described in the QoS Based on mClock section, the number of shards and the bluestore's throttle parameters have an impact on the mclock op queues. Therefore, it is critical to set these values carefully in order to maximize the impact of the mclock scheduler.

Number of Operational Shards

We recommend using the default number of shards as defined by the configuration options osd_op_num_shards, osd_op_num_shards_hdd, and osd_op_num_shards_ssd. In general, a lower number of shards will increase the impact of the mclock queues.

Bluestore Throttle Parameters

We recommend using the default values as defined by bluestore_throttle_bytes and bluestore_throttle_deferred_bytes. But these parameters may also be determined during the benchmarking phase as described below.

OSD Bench Command Syntax

The OSD Subsystem section describes the OSD bench command. The syntax used for benchmarking is shown below :

ceph tell osd.N bench [TOTAL_BYTES] [BYTES_PER_WRITE] [OBJ_SIZE] [NUM_OBJS]

where,

  • TOTAL_BYTES: Total number of bytes to write

  • BYTES_PER_WRITE: Block size per write

  • OBJ_SIZE: Bytes per object

  • NUM_OBJS: Number of objects to write

Benchmarking Test Steps Using OSD Bench

The steps below use the default shards and detail the steps used to determine the correct bluestore throttle values (optional).

  1. Bring up your Ceph cluster and login to the Ceph node hosting the OSDs that you wish to benchmark.

  2. Run a simple 4KiB random write workload on an OSD using the following commands:

    备注

    Note that before running the test, caches must be cleared to get an accurate measurement.

    For example, if you are running the benchmark test on osd.0, run the following commands:

    ceph tell osd.0 cache drop

    ceph tell osd.0 bench 12288000 4096 4194304 100

  3. Note the overall throughput(IOPS) obtained from the output of the osd bench command. This value is the baseline throughput(IOPS) when the default bluestore throttle options are in effect.

  4. If the intent is to determine the bluestore throttle values for your environment, then set the two options, bluestore_throttle_bytes and bluestore_throttle_deferred_bytes to 32 KiB(32768 Bytes) each to begin with. Otherwise, you may skip to the next section.

  5. Run the 4KiB random write test as before using OSD bench.

  6. Note the overall throughput from the output and compare the value against the baseline throughput recorded in step 3.

  7. If the throughput doesn't match with the baseline, increment the bluestore throttle options by 2x and repeat steps 5 through 7 until the obtained throughput is very close to the baseline value.

For example, during benchmarking on a machine with NVMe SSDs, a value of 256 KiB for both bluestore throttle and deferred bytes was determined to maximize the impact of mclock. For HDDs, the corresponding value was 40 MiB, where the overall throughput was roughly equal to the baseline throughput. Note that in general for HDDs, the bluestore throttle values are expected to be higher when compared to SSDs.

Specifying Max OSD Capacity

The steps in this section may be performed only if you want to override the max osd capacity automatically set during OSD initialization. The option osd_mclock_max_capacity_iops_[hdd, ssd] for an OSD can be set by running the following command:

ceph config set osd.N osd_mclock_max_capacity_iops_[hdd,ssd] <value>

For example, the following command sets the max capacity for a specific OSD (say "osd.0") whose underlying device type is HDD to 350 IOPS:

ceph config set osd.0 osd_mclock_max_capacity_iops_hdd 350

Alternatively, you may specify the max capacity for OSDs within the Ceph configuration file under the respective [osd.N] section. See Config file section names for more details.

mClock Config Options

osd_mclock_profile

This sets the type of mclock profile to use for providing QoS based on operations belonging to different classes (background recovery, scrub, snaptrim, client op, osd subop). Once a built-in profile is enabled, the lower level mclock resource control parameters [reservation, weight, limit] and some Ceph configuration parameters are set transparently. Note that the above does not apply for the custom profile.

type

str

default

high_client_ops

valid choices

  • balanced

  • high_recovery_ops

  • high_client_ops

  • custom

see also

osd_op_queue

osd_mclock_max_capacity_iops_hdd

Max IOPS capacity (at 4KiB block size) to consider per OSD (for rotational media)

type

float

default

315.0

osd_mclock_max_capacity_iops_ssd

Max IOPS capacity (at 4KiB block size) to consider per OSD (for solid state media)

type

float

default

21500.0

osd_mclock_cost_per_io_usec

Cost per IO in microseconds to consider per OSD (overrides _ssd and _hdd if non-zero)

type

float

default

0.0

osd_mclock_cost_per_io_usec_hdd

Cost per IO in microseconds to consider per OSD (for rotational media)

type

float

default

25000.0

osd_mclock_cost_per_io_usec_ssd

Cost per IO in microseconds to consider per OSD (for solid state media)

type

float

default

50.0

osd_mclock_cost_per_byte_usec

Cost per byte in microseconds to consider per OSD (overrides _ssd and _hdd if non-zero)

type

float

default

0.0

osd_mclock_cost_per_byte_usec_hdd

Cost per byte in microseconds to consider per OSD (for rotational media)

type

float

default

5.2

osd_mclock_cost_per_byte_usec_ssd

Cost per byte in microseconds to consider per OSD (for solid state media)

type

float

default

0.011

osd_mclock_force_run_benchmark_on_init

Force run the OSD benchmark on OSD initialization/boot-up

type

bool

default

false

see also

osd_mclock_max_capacity_iops_hdd, osd_mclock_max_capacity_iops_ssd

osd_mclock_skip_benchmark

Skip the OSD benchmark on OSD initialization/boot-up

type

bool

default

false

see also

osd_mclock_max_capacity_iops_hdd, osd_mclock_max_capacity_iops_ssd