Pool, PG and CRUSH Config Reference

When you create pools and set the number of placement groups (PGs) for each, Ceph uses default values when you don't specifically override the defaults. We recommend overriding some of the defaults. Specifically, we recommend setting a pool's replica size and overriding the default number of placement groups. You can specifically set these values when running pool commands. You can also override the defaults by adding new ones in the [global] section of your Ceph configuration file.

[global]

	# By default, Ceph makes 3 replicas of RADOS objects. If you want to maintain four
	# copies of an object the default value--a primary copy and three replica
	# copies--reset the default values as shown in 'osd_pool_default_size'.
	# If you want to allow Ceph to accept an I/O operation to a degraded PG,
	# set 'osd_pool_default_min_size' to a number less than the
	# 'osd pool default size' value.

	osd_pool_default_size = 3  # Write an object 3 times.
	osd_pool_default_min size = 2 # Accept an I/O operation to a PG that has two copies of an object.

	# Ensure you have a realistic number of placement groups. We recommend
	# approximately 100 per OSD. E.g., total number of OSDs multiplied by 100
	# divided by the number of replicas (i.e., osd pool default size). So for
	# 10 OSDs and osd pool default size = 4, we'd recommend approximately
	# (100 * 10) / 4 = 250.
        # always use the nearest power of 2

	osd_pool_default_pg_num = 256
	osd_pool_default_pgp_num = 256
mon_max_pool_pg_num

The maximum number of placement groups per pool.

type

uint

default

64Ki

mon_pg_stuck_threshold

Number of seconds after which PGs can be considered as being stuck.

type

int

default

1 minute

mon_pg_warn_min_per_osd

Raise HEALTH_WARN if the average number of PGs per in OSD is under this number. A non-positive number disables this.

type

uint

default

0

mon_pg_warn_min_objects

Do not warn if the total number of RADOS objects in cluster is below this number

type

int

default

10000

mon_pg_warn_min_pool_objects

Do not warn on pools whose RADOS object count is below this number

type

int

default

1000

mon_pg_check_down_all_threshold

Percentage threshold of down OSDs above which we check all PGs for stale ones.

type

float

default

0.5

mon_pg_warn_max_object_skew

Raise HEALTH_WARN if the average RADOS object count per PG of any pool is greater than mon_pg_warn_max_object_skew times the average RADOS object count per PG of all pools. Zero or a non-positive number disables this. Note that this option applies to ceph-mgr daemons.

type

float

default

10.0

mon_delta_reset_interval

Seconds of inactivity before we reset the PG delta to 0. We keep track of the delta of the used space of each pool, so, for example, it would be easier for us to understand the progress of recovery or the performance of cache tier. But if there's no activity reported for a certain pool, we just reset the history of deltas of that pool.

type

float

default

10.0

osd_crush_chooseleaf_type

The bucket type to use for chooseleaf in a CRUSH rule. Uses ordinal rank rather than name.

type

int

default

1

osd_crush_initial_weight

The initial CRUSH weight for newly added OSDs. The default value of this option is the size of a newly added OSD in TB. By default, the initial CRUSH weight for a newly added OSD is set to its device size in TB. See Weighting Bucket Items for details.

type

float

default

-1.0

osd_pool_default_crush_rule

The default CRUSH rule to use when creating a replicated pool. The default value of -1 means "pick the rule with the lowest numerical ID and use that". This is to make pool creation work in the absence of rule 0.

type

int

default

-1

osd_pool_erasure_code_stripe_unit

Sets the default size, in bytes, of a chunk of an object stripe for erasure coded pools. Every object of size S will be stored as N stripes, with each data chunk receiving stripe unit bytes. Each stripe of N * stripe unit bytes will be encoded/decoded individually. This option can is overridden by the stripe_unit setting in an erasure code profile.

type

size

default

4Ki

osd_pool_default_size

Sets the number of replicas for objects in the pool. The default value is the same as ceph osd pool set {pool-name} size {size}.

type

uint

default

3

allowed range

[0, 10]

osd_pool_default_min_size

Sets the minimum number of written replicas for objects in the pool in order to acknowledge an I/O operation to the client. If minimum is not met, Ceph will not acknowledge the I/O to the client, which may result in data loss. This setting ensures a minimum number of replicas when operating in degraded mode. The default value is 0 which means no particular minimum. If 0, minimum is size - (size / 2).

type

uint

default

0

allowed range

[0, 255]

see also

osd_pool_default_size

osd_pool_default_pg_num

The default number of placement groups for a pool. The default value is the same as pg_num with mkpool.

type

uint

default

32

see also

osd_pool_default_pg_autoscale_mode

osd_pool_default_pgp_num

The default number of placement groups for placement for a pool. The default value is the same as pgp_num with mkpool. PG and PGP should be equal (for now).

type

uint

default

0

see also

osd_pool_default_pg_num

osd_pool_default_pg_autoscale_mode

With default value on, the autoscaler starts a new pool with 1 pg, unless the user specifies the pg_num.

type

str

default

on

valid choices
  • off

  • warn

  • on

osd_pool_default_flags

The default flags for new pools.

type

int

default

0

osd_max_pgls

The maximum number of placement groups to list. A client requesting a large number can tie up the Ceph OSD Daemon.

type

uint

default

1Ki

osd_min_pg_log_entries

The minimum number of placement group logs to maintain when trimming log files.

type

uint

default

250

see also

osd_max_pg_log_entries, osd_pg_log_dups_tracked, osd_target_pg_log_entries_per_osd

osd_max_pg_log_entries

The maximum number of placement group logs to maintain when trimming log files.

type

uint

default

10000

see also

osd_min_pg_log_entries, osd_pg_log_dups_tracked, osd_target_pg_log_entries_per_osd

osd_default_data_pool_replay_window

The time (in seconds) for an OSD to wait for a client to replay a request.

type

int

default

45

osd_max_pg_per_osd_hard_ratio

The ratio of number of PGs per OSD allowed by the cluster before the OSD refuses to create new PGs. An OSD stops creating new PGs if the number of PGs it serves exceeds osd_max_pg_per_osd_hard_ratio * mon_max_pg_per_osd.

type

float

default

3.0

min

1

see also

mon_max_pg_per_osd