Logging and Debugging
Typically, when you add debugging to your Ceph configuration, you do so at
runtime. You can also add Ceph debug logging to your Ceph configuration file if
you are encountering issues when starting your cluster. You may view Ceph log
files under /var/log/ceph
(the default location).
小技巧
When debug output slows down your system, the latency can hide race conditions.
Logging is resource intensive. If you are encountering a problem in a specific area of your cluster, enable logging for that area of the cluster. For example, if your OSDs are running fine, but your metadata servers are not, you should start by enabling debug logging for the specific metadata server instance(s) giving you trouble. Enable logging for each subsystem as needed.
重要
Verbose logging can generate over 1GB of data per hour. If your OS disk reaches its capacity, the node will stop working.
If you enable or increase the rate of Ceph logging, ensure that you have sufficient disk space on your OS disk. See Accelerating Log Rotation for details on rotating log files. When your system is running well, remove unnecessary debugging settings to ensure your cluster runs optimally. Logging debug output messages is relatively slow, and a waste of resources when operating your cluster.
See Subsystem, Log and Debug Settings for details on available settings.
Runtime
If you would like to see the configuration settings at runtime, you must log in to a host with a running daemon and execute the following:
ceph daemon {daemon-name} config show | less
For example,:
ceph daemon osd.0 config show | less
To activate Ceph's debugging output (i.e., dout()
) at runtime, use the
ceph tell
command to inject arguments into the runtime configuration:
ceph tell {daemon-type}.{daemon id or *} config set {name} {value}
Replace {daemon-type}
with one of osd
, mon
or mds
. You may apply
the runtime setting to all daemons of a particular type with *
, or specify
a specific daemon's ID. For example, to increase
debug logging for a ceph-osd
daemon named osd.0
, execute the following:
ceph tell osd.0 config set debug_osd 0/5
The ceph tell
command goes through the monitors. If you cannot bind to the
monitor, you can still make the change by logging into the host of the daemon
whose configuration you'd like to change using ceph daemon
.
For example:
sudo ceph daemon osd.0 config set debug_osd 0/5
See Subsystem, Log and Debug Settings for details on available settings.
Boot Time
To activate Ceph's debugging output (i.e., dout()
) at boot time, you must
add settings to your Ceph configuration file. Subsystems common to each daemon
may be set under [global]
in your configuration file. Subsystems for
particular daemons are set under the daemon section in your configuration file
(e.g., [mon]
, [osd]
, [mds]
). For example
[global]
debug ms = 1/5
[mon]
debug mon = 20
debug paxos = 1/5
debug auth = 2
[osd]
debug osd = 1/5
debug filestore = 1/5
debug journal = 1
debug monc = 5/20
[mds]
debug mds = 1
debug mds balancer = 1
See Subsystem, Log and Debug Settings for details.
Accelerating Log Rotation
If your OS disk is relatively full, you can accelerate log rotation by modifying
the Ceph log rotation file at /etc/logrotate.d/ceph
. Add a size setting
after the rotation frequency to accelerate log rotation (via cronjob) if your
logs exceed the size setting. For example, the default setting looks like
this:
rotate 7
weekly
compress
sharedscripts
Modify it by adding a size
setting.
rotate 7
weekly
size 500M
compress
sharedscripts
Then, start the crontab editor for your user space.
crontab -e
Finally, add an entry to check the etc/logrotate.d/ceph
file.
30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1
The preceding example checks the etc/logrotate.d/ceph
file every 30 minutes.
Valgrind
Debugging may also require you to track down memory and threading issues.
You can run a single daemon, a type of daemon, or the whole cluster with
Valgrind. You should only use Valgrind when developing or debugging Ceph.
Valgrind is computationally expensive, and will slow down your system otherwise.
Valgrind messages are logged to stderr
.
Subsystem, Log and Debug Settings
In most cases, you will enable debug logging output via subsystems.
Ceph Subsystems
Each subsystem has a logging level for its output logs, and for its logs
in-memory. You may set different values for each of these subsystems by setting
a log file level and a memory level for debug logging. Ceph's logging levels
operate on a scale of 1
to 20
, where 1
is terse and 20
is
verbose 1 . In general, the logs in-memory are not sent to the output log unless:
a fatal signal is raised or
an
assert
in source code is triggered orupon requested. Please consult document on admin socket for more details.
A debug logging setting can take a single value for the log level and the
memory level, which sets them both as the same value. For example, if you
specify debug ms = 5
, Ceph will treat it as a log level and a memory level
of 5
. You may also specify them separately. The first setting is the log
level, and the second setting is the memory level. You must separate them with
a forward slash (/). For example, if you want to set the ms
subsystem's
debug logging level to 1
and its memory level to 5
, you would specify it
as debug ms = 1/5
. For example:
debug {subsystem} = {log-level}/{memory-level}
#for example
debug mds balancer = 1/20
The following table provides a list of Ceph subsystems and their default log and memory levels. Once you complete your logging efforts, restore the subsystems to their default level or to a level suitable for normal operations.
Subsystem |
Log Level |
Memory Level |
---|---|---|
|
0 |
5 |
|
0 |
1 |
|
0 |
1 |
|
1 |
1 |
|
1 |
5 |
|
1 |
5 |
|
1 |
5 |
|
1 |
5 |
|
1 |
5 |
|
1 |
5 |
|
0 |
1 |
|
0 |
1 |
|
0 |
1 |
|
0 |
1 |
|
0 |
1 |
|
0 |
5 |
|
0 |
5 |
|
0 |
5 |
|
0 |
5 |
|
0 |
5 |
|
0 |
5 |
|
0 |
5 |
|
1 |
5 |
|
0 |
5 |
|
0 |
5 |
|
1 |
3 |
|
1 |
3 |
|
0 |
5 |
|
1 |
5 |
|
0 |
10 |
|
1 |
5 |
|
0 |
5 |
|
1 |
5 |
|
1 |
5 |
|
1 |
1 |
|
1 |
1 |
|
1 |
5 |
|
1 |
5 |
|
1 |
5 |
|
1 |
5 |
|
1 |
10 |
|
1 |
5 |
|
1 |
5 |
|
1 |
1 |
|
0 |
0 |
|
1 |
5 |
|
1 |
5 |
|
1 |
5 |
|
1 |
3 |
|
1 |
5 |
|
4 |
5 |
|
4 |
5 |
|
4 |
5 |
|
1 |
5 |
|
1 |
5 |
|
1 |
5 |
|
1 |
5 |
|
1 |
5 |
Logging Settings
Logging and debugging settings are not required in a Ceph configuration file, but you may override default settings as needed. Ceph supports the following settings:
- log_file
The location of the logging file for your cluster.
- type
str
- see also
log_to_file
,log_to_stderr
,err_to_stderr
,log_to_syslog
,err_to_syslog
- log_max_new
The maximum number of new log files.
- type
int
- default
1000
- see also
- log_max_recent
The purpose of this option is to log at a higher debug level only to the in-memory buffer, and write out the detailed log messages only if there is a crash. Only log entries below the lower log level will be written unconditionally to the log. For example, debug_osd=1/5 will write everything <= 1 to the log unconditionally but keep entries at levels 2-5 in memory. If there is a seg fault or assertion failure, all entries will be dumped to the log.
- type
int
- default
500
- log_to_file
Determines if logging messages should appear in a file.
- type
bool
- default
true
- see also
- log_to_stderr
Determines if logging messages should appear in
stderr
.- type
bool
- default
true
- err_to_stderr
Determines if error messages should appear in
stderr
.- type
bool
- default
false
- log_to_syslog
Determines if logging messages should appear in
syslog
.- type
bool
- default
false
- err_to_syslog
Determines if error messages should appear in
syslog
.- type
bool
- default
false
- log_flush_on_exit
Determines if Ceph should flush the log files after exit.
- type
bool
- default
false
- clog_to_monitors
Determines if
clog
messages should be sent to monitors.- type
str
- default
default=true
- clog_to_syslog
Determines if
clog
messages should be sent to syslog.- type
str
- default
false
- mon_cluster_log_to_syslog
Determines if the cluster log should be output to the syslog.
- type
str
- default
default=false
- mon_cluster_log_file
The locations of the cluster's log files. There are two channels in Ceph:
cluster
andaudit
. This option represents a mapping from channels to log files, where the log entries of that channel are sent to. Thedefault
entry is a fallback mapping for channels not explicitly specified. So, the following default setting will send cluster log to$cluster.log
, and send audit log to$cluster.audit.log
, where$cluster
will be replaced with the actual cluster name.- type
str
- default
default=/var/log/ceph/$cluster.$channel.log cluster=/var/log/ceph/$cluster.log
- see also
mon_cluster_log_to_file
OSD
- osd_debug_drop_ping_probability
N/A
- type
float
- default
0.0
- osd_debug_drop_ping_duration
N/A
- type
int
- default
0
Filestore
- filestore_debug_omap_check
Debugging check on synchronization. This is an expensive operation.
- type
bool
- default
false
MDS
RADOS Gateway
- 1
there are levels >20 in some rare cases and that they are extremely verbose.