使用 systemd 限制系统资源的使用

<p>在基于 Linux-3.x 内核版本的很多发行版都提供了&nbsp;<a href="https://www.freedesktop.org/wiki/Software/systemd/" target="_blank">Systemd</a>&nbsp;来管理系统和服务. 同时也将 cgroup 功能加到了 slice, scope 和 service 三个单元中, 详见&nbsp;<a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/resource_management_guide/index#sec-Default_Cgroup_Hierarchies" target="_blank">sec-Default_Cgroup_Hierarchies</a>. 基于这些特性我们可以很方便的通过 systemd 来限制服务或者进程对系统资源的使用, 这在单主机多服务的场景下会很有用. 下面则以 MySQL 服务为例介绍如何使用 systemd 限制资源的使用, 其它服务的限制和此等同.</p>

<p>以 Centos7 系统为例, 从&nbsp;<a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/resource_management_guide/index" target="_blank">redhat-resource-control</a>&nbsp;的文档来看, 官方建议通过 service 来实现资源的限制, 所以这里我们增加可以带端口参数(如果单台主机有多个 MySQL 实例的话)启动的 mysql 服务:</p>

<pre>
<code class="language-ini"># cat /usr/lib/systemd/system/mysql@.service                                   
[Unit]
Description=MySQL Server node%i

[Service]
Type=forking
Environment="PORT_ARGS=%I"
PermissionsStartOnly=true
ExecStart=/usr/local/mysqlnode/bin/node ${PORT_ARGS} start
ExecStop=/usr/local/mysqlnode/bin/node ${PORT_ARGS} stop

[Install]
WantedBy=multi-user.target
</code></pre>

<p>上述服务以 fork 方式启动服务,&nbsp;<code>@</code>&nbsp;符号之后的端口号即为相应的端口参数, 启动后查看对应服务状态:</p>

<pre>
<code class="language-bash"># systemctl start mysql@3327 
# systemctl status mysql@3327
● mysql@3327.service - MySQL Server node3327
   Loaded: loaded (/usr/lib/systemd/system/mysql@.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-12-07 21:31:32 CST; 4s ago
  Process: 58084 ExecStart=/usr/local/mysqlnode/bin/node ${PORT_ARGS} start (code=exited, status=0/SUCCESS)
 Main PID: 58159 (mysqld_safe)
   CGroup: /system.slice/system-mysql.slice/mysql@3327.service
           ├─58159 /bin/sh /opt/Percona-Server-5.6.38-rel83.0-Linux.x86_64.ssl101//bin/mysqld_safe --defaults-file=/export/mysql/node3327/my.node.cnf
           ├─59489 /opt/Percona-Server-5.6.38-rel83.0-Linux.x86_64.ssl101/bin/mysqld --defaults-file=/export/mysql/node3327/my.node.cnf --basedir=/opt/Percona-Server-5.6.38-rel83.0-Linux.x86_64.ssl101 --datadir=/export/mysql/node3327/...
           └─59490 logger -t mysqld-3327 -p daemon.error
</code></pre>

<p>通过 systemd 增加 cgroup 限制，可以一次设置单项属性值, 也可以一次设置多项：</p>

<pre>
<code class="language-bash"># systemctl set-property mysql@3327.service MemoryLimit=5G       # 5G 内存
# systemctl set-property mysql@3327.service CPUQuota=150%        # 150% cpu 使用率
# systemctl set-property mysql@3327.service BlockIOWeight=1000   # IO 权重

# systemctl status mysql@3327
● mysql@3327.service - MySQL Server node3327
   Loaded: loaded (/usr/lib/systemd/system/mysql@.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mysql@3327.service.d
           └─50-MemoryLimit.conf
   Active: active (running) since Fri 2018-12-07 21:31:32 CST; 1min 18s ago
  Process: 58084 ExecStart=/usr/local/mysqlnode/bin/node ${PORT_ARGS} start (code=exited, status=0/SUCCESS)
 Main PID: 58159 (mysqld_safe)
   Memory: 1.3M (limit: 5G)            # 内存限制 5G
   CGroup: /system.slice/system-mysql.slice/mysql@3327.service
           ├─58159 /bin/sh /opt/Percona-Server-5.6.38-rel83.0-Linux.x86_64.ssl101//bin/mysqld_safe --defaults-file=/export/mysql/node3327/my.node.cnf
           ├─59489 /opt/Percona-Server-5.6.38-rel83.0-Linux.x86_64.ssl101/bin/mysqld --defaults-file=/export/mysql/node3327/my.node.cnf --basedir=/opt/Percona-Server-5.6.38-rel83.0-Linux.x86_64.ssl101 --datadir=/export/mysql/node3327/...
           └─59490 logger -t mysqld-3327 -p daemon.error
</code></pre>

<p>更多属性参见&nbsp;<code>man systemd.resource-control</code>, 不过一些参数没有对应的属性, 需要手动单独设置, 比如设置单独某个服务的&nbsp;<code>memory + swap</code>&nbsp;限制, 可以使用以下命令将限制的字节数写到对应的参数文件中:</p>

<pre>
<code class="language-bash">echo xxxxxx &gt; memory.memsw.limit_in_bytes
</code></pre>

<p>查看 MySQL 进程 cgroup 信息, 可以看到 memory, blkid, cpuacct 三个条目对应的信息:</p>

<pre>
<code class="language-bash"># cat /proc/59489/cgroup 
11:memory:/system.slice/system-mysql.slice/mysql@3327.service
10:devices:/system.slice/system-mysql.slice
9:cpuset:/
8:blkio:/system.slice/system-mysql.slice/mysql@3327.service
7:perf_event:/
6:hugetlb:/
5:freezer:/
4:cpuacct,cpu:/system.slice/system-mysql.slice
3:pids:/system.slice/system-mysql.slice
2:net_prio,net_cls:/
1:name=systemd:/system.slice/system-mysql.slice/mysql@3327.service
</code></pre>

<p>查看进程相关的限制值:</p>

<pre>
<code class="language-bash"># cat /sys/fs/cgroup/memory/system.slice/system-mysql.slice/mysql\@3327.service/memory.limit_in_bytes 
5368709120

# cat /sys/fs/cgroup/blkio/system.slice/system-mysql.slice/mysql\@3327.service/blkio.weight
1000

# cat /sys/fs/cgroup/cpu/system.slice/system-mysql.slice/mysql\@3327.service/cpu.cfs_quota_us 
150000
</code></pre>

<p>这里仅以不同的 CPUQuota 为例说明, 如下所示可以看到, MySQL 进程在不同&nbsp;<code>CPUQuota</code>&nbsp;限制下的不同表现,&nbsp;<code>benchyou</code>&nbsp;的压测结果也相差较大.</p>

<h4>限制进程 CPUQuota 为 150% 的情况下:</h4>

<pre>
<code class="language-bash"># top -p 59489
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                           
 59489 mysql     20   0 3412052 1.468g   8784 S 149.5  2.3   7:08.22 mysqld

# benchyou --oltp-tables-count=256 --read-threads=30 --update-threads 8 --write-threads 6 --delete-threads=5 --mysql-table-engine=innodb ..
time            thds              tps     wtps    rtps    rio    rio/op   wio    wio/op    rMB     rKB/op    wMB     wKB/op   cpu/op  freeMB  cacheMB   w-rsp(ms)  r-rsp(ms)    total-number
[13s]        [r:30,w:6,u:8,d:5]  3323     304     3019    0      0.00     0      0.00      0.00    0.00      0.00    0.00     0.00    0       0         62.56      9.92         40765

time            thds              tps     wtps    rtps    rio    rio/op   wio    wio/op    rMB     rKB/op    wMB     wKB/op   cpu/op  freeMB  cacheMB   w-rsp(ms)  r-rsp(ms)    total-number
[14s]        [r:30,w:6,u:8,d:5]  3319     325     2994    0      0.00     0      0.00      0.00    0.00      0.00    0.00     0.00    0       0         57.21      9.99         44084
</code></pre>

<h4>限制进程 CPUQuota 为 400% 情况下:</h4>

<pre>
<code class="language-bash"># top -p 59489
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                           
 59489 mysql     20   0 3416472 1.474g   8796 S 400.0  2.4   7:47.42 mysqld

# benchyou --oltp-tables-count=256 --read-threads=30 --update-threads 8 --write-threads 6 --delete-threads=5 --mysql-table-engine=innodb ..
time            thds              tps     wtps    rtps    rio    rio/op   wio    wio/op    rMB     rKB/op    wMB     wKB/op   cpu/op  freeMB  cacheMB   w-rsp(ms)  r-rsp(ms)    total-number
[10s]        [r:30,w:6,u:8,d:5]  10547    967     9580    0      0.00     0      0.00      0.00    0.00      0.00    0.00     0.00    0       0         19.49      3.12         96270

time            thds              tps     wtps    rtps    rio    rio/op   wio    wio/op    rMB     rKB/op    wMB     wKB/op   cpu/op  freeMB  cacheMB   w-rsp(ms)  r-rsp(ms)    total-number
[11s]        [r:30,w:6,u:8,d:5]  10829    1036    9793    0      0.00     0      0.00      0.00    0.00      0.00    0.00     0.00    0       0         18.23      3.04         107099
</code></pre>

<h2>会话级别限制</h2>

<p>如果没有以 service 启动, 可以按照 pid 查看会话级别的信息:</p>

<pre>
<code class="language-bash"># systemctl status 303   # 303 为 mysql 进程 id
● session-31208.scope - Session 31208 of user root
</code></pre>

<p><code>session-31208.scope</code>&nbsp;即为会话级别的信息, 一个会话可能包含多个 MySQL 进程, 使用上述的&nbsp;<code>systemctl set-property session-31208.scope XXXX</code>&nbsp;即可对整个会话的进程进行相关的限制. 当然使用这种方式即是对多个进程总体使用情况的限制.</p>

<p>如果要单独限制 1 个进程 id, 则需要借助&nbsp;<code>libcgroup</code>&nbsp;提供的工具, 具体的使用同 Centos6 的设置, 不过 Centos7 已经弃用了&nbsp;<code>libgcgroup-tools</code>, 不再建议使用, 如果要对单独进程进行限制, 最好使用&nbsp;<code>service</code>&nbsp;进行启动. 另外从&nbsp;<code>systemd</code>&nbsp;提供的&nbsp;<a href="https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/" target="_blank">ControlGroupInterface</a>&nbsp;来看, 其提供的控制选项还不够精细, 比如没有 memsw, cpuset 等设置的属性, 这种情况下如果有限制的需求, 还需要靠和以前一样的 libcgroup-tool 方式来设置, 比如以下设置, 限制 pid 为 59489 的进程仅使用 cpu 0 ~ 5, mem 可以是 cpu node0 或 node1, cgclassify 没有指定&nbsp;<code>--sticky</code>&nbsp;选项的话则 tasks 会包含指定进程的子进程信息:</p>

<pre>
<code class="language-bash"># cgcreate -g  cpuset:mysql@3327.service
# cgset -r cpuset.cpus=0-5 mysql@3327.service
# cgset -r cpuset.mems=0-1 mysql@3327.service
# cgclassify -g cpuset:mysql@3327.service 59489
</code></pre>

<p>当然也可以使用红帽知识库的方式在启动进程的时候就限制相应的 cpuset 等选项, 详见&nbsp;<a href="https://access.redhat.com/solutions/1445073" target="_blank">redhat-1445073</a></p>

<p><a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/resource_management_guide/index" target="_blank">resource_management_guide</a><br />
<a href="https://access.redhat.com/solutions/1445073" target="_blank">redhat-1445073</a><br />
<a href="https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html" target="_blank">systemd.resource-control</a><br />
<a href="https://www.freedesktop.org/software/systemd/man/systemctl.html" target="_blank">systemctl</a></p>

使用 systemd 限制系统资源的使用

标签云

网站信息

关于本站