您当前的位置: 首页   健康 > 快看点丨记一次线上bug:crontab 被意外清空
快看点丨记一次线上bug:crontab 被意外清空

时间:2023-07-05 05:26:39    来源:博客园

记一次线上bug:crontab 被意外清空目录记一次线上bug:crontab 被意外清空问题概述问题排查问题复现其他测试总结又是一次难忘的经历。问题概述

同事反馈,某台服务器的crontab被清空了.

看了cron运行日志来看,问题出在这里:


(资料图)

Jul  3 10:01:24 10-10-65-235 crontab[19333]: (root) REPLACE (root)

这一时刻被替换掉了,后续就没有任务执行了。

然后紧急恢复备份,并从日志中检查遗漏项。

问题排查

定位出问题后,就从history中找执行的操作。

可从history中没有看到 crontab -r 的操作被执行。

后在Google搜索问题找到一些类似的情况,是由于远程登录、加空格之类的导致的(https://cloud.tencent.com/developer/article/2222953)。

我看了后,就发现这一定是我的锅了,并且受影响的还有其他两台设备。

问题复现

我的操作是从远程服务器通过 ssh直接执行的命令,卡住之后 ctrl+C中断了操作,于是任务被清空。

复现流程:

创建任务(保证任务列表中有内容,不为空)。

╭─ ~/cmd ▓▒░·······························································░▒▓ ✔  10:29:52 ─╮╰─ crontab -l                                                                              ─╯# Minute   Hour   Day of Month       Month          Day of Week        who      Command# (0-59)  (0-23)     (1-31)    (1-12 or Jan-Dec)  (0-6 or Sun-Sat)20 17 * * 1-5 open "https://tengyun.qianxin-inc.cn/home/workspace/worklog"╭─ ~/cmd ▓▒░·······························································░▒▓ ✔  10:29:54 ─╮╰─                                                                                         ─╯

使用ssh命令执行能卡住的命令,如crontabcrontab -

╭─ ~/cmd ▓▒░·······························································░▒▓ ✔  10:29:54 ─╮╰─ ssh nanshao@127.0.0.1 crontab                                                           ─╯

命令卡住,此刻查看 crontab -l还是有的。

╭─ ~/cmd ▓▒░·······························································░▒▓ ✔  10:30:59 ─╮╰─ crontab -l                                                                              ─╯# Minute   Hour   Day of Month       Month          Day of Week        who      Command# (0-59)  (0-23)     (1-31)    (1-12 or Jan-Dec)  (0-6 or Sun-Sat)20 17 * * 1-5 open "https://tengyun.qianxin-inc.cn/home/workspace/worklog"╭─ ~/cmd ▓▒░·······························································░▒▓ ✔  10:31:02 ─╮╰─                                                                                         ─╯

使用ctrl+C结束掉后检查任务列表。

╭─ ~/cmd ▓▒░·······························································░▒▓ ✔  10:29:54 ─╮╰─ ssh nanshao@127.0.0.1 crontab                                                           ─╯^C%                                                                                           ╭─ ~/cmd ▓▒░··························································░▒▓ ✔  55s  10:31:30 ─╮╰─ crontab -l                                                                              ─╯╭─ ~/cmd ▓▒░·······························································░▒▓ ✔  10:31:33 ─╮╰─                                                                                         ─╯

问题复现成功。

其他测试后又分别在本机直接测试了这些指令,均没有清空任务列表。只有在ssh操作时,会被清空。ssh操作中 ssh XXX@XXX bash -c "crontab -l"该命令也会被卡住。总结

产生的操作,弄明白了,但是原理还是没搞明白。

避坑方法:

定时备份。在卡住时,不要中断,先备份,在停止掉。又是一次难忘的经历。
标签:

X 关闭

X 关闭