2020-07-15 09:30:27.93934416.0MbinfoStartLoop: disable caching
2020-07-15 09:30:27.96308916.0MbinfoStartLoop: check if we should take any measures before the actual scheduling loop
2020-07-15 09:30:27.96489916.0Mbinfopcntl functions found, installing signal handler
2020-07-15 09:30:27.96503916.0MbinfoStartLoop: check lock and status
2020-07-15 09:30:27.97229116.0MbinfoStartLoop: disconnected from DB
2020-07-15 09:30:27.97240216.0Mbinfo my PID: 7843
2020-07-15 09:30:27.97248116.0Mbinfoprepare infinite loop
2020-07-15 09:30:27.97256216.0MbsuccessLoop started, lock acquired
2020-07-15 09:30:27.97267816.0Mbinfowoke-up, at 1594794627.972658s
2020-07-15 09:30:27.97276016.0Mbinfo > loopDuration: 60.000114s
2020-07-15 09:30:27.97283516.0Mbinfo > reconnect
2020-07-15 09:30:28.37372316.0Mbsuccess -> run SendCalendarEventsEmails
2020-07-15 09:30:29.14652816.0Mbsuccess -> run ExportData
2020-07-15 09:30:29.49782716.0Mbsuccess -> run ReportsDataPopulate
2020-07-15 09:30:29.86283416.0Mbinfosleep duration: 31.137187s (until 1594794661.00s)
2020-07-15 09:31:01.00025516.0Mbinfowoke-up, at 1594794661.000214s
2020-07-15 09:31:01.00041616.0Mbinfo > loopDuration: 33.027556s
2020-07-15 09:31:01.00049816.0Mbinfo > reconnect
2020-07-15 09:31:01.04173116.0Mbinfosleep duration: 59.958299s (until 1594794721.00s)
2020-07-15 09:32:01.00024716.0Mbinfowoke-up, at 1594794721.000207s
2020-07-15 09:32:01.00041116.0Mbinfo > loopDuration: 59.999993s
2020-07-15 09:32:01.00049716.0Mbinfo > reconnect
2020-07-15 09:32:01.04046316.0Mbinfosleep duration: 59.959565s (until 1594794781.00s)
2020-07-15 09:33:01.00026616.0Mbinfowoke-up, at 1594794781.000227s
2020-07-15 09:33:01.00042716.0Mbinfo > loopDuration: 60.000020s
2020-07-15 09:33:01.00051416.0Mbinfo > reconnect
2020-07-15 09:33:01.04310216.0Mbinfosleep duration: 59.956926s (until 1594794841.00s)
2020-07-15 09:34:01.00025816.0Mbinfowoke-up, at 1594794841.000216s
2020-07-15 09:34:01.00042816.0Mbinfo > loopDuration: 59.999989s
2020-07-15 09:34:01.00051816.0Mbinfo > reconnect
2020-07-15 09:34:01.04360816.0Mbinfosleep duration: 59.956420s (until 1594794901.00s)
2020-07-15 09:35:01.00026216.0Mbinfowoke-up, at 1594794901.000223s
2020-07-15 09:35:01.00043216.0Mbinfo > loopDuration: 60.000007s
2020-07-15 09:35:01.00051916.0Mbinfo > reconnect
2020-07-15 09:35:01.04081816.0Mbinfosleep duration: 59.959211s (until 1594794961.00s)
2020-07-15 09:36:01.00027716.0Mbinfowoke-up, at 1594794961.000223s
2020-07-15 09:36:01.00043516.0Mbinfo > loopDuration: 60.000000s
2020-07-15 09:36:01.00052116.0Mbinfo > reconnect
2020-07-15 09:36:01.02430716.0Mbsuccess -> run SendCalendarEventsEmails
2020-07-15 09:36:01.80572116.0Mbinfosleep duration: 59.194303s (until 1594795021.00s)
2020-07-15 09:37:01.00023316.0Mbinfowoke-up, at 1594795021.000204s
2020-07-15 09:37:01.00034416.0Mbinfo > loopDuration: 59.999981s
2020-07-15 09:37:01.00039916.0Mbinfo > reconnect
2020-07-15 09:37:01.04561516.0Mbinfosleep duration: 59.954414s (until 1594795081.00s)
2020-07-15 09:38:01.00026816.0Mbinfowoke-up, at 1594795081.000221s
2020-07-15 09:38:01.00042216.0Mbinfo > loopDuration: 60.000017s
2020-07-15 09:38:01.00050216.0Mbinfo > reconnect
2020-07-15 09:38:01.03819316.0Mbinfosleep duration: 59.961834s (until 1594795141.00s)
2020-07-15 09:39:01.00254616.0Mbinfowoke-up, at 1594795141.002513s
2020-07-15 09:39:01.00267616.0Mbinfo > loopDuration: 60.002292s
2020-07-15 09:39:01.00276516.0Mbinfo > reconnect
2020-07-15 09:39:01.04654716.0Mbinfosleep duration: 59.953482s (until 1594795201.00s)
2020-07-15 09:40:01.00026816.0Mbinfowoke-up, at 1594795201.000231s
2020-07-15 09:40:01.00042216.0Mbinfo > loopDuration: 59.997718s
2020-07-15 09:40:01.00051316.0Mbinfo > reconnect
2020-07-15 09:40:01.04536116.0Mbinfosleep duration: 59.954669s (until 1594795261.00s)
2020-07-15 09:41:01.00025016.0Mbinfowoke-up, at 1594795261.000207s
2020-07-15 09:41:01.00042216.0Mbinfo > loopDuration: 59.999976s
2020-07-15 09:41:01.00051416.0Mbinfo > reconnect
2020-07-15 09:41:01.03290516.0Mbsuccess -> run SendCalendarEventsEmails
2020-07-15 09:41:01.38177616.0Mbsuccess -> run ReportsDataPopulate
2020-07-15 09:41:01.68539316.0Mbinfosleep duration: 59.314630s (until 1594795321.00s)
2020-07-15 09:42:01.00024116.0Mbinfowoke-up, at 1594795321.000185s
2020-07-15 09:42:01.00040316.0Mbinfo > loopDuration: 59.999978s
2020-07-15 09:42:01.00050416.0Mbinfo > reconnect
2020-07-15 09:42:01.04999216.0Mbinfosleep duration: 59.950036s (until 1594795381.00s)
2020-07-15 09:43:01.00026916.0Mbinfowoke-up, at 1594795381.000231s
2020-07-15 09:43:01.00042516.0Mbinfo > loopDuration: 60.000046s
2020-07-15 09:43:01.00050916.0Mbinfo > reconnect
2020-07-15 09:43:01.03977616.0Mbinfosleep duration: 59.960254s (until 1594795441.00s)
2020-07-15 09:44:01.00119916.0Mbinfowoke-up, at 1594795441.000933s
2020-07-15 09:44:01.00131916.0Mbinfo > loopDuration: 60.000702s
2020-07-15 09:44:01.00139916.0Mbinfo > reconnect
2020-07-15 09:44:01.04774516.0Mbinfosleep duration: 59.952281s (until 1594795501.00s)
2020-07-15 09:44:51.71003116.0MbcriticalScript was forcefully stopped
2020-07-15 09:44:51.71018116.0Mbcritical > received SIGTERM(15)
2020-07-15 09:44:51.71025816.0Mbcritical > signinfo: array (
'signo' => 15,
'errno' => 0,
'code' => 0,
)
2020-07-15 09:44:51.71451016.0MbcriticalReconnected to the DB
2020-07-15 09:44:51.71460216.0MbcriticalLock status:
2020-07-15 09:44:51.71542616.0Mbcritical > isLocked: true, DB status: 'should-stop'
2020-07-15 09:44:51.72849416.0Mbcritical > isLockedReasons: array (
)
2020-07-15 09:44:51.73022116.0MbcriticalMark as should-restart, released raw lock
2020-07-15 09:44:51.73641416.0MbcriticalGathering as much system info now:
2020-07-15 09:44:51.77429516.0Mbcritical(ps aux) 2>&1: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 221644 6132 ? Ss May21 1:13 /sbin/init nomdmonddf nomdmonisw
root 2 0.0 0.0 0 0 ? S May21 0:01 [kthreadd]
root 3 0.0 0.0 0 0 ? S May21 0:06 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< May21 0:00 [kworker/0:0H]
root 7 0.0 0.0 0 0 ? S May21 17:51 [rcu_sched]
root 8 0.0 0.0 0 0 ? S May21 0:00 [rcu_bh]
root 9 0.0 0.0 0 0 ? S May21 0:01 [migration/0]
root 10 0.0 0.0 0 0 ? S May21 0:13 [watchdog/0]
root 11 0.0 0.0 0 0 ? S May21 0:12 [watchdog/1]
root 12 0.0 0.0 0 0 ? S May21 0:02 [migration/1]
root 13 0.0 0.0 0 0 ? S May21 0:24 [ksoftirqd/1]
root 15 0.0 0.0 0 0 ? S< May21 0:00 [kworker/1:0H]
root 16 0.0 0.0 0 0 ? S May21 0:12 [watchdog/2]
root 17 0.0 0.0 0 0 ? S May21 0:01 [migration/2]
root 18 0.0 0.0 0 0 ? S May21 0:14 [ksoftirqd/2]
root 20 0.0 0.0 0 0 ? S< May21 0:00 [kworker/2:0H]
root 21 0.0 0.0 0 0 ? S May21 0:11 [watchdog/3]
root 22 0.0 0.0 0 0 ? S May21 0:01 [migration/3]
root 23 0.0 0.0 0 0 ? S May21 0:13 [ksoftirqd/3]
root 25 0.0 0.0 0 0 ? S< May21 0:00 [kworker/3:0H]
root 26 0.0 0.0 0 0 ? S May21 0:00 [kdevtmpfs]
root 27 0.0 0.0 0 0 ? S< May21 0:00 [netns]
root 28 0.0 0.0 0 0 ? S< May21 0:00 [perf]
root 29 0.0 0.0 0 0 ? S May21 0:01 [khungtaskd]
root 30 0.0 0.0 0 0 ? S< May21 0:00 [writeback]
root 31 0.0 0.0 0 0 ? SN May21 0:00 [ksmd]
root 32 0.0 0.0 0 0 ? SN May21 2:45 [khugepaged]
root 33 0.0 0.0 0 0 ? S< May21 0:00 [crypto]
root 34 0.0 0.0 0 0 ? S< May21 0:00 [kintegrityd]
root 35 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 36 0.0 0.0 0 0 ? S< May21 0:00 [kblockd]
root 37 0.0 0.0 0 0 ? S< May21 0:00 [ata_sff]
root 38 0.0 0.0 0 0 ? S< May21 0:00 [md]
root 39 0.0 0.0 0 0 ? S< May21 0:00 [devfreq_wq]
root 46 0.0 0.0 0 0 ? S May21 5:20 [kswapd0]
root 47 0.0 0.0 0 0 ? S< May21 0:00 [vmstat]
root 48 0.0 0.0 0 0 ? S May21 0:00 [fsnotify_mark]
root 49 0.0 0.0 0 0 ? S May21 0:00 [ecryptfs-kthrea]
root 65 0.0 0.0 0 0 ? S< May21 0:00 [kthrotld]
root 66 0.0 0.0 0 0 ? S< May21 0:00 [acpi_thermal_pm]
root 67 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 68 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 69 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 70 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 71 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 72 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 73 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 74 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 80 0.0 0.0 0 0 ? S< May21 0:00 [ipv6_addrconf]
root 93 0.0 0.0 0 0 ? S< May21 0:00 [deferwq]
root 94 0.0 0.0 0 0 ? S< May21 0:00 [charger_manager]
root 137 0.0 0.0 0 0 ? S May21 0:00 [scsi_eh_0]
root 138 0.0 0.0 0 0 ? S< May21 0:00 [scsi_tmf_0]
root 139 0.0 0.0 0 0 ? S May21 0:00 [scsi_eh_1]
root 140 0.0 0.0 0 0 ? S< May21 0:00 [scsi_tmf_1]
root 141 0.0 0.0 0 0 ? S May21 0:00 [scsi_eh_2]
root 142 0.0 0.0 0 0 ? S< May21 0:00 [scsi_tmf_2]
root 143 0.0 0.0 0 0 ? S May21 0:00 [scsi_eh_3]
root 144 0.0 0.0 0 0 ? S< May21 0:00 [scsi_tmf_3]
root 145 0.0 0.0 0 0 ? S May21 0:00 [scsi_eh_4]
root 146 0.0 0.0 0 0 ? S< May21 0:00 [scsi_tmf_4]
root 147 0.0 0.0 0 0 ? S May21 0:00 [scsi_eh_5]
root 148 0.0 0.0 0 0 ? S< May21 0:00 [scsi_tmf_5]
root 155 0.0 0.0 0 0 ? S< May21 0:00 [kfd_process_wq]
root 156 0.0 0.0 0 0 ? S< May21 0:00 [ttm_swap]
root 157 0.0 0.0 0 0 ? S< May21 0:00 [radeon-crtc]
root 158 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 159 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 160 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 167 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 168 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 171 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 176 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 177 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 181 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 184 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 185 0.0 0.0 0 0 ? S< May21 0:00 [bioset]
root 186 0.0 0.0 0 0 ? S May21 28:30 [md1_raid1]
root 187 0.0 0.0 0 0 ? S May21 4:08 [md0_raid1]
root 213 0.0 0.0 0 0 ? S< May21 0:00 [raid5wq]
root 243 0.0 0.0 0 0 ? S< May21 9:34 [kworker/3:1H]
root 244 0.0 0.0 0 0 ? S< May21 0:49 [kworker/0:1H]
root 245 0.0 0.0 0 0 ? S< May21 0:34 [kworker/1:1H]
root 249 0.0 0.0 0 0 ? S May21 1:28 [jbd2/md0-8]
root 250 0.0 0.0 0 0 ? S< May21 0:00 [ext4-rsv-conver]
root 291 0.0 0.0 0 0 ? S May21 0:00 [kauditd]
root 307 0.0 0.5 196876 48172 ? S&1
www-data 8151 0.0 0.0 34380 2840 ? RN 09:44 0:00 ps aux
root 12244 0.0 0.0 0 0 ? S< Jun09 0:00 [cifsiod]
root 18712 0.0 0.1 361456 12604 ? S Jun22 1:20 /usr/sbin/smbd -D
www-data 25579 0.0 0.4 367052 40316 ? SN Jul10 0:28 php -f /srv/www/devel/gabriel.grosu/TINA3/testing/tina/disp.php module=Admin/CronJobs/Main action=startLoop mainCronJobLogId=208115
root 28042 0.0 0.0 0 0 ? S 05:03 0:00 [kworker/3:2]
www-data 28082 0.0 0.3 362824 27500 ? SN May22 8:52 php -f /srv/www/devel/staging/instances/1u7-ujodu/tina/disp.php module=Admin/CronJobs/Main action=startLoop mainCronJobLogId=7
root 28234 0.0 0.0 0 0 ? S 05:09 0:02 [kworker/2:1]
root 32221 0.0 0.0 0 0 ? S 06:39 0:01 [kworker/0:2]
2020-07-15 09:44:51.81560216.0Mbcritical(w) 2>&1: 09:44:51 up 54 days, 19:52, 0 users, load average: 0.45, 0.22, 0.19
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
2020-07-15 09:44:51.84814016.0Mbcritical(who -a) 2>&1: system boot May 21 13:52
LOGIN tty1 May 21 13:52 1521 id=tty1
run-level 5 May 21 13:53
pts/0 Jul 13 15:54 20860 id=ts/0 term=0 exit=0
pts/3 Jul 13 10:56 9176 id=ts/3 term=0 exit=0
2020-07-15 09:44:51.88932116.0Mbcritical(dmesg | tail -n 50) 2>&1: [3219868.130480] [] do_filp_open+0x91/0x100
[3219868.130484] [] ? __alloc_fd+0xc7/0x190
[3219868.130487] [] do_sys_open+0x138/0x2a0
[3219868.130490] [] SyS_openat+0x14/0x20
[3219868.130494] [] entry_SYSCALL_64_fastpath+0x22/0xcb
[3824575.525947] INFO: task apache2:12843 blocked for more than 120 seconds.
[3824575.526041] Not tainted 4.4.0-145-generic #171-Ubuntu
[3824575.526118] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[3824575.526233] apache2 D ffff88006ca7bc58 0 12843 1815 0x00000000
[3824575.526239] ffff88006ca7bc58 0000000000000001 ffff880216360e00 ffff880101b07000
[3824575.526243] ffff88006ca7c000 ffff880022306614 ffff880101b07000 00000000ffffffff
[3824575.526246] ffff880022306618 ffff88006ca7bc70 ffffffff8185dea5 ffff880022306610
[3824575.526250] Call Trace:
[3824575.526258] [] schedule+0x35/0x80
[3824575.526262] [] schedule_preempt_disabled+0xe/0x10
[3824575.526266] [] __mutex_lock_slowpath+0xb7/0x130
[3824575.526269] [] mutex_lock+0x1f/0x30
[3824575.526275] [] path_openat+0x512/0x1360
[3824575.526278] [] ? putname+0x54/0x60
[3824575.526282] [] do_filp_open+0x91/0x100
[3824575.526287] [] ? __alloc_fd+0xc7/0x190
[3824575.526291] [] do_sys_open+0x138/0x2a0
[3824575.526294] [] SyS_openat+0x14/0x20
[3824575.526298] [] entry_SYSCALL_64_fastpath+0x22/0xcb
[3841660.077634] md: data-check of RAID array md0
[3841660.077639] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[3841660.077641] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[3841660.077644] md: using 128k window, over a total of 39028608k.
[3841660.201189] md: delaying data-check of md1 until md0 has finished (they share one or more physical units)
[3841982.038597] md: md0: data-check done.
[3841982.229431] md: data-check of RAID array md1
[3841982.229436] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[3841982.229438] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[3841982.229441] md: using 128k window, over a total of 390494016k.
[3847263.908565] md: md1: data-check done.
[4101498.004235] e1000e: eth0 NIC Link is Down
[4101498.004395] br0: port 1(eth0) entered disabled state
[4101574.696682] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[4101574.697042] br0: port 1(eth0) entered forwarding state
[4101574.697060] br0: port 1(eth0) entered forwarding state
[4101578.515944] e1000e: eth0 NIC Link is Down
[4101578.516100] br0: port 1(eth0) entered disabled state
[4101592.697511] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[4101592.697871] br0: port 1(eth0) entered forwarding state
[4101592.697890] br0: port 1(eth0) entered forwarding state
[4101595.268714] e1000e: eth0 NIC Link is Down
[4101595.268870] br0: port 1(eth0) entered disabled state
[4101599.697835] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[4101599.698194] br0: port 1(eth0) entered forwarding state
[4101599.698213] br0: port 1(eth0) entered forwarding state