[-- MARK -- Wed Mar 30 09:00:00 2005] CMAN: node cl032a has been removed from the cluster : Missed too many heartbeats SM: process_reply invalid id=6764 nodeid=4294967295 SM: process_reply invalid id=6764 nodeid=4294967295 SM: process_reply invalid id=6764 nodeid=4294967295 GFS: Trying to join cluster "lock_dlm", "gfs_cluster:stripefs" SM: process_reply invalid id=6765 nodeid=4294967295 GFS: fsid=gfs_cluster:stripefs.1: Joined cluster. Now mounting FS... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Trying to acquire journal lock... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Looking at journal... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Done SM: process_reply invalid id=6765 nodeid=4294967295 SM: process_reply invalid id=6765 nodeid=4294967295 SM: process_reply invalid id=5553 nodeid=4294967295 SM: process_reply invalid id=5553 nodeid=4294967295 SM: process_reply invalid id=5553 nodeid=4294967295 SM: process_reply invalid id=5553 nodeid=4294967295 SM: process_reply invalid id=5554 nodeid=4294967295 SM: process_reply invalid id=5554 nodeid=4294967295 GFS: Trying to join cluster "lock_dlm", "gfs_cluster:stripefs" SM: process_reply invalid id=5554 nodeid=4294967295 SM: process_reply invalid id=5554 nodeid=4294967295 GFS: fsid=gfs_cluster:stripefs.1: Joined cluster. Now mounting FS... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Trying to acquire journal lock... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Looking at journal... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Done SM: process_reply invalid id=6770 nodeid=4294967295 SM: process_reply invalid id=6770 nodeid=4294967295 SM: process_reply invalid id=6770 nodeid=4294967295 SM: process_reply invalid id=6771 nodeid=4294967295 SM: process_reply invalid id=6771 nodeid=4294967295 SM: process_reply invalid id=6771 nodeid=4294967295 SM: process_reply invalid id=5561 nodeid=4294967295 SM: process_reply invalid id=5561 nodeid=4294967295 SM: process_reply invalid id=5561 nodeid=4294967295 SM: process_reply invalid id=5561 nodeid=4294967295 SM: process_reply invalid id=5562 nodeid=4294967295 SM: process_reply invalid id=5562 nodeid=4294967295 SM: process_reply invalid id=5562 nodeid=4294967295 SM: process_reply invalid id=5562 nodeid=4294967295 [-- MARK -- Wed Mar 30 09:15:00 2005] SM: process_reply invalid id=6778 nodeid=4294967295 SM: process_reply invalid id=6778 nodeid=4294967295 SM: process_reply invalid id=6778 nodeid=4294967295 SM: process_reply invalid id=6779 nodeid=4294967295 SM: process_reply invalid id=6779 nodeid=4294967295 SM: process_reply invalid id=6779 nodeid=4294967295 GFS: Trying to join cluster "lock_dlm", "gfs_cluster:stripefs" SM: process_reply invalid id=5569 nodeid=4294967295 GFS: fsid=gfs_cluster:stripefs.1: Joined cluster. Now mounting FS... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Trying to acquire journal lock... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Looking at journal... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Done SM: process_reply invalid id=5569 nodeid=4294967295 SM: process_reply invalid id=5570 nodeid=4294967295 SM: process_reply invalid id=5570 nodeid=4294967295 SM: process_reply invalid id=5570 nodeid=4294967295 SM: process_reply invalid id=5570 nodeid=4294967295 SM: process_reply invalid id=6786 nodeid=4294967295 SM: process_reply invalid id=6786 nodeid=4294967295 SM: process_reply invalid id=6786 nodeid=4294967295 SM: process_reply invalid id=6787 nodeid=4294967295 SM: process_reply invalid id=6787 nodeid=4294967295 SM: process_reply invalid id=6787 nodeid=4294967295 GFS: Trying to join cluster "lock_dlm", "gfs_cluster:stripefs" SM: process_reply invalid id=5577 nodeid=4294967295 GFS: fsid=gfs_cluster:stripefs.1: Joined cluster. Now mounting FS... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Trying to acquire journal lock... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Looking at journal... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Done SM: process_reply invalid id=5577 nodeid=4294967295 SM: process_reply invalid id=5578 nodeid=4294967295 SM: process_reply invalid id=5578 nodeid=4294967295 SM: process_reply invalid id=5578 nodeid=4294967295 SM: process_reply invalid id=5578 nodeid=4294967295 SM: process_reply invalid id=6794 nodeid=4294967295 SM: process_reply invalid id=6794 nodeid=4294967295 SM: process_reply invalid id=6794 nodeid=4294967295 GFS: Trying to join cluster "lock_dlm", "gfs_cluster:stripefs" SM: process_reply invalid id=6795 nodeid=4294967295 GFS: fsid=gfs_cluster:stripefs.1: Joined cluster. Now mounting FS... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Trying to acquire journal lock... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Looking at journal... GFS: fsid=gfs_cluster:stripefs.1: jid=1: Done SM: process_reply invalid id=6795 nodeid=4294967295 SM: process_reply invalid id=6795 nodeid=4294967295 SM: process_reply invalid id=5583 nodeid=4294967295 SM: process_reply invalid id=5583 nodeid=4294967295 SM: process_reply invalid id=5583 nodeid=4294967295 SM: process_reply invalid id=5583 nodeid=4294967295 SM: process_reply invalid id=5584 nodeid=4294967295 SM: process_reply invalid id=5584 nodeid=4294967295 SM: process_reply invalid id=5584 nodeid=4294967295 SM: process_reply invalid id=5584 nodeid=4294967295 [-- MARK -- Wed Mar 30 09:30:00 2005] [-- MARK -- Wed Mar 30 09:45:00 2005] [-- MARK -- Wed Mar 30 10:00:00 2005] [-- MARK -- Wed Mar 30 10:15:00 2005] [-- MARK -- Wed Mar 30 10:30:00 2005] [-- MARK -- Wed Mar 30 10:45:00 2005] [-- MARK -- Wed Mar 30 11:00:00 2005] [-- MARK -- Wed Mar 30 11:15:00 2005] [-- MARK -- Wed Mar 30 11:30:00 2005] [-- MARK -- Wed Mar 30 11:45:00 2005] [-- MARK -- Wed Mar 30 12:00:00 2005] [-- MARK -- Wed Mar 30 12:15:00 2005] [-- MARK -- Wed Mar 30 12:30:00 2005] [-- MARK -- Wed Mar 30 12:45:00 2005] [-- MARK -- Wed Mar 30 13:00:00 2005] [-- MARK -- Wed Mar 30 13:15:00 2005] [-- MARK -- Wed Mar 30 13:30:00 2005] [-- MARK -- Wed Mar 30 13:45:00 2005] cat /proc/cluster/status > /tmp/cl030.status[root@cl031 root]# cat /proc/cluster/st[root@cl031 root]# cat /proc/cluster/sta[root@cl031 root]# cat /proc/cluster/stat[root@cl031 root]# cat /proc/cluster/status > /tmp/cl030.status[root@cl031 root]# cat /proc/cluster/statu[root@cl031 root]# cat /proc/cluster/status[root@cl031 root]# cat /proc/cluster/status [root@cl031 root]# cat /proc/cluster/status >[root@cl031 root]# cat /proc/cluster/status > [root@cl031 root]# cat /proc/cluster/status > /[root@cl031 root]# cat /proc/cluster/status > /t[root@cl031 root]# cat /proc/cluster/status > /tm[root@cl031 root]# cat /proc/cluster/status > /tmp[root@cl031 root]# cat /proc/cluster/status > /tmp/[root@cl031 root]# cat /proc/cluster/status > /tmp/c[root@cl031 root]# cat /proc/cluster/status > /tmp/cl[root@cl031 root]# cat /proc/cluster/status > /tmp/cl0[root@cl031 root]# cat /proc/cluster/status > /tmp/cl031.status You have new mail in /var/spool/mail/root [root@cl031 root]# cat /proc/cluster/status > /tmp/cl031.status[root@cl031 root]# [root@cl031 root]# cat [root@cl031 root]# cat /[root@cl031 root]# cat /proc[root@cl031 root]# cat /proc/[root@cl031 root]# cat /proc/cluster[root@cl031 root]# cat /proc/cluster/ > /tmp/cl031.statusn > /tmp/cl031.statuso > /tmp/cl031.statusd > /tmp/cl031.statuse > /tmp/cl031.statuss > /tmp/cl031.status[root@cl031 root]# cat /proc/cluster/nodes > /tmp/cl031.statunodes [root@cl031 root]# cat /proc/cluster/nodes > /tmp/cl031.nodes[root@cl031 root]# [root@cl031 root]# c[root@cl031 root]# ca[root@cl031 root]# cat[root@cl031 root]# cat [root@cl031 root]# cat /[root@cl031 root]# cat /p[root@cl031 root]# cat /pr[root@cl031 root]# cat /pro[root@cl031 root]# cat /proc[root@cl031 root]# cat /proc/[root@cl031 root]# cat /proc/c[root@cl031 root]# cat /proc/cl[root@cl031 root]# cat /proc/clu[root@cl031 root]# cat /proc/clus[root@cl031 root]# cat /proc/clust[root@cl031 root]# cat /proc/cluste[root@cl031 root]# cat /proc/cluster[root@cl031 root]# cat /proc/cluster/[root@cl031 root]# cat /proc/cluster/n[root@cl031 root]# cat /proc/cluster/no[root@cl031 root]# cat /proc/cluster/ > /tmp/cl031.nodess > /tmp/cl031.nodesm > /tmp/cl031.nodes_ > /tmp/cl031.nodesdebug > /tmp/cl031.nodes[root@cl031 root]# cat /proc/cluster/sm_debug > /tmp/cl031.node[root@cl031 root]# cat /proc/cluster/sm_debug > /tmp/cl031.[root@cl031 root]# cat /proc/cluster/sm_debug > /tmp/cl031.sm_debug [root@cl031 root]# cat /proc/cluster/sm_debug > /tmp/cl031.sm_debug[root@cl031 root]# [root@cl031 root]# cat [root@cl031 root]# cat /[root@cl031 root]# cat /p[root@cl031 root]# cat /pr[root@cl031 root]# cat /pro[root@cl031 root]# cat /proc[root@cl031 root]# cat /proc/[root@cl031 root]# cat /proc/c[root@cl031 root]# cat /proc/cl[root@cl031 root]# cat /proc/clu[root@cl031 root]# cat /proc/clus[root@cl031 root]# cat /proc/clust[root@cl031 root]# cat /proc/cluste[root@cl031 root]# cat /proc/cluster[root@cl031 root]# cat /proc/cluster/[root@cl031 root]# cat /proc/cluster/s[root@cl031 root]# cat /proc/cluster/sm[root@cl031 root]# cat /proc/cluster/sm_ > /tmp/cl031.sm_debugs > /tmp/cl031.sm_debuge > /tmp/cl031.sm_debugr > /tmp/cl031.sm_debugv > /tmp/cl031.sm_debugi > /tmp/cl031.sm_debugc > /tmp/cl031.sm_debuge > /tmp/cl031.sm_debugs > /tmp/cl031.sm_debug[root@cl031 root]# cat /proc/cluster/services [root@cl031 root]# cat /proc/cluster/services > [root@cl031 root]# cat /proc/cluster/services > /[root@cl031 root]# cat /proc/cluster/services > /tmp[root@cl031 root]# cat /proc/cluster/services > /tmp/[root@cl031 root]# cat /proc/cluster/services > /tmp/cl031[root@cl031 root]# cat /proc/cluster/services > /tmp/cl031.[root@cl031 root]# cat /proc/cluster/services > /tmp/cl031.services [root@cl031 root]# cat /proc/cluster/services > /tmp/cl031.services[root@cl031 root]# [root@cl031 root]# cat [root@cl031 root]# cat /[root@cl031 root]# cat /proc[root@cl031 root]# cat /proc/[root@cl031 root]# cat /proc/cluster[root@cl031 root]# cat /proc/cluster/ > /tmp/cl031.servicesd > /tmp/cl031.servicesl > /tmp/cl031.servicesm > /tmp/cl031.services_ > /tmp/cl031.servicesd > /tmp/cl031.servicese > /tmp/cl031.servicesb > /tmp/cl031.servicesu > /tmp/cl031.servicesg > /tmp/cl031.servicest > /tmp/cl031.services > /tmp/cl031.services[root@cl031 root]# cat /proc/cluster/dlm_debug > /tmp/cl031.service[root@cl031 root]# cat /proc/cluster/dlm_debug > /tmp/cl031.dlm_debug [root@cl031 root]# /tail /varl/log/messages[root@cl031 root]# tail /var/log/messages[root@cl031 root]# Mar 30 09:21:04 cl031 kernel: GFS: fsid=gfs_cluster:stripefs.1: jid=1: Looking at journal... Mar 30 09:21:04 cl031 kernel: GFS: fsid=gfs_cluster:stripefs.1: jid=1: Done Mar 30 09:21:04 cl031 sshd(pam_unix)[7454]: session closed for user root Mar 30 09:21:04 cl031 kernel: SM: process_reply invalid id=6795 nodeid=4294967295 Mar 30 09:21:05 cl031 sshd(pam_unix)[7491]: session opened for user root by (uid=0) Mar 30 09:21:05 cl031 sshd(pam_unix)[7491]: session closed for user root Mar 30 09:21:05 cl031 kernel: SM: process_reply invalid id=6795 nodeid=4294967295 Mar 30 09:21:06 cl031 kernel: SM: process_reply invalid id=5583 nodeid=4294967295 Mar 30 09:21:09 cl031 last message repeated 3 times Mar 30 09:21:10 cl031 kernel: SM: process_reply invalid id=5584 nodeid=4294967295 [root@cl031 root]# [-- MARK -- Wed Mar 30 14:00:00 2005] [-- MARK -- Wed Mar 30 14:15:00 2005] [-- MARK -- Wed Mar 30 14:30:00 2005] [-- MARK -- Wed Mar 30 14:45:00 2005] [-- MARK -- Wed Mar 30 15:00:00 2005] [-- MARK -- Wed Mar 30 15:15:00 2005] [-- MARK -- Wed Mar 30 15:30:00 2005] ls -ltr /tmp total 36 drwxr-xr-x 2 root root 16384 Sep 3 2004 lost+found srwxrwxrwx 1 wnn wnn 0 Mar 24 15:09 jd_sockV4 -rw-r--r-- 1 root root 265 Mar 30 13:49 cl031.status -rw-r--r-- 1 root root 110 Mar 30 13:49 cl031.nodes -rw-r--r-- 1 root root 256 Mar 30 13:49 cl031.sm_debug -rw-r--r-- 1 root root 312 Mar 30 13:49 cl031.services -rw-r--r-- 1 root root 1024 Mar 30 13:49 cl031.dlm_debug [root@cl031 root]# [root@cl031 root]# [-- MARK -- Wed Mar 30 15:45:00 2005] [-- MARK -- Wed Mar 30 16:00:00 2005] [-- MARK -- Wed Mar 30 16:15:00 2005] [-- MARK -- Wed Mar 30 16:30:00 2005] [-- MARK -- Wed Mar 30 16:45:00 2005]