公司服务器网络割接,服务器换了新的IP地址,Hadoop服务器在换完网络后需要修改IP恢复使用,但是遇到一个问题.

起因是这样子的,由于服务器前面新加了交换机和路由器,网关掩码和IP都需要修改,但是我的集群是三台Hadoop节点,通过/etc/hosts中添加解析(如: 192.168.1.10 h1)这种,并且修改IP前没有停止Hadoop服务,就直接断网开始割接.

网络割接完成后修改IP重启了网卡,然后我使用命令

for i in `jps |awk '{print $1}';do kill -9 $i ;done`

杀死了所有进程,确认无误后开始依次启动.

我的另一个集群就是这么做的,简直不要太完美的成功了,但是在这里遇到了问题

在我启动完zookeeper后启动Hadoop,打开页面访问了 8088端口正常,但是我继续访问50070 端口发现大小全部都是0
也就是hdfs没有启动成功,然后我就开始看日志(大部分日志放在文件结尾),看到这么一条,然后就停止了.

java.io.IOException: NameNode is not formatted.

他居然提示我格式化namenode?
百度了一下,大部分是小白搭建集群忘记格式化,执行 format namenode,但是我绝不会是这种情况.
在百度和bing的海洋里遨游,我发现所有的解决方式都是格式化namenode 后重启,但我的是线上环境,里面存了好多客户信息,动不动就格式化还怎么玩

解决方法

怀疑是在停止的时候,或者割接网络的时候对Hadoop还有写入数据操作,导致数据结构损坏,通过寻找大佬,大佬早年看Hadoop原理的时候,看到过可能的解决方式.

我的数据是放在 /data 目录下,这个可以再配置文件中找到

# 我的目录结构
[root@h1 data]# cd /data/
[root@h1 data]# ls
hadoop  hdfs  namenode  yarn

然后进入Hadoop目录

[root@h1 data]# cd hadoop/tmp/dfs/namesecondary
[root@h1 dfs]# ls
current  current.tar.gz  in_use.lock

这里有个current的目录,里面存储的是备份后的Hadoop namenode节点的数据
示例:

[root@h1 namesecondary]# ls current|head 
edits_0000000000000749853-0000000000000749948
edits_0000000000000749949-0000000000000749986
edits_0000000000000749987-0000000000000750115
edits_0000000000000750116-0000000000000750173
edits_0000000000000750174-0000000000000750301
edits_0000000000000750302-0000000000000750333
edits_0000000000000750334-0000000000000750455
edits_0000000000000750456-0000000000000750513
edits_0000000000000750514-0000000000000750590
edits_0000000000000750591-0000000000000750681

不确定数据是否可用,我先将namenode数据备份,直接复制目录,然后将这里的所有文件移动到了namenode目录下覆盖,再重启Hadoop

完美启动成功.

再次百度,感觉这是个类似MySQL的 binlog,又有点类似MySQL主从,Hadoop会先将写入数据加载到内存,然后修改,修改完成后重新写入磁盘,并且将操作复制给current一份,作为备份.具体我也就不太懂了…因为是后来补的可能会错过写细节,大体是这么个步骤.记录下这个问题

Hadoop 报错部分日志

2020-05-29 09:42:37,362 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2020-05-29 09:42:37,609 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2020-05-29 09:42:37,609 INFO org.apache.hadoop.util.GSet: VM type       = 64-bit
2020-05-29 09:42:37,609 INFO org.apache.hadoop.util.GSet: 1.0% max memory 889 MB = 8.9 MB
2020-05-29 09:42:37,609 INFO org.apache.hadoop.util.GSet: capacity      = 2^20 = 1048576 entries
2020-05-29 09:42:37,610 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: ACLs enabled? false
2020-05-29 09:42:37,610 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: XAttrs enabled? true
2020-05-29 09:42:37,610 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: Maximum size of an xattr: 16384
2020-05-29 09:42:37,610 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2020-05-29 09:42:37,617 INFO org.apache.hadoop.util.GSet: Computing capacity for map cachedBlocks
2020-05-29 09:42:37,617 INFO org.apache.hadoop.util.GSet: VM type       = 64-bit
2020-05-29 09:42:37,617 INFO org.apache.hadoop.util.GSet: 0.25% max memory 889 MB = 2.2 MB
2020-05-29 09:42:37,617 INFO org.apache.hadoop.util.GSet: capacity      = 2^18 = 262144 entries
2020-05-29 09:42:37,618 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2020-05-29 09:42:37,618 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2020-05-29 09:42:37,618 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
2020-05-29 09:42:37,621 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2020-05-29 09:42:37,621 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2020-05-29 09:42:37,621 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2020-05-29 09:42:37,624 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2020-05-29 09:42:37,625 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2020-05-29 09:42:37,626 INFO org.apache.hadoop.util.GSet: Computing capacity for map NameNodeRetryCache
2020-05-29 09:42:37,626 INFO org.apache.hadoop.util.GSet: VM type       = 64-bit
2020-05-29 09:42:37,626 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
2020-05-29 09:42:37,627 INFO org.apache.hadoop.util.GSet: capacity      = 2^15 = 32768 entries
2020-05-29 09:42:37,636 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /data/namenode/in_use.lock acquired by nodename 4270@h1
2020-05-29 09:42:37,638 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:243)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:984)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:686)at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:586)at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:646)at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:820)at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:804)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1516)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1582)
2020-05-29 09:42:37,640 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:50070
2020-05-29 09:42:37,741 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2020-05-29 09:42:37,741 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2020-05-29 09:42:37,742 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2020-05-29 09:42:37,742 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: NameNode is not formatted.at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:243)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:984)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:686)at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:586)at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:646)at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:820)at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:804)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1516)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1582)
2020-05-29 09:42:37,743 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2020-05-29 09:42:37,744 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at h1/20.88.2.114
************************************************************/