java.io.IOException: Incompatible clusterIDs

Question

I am installing Hadoop 2.7.2 (1 master NN -1 second NN-3 datanode) and cannot start the datanodes!!! After trouble shouting the logs (see below), the fatal error is due to ClusterID mismatch... easy! just change the IDs. WRONG... when I check my VERSION files on the NameNode and the DataNodes they are identical..

So the question is simple: INTO the log file --> Where the ClusterID of the NameNode is coming From????

LOG FILE:

WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /home/hduser/mydata/hdfs/datanode: namenode clusterID = **CID-8e09ff25-80fb-4834-878b-f23b3deb62d0**; datanode clusterID = **CID-cd85e59a-ed4a-4516-b2ef-67e213cfa2a1**
org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to master/172.XX.XX.XX:9000. Exiting.
java.io.IOException: All specified directories are failed to load.
atorg.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
atorg.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1358)
atorg.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1323)
atorg.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
atorg.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
atorg.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
at java.lang.Thread.run(Thread.java:745)
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to master/172.XX.XX.XX:9000
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode

COPY of THE VERSION FILE

the master

storageID=DS-f72f5710-a869-489d-9f52-40dadc659937
clusterID=CID-cd85e59a-ed4a-4516-b2ef-67e213cfa2a1
cTime=0
datanodeUuid=54bc8b80-b84f-4893-8b96-36568acc5d4b
storageType=DATA_NODE
layoutVersion=-56

THE DataNode

storageID=DS-f72f5710-a869-489d-9f52-40dadc659937
clusterID=CID-cd85e59a-ed4a-4516-b2ef-67e213cfa2a1
cTime=0
datanodeUuid=54bc8b80-b84f-4893-8b96-36568acc5d4b
storageType=DATA_NODE
layoutVersion=-56

lucTiber · Accepted Answer · 2016-01-31 18:19:47Z

13

Just to summarize (and close) this issue, I would like to share how I fixed this issue.

On the MASTER and the 2nd Namenode the Namenode VERSION file is under ~/.../namenode/current/VERSION.

BUT for DATANODES the path is different. it should look something like this ~/.../datanode/current/VERSION

ClusterIDs between the 2 VERSION files should be identical

Hope it helps!

answered Jan 31, 2016 at 18:19

lucTiber

4411 gold badge4 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

srinivas amara · Accepted Answer · 2017-10-05 12:45:00Z

9

I also faced the same issue while installing 2.7.2. Data node is not coming up. Error shown in the datanode log file is

java.io.IOException: Incompatible clusterIDs in /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode: namenode clusterID = CID-XXX; datanode clusterID = CID-YYY

What i have done is

HADOOP_DIR/bin/hadoop namenode -format -clusterID CID-YYY

(No quotes required for cluster id)

answered Oct 5, 2017 at 12:45

srinivas amara

1652 silver badges10 bronze badges

1 Comment

Harshit H Over a year ago

This FIXED the error for me. Seems like we need more info on why this happens.

KayV · Accepted Answer · 2017-05-19 09:11:54Z

3

Just to add one more thing.

First, stop the dfs and delete the namenode and datanode directory/folders as specified in the hfs-site.xml.

And after that go to the ../namenode/current/VERSION file and copy the clusterId and replate the clusterID in ../datanode/current/VERSION file with the previously copied clusterID.

answered May 19, 2017 at 9:11

KayV

14k17 gold badges106 silver badges156 bronze badges

Collectives™ on Stack Overflow

java.io.IOException: Incompatible clusterIDs

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related