Could Only Be Replicated To ...
A common message people see is "could only be replicated to 0 nodes, instead of ...".
What does this mean? It means that the Block Replication mechanism of HDFS could not make any copies of a file it wanted to create. This can be caused by
No DataNode instances being up and running. Action: look at the servers, see if the processes are running.
Your DataNode instances have no hard disk space in their configured data directories. Action: look at the dfs.data.dir list in the node configurations, verify that at least one of the directories exists, and is writeable by the user running the Hadoop processes. Then look at the logs.
Your DataNode instances have run out of space. Look at the disk capacity via the Namenode web pages. Delete old files. Compress under-used files. Buy more disks for existing servers (if there is room), upgrade the existing servers to bigger drives, or add some more servers.
- The reserved space for a DN (as set in dfs.datanode.du.reserved is greater than the remaining free space, so the DN thinks it has no free space
- You may also get this message due to permissions, eg if JT can not create jobtracker.info on startup.
This is not a problem in Hadoop, it is a problem in your cluster that you are going to have to fix on your own. Sorry.