spark 中写hdfs的异常解决

报错信息：

 

ERROR DFSClient: Failed to close inode 16604
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.15.201:50010,DS-00e134aa-a7c3-44be-b1e2-0c75ccb7c4bc,DISK]], original=[DatanodeInfoWithStorage[192.168.15.201:50010,DS-00e134aa-a7c3-44be-b1e2-0c75ccb7c4bc,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via "dfs.client.block.write.replace-datanode-on-failure.policy" in its configuration.
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:925)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:988)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1156)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454)

按照网上说明该了hdfs-site.xml 设置了

1. dfs.client.block.write.replace-datanode-on-failure.policy : NEVER  
2. dfs.client.block.write.replace-datanode-on-failure.enable : true 

不过没有用，要改客户端。

我是spark-shell 里面执行的 ，最后在scala环境中执行下面两句搞定

scala> System.setProperty("dfs.client.block.write.replace-datanode-on-failure.enable", "true")
res1: String = true

scala> System.setProperty("dfs.client.block.write.replace-datanode-on-failure.policy","NEVER")
res3: String = never

ps：我在so。com里面搜索出来的，baidu里面冗余无效信息太多