How to delete files from the HDFS?
Your problem is inside of the basis of HDFS. In HDFS (and in many other file systems) physical deleting of files isn't the fastest operations. As HDFS is distributed file system and usually replicate at least 3 replicas on different servers of the deleted file then each replica (which may consist of many blocks on different hard drives) must be deleted in the background after your request to delete the file.
Official documentation of Hadoop tells us the following:
The deletion of a file causes the blocks associated with the file to be freed. Note that there could be an appreciable time delay between the time a file is deleted by a user and the time of the corresponding increase in free space in HDFS.
what works for me :
hadoop fs -rmr -R <your Directory>
Try hadoop fs -rm -R URI
-R option deletes the directory and any content under it recursively.
You can use
hdfs dfs -rm -R /path/to/HDFS/file
since hadoop dfs
has been deprecated.