How to directly download a file into HDFS?

Question

Is there any way to download a file directly into HDFS?

Answer 1

Yes, please use curl as follows:

curl -sS https://www.google.com/robots.txt | hadoop fs -put - /robots.txt

It downloads the robots.txt file and put it directly to /robots.txt on your HDFS. You can check it by running the following command afterward:

hadoop fs -ls /