No response for my wget command

[Nipponho]

Hi,

Any kind linux experts, please help me.

I am following the book Hadoop for Dummies learning how to set up a VM. The book instructs us to go to root access and use wget to request for a repository file needed for Apache Bigtop installation.

I copied the command and it gave me a response saying scheme is missing. So I try again by typing inverted commas on the link address. This time round, there is no scheme is missing error message. But nothing happens and I am stuck. What did I do wrong?

[/URL][/IMG]

Thanks!

 

[davidktw]

Hi,

Any kind linux experts, please help me.

I am following the book Hadoop for Dummies learning how to set up a VM. The book instructs us to go to root access and use wget to request for a repository file needed for Apache Bigtop installation.

I copied the command and it gave me a response saying scheme is missing. So I try again by typing inverted commas on the link address. This time round, there is no scheme is missing error message. But nothing happens and I am stuck. What did I do wrong?

[/URL][/IMG]

Thanks!

The command should be

wget -O ... "http://www.apache.org/dist/bigtop/bigtop-0.7.0/repos/centos6/bigtop.repo"

However, in any case, the file is no longer there

Use your browser and access http://www.apache.org/dist/bigtop/

 

[localITguy]

Hi,

Any kind linux experts, please help me.

I am following the book Hadoop for Dummies learning how to set up a VM. The book instructs us to go to root access and use wget to request for a repository file needed for Apache Bigtop installation.

I copied the command and it gave me a response saying scheme is missing. So I try again by typing inverted commas on the link address. This time round, there is no scheme is missing error message. But nothing happens and I am stuck. What did I do wrong?

[/URL][/IMG]

Thanks!

try

wget https://www.apache.org/dist/bigtop/bigtop-1.1.0/repos/centos6/bigtop.repo

works for me.

 

[Nipponho]

The command should be

wget -O ... "http://www.apache.org/dist/bigtop/bigtop-0.7.0/repos/centos6/bigtop.repo"

However, in any case, the file is no longer there

Use your browser and access http://www.apache.org/dist/bigtop/

Hi David, thanks for your reply! So stupid of me not to check the existence of the file

 

[Nipponho]

still cannot lay

try

wget https://www.apache.org/dist/bigtop/bigtop-1.1.0/repos/centos6/bigtop.repo

works for me.

HI LocalITGuy,

Thanks for your reply!

The book’s instruction was :
wget -O /etc/yum.repos.d/bigtop.repo \ http://www.apache.org/dist/bigtop/bigtop-0.7.0/repos/centos6/bigtop.repo

Since the 0.7.0 file is no longer there as David pointed out, I modified it :
wget -O /etc/yum.repos.d/bigtop.repo \ https://www.apache.org/dist/bigtop/bigtop-1.1.0/repos/centos6/bigtop.repo
It still gave me the Scheme is missing error message. I tried adding inverted comma, it gave no response at all.

I tried your method :
wget https://www.apache.org/dist/bigtop/bigtop-1.1.0/repos/centos6/bigtop.repo

It gave me another error message :
Certificate common name “*.openoffice.org” doesn’t match requested host name www.apache.org

How?

This book does not live up to its claims of being "for dummies".

 

[Nipponho]

like this means successful?

The command should be

wget -O ... "http://www.apache.org/dist/bigtop/bigtop-0.7.0/repos/centos6/bigtop.repo"

http://www.apache.org/dist/bigtop/

Hi again David. Seems like your code can! No error message. Like this means succeed already? Sorry first time using linux, seeing stars.

[/URL][/IMG]

 

[Extremism]

-

bumping this thread for my post count to reach 10. lol.

 

[ykgoh]

Hi again David. Seems like your code can! No error message. Like this means succeed already? Sorry first time using linux, seeing stars.

[/URL][/IMG]

You see HTTP RESPONSE 200 OK means something got downloaded.

But you saved it to "..."

Are you blindly copying and pasting commands without knowing what they are supposed to do?

 

[Nipponho]

You see HTTP RESPONSE 200 OK means something got downloaded.

But you saved it to "..."

Are you blindly copying and pasting commands without knowing what they are supposed to do?

Thanks! Great! Yeah, hehe, i am struggling, lots of sweat now, please forgive me, i have zero knowledge about linux prior to embarking on this hadoop thing.

 

[ykgoh]

OK anyway since you have the repo file downloaded. You probably want to rename to something easier using the mv command.

mv '...' abc.repo

Then add this repo as one of package source for yum. See this for more details:

https://www.centos.org/docs/5/html/yum/sn-using-repositories.html

The repo file you have is simply a text configuration find pointing to a URL where Hadoop BigTop packages can be downloaded and installed.

Probably you need to run yum to download and install the packages after adding the repo. See your original dummies guide.

 

[Nipponho]

OK anyway since you have the repo file downloaded. You probably want to rename to something easier using the mv command.

mv '...' abc.repo

Then add this repo as one of package source for yum. See this for more details:

https://www.centos.org/docs/5/html/yum/sn-using-repositories.html

The repo file you have is simply a text configuration find pointing to a URL where Hadoop BigTop packages can be downloaded and installed.

Probably you need to run yum to download and install the packages after adding the repo. See your original dummies guide.

Hi YK,

Book says run this :

yum install hadoop\* mahout\* oozie\* hbase\* hive\* hue\* pig\* zookeeper\*

i followed accordingly to install all these hadoop related components. After that it says "Complete".

Since the components are already installed, is it still necessary to "add the repo as one of package source for yum" that you taught me?

 

[ykgoh]

Previously, the repo file was not even downloaded n installed successfully. Not sure how you managed to install the packages without it.

Anyway I think you're following the instructions here: http://m.dummies.com/how-to/content/set-up-the-hadoop-environment-with-apache-bigtop.html

If you're sure the packages are installed properly, just proceed as per normal. Install Java and start Hadoop.

 

[Nipponho]

Previously, the repo file was not even downloaded n installed successfully. Not sure how you managed to install the packages without it.

Anyway I think you're following the instructions here: http://m.dummies.com/how-to/content/set-up-the-hadoop-environment-with-apache-bigtop.html

If you're sure the packages are installed properly, just proceed as per normal. Install Java and start Hadoop.

i think somehow i must have unknowingly and blindly installed the repo successfully, hehe. Just to be sure, I ran the yum again a while ago, and it tells me all the components "already installed and latest version, nothing to do".

Ya, i am following the hard copy of this book that u showed. OK i proceed on.

Thanks for your help. And also thanks David and LocalITGuy too. Sweating....

 

[localITguy]

i think somehow i must have unknowingly and blindly installed the repo successfully, hehe. Just to be sure, I ran the yum again a while ago, and it tells me all the components "already installed and latest version, nothing to do".

Ya, i am following the hard copy of this book that u showed. OK i proceed on.

Thanks for your help. And also thanks David and LocalITGuy too. Sweating....

Please verify that the repo is installed correctly

yum repolist all

make sure that the repo is listed there

 

[Nipponho]

like this means ok already?

Please verify that the repo is installed correctly
yum repolist all
make sure that the repo is listed there

It shows up there as BigTop and status enabled. Means kao tim already?

[/URL][/IMG]

 

[localITguy]

It shows up there as BigTop and status enabled. Means kao tim already?

[/URL][/IMG]

Looks good to me..

 

[Nipponho]

Good morning LocalITGuy/ykGoh/Davidktw,

Problem again.

I have a file in VM desktop named “1987.csv” that I am supposed to copy and paste to Hadoop. Location of this file is /home/tom/Desktop

Book’s command is :
hdfs dfs -copyFromLocal 1987.csv /user/root

But it keeps saying “1987.csv” no such file or directory.

I tried
hdfs dfs -copyFromLocal home/tom/Desktop/1987.csv /user/root.
Still fail.

I tried doing this too:
su - hdfs
hdfs dfs -mkdir /user/root
hdfs dfs -chown root:hadoop /user/root
hdfs dfs -chmod 755 /user/root
Also failed.

How to solve this problem?

 

[ykgoh]

Your command:

hdfs dfs -copyFromLocal home/tom/Desktop/1987.csv /user/root

Try this:

hdfs dfs -copyFromLocal /home/tom/Desktop/1987.csv /user/root

Your command seems to be missing a backslash in front of home. And check your current working directory using pwd.

 

[Nipponho]

Worked

Your command:

Try this:

hdfs dfs -copyFromLocal /home/tom/Desktop/1987.csv /user/root

Your command seems to be missing a backslash in front of home. And check your current working directory using pwd.

kao tim. Thanks!

[/URL][/IMG]

 

[Nipponho]

Hi Gurus,

Problem again.

Objective : Use Fsck line command to list the blocks that make up each file in HDFS.

Book’s command : % hadoop fsck / -files –blocks
Error message : Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. Fsck on path “/” failed

So I removed the % hadoop and / and tried this :
Hdfs fsck –files –blocks

And I got this response that does not seem to achieve the objective :

Usage: hdfs fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]]
<path>start checking from this path
-movemove corrupted files to /lost+found
-deletedelete corrupted files
-filesprint out files being checked
-openforwriteprint out files opened for write
-includeSnapshotsinclude snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it
-list-corruptfileblocksprint out list of missing blocks and files they belong to
-blocksprint out block report
-locationsprint out locations for every block
-racksprint out network topology for data-node locations
-storagepoliciesprint out storage policy summary for the blocks

-blockIdprint out which file this blockId belongs to, locations (nodes, racks) of this block, and other diagnostics info (under replicated, corrupted or not, etc)

Please Note:
1. By default fsck ignores files opened for write, use -openforwrite to report such files. They are usually tagged CORRUPT or HEALTHY depending on their block allocation status
2. Option -includeSnapshots should not be used for comparing stats, should be used only for HEALTH check, as this may contain duplicates if the same file present in both original fs tree and inside snapshots.

Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenodeort> specify a namenode
-jt <local|resourcemanagerort> specify a ResourceManager
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenodeort> specify a namenode
-jt <local|resourcemanagerort> specify a ResourceManager
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

How? Please advise......