No response for my wget command

[davidktw]

Hi Gurus,

Problem again.

Objective : Use Fsck line command to list the blocks that make up each file in HDFS.

Book’s command : % hadoop fsck / -files –blocks
Error message : Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. Fsck on path “/” failed

So I removed the % hadoop and / and tried this :
Hdfs fsck –files –blocks

And I got this response that does not seem to achieve the objective :

Usage: hdfs fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]]
<path>start checking from this path
-movemove corrupted files to /lost+found
-deletedelete corrupted files
-filesprint out files being checked
-openforwriteprint out files opened for write
-includeSnapshotsinclude snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it
-list-corruptfileblocksprint out list of missing blocks and files they belong to
-blocksprint out block report
-locationsprint out locations for every block
-racksprint out network topology for data-node locations
-storagepoliciesprint out storage policy summary for the blocks

-blockIdprint out which file this blockId belongs to, locations (nodes, racks) of this block, and other diagnostics info (under replicated, corrupted or not, etc)

Please Note:
1. By default fsck ignores files opened for write, use -openforwrite to report such files. They are usually tagged CORRUPT or HEALTHY depending on their block allocation status
2. Option -includeSnapshots should not be used for comparing stats, should be used only for HEALTH check, as this may contain duplicates if the same file present in both original fs tree and inside snapshots.

Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenodeort> specify a namenode
-jt <local|resourcemanagerort> specify a ResourceManager
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenodeort> specify a namenode
-jt <local|resourcemanagerort> specify a ResourceManager
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

How? Please advise......

hdfs fsck / -files -blocks

 

[Nipponho]

hdfs fsck / -files -blocks

hi David, still have error message : fsck on path "/" failed.

[/URL][/IMG]

 

[ykgoh]

hi David, still have error message : fsck on path "/" failed.

https://community.cloudera.com/t5/Storage-Random-Access-HDFS/hdfs-fsck-command-issue/td-p/25870

See above link for similar problem.

The problem is your user account does not have the required permissions to do fsck on /tmp/hive. If my interpretation is correct, your current user cannot read and execute in /tmp/hive. The other directories before it were OK.

It might be that you are logged in as the wrong account to do the fsck, or you need to grant yourself the appropriate permissions.

But do you know what is fsck supposed to do btw?

 

[localITguy]

TS, do you even know the basics of Linux?

do you even know what is fsck and what its purpose?

i suggest you to learn the basics of Linux first before jumping into advance stuff like fsck and hadoop

seems like you didn't get the basics right.

Linux is not that simple

 

[Nipponho]

https://community.cloudera.com/t5/Storage-Random-Access-HDFS/hdfs-fsck-command-issue/td-p/25870

See above link for similar problem.

The problem is your user account does not have the required permissions to do fsck on /tmp/hive. If my interpretation is correct, your current user cannot read and execute in /tmp/hive. The other directories before it were OK.

It might be that you are logged in as the wrong account to do the fsck, or you need to grant yourself the appropriate permissions.

But do you know what is fsck supposed to do btw?

Thanks! I ran this command at root. I only have one account and one user. I remembered there was some complication when i was installing hive 2 days ago. Something must have been wrong. I will figure out how to uninstall and reinstall Hive again later on.

Yeah i think so i know. Fsck is supposed to show us details for each file that is being split up, which parts of the file landed up on which blocks and on which data nodes in the cluster. No?

 

[Nipponho]

TS, do you even know the basics of Linux?
do you even know what is fsck and what its purpose?
i suggest you to learn the basics of Linux first before jumping into advance stuff like fsck and hadoop
seems like you didn't get the basics right.
Linux is not that simple

i have zero experience in linux before. Ya, linux is not that simple, that's why i am struggling now.

Before i embark on this, i called one popular training institute to enquire about the course. They told me java and linux knowledge is good to have but not compulsory as everything will be taught during the course. $2000+ for 5 saturday afternoons. So i decided to pick up a book to learn at my own pace and plan later on take the test conducted by IBM.

Actually i can understand most of the hadoop concepts explained in the book. Just that authors' commands contain errors and some important steps were totally omitted. That's why i got stuck. i managed to solve some of them as there were people before me who encountered the same situation and they posted their problem and solutions on some forums like stackoverflow.

Thanks so much for your guidance Learning curve, slowly pick up.

 

[localITguy]

Thanks! I ran this command at root. I only have one account and one user. I remembered there was some complication when i was installing hive 2 days ago. Something must have been wrong. I will figure out how to uninstall and reinstall Hive again later on.

Yeah i think so i know. Fsck is supposed to show us details for each file that is being split up, which parts of the file landed up on which blocks and on which data nodes in the cluster. No?

The system utility fsck (for "file system consistency check") is a tool for checking the consistency of a file system in Unix and Unix-like operating systems, such as Linux and OS X. It is similar to the chkdsk command in Microsoft Windows.

start with the basics, seems like you running around typing in commands that you don't understand, you will not learn anything that way

 

[Nipponho]

The system utility fsck (for "file system consistency check") is a tool for checking the consistency of a file system in Unix and Unix-like operating systems, such as Linux and OS X. It is similar to the chkdsk command in Microsoft Windows.

start with the basics, seems like you running around typing in commands that you don't understand, you will not learn anything that way

i see... ok... Thanks! . The book didn't explain this thoroughly. It only ask us to run fsck in order to see the details of the split.

Luckily got you around here. Can one. 硬硬来。I just have to struggle during troubleshooting.

 

[localITguy]

i see... ok... Thanks! . The book didn't explain this thoroughly. It only ask us to run fsck in order to see the details of the split.

Luckily got you around here. Can one. 硬硬来。I just have to struggle during troubleshooting.

the problem is you don't seem to understand the basics to start with.

take my advise and start slow.

junping in this way is not helping at all, been there done that.

i can honestly tell you there are lots of things that i do not know.

 

[Nipponho]

the problem is you don't seem to understand the basics to start with.
take my advise and start slow.
junping in this way is not helping at all, been there done that.
i can honestly tell you there are lots of things that i do not know.

ok will do. I will grab a book from library this weekend. Linux Essentials. 300 over pages. Thanks!

 

[ykgoh]

ok will do. I will grab a book from library this weekend. Linux Essentials. 300 over pages. Thanks!

Maybe you would like to consider another book for learning Hadoop since your dummies guide has so many errors and outdated info.

 

[Nipponho]

Maybe you would like to consider another book for learning Hadoop since your dummies guide has so many errors and outdated info.

hahahha, ya quite true. A few other ang mo learners who are using this book also complaining on forums that they cannot execute their commands due to these errors and outdated info and omitted steps. 5 co-authors = 5 pairs of eyes, but yet still so many errors, one of the authors is PHD holder somemore. For theories and concepts, they explain it quite well and thoroughly, but their practicals is headache.

 

[localITguy]

hahahha, ya quite true. A few other ang mo learners who are using this book also complaining on forums that they cannot execute their commands due to these errors and outdated info and omitted steps. 5 co-authors = 5 pairs of eyes, but yet still so many errors, one of the authors is PHD holder somemore. For theories and concepts, they explain it quite well and thoroughly, but their practicals is headache.

i reco Michael Jiang's books

 

[Nipponho]

i reco Michael Jiang's books

ok i keep a look out. A hell lot of related things to learn. I still haven't touch on analytics yet.