Sunday 15 January 2012

apache pig - hadoop pig cannot mkdir java throw IO exception -


I have a very simple script example from Hewup real world solution cookbook, and I call it on Amazon Claudera Clertartogov 04 Amy I try and it gives me the Mkdir not able to Java exception ??? But do I have enough disk space ??

  [ec2-user] $ df Filesystem use available 1K- used to block %% / dev / xvde1 8255928 3307252 4529300 43% / tmpfs 3757068 0 3757068 0  < Pre>  web logs = load '/% / dev / shm / dev / xvdk 103212320 192116 97777324 1% / data   

Data2 / weblogs / weblog_entries.txt' ( Md5: chararray, url: chararray, date: chararray, time: chararray, ip: chararray); Md5_grp = Group Weblogs MD5 parallel 4; Store md5_grp in '/data/weblogs/weblogs_md5_groups.bcp'; Pig-S Local-f pig02 2 & gt; Err02

2013-06-20 19: 57: 29,49 9 [Thread-4] information org.apache.hadoop.mapred.Mger - Below previous merge pass, 1 segment balance from the total size: 291932 bytes 2013-06-20 19: 57: 29,49 9 [Thread-4] Information organization. Hap.apap.mapred.LocalJobRunner - 2013- 06-20 19: 57: 29,519 [Thread -4] warning org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 java.io.IOException: Failed to create Mkdirs file: At /data/weblogs/weblogs_md5_groups.bcp/_temporary/ _attempt_local_0001_r_000000_0 at org.apache.hadoop.fs.ChecksumFileSystem.create (ChecksumFileSystem.java:434) at org.apache.hadoop.fs.ChecksumFileSystem.create (ChecksumFileSystem .java: 420) .fs.FileSystem.create on org.apache.hadoop (FileSystem.java:805) org.apache.hadoop.fs.FileSystem.create (FileSystem.java:786) org.apache.hadoop.fs. on FileSystem.create (FileSystem.java:685) on Org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextOutputFormat.getRecordWriter (PigTextOu) TputFormat.java:98) org.apache.pig.backend.hadoop.executionengine Org.ap on .mapReduceLayer.PigOutputFormat.getRecordWriter (pigOutputFormat.java:84) ache.hadoop.mapred.ReduceTask.RunNewReducer (ReduceTask.java Letter 82) org.apache.hadoop.mapred.ReduceTask.run on org.apache.hadoop.mapred.LocalJobRunner $ Job.run (LocalJobRunner.java:309) (ReduceTask .java: 433) on 2013-06-20 19: 57: 33,176 [main] info org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Job job_local_0001 has failed! Prevent all dependent jobs from running
2013-06-20 19: 57: 33,180 [Main] Information organization.PackagePP.B & Aid HDOP Execution Engine. Map Radius Layer Mapreds Launcher - 100% complete
2013-06-20 19: 57: 33,182 [Main] error org.apache.pig.tools.pigstats.PigStatsUtil - 1 map failed to reduce job!
2013-06-20 19: 57: 33,182 [Main] Information org.apache. Pig Tools. Pistons SimplePastatsTits - Detected local mode. Statistics may be reported incomplete
2013-06-20 19: 57: 33,185 [Main] info org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features- 2.0.0 -CDH 4.1.2 0.10.0-CDH 4.1.2 EC2-User 2013-06-20 19:57:27 2013-06-20 19:57:33 Group_byY

failed!

Pig Stack Trace ---------------
Error 2244: Job unsuccessful, Hape did not return any error message

Organization apache.pig.backend.executionengine.ExecException: error 2244: failed job, Hadoop org.apache any error message org.apache.pig.tools.grunt.GruntParser.executeBatch (GruntParser.java:140). does not return the pig .tools.grunt.GruntParser.parseStopOnError (GruntParser.java:193) org.apache.pig.tools.grunt.GruntParser.parseStopOnError on org.apache.pig.tools.grunt (GruntParser.java:165 ) On
. Grnt.akssi (Grntkjawa: 84)
at org.apache.pig.Main.run (main. Java: 430)
at org.apache.pig.Main.main (Man. Java 11) java on sun.reflect.NativeMethodAccessorImpl.invoke0 (Native method) (NativeMethodAccessorImpl.java:39) on sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25) on sun.reflect.NativeMethodAccessorImpl.invoke on .lang.reflect.Method .invoke (Method.java:597)

org.apache.hadoop.util.RunJar.main (RunJar.java208)

You can not make your Hadoop job look like that you specified in your directory store

Sh Get like trying to storing produce at a different location, such as your home directory?

Also FYI, the pig is not its production called "weblogs_md5_groups.bcp", it really will create a directory with that will save to a file name.

No comments:

Post a Comment