13518219792
本篇内容介绍了“hadoop的wordcount java举例分析”的有关知识，在实际案例的操作过程中，不少人都会遇到这样的困境，接下来就让小编带领大家学习一下如何处理这些情况吧！希望大家仔细阅读，能够学有所成！
1.导入hadoop需要用到的包
hadoop-2.4.2/share/hadoop/mapreduce/*.jar
hadoop-2.4.2/share/hadoop/mapreduce/lib/*.jar
hadoop-2.4.2/share/hadoop/common/*.jar
hadoop-2.4.2/share/hadoop/common/lib/*.jar
2.编写java程序
package demo;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
public class WordCount {
public static class Map extends MapReduceBase implements Mapper{
private final static IntWritable one=new IntWritable(1);
private Text word=new Text();
@Override
public void map(LongWritable key, Text value,
OutputCollector output, Reporter reporter)
throws IOException {
// TODO Auto-generated method stub
String line=value.toString();
StringTokenizer tokenizer=new StringTokenizer(line);
while (tokenizer.hasMoreTokens()){
word.set(tokenizer.nextToken());
output.collect(word,one);
}
}
}
//旧版本
public static class Reduce extends MapReduceBase implements Reducer{
@Override
public void reduce(Text key, Iterator values,
OutputCollector output, Reporter reporter)
throws IOException {
// TODO Auto-generated method stub
int sum=0;
while(values.hasNext()){
sum+=values.next().get();
}
output.collect(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception{
// TODO Auto-generated method stub
//System.setProperty("HADOOP_USER_NAME","root");
JobConf conf=new JobConf(WordCount.class);
//conf.set("fs.defaultFS","hdfs://192.168.1.120:9000");
conf.setJobName("wordcount");
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(Map.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.setInputPaths(conf,new Path(args[0]));
FileOutputFormat.setOutputPath(conf,new Path(args[1]));
JobClient.runJob(conf);
}
}
3.导出为jar文件
4.上传到linux系统中。
5.新建input目录，如果有output目录，先删除
6.上传jar包后，到jar包的目录下，执行
hadoop jar WordCount.jar demo.WordCount /input/* /output/
7.如果执行时不带“/”，会在hadoop目录中新建/user/root下新建两个文件夹，会提示
Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file: /input Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://master:9000/user/root/input
只需要在执行的时候带上“/”就行。
8.获取分离后的文件
hadoop fs -get /output/* output/
“hadoop的wordcount java举例分析”的内容就介绍到这里了，感谢大家的阅读。如果想了解更多行业相关的知识可以关注创新互联-成都网站建设公司网站，小编将为大家输出更多高质量的实用文章！
