13518219792
这篇文章主要介绍eclipse中如何运行spark机器学习代码，文中介绍的非常详细，具有一定的参考价值，感兴趣的小伙伴们一定要看完！
成都创新互联主营钦南网站建设的网络公司,主营网站建设方案,成都app软件开发公司,钦南h5重庆小程序开发搭建,钦南网站营销推广欢迎钦南等地区企业咨询
直接在eclipse运行，不需要hadoop，不需要搭建spark，只需要pom.xml中的依赖完整
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.mllib.classification.LogisticRegressionWithSGD
import org.apache.spark.mllib.feature.HashingTF
import org.apache.spark.mllib.regression.LabeledPoint
object MLlib {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName(s"Book example: Scala").setMaster("local[2]")
val sc = new SparkContext(conf)
// Load 2 types of emails from text files: spam and ham (non-spam).
// Each line has text from one email.
val spam = sc.textFile("file:/Users/xxx/Documents/hadoopTools/scala/eclipse/Eclipse.app/Contents/MacOS/workspace/spark_ml/src/main/resources/files/spam.txt")
val ham = sc.textFile("file:/Users/xxx/Documents/hadoopTools/scala/eclipse/Eclipse.app/Contents/MacOS/workspace/spark_ml/src/main/resources/files/ham.txt")
// val abc=sc.parallelize(seq, 2)
// Create a HashingTF instance to map email text to vectors of 100 features.
val tf = new HashingTF(numFeatures = 100)
// Each email is split into words, and each word is mapped to one feature.
val spamFeatures = spam.map(email => tf.transform(email.split(" ")))
val hamFeatures = ham.map(email => tf.transform(email.split(" ")))
// Create LabeledPoint datasets for positive (spam) and negative (ham) examples.
val positiveExamples = spamFeatures.map(features => LabeledPoint(1, features))
val negativeExamples = hamFeatures.map(features => LabeledPoint(0, features))
val trainingData = positiveExamples ++ negativeExamples
trainingData.cache() // Cache data since Logistic Regression is an iterative algorithm.
// Create a Logistic Regression learner which uses the LBFGS optimizer.
val lrLearner = new LogisticRegressionWithSGD()
// Run the actual learning algorithm on the training data.
val model = lrLearner.run(trainingData)
// Test on a positive example (spam) and a negative one (ham).
// First apply the same HashingTF feature transformation used on the training data.
val posTestExample = tf.transform("O M G GET cheap stuff by sending money to ...".split(" "))
val negTestExample = tf.transform("Hi Dad, I started studying Spark the other ...".split(" "))
// Now use the learned model to predict spam/ham for new emails.
println(s"Prediction for positive test example: ${model.predict(posTestExample)}")
println(s"Prediction for negative test example: ${model.predict(negTestExample)}")
sc.stop()
}
}sc.textFile里的参数是文件在本地的绝对路径。
setMaster("local[2]") 表示是本地运行，只使用两个核
HashingTF 用来从文档中创建词条目的频率特征向量，这里设置维度为100.
TF-IDF(Term frequency-inverse document frequency ) 是文本挖掘中一种广泛使用的特征向量化方法。TF-IDF反映了语料中单词对文档的重要程度。假设单词用t表示，文档用d表示，语料用D表示，那么文档频度DF(t, D)是包含单词t的文档数。如果我们只是使用词频度量重要性，就会很容易过分强调重负次数多但携带信息少的单词，例如：”a”, “the”以及”of”。如果某个单词在整个语料库中高频出现，意味着它没有携带专门针对某特殊文档的信息。逆文档频度(IDF)是单词携带信息量的数值度量。
pom.xml
