在Spark 1.6.0中,我有一个数据框,其中包含一个包含作业描述的列,如：
Description bartender bartender employee taxi-driver ...
我从该列检索一个唯一值列表：
val jobs = people.select("Description").distinct().rdd.map(r => r(0).asInstanceOf[String]).repartition(4)
然后,我尝试为每个工作描述检索具有该工作的人并做一些事情,但我得到一个NullPointerException：
jobs.foreach {
ajob =>
var peoplewithjob = people.filter($"Description" === ajob)
// ... do stuff
}
我不明白为什么会发生这种情况,因为每个工作都是从人员数据框中提取出来的,所以至少应该有一个工作…任何提示更多的欢迎！这是堆栈跟踪：
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 4.0 Failed 1 times,most recent failure: Lost task 3.0 in stage 4.0 (TID 206,localhost): java.lang.NullPointerException at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:131) at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$withPlan(DataFrame.scala:2165) at org.apache.spark.sql.DataFrame.filter(DataFrame.scala:799) at jago.Run$$anonfun$main$1.apply(Run.scala:89) at jago.Run$$anonfun$main$1.apply(Run.scala:82) at scala.collection.Iterator$class.foreach(Iterator.scala:742) at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$32.apply(RDD.scala:912) at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$32.apply(RDD.scala:912) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)