Chapter 19: Spark build failure

[joelrorseth]

Hi, I have been unable to get Maven to build Chapter 19. I am getting the following message with the build error:

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.6:compile (default) on project ch19-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

The issue appears to be related to the scala plugin. I am running macOS 10.14.4, and have Java 8 and 12 installed (OpenJDK).

I have tried removing the scala-reflect install, did not fix.
rm -R /Users/Joel/.m2/repository/org/scala-lang/scala-reflect/2.10.4

Here is the full trace

[ERROR] error: error while loading package, Missing dependency 'object java.lang.Object in compiler mirror', required by /Users/Joel/.m2/repository/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar(scala/package.class)
[ERROR] error: error while loading package, Missing dependency 'object java.lang.Object in compiler mirror', required by /Users/Joel/.m2/repository/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar(scala/runtime/package.class)
[ERROR] error: scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.
[ERROR] at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
[ERROR] at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
[ERROR] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
[ERROR] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
[ERROR] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
[ERROR] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
[ERROR] at scala.reflect.internal.Mirrors$RootsBase.getClassByName(Mirrors.scala:99)
[ERROR] at scala.reflect.internal.Mirrors$RootsBase.getRequiredClass(Mirrors.scala:102)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:264)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:264)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.AnyRefClass$lzycompute(Definitions.scala:263)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.AnyRefClass(Definitions.scala:263)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.specialPolyClass(Definitions.scala:1120)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.RepeatedParamClass$lzycompute(Definitions.scala:407)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.RepeatedParamClass(Definitions.scala:407)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses$lzycompute(Definitions.scala:1154)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses(Definitions.scala:1152)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode$lzycompute(Definitions.scala:1196)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode(Definitions.scala:1196)
[ERROR] at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1261)
[ERROR] at scala.tools.nsc.Global$Run.(Global.scala:1290)
[ERROR] at scala.tools.nsc.Driver.doCompile(Driver.scala:32)
[ERROR] at scala.tools.nsc.Main$.doCompile(Main.scala:79)
[ERROR] at scala.tools.nsc.Driver.process(Driver.scala:54)
[ERROR] at scala.tools.nsc.Driver.main(Driver.scala:67)
[ERROR] at scala.tools.nsc.Main.main(Main.scala)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:567)
[ERROR] at scala_maven_executions.MainHelper.runMain(MainHelper.java:164)
[ERROR] at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
[ERROR]

[RuVT]

I was having the same problem.
Just started to play around with all the hadoop stuff, I know Java but I'm by far not an expert in the language and all the tools that come with it, one of those tools is Maven.

So after a week of trying to build the code getting the same error I started to read more about maven and found this:
https://maven.apache.org/guides/mini/guide-using-toolchains.html

Long story short, if your are new to all this Maven+Java thing then you probably have not created the ~/.m2/toolchains.xml file as in the guide; once you created that file, Maven will use the path you specify there to look for the needed JDK version.

With the toolchains.xml file in place I'm now able to compile all the sub modules.

This is just a problem when you are a newbie ( like me 🤦‍♂️ ), is common knowledge so is not written in the instructions.

[RuVT]

I created a PR (#39 ) just to explicitly tell the toolchains.xml file part.
@tomwhite I know that this is just common knowledge, but a little hint can help a lot to the newcomers.

[woozway]

I'm a newbie, can you show me how...

[th3-w1tn3ss]

I installed Open-jdk-8 then my problem was solved