Some questions about compiling carbondata

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Some questions about compiling carbondata

李寅威
Hi all,

  I've tried the following two ways to compile carbondata, but they finally all failed, if anyone could help me:


  1. with the latest verson carbondata at github and spark 2.0.0, use the command


      mvn -DskipTests -Pspark-2.0 -Dspark.version=2.0.0 clean package


     and it compiled successly.


      but when I run the script   ./bin/carbon-spark-shell  ,  it turns out:
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ./bin/carbon-spark-shell: line 78: /bin/spark-submit: No such file or directory


      and when I run the script  ./bin/carbon-spark-sql, it turns out:
      $SPARK_HOME is not set
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ./bin/carbon-spark-sql: line 79: cmd: command not found
      ./bin/carbon-spark-sql: line 89: /bin/spark-submit: No such file or directory
      ./bin/carbon-spark-sql: line 89: exec: /bin/spark-submit: cannot execute: No such file or directory


    2.with the carbondata-0.2.0 and spark 1.5.0, use the command


      mvn -DskipTests -Pspark-1.5 -X -Dspark.version=1.5.0 clean package


     and it compiled failed. the error message is :


        [ERROR] /home/hadoop/carbondata-0.2.0/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala:46: error: not found: value SqlParser
        [INFO]     val tableIdent = SqlParser.parseTableIdentifier(tableWithDb)
        [INFO]                      ^
        [WARNING] one warning found
        [ERROR] one error found
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO]
        [INFO] Apache CarbonData :: Parent ........................ SUCCESS [  0.971 s]
        [INFO] Apache CarbonData :: Common ........................ SUCCESS [  1.702 s]
        [INFO] Apache CarbonData :: Core .......................... SUCCESS [  4.428 s]
        [INFO] Apache CarbonData :: Processing .................... SUCCESS [  1.587 s]
        [INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [  1.104 s]
        [INFO] Apache CarbonData :: Spark ......................... FAILURE [ 11.456 s]
        [INFO] Apache CarbonData :: Assembly ...................... SKIPPED
        [INFO] Apache CarbonData :: Examples ...................... SKIPPED
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 21.432 s
        [INFO] Finished at: 2016-12-15T19:41:29+08:00
        [INFO] Final Memory: 69M/995M
        [INFO] ------------------------------------------------------------------------
        [ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1]
        org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
                at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
                at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
                at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
                at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
                at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
                at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
                at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
                at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
                at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
                at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:497)
                at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
                at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
                at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
                at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
        Caused by: org.apache.maven.plugin.MojoExecutionException: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:350)
                at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
                ... 20 more
        Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:346)
                at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:149)
                at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:136)
                at org_scala_tools_maven_executions.JavaMainCallerByFork.run(JavaMainCallerByFork.java:80)
                at org_scala_tools_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:124)
                at org_scala_tools_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:80)
                at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:342)
                ... 22 more
        [ERROR]
        [ERROR]
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
        [ERROR]
        [ERROR] After correcting the problems, you can resume the build with the command
        [ERROR]   mvn <goals> -rf :carbondata-spark
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Some questions about compiling carbondata

cenyuhai
Carbon don't support spark 2.x




------------------ Original ------------------
From:  "李寅威";<[hidden email]>;
Date:  Thu, Dec 15, 2016 07:45 PM
To:  "dev"<[hidden email]>;

Subject:  Some questions about compiling carbondata



Hi all,

  I've tried the following two ways to compile carbondata, but they finally all failed, if anyone could help me:


  1. with the latest verson carbondata at github and spark 2.0.0, use the command


      mvn -DskipTests -Pspark-2.0 -Dspark.version=2.0.0 clean package


     and it compiled successly.


      but when I run the script   ./bin/carbon-spark-shell  ,  it turns out:
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ./bin/carbon-spark-shell: line 78: /bin/spark-submit: No such file or directory


      and when I run the script  ./bin/carbon-spark-sql, it turns out:
      $SPARK_HOME is not set
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ls: cannot access /home/hadoop/incubator-carbondata/assembly/target/scala-2.10: No such file or directory
      ./bin/carbon-spark-sql: line 79: cmd: command not found
      ./bin/carbon-spark-sql: line 89: /bin/spark-submit: No such file or directory
      ./bin/carbon-spark-sql: line 89: exec: /bin/spark-submit: cannot execute: No such file or directory


    2.with the carbondata-0.2.0 and spark 1.5.0, use the command


      mvn -DskipTests -Pspark-1.5 -X -Dspark.version=1.5.0 clean package


     and it compiled failed. the error message is :


        [ERROR] /home/hadoop/carbondata-0.2.0/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala:46: error: not found: value SqlParser
        [INFO]     val tableIdent = SqlParser.parseTableIdentifier(tableWithDb)
        [INFO]                      ^
        [WARNING] one warning found
        [ERROR] one error found
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO]
        [INFO] Apache CarbonData :: Parent ........................ SUCCESS [  0.971 s]
        [INFO] Apache CarbonData :: Common ........................ SUCCESS [  1.702 s]
        [INFO] Apache CarbonData :: Core .......................... SUCCESS [  4.428 s]
        [INFO] Apache CarbonData :: Processing .................... SUCCESS [  1.587 s]
        [INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [  1.104 s]
        [INFO] Apache CarbonData :: Spark ......................... FAILURE [ 11.456 s]
        [INFO] Apache CarbonData :: Assembly ...................... SKIPPED
        [INFO] Apache CarbonData :: Examples ...................... SKIPPED
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 21.432 s
        [INFO] Finished at: 2016-12-15T19:41:29+08:00
        [INFO] Final Memory: 69M/995M
        [INFO] ------------------------------------------------------------------------
        [ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1]
        org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project carbondata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
                at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
                at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
                at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
                at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
                at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
                at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
                at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
                at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
                at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
                at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:497)
                at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
                at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
                at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
                at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
        Caused by: org.apache.maven.plugin.MojoExecutionException: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:350)
                at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
                ... 20 more
        Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1)
                at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:346)
                at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:149)
                at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:136)
                at org_scala_tools_maven_executions.JavaMainCallerByFork.run(JavaMainCallerByFork.java:80)
                at org_scala_tools_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:124)
                at org_scala_tools_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:80)
                at org_scala_tools_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:342)
                ... 22 more
        [ERROR]
        [ERROR]
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
        [ERROR]
        [ERROR] After correcting the problems, you can resume the build with the command
        [ERROR]   mvn <goals> -rf :carbondata-spark
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Some questions about compiling carbondata

Jacky Li
In reply to this post by 李寅威
Hi,

You do not need to specify spark.version variable, you can try these:
mvn clean package -DskipTests -Pspark-2.0  (to build carbon with spark-2.0.2)
mvn clean package -DskipTests  (to build carbon with spark-1.5.2, which is default profile)

Regards,
Jacky
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Some questions about compiling carbondata

lzzuo2
In reply to this post by cenyuhai
请问这个问题现在解决了么?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Some questions about compiling carbondata

Erlu Chen
In reply to this post by 李寅威
Hi

For Question one, I have raised discussion about carbon-spark-shell for spark2.x in following link.

http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-Whether-Carbondata-should-keep-carbon-spark-shell-script-td14077.html

Actually there is PR to fix carbon-spark-shell for spark2.x, but I think this script is useless.

You can give your opinion, we can vote for this.

Regards.
Chenerlu
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Some questions about compiling carbondata

ravipesala
Carbon does not support Spark 2.0, please use Spark 2.1

Use the following command to compile
   mvn clean  -Pspark-2.1  package

Regards,
Ravindra.

On 15 June 2017 at 14:44, Erlu Chen <[hidden email]> wrote:

> Hi
>
> For Question one, I have raised discussion about carbon-spark-shell for
> spark2.x in following link.
>
> http://apache-carbondata-dev-mailing-list-archive.1130556.
> n5.nabble.com/DISCUSSION-Whether-Carbondata-should-
> keep-carbon-spark-shell-script-td14077.html
>
> Actually there is PR to fix carbon-spark-shell for spark2.x, but I think
> this script is useless.
>
> You can give your opinion, we can vote for this.
>
> Regards.
> Chenerlu
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/Some-questions-
> about-compiling-carbondata-tp4498p15139.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>



--
Thanks & Regards,
Ravi
Loading...