amazon web services - How to run a Spark jar file from AWS Console without Spark-Shell -


i'm trying run spark application on aws emr console (amazon web services). scala script compiled in jar takes sparkconf settings parameters or strings:

val sparkconf = new sparkconf()   .setappname("wikipediagraphxpagerank")   .setmaster(args(1))   .set("spark.executor.memory","1g")   .registerkryoclasses(array(classof[prvertex], classof[prmessage]))

however, don't know how pass master-url parameter , other parameters jar when it's uploaded , set-up cluster. clear, i'm aware if running spark-shell way, i'm windows user , current set-up , work i've done, useful have way pass master url emr cluster in 'steps'.

i don't want use spark-shell, have close deadline , have set-up way , feels small issue of passing master url parameter should possible, considering aws have guide running stand-alone spark applications on emr.

help appreciated!

here instructions on using spark-submit via emr step: https://github.com/awslabs/emr-bootstrap-actions/blob/master/spark/examples/spark-submit-via-step.md


-

Comments

Post a Comment

Popular posts from this blog

css - SVG using textPath a symbol not rendering in Firefox -

using textpath within symbol doesn't render in firefox. renders fine in latest chrome , ie, when try reference symbol, svg text doesn't render in firefox (37.0.1) - first box appears empty. code below (no external dependencies), there should 2 boxes word test flowing vertically in centre of each. edit: thought issue somehow involved flexbox layout, until paul pointed out issue exists without flexbox layout. the html is: <div> <svg id="not_working" viewbox="0 0 250 1200" preserveaspectratio="xmidymid meet"> <symbol id="test_symbol1" preserveaspectratio="xmidymid meet" viewbox="0 0 250 1200"> <path id="test_symbol_path" d="m 100 1200 l 100 0" /> <text font-size="100" fill="red"> <textpath text-anchor="middle" startoffset="50%" xlink:href="#test_symbol_path&q
Read more

Java 8 + Maven Javadoc plugin: Error fetching URL -

i attempting generate javadoc links javadoc dependencies. have tried various means generate javadoc not produce qualified class names references classes dependencies. wanted links java doc simplified class names. however, java api classnames, no links , have qualified class names. working java 8. have following configuration: <plugin> <groupid>org.apache.maven.plugins</groupid> <artifactid>maven-javadoc-plugin</artifactid> <version>2.10.2</version> <configuration> <reportoutputdirectory>${project.basedir}/target</reportoutputdirectory> <destdir>javadoc</destdir> <windowtitle>epiphany</windowtitle> <doctitle>epiphany</doctitle> <show>private</show> <detectlinks>false</detectlinks> <detectoffline
Read more

node.js - How to abort query on demand using Neo4j drivers -

i making search engine application using node.js , neo4j allows users submit graph traversal query via web-based user interface. want give users option cancel query after has been submitted (i.e. if user decides change query parameters). thus, need way abort query using either command node.js-to-neo4j driver or via cypher query. after few hours of searching, haven't been able find way using of node.js-to-neo4j drivers. can't seem find cypher query allows killing of query. overlooking something, or not possible neo4j? using neo4j 2.0.4, willing upgrade newer version of neo4j if has query killing capabilities. starting neo4j version 2.2.0, it's possible kill queries ui , via rest interface. don't know if existing nodejs drivers support feature, if not can still achieve same functionality making http request rest interface. if run query in browser in version 2.2.x or later, notice there (x) close link @ top-right corner of area queries executed , displayed
Read more