![]()
# Run on a YARN cluster in cluster deploy mode export HADOOP_CONF_DIR =XXX bin/spark-submit \ -class .SparkPi \ -master spark://207.184.161.138:7077 \ -deploy-mode cluster \ -supervise \ -executor-memory 20G \ -total-executor-cores 100 \ # Run on a Spark standalone cluster in cluster deploy mode with supervise # Run on a Spark standalone cluster in client deploy mode bin/spark-submit \ -class .SparkPi \ -master local \ Here are a few examples of common options: # Run application locally on 8 cores To enumerate all such options available to spark-submit, #Automatically download attachments spark for mac driver#You can also specify -supervise to make sure that the driver is automatically restarted if itįails with a non-zero exit code. There are a few options available that are specific to theįor example, with a Spark standalone cluster with cluster deploy mode, ![]() py files to the search path with -py-files. Currently, the standalone mode does not support cluster mode for Pythonįor Python applications, simply pass a. Locally on your laptop), it is common to use cluster mode to minimize network latency between Spark shell).Īlternatively, if your application is submitted from a machine far from the worker machines (e.g. Thus, this mode is especially suitableįor applications that involve the REPL (e.g. Output of the application is attached to the console. Within the spark-submit process which acts as a client to the cluster. In client mode, the driver is launched directly In this setup, client mode is appropriate. Master node in a standalone EC2 cluster). Physically co-located with your worker machines (e.g. † A common deployment strategy is to submit your application from a gateway machine
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |