A few days back I tried to set up a spark standalone cluster in my own machine with the following specification: two workers (balanced cores) within a single node.
Here’s how you’d do it.
Step 1. Start the master instance via
Step 2. Configure the spark environment properties.
Go to the
conf directory and add the following lines to
export SPARK_WORKER_INSTANCES=<number_of_workers> export SPARK_WORKER_CORES=<total_cores_for_the_workers>
The property names should be intuitive. We specified the number of worker instances and the total number of cores for the workers.
Step 3. Start the slave node via
Step 4. To check the configuration results, just go to