Dataproc: Qwik Start - Console

Dataproc: Qwik Start - Console


Dataproc is a fast, easy-to-use, fully-managed cloud service for running Apache Spark and Apache Hadoop clusters in a simpler, more cost-efficient way. Operations that used to take hours or days take seconds or minutes instead. Create Dataproc clusters quickly and resize them at any time, so you don't have to worry about your data pipelines outgrowing your clusters.

This lab shows you how to use the Google Cloud console to create a Dataproc cluster, run a simple Apache Spark job in the cluster, and then modify the number of workers in the cluster.

Task 1. Create a cluster

  1. In the Cloud Platform Console, select Navigation menu > Dataproc > Clusters, then click Create cluster.

  2. Click Create for Cluster on Compute Engine.

  3. Set the following fields for your cluster and accept the default values for all other fields:

Note: In the Configure nodes section ensure both the Master node and Worker nodes are set to the correct Machine Series and Machine Type

Machine SeriesE2
Machine Typee2-standard-2
Number of Worker Nodes2
Primary disk size30 GB
Internal IP onlyDeselect "Configure all instances to have only internal IP addresses"

Note: A Zone is a special multi-region namespace that is capable of deploying instances into all Google Compute zones globally. You can also specify distinct regions, such as us-central1 or europe-west1, to isolate resources (including VM instances and Cloud Storage) and metadata storage locations utilized by Cloud Dataproc within the user-specified region.

  1. Click Create to create the cluster.

Your new cluster will appear in the Clusters list. It may take a few minutes to create, the cluster Status shows as Provisioning until the cluster is ready to use, then changes to Running.


export REGION=
export ZONE=
gcloud dataproc clusters create example-cluster \
--region=$REGION \
--zone=$ZONE \
--master-machine-type=e2-standard-2 \
--master-boot-disk-size=50GB \
--num-workers=2 \
--worker-machine-type=e2-standard-2 \

Task 2. Submit a job

To run a sample Spark job:

  1. Click Jobs in the left pane to switch to Dataproc's jobs view, then click Submit job.

  2. Set the following fields to update Job. Accept the default values for all other fields:

Job typeSpark
Main class or jarorg.apache.spark.examples.SparkPi
Jar filesfile:///usr/lib/spark/examples/jars/spark-examples.jar
Arguments1000 (This sets the number of tasks.)
  1. Click Submit.

Note: How the job calculates Pi: The Spark job estimates a value of Pi using the Monte Carlo method. It generates x,y points on a coordinate plane that models a circle enclosed by a unit square. The input argument (1000) determines the number of x,y pairs to generate; the more pairs generated, the greater the accuracy of the estimation. This estimation leverages Cloud Dataproc worker nodes to parallelize the computation. For more information, see Estimating Pi using the Monte Carlo Method and see on GitHub.

Your job should appear in the Jobs list, which shows your project's jobs with its cluster, type, and current status. Job status displays as Running, and then Succeeded after it completes.


gcloud dataproc jobs submit spark \
  --region=$REGION \
  --cluster=example-cluster \
  --class=org.apache.spark.examples.SparkPi \
  --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar \
  -- 1000

Task 3. View the job output

To see your completed job's output:

  1. Click the job ID in the Jobs list.

  2. Select LINE WRAP to ON or scroll all the way to the right to see the calculated value of Pi. Your output, with LINE WRAPON, should look something like this:


Your job has successfully calculated a rough value for pi!

Task 4. Update a cluster to modify the number of workers

To change the number of worker instances in your cluster:

  1. Select Clusters in the left navigation pane to return to the Dataproc Clusters view.

  2. Click example-cluster in the Clusters list. By default, the page displays an overview of your cluster's CPU usage.

  3. Click Configuration to display your cluster's current settings.

  4. Click Edit. The number of worker nodes is now editable.

  5. Enter 4 in the Worker nodes field.

  6. Click Save.

Your cluster is now updated. Check out the number of VM instances in the cluster.


gcloud dataproc clusters update example-cluster \
--num-workers=4 \