Lesson 2 of 6
Start and configure a cluster
Launch a small all-purpose cluster and understand how it runs your code.
4 minDatabricks
After this lesson, you will be able to launch a small all-purpose cluster in Databricks, choose the right settings for a beginner workload, and explain how the cluster runs your code.
What is a cluster?
A cluster is a set of cloud computers that work together to run your code. In Databricks, a cluster has a driver node and one or more worker nodes. The driver coordinates the work. The workers do the heavy processing. When you run a query or a notebook, Databricks sends it to the driver, which splits the work among the workers.
For this course, you will use an all-purpose cluster. This type of cluster is designed for interactive work, such as running queries one at a time in a notebook. It stays running until you stop it, so you can run several commands without waiting for a new cluster each time.
Create a cluster
In your Databricks workspace, click Compute in the sidebar. Then click Create compute. You will see a form with many options. For now, you only need to change a few.
Give the cluster a name, such as My First Cluster. Under Cluster mode, keep the default Single node. A single-node cluster has one driver and no separate workers. The driver does all the work. This is cheaper and simpler for learning, and it is enough for small data.
Under Databricks runtime version, keep the default. The runtime is the set of software that Databricks installs on the cluster, including Apache Spark and other libraries. The default is a recent stable version with long-term support.
Under Node type, choose the smallest available instance. The exact name depends on your cloud provider, but look for something like Standard_DS3_v2 on Azure or m5.large on AWS. These are general-purpose machines with a few CPUs and a few gigabytes of memory. They are fine for the small CSV file you will use later.
Leave all other settings at their defaults. Scroll down and click Create compute. Databricks will start to provision the cluster. This can take a few minutes. You will see the cluster state change from Pending to Running.
How a cluster runs your code
When you attach a notebook to a cluster and run a cell, here is what happens. The notebook sends the code to the driver. The driver parses the code and builds a plan. If the cluster has workers, the driver sends tasks to them. With a single-node cluster, the driver runs the tasks itself.
The driver also keeps track of the state of your session. For example, if you create a temporary view in one cell, the driver remembers it for the next cell. This is why you can run cells in order and build on earlier results.
If you stop the cluster, the driver shuts down and all session state is lost. Your notebooks and data files are stored separately, so they are safe. But any temporary views or variables you created are gone. When you start the cluster again, you get a fresh driver.
Monitor your cluster
While the cluster is running, you can see its status on the Compute page. Click the cluster name to open its details. You will see tabs for metrics, event logs, and configuration. The metrics tab shows CPU and memory usage. For now, it will be mostly flat because you are not running heavy work.
The event log shows what Databricks is doing, such as starting the cluster or installing libraries. If something goes wrong, this is a good place to look.
Stop the cluster
When you are not using the cluster, stop it to save money. On the Compute page, click the square stop icon next to the cluster name. The cluster will shut down. You can start it again later with the play icon. Starting a stopped cluster is faster than creating a new one, because Databricks keeps the configuration.
For this lesson, leave the cluster running. You will use it in the next lesson to run your first query.
Key takeaways
- A cluster is a set of cloud computers that run your code, with a driver that coordinates work and workers that process data.
- A single-node all-purpose cluster is a good choice for learning: it is simple, cheap, and keeps session state between notebook cells.
- The driver holds session state, so temporary views and variables disappear when you stop the cluster.
- Stop your cluster when you are not using it to avoid unnecessary cloud costs.
Try it
Create a new single-node all-purpose cluster with the smallest available node type. Name it Exercise Cluster. Wait for it to reach the Running state. Then open the cluster details and find the Spark UI link. Click it and look at the Jobs tab. You should see no jobs yet, because you have not run any code. Stop the cluster when you are done. A good result is that the cluster starts, you can open the Spark UI, and the Jobs tab shows an empty list.
Official documentation
Written with AI and checked automatically. Tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.