# Hardening Default GKE Cluster Configurations - GSP496

## Overview

This lab demonstrates some of the security concerns of a default GKE cluster configuration and the corresponding hardening measures to prevent multiple paths of pod escape and cluster privilege escalation. These attack paths are relevant in the following scenarios:

1. An application flaw in an external facing pod that allows for Server-Side Request Forgery (SSRF) attacks.
    
2. A fully compromised container inside a pod allowing for Remote Command Execution (RCE).
    
3. A malicious internal user or an attacker with a set of compromised internal user credentials with the ability to create/update a pod in a given namespace.
    

This lab was created by GKE Helmsman engineers to help you grasp a better understanding of hardening default GKE cluster configurations.

*The example code for this lab is provided as-is without warranty or guarantee*

### Objectives

Upon completion of this lab you will understand the need for protecting the GKE Instance Metadata and defining appropriate PodSecurityPolicy policies for your environment.

You will:

1. Create a small GKE cluster using the default settings.
    
2. Validate the most common paths of pod escape and cluster privilege escalation from the perspective of a malicious internal user.
    
3. Harden the GKE cluster for these issues.
    
4. Validate the cluster so that those actions are no longer allowed.
    

## Setup and requirements

### Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click **Start Lab**, shows how long Google Cloud resources are made available to you.

This hands-on lab lets you do the lab activities in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

* Access to a standard internet browser (Chrome browser recommended).
    

**Note:** Use an Incognito (recommended) or private browser window to run this lab. This prevents conflicts between your personal account and the student account, which may cause extra charges incurred to your personal account.

* Time to complete the lab—remember, once you start, you cannot pause a lab.
    

**Note:** Use only the student account for this lab. If you use a different Google Cloud account, you may incur charges to that account.

### How to start your lab and sign in to the Google Cloud console

1. Click the **Start Lab** button. If you need to pay for the lab, a dialog opens for you to select your payment method. On the left is the Lab Details pane with the following:
    
    * The Open Google Cloud console button
        
    * Time remaining
        
    * The temporary credentials that you must use for this lab
        
    * Other information, if needed, to step through this lab
        
2. Click **Open Google Cloud console** (or right-click and select **Open Link in Incognito Window** if you are running the Chrome browser).
    
    The lab spins up resources, and then opens another tab that shows the Sign in page.
    
    ***Tip:*** Arrange the tabs in separate windows, side-by-side.
    
    **Note:** If you see the **Choose an account** dialog, click **Use Another Account**.
    
3. If necessary, copy the **Username** below and paste it into the **Sign in** dialog.
    
    ```apache
    student-04-b88292ee0840@qwiklabs.net
    ```
    
    You can also find the Username in the Lab Details pane.
    
4. Click **Next**.
    
5. Copy the **Password** below and paste it into the **Welcome** dialog.
    
    ```apache
    JRuJ137fFPOA
    ```
    
    You can also find the Password in the Lab Details pane.
    
6. Click **Next**.
    
    **Important:** You must use the credentials the lab provides you. Do not use your Google Cloud account credentials.
    
    **Note:** Using your own Google Cloud account for this lab may incur extra charges.
    
7. Click through the subsequent pages:
    
    * Accept the terms and conditions.
        
    * Do not add recovery options or two-factor authentication (because this is a temporary account).
        
    * Do not sign up for free trials.
        

After a few moments, the Google Cloud console opens in this tab.

**Note:** To access Google Cloud products and services, click the **Navigation menu** or type the service or product name in the **Search** field.

![Navigation menu icon and Search field](https://cdn.qwiklabs.com/9Fk8NYFp3quE9mF%2FilWF6%2FlXY9OUBi3UWtb2Ne4uXNU%3D align="left")

### Activate Cloud Shell

Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources.

1. Click **Activate Cloud Shell** at the top of the Google Cloud console.
    
2. Click through the following windows:
    
    * Continue through the Cloud Shell information window.
        
    * Authorize Cloud Shell to use your credentials to make Google Cloud API calls.
        

When you are connected, you are already authenticated, and the project is set to your **Project\_ID**, `qwiklabs-gcp-04-99daceb10f6f`. The output contains a line that declares the **Project\_ID** for this session:

```apache
Your Cloud Platform project in this session is set to qwiklabs-gcp-04-99daceb10f6f
```

`gcloud` is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.

3. (Optional) You can list the active account name with this command:
    

```apache
gcloud auth list
```

4. Click **Authorize**.
    

**Output:**

```apache
ACTIVE: *
ACCOUNT: student-04-b88292ee0840@qwiklabs.net

To set the active account, run:
    $ gcloud config set account `ACCOUNT`
```

5. (Optional) You can list the project ID with this command:
    

```apache
gcloud config list project
```

**Output:**

```apache
[core]
project = qwiklabs-gcp-04-99daceb10f6f
```

**Note:** For full documentation of `gcloud`, in Google Cloud, refer to [the gcloud CLI overview guide](https://cloud.google.com/sdk/gcloud).

## Task 1. Create a simple GKE cluster

1. Set a zone into an environment variable called MY\_ZONE. This lab is using "`us-central1-f`", you can select a [zone](https://cloud.google.com/compute/docs/regions-zones/) if you prefer:
    

```apache
export MY_ZONE=us-central1-f
```

2. Run this to start a Kubernetes cluster managed by Kubernetes Engine named `simplecluster` and configure it to run 2 nodes:
    

```apache
gcloud container clusters create simplecluster --zone $MY_ZONE --num-nodes 2 --metadata=disable-legacy-endpoints=false
```

It takes several minutes to create a cluster as Kubernetes Engine provisions virtual machines for you. The warnings about features available in new versions can be safely ignored for this lab.

3. After the cluster is created, check your installed version of Kubernetes using the `kubectl version` command:
    

```apache
kubectl version
```

The `gcloud container clusters create` command automatically authenticated `kubectl` for you.

4. View your running nodes in the Cloud Console. On the **Navigation menu**, click **Compute Engine** > **VM Instances**.
    

Your Kubernetes cluster is now ready for use.

Click *Check my progress* to verify the objective.

Create a simple GKE cluster

**Check my progress**

## Task 2. Run a Google Cloud-SDK pod

1. From your Cloud Shell prompt, launch a single instance of the Google Cloud-SDK container:
    

```apache
kubectl run -it --rm gcloud --image=google/cloud-sdk:latest --restart=Never -- bash
```

This will take a few minutes to complete.

**Note:** If you get a timed out error, run the command again.

2. You should now have a bash shell inside the pod's container:
    

```apache
root@gcloud:/#
```

It may take a few seconds for the container to be started and the command prompt to be displayed. If you don't see a command prompt, try pressing **Enter**.

### Explore the Compute Metadata endpoint

1. Run the following command to access the `v1` Compute Metadata endpoint:
    

```apache
curl -s http://metadata.google.internal/computeMetadata/v1/instance/name
```

Output looks like:

```apache
...snip...
Your client does not have permission to get URL <code>/computeMetadata/v1/instance/name</code> from this server. Missing Metadata-Flavor:Google header.
...snip...
```

Notice how it returns an error stating that it requires the custom HTTP header to be present.

2. Add the custom header on the next run and retrieve the Compute Engine instance name that is running this pod:
    

```apache
curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name
```

Output looks like:

```apache
gke-simplecluster-default-pool-b57a043a-6z5v
```

**Note:** If a custom HTTP header is not required to access a Compute Engine Instance Metadata endpoint, an attacker would need only an application flaw to trick a web URL to provide user credentials. By requiring a custom HTTP header, an attack is more difficult as the attacker would need both an application flaw and the custom header to be successful.

Keep this shell inside the pod available for the next step.

3. If you accidentally exit from the pod, simply re-run:
    

```apache
kubectl run -it --rm gcloud --image=google/cloud-sdk:latest --restart=Never -- bash
```

### Explore the GKE node bootstrapping credentials

1. From inside the same pod shell, run the following command to list the attributes associated with the underlying Compute Engine instances. Be sure to include the trailing slash:
    

```apache
curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/attributes/
```

Perhaps the most sensitive data in this listing is `kube-env`. It contains several variables which the `kubelet` uses as initial credentials when attaching the node to the GKE cluster. The variables `CA_CERT`, `KUBELET_CERT`, and `KUBELET_KEY` contain this information and are therefore considered sensitive to non-cluster administrators.

2. To see the potentially sensitive variables and data, run the following command:
    

```apache
curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-env
```

Therefore, in any of the following situations:

* A flaw that allows for SSRF in a pod application
    
* An application or library flaw that allow for RCE in a pod
    
* An internal user with the ability to create or exec into a pod
    

There exists a high likelihood for compromise and exfiltration of sensitive `kubelet` bootstrapping credentials via the Compute Metadata endpoint. With the `kubelet` credentials, it is possible to leverage them in certain circumstances to escalate privileges to that of `cluster-admin` and therefore have full control of the GKE Cluster including all data, applications, and access to the underlying nodes.

### Leverage the Permissions assigned to this Node Pool's service account

By default, Google Cloud projects with the Compute API enabled have a default service account in the format of `NNNNNNNNNN-compute@developer.gserviceaccount.com` in the project and the `Editor` role attached to it. Also by default, GKE clusters created without specifying a service account will utilize the default Compute service account and attach it to all worker nodes.

1. Run the following `curl` command to list the OAuth scopes associated with the service account attached to the underlying Compute Engine instance:
    

```apache
curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/scopes
```

(output)

```apache
 https://www.googleapis.com/auth/devstorage.read_only https://www.googleapis.com/auth/logging.write https://www.googleapis.com/auth/monitoring https://www.googleapis.com/auth/service.management.readonly https://www.googleapis.com/auth/servicecontrol https://www.googleapis.com/auth/trace.append 
```

The combination of authentication scopes and the permissions of the service account dictates what applications on this node can access. The above list is the minimum scopes needed for most GKE clusters, but some use cases require increased scopes.

**Warning:** If, during cluster creation, you configured the authentication scope to include `https://www.googleapis.com/auth/cloud-platform`, any Google Cloud API would be in scope and only IAM permissions assigned to the service account would determine access.

Further, if the default service account with the default IAM Role of `Editor` is in use, any pod on this node pool has `Editor` permissions to the Google Cloud project where the GKE cluster is deployed. As the `Editor` IAM Role has a wide range of read/write permissions to interact with project resources such as Compute instances, Cloud Storage buckets, GCR registries, and more, this is a significant security risk.

2. Exit out of this pod by typing:
    

```apache
exit
```

**Note:** If did not return to cloud shell press ctrl+c

## Task 3. Deploy a pod that mounts the host filesystem

One of the simplest paths for "escaping" to the underlying host is by mounting the host's filesystem into the pod's filesystem using standard Kubernetes `volumes` and `volumeMounts` in a `Pod` specification.

1. To demonstrate this, run the following to create a Pod that mounts the underlying host filesystem `/` at the folder named `/rootfs` inside the container:
    

```apache
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: hostpath
spec:
  containers:
  - name: hostpath
    image: google/cloud-sdk:latest
    command: ["/bin/bash"]
    args: ["-c", "tail -f /dev/null"]
    volumeMounts:
    - mountPath: /rootfs
      name: rootfs
  volumes:
  - name: rootfs
    hostPath:
      path: /
EOF
```

2. Run `kubectl get pod` and re-run until it's in the "Running" state:
    

```apache
kubectl get pod
```

(Output)

```apache
NAME       READY   STATUS    RESTARTS   AGE
hostpath   1/1     Running   0          30s
```

Click *Check my progress* to verify the objective.

Deploy a pod that mounts the host filesystem

**Check my progress**

## Task 4. Explore and compromise the underlying host

1. Run the following to obtain a shell inside the pod you just created:
    

```apache
kubectl exec -it hostpath -- bash
```

2. Switch to the pod shell's root filesystem point to that of the underlying host:
    

```apache
chroot /rootfs /bin/bash
```

With those simple commands, the pod is now effectively a `root` shell on the node. You are now able to do the following:

| run the standard docker command with full permissions | `docker ps` |
| --- | --- |
| list docker images | `docker images` |
| `docker run` a privileged container of your choosing | `docker run --privileged <imagename>:<imageversion>` |
| examine the Kubernetes secrets mounted | `mount | grep volumes | awk '{print $3}' | xargs ls` |
| `exec` into any running container (even into another pod in another namespace) | `docker exec -it <docker container ID> sh` |

Nearly every operation that the `root` user can perform is available to this pod shell. This includes persistence mechanisms like adding SSH users/keys, running privileged docker containers on the host outside the view of Kubernetes, and much more.

3. To exit the pod shell, run `exit` twice - once to leave the `chroot` and another to leave the pod's shell:
    

```apache
exit
```

```apache
exit
```

**Note:** If did not return to cloud shell press ctrl+c

4. Now you can delete the `hostpath` pod:
    

```apache
kubectl delete pod hostpath
```

### Understand the available controls

The next steps of this demo will cover:

* **Disabling the Legacy Compute Engine Metadata API Endpoint** - By specifying a custom metadata key and value, the `v1beta1` metadata endpoint will no longer be available from the instance.
    
* **Enable Metadata Concealment** - Passing an additional configuration during cluster and/or node pool creation, a lightweight proxy will be installed on each node that proxies all requests to the Metadata API and prevents access to sensitive endpoints.
    
* **Enable and Utilize Pod Security Admission** - Enable the Pod Security Admission (PSA) controller in your GKE cluster. This provides the ability to enforce pod security standards that enhance your cluster's security posture.
    

## Task 5. Deploy a second node pool

To enable you to experiment with and without the Metadata endpoint protections in place, you'll create a second node pool that includes two additional settings. Pods that are scheduled to the generic node pool will not have the protections, and Pods scheduled to the second node pool will have them enabled.

**Note:** Legacy endpoints were deprecated on September 30, 2020. In GKE versions 1.12 and newer, the `--metadata=disable-legacy-endpoints=true` setting is automatically enabled. The next command below explicitly defines it for clarity.

* Create the second node pool:
    

```apache
gcloud beta container node-pools create second-pool --cluster=simplecluster --zone=$MY_ZONE --num-nodes=1 --metadata=disable-legacy-endpoints=true --workload-metadata-from-node=SECURE
```

Click *Check my progress* to verify the objective.

Deploy a second node pool

**Check my progress**

## Task 6. Run a Google Cloud-SDK pod

1. In Cloud Shell, launch a single instance of the Google Cloud-SDK container that will be run only on the second node pool with the protections enabled and not run as the root user:
    

```apache
kubectl run -it --rm gcloud --image=google/cloud-sdk:latest --restart=Never --overrides='{ "apiVersion": "v1", "spec": { "securityContext": { "runAsUser": 65534, "fsGroup": 65534 }, "nodeSelector": { "cloud.google.com/gke-nodepool": "second-pool" } } }' -- bash
```

**Note:** If you get a timed out error, run the command again.

2. You should now have a bash shell inside the pod's container running on the node pool named `second-pool`. You should see the following:
    

```apache
nobody@gcloud:/$
```

It may take a few seconds for the container to start and the command prompt to open.

If you don't see a command prompt, press **Enter**.

### Explore various blocked endpoints

1. With the configuration of the second node pool set to `--workload-metadata-from-node=SECURE` , the following command to retrieve the sensitive file, `kube-env`, will now fail:
    

```apache
curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-env
```

(Output)

```apache
This metadata endpoint is concealed.
```

2. But other commands to non-sensitive endpoints will still succeed if the proper HTTP header is passed:
    

```apache
curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name
```

(Example Output)

```apache
gke-simplecluster-second-pool-8fbd68c5-gzzp
```

3. Exit out of the pod:
    

```apache
exit
```

You should now be back in Cloud Shell.

## Task 7. Enforce Pod Security Standards

1. In order to have the necessary permissions to proceed, grant explicit permissions to your own user account to become `cluster-admin:`
    

```apache
kubectl create clusterrolebinding clusteradmin --clusterrole=cluster-admin --user="$(gcloud config list account --format 'value(core.account)')"
```

(Output)

```apache
clusterrolebinding.rbac.authorization.k8s.io/clusteradmin created
```

2. Now you will enforce a pod security standard. Choose the most appropriate security standard for your 'default' namespace. The 'restricted' profile offers stronger security:
    

```apache
kubectl label namespace default pod-security.kubernetes.io/enforce=restricted
```

3. Next you will create a ClusterRole. If you want to control who can modify Pod Security Admission levels on namespaces, create a ClusterRole called `pod-security-manager`:
    

```apache
cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
   name: pod-security-manager
rules:
- apiGroups: ['policy']
  resources: ['podsecuritypolicies']
  resourceNames: ['privileged', 'baseline', 'restricted']
  verbs: ['use']
- apiGroups: ['']
  resources: ['namespaces']
  verbs: ['get', 'list', 'watch', 'label']
EOF
```

4. Next, you will create a RoleBinding. To restrict who can change namespace labels related to Pod Security Admission, create a RoleBinding in the 'default' namespace:
    

```apache
cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
   name: pod-security-modifier
   namespace: default
subjects:
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:authenticated
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pod-security-manager
EOF
```

**Note:** For highly customized and dynamic pod security enforcement, consider tools like OPA Gatekeeper or Kyverno.

Click *Check my progress* to verify the objective.

Deploy PodSecurityPolicy objects

**Check my progress**

## Task 8. Deploy a blocked pod that mounts the host filesystem

Because the account used to deploy the GKE cluster was granted cluster-admin permissions in a previous step, it's necessary to create another separate "user" account to interact with the cluster and validate the PodSecurityPolicy enforcement.

1. To do this, run:
    

```apache
gcloud iam service-accounts create demo-developer
```

(Output)

```apache
Created service account [demo-developer].
```

2. Next, run these commands to grant these permissions to the service account - the ability to interact with the cluster and attempt to create pods:
    

```apache
MYPROJECT=$(gcloud config list --format 'value(core.project)')
```

```apache
gcloud projects add-iam-policy-binding "${MYPROJECT}" --role=roles/container.developer --member="serviceAccount:demo-developer@${MYPROJECT}.iam.gserviceaccount.com"
```

3. Obtain the service account credentials file by running:
    

```apache
gcloud iam service-accounts keys create key.json --iam-account "demo-developer@${MYPROJECT}.iam.gserviceaccount.com"
```

4. Configure `kubectl` to authenticate as this service account:
    

```apache
gcloud auth activate-service-account --key-file=key.json
```

5. To configure `kubectl` to use these credentials when communicating with the cluster, run:
    

```apache
gcloud container clusters get-credentials simplecluster --zone $MY_ZONE
```

6. Now, try to create another pod that mounts the underlying host filesystem `/` at the folder named `/rootfs` inside the container:
    

```apache
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: hostpath
spec:
  containers:
  - name: hostpath
    image: google/cloud-sdk:latest
    command: ["/bin/bash"]
    args: ["-c", "tail -f /dev/null"]
    volumeMounts:
    - mountPath: /rootfs
      name: rootfs
  volumes:
  - name: rootfs
    hostPath:
      path: /
EOF
```

7. This output validatates that it's blocked by the pod security standard:
    

```apache
Error from server (Forbidden): error when creating "STDIN": pods "hostpath" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "hostpath" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "hostpath" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "rootfs" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "hostpath" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "hostpath" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
```

8. Deploy another pod that meets the criteria of the `restrictive-psp`:
    

```apache
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: hostpath
spec:
  securityContext:
    runAsNonRoot: true  # Ensure a non-root user
    runAsUser: 1000
    fsGroup: 2000
    seccompProfile:  # Add a seccomp profile
      type: RuntimeDefault
  containers:
  - name: hostpath
    image: google/cloud-sdk:latest
    command: ["/bin/bash"]
    args: ["-c", "tail -f /dev/null"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
EOF
```

(Output)

```apache
pod/hostpath created
```

Click *Check my progress* to verify the objective.

Deploy a blocked pod that mounts the host filesystem

**Check my progress**

---

## Solution of Lab

%[https://youtu.be/WRqTWnT0L-0] 

```apache
curl -LO raw.githubusercontent.com/ePlus-DEV/storage/refs/heads/main/labs/GSP496/lab.sh
source lab.sh
```

**Script Alternative**

```apache
curl -LO https://raw.githubusercontent.com/Itsabhishek7py/GoogleCloudSkillsboost/refs/heads/main/Hardening%20Default%20GKE%20Cluster%20Configurations/abhishek.sh
sudo chmod +x abhishek.sh
./abhishek.sh
```