Kubernetes

Data Collection Setup

Metrics are collected via the Kubernetes Host API and Kubelet API.

Blue Medora has created a LPU file for secure data collection. See the Least Privileged User Section below for detailed instructions

In-depth information around Kuberenetes Authentication mechanisms can be found here: https://kubernetes.io/docs/reference/access-authn-authz/authentication/

Network Requirements

Ports for Kubernetes Host:

• Non-SSL: 8080 (TCP) Default
• SSL: 443 (TCP) Default

Ports for Kubelet API:

• Non-SSL: 10255 (TCP) Default
• SSL: 10250 (TCP) Default

🚧

Configurable Ports

The network ports for the Kubernetes servers may differ depending on your deployment.

Least Privileged User

Credentials are supported when using a Bearer token.

Install the bindplane-monitoring role

1. Verify you are pointed to the correct cluster and your kubectl environment is setup properly

kubectl cluster-info

2. Next, run the following commands to install the bindplane-monitoring role

kubectl apply -f https://raw.githubusercontent.com/BlueMedoraPublic/bm-kube-lpu/master/bluemedora-lpu/bm-clusterrole.yaml
kubectl apply -f https://raw.githubusercontent.com/BlueMedoraPublic/bm-kube-lpu/master/bluemedora-lpu/bm-clusterrolebinding.yaml
kubectl apply -f https://raw.githubusercontent.com/BlueMedoraPublic/bm-kube-lpu/master/bluemedora-lpu/bm-role.yaml
kubectl apply -f https://raw.githubusercontent.com/BlueMedoraPublic/bm-kube-lpu/master/bluemedora-lpu/bm-rolebinding.yaml
kubectl apply -f https://raw.githubusercontent.com/BlueMedoraPublic/bm-kube-lpu/master/bluemedora-lpu/bm-serviceaccount.yaml

Obtain the Bearer Token

3. Finally, after the bindplane-monitoring role has been installed, you can get the Bearer Token needed when configuring the source in BindPlane.

You can get the bearer token by running:

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep bluemedora | awk '{print $1}')

Use this Bearer Token In the Source configuration.

🚧

Enable Role-Based Access Control (RBAC) authorization mode for Custom Installations

For manually installed Kubernetes clusters, ensure that you RBAC enabled. Most Kubernetes deployments will have this enabled by default.

Before you can install the bluemedora-k8s-lpu.tar.gz file, your cluster must have RBAC authorization enabled.

1.Make sure the apiserver has been started with --authorization-mode=RBAC

2.If you have to change your configuration in order to enable this you will also need to pass --authorization-rbac-super-user=admin replacing 'admin' with whatever your admin account is.

3.Most Kubernetes deploy tools have RBAC enabled by default. If yours was not, you will need to do some additional setup in order to get everything running smoothly. This blog post has a step by step walkthrough: http://blog.screwdriver.cd/post/150999692902/how-we-got-rbac-working-in-kubernetes

4.You will need to restart the apiserver if you had to update the configuration to enable this.

Supported Versions

Kubernetes: 1.16 - 1.20

How to Leverage BindPlane Metrics for Kubernetes HPA Setup

📘

Pre-Requisites

1. Must be using the BindPlane for Stackdriver offering

2. The Stackdriver Project being used for the BindPlane Destination must contain the GKE cluster.

3. The Stackdriver Destination in BindPlane must have the bindplane_id label enabled. For more information please read Enabling the bindplane_id in Google Stackdriver

Setup Kubernetes for GKE Metrics

1. Make sure you are a user that has admin privilege to the GKE cluster.
You can add yourself with:

kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)

2. Verify the cluster has the correct access
For GKE cluster <my_cluster>, use following command:

gcloud container clusters describe <my_cluster>
For each node pool check the section oauthScopes - there should be https://www.googleapis.com/auth/monitoring scope listed there.

3. Deploy the Stackdriver Collector
###Using the legacy resource model (Under your GKE settings, titled "Legacy Stackdriver Monitoring"), run the following:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml

Using the new resource model, run the following:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml

4. Once deployed, you can view all metrics by running:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq

5. Record the path given by this API to build the HPA configuration.
You can verify an individual metric by running the following, and gather the data that comes back

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/*/external.googleapis.com|bluemedora|generic_node|aerospike_server|namespace|memory|usage" | jq

6. Create a HPA resource.
Make sure you create a HPA using the v2beta1 API, so that external metrics can be leveraged. An example configuration is below, but will be heavily dependent on the specific metric and resource

Example Configuration

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myservice
  namespace: mynamespace
spec:
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - external:
  	metricName: external.googleapis.com|bluemedora|generic_node|aerospike_server|namespace|memory|usage
  	metricSelector:
    	matchLabels:
      	metric.labels.bindplane_id: 35691764da932e9cabd24ae124de5577
  	targetAverageValue: 300M
	type: External
  scaleTargetRef:
	apiVersion: apps/v1
	kind: Deployment
	name: myservice

Connection Parameters

Metrics

Cluster

Container

Namespace

Node

Pod

Volume