How to setup monitoring on AKS Cluster using Prometheus and Grafana?
Prometheus Architecture
What is Prometheus?
- Prometheus is an open source monitoring tool
- Provides out-of-the-box monitoring capabilities for the Kubernetes container orchestration platform. It can monitor servers and databases as well.
- Collects and stores metrics as time-series data, recording information with a timestamp
- It is based on pull and collects metrics from targets by scraping metrics HTTP endpoints.
What is Grafana?
- Grafana is an open source visualization and analytics tool.
- It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored.
Key components:
Installation Method:
The are are many ways you can setup Prometheus and Grafana. You can install in following ways:
1. Create all configuration files of both Prometheus and Grafana and execute them in right order.
2. Prometheus Operator - to simplify and automate the configuration and management of the Prometheus monitoring stack running on a Kubernetes cluster
3. Helm chart (Recommended) - Using helm to install Prometheus Operator including Grafana
Why to use Helm?
Helm is a package manager for Kubernetes. Helm simplifies the installation of all components in one command. Install using Helm is recommended as you will not be missing any configuration steps and very efficient.
Pre-requisites:
Create AKS Cluster
Make sure you are login to Azure portal first.
az login
Create a resource group first
Create AKS cluster with 2 worker nodes
az aks create --resource-group myResourceGroup --name myAKSCluster --node-count 2 --enable-addons monitoring
Display Details of Cluster
az aks show --name myAKSCluster --resource-group myResourceGroup
The above command will display Cluster details.
Connect to the cluster
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster --overwrite-existing
To verify the connection to your cluster, use the kubectl get command to return a list of the cluster nodes.
kubectl get nodes
Connect to the cluster
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster --overwrite-existing
To verify the connection to your cluster, use the kubectl get command to return a list of the cluster nodes.
kubectl get nodes
Implementation steps
We need to add the Helm Stable Charts for your local client. Execute the below command:
helm repo add stable https://charts.helm.sh/stable
# Add prometheus Helm repohelm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# Search for newly installed repositorieshelm repo listhelm search repo prometheus-communityPrometheus and grafana helm chart moved to kube prometheus stack
We need to add the Helm Stable Charts for your local client. Execute the below command:
helm repo add stable https://charts.helm.sh/stable
Prometheus and grafana helm chart moved to kube prometheus stack
Install kube-prometheus-stack
Below is helm command to install kube-prometheus-stack. The helm repo kube-stack-prometheus (formerly prometheus-operator) comes with a grafana deployment embedded.
helm install stable prometheus-community/kube-prometheus-stack -n prometheus
Lets check if prometheus and grafana pods are running alreadykubectl get pods -n prometheus
kubectl get svc -n prometheus
This confirms that prometheus and grafana have been installed successfully using Helm.
In order to make prometheus and grafana available outside the cluster, use LoadBalancer or NodePort instead of ClusterIP.
Below is helm command to install kube-prometheus-stack. The helm repo kube-stack-prometheus (formerly prometheus-operator) comes with a grafana deployment embedded.
kubectl get pods -n prometheus
kubectl get svc -n prometheus
Edit Prometheus Service
Edit Grafana Service
kubectl edit svc stable-grafana -n prometheus
Verify if service is changed to LoadBalancer and also to get the Load Balancer URL.
Access Grafana UI in the browser
Get the URL from the above screenshot and put in the browser
UserName: admin Password: prom-operatorCreate Dashboard in Grafana
In Grafana, we can create various kinds of dashboards as per our needs.
Create Dashboard in Grafana
How to Create Kubernetes Monitoring Dashboard?
For creating a dashboard to monitor the cluster:
Click '+' button on left panel and select ‘Import’.
Enter 12740 dashboard id under Grafana.com Dashboard.
Click ‘Load’.
Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.
Click ‘Import’.
This will show monitoring dashboard for all cluster nodes
How to Create Kubernetes Cluster Monitoring Dashboard?
For creating a dashboard to monitor the cluster:
Click '+' button on left panel and select ‘Import’.
Enter 3119 dashboard id under Grafana.com Dashboard.
Click ‘Load’.
Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.
Click ‘Import’.
This will show monitoring dashboard for all cluster nodes
Create POD Monitoring Dashboard
For creating a dashboard to monitor the cluster:
Click '+' button on left panel and select ‘Import’.
Enter 12740 dashboard id under Grafana.com Dashboard.
Click ‘Load’.
Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.
Click ‘Import’.
This will show monitoring dashboard for all cluster nodes
How to Create Kubernetes Cluster Monitoring Dashboard?
For creating a dashboard to monitor the cluster:
Click '+' button on left panel and select ‘Import’.
Enter 3119 dashboard id under Grafana.com Dashboard.
Click ‘Load’.
Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.
Click ‘Import’.
This will show monitoring dashboard for all cluster nodes
Create POD Monitoring Dashboard
For creating a dashboard to monitor the cluster:
Click '+' button on left panel and select ‘Import’.
Enter 6417 dashboard id under Grafana.com Dashboard.
Click ‘Load’.
Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.
Click ‘Import’.
This will show monitoring dashboard for all cluster nodes.
Clean up AKS Cluster
To avoid Azure charges, you should clean up the resources created. We can use the az group delete command to remove the resource group, AKS cluster, and all related resources.
az group delete --name myResourceGroup --yes --no-wait
For creating a dashboard to monitor the cluster:
Click '+' button on left panel and select ‘Import’.
Enter 6417 dashboard id under Grafana.com Dashboard.
Click ‘Load’.
Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.
Click ‘Import’.
This will show monitoring dashboard for all cluster nodes.
Clean up AKS Cluster
To avoid Azure charges, you should clean up the resources created. We can use the az group delete command to remove the resource group, AKS cluster, and all related resources.
az group delete --name myResourceGroup --yes --no-wait
No comments:
Post a Comment