Saturday, March 18, 2023

How to Automate AKS cluster creation using Terraform and Azure DevOps | Create AKS Cluster using Terraform

How to Automate AKS cluster using Terraform and Azure Pipelines


What is Azure Kubernetes Service (AKS)

Azure Kubernetes Service (AKS) is a managed container orchestration service, based on the open source Kubernetes system, which is available on the Microsoft Azure public cloud. AKS allows you to quickly deploy a production ready Kubernetes cluster in Azure, deploy and manage containerized applications more easily with a fully managed Kubernetes service. We will see how to create AKS cluster in Azure cloud using Terraform.

AKS cluster can be created by many ways as mentioned below:

1. Create AKS cluster in Azure portal directly

2. Create AKS cluster using Azure CLI

3. Create AKS cluster using Terraform. 

Creating an AKS resource with Terraform is incredibly easy, it only requires a single resource azurerm_kubernetes_cluster and in this post, we are going to walk through the necessary steps to create this with Terraform. We will create ACR and create a role with ACRpull assignment as well

Pre-requisites:

Thursday, March 2, 2023

How to setup monitoring on AKS Cluster using Prometheus and Grafana | Setup monitoring on AKS Cluster using Prometheus and Grafana

How to setup monitoring on AKS Cluster using Prometheus and Grafana?

Prometheus Architecture

What is Prometheus?

  • Prometheus is an open source monitoring tool
  • Provides out-of-the-box monitoring capabilities for the Kubernetes container orchestration platform. It can monitor servers and databases as well.
  • Collects and stores metrics as time-series data, recording information with a timestamp 
  • It is based on pull and collects metrics from targets by scraping metrics HTTP endpoints.

What is Grafana?

  • Grafana is an open source visualization and analytics tool. 
  • It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored.

Key components:

    1. Prometheus server - Processes and stores metrics data
    2. Alert Manager - Sends alerts to any systems/channels
    3. Grafana - Visualize scraped data in UI

Installation Method:

The are are many ways you can setup Prometheus and Grafana. You can install in following ways:

1. Create all configuration files of both Prometheus and Grafana and execute them in right order.

2. Prometheus Operator - to simplify and automate the configuration and management of the Prometheus monitoring stack running on a Kubernetes cluster

3. Helm chart (Recommended) - Using helm to install Prometheus Operator including Grafana

Why to use Helm?

Helm is a package manager for Kubernetes. Helm simplifies the installation of all components in one command. Install using Helm is recommended as you will not be missing any configuration steps and very efficient. 

Pre-requisites:

Create AKS Cluster

Make sure you are login to Azure portal first.

az login

enter your Microsoft credentials.


Create a resource group first

az group create --name myResourceGroup --location southcentralus

Create AKS cluster with 2 worker nodes

az aks create --resource-group myResourceGroup --name myAKSCluster --node-count 2 --enable-addons monitoring

Display Details of Cluster

az aks show --name myAKSCluster --resource-group myResourceGroup

The above command will display Cluster details.

Connect to the cluster

az aks get-credentials --resource-group myResourceGroup --name myAKSCluster --overwrite-existing

To verify the connection to your cluster, use the kubectl get command to return a list of the cluster nodes.

kubectl get nodes

 

 


Implementation steps

We need to add the Helm Stable Charts for your local client. Execute the below command:

helm repo add stable https://charts.helm.sh/stable

# Add prometheus Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Search for newly installed repositories
helm repo list

helm search repo prometheus-community

Prometheus and grafana helm chart moved to kube prometheus stack

Create Prometheus namespace
kubectl create namespace prometheus

Install kube-prometheus-stack

Below is helm command to install kube-prometheus-stack. The helm repo kube-stack-prometheus (formerly prometheus-operator) comes with a grafana deployment embedded.

helm install stable prometheus-community/kube-prometheus-stack -n prometheus

Lets check if prometheus and grafana pods are running already

kubectl get pods -n prometheus


kubectl get svc -n prometheus


This confirms that prometheus and grafana have been installed successfully using Helm.

In order to make prometheus and grafana available outside the cluster, use LoadBalancer or NodePort instead of ClusterIP.

Edit Prometheus Service

kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus

Edit Grafana Service

kubectl edit svc stable-grafana -n prometheus

Verify if service is changed to LoadBalancer and also to get the Load Balancer URL.

kubectl get svc -n prometheus


Access Grafana UI in the browser

Get the URL from the above screenshot and put in the browser


UserName: admin
Password: prom-operator

Create Dashboard in Grafana

In Grafana, we can create various kinds of dashboards as per our needs.

How to Create Kubernetes Monitoring Dashboard?

For creating a dashboard to monitor the cluster:


Click '+' button on left panel and select ‘Import’.

Enter 12740 dashboard id under Grafana.com Dashboard.

Click ‘Load’.

Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.

Click ‘Import’.


This will show monitoring dashboard for all cluster nodes



How to Create Kubernetes Cluster Monitoring Dashboard?

For creating a dashboard to monitor the cluster:


Click '+' button on left panel and select ‘Import’.

Enter 3119 dashboard id under Grafana.com Dashboard.

Click ‘Load’.

Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.

Click ‘Import’.

This will show monitoring dashboard for all cluster nodes




Create POD Monitoring Dashboard

    For creating a dashboard to monitor the cluster:


    Click '+' button on left panel and select ‘Import’.

    Enter 6417 dashboard id under Grafana.com Dashboard.

    Click ‘Load’.



    Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.

    Click ‘Import’.


    This will show monitoring dashboard for all cluster nodes.


    Clean up AKS Cluster

    To avoid Azure charges, you should clean up the resources created. We can use the az group delete command to remove the resource group, AKS cluster, and all related resources. 

    az group delete --name myResourceGroup --yes --no-wait

    Friday, February 17, 2023

    How to Create a Docker Image for Springboot App and Push Docker image into Amazon ECR from Azure DevOps Pipelines | Azure DevOps Pipelines to Build and Push a Docker image to AWS ECR

    Amazon Elastic Container Registry (ECR) is a fully managed Docker container registry that makes it easy to store, share, and deploy container images. We will learn how to build docker image for a springboot microservices app using Azure DevOps(ADO) build pipeline and push docker image into AWS ECR.



    What are we going to do in this lab?
    1. Create a Repository in AWS ECR for storing docker images
    2. Create an IAM user and AmazonEC2ContainerRegistryFullAccess policy.
    3. Create access keys for IAM user in AWS console
    4. Create service connection in AzureDevOps to connect to AWS using IAM user access & secret keys.
    5. Create Azure DevOps Build pipeline with below tasks:
    • maven build for building JAR
    • build docker image
    • push docker image into ECR
    6. Verify if docker image has been pushed to AWS ECR

    Pre-requisites:

    Watch Steps in YouTube channel:

    Step 1 - Create a repo in ECR 

    Go to AWS console, type ECR



    Click on Create Repository

    Enter name for your repo - all lower case and Click create repository


    Create an IAM user

    Go to AWS console --> IAM --> Add Users

    Enter name for the user
    Search for EC2 and choose AmazonEC2ContainerRegistryFullAccess
    Click on Create User

    Create security credentials

    Click on user name ecr-user
    Click on Security credentials



    Create Access key

    Create Service connection 

    Go to Project settings --> Service Connections


    Enter Access keys and Secret keys

    Enter Connect name and select Grant access to all pipelines
    Click on save

    Create a classic Azure Build pipeline

    Click on use the classic editor

    Select GitHub and choose your spring-boot project and click continue



    Choose a template for the pipeline, type docker and select docker container

    Click on Apply

    Select build Agent for the Pipeline
    Choose Ubuntu latest as build agent


    Add Pipeline variables
    imageName as springboot-app
    repoName as my-springboot-repo

    Add Maven task for building springboot JAR file
    Make sure Maven task is moved up..it should be a first task

    enter maven goas as install


    Customize build an image Task 

    start customizing the task, choose the version as 2.0
    Enter $(imageName) as  Container repository 
    Select build as command from drop down
    Tags as it have shown below
    Remove push an image task



    Add ECR push task


    now configure the task

    Select as shown in screenshot
    Choose aws service connection from drop down
    select region as per your settings.
    select Image ID
    enter as $(imageName):$(Build.BuildId)


    Repo Name as $(repoName)
    $(Build.BuildId) as tar repo tag


    Save and Queue
    Select ubuntu latest as build agent..do NOT select window agents.

    Now make sure build is successful.



    Verify if Docker image has been pushed into AWS ECR

    Now login to AWS console --> Go to ECR--> select your repo. verify if image has been uploaded successfully.

    Monday, February 13, 2023

    How to integrate SonarQube with Azure DevOps | SonarQube Integration with Azure DevOps | Automate Code Scan using SonarQube In Azure Pipelines

    Please find steps below for integrating SonarQube with Azure DevOps, Previously known as Visual Studio Team Services:


    Pre-requisites:

    https://marketplace.visualstudio.com/acquisition?itemName=SonarSource.sonarqube


    Once added SonarQube plug-in, click on proceed to Organization..



    Watch Steps in YouTube channel:

    How to integrate SonarQube with Azure DevOps:

    Create Token in SonarQube to authenticate with Azure DevOps
    You need to login to SonarQube using your admin password. admin/admin123 and click on Admin on your top side.
    Click on My Account, Security. 
    Under Tokens, Give some value for token name and choose Global analysis token, click on generate Tokens. Copy the token value generated.


    Create Service Connections in Azure DevOps 

    Login to Azure DevOps. Select your project dashboard.



    Click on Project settings --> Service connections


    click on New service connection

    Type SonarQube and Click Next

    Enter SonarQube server url and enter Token created 
    Give name for service connection and select Grant access permission to all pipelines.
    Click on Save.

    Create a Pipeline in Azure DevOps

    1. Login to Azure DevOps. Go to Azure Pipelines. Click on create a new pipeline, use classic editor 
    Select your Azure Repos
    Type Maven and choose maven template as shown below


    2. Click on Add tasks
    3. Type Sonar











    4. Add Prepare Analysis on SonarQube task
    5. move up this task to all the way up.
    And also search for Java tool installer task and add it


    It should be like shown below:



    6. Click on prepare sonar analysis configuration task

    select Integrate with Maven or Gradle option

    7.Click on Java tool installer and change to 11 instead of 8


    8. Edit maven task & add install sonar:sonar and also select the path of pom.xml under MyWebApp/pom.xml



















    Click on Save and Queue to kick start build.
    9. Now login to SonarQube dashboard, click on Projects


    How to Automate AKS cluster creation using Terraform and Azure DevOps | Create AKS Cluster using Terraform

    How to Automate AKS cluster using Terraform and Azure Pipelines What is Azure Kubernetes Service (AKS) Azure Kubernetes Service (AKS) is a m...