How to set up AWS Kubernetes Jenkins pipeline? - Part 13

How to set up AWS Kubernetes Jenkins pipeline?
▬▬▬▬▬▬ * Instruction's Guides * ▬▬▬▬▬▬
1. AWS Kubernetes Jenkins Pipeline - jhooq.com/aws-kubernetes-jenk...
2. GitHub Repo - github.com/rahulwagh/k8s-jenk...
In this lab session, we are going to set up the AWS Kubernetes Jenkins pipeline. We will be using eksctl for creating an EKS cluster on AWS.
Setting up a pipeline on AWS requires you to have an active AWS account and you can not use the free tier of AWS because you can not use the EKS cluster on the free tier of AWS.
Here are steps which we will go through -
1. Setting up AWS EC2 Machine - First we need to set up EC2 machine where we will be installing the Jenkins along with Gradle, kubectl, eksctl, docker. We can not use t2.micro instance because we will need more CPU and memory to run Jenkins and Gradle.
2. Connect to EC2 machine - After starting the EC2 machine we will use the private key-pair file to connect to EC2 instance.
3. Install JDK - As we need to install Jenkins so we need to have JDK(adoptopenjdk.net/) pre-installed onto the EC2 machine. For this lab session I will be using adopt open JDK 11.
4. Install and Setup Jenkins - In this step we are going to install Jenkins and also going to the setup admin user
5. Assign administrative privileges to Jenkins user - After installing and setting up the Jenkins on EC2 machine we need to assign root privileges so that we can execute Docker and Gradle commands inside the Jenkins pipeline
6. Install Docker - The next step would be to install docker so that we can create a docker image and push the docker image to docker hub
7. Install and Configure AWS CLI - After installing the docker we need to install and configure AWS CLI so that we can use EKSCTL to set up and start the Kubernetes cluster.
8. Install kubectl - Also along with EKSCTL we need to install Kubectl so that we can view the deployments and services
9. Install eksctl - Now we need to install eksctl so that we can set up and start the Kubernetes cluster without using AWS UI
10. Add docker hub and GITHUB credentials - As we need to clone the github repo and push the docker image to docker hub we need to store credentials of github and docker inside Jenkins.
11. Jenkins script for stages - Once we are done all the above steps we are going to write the Jenkins pipeline scripts for all the stages
▬▬▬▬▬▬ ⭐️ 🕘Timestamps ⭐️ ▬▬▬▬▬▬
2:21 Setup an AWS EC2 Instance
7:17 Connect to EC2 Instance
10:05 Install JDK on AWS EC2 Instance
11:54 Install and Setup Jenkins
17:41 Update visudo and assign administrative privileges to Jenkins user
19:24 Install Docker
25:44 Install and Setup AWS CLI
29:32 Install and Setup Kubectl
31:01 Install and Setup eksctl
33:47 Create eks cluster using eksctl
48:25 Add Docker and GitHub Credentials into Jenkins
50:25 Add Jenkins stages
52:54 Build, deploy and test CI CD pipeline
To learn more on DevOps visit - jhooq.com
Disclaimer/Policy: All the content/instructions are solely mine. The source is completely open-source.
Video is copyrighted and can not be re-distributed on any platform.
#devops #aws #jenkins #cloud #kubernetes #eksctl #docker

[@anilkommalapati6248]

I repent why i have not come across such a wonderful tutorial all these years. Brilliant Rahul. Great work and clear concise and pretty simple. Thanks a million Rahul

[@JaiboDelNorte]

It will great to see logs for POD on the different instance in your AWS each time your hit the URL, amazing video!!

[@RahulWagh]

Thanks for the idea! Generally this session tends to get very long so its really hard to keep the track of every aspect. But i will make a not for the next time

[@abhishekgowlikar]

Most wonder video I ever seen on youtube on CICD with simple explanation. Kudos

[@RahulWagh]

Glad you liked it

[@kullayappatanguturi5761]

Thanks a ton Rahul for the video, got clarified many doubts in this video GREAT EXPLANATION

[@dileepu2157]

Thank you so much brother amazing and detailed explanation. Just started to learn about devops and this was very helpful.

[@daniellet9180]

Thank you so much @Rahul for the amazing content and for starting from scratch.

[@RahulWagh]

You are welcome

[@nikunjrabadiya436]

This is the best video I found in recent day on any devops topics Thanks Rahul to make life easy :)

[@RahulWagh]

I hope it was easy to understand

[@mohanbabu9198]

Very nice explanation by detailing each step.

[@gopib4324]

I have become ur fan Boss!! u r teaching is out of this world!! Keep up ur good worK!!

[@RahulWagh]

Thank you so much 😀

[@attanpashtonculture6870]

You deserve a millions of followers.such a great tutorials. Appreciate your great work. Keep it up

[@RahulWagh]

Glad you like them!

[@user-bq1ze4zl7p]

wow...!!Crystal clear ,simple ,easy ..I was looking for this pipeline since long ,,,,first time ever seen this type of explanation about CI/CD pipeline .. you are real champion of DevOps this video boosted my confidence

[@RahulWagh]

Glad to hear that

[@tharunjulme3002]

Thanks a Bunch Rahul for sharing content

[@RahulWagh]

So nice of you

[@AkshanshGusain]

You are a life save Rahul. Awesome tutorial.

[@RahulWagh]

Happy to help

[@poonamkamboj5264]

nicely explained, Rahul. thanks, I was able to follow the steps :)

[@santoshs3783]

Thank you Rahul sir..... :) Very nice and simple steps in this video as you shown.

[@RahulWagh]

Keep watching there is more to come

[@pravim2651]

Rahul Bhai amazing videos Your my Guruji , thanks a lot ..... 🙏🙏

[@hrishikeshshrikant4230]

its very much impressive Rahul. you can make it very easy. Thank you

[@RahulWagh]

So nice of you, stay tuned for more similar sessions

[@PrashanthBharadwaj-mq5tf]

Bro... fantastic Explanation.. Got the clear picture !! :)

[@RahulWagh]

Glad to know

[@CelebsVideopidia]

Thankyou Rahul such a great step by step knowledge sharing .... Thank a lot

[@RahulWagh]

Thanks and welcome

[@derekmab7734]

Fantastic tutorial. I really enjoyed it and learned a lot. Thank you very much.

[@RahulWagh]

You're very welcome!

[@PoojaSharma-hv7yw]

I was looking for this pipeline since long. Thanks a lot . It's the best tutorial.

[@RahulWagh]

Glad to know that

[@PoojaSharma-hv7yw]

@@RahulWagh Just wanted to thank you again for this video. I cracked an interview today :)

[@RahulWagh]

Sorry for the late reply and but really glad to know you cleared the interview. Best of luck with your new job

[@kamkum2k]

Rahul, you are amazing, BEST of BEST. Thanks a lot

[@RahulWagh]

Thanks

[@krishnakrgupta]

Brilliant tutorial.

[@PrasadD-fe1fr]

Thank you so much for sharing such wonderful video bro.. Please keep posting. It is very useful video for everybody.

[@RahulWagh]

So nice of you

[@eduarmoran]

excellent video !!!

[@RahulWagh]

You welcome

[@akshaychaudhari8243]

Excellent session Boss!!

[@RahulWagh]

Glad you enjoyed it!

[@akshaychaudhari8243]

@@RahulWagh when do u have plan to make video on complete automation with Terraform, Ansible, Docker, K8s, Jenkins etc.

[@RahulWagh]

Let see when i get little more time in my schedule may be in the next month you will see

[@Nachiketde]

Which kube context are you using and how did you create that? Did you update aws-auth? any AWS role attached to the machine?

[@roshanranasinghe6690]

Great work Rahul.. Thank you..

[@RahulWagh]

Glad you liked it

[@Amanullah-lt6fq]

awesome but you forgot to explain how to connect Kubernetes cluster in Jenkins pipeline

[@preethi1530]

Best and clean explanation 👍

[@RahulWagh]

Glad you think so!

[@uradiprashanthmudiraj5942]

greate video and crestal clear explation ever👏

[@RahulWagh]

Glad it was helpful!

[@deepaksabane3068]

Awesome videos whenever I see your videos first I will like then I will see because I know u will explain every steps with details waiting for openshift lecture for deploying application

[@RahulWagh]

Thanks and welcome

[@evgen5647]

Does `kubectl apply` inside pipeline work because you configured credentials with `aws configure` previously?

[@keshavpisal6777]

Tremendous explanation 💥🤘❣️

[@RahulWagh]

Thanks a lot 😊

[@krishm5116]

Awesome sir

[@jesusvalencia3332]

Excellent!

[@vipinsoni3882]

Nice explanation

[@RahulWagh]

Thanks for liking

[@LeGabruRJN]

how do i change the final message in the rest end point?

[@WaqasAhmed-io6fn]

very nice and helpful

[@RahulWagh]

Thanks and welcome

[@sunilsubramanya3152]

Great tutorial!

[@RahulWagh]

Thank you! Cheers!

[@lakshmanakumar1594]

36:50 eks after execution of that command i got an error called...invalid apiversion version string is empty...how can I resolve the issue

[@qe6919]

Amazing video !!! When do you plan to create a video more automation focused with with terraform, Ansible, docker, EKS, Jenkins, CI/CD all combined ?

[@RahulWagh]

Soon I will prepare something around that just not getting sufficient time

[@royking7370]

Rahul, am new to devops and your videos are awesome very easy to understand, am in the US and is there anyway I can get in contact with you for some guidance, thanks

[@himanshusrivastava6540]

Thanks Really Helpful

[@RahulWagh]

Glad it helped

[@guillaumengoulle7150]

Awesome you are the best

[@RahulWagh]

Cheer’s

[@mukundkomandur4430]

Really this video is great !!

[@RahulWagh]

Glad to know that

[@fayeza8279]

instead of installing aws cli on 24:51, you could attach a role to the jenkins instances with ec2 assume role permissions, saves u the trouble of creating access keys

[@RahulWagh]

Great suggestion

[@navnathdahibhate2464]

I have completed this pipeline today, great step by step guide. Do you have guide for Jenkins master slave architecture and deploy to EKS.

[@RahulWagh]

Not yet navnath but may be in future i can prepare around it

[@shwetasingh-qi7kq]

Great tutorial , but now i am more confuse , i have my jenkins application running on Ec2 instance , and i have multiple eks clusters in different aws regions , and i want to connect to my eks cluster , for that i am using Kubernetes plugins in jenkins ans providing, but getting error. connection details in cloud configuration section. also kubctl and eksctl is installed on my local machine., so accessing Eks cluster from my local machine , you seems to skip this step , or am I missing something here?

[@RahulWagh]

Well in past I tried using Jenkins kubernetes plugin but was not satisfied with it. It’s buggy and I think it’s not being maintained by the community

[@cherukurimaniharika7031]

Thank you so much for detailed tutorial. Have a doubt! How are we connecting to AWS k8 cluster without providing any AWS creds either in jenkins or jenkins file? Can you please clarify on this. Also, Please tell me the plugin names for docker and k8 that needs to be installed.

[@RahulWagh]

You need to install the AWS CLI onto your machine so that you can connect to AWS. Also after installing the AWS CLI you need to run the command aws configure where you need to enter access key and secret id

[@healthybrain9072]

Hi Rahul. Trust you are doing great. Your video has been a good refrence for me. Question: I was just thinking about the installation of the docker using jenkins user. Would it give us same result if we spin up a new EC2 instance for installing docker? Would our k8 cluster not be affected if jenkins server has issue? Anticipating your response. Thank you

[@RahulWagh]

Thanks you liked my videos. Answering to you question and as per my experience I would keep isolation between kubernetes cluster and the EC2 instance where i have installed Docker,Jenkins. There is no problem using docker using jenkins user or install docker and jenkins to docker user group but I would always install jenkins on separate server(ec2) and kubernetes cluster on separate(EKS cluster.) If you keep the isoation the way i have explained in the video then there should not be any problem

[@palanisamy-dl9qe]

did you created any RBAC role for jenkins deployment because i'm getting error while running deployment stage like Error from server (Forbidden): deployments.apps is forbidden and also kubectl auth can-i create deployments --namespace default no

[@RahulWagh]

no i did not create any RBAC for jenkins

[@rohitgupta-ep3nk]

It was a great video

[@RahulWagh]

Glad you liked it

[@lakshmanakumar1594]

Hii...Can you make video on cluster autos calling...it'll help us so much...bcs your way of explanation is Excellent tq

[@RahulWagh]

Will try surely on cluster auto scalling

[@isaacambi1914]

It says this is 13, where is the beginning please. I want to start and follow from the beginning.

[@preethi1530]

When I build in jenkins I got an error like kubectl command not found.. But in my ubuntu I'm able to get commands

[@RahulWagh]

I think either your jenkins user does not have permission to run kubectl command or you did not install the kubectl on jenkins virtual machine

[@vijaysarde385]

Hi Rahul, Big thank you for taking time and explaining DevOps processes step by step. Appreciate your efforts. I have a one concern - I am not sure from where this "jenkins" user came in picture. Did it get created after installation of Jenkins on EC2 machine? - धन्यवाद !

[@RahulWagh]

Glad you liked it. Jenkins user is created at the time of installation

[@lakshmanakumar1594]

When we install Jenkins by default Jenkins User has been created... cd /etc/passwd In last we can see the default Jenkins User...then set the password for the user with passwd command

[@vijaysarde385]

@@lakshmanakumar1594 Thank you very much :)

[@wladimirlobato9960]

Great thank you so much

[@RahulWagh]

You welcome

[@shashankchouksey2425]

can you tell us how you connected Kubernetes please let me know plugin used

[@RahulWagh]

You just need to install kubectl utility and it will take care of the connection

[@nilavasen8631]

Dear Mr. Rahul, Hope you are doing well. Thanks a lot for your kind video. This is really helpful. May I request you to please upload a new video on an end to end Microservice application deployment in AWS EKS .. like from pushing the code in GIT to EKS Deployment ... and the complete flow will be in a pipeline calling BuildSpec.yaml ( including Docker build to create Image) , Kubernetes Manifest file , services.yaml file etc.. I will be highly grateful if you can arrange something like this !! 🙂

[@RahulWagh]

Will try to put it onto my todo list. Hopefully i will come up something around that in upcoming months

[@nilavasen8631]

@@RahulWagh Happy New Year Rahul !! Hope all is well . Kindly can you please look into this request ? I have some upcoming interviews so will be grateful if you kindly provide something on this . 🙂

[@gk4976]

Hi Rahul it's really amazing and you way to explain is very nice u made simple steps, will you pls share me playlist link for this whole task, as I am unable to find playlist ☺️☺️☺️☺️☺️

[@RahulWagh]

Here are the playlist - 1. Terraform - kzbin.info/aero/PL7iMyoQPMtAOz187ezONf7pL8oGZRobYl 2. Kubernetes - kzbin.info/aero/PL7iMyoQPMtAP4nNc7IqQncWm6DGlIee66 3. Helm chart - kzbin.info/aero/PL7iMyoQPMtANm_35XWjkNzDCcsw9vy01b

[@vishalingle7024]

Thanks 👍 🙏

[@RahulWagh]

Welcome 👍

[@nirmalanirmala-vn5wo]

Thanks 👍👍👌

[@RahulWagh]

Welcome 😊

[@isaacambi1914]

Where is the beginning

[@rahulvarma2100]

Rahul, where did you provide Kube config details in Jenkins server? How is it able to connect to & authenticate with kibe-api server from Jenkins ?

[@RahulWagh]

I have installed the kubectl utility on my machine which will take of kube config details

[@rahulvarma2100]

@@RahulWagh Yes, having the Kubernetes config details on profile will work when kubectl are run from a terminal. But how Jenkins is able to detect those details. It must know some how to look at default Kubernetes config, there must be a plugin installed in your Jenkins setup which is doing that job. In my case (unluckily windows env.), I have kubectl installed & in system path variable. From a command prompt, I'm able to query kibe-api server. But the same kubectl commands are not recognised by Jenkins. I think I miss a plugin here, but no idea which one.

[@rahulvarma2100]

@@RahulWagh Found the issue. Its because Jenkins is running as system account instead of its own/admin. Due to this, the relevant kubeconfig files are not used by Jenkins.

[@RahulWagh]

Great to know you found the issue

[@kazishafin1]

Sir , how about testing within the CI/CD pipeline ?

[@RahulWagh]

Yes off course you could write unit tests or integration test which can be run and tested along with pipeline. As the video was quite long so it was really hard to incorporate everything into one session.

[@RahulWagh]

I can see you are having trouble tagging the correct docker image name. I would recommend follow these steps and use the correct docker image name along the docker repository name which you have created on docker.hub.com . here is link on how to push correct docker image - jhooq.com/deploy-spring-boot-microservices-on-kubernetes/#step-6-create-docker-registry-at-httpshubdockercom

[@RahulWagh]

Also did you created repository with the name - kazishafin/jhooq-docker-demo on docker hub?

[@parames3039]

So. Smoothly explanation Can you please explain pipeline how to write. In your way

[@RahulWagh]

Hey paramesh thanks for the comment. sorry i did not get your question about how to write?

[@daniellet9180]

@@RahulWagh He means how to write the JenkinsFile of our pipeline

[@rajkishor49]

sir, pipeline was not able to use kubectl command . It says " rror from server (Forbidden): window.location.replace('/login?from=%2Fswagger-2.0.0.pb-v1%3Ftimeout%3D32s'); " and authentication required ! Help me out of this sir.

[@RahulWagh]

Probably something wrong with your certificates

[@anubhavsaxena5079]

bro same error....how did you solve it??

[@bababackchodji4084]

@@RahulWagh sir same error please help

[@glimglobe]

Thank you for this great tutorial. It was so clear. However, my pipeline fails during Kubernetes deployment. I get the error below + kubectl apply -f k8s-spring-boot-deployment.yml Error from server (Forbidden): window.location.replace('/login?from=%2Fswagger-2.0.0.pb-v1%3Ftimeout%3D32s'); Authentication required . Please can someone help me here?

[@RahulWagh]

Probably you are not able to pull the docker image from docker hub. Need to see the logs there could be many things behind it

[@glimglobe]

@@RahulWagh Thanks so much for your reply. I was able to fix the issue. The problem was with my kubectl version. I just had to update my cluster version using the command aws eks update-kubeconfig --name cluster-name It worked and everything built perfectly.

[@anjanishrivastava1273]

Gradle all dont use If I use maven then IMAGE CREATION and then check then image then go to ECR registry so no need to pull only push is OK

[@sebasnangsong4144]

Thank you very much Rahul. This video is just perfect and well explained.

[@RahulWagh]

Most welcome!

[@navnathdahibhate2464]

Great tutorial!

[@RahulWagh]

Glad you liked it.