DevOps Essentials!

1. DevOps

• Devops Tutorials:CI/CD with devops tools Git,Maven,Jenkins,Docker,Docker macine,Docker Swarm,kubernetes on AWS with EKS

1.1 Why DevOps?

• Development Team (Dev / QA)
  • Task : Coding, Design, Testing (Product Development)
  • Look for agility
• Operations (Admins / Networking / DBA)
  • Task : Manage Servers, Services, Networks, Access, Logins (Infrastructure Management)
  • Look for stability
• Dev team will request for infra changes to Ops team

1.2 Waterfall Model (only one direction)

• Order
  • Requirment Gathering & Analysis
  • Design
  • Implement
  • Test
  • Deploy
  • Maintenance
• Slow, Bug fixing is slow, Not flexible to add modifications, Software is delivered only after complete software is ready
• Dev, QA & Ops team working in isolation

1.3 Agile Methodology

• CD / CT / CI
• Order
  • Continuous Development
  • Continuous Testing
  • Continuous Integration (Delivery)
  • Monitoring
• Development of small modules and testing in quick iterations
• Development & testing going on parallely

• Interatively merging and releasing

• Dev + QA teams work together but Ops team still work in isolation

1.4 DevOps Methodology

• Reduces gaps between the teams
• Faster Delivery, Quality Product, Less Resources

1.5 What is DevOps?

• Order
  • Continuous Development
  • Continuous Testing
  • Continuous Integration
  • Continuous Deployment (or Delivery i.e. Release with Manual Approval)
  • Continuous Monitoring
• Dev + QA + Ops teams work in collaboration
• Automating the workflow
• Collaboration between the teams

1.6 DevOps Tools

• Version Control System:
  • Git, SVN, TFS, Perforce
• Build Tools:
  • Maven, Ant, Gradle
• Continuous Integration Tools:
  • Jenkins, Bamboo, Hudson
• Continuous Testing Tools:
  • Delenium, Junit TestNG
• Containerisation Tools:
  • Docker, Rkt
• Configuration Management Tools:
  • Puppet, Ansible, Chef, Salt Stack
• Continuous Monitoring Tools:
  • Splunk, ELK, Nagios

1.7 Tools Covered in the Course

• Git, Maven, Jenkins, Docker, Docker machine, Docker swarm

1.8 What is Continuos Integration and Continuos Deployment?

• VCS –> Sensed by the system and trigger automated build (building the code = compiling the source code to executable code, and then packaging all the executable code into single deployable package)
• Build –> Testing
• Testing –> Deployment

2. Version Control System

2.1 What is VCS?

• Software which allows developers to work simultaneously
• For resolving conflicts
• Detailed log of changes and history
• Maintain backup for previous versions

2.2 Types of VCS

• Local VCS (for only single user)
• Centralized VCS (can be used for team to collaborate but single server & developers always have to be connected to server over network connection)
  • Example: SVN, TFS, Perforce
• Distributed VCS (remote repos and local repos, so chances of recovery is higher; work mostly happen on local repo and less susceptible to bad network connection)
  • Example: Git

2.3 What is Git?

• Branching & Merging
• Small & Fast (written in C and can run in Linux, Windows, Mac)
• Distributed (multiple backups and no single point of failure)
• Data Assurance (commit id)

• Staging Area (layer between working directory and local repo, check/format commits before they go to local repo)

• Architecture
  • Working directory
    • git add
  • Staging area (indexing area)
    • git commit
  • Local repo (physically same as working directory)
    • git push
    • git clone/pull
  • Remote repo

2.4 Install Git

sudo apt install git

2.5 Creating git local repo

` root@debiannew:~# mkdir gitprojects root@debiannew:~# cd gitprojects/ root@debiannew:~/gitprojects# mkdir project1 root@debiannew:~/gitprojects# cd project1/ root@debiannew:~/gitprojects/project1# git init `

Initialized empty Git repository in /root/gitprojects/project1/.git/

2.6 Understanding working directory, staging area and local repository

Move the changes to the staging area git add

Move the file to the local repo git commit

2.6 git log

git log

• commit id, author, date

git log file1.txt

• commits for file

git log --help

2.7 git diff file1.txt

git diff file1.txt

• will highlight the unstaged changes

git add file1.txt

git diff --staged file1.txt

• will highlight the staged changes

2.8 Removing files from wd and local repo

• remove the file from working directory and local repo git rm file3.txt

ls

git ls-files

git status

git commit -m "del the file"

• remove the file only from local repo git rm --cached file2.txt

git ls-files

git commit -m "del only from local repo"

git status

2.8 git ignore

vim .gitignore

• give filenames

` git status git add .gitignore git commit -m “ignore file added” `

2.9 git checkout

• to discard changes in working directory
• opposite of ‘git add’ which moves changes from working directory to the staging area

git checkout -- file1.txt

2.10 git reset (at file level)

• will remove the changes to the file from staging area
• i.e. unstage the changes

git reset HEAD file1.txt

2.10 git reset (at commit level)

• remove recent 2 commits

git reset HEAD~2

git status

2.11 git reset –mixed

• git reset is same as git reset --mixed as mixed is the default option

2.12 git reset –soft

• remove the recent two commits

git reset --soft HEAD~2

git status

• changes still in the staging area

2.13 git reset –hard

• remove the recent two commits

git reset --hard HEAD~2

git log --oneline

git status

• changes will be removed from staging area and working directory
• reset only used for local commits
• if commits are pushed use git revert

2.14 Branching in git

• list all the branches

git branch

git status

• creating a new branch

git checkout -b new_branch_name old_branch_name

• default old_branch_name = master

git checkout -b test

2.15 Merging in git

git checkout master

• syntax : git merge <source branch> <destination branch>

git merge test master

2.16 Conflicts in git

• manually open the file with conflict and then delete or keep the preferred changes

2.17 git stash

• temporary shelfs our incomplete changes
• so, we cam work on something else and commit the completed changes
• later come back and work on the stashed changes

• and then commit again to the local repository

• syntax : git stash

2.18 stash pop

git stash list git show stash@{0} git stash pop stash@{0}

2.19 stash apply

• can also use git stash apply instead of git stash pop
• difference is apply will not delete the stash. And stash can be applied to other branches

2.20 Partial stash

• stash only some files

git stash -p

2.21 Stashing unstaged files

git stash -p

2.22 Delete the stash

git stash clear

git stash drop stash@{0}

2.23 Remote repository Github

git remote add origin "https://github.com/hi5sahil/localrepo1.git"

• how to delete a merged branch from local

git branch -d test

• how to delete an unmerged branch from local

git branch -D test

• how to delete a branch from remote

git push origin --delete test

2.24 Pull requests on Github

• raise a pull request to merge master with test

2.25 git clone

` mkdir project2 cd project2 git clone cd dockerdemo `

2.26 git pull

• pull from origin (i.e. local repo) from master branch

git pull origin master

2.27 git fetch

• difference between git pull and git fetch
• git fetch origin will just show us the chages
• git pull origin will also merge the changes
• pull = fetch + merge

2.28 Mege conflicts on remote repository

` git pull origin master `

• collaborate will other developers to discuss what changes to keep

` git add . git commit -m “resolved the conflicts” git push origin master `

3. Build Automation Tool - Maven

• Dynamically add dependencies
• Continuous integration, continuous builds, continuous testing
• Automate the build process
• Document the project

3.1 Why and What is Maven?

• Maven Life Cycle
  • Clean : clean
  • Default : compile, package, install, deploy
  • Site : site-deploy

mvn compiler:compile

3.2 Maven Architecture

• Maven is based on project object model
• POM file is placed along with the source code
• POM file - project object model
• Maven has 3 repos - local, central, remote
• Maven downloads dependencies and plugins from the central repo to the local repo over the internet
• If Maven doesn’t finds dependencies in central repo, then it will search the custom remote repo (having the dependencies) created by the developer
• Plugins will be downloaded from central repo to the local cache and will be used to execute the task

3.3. Maven parameters

• modelVersion
• groupId
• artifactId
• plugins

4. Continuous Integration Tool (or Orchestrator) - Jenkins

• CI is a development practice that requires developers to integrate code into the shared repository at regular intervals several times a day
• Continuous Build -> Sense the changes, Pull the changes & Build the changes -> Continuous Testing –> Fail (email developers) / Pass
• Continuous Testing –> Continuous Deployment
• Continuous Integration : Continuous Build -> Continuous Testing –> Continuous Deployment

4.1 Benefits of CI

• Fixing the bugs is faster
• Less rework
• Less people
• Automation reduces risk of human errors
• More flexible for changes

4.2 What is Jenkins?

• Jenkins is a self-contained, open source automation server (or orchestrator) which can be used to automate all sorts of tasks related to building, testing and delivering or deploying the software
• Other CI Tools:
  • Jenkins
  • Bamboo
  • Travis CI
  • Buildbot
  • Gitlab CI

4.3 Features of Jenkins

• Easy to setup
• Easy to configure
• Plugins
• Extensible
• Distributed

4.4 Setup & Launch Jenkins

• Install Jenkins
• Jenkins is Plugins based tool
• Install suggested plugins

4.5 Creating first job in Jenkins

Sections

• Build
• Build Triggers
• Build Environment
• Post-Build
• Source Code Management
• General

4.6 Notifications in Jenkins

• Uses SMTP (simple mail transfer protocol) server to send emails
• Example : smtp.gmail.com
• Give SMTP authentication : give your personal emal & pwd as using gmail SMTP
• Use SSL
• SMTP port : 465
• Default Subject : $PROJECT_NAME, $BUILD_STATUS (environement variables)
• Default Content :

4.7 CI/CD Pipeline in DevOps

Stages in a CI/CD Pipeline

• Build Automation & CI
  • Compile
  • Code Review
  • Package
• Test Automation
  • Unit Test
  • Integration Tests
  • Code Coverage Analysis
• Deployment Automation
  • Test Env
  • Staging
  • PreProd and Prod

Example of DevOps CI/CD Pipeline

• Compile –> Converting the source code into executable code (example: convert .java file to .class file)
  • any changes in github will trigger the compile job
  • if no build errors or syntax error, will proceed to next job i.e. Unit Test
• UnitTest –> Execute unit test cases for the modules
  • write unite test cases and place them in the same repo as the souce code
• Package –> All executable code (100 .class files corresponsing to a 100 .java files) is converted into a single deployable file (example : .jar file or .war file etc)
  • deployable file will be deployed on different environements (Docker Containers or VMs or Physical Servers)
• Deploy on Test Env –>
• Functional Testing –>
• Deploy to Staging –>

4.8 Maven Compile job in Jenkins

mvn compile
mvn test
mvn package

• create a maven job in jenkins
• should have java, maven, git

4.9 Maven Test job in Jenkins

mvn compile
mvn test
mvn package

• Execute test cases (example - junit / selenium test caes)

4.10 Maven Package job in Jenkins

mvn compile
mvn test
mvn package

• All executable code (100 .class files corresponsing to a 100 .java files) is converted into a single deployable file (example : .jar file or .var file etc)

4.11 Create a CI/CD Pipeline in Jenkins

• Any change will be compiled, tested and deployed

4.12 Scheduling a job in Jenkins using cron tab

• watch the repository in github (every two mins)
• compare the local repo (in jenkins workspace) with remote repo
• trigger the job (pull & compile) when there is a commit

4.13 Scheduling a job in Jenkins using Poll SCM

• watch the repository in github (every two mins)
• compare the local repo (in jenkins workspace) with remote repo
• trigger the job (pull & compile) when there is a commit

4.14 Scheduling a job through Webhooks in Github

• webhooks - any event happening on Git repo will be informed to jenkins

4.15 Distributed Architecture in Jenkins

• Jenkins Master (runs on multiple OS)
  • schedule the jobs
  • display the build results
  • monitor the jobs
• Jenkins Slave (you can distribute jobs to multiple slaves and share the load from jenkins master)
  • to distribute load
  • for multiple environment (linux, mac, windows)

4.16 Adding windows slave

• Manage Jenkins –> Manage Nodes –> New Node –> Give Name & choose Permanent Agent –> Give Workspace Location, Number of executors, Label, Usage, Launch Methods (SSH, Java Webstack Method,

4.17 Adding linux slave

4.18 What is Jenkins Pipeline?

• Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into jenkins
• Intersection point between Dev team & Ops team

4.19 What is Pipeline as a code?

• Pipeline as code describes a set of features that allow Jenkins users to define pipelined job processes with code
• These features allow jenkins to discover, manage and run jobs for muliple source repositories and branches - eliminating the need for manual job creation and management

4.20 Creating job using Pipeline as a code

• writing code (pipeline DSL - groovy based) to create this job

4.21 What is jenkins file?

• Pipile as code is implemented via jenkins file, Pipeline domain specific language which is groovy based
• It is a text file (Jenkinsfile) committed to a projects source control repository
• Jenkins File stored on source code management system or on Jenkins UI

4.22 benefits of Jenkins file on SCM

• Automatically creates a pipeline build process
• Code review/iteration on the pipeline
• Audit trail for the pipeline
• Single source of truth

4.23 Multiconfiguration project type in Jenkins

• Testing on multiple environements (linux, windows, mac) or different configurations for the same environment

5. Containerisation Tool - DOCKER

5.1 What is Docker?

• It is container technology launched in 2013 as an open source Docker Engine

5.2 VMs versus Containers

• VMs : Server –> Host OS –> Hypervisor (VMware, Oracle Virtual Box)
  • Every VM has its on OS
  • Consumes more resources of the Host
  • VMs operating system is independent of the Host operating system
  • Virtualisation is done to the hardware
  • Turning one server into multiple server
  • VMs are slow to boot
  • VMs take 10s of GBs (have their own OS, binary files etc)
• Containers : Server –> Host OS –> Docker Engine (Containerisation Software)
  • Container don’t have their own operating system and will share the operating system of the host
  • Lightweight compared to VMs as no separate OS
  • Virtualisation is done at the OS level
  • Containers are an abstraction at the application layer and share the same OS
  • Container images are 10s of MBs

5.3 What are Docker containers?

• A container is a standard unit of software
• It is a light weight standalone executable package of software
• Container isolate the software from the environment

5.4 Docker Architecture

• Docker Client (commands : docker build, docker pull, docker run)
• Docker Host
  • Docker Daemon (containers and images)
• Docker Registry
  • Docker Hub

yum install docker
docker run image_name

5.4 Container Life Cycle

docker --version
docker images
docker ps
docker ps -a
docker run -it unbuntu
exit
docker start container_name
docker stop container_name
docker rm container_name
docker rmi image_name
docker run -itd ubuntu

5.5 What are Docker volumes?

• Save or backup the data inside the container even after container is removed

docker volume create vol1

5.6 Attaching volume to a container

docker run -it --name container1 -v vol1:/data ubuntu

5.7 Deploying docker containers via jenkins

• Deploy Job (deploy the packaged .jar or .war file)
• Write a docker file to create a docker iamge

rm -rf jenkins-dockerimages
mkdir jenkins-dockerimages
cd jenkins-dockerimages
cp <path of war file>
touch dockerfile
cat <<EOT>> dockerfile
From tomcat
ADD filename.war <location in tomcat>
CMD "catalina.sh"
EXPOSE 8080
EOT
sudo docker build -t mydeploy:$BUILD_NUMBER
sudo docker run -itd -P mydeploy:$BUILD_NUMBER

6. Docker Swarm

• Container Orchestration & Clustering Tool
• Docker Swarm is a technique which can manage multiple containers on a cluster of docker host
• Docker host is the VM where docker engine is running

7. Docker Machine

• Using docker-machine we can create docker hosts on local computer, or on company’s datacenter or on cloud providers
• To provision and manage mutiple docker hosts
• To provision docker swarm clusters

8. Container Orchestrator - Kubernetes with Amazon EKS

8.1 What is Kubernetes?

• K8S is an open source project developed by Google now maintained by CNCF
• Kubernetes is a Container Orchestration tool which can manage containers on a cluster of nodes
• Other top container orchestrator
  • Docker Swarm
  • Apache Mesos
• Kubernetes is most popular and advanced. Can manage different containers - docker & rocket

8.2 Container Orchestration

• helps in scheduling the containers
• it provides HA for apps
• resilient systems
• scaling solution
• self healing
• automatic rolling updates and rollbacks

8.3 What is Amazon EKS?

• Amazon EKS is Amazon Elastic Kubernetes Service
• It is a managed service by AWS that helps in running Kubernetes on AWS by managing the control plane on the cluster