DevOps-Projects Documentation

File: README.md Copy Chapter

**Real-World DevOps/Cloud Projects For Learning from Beginner to Advanced** ♐

👥 **Project Ownership** Copy Chapter

🌟 **Star History** Copy Chapter

Welcome to the ultimate resource for learning DevOps through hands-on projects! This repository is designed to cater to aspiring DevOps engineers of all skill levels, from beginners taking their first steps in the field to advanced users looking to deepen their knowledge and expertise.

🧠 **Purpose of the Repository** Copy Chapter

This repository serves as a comprehensive resource for aspiring DevOps engineers to learn and implement real-world DevOps projects. It includes guides and solutions for deploying scalable systems, such as deploying a Java application on AWS using a 3-tier architecture and setting up scalable VPC architectures in the cloud.

The README files provide detailed instructions for implementing these projects, emphasizing practical deployment steps, pre-requisites, and validation processes. For example, one project focuses on deploying a Java-based login application integrated with a MySQL database, while another covers creating modular VPC network setups leveraging AWS services.

🔍 **Analysis of Features and Technologies** Copy Chapter

The repository demonstrates extensive use of DevOps concepts and tools, focusing on AWS cloud infrastructure and automation. It features technologies such as:

• EC2, RDS, VPC, Auto Scaling, IAM roles
• Maven, SonarCloud, JFrog Artifactory
• Monitoring via CloudWatch
• Custom AMIs, automation scripts

These elements showcase a robust implementation of scalable, secure, and automated systems aligned with real-world DevOps practices.

🌐 **Real-time DevOps/Cloud Projects Showcase** Copy Chapter

To improve readability and accessibility for users, this repository is also available as a modern and responsive web interface.

A website showcasing a curated list of major real-time DevOps and Cloud projects, ranging from beginner to advanced levels. Built using Next.js and styled with Tailwind CSS, this project leverages a modern starter template for fast and responsive development. Perfect for learning and exploring hands-on DevOps and Cloud concepts!

🔗 Explore the site: projects.prodevopsguytech.com

🔗 **Related AWS Projects Repository** Copy Chapter

For comprehensive AWS-specific projects and hands-on learning experiences, visit our dedicated AWS Projects repository:

AWS Projects Repository Highlights:

• Real-world AWS Projects from beginner to advanced levels
• AWS DevOps Focus with practical implementation guides
• Hands-on Learning with AWS services and best practices
• Industry-Relevant projects covering EC2, VPC, RDS, Lambda, and more
• Community Driven with active contributions and AWS expertise

🔗 Visit the AWS repository: AWS-Projects

**Repository Contents for DevOps Projects from Beginner to Advanced Levels** Copy Chapter

[!IMPORTANT]

This repository contains a comprehensive collection of DevOps projects, each meticulously crafted to provide a hands-on learning experience. The projects are categorized into different skill levels to ensure that everyone, regardless of their current expertise, can find a suitable starting point and progressively enhance their skills.

• Beginner Projects: Simple, foundational projects that introduce basic DevOps concepts and tools.
• Intermediate Projects: More complex projects that require a good understanding of DevOps fundamentals.
• Advanced Projects: Challenging projects designed to push your limits and deepen your understanding of sophisticated DevOps practices.

📊 **Complete Projects Overview Table** Copy Chapter

**Integration of DevOps Technology with Other Technologies** Copy Chapter

[!NOTE]
In the modern tech landscape, DevOps doesn't exist in isolation. It intersects with a variety of other technologies, enhancing and being enhanced by them. This repository includes projects that integrate DevOps with several key technologies, allowing you to see how these integrations work in real-world scenarios.

• Machine Learning: Implement DevOps practices to manage and deploy machine learning models efficiently.
• Version Control with Git & GitHub: Learn how to manage code versions and collaborate with others using Git and GitHub.
• CI/CD Pipelines: Set up continuous integration and continuous deployment pipelines to automate the testing and deployment of your applications.
• Cloud Platforms (AWS, Azure, GCP): Deploy applications on cloud platforms and leverage their services for scalability and reliability.
• Containerization (Docker, Kubernetes): Use container technologies to ensure that your applications run consistently across different environments.

**Project Scope** Copy Chapter

[!IMPORTANT]
The projects span a wide array of topics within the DevOps domain, each designed to provide practical experience and insights into real-world scenarios. Here’s a detailed look at the areas covered:

• Automated Deployment: Learn how to automate the deployment process to ensure that your applications are deployed quickly and reliably.
• Continuous Integration & Continuous Deployment (CI/CD): Understand how to set up and manage CI/CD pipelines to automate the testing and deployment of your code.
• Infrastructure as Code (IaC): Use tools like Terraform and CloudFormation to manage your infrastructure through code, ensuring consistency and scalability.
• Monitoring & Logging: Implement monitoring and logging solutions to keep track of your applications’ performance and troubleshoot issues.
• Security & Compliance: Learn how to incorporate security practices into your DevOps workflows to ensure that your applications are secure and compliant with regulations.
• Scalability & Performance Optimization: Understand how to scale your applications and optimize their performance to handle increasing loads.

**Why Explore This Repository?** Copy Chapter

[!NOTE]
This repository is a treasure trove of learning opportunities, tailored to help you grow in the DevOps field. Here's why you should dive in:

• Hands-on Experience: Each project is designed to provide you with practical, hands-on experience. You'll work through real-world challenges and gain the skills you need to succeed in the industry.
• Skill Enhancement: Whether you're just starting or looking to build on existing skills, the projects are structured to guide you through a learning path that will enhance your capabilities.
• Industry Relevance: Stay up-to-date with the latest trends and technologies in DevOps. The projects reflect current industry practices, ensuring that what you learn is relevant and applicable.
• Community Engagement: Join a community of like-minded learners and professionals. Share your projects, seek feedback, and collaborate on exciting DevOps initiatives.

**Code of Conduct** Copy Chapter

[!CAUTION]

We are committed to fostering a welcoming and respectful environment for all contributors. Please take a moment to review our Code of Conduct before participating in this community.

**Security Policy** Copy Chapter

[!IMPORTANT]

We prioritize security in all our DevOps projects. Please review our Security Policy to understand our security practices, vulnerability reporting process, and how we maintain secure educational environments.

**Contribute and Collaborate** Copy Chapter

[!TIP]
This repository thrives on community contributions and collaboration. Here’s how you can get involved:

• Fork the Repository: Create your own copy of the repository to work on.
• Submit Pull Requests: Contribute your projects or improvements to existing projects by submitting pull requests.
• Engage with Others: Participate in discussions, provide feedback on others’ projects, and collaborate to create better solutions.
• Share Your Knowledge: If you’ve developed a new project or learned something valuable, share it with the community. Your contributions can help others in their learning journey.

We follow best practices for contribution.

🌍 **Join the Community** Copy Chapter

[!IMPORTANT]
Be a part of our active DevOps community:

⭐ **Hit the Star!** Copy Chapter

If you find this helpful, don’t forget to give this repository a star. Your support matters! ⭐

🛠️ **Author & Community** Copy Chapter

This project is crafted by Harshhaa 💡
I’d love to hear your feedback! Feel free to share your thoughts.

📧 **Connect with me:** Copy Chapter

📢 **Stay Connected** Copy Chapter

📜 **Copyright & License** Copy Chapter

File: DevOps-Project-01/Java-Login-App/README.md Copy Chapter

Java Login App Copy Chapter

Testing

Sample Java Login application uses "UserDB" database and Table schema to store the Employee Login details. Copy Chapter

How to see list of Databases Copy Chapter

SHOW DATABASES;

How to create Database Copy Chapter

CREATE DATABASE UserDB;

How to list Tables Copy Chapter

USE UserDB;

SHOW TABLES;

How to create Table Copy Chapter

Below Query to create require TABLE schema to store Employee records Copy Chapter

CREATE TABLE Employee (
id int unsigned auto_increment not null,
first_name varchar(250),
last_name varchar(250),
email varchar(250),
username varchar(250),
password varchar(250),
regdate timestamp,
primary key (id)
);

List Table data Copy Chapter

SELECT * FROM Employee;

Describe Table schema Copy Chapter

DESCRIBE Employee;

File: DevOps-Project-01/README.md Copy Chapter

Deploy Java Application on AWS 3-Tier Architecture

Table of Contents Copy Chapter

1. Project Overview
2. Architecture Overview
3. Pre-Requisites
4. Infrastructure Setup
   • VPC and Networking
   • Security Configuration
   • Database Layer
5. Application Setup
   • Build Environment
   • Application Deployment
   • Load Balancing and Auto Scaling
6. Monitoring and Maintenance
7. Security Best Practices
8. Troubleshooting Guide
9. Contributing

Project Overview

Introduction Copy Chapter

This project demonstrates the deployment of a production-grade Java web application using AWS's robust 3-tier architecture. The implementation follows cloud-native best practices, ensuring high availability, scalability, and security across all application tiers.

Key Features Copy Section

• High Availability: Multi-AZ deployment with automated failover
• Auto Scaling: Dynamic resource allocation based on demand
• Security: Defense-in-depth approach with multiple security layers
• Monitoring: Comprehensive logging and monitoring setup
• Cost Optimization: Efficient resource utilization and management

Architecture Overview Copy Chapter

Infrastructure Components Copy Section

1. Presentation Tier (Frontend)

   • Nginx web servers in Auto Scaling Group
   • Public-facing Network Load Balancer
   • CloudFront Distribution for static content

2. Application Tier (Backend)

   • Apache Tomcat servers in Auto Scaling Group
   • Internal Network Load Balancer
   • Session management with Amazon ElastiCache

3. Data Tier

   • Amazon RDS MySQL in Multi-AZ configuration
   • Automated backups and point-in-time recovery
   • Read replicas for read-heavy workloads

Network Architecture Copy Section

• VPC Design
  • Two separate VPCs (192.168.0.0/16 and 172.32.0.0/16)
  • Public and private subnets across multiple AZs
  • Transit Gateway for inter-VPC communication

Pre-Requisites

Required Accounts and Tools Copy Chapter

1. AWS Account Setup Copy Section

• Create an AWS Free Tier Account
• Install AWS CLI v2
  bash Copy

  # For Linux
  curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
  unzip awscliv2.zip
  sudo ./aws/install
  
  # For macOS
  brew install awscli
  
  # Configure AWS CLI
  aws configure

2. Development Tools Copy Section

• Git: Version control system
  bash Copy

  # For Linux
  sudo apt-get update
  sudo apt-get install git
  
  # For macOS
  brew install git

3. CI/CD Integration Copy Section

• SonarCloud Account

  • Sign up at SonarCloud
  • Generate authentication token
  • Configure project settings:
    bash Copy

    # Add to pom.xml
    
        <sonar.projectKey>your_project_key</sonar.projectKey>
        <sonar.organization>your_organization</sonar.organization>
        <sonar.host.url>https://sonarcloud.io</sonar.host.url>

• JFrog Artifactory

  • Create account on JFrog Cloud
  • Set up Maven repository
  • Configure authentication:
    xml Copy

    <servers>
        <server>
            jfrog-artifactory
            ${env.JFROG_USERNAME}
            ${env.JFROG_PASSWORD}
        </server>
    </servers>

Infrastructure Setup

VPC and Networking Copy Chapter

1. VPC Creation Copy Section

# Create primary VPC
aws ec2 create-vpc \
    --cidr-block 192.168.0.0/16 \
    --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=PrimaryVPC}]' \
    --region us-east-1

# Create secondary VPC
aws ec2 create-vpc \
    --cidr-block 172.32.0.0/16 \
    --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=SecondaryVPC}]' \
    --region us-east-1

2. Subnet Configuration Copy Section

# Create public subnet
aws ec2 create-subnet \
    --vpc-id vpc-xxx \
    --cidr-block 192.168.1.0/24 \
    --availability-zone us-east-1a \
    --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=PublicSubnet1}]'

# Create private subnet
aws ec2 create-subnet \
    --vpc-id vpc-xxx \
    --cidr-block 192.168.2.0/24 \
    --availability-zone us-east-1b \
    --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=PrivateSubnet1}]'

3. Gateway Setup Copy Section

# Create and attach Internet Gateway
aws ec2 create-internet-gateway
aws ec2 attach-internet-gateway --vpc-id vpc-xxx --internet-gateway-id igw-xxx

# Create NAT Gateway
aws ec2 create-nat-gateway \
    --subnet-id subnet-xxx \
    --allocation-id eipalloc-xxx \
    --tag-specifications 'ResourceType=natgateway,Tags=[{Key=Name,Value=PrimaryNATGateway}]'

Security Configuration Copy Chapter

1. Security Groups Copy Section

# Create frontend security group
aws ec2 create-security-group \
    --group-name FrontendSG \
    --description "Security group for frontend servers" \
    --vpc-id vpc-xxx

# Allow inbound HTTP/HTTPS
aws ec2 authorize-security-group-ingress \
    --group-id sg-xxx \
    --protocol tcp \
    --port 80 \
    --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \
    --group-id sg-xxx \
    --protocol tcp \
    --port 443 \
    --cidr 0.0.0.0/0

2. IAM Roles and Policies Copy Section

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::your-bucket/*"
        }
    ]
}

Database Layer Copy Chapter

1. RDS Instance Creation Copy Section

aws rds create-db-instance \
    --db-instance-identifier prod-mysql \
    --db-instance-class db.t3.medium \
    --engine mysql \
    --master-username admin \
    --master-user-password "YourSecurePassword" \
    --allocated-storage 20 \
    --multi-az \
    --vpc-security-group-ids sg-xxx \
    --db-subnet-group-name your-db-subnet-group

2. Database Initialization Copy Section

-- Connect to database
mysql -h your-rds-endpoint -u admin -p

-- Create application database
CREATE DATABASE javaapp;
USE javaapp;

-- Create users table
CREATE TABLE users (
    id INT AUTO_INCREMENT PRIMARY KEY,
    username VARCHAR(50) NOT NULL UNIQUE,
    password VARCHAR(255) NOT NULL,
    email VARCHAR(100) NOT NULL UNIQUE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Create necessary indexes
CREATE INDEX idx_username ON users(username);
CREATE INDEX idx_email ON users(email);

Application Setup

Build Environment Copy Chapter

1. Maven Configuration Copy Section

<java.version>11</java.version>
        <spring.version>2.5.12</spring.version>
    
    
    <dependencies>
        
    </dependencies>
    
    
        
            
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>

2. Build Process Copy Section

# Clean and build project
mvn clean package -DskipTests

# Run tests
mvn test

# Deploy to JFrog
mvn deploy

Application Deployment Copy Chapter

1. Tomcat Configuration Copy Section

# Create tomcat.service
sudo tee /etc/systemd/system/tomcat.service << EOF
[Unit]
Description=Apache Tomcat Web Application Container
After=network.target

[Service]
Type=forking
Environment=JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid
Environment=CATALINA_HOME=/opt/tomcat
Environment=CATALINA_BASE=/opt/tomcat
Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh

User=tomcat
Group=tomcat
UMask=0007
RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target
EOF

2. Nginx Configuration Copy Section

# /etc/nginx/conf.d/app.conf
upstream backend {
    server internal-nlb-xxx.elb.amazonaws.com:8080;
}

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location /static/ {
        proxy_pass https://your-cloudfront-distribution.cloudfront.net;
    }
}

Load Balancing and Auto Scaling Copy Chapter

1. Launch Template Configuration Copy Section

aws ec2 create-launch-template \
    --launch-template-name WebServerTemplate \
    --version-description WebServerVersion1 \
    --launch-template-data '{
        "ImageId": "ami-xxx",
        "InstanceType": "t3.micro",
        "SecurityGroupIds": ["sg-xxx"],
        "UserData": "IyEvYmluL2Jhc2gKCiMgSW5zdGFsbCBOZ2lueApzdWRvIHl1bSBpbnN0YWxsIG5naW54IC15Cg=="
    }'

2. Auto Scaling Group Copy Section

aws autoscaling create-auto-scaling-group \
    --auto-scaling-group-name WebServerASG \
    --launch-template LaunchTemplateName=WebServerTemplate,Version='$Latest' \
    --min-size 2 \
    --max-size 6 \
    --desired-capacity 2 \
    --vpc-zone-identifier "subnet-xxx,subnet-yyy" \
    --target-group-arns "arn:aws:elasticloadbalancing:region:account-id:targetgroup/your-target-group/xxx" \
    --health-check-type ELB \
    --health-check-grace-period 300

Monitoring and Maintenance

CloudWatch Setup Copy Chapter

1. Metrics Configuration Copy Section

# Create custom metric for memory usage
cat << EOF > /opt/aws/scripts/memory-metrics.sh
#!/bin/bash
MEMORY_USAGE=\$(free | grep Mem | awk '{print \$3/\$2 * 100.0}')
aws cloudwatch put-metric-data \
    --metric-name MemoryUsage \
    --namespace CustomMetrics \
    --value \$MEMORY_USAGE \
    --dimensions InstanceId=\$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
EOF

# Add to crontab
echo "* * * * * /opt/aws/scripts/memory-metrics.sh" | crontab -

2. Log Management Copy Section

# Configure CloudWatch agent
cat << EOF > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
{
    "agent": {
        "metrics_collection_interval": 60,
        "run_as_user": "root"
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/opt/tomcat/logs/catalina.out",
                        "log_group_name": "/aws/tomcat/application",
                        "log_stream_name": "{instance_id}",
                        "timezone": "UTC"
                    }
                ]
            }
        }
    },
    "metrics": {
        "metrics_collected": {
            "mem": {
                "measurement": [
                    "mem_used_percent"
                ]
            },
            "swap": {
                "measurement": [
                    "swap_used_percent"
                ]
            }
        }
    }
}
EOF

Security Best Practices

1. Network Security Copy Chapter

• Implement network ACLs
• Use security groups effectively
• Enable VPC Flow Logs
• Configure AWS WAF

2. Application Security Copy Chapter

• Regular security patches
• Implement AWS Shield
• Use AWS Secrets Manager
• Enable AWS GuardDuty

3. Data Security Copy Chapter

• Enable encryption at rest
• Use SSL/TLS for data in transit
• Regular security audits
• Implement backup strategies

Troubleshooting Guide

Common Issues and Solutions Copy Chapter

1. Connection Issues Copy Section

# Check connectivity
telnet database-endpoint 3306

# Verify security group rules
aws ec2 describe-security-groups --group-ids sg-xxx

# Test load balancer health
aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:region:account-id:targetgroup/your-target-group/xxx

2. Performance Issues Copy Section

# Check CPU usage
top -bn1

# Monitor memory usage
free -m

# Check disk usage
df -h

# Monitor Tomcat threads
ps -eLf | grep java | wc -l

Contributing

How to Contribute Copy Chapter

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

Development Setup Copy Chapter

# Clone repository
git clone https://github.com/yourusername/your-repo.git

# Install dependencies
mvn install

# Run tests
mvn test

🛠️ Author & Community Copy Chapter

This project is maintained by Harshhaa 💡.
Your feedback and contributions are welcome!

📧 Connect with me:

• GitHub: @NotHarshhaa
• Blog: ProDevOpsGuy
• Telegram Community: Join Here
• LinkedIn: Harshhaa Vardhan Reddy

⭐ Support the Project Copy Chapter

If you found this project helpful, please consider:

• Starring ⭐ the repository
• Sharing it with your network
• Contributing to its improvement

📢 Stay Connected Copy Section

[!Important]
This documentation is continuously evolving. For the latest updates, please check the repository regularly.

File: DevOps-Project-02/html-web-app/README.md Copy Chapter

Edit a file, create a new file, and clone from Bitbucket in under 2 minutes

When you're done, you can delete the content in this README and update the file with details for others getting started with your repository.

We recommend that you open this README in another tab as you perform the tasks below. You can watch our video for a full demo of all the steps in this tutorial. Open the video in a new tab to avoid leaving Bitbucket.

Edit a file Copy Chapter

You'll start by editing this README file to learn how to edit a file in GitHub.

1. Click Source on the left side.
2. Click the README.md link from the list of files.
3. Click the Edit button.
4. Delete the following text: Delete this line to make a change to the README from GitHub.
5. After making your change, click Commit and then Commit again in the dialog. The commit page will open and you'll see the change you just made.
6. Go back to the Source page.

Create a file Copy Chapter

Next, you'll add a new file to this repository.

1. Click the New file button at the top of the Source page.
2. Give the file a filename of contributors.txt.
3. Enter your name in the empty file space.
4. Click Commit and then Commit again in the dialog.
5. Go back to the Source page.

Before you move on, go ahead and explore the repository. You've already seen the Source page, but check out the Commits, Branches, and Settings pages.

Clone a repository Copy Chapter

Use these steps to clone from SourceTree, our client for using the repository command-line free. Cloning allows you to work on your files locally. If you don't yet have SourceTree, download and install first. If you prefer to clone from the command line, see Clone a repository.

1. You'll see the clone button under the Source heading. Click that button.
2. Now click Check out in SourceTree. You may need to create a SourceTree account or log in.
3. When you see the Clone New dialog in SourceTree, update the destination path and name if you like to and then click Clone.
4. Open the directory you just created to see your repository files.

Now that you're more familiar with your GitHub repository, go ahead and add a new file locally. You can push your change back to GitHub with SourceTree, or you can add, commit, and push from the command line.

File: DevOps-Project-02/README.md Copy Chapter

Deploy Scalable VPC Architecture on AWS Cloud

TABLE OF CONTENTS Copy Section

1. Goal
2. Pre-Requisites
3. Pre-Deployment
4. VPC Deployment
5. Validation

Goal Copy Chapter

Deploy a Modular and Scalable Virtual Network Architecture with Amazon VPC.

Pre-Requisites Copy Chapter

1. You must be having an AWS account to create infrastructure resources on AWS cloud.
2. Source Code

Pre-Deployment Copy Chapter

Customize the application dependencies mentioned below on AWS EC2 instance and create the Golden AMI.

1. AWS CLI
2. Install Apache Web Server
3. Install Git
4. Cloudwatch Agent
5. Push custom memory metrics to Cloudwatch.
6. AWS SSM Agent

VPC Deployment Copy Chapter

1. Build VPC network ( 192.168.0.0/16 ) for Bastion Host deployment as per the architecture shown above.
2. Build VPC network ( 172.32.0.0/16 ) for deploying Highly Available and Auto Scalable application servers as per the architecture shown above.
3. Create NAT Gateway in Public Subnet and update Private Subnet associated Route Table accordingly to route the default traffic to NAT for outbound internet connection.
4. Create Transit Gateway and associate both VPCs to the Transit Gateway for private communication.
5. Create Internet Gateway for each VPC and Public Subnet associated Route Table accordingly to route the default traffic to IGW for inbound/outbound internet connection.
6. Create Cloudwatch Log Group with two Log Streams to store the VPC Flow Logs of both VPCs.
7. Enable Flow Logs for both VPCs and push the Flow Logs to Cloudwatch Log Groups and store the logs in the respective Log Stream for each VPC.
8. Create Security Group for bastion host allowing port 22 from public.
9. Deploy Bastion Host EC2 instance in the Public Subnet with EIP associated.
10. Create S3 Bucket to store application specific configuration.
11. Create Launch Template with below configuration (replacing deprecated Launch Configuration).
    1. Golden AMI
    2. Instance Type – t3.micro (updated from t2.micro for better performance)
    3. Userdata to pull the code from Git Repository to document root folder of webserver and start the httpd service.
    4. IAM Role granting access to Session Manager and to S3 bucket created in the previous step to pull the configuration. (Do not grant S3 Full Access)
    5. Security Group allowing port 22 from Bastion Host and Port 80 from Public.
    6. Key Pair
12. Create Auto Scaling Group with Min: 2 Max: 4 with two Private Subnets associated to multiple AZs for high availability.
13. Create Target Group and associate it with ASG.
14. Create Application Load Balancer (ALB) in Public Subnets for better layer 7 routing and add Target Group as target.
15. Update route53 hosted zone with CNAME record routing the traffic to ALB.

Validation Copy Chapter

1. As DevOps Engineer login to Private Instances via Bastion Host.
2. Login to AWS Session Manager and access the EC2 shell from console.
3. Browse web application from public internet browser using domain name and verify that page loaded.

🛠️ Author & Community Copy Chapter

This project is crafted by Harshhaa 💡.
I’d love to hear your feedback! Feel free to share your thoughts.

📧 Connect with me:

• GitHub: @NotHarshhaa
• Blog: ProDevOpsGuy
• Telegram Community: Join Here
• LinkedIn: Harshhaa Vardhan Reddy

⭐ Support the Project Copy Chapter

If you found this helpful, consider starring ⭐ the repository and sharing it with your network! 🚀

📢 Stay Connected Copy Section

File: DevOps-Project-03/README.md Copy Chapter

Fun with Linux for Cloud & DevOps Engineers

New to Linux? Below assignment covering all the required basics of Linux to be familiar for an DevOps engineer.

Skills Copy Chapter

Below skills are required to complete the deployment steps:

Linux User Management, Permissions, Directory Structure, File Systems, File Management

Pre-Requisites Copy Chapter

Login to AWS cloud and create Linux based EC2 instance to complete the below assignment.

Deployment Copy Chapter

1. Login to the server as super user and perform below
   1. Create users and set passwords – user1, user2, user3
   2. Create Groups – devops, aws
   3. Change primary group of user2, user3 to ‘devops’ group
   4. Add ‘aws’ group as secondary group to the ‘user1’
   5. Create the file and directory structure shown in the above diagram.
   6. Change group of /dir1, /dir7/dir10, /f2 to “devops” group
   7. Change ownership of /dir1, /dir7/dir10, /f2 to “user1” user.
2. Login as user1 and perform below
   1. Create users and set passwords – user4, user5
   2. Create Groups – app, database
3. Login as ‘user4’ and perform below
   1. Create directory – /dir6/dir4
   2. Create file – /f3
   3. Move the file from “/dir1/f1” to “/dir2/dir1/dir2”
   4. Rename the file ‘/f2′ to /f4’
4. Login as ‘user1’ and perform below
   1. Create directory – “/home/user2/dir1”
   2. Change to “/dir2/dir1/dir2/dir10” directory and create file “/opt/dir14/dir10/f1” using relative path method.
   3. Move the file from “/opt/dir14/dir10/f1” to user1 home directory
   4. Delete the directory recursively “/dir4”
   5. Delete all child files and directories under “/opt/dir14” using single command.
   6. Write this text “Linux assessment for an DevOps Engineer!! Learn with Fun!!” to the /f3 file and save it.
5. Login as ‘user2’ and perform below
   1. Create file “/dir1/f2”
   2. Delete /dir6
   3. Delete /dir8
   4. Replace the “DevOps” text to “devops” in the /f3 file without using editor.
   5. Using Vi-Editor copy the line1 and paste 10 times in the file /f3.
   6. Search for the pattern “Engineer” and replace with “engineer” in the file /f3 using single command.
   7. Delete /f3
6. Login as ‘root’ user and perform below
   1. Search for the file name ‘f3’ in the server and list all absolute paths where f3 file is found.
   2. Show the count of the number of files in the directory ‘/’
   3. Print last line of the file ‘/etc/passwd’
7. Login to AWS and create 5GB EBS volume in the same AZ of the EC2 instance and attach EBS volume to the Instance.
8. Login as ‘root’user and perform below
   1. Create File System on the new EBS volume attached in the previous step
   2. Mount the File System on /data directory
   3. Verify File System utilization using ‘df -h’ command – This command must show /data file system
   4. Create file ‘f1’ in the /data file system.
9. Login as ‘user5’ and perform below
   1. Delete /dir1
   2. Delete /dir2
   3. Delete /dir3
   4. Delete /dir5
   5. Delete /dir7
   6. Delete /f1 & /f4
   7. Delete /opt/dir14
10. Logins as ‘root’ user and perform below
11. Delete users – ‘user1, user2, user3, user4, user5’
12. Delete groups – app, aws, database, devops
13. Delete home directories of all users ‘user1, user2, user3, user4, user5’ if any exists still.
14. Unmount /data file system
15. Delete /data directory
16. Login to AWS and detach EBS volume to the EC2 Instance and delete the volume and then terminate EC2 instance.

All done? still not confident? repeat the steps!

Happy Learning!

🛠️ Author & Community Copy Chapter

This project is crafted by Harshhaa 💡.
I’d love to hear your feedback! Feel free to share your thoughts.

📧 Connect with me:

• GitHub: @NotHarshhaa
• Blog: ProDevOpsGuy
• Telegram Community: Join Here
• LinkedIn: Harshhaa Vardhan Reddy

⭐ Support the Project Copy Chapter

If you found this helpful, consider starring ⭐ the repository and sharing it with your network! 🚀

📢 Stay Connected Copy Section

File: DevOps-Project-04/README.md Copy Chapter

Production-Ready Django Deployment on AWS: Complete ECS & ECR DevOps Pipeline

This comprehensive guide demonstrates how to deploy a Django-based production application onto AWS using ECS (Elastic Container Service) and ECR (Elastic Container Registry). We'll cover the complete DevOps pipeline from containerization to deployment, including security best practices, monitoring setup, and production optimization.

📋 Table of Contents Copy Chapter

• Overview
• Prerequisites
• Project Structure
• Django Web Framework
• Docker & Containerization
• AWS ECR Setup
• AWS ECS Deployment
• Security & Best Practices
• Monitoring & Logging
• Troubleshooting
• Cost Optimization
• Alternative Deployments

🎯 Overview Copy Chapter

This project provides a complete DevOps pipeline for deploying Django applications on AWS cloud infrastructure. The solution includes:

• Containerization: Docker-based application packaging
• Registry Management: ECR for secure image storage
• Orchestration: ECS for container management
• Scalability: Auto-scaling and load balancing
• Security: IAM roles, security groups, and secrets management
• Monitoring: CloudWatch integration and health checks

📚 Prerequisites Copy Chapter

Technical Requirements Copy Section

• Python 3.9+ installed locally
• Docker Desktop or Docker Engine
• AWS Account with appropriate permissions
• AWS CLI configured with credentials
• Django framework knowledge
• Basic understanding of containers and cloud concepts

AWS Permissions Required Copy Section

• Amazon ECR: Full access
• Amazon ECS: Full access
• Amazon EC2: Full access
• IAM: Basic permissions for role creation
• CloudWatch: Read/Write permissions
• VPC: Network configuration permissions

Development Environment Setup Copy Section

# Install Django and create project
pip install django
django-admin startproject myproject
cd myproject

# Create requirements file
pip freeze > requirements.txt

# Test locally
python manage.py runserver

🏗️ Project Structure Copy Chapter

myproject/
├── myproject/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── apps/
│   ├── __init__.py
│   └── [your-apps]
├── static/
├── media/
├── templates/
├── requirements.txt
├── Dockerfile
├── docker-compose.yml
├── .dockerignore
├── .env.example
└── README.md

🐍 Django Web Framework Copy Chapter

Django is a high-level Python web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of web development, so you can focus on writing your app without needing to reinvent the wheel.

Key Features Copy Section

• Batteries-included: ORM, authentication, admin panel, forms
• Security: Built-in protection against CSRF, XSS, SQL injection
• Scalability: Designed for high-traffic applications
• Documentation: Comprehensive docs and active community
• MVT Architecture: Model-View-Template pattern

Production Considerations Copy Section

• Static files handling: Use AWS S3 + CloudFront
• Database: Amazon RDS (PostgreSQL/MySQL)
• Caching: Redis/ElastiCache
• Session storage: Redis or database
• Environment variables: AWS Secrets Manager

🐳 Docker & Containerization Copy Chapter

What is Docker? Copy Section

Docker is an open platform for developing, shipping, and running applications in containers. Containerization provides a lightweight, portable way to package applications with all their dependencies, ensuring consistency across different environments.

Benefits of Containerization Copy Section

• Portability: Run anywhere Docker is installed
• Isolation: Applications don't interfere with each other
• Scalability: Easy to scale horizontally
• Version Control: Image versioning and rollbacks
• Resource Efficiency: Shared OS kernel, lightweight

Docker Workflow Copy Section

1. Write Dockerfile: Define application environment
2. Build Image: Create container image with dependencies
3. Test Locally: Verify container works as expected
4. Push to Registry: Store in ECR for deployment
5. Deploy: Run containers in ECS

Dockerfile Best Practices Copy Section

• Use multi-stage builds for smaller images
• Leverage layer caching effectively
• Use specific base image versions
• Minimize attack surface
• Optimize for production performance

📦 AWS Elastic Container Registry (ECR) Copy Chapter

Amazon Elastic Container Registry (Amazon ECR) is a fully managed container image registry service that makes it easy to store, manage, share, and deploy your container images. ECR eliminates the need to operate your own container repositories or worry about scaling the underlying infrastructure.

Key Features Copy Section

• Fully Managed: No infrastructure to maintain
• Secure: IAM integration and encryption
• Scalable: Automatic scaling based on demand
• Integrated: Works seamlessly with ECS and EKS
• Cost-effective: Pay only for storage and data transfer
• Vulnerability Scanning: Automated security scans

ECR Repository Setup Copy Section

Step 1: Create Repository

# Using AWS CLI
aws ecr create-repository \
    --repository-name django-app \
    --image-scanning-configuration scanOnPush=true \
    --image-tag-mutability MUTABLE \
    --region us-east-1

Step 2: Configure Repository Policies

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "AllowPull",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::account-id:user/ecs-user"
      },
      "Action": [
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "ecr:BatchCheckLayerAvailability"
      ]
    }
  ]
}

ECR Image Management Copy Section

Build Docker Image

# Build the Docker image
docker build -t django-app:latest .

# Verify image creation
docker images | grep django-app

Authenticate with ECR

# Get ECR login password
aws ecr get-login-password --region us-east-1 | \
    docker login --username AWS --password-stdin \
    123456789012.dkr.ecr.us-east-1.amazonaws.com

Tag and Push Image

# Tag image for ECR
docker tag django-app:latest \
    123456789012.dkr.ecr.us-east-1.amazonaws.com/django-app:latest

# Push to ECR
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/django-app:latest

Lifecycle Policies

# Create lifecycle policy to clean up old images
aws ecr put-lifecycle-policy \
    --repository-name django-app \
    --lifecycle-policy-text 'file://lifecycle-policy.json'

lifecycle-policy.json:

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Keep last 10 images",
      "selection": {
        "tagStatus": "tagged",
        "tagPrefixList": ["v"],
        "countType": "imageCountMoreThan",
        "countNumber": 10
      },
      "action": {
        "type": "expire"
      }
    }
  ]
}

🚀 AWS Elastic Container Service (ECS) Copy Chapter

Amazon Elastic Container Service (ECS) is a highly scalable, high-performance container orchestration service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances or AWS Fargate.

ECS Launch Types Copy Section

EC2 Launch Type

• Full Control: Access to underlying EC2 instances
• Cost Optimization: Better for sustained workloads
• Customization: Can install additional software
• Networking: Direct control over networking configuration

Fargate Launch Type

• Serverless: No EC2 instances to manage
• Pay-per-use: Billing based on vCPU and memory
• Isolation: Each task gets its own isolated environment
• Simplicity: Reduced operational overhead

Key ECS Components Copy Section

1. ECS Cluster

# Create ECS cluster using AWS CLI
aws ecs create-cluster \
    --cluster-name django-cluster \
    --service-connect default \
    --region us-east-1

2. Task Definition

{
  "family": "django-task",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "executionRoleArn": "arn:aws:iam::account:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "django-container",
      "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/django-app:latest",
      "portMappings": [
        {
          "containerPort": 8000,
          "protocol": "tcp"
        }
      ],
      "environment": [
        {
          "name": "DJANGO_SETTINGS_MODULE",
          "value": "myproject.settings.production"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/django-app",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}

3. Service Definition

# Create ECS service
aws ecs create-service \
    --cluster django-cluster \
    --service-name django-service \
    --task-definition django-task \
    --desired-count 2 \
    --launch-type FARGATE \
    --network-configuration "awsvpcConfiguration={subnets=[subnet-12345,subnet-67890],securityGroups=[sg-12345],assignPublicIp=ENABLED}" \
    --deployment-configuration "maximumPercent=200,minimumHealthyPercent=100" \
    --health-check-grace-period-seconds 30

ECS Deployment Steps Copy Section

Step 1: Create VPC and Networking

# Create VPC
aws ec2 create-vpc --cidr-block 10.0.0.0/16 --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=django-vpc}]'

# Create subnets
aws ec2 create-subnet --vpc-id vpc-12345 --cidr-block 10.0.1.0/24 --availability-zone us-east-1a
aws ec2 create-subnet --vpc-id vpc-12345 --cidr-block 10.0.2.0/24 --availability-zone us-east-1b

# Create security groups
aws ec2 create-security-group --group-name django-sg --description "Security group for Django app" --vpc-id vpc-12345

Step 2: Create IAM Roles

# Create task execution role
aws iam create-role --role-name ecsTaskExecutionRole --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --role-name ecsTaskExecutionRole --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

Step 3: Register Task Definition

aws ecs register-task-definition --cli-input-json file://task-definition.json

Step 4: Create Service with Load Balancer

# Create Application Load Balancer
aws elbv2 create-load-balancer \
    --name django-alb \
    --subnets subnet-12345 subnet-67890 \
    --security-groups sg-12345

# Create target group
aws elbv2 create-target-group \
    --name django-tg \
    --protocol HTTP \
    --port 8000 \
    --vpc-id vpc-12345 \
    --target-type ip

# Create service with load balancer
aws ecs create-service \
    --cluster django-cluster \
    --service-name django-service \
    --task-definition django-task \
    --desired-count 2 \
    --launch-type FARGATE \
    --load-balancers targetGroupArn=arn:aws:elasticloadbalancing:region:account:targetgroup/django-tg,containerName=django-container,containerPort=8000

Step 5: Configure Auto Scaling

# Create auto scaling target
aws application-autoscaling register-scalable-target \
    --service-namespace ecs \
    --resource-id service/django-cluster/django-service \
    --scalable-dimension ecs:service:DesiredCount \
    --min-capacity 1 \
    --max-capacity 10

# Create scaling policy
aws application-autoscaling put-scaling-policy \
    --service-namespace ecs \
    --resource-id service/django-cluster/django-service \
    --scalable-dimension ecs:service:DesiredCount \
    --policy-name django-scale-out \
    --policy-type TargetTrackingScaling \
    --target-tracking-scaling-policy-configuration file://scaling-policy.json

🔒 Security & Best Practices Copy Chapter

Container Security Copy Section

• Use non-root users in containers
• Scan images for vulnerabilities
• Implement resource limits to prevent DoS attacks
• Use secrets management instead of environment variables

AWS Security Copy Section

• IAM Roles: Principle of least privilege
• Security Groups: Network-level security
• VPC: Isolated network environment
• Encryption: Data at rest and in transit

Django Security Settings Copy Section

# production.py
SECURE_SSL_REDIRECT = True
SECURE_HSTS_SECONDS = 31536000
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_BROWSER_XSS_FILTER = True
X_FRAME_OPTIONS = 'DENY'
CSRF_COOKIE_SECURE = True
SESSION_COOKIE_SECURE = True

📊 Monitoring & Logging Copy Chapter

CloudWatch Integration Copy Section

{
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "/ecs/django-app",
      "awslogs-region": "us-east-1",
      "awslogs-stream-prefix": "ecs",
      "awslogs-datetime-format": "%Y-%m-%dT%H:%M:%S.%fZ"
    }
  }
}

Health Checks Copy Section

# Configure health checks in task definition
"healthCheck": {
  "command": ["CMD-SHELL", "curl -f http://localhost:8000/health/ || exit 1"],
  "interval": 30,
  "timeout": 5,
  "retries": 3,
  "startPeriod": 60
}

Metrics to Monitor Copy Section

• CPU and Memory utilization
• Request/response times
• Error rates
• Database connections
• Container restart counts

🛠️ Troubleshooting Copy Chapter

Common Issues Copy Section

Container Won't Start

# Check task logs
aws logs get-log-events \
    --log-group-name /ecs/django-app \
    --log-stream-name ecs/django-container/abcdef123456

# Describe task failure
aws ecs describe-tasks --cluster django-cluster --tasks abcdef123456

Network Issues

# Check security group rules
aws ec2 describe-security-groups --group-ids sg-12345

# Test connectivity
aws ecs execute-command \
    --cluster django-cluster \
    --task abcdef123456 \
    --container django-container \
    --command "curl -I http://localhost:8000" \
    --interactive

Performance Issues

• Monitor CloudWatch metrics
• Check database connections
• Review container resource limits
• Analyze application logs

💰 Cost Optimization Copy Chapter

ECS Cost Saving Tips Copy Section

• Use Fargate Spot for non-critical workloads
• Implement auto-scaling to match demand
• Right-size containers based on actual usage
• Schedule scaling for predictable patterns

ECR Cost Management Copy Section

• Implement lifecycle policies to clean up old images
• Use image scanning only when needed
• Optimize image sizes with multi-stage builds

🔄 Alternative Deployments Copy Chapter

AWS Elastic Beanstalk Copy Section

• Simplified deployment process
• Built-in load balancing and auto-scaling
• Managed platform updates
• Less configuration required

Kubernetes (EKS) Copy Section

• More complex but highly flexible
• Portability across cloud providers
• Rich ecosystem and tooling
• Better for microservices architecture

Serverless Options Copy Section

• AWS Lambda for API endpoints
• Amplify for full-stack applications
• App Runner for containerized apps

🎉 Success! 🎉 Copy Chapter

Congratulations! You have successfully deployed your Django Application on AWS cloud using ECS and ECR with production-ready configurations.

Verification Steps Copy Section

1. Check Load Balancer DNS in browser
2. Verify SSL certificate is working
3. Test application functionality
4. Monitor CloudWatch metrics
5. Review security configurations

Post-Deployment Checklist Copy Section

• SSL/TLS certificates configured
• Database backups enabled
• Monitoring alerts set up
• Log rotation configured
• Security groups reviewed
• Auto-scaling policies tested
• Disaster recovery plan documented

Next Steps Copy Section

• Implement CI/CD pipeline with AWS CodePipeline
• Add comprehensive testing suite
• Set up staging environment
• Implement blue-green deployments
• Add performance monitoring with APM tools

Happy Learning and Happy Deploying! 🚀

🛠️ Author & Community Copy Chapter

This project is crafted by Harshhaa 💡.
I’d love to hear your feedback! Feel free to share your thoughts.

📧 Connect with me:

• GitHub: @NotHarshhaa
• Blog: ProDevOpsGuy
• Telegram Community: Join Here
• LinkedIn: Harshhaa Vardhan Reddy

⭐ Support the Project Copy Chapter

If you found this helpful, consider starring ⭐ the repository and sharing it with your network! 🚀

📢 Stay Connected Copy Section

File: DevOps-Project-05/README.md Copy Chapter

Deploy your code on a Docker Container using Jenkins on AWS

In this blog, we are going to deploy a Java Web app on a Docker Container built on an EC2 Instance through the use of Jenkins.

Agenda Copy Section

• Setup Jenkins
• Setup & Configure Maven and Git
• Integrating GitHub and Maven with Jenkins
• Setup Docker Host
• Integrate Docker with Jenkins
• Automate the Build and Deploy process using Jenkins
• Test the deployment

Prerequisites Copy Section

• AWS Account
• Git/ Github Account with the Source Code
• A local machine with CLI Access
• Familiarity with Docker and Git

Step 1: Setup Jenkins Server on AWS EC2 Instance Copy Chapter

• Setup a Linux EC2 Instance
• Install Java
• Install Jenkins
• Start Jenkins
• Access Web UI on port 8080

Log in to the Amazon management console, open EC2 Dashboard, click on the Launch Instance drop-down list, and click on Launch Instance as shown below:

Once the Launch an instance window opens, provide the name of your EC2 Instance:

Choose an Instance Type. Here you can select the type of machine, number of vCPUs, and memory that you want to have. Select t2.micro which is free-tier eligible.

For this demo, we will select an already existing key pair. You can create new key pair if you don’t have:

Now under Network Settings, Choose the default VPC with Auto-assign public IP in enable mode. Create a new Security Group, provide a name for your security group, allow ssh traffic, and custom default TCP port of 8080 which is used by Jenkins.

Rest of the settings we will keep them at default and go ahead and click on Launch Instance

On the next screen you can see a success message after the successful creation of the EC2 instance, click on Connect to instance button:

Now connect to instance wizard will open, go to SSH client tab and copy the provided chmod and SSH command:

Open any SSH Client in your local machine, take the public IP of your EC2 Instance, and add the pem key and you will be able to access your EC2 machine in my case I am using MobaXterm on Windows:

After logging in to our EC2 machine we will install Jenkins following the instructions from the official Jenkins website:
https://pkg.jenkins.io/redhat-stable/

To use this repository, run the following command Copy Section

sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo

sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key

Output:

Now let’s install epel packages for Amazon Linux AMI:

After installing epel packages, let’s install java-openjdk11:

Let’s check the version of Java now:

Now let’s install Jenkins with the below command as shown in the output:

After successful installation Let’s enable and start Jenkins service in our EC2 Instance:

Now let’s try to access the Jenkins server through our browser. For that take the public IP of your EC2 instance and paste it into your favorite browser and should see something like this:

To unlock Jenkins we need to go to the path /var/lib/jenkins/secrets/initialAdminPassword and fetch the admin password to proceed further:

Now on the Customize Jenkins page, we can go ahead and install the suggested plugins:

Now we can create our first Admin user, provide all the required data and proceed to save and continue.

Now we are ready to use our Jenkins Server.

Step 2: Integrate GitHub with Jenkins Copy Chapter

• Install Git on Jenkins Instance
• Install Github Plugin on Jenkins GUI
• Configure Git on Jenkins GUI

Let’s first install Git on our EC2 instance with the below command:

We can check the version as shown in the below screenshot:

To install the GitHub plugin lets go to our Jenkins Dashboard and click on manage Jenkins as shown:

On the next page, click on manage plugins:

Now in order to install any plugin we need to select Available Plugins, search for Github Integration, select the plugin, and finally click on Install without restart as shown below:

Now let’s configure Git on Jenkins. Go to Manage Jenkins, and click on Global Tool Configuration.

Under Git installations, provide the name Git, and under Path, we can either provide the complete path where our Git is installed on the Jenkins machine or just put any name, in my case I put Git to allow Jenkins to automatically search for Git. Then click on Save to complete the installation.

Step 3: Integrate Maven with Jenkins Copy Chapter

• Setup Maven on Jenkins Server
• Setup Environment Variables
  JAVA_HOME,M2,M2_HOME
• Install Maven Plugin
• Configure Maven and Java

To install Maven on our Jenkins Server we will switch to the /opt directory and download the Maven package:

Now we will extract the tar.gz file:

Now we will set up Environment Variables for our root user in bash_profile in order to access Maven from any location in our Server Go to the home directory of your Jenkins server and edit the bash_profile file as shown in the below steps:

In the .bash_profile file, we need to add Maven and Java paths and load these values.

To verify follow the below steps:

With this setup, we can execute maven commands from anywhere on the server:

Now we need to update the paths where Java and Maven have been installed in the Jenkins UI. We will first install the Maven Integration Plugin as shown below:

After clicking on Install without restart, go again to manage Jenkins and select Global Tool configuration to set the paths for Java and Maven.

For JAVA:

For MAVEN:

Click on save and hence we have successfully Integrated Java and Maven with Jenkins.

Step 4: Setup a Docker Host Copy Chapter

• Setup a Linux EC2 Instance
• Install Docker
• Start Docker Services
• Run Basic Docker Commands

Let's first launch an EC2 Instance. We will skip the steps here as we have already shown earlier how to create an EC2 Instance.

Below is the screenshot of our newly created EC2 Instance on which we will install Docker:

We will first install Docker on this EC2 Instance:

After the successful installation of Docker, let’s verify the version of Docker:

Also, let’s enable and start the Docker Service:

Create Tomcat Docker Container Copy Section

In my previous blog, I deployed a Java code on a Tomcat VM, here in this blog we will deploy on a Tomcat Docker container.

We will first pull the official Tomcat docker image from the Docker Hub and then run the container out of the same image.

Let’s now create a Container from the same Image with the command:

docker run -d --name tomcat-container -p 8081:8080 tomcat

The above command runs a docker container in detached mode with the name tomcat-container and we are exposing port 8081 of our host machine with port 8080 of our container and it's using the latest image of tomcat.

Let's verify the running container on our EC2 machine:

Before accessing our container from the browser we need to allow port 8081 in the Security Group of our EC2 docker-host machine.

Go to the Security group of your EC2 machine and click on Edit inbound rules:

Click on Add rule, select Custom TCP as type, and in port range put 8081–9000 in case we need it in the future, and under source select from anywhere and then click on Save rules to proceed:

Now let’s take the public IP of our Docker-host EC2 machine and with port 8081 access it from our browser:

From the above screenshot, you can see that although there is a 404 error, it also displays Apache Tomcat at the bottom which means the installation is successful however this is a known issue with the Tomcat docker image that we will fix in the next steps.

The above issue occurs because whenever we try to access the Tomcat server from the browser it will look for the files in /webapps directory which is empty and the actual files are being stored in /webapps.dist.

So in order to fix this issue we will copy all the content from webapps.dist to webapps directory and that will resolve the issue.

Let's access our tomcat container and perform the steps as shown below:

Once we are in the tomcat container go to the /webapps.dist directory and copy all the content to webapps directory:

After that we should be able to access our tomcat docker container:

Somehow if we stop this container and start another container with the same Image on a different port we will face the same issue of 404 error.
This happens because every time we launch a new container we are using a new Image as the previous container gets deleted.

This issue can be solved by creating our own Docker Image with the appropriate changes required to run our container. This can be achieved by creating a Dockerfile on which we can mention the steps required to build the docker image and from that Image, we can run the container.

Create a Customized Dockerfile for Tomcat:

To create the Dockerfile we will use the official Image of Tomcat and with it will mention the step to copy the contents from the directory /webapps.dist to /webapps:

FROM  tomcat:latest
RUN cp -R /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps

Now let's build the Docker Image using this Dockerfile using the below command:

docker build -t tomcatserver .

Let’s verify with the docker images command:

Now is the time to run the docker container out of our customized docker image which we built using our dockerfile. The command is as:

docker run -d --name tomcat-server -p 8085:8080 tomcatserver

Here you should remember that we have already allowed port range 8081–9000 in the security group of our docker EC2 Instance.

Let’s verify using the docker ps command:

Also, let's try to access the Tomcat server on port 8085 from the browser:

Hence we should be now able to launch as many times the same container without facing any issues by utilizing our customizable Dockerfile.

Step 5: Integrate Docker with Jenkins Copy Section

• Create a dockeradmin user
• Install the “Publish Over SSH” plugin
• Add Dockerhost to Jenkins “configure systems”

Let's first create a dockeradmin user and create a password for it as well.

Now let’s add this user to the Docker group with the below command:

usermod -aG docker dockeradmin

Now in order to access the server using the newly created User we need to allow password-based authentication. For that, we need to do some changes in the /etc/ssh/sshd_config file.

As you can see above we need to uncomment PasswordAuthentication yes and comment out PasswordAuthentication no.

Once we have updated the config file we need to restart the services with the command:

service sshd reload
Redirecting to /bin/systemctl reload sshd.service

Now we would be able to log in using the dockeradmin user credentials:

Now next step is to integrate Docker with Jenkins, for that we need to install the “Publish Over SSH” plugin:

Go to Manage Jenkins > Manage Plugins > Available plugins and search for publish over ssh plugin:

Click on Install without restart to install the plugin.

Now we need to configure our dockerhost in Jenkins. For that go to Manage Jenkins > Configure System:

On the next page after scrolling down you would be able to see Publish over SSH section where you need to add a new SSH server with the info as shown below:

It should be noted that it's best practice to use ssh keys however for this demo we are using password-based authentication. In the above screenshot, we have provided details of our Docker host which we created on EC2 Instance. Also, note the use of private IP under Hostname as both our Jenkins Server and Dockerhost are on the same subnet. We can also use Public IP here.

Click on Apply and Save to proceed. With this, our Docker integration with Jenkins is successfully accomplished.

Step 6: Create Jenkins Job to Build and Copy Artifacts on to Docker Host Copy Section

In this section, we would create a new job in Jenkins however we would copy from the existing job

Click on Ok to configure the job.

On the configure settings you will notice that our new job has inherited all the settings from our previous build job. However, you can change the description and other settings:

Now one important change we need to do is to delete the previous job post-build step and select Send files or execute commands over SSH option.

Under the SSH server, it will display the already created ssh server which we created in our earlier steps. Then we need to provide the path from where the WAR file will be copied and then deployed on the remote server for which we also need to provide a path under the Remote directory. Then finally Apply and Save to proceed.

Once we save the project the build will be triggered automatically due to Poll by SCM feature which we have enabled.

Now if we check the console output of our job we should see the Job has been successfully finished.

We can also verify by checking the presence webapp.war file on our docker EC2 machine:

Step 7: Update Dockerfile to copy Artifacts to launch New Container Copy Chapter

In this step, we will create a Dockerfile to include the webapp.war file to launch a new container using our Java web Application. For that, we need to copy our artifacts to the location where we have our Dockerfile.

We will create a separate directory named docker under the root user of our dockerhost inside /opt.

As you can in the above screenshot that this new docker directory is owned by root and in Jenkins configuration settings we have mentioned that the artifacts are owned by the dockeradmin user. So let’s give the ownership of this directory to dockeradmin user.

Now let's copy the Dockerfile which we created in earlier steps to this newly created docker directory and change its ownership to dockeradmin as well:

mv Dockerfile /opt/docker/
cd /opt/docker/
chown -R dockeradmin:dockeradmin /opt/docker/

ll
total 4
-rw-r--r-- 1 dockeradmin dockeradmin 89 May 10 12:08 Dockerfile

Now we need to configure our Jenkins job to change the remote directory from /home/dockeradmin to //opt//docker:

Click on Apply and Save to build the Job manually.

If the build is successful we can see the webapp.war file in the /opt/docker directory of our dockerhost:

Now in our Dockerfile, we need to mention the location of this WAR file and copy this file onto the /usr/local/tomcat/webapps location in the container.

Dockerfile:

FROM  tomcat:latest
RUN cp -R /usr/local/tomcat/webapps.dist/* /usr/local/tomcat/webapps
COPY ./*.war /usr/local/tomcat/webapps

Let’s now build a new image using this updated Dockerfile with the command:

docker build -t tomcat:v1 .

Output:

In the next step let’s now create a container out of this image with the command:

docker run -d --name tomcatv1 -p 8086:8080 tomcat:v1

Output:

Now let’s access this application from our browser using URL http://54.173.227.226:8086/webapp/

So far we have successfully copied the artifacts to our dockerhost and then manually used docker commands like docker build and docker run to deploy our application on the docker container.

Step 8: Automate Build and Deployment on Docker Container Copy Chapter

In this step, we will try to automate end to end Jenkins pipeline right from when we commit our code to GitHub, it should build it, create an artifact and then copy the artifacts to the docker host, create a docker image, and finally create a docker container to deploy the project.

For this automation to happen we need to go to our Jenkins job, select configure, and under Send files or execute commands over SSH there is Exec command field where we need to put some commands as shown below:

cd /opt/docker;
docker build -t regapp:v1 .;
docker run -d --name registerapp -p 8087:8080 regapp:v1

Now let’s do some minor changes in our code and commit the changes which will trigger the Jenkins build process and we can see the results of automation.

As soon as I made some changes in the Readme file in the GitHub repository the build got triggered in our Jenkins Job.

If the build is successful we should see our new docker image and docker container as shown in the below screenshot:

Also, we if access our new dockerized app from our browser on port 8087, the result should be something like this:

Conclusion Copy Section

In this blog, we learned how to automate the build and deploy process using GitHub, Jenkins, Docker, and AWS EC2.

Happy Learning!

🛠️ Author & Community Copy Chapter

This project is crafted by Harshhaa 💡.
I’d love to hear your feedback! Feel free to share your thoughts.

📧 Connect with me:

• GitHub: @NotHarshhaa
• Blog: ProDevOpsGuy
• Telegram Community: Join Here
• LinkedIn: Harshhaa Vardhan Reddy

⭐ Support the Project Copy Chapter

If you found this helpful, consider starring ⭐ the repository and sharing it with your network! 🚀

📢 Stay Connected Copy Section

File: DevOps-Project-06/README.md Copy Chapter

🚀 Implementation of the Entire Advanced CI/CD Pipeline with Major DevOps Tools

A comprehensive end-to-end CI/CD pipeline leveraging Terraform, Ansible, Jenkins, SonarQube, JFrog, Docker, EKS, Prometheus, and Grafana.

🧩 Pipeline Architecture Overview Copy Chapter

Below is a step-by-step implementation breakdown of the complete DevOps CI/CD process.

🔧 Step 1: Infrastructure Provisioning (Terraform)

• Provisioned VPC, Security Groups, Ansible Controller, Jenkins Master, and Agent Instances using Terraform.
  🔗 Step Details

🔐 Step 2: SSH Configuration

• Set up password-less authentication between Ansible Controller and Agent nodes.
  🔗 Step Details

⚙️ Step 3: Jenkins Setup (Ansible)

• Configured Jenkins Master and Agent nodes.
• Agent configured as Maven Build Server.
  🔗 Step Details

🔗 Step 4: Jenkins Master-Agent Integration

• Connected Jenkins Master to Agent with credentials.
  🔗 Step Details

🌐 Step 5: GitHub Integration

• Added GitHub credentials.
• Created Multibranch Pipeline Job.
  🔗 Step Details

🚨 Step 6: Webhook Trigger Setup

• Configured GitHub webhook trigger using Multibranch Scan Webhook Trigger Plugin.
  🔗 Step Details

🧪 Step 7: SonarQube Integration

• Generated access token in SonarCloud.
• Installed SonarQube scanner plugin.
• Added SonarQube server and scanner to Jenkins.
• Configured sonar-project.properties.
• Added code quality, unit test, and build stages in Jenkinsfile.
  🔗 Step-by-step Setup

📦 Step 8: JFrog Artifactory Integration

• Configured JFrog credentials.
• Installed Artifactory plugin in Jenkins.
  🔗 Details

🐳 Step 9: Docker Image & Push to JFrog

• Built Docker image from .jar.
• Pushed to JFrog Artifactory using Docker Pipeline Plugin.
• Added Docker Build & Publish stage to Jenkinsfile.
  🔗 Details

☸️ Step 10: EKS Cluster Setup

• Created EKS cluster via Terraform.
• Installed kubectl and AWS CLI in Jenkins slave.
• Configured Kube credentials using:
  text Copy

  aws eks update-kubeconfig --region <region_name> --name <cluster_name>

🔗 Details

🚀 Step 11: Kubernetes Deployment

• Pulled Docker image using Kubernetes secrets.
• Deployed it to EKS cluster using Deployment and Service resources.
• Exposed via LoadBalancer.
  🔗 Details

📊 Step 12: Monitoring (Prometheus & Grafana)

• Added Prometheus Helm repo.
• Enabled monitoring in EKS cluster.
• Changed Prometheus and Grafana services to LoadBalancer for browser access.
  🔗 Details

👨‍💻 Author & Community Copy Chapter

This project is maintained by Harshhaa 🚀

💬 Connect with Me: Copy Section

• 🐙 GitHub: @NotHarshhaa
• 📝 Blog: ProDevOpsGuy
• 💬 Telegram: Join Community
• 👔 LinkedIn: Harshhaa Vardhan Reddy

⭐ Support the Project Copy Chapter

If you found this helpful:

• ⭐ Star this repo
• 🔁 Share it with your peers
• ✅ Follow for more DevOps content

📢 Stay Updated Copy Chapter

File: DevOps-Project-07/labs/7-Monitoring-and-Alerting/README.md Copy Chapter

Monitoring and alerting

The purpose of this lab is to review and setup sample monitoring and alerting.

In this lab, you will:

• Use and review Application Insights logs and metrics
• Setup an Application Insights Availability test
• Use and review Log Analytics along with container insights