Tutorial - Automating a Web Application CI/CD Tasks with Linux Commands & Shell Scripting

 Scenario

You are the DevOps engineer in a software development company. One of the development teams is working on a web application but has issues automating a series of time-consuming manual tasks as part of the build and deployment process. The DevOps Engineering Manager asked for your help. You need to identify the best Linux commands for the different tasks that need to be handled. The ultimate goal of this assignment is to provide the development team with a single script they can use on their CI server, which will build & deploy the web application

What’s CI/CD?

Continuous integration and Continuous Development is a software development/DevOps practice where code changes are made frequently and reliably.

Clone the Git repository

1. Navigate to the code directory

cd ~/code

2. Verify you're in the correct directory

pwd

3. Clone the project

got clone https://github.com/vdespa/customer-web-portal2. Create the directory structure

4. Navigate to the new directory

cd customer-web-portal

5. List the files and directories

ls -l

6. List all the files and directories, even the files with “.”

ls -la

Create the directory structure

This directory structure will contain all of the application files. You should already be in the /home/labuser/code directory

1. Use mkdir to create the directory structure. The -p option creates additional directories

mkdir -p build/public/js

mkdir -p build/public/css

2. Verify the folders have been created correctly. The -R option displays all the subdirectories

ls -R build

Create a build script in Bash

To create the bash script will be using Vim editor and adding all the commands we used in the previous steps. The following are essential steps the script needs to complete:

• The build & customer-web-portal directories need to be removed as soon as the script starts
• The script indicates which tasks are being fulfilled by displaying a message
• The script has execution permissions needed
• The script is in debug mode

1. Navigate to the ~/code directory

cd ~/code

2. Create a new file using Vim called build-script.sh in the current directory

vim build-script.sh

• Press i to switch to insert mode

3. Reference the Bash shell at the top of the file

#!/bin/bash

4. Write a display message like below to indicate when the script has started

echo "Starting build..."

5. Remove all existing directories

rm -rf {build, customer-web-portal}

6. Clone the git repository

echo "Cloning the project"
git clone https://github.com/vdespa/customer-web-portalAdd empty index.html files to public directories

7. Create the directory structure

echo "Create the directory structure"
mkdir -p build/public/{js,css}

8. Display a message stating that the build is complete

echo "Build completed!"
exit

• Use the ESC key to exit the insert mode
• type :x to save the file and exit

9. Add the execute permission

chmod +x build-script.sh

10. Run Bash script

./build-script.sh

11. Navigate back to Vim editor and add -x to run all the commands in debug mode

#!/bin/bash -x

12. Run the script again

Add empty index.html files to the public directory

Without an empty index.html file, the web server would display all the content of the web application. Creating the empty index.html files adds an extra layer of security.

1. Navigate to the correct directory

cd ~/code

2. Navigate to the public directory

cd build/public

3. Use the touch command to create an empty file in the current directory

touch index.html

4. Use the touch command to create empty files in every subdirectory

touch js/index.html

touch css/index.html

5. Navigate back to the code directory

cd ~/code

6. Verify that all index.html files have been created, use the -R to display all subdirectories and files

ls -R build

7. Navigate to the bash script and add a task that navigates using the relative path from the current directory.

echo "Add empty index.html files to public directories"
touch build/public/index.html 
touch build/public/js/index.html 
touch build/public/css/index.html 

Add build information

To track the build information like when the build and by whom.

1. Navigate to the ~/code
2. Combine the Created: and the output date using the echo command

echo "Created: $(date)"

3. Redirect the output to the build-info.txt file

echo "Created: $(date)" > build/build-info.txt

4. Use the echo command to combine the text Created by: the environment variable USER. Append the output to the build-info.txt file.

echo "Created by: $USER" >> build/build-info.txt

5. Verify the content of the file

cat build/build-info.txt

6. Add the following to the bash script

echo "Add build information"
echo "Created: $(date)" > build/build-info.txt
echo "Created by: $USER" >> build/build-info.txt

Compile the application

The web app is written in C++, we'll use the g++ compiler command

1. Navigate to the code directory

cd ~/code

2. Compile the C++ file and store the result in the build/public folder:

g++ customer-web-portal/web-app.cpp -o build/public/web-app.cgi

3. Copy all files from the project js folder into build/public/js

cp -r customer-web-portal/js/. build/public/js

4. Copy any files from the project css folder into build/public/css

cp -r customer-web-portal/css/. build/public/css

5. Expand the Bash build script with the commands used to complete this task.

ls -l build/public

Test the application

1. Navigate to the code directory

cd ~/code

2. Run the application to see the html

build/public/web-app.cgi

3. Pipe the output to the grep command to search for “Customer Web Portal”

build/public/web-app.cgi

4. Add to the Bash script

echo "Test the application"
build/public/web-app.cgi | grep "Customer Web Portal"
if [ $? -eq 0 ]
then
  echo "Test successful."
else
  echo "Aborting build. Test failed."
  exit 1
fi

5. Run the Bash script and verify its output

./build-script.sh

Generate a new Git tag

With new releases, there should always be a unique version number associated with a Git tag. The git tag number increases when a new git tag is generated.

1. Get the most recent git tag number

git describe --tags --abbrev=0

2. Navigate to the project directory

cd ~/code/customer-web-portal

3. Retrieve the previous Git tag number & store it inside a variable

latest_git_tag="$(git describe --tags --abbrev=0)"

4. Print the contents of the variable

echo "Latest Git tag: $latest_git_tag"

3. Remove the v in front of the version. Use substring expansion.

latest_version="${latest_git_tag:1}"

4. Display the contents of the variable.

echo "Latest version: $latest_version"

5. Increment the version

next_version="$((latest_version + 1))"

6. Display the contents of the variable.

echo "Next version: $next_version"

7. Create a new Git tag:

git tag "v$next_version"

8. Add all commands to the Bash script.

echo "Generate a new Git tag"
cd ~/code/customer-web-portal
latest_git_tag="$(git describe - tags - abbrev=0)"
echo "Latest Git tag: $latest_git_tag"
latest_version="${latest_git_tag:1}"
echo "Latest version: $latest_version"
next_version="$((latest_version + 1))"
echo "Next version: $next_version"
git tag "v$next_version"

9. Navigate to the code directory and run the Bash script. After the execution, check its output.

cd ~/code
./build-script.sh

Create release notes from the remote template

1. Navigate to the code directory within the user home directory using the tilde shortcut:

cd ~/code

2. Use the wget utility to download the configuration file by providing the URL. Use the -O option to save the file to a different location and different name.

wget -O build/release-notes.txt https://gist.githubusercontent.com/vdespa/f0fdbfe2de231651fc7bcbce2e02c66d/raw/b153e22c76c7ba1d741e2ba017797221673b7e79/release-notes.template.txt

3. Use sed to replace the #DATE# placeholder in the file with the current date.

sed -i "s/#DATE#/$(date)/g" build/release-notes.txt

4. Use sed to replace the #VERSION# placeholder in the file with the new version.

sed -i "s/#VERSION#/$next_version/g" build/release-notes.txt

5. Check that the placeholders have been replaced.

cat build/release-notes.txt

6. Add all commands to the Bash script.

echo "Create release notes"
cd ~/code
wget -O build/release-notes.txt https://gist.githubusercontent.com/vdespa/f0fdbfe2de231651fc7bcbce2e02c66d/raw/b153e22c76c7ba1d741e2ba017797221673b7e79/release-notes.template.txt
sed -i "s/#DATE#/$(date)/g" build/release-notes.txt
sed -i "s/#VERSION#/$next_version/g" build/release-notes.txtList changes since the last Git tag

The developers have used the following command to get the list of changes since the last Git tag.

Get all changes since a Git tag:

git log <latest_git_tag_name>..HEAD --oneline

Use this command to get a list of changes since the last Git tag and append them to the release-notes.txt file.

Expand the Bash script with the commands used for this task.

1. Navigate to the code directory within the user home directory using the tilde shortcut:

cd ~/code

2. Use the variable in latest_git_tag in the git log command. Use the option -C to specify a different working directory for Git.

git -C customer-web-portal log $latest_git_tag..HEAD --oneline

3. Use the append operator (>>) to append the output of the git log command to the release-notes.txt file.

git -C customer-web-portal log $latest_git_tag..HEAD --oneline >> build/release-notes.txt

4. Verify the content of the release-notes.txt file.

cat build/release-notes.txt

5. Add all commands to the Bash script.

git -C customer-web-portal log $latest_git_tag..HEAD - oneline
git -C customer-web-portal log $latest_git_tag..HEAD - oneline >> build/release-notes.txt

Package application

1. Navigate to the code directory within the user home directory using the tilde shortcut:

cd ~/code

2. Create the releases directory

mkdir -p releases

3. Use the tar utility with the following options:

c — creating an archive

z — filter the archive through gzip

v — verbose operation

f — sets the name of the archive

tar -czvf releases/release-v$next_version.tar.gz build

4. Get the file size and store it in a variable

archive_size=$(ls -l releases/release-v$next_version.tar.gz | awk '{print $5}')

5. Check if the size is within the acceptable limits

if [ $archive_size -lt 50000 ] 
then 
 echo "The tar.gz file is too small"
 exit 1 
fi

5. Add all commands to the Bash script

echo "Package application"
mkdir -p releases
tar -czvf releases/release-v$next_version.tar.gz build
archive_size=$(ls -l releases/release-v$next_version.tar.gz | awk '{print $5}')
if [ $archive_size -lt 50000 ] 
then 
 echo "The tar.gz file is too small"
 exit 1 
fi

6. Run the Bash script and check its output.

./build-script.sh

And there you have it! We've successfully automated a web application CI/CD tasks using Linux & shell scripting!