Skip to the content.
Home Accounts Setup Verify Play Hacks

Tools Verify using Shell

Linux and the shell is used in this example to setup and verify the installation of the tools. Additionally, a few programming exercises are included.

DevOps

Computers and Terminals

A brief overview of Terminal and Linux is a step on your way to becoming a Linux expert. When a computer boots up, a kernel (MacOS, Windows, Linux) is started. This kernel is the core of the operating system and manages hardware resources. Above the kernel, various applications run, including the shell and terminal, which allow users to interact with the system using a basic set of commands provided by the kernel.

Typically, casual users interact with the system through a Desktop User Interface (UI) that is started by the computer’s boot-up processes. However, to interact directly with the shell, users can run a “terminal” application through the Desktop UI. Additionally, VS Code provides the ability to activate a “terminal” within its editing environment, making it convenient for developers to execute commands without leaving the code editor.

In this next phase, we will use a Jupyter notebook to perform Linux commands through a terminal. The Jupyter notebook is an application that runs above the kernel, providing an interactive environment for writing and executing code, including shell commands. This setup allows us to seamlessly integrate code execution, data analysis, and documentation in one place, enhancing our productivity and learning experience.

Setup a Personal GitHub Pages Project

You will be making a personal copy of the course repository. Be sure to have a GitHub account!!!

  • Use the Green “Use this Template” button on the portfolio_2025 repository page to set up your personal GitHub Pages repository.
  • Create a new repository.
  • Fill in the dialog and select the Repository Name to be under your GitHub ID ownership.

    create repo

  • After this is complete, use the Green “Code” button on the newly created repository page to capture your “Project Repo” name.

In the next few code cells, we will run a bash (shell) script to pull a GitHub project.

Shell Script and Variables

We will ultimately run a bash (shell) script to pull a GitHub project. This next script simply sets up the necessary environment variables to tell the script the location of repository from GitHub and where to copy the output.

For now, focus on each line that begins with export. These are shell variables. Each line has a name (after the keyword export) and a value (after the equal sign).

Here is a full description:

  • Creates a temporary file /tmp/variables.sh to store environment variables.
  • Sets the project_dir variable to your home directory with a subdirectory named nighthawk. You can change nighthawk to a different name to test your git clone.
  • Sets the project variable to a subdirectory within project_dir named portfolio_2025. You can change portfolio_2025 to the name of your project.
  • Sets the project_repo variable to the URL of the GitHub repository. Change this to the project you created from the portfolio_2025 template.

By running this script, you will prepare your environment for cloning and working on your GitHub project. This is an essential step in setting up your development environment and ensuring that all dependencies are correctly configured.

%%script bash

# Dependency Variables, set to match your project directories

cat <<EOF > /tmp/variables.sh
export project_dir=$HOME/nighthawk  # change nighthawk to different name to test your git clone
export project=\$project_dir/Keerthan_2025  # change student_2025 to name of project from git clone
export project_repo="https://github.com/Githubneos/Keerthan_2025.git"  # change to project you created from portfolio_2025 template 
EOF

Describing the Outputs of the Variables

The next script will extract the saved variables and display their values. Here is a description of the commands:

  • The source command loads the variables that we saved in the /tmp/variables.sh file in the previous code cell.
  • The echo commands display the contents of the named variables:
    • project_dir: The directory where your project is located.
    • project: The specific project directory within project_dir.
    • project_repo: The URL of the GitHub repository.

By running this script, you can verify that the environment variables are correctly set in your development environment. If they don’t match up, go back to the previous code cell and make the necessary corrections.

%%script bash

# Extract saved variables
source /tmp/variables.sh

# Output shown title and value variables
echo "Project dir: $project_dir"
echo "Project: $project"
echo "Repo: $project_repo"
Project dir: /Users/keerthan/nighthawk
Project: /Users/keerthan/nighthawk/Keerthan_2025
Repo: https://github.com/Githubneos/Keerthan_2025.git

Project Setup and Analysis with Bash Scripts

The bash scripts that follow automate what was done in the Tools Installation procedures with regards to cloning a GitHub project. Doing this in a script fashion adds the following benefits:

  • After completing these steps, we will have notes on how to set up and verify a project.
  • By reviewing these commands, you will start to learn the basics of Linux.
  • By setting up these code cells, you will be learning how to develop automated scripts using Shell programming.
  • You will learn that pretty much anything we type on a computer can be automated through the use of variables and a coding language.

Pull Code

Pull code from GitHub to your machine. This is a bash script, a sequence of commands, that will create a project directory and add the “project” from GitHub to the vscode directory. There is conditional logic to make sure that the clone only happens if it does not (!) exist. Here are some key elements in this code:

  • cd command (change directory), remember this from the terminal session.
  • if statements (conditional statements, called selection statements by College Board), code inside only happens if the condition is met.

Run the script two times and you will see that the output changes. In the second run, the files exist and it impact the flow of the code.

%%script bash

# Extract saved variables
source /tmp/variables.sh

echo "Using conditional statement to create a project directory and project"

cd ~    # start in home directory

# Conditional block to make a project directory
if [ ! -d $project_dir ]
then 
    echo "Directory $project_dir does not exist... making directory $project_dir"
    mkdir -p $project_dir
fi
echo "Directory $project_dir exists." 

# Conditional block to git clone a project from project_repo
if [ ! -d $project ]
then
    echo "Directory $project does not exist... cloning $project_repo"
    cd $project_dir
    git clone $project_repo
    cd ~
fi
echo "Directory $project exists."
Using conditional statement to create a project directory and project
Directory /Users/keerthan/nighthawk exists.
Directory /Users/keerthan/nighthawk/Keerthan_2025 exists.

Look at Files in GitHub Project

All computers contain files and directories. The clone brought more files from the cloud to your machine. Review the bash shell script, observe the commands that show and interact with files and directories. These were used during setup.

  • ls lists computer files in Unix and Unix-like operating systems.
  • cd offers a way to navigate and change the working directory.
  • pwd prints the working directory.
  • echo is used to display a line of text/string that is passed as an argument.
%%script bash

# Extract saved variables
source /tmp/variables.sh

echo "Navigate to project, then navigate to area wwhere files were cloned"
cd $project
pwd

echo ""
echo "list top level or root of files with project pulled from github"
ls

Navigate to project, then navigate to area wwhere files were cloned
/Users/keerthan/nighthawk/Keerthan_2025

list top level or root of files with project pulled from github
404.html
Gemfile
Gemfile.lock
Keerthan_2025
LICENSE
Makefile
README.md
README4YML.md
_config.yml
_includes
_layouts
_notebooks
_posts
_sass
_site
assets
images
index.md
navigation
requirements.txt
scripts
venv

Look at File List with Hidden and Long Attributes

Most Linux commands have options to enhance behavior. The enhanced listing below shows permission bits, owner of the file, size, and date.

Some useful ls flags:

  • -a: List all files including hidden files.
  • -l: List in long format.
  • -h: Human-readable file sizes.
  • -t: Sort by modification time.
  • -R: Reverse the order of the sort.

ls reference

%%script bash

# Extract saved variables
source /tmp/variables.sh

echo "Navigate to project, then navigate to area wwhere files were cloned"
cd $project
pwd

echo ""
echo "list all files in long format"
ls -al   # all files -a (hidden) in -l long listing
Navigate to project, then navigate to area wwhere files were cloned
/Users/keerthan/nighthawk/Keerthan_2025

list all files in long format
total 144
drwxr-xr-x  28 keerthan  staff    896 Aug 28 11:37 .
drwxr-xr-x   5 keerthan  staff    160 Aug 27 10:38 ..
drwxr-xr-x  15 keerthan  staff    480 Sep  2 17:43 .git
drwxr-xr-x   3 keerthan  staff     96 Aug 27 10:38 .github
-rw-r--r--   1 keerthan  staff    251 Aug 27 10:38 .gitignore
drwxr-xr-x   3 keerthan  staff     96 Aug 27 10:38 .vscode
-rw-r--r--   1 keerthan  staff    436 Aug 27 10:38 404.html
-rw-r--r--   1 keerthan  staff    122 Aug 27 10:38 Gemfile
-rw-r--r--   1 keerthan  staff   7661 Aug 28 11:37 Gemfile.lock
drwxr-xr-x  27 keerthan  staff    864 Sep  3 10:19 Keerthan_2025
-rw-r--r--@  1 keerthan  staff  11357 Aug 27 10:38 LICENSE
-rw-r--r--   1 keerthan  staff   3549 Aug 27 10:38 Makefile
-rw-r--r--   1 keerthan  staff  14171 Aug 27 10:38 README.md
-rw-r--r--   1 keerthan  staff     79 Aug 27 10:38 README4YML.md
-rw-r--r--   1 keerthan  staff    841 Aug 28 11:38 _config.yml
drwxr-xr-x  19 keerthan  staff    608 Aug 27 10:38 _includes
drwxr-xr-x   8 keerthan  staff    256 Aug 27 10:38 _layouts
drwxr-xr-x   3 keerthan  staff     96 Aug 27 10:38 _notebooks
drwxr-xr-x   3 keerthan  staff     96 Aug 27 10:38 _posts
drwxr-xr-x   6 keerthan  staff    192 Aug 27 10:38 _sass
drwxr-xr-x  20 keerthan  staff    640 Aug 29 10:03 _site
drwxr-xr-x   5 keerthan  staff    160 Aug 27 10:38 assets
drwxr-xr-x   5 keerthan  staff    160 Aug 27 10:38 images
-rw-r--r--   1 keerthan  staff   5194 Aug 29 10:36 index.md
drwxr-xr-x   5 keerthan  staff    160 Aug 27 10:38 navigation
-rw-r--r--@  1 keerthan  staff     57 Aug 27 10:38 requirements.txt
drwxr-xr-x   9 keerthan  staff    288 Aug 28 11:36 scripts
drwxr-xr-x   8 keerthan  staff    256 Aug 28 11:35 venv
%%script bash

# Extract saved variables
source /tmp/variables.sh

echo "Look for posts"
export posts=$project/_posts  # _posts inside project
cd $posts  # this should exist per fastpages
pwd  # present working directory
ls -lR  # list posts recursively
Look for posts
/Users/keerthan/nighthawk/Keerthan_2025/_posts
total 0
drwxr-xr-x  5 keerthan  staff  160 Aug 28 11:37 Foundation

./Foundation:
total 8
-rw-r--r--  1 keerthan  staff  2683 Aug 28 11:37 2024-08-21-sprint1_plan_IPYNB_2_.md
drwxr-xr-x  5 keerthan  staff   160 Aug 28 11:36 A-pair_programming
drwxr-xr-x  8 keerthan  staff   256 Aug 28 11:37 B-tools_and_equipment

./Foundation/A-pair_programming:
total 48
-rw-r--r--  1 keerthan  staff   5433 Aug 27 10:38 2023-08-16-pair_programming.md
-rw-r--r--  1 keerthan  staff   3242 Aug 28 11:36 2023-08-16-pair_showcase_IPYNB_2_.md
-rw-r--r--  1 keerthan  staff  10043 Aug 28 11:36 2023-08-17-pair_habits_IPYNB_2_.md

./Foundation/B-tools_and_equipment:
total 184
-rw-r--r--  1 keerthan  staff   8196 Aug 28 11:37 2023-08-19-devops_accounts_IPYNB_2_.md
-rw-r--r--  1 keerthan  staff   4949 Aug 28 11:37 2023-08-21-devops_tools-home_IPYNB_2_.md
-rw-r--r--  1 keerthan  staff  17367 Aug 28 11:37 2023-08-21-devops_tools-setup_IPYNB_2_.md
-rw-r--r--  1 keerthan  staff  16130 Aug 28 11:37 2023-08-22-devops_tools-verify_IPYNB_2_.md
-rw-r--r--  1 keerthan  staff  25977 Aug 28 11:37 2023-08-23-devops-githhub_pages-play_IPYNB_2_.md
-rw-r--r--  1 keerthan  staff   7506 Aug 28 11:37 2023-08-23-devops-hacks_IPYNB_2_.md
%%script bash

# Extract saved variables
source /tmp/variables.sh

echo "Look for notebooks"
export notebooks=$project/_notebooks  # _notebooks is inside project
cd $notebooks   # this should exist per fastpages
pwd  # present working directory
ls -lR  # list notebooks recursively
Look for notebooks
/Users/keerthan/nighthawk/Keerthan_2025/_notebooks
total 0
drwxr-xr-x  5 keerthan  staff  160 Aug 27 10:38 Foundation

./Foundation:
total 8
-rw-r--r--  1 keerthan  staff  3509 Aug 27 10:38 2024-08-21-sprint1_plan.ipynb
drwxr-xr-x  4 keerthan  staff   128 Aug 27 10:38 A-pair_programming
drwxr-xr-x  8 keerthan  staff   256 Aug 27 10:38 B-tools_and_equipment

./Foundation/A-pair_programming:
total 32
-rw-r--r--  1 keerthan  staff   3918 Aug 27 10:38 2023-08-16-pair_showcase.ipynb
-rw-r--r--  1 keerthan  staff  11624 Aug 27 10:38 2023-08-17-pair_habits.ipynb

./Foundation/B-tools_and_equipment:
total 224
-rw-r--r--  1 keerthan  staff   9767 Aug 27 10:38 2023-08-19-devops_accounts.ipynb
-rw-r--r--  1 keerthan  staff   5931 Aug 27 10:38 2023-08-21-devops_tools-home.ipynb
-rw-r--r--  1 keerthan  staff  22859 Aug 27 10:38 2023-08-21-devops_tools-setup.ipynb
-rw-r--r--  1 keerthan  staff  23150 Aug 27 10:38 2023-08-22-devops_tools-verify.ipynb
-rw-r--r--  1 keerthan  staff  32309 Aug 27 10:38 2023-08-23-devops-githhub_pages-play.ipynb
-rw-r--r--  1 keerthan  staff   9478 Aug 27 10:38 2023-08-23-devops-hacks.ipynb
%%script bash

# Extract saved variables
source /tmp/variables.sh

echo "Look for images, print working directory, list files"
cd $project/images  # this should exist per fastpages
pwd
ls -lR

Look inside a Markdown File

“cat” reads data from the file and gives its content as output

%%script bash

# Extract saved variables
source /tmp/variables.sh

echo "Navigate to project, then navigate to area wwhere files were cloned"

cd $project
echo "show the contents of README.md"
echo ""

cat README.md  # show contents of file, in this case markdown
echo ""
echo "end of README.md"

Env, Git, and GitHub

Env(ironment) is used to capture things like the path to the Code or Home directory. Git and GitHub are not only used to exchange code between individuals but are also often used to exchange code through servers, in our case for website deployment. All tools we use have behind-the-scenes relationships with the system they run on (MacOS, Windows, Linux) or a relationship with servers to which they are connected (e.g., GitHub). There is an “env” command in bash. There are environment files and setting files (e.g., .git/config) for Git. They both use a key/value concept.

  • env shows settings for your shell.
  • git clone sets up a directory of files.
  • cd $project allows the user to move inside that directory of files.
  • .git is a hidden directory that is used by Git to establish a relationship between the machine and the Git server on GitHub.
%%script bash

# This command has no dependencies

echo "Show the shell environment variables, key on left of equal value on right"
echo ""

env
Show the shell environment variables, key on left of equal value on right

VSCODE_CRASH_REPORTER_PROCESS_TYPE=extensionHost
GEM_HOME=/Users/keerthan/gems
TERM=xterm-color
SHELL=/bin/zsh
CLICOLOR=1
TMPDIR=/var/folders/jp/ylbd71md1pbcln8rb67lfykr0000gn/T/
HOMEBREW_REPOSITORY=/opt/homebrew
PYTHONUNBUFFERED=1
ORIGINAL_XDG_CURRENT_DESKTOP=undefined
MallocNanoZone=0
PYDEVD_USE_FRAME_EVAL=NO
PYTHONIOENCODING=utf-8
USER=keerthan
COMMAND_MODE=unix2003
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.IpcARf0Kx1/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PAGER=cat
VIRTUAL_ENV=/Users/keerthan/nighthawk/Keerthan_2025/Keerthan_2025/venv
ELECTRON_RUN_AS_NODE=1
VSCODE_AMD_ENTRYPOINT=vs/workbench/api/node/extensionHostProcess
PATH=/Users/keerthan/nighthawk/Keerthan_2025/Keerthan_2025/venv/bin:/Users/keerthan/gems/bin:/Users/keerthan/gems/bin:/Users/keerthan/gems/bin:/Users/keerthan/gems/bin:/Users/keerthan/.rbenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
__CFBundleIdentifier=com.microsoft.VSCode
PWD=/Users/keerthan/nighthawk/Keerthan_2025/Keerthan_2025/_notebooks/Foundation/B-tools_and_equipment
VSCODE_HANDLES_UNCAUGHT_ERRORS=true
MPLBACKEND=module://matplotlib_inline.backend_inline
PYTHON_FROZEN_MODULES=on
XPC_FLAGS=0x0
FORCE_COLOR=1
RBENV_SHELL=zsh
XPC_SERVICE_NAME=0
SHLVL=2
HOME=/Users/keerthan
VSCODE_NLS_CONFIG={"userLocale":"en-us","osLocale":"en-us","resolvedLanguage":"en","defaultMessagesFile":"/private/var/folders/jp/ylbd71md1pbcln8rb67lfykr0000gn/T/AppTranslocation/1579BAA9-3CD6-47FB-B7E6-FFB9909FE36D/d/Visual Studio Code 3.app/Contents/Resources/app/out/nls.messages.json","locale":"en","availableLanguages":{}}
PYDEVD_IPYTHON_COMPATIBLE_DEBUGGING=1
HOMEBREW_PREFIX=/opt/homebrew
LOGNAME=keerthan
LC_CTYPE=UTF-8
VSCODE_IPC_HOOK=/Users/keerthan/Library/Application Support/Code/1.92-main.sock
VSCODE_CODE_CACHE_PATH=/Users/keerthan/Library/Application Support/Code/CachedData/fee1edb8d6d72a0ddff41e5f71a671c23ed924b9
CLICOLOR_FORCE=1
VSCODE_PID=21960
INFOPATH=/opt/homebrew/share/info:
HOMEBREW_CELLAR=/opt/homebrew/Cellar
GIT_PAGER=cat
VSCODE_L10N_BUNDLE_LOCATION=
VSCODE_CWD=/
VIRTUAL_ENV_PROMPT=(venv) 
_=/usr/bin/env
%%script bash

# Extract saved variables
source /tmp/variables.sh

cd $project

echo ""
echo "show the secrets of .git config file"
cd .git
ls -l config

echo ""
echo "look at config file"
cat config
show the secrets of .git config file
-rw-r--r--@ 1 keerthan  staff  346 Sep  2 17:34 config

look at config file
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	ignorecase = true
	precomposeunicode = true
[remote "origin"]
	url = https://github.com/Githubneos/Keerthan_2025.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
	remote = origin
	merge = refs/heads/main
	vscode-merge-base = origin/main

Advanced Shell project

This example was requested by a student (Jun Lim, CSA). The request was to make a Jupyter file using bash; I adapted the request to markdown. This type of thought will have great extrapolation to coding and possibilities of using Lists, Arrays, or APIs to build user interfaces. JavaScript is a language where building HTML is very common.

To get more interesting output from the terminal, this will require using something like mdless (https://github.com/ttscoff/mdless). This enables seeing markdown in rendered format.

Output of the example is much nicer in “Jupyter”

This is starting the process of documentation.

%%script bash

# This example has an error in VSCode; it runs best on Jupyter
cd /tmp

file="sample.md"
if [ -f "$file" ]; then
    rm $file
fi

# Create a markdown file using tee and here document (<<EOF)
tee -a $file >/dev/null <<EOF
# Show Generated Markdown
This introductory paragraph and this line and the title above are generated using tee with the standard input (<<) redirection operator.
- This bulleted element is still part of the tee body.
EOF

# Append additional lines to the markdown file using echo and redirection (>>)
echo "- This bulleted element and lines below are generated using echo with standard output (>>) redirection operator." >> $file
echo "- The list definition, as is, is using space to separate lines. Thus the use of commas and hyphens in output." >> $file

# Define an array of actions and their descriptions
actions=("ls,list-directory" "cd,change-directory" "pwd,present-working-directory" "if-then-fi,test-condition" "env,bash-environment-variables" "cat,view-file-contents" "tee,write-to-output" "echo,display-content-of-string" "echo_text_>\$file,write-content-to-file" "echo_text_>>\$file,append-content-to-file")

# Loop through the actions array and append each action to the markdown file
for action in ${actions[@]}; do
  action=${action//-/ }  # Convert dash to space
  action=${action//,/: } # Convert comma to colon
  action=${action//_text_/ \"sample text\" } # Convert _text_ to "sample text", note escape character \ to avoid "" having meaning
  echo "    - ${action//-/ }" >> $file  # Append action to file
done

echo ""
echo "File listing and status"
ls -l $file # List file details
wc $file   # Show word count
mdless $file  # Render markdown from terminal (requires mdless installation)

rm $file  # Clean up temporary file
File listing and status
-rw-r--r--  1 keerthan  wheel  808 Sep  3 10:25 sample.md
      15     132     808 sample.md


Config file saved to /Users/keerthan/.config/mdless/config.yml




Show Generated Markdown ======================================================

This introductory paragraph and this line and the title above are generated using tee with the standard input (<<) redirection operator.

 * This bulleted element is still part of the tee body.
 * This bulleted element and lines below are generated using echo with standard output (>>) redirection operator.
 * The list definition, as is, is using space to separate lines. Thus the use of commas and hyphens in output.
   * ls: list directory
   * cd: change directory
   * pwd: present working directory
   * if then fi: test condition
   * env: bash environment variables
   * cat: view file contents
   * tee: write to output
   * echo: display content of string
   * echo "sample text" >$file: write content to file
   * echo "sample text" >>$file: append content to file



Display Shell commands help using man

The previous example used a markdown file to store a list of actions and their descriptions. This example uses the man command to generate a markdown file with descriptions of the commands. The markdown file is then displayed using mdless.

In coding, we should try to get data from the content creators instead of creating it on our own. This approach has several benefits:

  • Accuracy: Descriptions from man pages are authoritative and accurate, as they come directly from the documentation provided by the command’s developers.
  • Consistency: Automatically generating descriptions ensures consistency in formatting and terminology.
  • Efficiency: It saves time and effort, especially when dealing with a large number of commands.
  • Up-to-date Information: man pages are regularly updated with the latest information, ensuring that the descriptions are current.
%%script bash

# This example has an error in VSCode; it runs best on Jupyter
cd /tmp

file="sample.md"
if [ -f "$file" ]; then
    rm $file
fi

# Set locale to C to avoid locale-related errors
export LC_ALL=C

# Create a markdown file using tee and here document (<<EOF)
tee -a $file >/dev/null <<EOF
# Show Generated Markdown
This introductory paragraph and this line and the title above are generated using tee with the standard input (<<) redirection operator.
- This bulleted element is still part of the tee body.
EOF

# Append additional lines to the markdown file using echo and redirection (>>)
echo "- This bulleted element and lines below are generated using echo with standard output (>>) redirection operator." >> $file
echo "- The list definition, as is, is using space to separate lines. Thus the use of commas and hyphens in output." >> $file

# Define an array of commands
commands=("ls" "cat" "tail" "pwd" "env" "grep" "awk" "sed" "curl" "wget")

# Loop through the commands array and append each command description to the markdown file
for cmd in ${commands[@]}; do
  description=$(man $cmd | col -b | awk '/^NAME/{getline; print}')
  echo "    - $description" >> $file
done

echo ""
echo "File listing and status"
ls -l $file # List file details
wc $file   # Show word count
mdless $file  # Render markdown from terminal (requires mdless installation)

rm $file  # Clean up temporary file
No manual entry for wget



File listing and status
-rw-r--r--  1 keerthan  wheel  919 Sep  3 10:24 sample.md
      15     145     919 sample.md


bash: line 37: mdless: command not found