An Overview of Linux

The Linux command line will always hold a special place in my heart. I love the black screen. I love the rolling waves of text when updating packages. I love the command syntax that can be written in so many different ways it feels like a living breathing language. Still, linux can be intimidating. In an attempt to help with that, I’ve taken some of the notes I’ve gathered over time and organized them here.

Introduction

You could write an entire book on the amount of knowledge there is to cover on Linux - and many people have. I spent a lot of time gathering and compiling all the information you’ll find here. While not a complete guide to linux, there is a massive amount of information in this guide. As such, I highly recommend you make use of the table of contents on the right hand side.

I’m a big fan of learnlinux.tv as a website. Jay makes absolutely great content for everything linux. If you’ve never installed a linux system before, here is his guide on Ubuntu linux so you have a baseline to work with.

Linux Distributions

Two of the concepts I wanted to cover are distributions and streams. Linux comes from the open source community, meaning it was created for free as an operating system. Since it was free and the code was available, many people created their own versions or forks of linux. We call these different coding forks distributions. The interesting thing is it didn’t stop there. People then forked these distributions, which made forks of forks. When you have a chain of distributions which depend on each other we call this a stream. You can refer to a linux distribution as being upstream or downstream of another distribution. Let’s look at the most famous linux stream as a concrete example.

Redhat Enterprise Linux or RHEL for short, is the de facto enterprise linux distribution. If linux is free you ask, how can a distribution be enterprise? Well, support is your answer. Redhat as a company puts features and customer support infrastructure into their product so that they can charge money for a more substantial product. Big companies like to know if something breaks someone will stand by it and help them, and for this reason they like RHEL.

Upstream of RHEL sits Fedora, the open source version of RHEL. The community, along with RHEL developers, actively contribute to the Fedora operating system. This code is then used and modified to become RHEL. This coding cycle continues, with one release based off another within the chain. Downstream of RHEL sits several other distributions such as Oracle Linux, Rocky Linux, and Alma Linux. The obvious question would be how can distributions like Rocky Linux take code from RHEL an enterprise version? This is exactly a problem in the linux distribution world. Another distribution, CentOS, used to be the official open-source downstream of RHEL and Rocky and Oracle were based off that. Recently RedHat made very controversial decisions to move CentOS upstream, and no longer make enterprise code available downstream. This has put distributions in a tricky spot where they want to remain 1:1 with RHEL, but struggle to gain insight into RHEL’s codebase.

A list of some commonly known Linux distributions sometimes refered to as flavors:

• Rocky https://rockylinux.org/: Rocky strives to be a 1:1 downstream clone of RHEL.
• Alma https://almalinux.org/: ABI compatible clone of RHEL.
• Oracle https://www.oracle.com/linux/: Oracle’s enhanced security Linux downstream of RHEL.
• Fedora https://fedoraproject.org/: Upstream open-source of RHEL.
• CentOS Stream https://www.centos.org/: Positioned midstream between Fedora and RHEL.
• Debian https://www.debian.org/: Major RHEL alternative open-source stream which forms Ubuntu.
• Ubuntu https://ubuntu.com/: Solid linux alternative to RHEL made by Canonical. First choice for a lot of homelabbers and downstream of debian.
• Arch https://archlinux.org/: A distro based around constant rolling updates.
• Kali https://www.kali.org/: A distro designed around hacking. Popular among penetration testers.

Most of these distros can come with a GUI or they can come as what we refer to as headless which basically means terminal only.

Navigating linux

So you’re sitting at a command prompt. You’ll need to get the lay of the land a bit. Navigating is the first step so here is a great set of commands to begin moving around by typing them and pressing enter within your terminal.

• cd : change directory. Ex. cd /var/log
• pwd : lists your current directory.
• ls : lists out files in directory. Can be used with ls -a for hidden files. Ex. ls -a /var/log

It’s important to note here that some characters in linux have special meanings in the terminal. See below.

(.) The dot represents the current directory in the filesystem.
(..) The dot-dot represents one level above the current directory.
(/) The forward slash represents the “root” of the filesystem. (Every directory/file in the Linux filesystem is nested under the root / directory.)
(~) The tilde represents the home directory of the currently logged in user.
(*) The asterisk represents a wildcard and is basically used to extend search matches in commands. We’ll see that below a couple times.

So in other words you could change to your home directory by typing cd ~ or go one directory backwards with cd ..

Key directories

Linux has some file paths that are in common amongst its distributions. It’s important to have a general sense of where things are stored so that if you’re looking for a file type you know where to start.

Root directories

/ETC: configuration files
/HOME: individual user’s login directories
/BOOT: Boot loader files
/ROOT: The root directory
/OPT: Third-party applications
/DEV: Device files
/BIN: User binaries
/SBIN: System binaries
/USR: User Applications
/PROC: Process information
/MNT: Mount directory
/SYS: Virtual file system
/MEDIA: Removable devices
/RUN: Temporary file system
/TMP: Temporary files
/LIB: System libraries
/SRV: Service data directory

logging

In general logs can be found within the /var/logs directory. Important log files here generally include:

/var/log/messages – Contains global system messages, including the messages that are logged during system startup.
/var/log/boot.log – Contains information logged when the system boots
/var/log/lastlog – Displays the recent login information for all the users. Can use LASTLOG command.
/var/log/yum.log – Contains logs for packages installed using yum. Or /var/log/dnf.log for example.

FIREWALLD and DNF, and Cron job logs are located here as well.

Important commands

General

The MAN command is a searchable manual of every command in linux.

man
ex. man grep # display a page on using the command grep
ex. man -f ls # searching within the manual for entries concerning "ls"

The PWD command prints the absolute path of where you’re located. This is great if you’re lost within the file system, or if you need to copy and paste a path.

pwd
ex. pwd # displays current directory
/home/Ubuntu/Docker

The CD command changes the current working directory.

cd
ex. cd /home/user/docker # changes directly to /home/user/docker directory
ex. cd ~ # Using tilde changes to home directory
ex. cd .. # Using two periods changes to one directory up
ex. cd - # Using dash changes back to last working directory

The SSH command is used to connect remotely to a server using secure shell protocol.

ssh
ex. ssh servername # will simply login to server and will ask for a username and password
ex. ssh -i ~/.ssh/key.pem username@servername # -i specifies location of SSH key - You can also specify username

The HISTORY command can be used in conjunction with the exclamation mark to reuse any recent series of linux commands.

history
ex. history # displays last commands used
ex. !27 # use a command from displayed list

The LS LL commands lists files and directories.

ls/ll
ex. ls -a  # This shows all files including hidden directories and system files
ex. ls -sh # Shows file sizes, the h portion makes it human-readable
ex. ls -d */ # list only folders
ex. ls -l # show ownership of a file
ex. ll /etc/sssd/ # the ll or long list command is useful for displaying permissions

The MKDIR command creates a folder.

mkdir
ex. mkdir newfolder # mkdir followed by folder name creates a directory
ex. mkdir this/series/of/directories -p # The -p command will create all parent directories

The MV command is equivalent of the cut command. This is also how Linux renames a file or folder.

mv
ex. mv ~/Documents/Ukulele/Apache.pdf /etc/ # mv command works by specifying source path and destination path

The CP command is equivalent of the copy command.

cp
ex. cp file.a file.b /home/usr/rapid/ # copies file.a and file.b to /home/usr/rapid folder
ex. cp -r /source/folder/ /remote/place/ # source, destination -R command recursively copies all files in directory
ex. cp *.c dest/folder # copies all files with .c extension to dest/folder

The SCP command is the secure remote transfer version of copy.

scp -i .ssh/mykey -v "somefile.txt" ubuntu@sysblob:/home/ubuntu/

The RM command can delete files, and RMDIR removes a directory. RM is often used with -r to recursively delete directories.

rm somefile.exe
rm -rf /somedirectory # -r is recursive -f is force to ignore errors such as empty directories
rmdir /someemptydirectory #rmdir does not work if folder contains files

The ALIAS command can shorten a longer command or a series of commands.

alias
ex. alias cls=clear
ex. alias pf="ps -e | grep $1" # Make pf command and whatever follows it search for a process

Searching and piping

These commands are for searching for files or through content, or for performing piping. In Linux it’s possible to feed the results of one command into the next command and we call that piping.

The FIND command will search for folders or files.

find
ex. find . -name *ones* -type f # Searches for files. -name specifies the sequence -type can return d(irectory) or f(ile)
ex. find /home/user -name *.txt
ex. find Pictures/ type -f -exec chmod 600 {} + # can be used to mass edit permissions. {} holds value per line and + ends/loops line

The GREP and PIPE ‘|’ command are extremely useful and are often used together. Grep allows for searching and pipe is how you feed a command into another. Using pipe you can chain together the results of any linux command into another linux command to filter results.

grep and |
ex. ip addr | grep "inet" # only display the ip addresses on a linux system without the junk
ex. grep "string" filename.txt # search a file for a string
ex. grep -r "192.168.1.5" /etc/ # displays lines that contain that IP within a file
ex. cat /proc/cpuinfo | grep -i 'Model' # reads a directory and only prints out lines with Model

Processes, CPU, and Memory

The PS and KILL commands can be used with grep to check quickly for processes and end them.

ps
ex1. ps -e | grep shutter # first search for a process
     kill 1692 # then kill it by id

The TOP and HTOP commands are useful for displaying data on CPU and memory usage. HTOP is colored and real-time. You can also kill processes and run other interactive commands using the function keys.

htop
ex. htop -u username # displays processes of specific user

The FREE command simply displays memory statistics.

free
ex. free -h # memory statistics in human readable

Storage

The DF command can be used to check total storage. It’s best used with the human readable option.

df
ex. df -h # display filesystem and usage in human readable

The LSBLK command lists block devices on system. Block devices are typically SSDs or HDDs.

lsblk
ex. lsblk

The DU command can be used to check particular directories for space usage.

du
Ex. du -h /var/log # find out what's taking up space in /var/log

Content

The TOUCH command can be used to quickly create a file without content.

touch
ex. touch test.txt
ex. touch index.html

The CAT command is great for working with anything text. It has a multitude of uses from copying, reading, or creating content.

cat
ex. cat "put this string in there" > example.txt # create a file and put some text in there
ex. cat example.txt # display file contents
ex. cat file1.txt >> file2.txt # append one file onto another
ex. cat file1.txt file2.txt file3.txt > combinedfile.txt # combine multiple files into one big file
ex. cat >> file.txt # type multiple lines into a file
ex. tac file.txt # tac is like cat but displays content backwards

The ECHO command is similar to CAT but is used to mostly display variables within linux.

echo
ex. echo $USER # display the user variable
ex. echo "This is the list of directories: $(ls)" # echo can even be used in scripts to make things readable.

The WGET command is used to retrieve content across a network.

wget
ex. wget http://www.filesite.com/downloads/files.zip

The CURL command sends a GET request to a web server. This can be used to retrieve any content the web server responds with.

curl
ex. curl www.google.com
ex. curl http://www.filesite.com/downloads/files.zip

The TAR command is used for archiving a directory into a singular file called a tarball. It has many important options:

• f specifies filename
• c creates
• x extracts
• t displays files inside zip
• z declares file as gzip type
• v verbose

tar 
ex. tar -cvf etc_backup.tar etc # creates tarball
ex. tar -tvf etc_backup.tar # view files inside tarball
ex. tar -xvf etc_backup.tar # extract the contents of a tarball
ex. tar -czvf etc_backup.tar.gz etc # use both tar and gzip on a file to compress. add .gz best practice.

The GZIP command is typically used within the TAR command but is used to compress a file.

gzip
ex. gzip etc_backup.tar
ex. gunzip etc_backup.tar.gz # unzip

The TAIL command displays the last 10 lines of a file. Often useful for log files. You can specify how many lines to display.

tail
ex. tail -n 5 file.txt # display last 5 lines of a file

The TRUNCATE command can shorten a file for you. Useful for emptying a file but not deleting it.

truncate
ex. truncate /var/log/cloudinit.log -s 0 # -s is file size
ex. sudo find /var/log/ -type f *.log -exec truncate -s 0 {} + # clear contents of all var .log files

Su and Sudo

Much like Windows makes use of Administrator groups, Linux uses Sudo. Using the SUDO command allows you to temporarily assume super user rights in order to perform an operation. The SU command allows you to login as a new user.

sudo chmod 700 myfile # temporarily assume super rights to perform a single operation
su username # Assume the permission rights of sudo user moving forward

In order to use the sudo command we need to be part of the sudoers group. There are two ways to modify this group to add rights. You can directly add the user via nano and the /etc/sudoers file, or you can use the usermod or visudo commands. The visudo command does the same thing, except designed specifically with safety in mind. For example, it uses an editor which locks the sudoers file and prevents other users from editing it while you have it open.

Using USERMOD to add a user to sudo group:

sudo usermod -aG sudo username # append user to sudo group

Using VISUDO to add a user to sudo group.

sudo visudo
## add similar line inside file
daniel ALL=(ALL:ALL) mytop,cat,tail # you can also give a user sudo rights only for certain commands

Users, Groups and permissions

The LASTLOG command displays the last login time for a user.

lastlog
ex. lastlog -u username # displays last login time for a user

The CHOWN command changes ownership of a linux file. (For more details see the Linux file permissions section).

chown
ex. chown user file.txt # make user the new owner of file
ex. chown user:group file.txt # change user and group
ex. chown :group file.txt # just change group
ex. chown -R user:group /that/folder/directory # recursively give ownership to directory and files

The USERADD command adds a user.

sudo useradd username
ex. sudo useradd -d /var/mike -m mike # -d option specifies a home directory otherwise it defaults to /home/username. -m creates a home directory.
ex. sudo useradd -e 2020-05-30 test_user # add expire date
# can also use -u and -g to specify own userid or groupid
ex. sudo useradd -r systemuser # use -r to create system user

The USERDEL command deletes a user.

userdel
ex. sudo userdel -r username # -r deletes home directory

The PASSWD command sets a password for a user.

passwd
ex. sudo passwd username
ex. sudo passwd -l username # can also be used to lock a user's account. -u to unlock.

The CHAGE command can be used to deal with account and password expirations.

chage
ex. sudo chage -l username # lists expiration info for account
ex. sudo chage -E 2022-12-10 username # set account expiration
ex. sudo chage -M 30 username # set password expiration in days. set to -1 for never expire

The GPASSWD command is used to perform admin tasks on groups. You can add or remove users from secondary groups for example.

sudo gpasswd –d username groupname # remove a user from a group
sudo gpasswd -a groupname # add a user to a group

The GROUPADD command is used to create a new group.

sudo groupadd mynewgroup

The GROUPS command displays current groups of a user.

groups username # displays groups for a user

The USERMOD command lets you modify an existing user data. USERMOD, for example, is the preferred way of modifying primary group membership.

sudo usermod -a -G alpha username # adds username to alpha primary group
sudo usermod -l newname username # rename a username newname

The CHMOD command lets you modify file permissions. (For more details see the Linux file permissions section)

chmod
ex. sudo chmod 700 .ssh # allow read, write, and execute only for user
ex. sudo chmod 600 .ssh/authorized_keys # allow read and write for user only 
ex. sudo chmod u=rw,og=r new_file.txt # chmod also allows for symbolic notation. 
ex. sudo chmod g-x file.txt # take away execute rights from group
ex. sudo chmod -R directory/otherdirectory # recursively change permissions with -R

The CAT /ETC/PASSWORD commands are best used to list users for a server. This can also be used with grep. This is displayed in the format of: username:x:userid:groupid: , , , :/home/username:/bin/bash – each portion defining a different property of the user and the last two being the user’s home directory and default shell.

cat /etc/passwd
ex. cat /etc/passwd | grep "bob"

File permissions

Linux uses a permissions system that at first glance can be pretty confusing. However, simply put they can be broken down into two categories: Permission Group, and Permission type.

PERMISSION GROUP
u – Owner
g – Group
a – All users

PERMISSION TYPES r – Read
w – Write
x – Execute - It’s important to note that execute rights are required to enter/CD into a directory

Let’s look at the output of an ls -l command (which displays files and their permissions):

-rw-rw-r-- 1 lucyadmin lucyadmin  203 Oct  6 07:03 mylogo.txt

In this example the -rw-rw-r– gives us a lot of information. The first letter ( - ) represents that this is a file. A dash represents a file or a d represents a directory. From here everything is grouped in 3s. The second 3 letters rw- represents the owner’s rights. This owner has read, write, but does not have execute rights (represented by the dash). The third portion rw- represents the permissions of the group which has read and write access. The final portion of r– represents the any user category, which only has read permissions.

OCTAL NOTATION

Now that we have a good grasp on how Linux deals with and writes permissions it’s important to learn octal notation. There are only so many ways a file or directory can have permissions. To make things easier on ourselves we use numbers to represent certain permission combinations. We always know the order will be USER:GROUP:ANY with a possible 7 combinations so we represent permissions by a 3 digit number with each digit being 1-7. The chart below explains:

Number	Permission	Symbol
0	No permission	—
1	execute	–x
2	write	-w-
3	execute + write	-wx
4	read	r–
5	read + execute	r-x
6	read + write	rw-
7	read + write + execute	rwx

We then use OCTAL NOTATION in combination with CHMOD to change file and directory permissions.

chmod 764 myfile # changes to permissions 7, 6, and 4 for user, group, and any respectively.

Primary and secondary groups

Every user belongs to both a primary and secondary group. Simply put, primary groups define what group assigned ownership for newly created files, while secondary groups allow a user additional permission. We represent these group types by denoting either -g for primary or -G for secondary.

sudo useradd -g primarygroup -G secondarygroup1,secondarygroup2,secondarygroup3 username

UID and GID

You can determine a user’s GID and UID by using the ID command. You can also CAT /ETC/GROUPS.

id username
cat /etc/groups

USERMOD and GROUPMOD commands can be used to edit UID and GID respectively. The -n option can rename. GROUPDEL can be used to delete a group.

usermod -u 2005 username # changing a users UID
groupmod -g 3000 groupname # changing a users GID
groupdel groupname # delete a group

When a user’s UID or GID are changed the ownership of home directory files are changed automatically. However, files outside this directory need to be changed manually.

Expiring or Unlocking

You can instantly expire a user’s password with the -e option. The next time the user logs in they will need to reset their password.

sudo passwd -e username # Expire a password
sudo passwd -u username # Unlock an account

Managing SSH

Secure Shell, or SSH, is a secure protocol used to connect to devices across unsecure networks. SSH makes use of public and private keys in order to secure both the client and server-side connection. In order to make use of SSH you first need to generate a public and private key to use.

Generating an SSH key

For generating SSH keys we prefer the ED25519 format as it’s the most secure.

Use the SSH-KEYGEN command to generate public and private SSH keys.

ssh-keygen -t ed25519 -C "key comment"

If your host already has SSH keys, you may be asked if you wish to overwrite your previous key. If you choose yes, your previous key will be overwritten, and you will no longer be able to log in to servers using that key. Because of this, be sure to overwrite keys with caution.

.PEM vs .PPK vs .PUB

.pub file format is used by SSH for public key store, this is the key you need to share.

.pem(Privacy Enhanced Mail) is a base64 container format for encoding keys and certificates. A PEM file can contain any string of text specified in its header. Often found in openssh format.

.ppk(Putty Private Key) is a windows ssh client, it does not support .pem format so private keys need to be converted to ppk.

Disabling password Auth

If you have SSH keys configured, tested, and working properly, it’s likely a good idea to disable password authentication. This will prevent any user from signing in with SSH using a password.

To do this, connect to your remote server and open the /etc/ssh/sshd_config file with root or sudo privileges:

sudo nano /etc/ssh/sshd_config

Inside of the file, search for the PasswordAuthentication directive. If it is commented out, uncomment it. Set it to no to disable password logins.

After you have made the change, save and close the file. To implement the changes, you should restart the SSH service.

sudo service ssh restart # ubuntu
sudo systemctl restart ssh # rhel based

Managing Linux SSH via .ssh/config file

If you’re remoting from a Linux based OS, Linux allows for a creation of a config file to make the process easier. Go into the .ssh folder and create a file called config. Use nano/vim to edit the file and the format per host looks like:

Host nextcloud
HostName 192.168.1.20
User vega
IdentityFile ~/.ssh/mykey.pem

This will allow you to then use the ssh command to an alias in the form of:

ssh nextcloud

The .ssh/config file is particularly important when using in combination with Github.com and private repositories. In order to pull a repository remotely via SSH github needs to know which key to use. We specify this in the host file as such:

Host github.com
IdentityFile ~/.ssh/private_key.pem

Networking

The IFCONFIG and IP commands can both be used to display network interfaces. The IP command is the newer preferred command.

ifconfig # display network interface details
ifconfig eth0
ifconfig | grep "inet" # display ipv4/ipv6 details

ip addr # display network interface details
ip addr show dev em1 # show one device
ip route # display and edit route table entries
ip route add default via 192.168.1.1 dev em1 # add a route to table
ip addr add 192.168.1.1/24 dev em1 # add an ip

One of the commands I tend to use to list any addresses the host knows itself as is:

ip addr | grep "inet"

Firewalld and Iptables

The default firewall of RHEL based distributions is IPTABLES, however we wrap that in FIREWALLD for ease of use. Through this firewall you can manage which ports are accessible on the server.

firewall-cmd --list-all # check firewall status

### FIREWALLD has a runtime config and a permanent config. Permanent config is loaded at each boot and replaces runtime. You can make permanent changes to the firewall by either editing the permanent config and reloading the firewall, or setting new runtime configs and making them permanent.
# runtime to perm
sudo firewall-cmd <options>
sudo firewall-cmd --runtime-to-permanent
# perm and reload
sudo firewall-cmd --permanent <options>
sudo firewall-cmd --reload
### add/remove a port
firewall-cmd --permanent --add-port=22/TCP
firewall-cmd --permanent --remove-port=444/tcp
### add/remove a service
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --remove-service=mysql
### add/remove ip or range
firewall-cmd --permanent --add-source=192.168.1.9 or /24
firewall-cmd --permanent --remove-source=192.168.1.100

Linux’s legacy firewall IPTABLES can be found below:

iptables -nvL # list firewall rules
iptables -A INPUT -s 192.168.1.1 -j ACCEPT # white list an ip address
iptables -A INPUT -s 192.168.0.1 -j DROP # block ip address

iptables -I INPUT -p tcp --dport 25 -j ACCEPT # add port inbound
iptables -I OUTPUT -p tcp --sport 25 -j ACCEPT # add port outbound
iptables -A INPUT  -p tcp -m multiport --dports 80,443 -j ACCEPT # grant access to multiple ports inbound
iptables -A OUTPUT -p tcp -m multiport --sports 80,443 -j ACCEPT # grant access to multiple ports outbound
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 443 # setup port forwarding

iptables-save # save changes

Checking ports

Two popular commands for checking open ports are NMAP and NETSTAT.

sudo nmap -sT -O localhost # show ports in nmap locally
sudo netstat -tulpn | grep LISTEN # show listening ports in netstat locally

Sometimes it’s useful to know when a port on a remote server is open. We can also scan for open ports remotely using using NMAP:

sudo nmap 192.168.0.1 -p 22 # checking ip to see if port 22 is open

Network files

Often network files need to be modified for reasons such as changing the hostname. The four main network files are described by Redhat as:

• /etc/hosts: The main purpose of this file is to resolve host names. You specify hostname and IP address much like an ARP table. It can also be used to resolve host names on small networks with no DNS server. Regardless of the type of network the computer is on, this file should contain a line specifying the IP address of the loopback device (127.0.0.1) as localhost.localdomain.
• /etc/resolv.conf: This file specifies the IP addresses of DNS servers and the search domain. Unless configured to do otherwise, the network initialization scripts populate this file.
• /etc/sysconfig/network: This file specifies routing and host information for all network interfaces. It is used to contain directives which are to have global effect and not to be interface specific.
• /etc/hostname: This stores the name of your host.

Changing a hostname

You can change a hostname in RHEL Linux using the HOSTNAMECTL command.

hostnamectl # display host statistics
hostnamectl set-hostname hostname.net # set a new hostname

Afterwards we also add the new hostname to the /etc/hosts file in a format similar to this.

# /etc/hosts
# other stuff here
192.168.1.55 hostname.domain.com hostname

So as a checklist you should –

1. Edit the name of the VM in hypervisor
2. Log into host and run command hostnamectl set-hostname example.domain.com
3. Edit the file /etc/hosts to add your IP and FQDN and hostname as a line and edit anywhere it lists old name
4. Renew the lease with DNS server to replace hostname on network using sudo dhclient command.

After finishing these tasks you will commonly run into two issues. Any hosts that previously logged into this host via SSH will need the old host fingerprint entry deleted from .ssh/authorized_keys file. Any hosts that contacted this host previously will have an entry in their DNS cache for the old hostname pointing to this IP. You will need to clear the DNS cache for it to memorize new hostname.

Troubleshooting

When checking the network path between two devices the TRACEROUTE command can be useful as it shows each hop.

traceroute google.com
traceroute -n google.com # shows network delays

When dealing with DNS issues the DIG and NSLOOKUP commands can display DNS results.

dig google.com
dig google.com ANY # query servers in /etc/resolv.conf

nslookup google.com
nslookup 192.168.10.10 # retrieve a hostname from an ip

When needing to clear the cache of looked up DNS on a server

sudo killall -HUP mDNSResponder; echo "dns cleared successfully"

Package Managers

Much like windows installs software Linux installs packages. For managing packages within a Linux, we can divide Linux distributions between two software managers: RPM and DKPG. These package managers both use their own popular utility to access these package managers. For RPM, YUM is typically used. For DKPG, APT is used. APT is associated with Debian based Linux distributions such as Ubuntu. YUM is used by Redhat distributions such as Fedora and Rocky.

Technically, YUM is now deprecated and the new preferred utility is DNF and will be covered briefly at the end of this guide.

Yum commands

yum update package_name --security # update a package, --security option can just install security updates
yum upgrade # update packages but removes old unneeded packages
yum list package_name # search for a package
yum list available # show available packages
yum list installed # show installed packages
yum list all # show all packages
yum repolist # show repos available
yum install package_name -y # install a package, -y agrees automatically
yum remove package_name # remove a package
yum search package_name # search for a package

DNF commands

sudo dnf repolist all # # list all repos
# list all packages installed -- or list all possible packages (best to grep)
dnf list installed
dnf lsit | grep "package"
# install/remove package
sudo dnf install <package>
sudo dnf remove <package>
# check for updates to all enabled repos then upgrade them
sudo dnf check-update
sudo dnf upgrade --refresh
# Searching
dnf search searchterm

APT commands

sudo apt update && sudo apt upgrade -y # typical command structure for updating all packages
apt update  # update package index
apt upgrade # update packages to latest versions
apt search package_name # search for a package
apt list --installed # show installed packages
apt list # show all packages
yum repolist # show repos available
apt install package_name -y # install a package, -y agrees automatically
apt remove package_name # remove a package
apt autoremove # removes unused packages
apt show package_name # show information about a package

File systems

Linux filesystems can be very confusing to manage. As an introductory it might be helpful to run through the steps of adding a new drive while explaining some concepts and commands.

If you added a new drive determine its name by using lsblk command.

1. First, we need to partition a drive. We logically divide up a hard drive using partitions for organization and isolation.

fdisk /dev/sda
lsblk # after using fdisk we can check drive capacity and partitions using lsblk

A word on drives:

• SD represents SSD or SCSI device.
• HD represents IDE hard drive

When you see a storage device within linux it will be referenced this way such as /dev/sda. These drives are incremented by letter as they are discovered by the operating system, and by number for partition. For example, if an SSD drive were detected second, and we created 1 partition on it as the primary, this would be referenced by /dev/sdb1.

1. Next, we can create a file system on our partition with the MKFS command.

mkfs.ext4 /dev/sda1

We now create a mount point and mount the file system.

mkdir /newstorage # create directory
mount /dev/sdb1 /newstorage # mount storage at directory point
du -h /newstorage # confirm storage is accessible and right size

FSTAB stands for file system table and defines the way file systems are treated when they are introduced into the operating system. In other words, fstab controls how drives are automatically mounted. It’s important if you wish to mount a file system at boot to add an entry for it in /etc/fstab

This is an example of an fstab with a couple entries:

# <file system>     <dir>       <type>   <options>   <dump>	<pass>
10.10.0.10:/backups /var/backups  nfs      defaults    0       0
# automount from synology box
192.168.1.120:/volume2/Media /mnt/media nfs nouser,rsize=8192,wsize=8192,atime,auto,rw,dev,exec,suid 0 0
# as another example below
192.168.1.10:/volume1/sharedfolder /localfolder nfs defaults 0 0

You can mount all fstab entries live by using the mount -a command.
We can check to see if the drive is successfully mounted using the df -h command.

mkfs.ext4 /dev/sda1

Introducing LVM

Another way of designing storage is through volumes. LVM can be thought of as bringing virtualization concepts to storage. Instead of seeing many physical devices, we combine them all into a single virtual device which makes dividing up resources easy and efficient. Let’s run through the same process but this time combine our storage using LVM or Logical Volume Manager.

First, we create a PHYSICAL VOLUME. Physical volumes are disks or partitions that you want to make available to LVM as possible storage.

sudo pvcreate /dev/sdb1 # here we create a physical volume out of the primary partition on our 2nd SSD

We can check our physical volumes with the PVDISPLAY command.

pvdisplay

Now we can create a VOLUME GROUP using the VGCREATE command. Volume groups are the virtual pool we use that contains all the physical volumes. A volume group represents usually the totality of our storage, but often can be used to divide or organize directories. By default, linux machines come with vg00 which is the first volume group. You would most likely be creating vg01.

vgcreate vg00 /dev/sdb1 /dev/sdc # create volume group vg00 and add sdb1 and sdc physical volumes to it

To display Volume group information, we can use the VGDISPLAY command.

vgdisplay
vgdisplay vg00 # display specific group

We then divide the volume group up into any number of LOGICAL VOLUMES. Logical volumes can be thought of as partitions of the virtual storage pool. We create a logical volume using the LVCREATE command. We don’t have to, but it’s best practice to name the logical volume the same or similar to the directory you intend to mount it to. Here I would be mounting to /home/sales.

lvcreate -L 10G -n home_sales vg00 # create a 10 GB Logical Volume named home_sales carved from the vg00 Volume Group

We can now view our newly created logical volume with the LVDISPLAY command.

lvdisplay /dev/vg00/sales-lv

Before mounting we then need to assign a file system to the logical volume and format it. This is done using the MKFS command. All links to logical volumes are stored within the /dev/mapper folder which we check to make sure we are pointing at the correct volume. We then format the file system.

ls -l /dev/mapper # Confirm your logical volume is in the mapper folder which is where all volumes are stored.
mkfs.ext4 /dev/mapper/vg00-home_sales # format our logical volume's filesystem to ext4

Once the logical volume is created it can be mounted like any other file system using the mount command. However, don’t forget to add the mount point to fstab for auto-mounting otherwise next reboot it won’t be there. We can do both of these easily by doing some echo commands into fstab and running a mount command.

echo "/dev/mapper/vg00-home_sales   /home/sales                  ext4    defaults        1 2"  >> /etc/fstab

Then run the mount -a command which mounts everything in fstab. We do it this way as validation as it will produce an error message if our echo to fstab was done incorrectly.

mount -a

Expanding a logical volume

The general steps for adding a storage device to a LV are:

1. Add a disk and configure it as a physical volume.
2. Add it to a volume group.
3. Add the capacity to the logical volume and then extend the filesystem.

Add the new physical volume to the volume group using VGEXTEND command

vgextend vg00 /dev/sdb2

Grow the size of the specific logical volume by extending it with the LVEXTEND command.

lvextend -r -L +1G /dev/vg00/sales-lv # grows size by 1 GB
lvextend -r -l +100%FREE ubuntu-vg/ubuntu-lv # alternatively you can use this command to claim all possible space

Let’s bring it all together and see a full example of creating a new physical volume, adding the physical volume to an existing volume group, allocating space for and creating two new logical volumes, and then mounting them to new directories:

Cron jobs

Cron is a misspelling on the part of a developer, but was intended to represent the greek work Chronos or Chronological. A Cron job is a command or series of commands that are meant to run in the future. This can be a one-off command or a repeated service.

Cron jobs are handled under Linux with the command CRONTAB.

Crontab commands

crontab
ex. sudo crontab -l # list cronjobs for logged in user
ex. sudo cronjob -e # enters crontab editor

Cron jobs are controlled with crontab and editing it using the crontab -e command. Much like the /etc/sudoers file and visudo command, the crontab -e command is purposely designed for editing the /var/spool/cron/crontabs file to avoid mistakes and corruption. After opening this file via the crontab -e command, we can add a cron job as a single line at the end of the file.

~

* * * * * apt install tmux -y
# m h dom mon dow

The first thing you’ll notice is the asterisks. Essentially the crontab file handles each entry by a space and number representing a different control for the job. These are all numerical and they can be found below:

Syntax	Description
Asterisk #	Represents
1	minute
2	hour
3	day of month
4	month
5	day of week
6	command

So for example if we wanted a job to run on the 15th of the month at 11am to update and upgrade all packages we would simply edit in a line similar to this:

0 11 15 * * sudo apt update && sudo apt upgrade -y

Commonly though we will call a script. This is how you’d call a bash script.

0 11 15 * * /usr/bin/bash /home/username/script.sh

System users and cron jobs

Typically, we don’t run Cron jobs as our day to day user. We usually set these tasks aside for a specific user created to run the job. These are system users. System users are denoted usually by a UID below 1000. We can create a system user and assign it a cron job as shown below:

sudo adduser -r -s /bin/nologin apt_runner # -r is system user and -s specifies shell which we say none
sudo crontab -u apt_runner -e # specify the new user and edit their crontab document

Easy crontabs

If you find the process of writing a crontab difficult with the asterisks and timing, there are websites which help you generate the line based off easy to use templates. The website below can provide that:

crontab-generator.org

Journalctl

Journalctl is used to log all things controlled by systemd. It’s extremely handy to check logs to find out what went wrong. Here are some common commands that can be used.

The basic command is journalctl but this dumps all possible logs from all possible time ranges. Instead, we have many ways to sort, filter, and manipulate this log data.

Sorting by time

journalctl --list-boots #list logs by boot
journalctl -b #display current boot log
journalctl -b -1 #display boot log before it

# other ways to sort by time
journalctl --since "1 hour ago" 
journalctl --since "2 days ago"
journalctl --since "2015-06-26 23:15:00" --until "2015-06-26 23:20:00"
journalctl -n 50 --since "1 hour ago" #last 50 messages last hour

Sorting by service

journalctl -u nginx.service #sort by service
journalctl -u nginx.service -u mysql.service #multiple

Live logging

journalctl -f #print log messages realtime
journalctl -u mysql.service -f #for specific service

Bash scripting

Automation is one of the keys of linux administration and one of the basic ways to automate a series of commands is a bash script. Here are the basics of making and using a bash script.

Create a script using whatever text editor. For bash scripts we use the .sh extension.

sudo nano script.sh
sudo vi script.sh

Paste in

#!/usr/bin/env bash
#
# Author : Sysblob
# Date: March 2022
# Version 1.0.0: Displays the text "Hello world!"
#

# Displays a text on the screen :
echo "Hello world!"

We can then run this bash script by using the bash command.

bash script.sh

We can also give the script proper permissions to be executed and run it directly.

chmod 775 script.sh

Then run it.

./script.sh

Scripts require a full path to execute unless they are added to your path which is why we specify ./ in front. We could also specify /testdir/script.sh to run it directly.

Remember in order to execute scripts you need to give the script execute permissions. Try the command chmod 775 myscript.sh

The line #!/usr/bin/env bash identifies the binary used to interpret the code. You could also specify the interpreter to be python for example using #!/usr/bin/env python.

Variables

Here is a good script to study to get an idea of how variables work.

#!/usr/bin/env bash

#
# Author : Sysblob
# Date: July 2023
# Version 1.0.0: Save in /root the files passwd, shadow, group, and gshadow
# Version 1.0.1: Adding what we learned about variables
#

# Global variables
FILE1=/etc/passwd
FILE2=/etc/shadow
FILE3=/etc/group
FILE4=/etc/gshadow

# Destination folder
DESTINATION=/root

## Readonly variables
readonly FILE1 FILE2 FILE3 FILE4 DESTINATION

# A folder name with the day's number
dir="backup-$(date +%j)"

# Clear the screen
clear

# Launch the backup
echo "****************************************************************"
echo "     Backup Script - Backup on ${HOSTNAME}                      "
echo "****************************************************************"
echo "The backup will be made in the folder ${dir}."
echo "Creating the directory..."
mkdir -p ${DESTINATION}/${dir}

echo "Starting the backup of ${FILE1}, ${FILE2}, ${FILE3}, ${FILE4} to ${DESTINATION}/${dir}:"

cp ${FILE1} ${FILE2} ${FILE3} ${FILE4} ${DESTINATION}/${dir}

echo "Backup ended!"

# The backup is noted in the system event log:
logger "Backup of system files by ${USER} on ${HOSTNAME} in the folder ${DESTINATION}/${dir}."

System variables

Some variables come built-in to use:

Variable	Description
HOSTNAME	Host name of the machine
PATH	Path to find the commands
PWD	Current directory, updated each time the cd command is executed
HOME	Login directory of user
$$	Process id of the script execution
$?	Return code of the last command executed
USER, USERNAME and LOGNAME	Name of the user connected to the session.

We can print environment variables using the printenv command or all variables using the set command.

printenv # just global system variables
printenv HOME # Specific variables
set # display all possible variables

There are also Shell variables which exist inside the current shell session only. Some shell variable commands.

MY_VAR="text here" # Setting a shell variable
echo $MY_VAR # Print variable
export MY_VAR # Mae shell variable into a permanent global environment variable
export MY_NEW_VAR="My New Var" # Create a shell variable and instantly make it a global environment variable

Permanent variables

Previous mentioned variables are only temporary to your session. To make permanent variables you need to edit some files.

Variables set in the file /etc/environment whenever a bash login shell is entered.

FOO=bar
VAR_TEST="Test Var"

/etc/profile Use this file to set up system-wide environment variables.

export JAVA_HOME="/path/to/java/home"
export PATH=$PATH:$JAVA_HOME/bin

If you add new variables make sure to reload them into your current session

source ~/.bashrc

Storing commands as variables

It’s possible to put commands in variables. The syntax is:

variable=`command`
variable=$(command) # Preferred syntax

Examples below.

$ day=`date +%d`
$ homedir=$(pwd)