The following article is part of Diogo Peralta Cordeiro's Logbook.

UNIX

Published in 2018-09-05 by Diogo Peralta Cordeiro.

“UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity.” - Dennis Ritchie

In this document, a overview of modern Unix-like systems is presented. These are my study notes when learning about Linux, they are largely based on https://documentation.suse.com/sles/15-SP4/single-html/SLES-administration/ and https://www.cl.cam.ac.uk/teaching/2122/UnixTools/.

Becoming excited

Brief History of Unix

“First Edition” was developed at AT&T Bell Labs during 1968–71 by Ken Thompson and Dennis Ritchie for a PDP 11

User Interfaces and Design Philosophy

Original Unix user interfaces

Photo: Teletype terminals from Bell Labs

VT100 terminals

Video display terminals such as the DEC VT100, all provide 80 characters-per-line fixed-width ASCII output. Their communications protocol is still used today in graphical windowing environments via “terminal emulators” (e.g., xterm, konsole).

Photo: http://www.catb.org/esr/writings/taouu/html/ The VT100 was the first video terminal with a microprocessor, and the first to implement the ANSI X3.64 (= ECMA-48) control functions.

For instance, “ESC[7m” activates inverse mode and “ESC[0m” returns to normal, where ESC is the ASCII “escape” control character (27 = 0x1B).

Unix tools design philosophy

  • Compact and concise input syntax, making full use of ASCII repertoire to minimise keystrokes
  • Output format should be simple and easily usable as input for other programs
  • Programs can be joined together in “pipes” and “scripts” to solve more complex problems
  • Each tool originally performed a simple single function
  • Prefer reusing existing tools with minor extension to rewriting a new tool from scratch
  • The main user-interface software (“shell”) is a normal replaceable program without special privileges
  • Support for automating routine tasks from Brian W. Kernighan, Rob Pike: The Unix Programming Environment. Prentice-Hall, 1984.

Unix Documentation

man[ual] pages

Most Unix documentation can be read from the command line. Classic manual sections: user commands (1), system calls (2), library functions (3), devices (4), file formats (5).

The man tool searches for the manual page file (→ $MANPATH) and activates two further tools (nroff text formatter and more text-file viewer). Add optional section number to disambiguate:

$ man 3 printf # C subroutine, not command

Honesty in the documentation: Unix manual pages traditionally include a BUGS section.

Part Contents
NAME Name and short description of the command
SYNOPSIS Description of the syntax
DESCRIPTION Detailed description of the command
OPTIONS Description of all available options
COMMANDS Instruction that can be given to the program while it is running
FILES Files connected in some way to the command
SEE ALSO Hints on related commands
DIAGNOSTICS Possible error messages of the program
EXAMPLES Examples of calling up a command
BUGS Known errors and problems with the command

The less command is used automatically to view one screen of information at a time while viewing man pages (it is invoked in more emulation mode). The following keys can be used with the less command:

Key Command Description
Space Page one screen forward.
b Page one screen backward.
PageDown Page half a screen forward.
PageUp Page half a screen backward
Down-arrow, Enter Jump one line forward.
Up-arrow Jump one line backward.
End Go to end of the manual page.
Home Go to beginning of manual page.
/expression Search forward from the current cursor position for expression; matching line is displayed as first line on the screen.
?expression Search backwards from current cursor position for expression; matching line is displayed as first line on the screen.
n Move to next instance of expression in the search.
N Move to previous instance of expression in the search
q End display of the manual page

The manual pages are organized in the following sections:

Section Contents
1 Executable programs and shell commands (user commands)
2 System calls
3 Functions and library routines
4 Device files
5 Configuration files and file formats
6 Games
7 Macro packages and file formats
8 System administration commands
p Programmer’s manual

For example, entering the following display general information about mcrontab command:

man 1 crontab

Entering the following displays information about the configuration file for the crontab command (the configuration file is also named crontab):

man 5 crontab

It is especially important to know to which section a command belongs when there is more than one manual for a command.

For example, the uname command is both a user command and a system call.

Entering the following displays information about the user command:

man 1 uname

Entering the following displays information about the system call (such as name and information about the current kernel).

man 2 uname

You can display a brief description of all the available manual pages for a command or utility by using the whatis command.

If you enter man -k keyword or apropos keyword, a list of manual pages in which the keyword appears in the NAME section is displayed.

xman

X11 GUI variant offers a table of contents

info

Many programs are no longer provided with manual pages. Instead, info files are used, which can be read with the info command. It is an alternative GNU hypertext documentation system.

Invoke with info from the shell of with C-h i from emacs. Use M(enu) key to select topic or [Enter] to select hyperlink under cursor, N(ext)/P(rev)/U(p)/D(irectory) to navigate the document tree, Emacs search function (Ctrl-S), and finally Q(uit).

The following are advantages of the info format:

Release Notes and White Papers

Check /usr/share/doc/ and Web for further documentation.

File System

Understanding the File System Hierarchy Standard (FHS)

The Hierarchical Structure of the File System

The file system concept of Linux involves a hierarchical file system that can be shown in the form of a tree.

This tree is not limited to a local partition. It can stretch over several partitions, which can be located on different computers in a network. It begins at the root directory (/), from which the name for the system administrator comes, and branches out like the branches of a tree.

The following shows part of a typical file system tree:

The Top Level of the Linux Directory Tree

A file in this directory tree is uniquely defined by its path. A path refers to the directory names which lead to this file.

The separation character between individual directory names is the slash (“/”). The path can be specified in two ways:

Relative Paths

In this example, the current position in the file system is the user’s home directory. To change to the /etc directory, you can use either one of the following commands:

Sometimes it is necessary to specify the absolute path because certain files can only be uniquely addressed in this way. The length of the path cannot exceed 4096 characters, including the slashes.

Each directory contains two directories that allow relative path specifications.

One of these entries (“.”) points to the directory itself. The other entry (“..”) points to the entry one level higher in the hierarchy.

FHS (File System Hierarchy Standard)

The structure of the file system is described in the File System Hierarchy Standard (FHS). The FHS specifies which directories must be located on the first level after the root directory and what they contain. The current version of FHS is 2.3 (January 2004), and a description is available at http://www.pathname.com/fhs/pub/fhs- 2.3.html. The FHS does not dictate all details. In some areas, it allows for your own definitions. The FHS defines a two-layered hierarchy:

Root Directory (/)

Similar to the root of the C: drive (C:\) in Windows, the root directory refers to the highest layer of the file system tree. Normally only directories (not files) are located here. When the system is booted, the partition on which this directory is located in the first one mounted.

As the kernel cannot fulfil all the tasks of the operating system, all programs that are run at the system start must be available on this partition (they cannot be located on another partition).

The following directories always have to be on the same partition as the root directory: /bin, /dev, /etc, /lib, and /sbin.

Essential Binaries for Use by All Users (/bin)

The /bin directory contains important binaries (executable programs) that are required when no other file systems are mounted, such as all programs necessary for the system start.

These include the various shells, the most important commands for working with files, and several commands for system analysis and configuration.

Overview of the Contents of the /bin Directory

File Description
/bin/bash The bash shell
/bin/cat Displaying files
/bin/cp Copying files
/bin/dd Copying files byte-wise
/bin/gzip Compressing files
/bin/rm Deleting files
/bin/vi vi editor

Boot Directory (/boot)

The /boot directory contains system files. Specifically, it contains

Device Files (/dev)

Each hardware component in the system (such as hard drive partitions, CD drives, printer, and mouse) is represented as a file in the /dev directory.

The hardware components are addressed via these files by writing to or reading from one of these files. Two kinds of device files are included:

The connection to device drivers in the kernel is implemented via numbered channels, which correspond to the number of the device driver in question. These are referred to as major device numbers.

A driver might be responsible for several devices of the same type. To distinguish between these devices, the minor device number is used.

Instead of the size of the files, these two numbers are displayed (the files do not occupy any space on the hard drive anyway):

~ls -l /dev/sda*
root disk 8, 0 20. Feb 08:40 /dev/sda
root disk 8, 1 20. Feb 05:09 /dev/sda1
root disk 8, 2 20. Feb 05:09 /dev/sda2
~

In this example, you want a long list of all SCSI and SATA hard drives in the /dev directory. You enter

ls -l /dev/sda*

Many device files are already available by default. Some of these, however, are never needed. If special device files are required for specific devices, you can generate these with the mknod command. The necessary parameters must be provided by the hardware manufacturer.

The null device /dev/null is also located in this directory. The null device is a special file that discards all data written to it (but reports that the write operation succeeded) and provides no data to any process that reads from it. Program output that would normally be sent to the screen can be redirected to this device (for example, using redirects). The redirected data will be deleted.

Important Device Files:

Device Device File Description
Terminals /dev/
console
/dev/tty1
The system console
The first virtual console, reachable with Ctrl+Alt+F1.
Serial ports /dev/ttyS0
/dev/ttyS*
The first serial port.
Parallel ports /dev/lp0
/dev/lp*
The first parallel port.
Floppy disk drives /dev/fd0
/dev/fd*
The first floppy disk drive. If the drives are addressed via the device files fd0 and fd1, the kernel tries to recognize the floppy disk format itself.
SSD Hard drive /dev/nvme* /dev/nvme0p1 is the first partition of the first ssd.
Sata Hard drives /dev/sda
/dev/sda*
/dev/sda1 is the first primary partition on the first hard drive.
The limit is the number of partitions for SCSI/SATA/IDE drives is 15.
CD-ROM drives /dev/scd0
/dev/scd*
The first CD-ROM drive.

Configuration Files (/etc)

This directory and its subdirectories contain system configuration files. Almost all these files are ASCII files, which can be processed with any editor.

Normal users can read nearly all of these files, but only root can edit them. According to the FHS, no executable programs can be located here.

However, the subdirectories contain many shell scripts.

Important Configuration Files

File Description
/etc/inittab Configuration file for the init process
/etc/init.d/* Scripts for starting services
/etc/modprobe.conf Configuration file of the kernel modules
/etc/DIR_COLORS Specifies the colors for directory listings (ls)
/etc/X11/xorg.conf Configuration file of the X Window System
/etc/fstab Table of the file systems automatically mounted at the system start
/etc/profile Login script of the shell
/etc/passwd User database; all information except passwords
/etc/shadow Encrypted passwords of users
/etc/group Database of user groups
/etc/cups/* Files for the CUPS printing system
/etc/hosts Allocation of computer names to IP addresses
/etc/motd Welcome message after a user logs in (message of the day)
/etc/issue Linux welcome message before the login prompt
/etc/sysconfig/* Central configuration files of the system

Nearly every installed service has at least one configuration file in the /etc directory or a subdirectory.

User Directories (/home)

Every user on a Linux system has his area in which to work with files. This area is called the home directory of the user. When a user logs in, he is in his home directory.

Individual configuration files can be found in the user’s home directory. These configuration files are hidden files because they are normally not displayed by the ls command. All of these files have names that begin with a dot.

Important Files in a User’s Home Directory

File Description
.profile Private login script of the user
.bashrc Configuration file for bash
.bash_history List of commands previously run in bash

If there are no special settings, the home directories of all users are located beneath the /home directory. The home directory of a user can also be addressed via the shortcut “~”, so ~/.bashrc refers to the .bashrc file in the user’s home directory.

In many cases, the /home directory is located on a different partition or can even be located on a different computer (with the central administration of home directories).

Libraries (/lib)

Many programs use specific functions that are also used by other programs. Such standard functions are removed from the actual program, stored in the system, and only called up when the program runs. They are called shared libraries.

The /lib directory contains the libraries that are used by programs in the /bin and /sbin directories. The kernel modules (hardware drivers not compiled into the kernel) are located in the /lib/modules/ directory.

You can find additional libraries below the /usr directory.

Application Directory (/opt)

Installed programs can store their static files in the /opt directory. First, a directory with the name of the application is created. The files are then stored in that directory.

Administrator’s Home Directory (/root)

The home directory of the system administrator is not located beneath /home as are the home directories of normal users. Preferably, it should be on the same partition as the root directory (/) so that it is protected from other users, whose home directories should be on a different partition. Only then is it guaranteed that the user named root can always log in without a problem and have his or her own configured environment available.

System Binaries (/sbin)

The /sbin directory contains important programs for system administration. By contrast, programs that are run by normal users are located in /bin.

Programs in the /sbin directory can also, as a rule, be run by normal users but only to display the configured values. Changes to the configuration can only be made by the user root.

Important Files in the /sbin Directory

File Description
/sbin/fdisk Modifies partitions.
/sbin/fsck* Checks file systems (file system check).
/sbin/init Modifies partitions.
/sbin/fdisk Initializes the system.
/sbin/mkfs* Creates a file system (formatting).
/sbin/shutdown Shuts down the system.

Data Directories for Services (/srv)

The /srv directory contains subdirectories designed for containing data of various services.

This is where server-related physical storage meant to be served should be mounted.

Temporary Area (/tmp)

Various programs create temporary files that are stored in the /tmp directory until they are deleted.

The Hierarchy Below /usr

The /usr directory, in accordance with the FHS, represents a second hierarchical layer (/usr stands for Unix Specific Resources or Unix System Resources).

This is the location for all application programs, graphical interface files, additional libraries, locally installed programs, and commonly shared directories containing documentation.

Subdirectories of the /sbin Directory

Directory Description
/usr/X11R6/ Files of the X Window System
/usr/bin/ Almost all executable programs
/usr/lib/ Libraries
/usr/local/ Locally installed programs, now frequently found in the /opt/ directory
/usr/sbin/ Programs for system administration
/usr/share/doc/ Documentation
/usr/share/man/ The manual pages (command descriptions)

Variable Files (/var)

This directory and its subdirectories contain files that will be modified while the system is running.

Important Directories Beneath /var:

Directory Description
/var/lib/ Variable libraries (such as databases for the locate command)
/var/log/ Log files for most services
/run/ Files with information on running processes (used to be /var/run)
/var/spool/ Directory for queues (printers, e-mail)
/var/lock/ Lock files that are used to protect devices from multiple use

Process Files (/proc)

Linux handles process information that is made available to users via the /proc directory. This directory does not contain any real files and, therefore, does not occupy any space on the hard disk.

/proc is generated dynamically when it is accessed (for example, with ls /proc). Each process has its directory. The values in these directories can be read as if they were in a file, like a “virtual” file. Some values can also be set by writing to the corresponding “files.” Changes to this virtual file system only have an effect as long as the system is running.

For example, the init process always has the process number “1”. Information about it is, therefore, found in the /proc/1/ directory. Each numbered directory corresponds to a running process.

You can view the contents of the files with the cat command, which shows the status of the process, as in the following example:

da1:~ # cat /proc/1/status
Name: init
State: S (sleeping)
Tgid: 1
Pid: 1
PPid: 0
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 256
Groups:
VmPeak: 2308 kB
VmSize: 2280 kB
VmLck: 0 kB
...
                        

In this example, a list is displayed of what the process is called (init), what state it is in (sleeping), and to which user it belongs (Uid: 0 for root).

In addition to directories for each process, /proc also includes directories and files containing information about the state of the system.

Important Files and Directories inside /proc

File Description
/proc/cpuinfo Information about the processor
/proc/dma Use of the Direct Memory Access (DMA) ports
/proc/interrupts Use of the interrupt
/proc/ioports Use of the intrasystem I/O ports
/proc/filesystems File system formats that the kernel understands
/proc/modules Active modules
/proc/mounts Mounted file systems
/proc/net/* Network-specific information and statistics in human-readable form
/proc/partitions Existing partitions
/proc/bus/pci Existing PCI devices
/proc/bus/scsi/ Connected SCSI devices
/proc/sys/* System and kernel information
/proc/version Kernel version

System Information Directory (/sys)

The /sys directory provides information in the form of a tree structure on various hardware buses, hardware devices, active devices, and their drivers. Similar to the / proc directory, /sys is a virtual directory.

Directories for Mounting Other File Systems

Other file systems such as other hard drive partitions, directories from other computers via the network, or removable media (floppy disk, CD-ROM, DVD removable hard drive) can be mounted to the file system at any point.

A directory must exist at the point where you intend to mount the file system. This directory is referred to as the mount point. The complete directory structure of the mounted file system can be found beneath this directory.

In most cases, only the user root can mount and unmount directories. Removable media, such as CDs, can be mounted by a normal user.

To mount a file system, enter the mount command, specifying the device file and the directory to which the file system should be mounted.

A file system can be removed again with the umount command. (Note that the command is NOT called unmount, but umount.) The /etc/mtab file, which is updated by the command mount, shows which file systems are currently mounted. It is possible to mount one file system at different positions.

You can mount file systems in directories that are occupied. The existing contents of these directories, however, will no longer be accessible. After the file system is removed, the data becomes available again.

You can also share certain directories with many computers. This approach is often used for the home directories of users, which are then located centrally on one machine and exported to other computers in the network.

Mount Point for Temporarily Mounted File Systems by system administrators (/mnt)

To access file systems (partitions and devices) you have to integrate or “mount” them before you can access them. You can mount files system anywhere, but the standard directory for mounting is /mnt. It should only be used for temporary purposes. For permanent mounts, you should create an appropriately named directory.

In the following example, the hard drive partition /dev/sda7 is mounted at the position /mnt in the directory tree using the mount command:

da1:~# mount /dev/sda7 /mnt All files on this partition can now be reached via the /mnt directory. To remove this partition again, you use the umount command:

da1:~# umount /mnt If you do not include any options with the mount command, the program tries out several file system formats. If you want to specify a specific file system, use the option -t.

If the file system format is not supported by the kernel, the command is aborted and you receive an error message. In this case, you either load the appropriate module manually or create a new initrd containing the module. Using an updated initrd is the preferred way.

Mount Point for Removable Media (/run/media/$USER/*)

These are auto mounts for removable media. Used to be /media.

Mount Point for Permanent Mounts by system administrators (/vol)

This is a convention, also, if you want to serve the mounted unit, then consider using /srv.

Directories That Can Be Shared

Directory Description
/home Home directories
/opt Applications
/usr The hierarchy below /usr

Directories That Can’t Be Shared

The following directories cannot be imported from other computers. They must always be present locally on each computer:

Directory Description
/bin Important programs
/boot Kernel and boot files
/dev Device files
/etc Configuration files
/lib Libraries
/sbin Important programs for system administration

File Fragments (/lost+found)

This directory is created only on EXT-based file systems (not on Reiser on XFS file systems).

If you have to repair your file system, files and file fragments that can not be restored correctly are saved at /lost+found.

find – traverse directory trees

find directories expression — recursively traverse the file trees rooted at the listed directories. Evaluate the Boolean expression for each file found. Examples:

Print the relative pathname of each file below the current directory:

$ find . -print Erase each file named “core” below home directory if it was not modified in the last 10 days:

$ find ~ -name core -mtime +10 -exec rm -i {} \; The test “-mtime +10” is true for files older than 10 days, concatenation of tests means “logical and”, so “-exec” will only be executed if all earlier terms were true. The “{}” is substituted with the current filename, and “\;” terminates the list of arguments of the shell command provided to “-exec”.

File Types in the Unix System

Identifying File Types in the Unix System

Normal Files

Normal files refer to files as they are also known in other operating systems: a set of contiguous data addressed with one name. This includes files such as ASCII text files, executable programs, and graphics files.

The names for such files can be freely chosen and there is no division into the file name and file type (such as report.txt). Many file names still retain this structure, but these are requirements of the corresponding applications, such as word processing programs or compilers.

Two Special Directories (.) and (..)

Each directory contains two directories that allow relative path specifications.

One of these entries (“.”) points to the directory itself. The other entry (“..”) points to the entry one level higher in the hierarchy.

Device Files

Each piece of hardware in a Linux system is represented by a device file. These files represent links between the hardware components or the device drivers in the kernel and the applications.

Every program that wants to access hardware must access it through the corresponding device file. The programs write to or read from a device file. The kernel then ensures that the data finds its way to the hardware or can be read from the file.

Links are references to files located at other points in the file system. Data maintenance is simplified through the use of such links. Changes only need to be made to the original file. The changes are then automatically valid for all links. There are two types of links: symbolic links and hard links.

Sockets

A socket refers to a special file with which data exchange can be implemented through the file system between two locally running processes.

First In, First Out (FIFO)

FIFO (first in, first out) or named pipe is a term used for files used to exchange data between processes. However, the file can only exchange data in one direction.

File Permissions and Ownership

Managing File Permissions and Ownership

Both files and processes in Unix have an access permissions system. Every file (or process belongs, always, to a user. A process (that belongs to a user) can only access a file if that file permission allows it. On the other hand, users are organized in groups. The permissions may be:

The permission type may be:

Permission Files Directories Who Octal
r read the content list u
g
o
400
040
004
w write add/copy/remove files u
g
o
200
020
002
x execute access to files and subdirectories u
g
o
100
010
001

The permission to the user, group and others are grouped in sequences of 3 characters and match, respectively, the reading, writing and execution, and where - means the absence of given permission. For example, a file with:

rw-r–r-

Allows the reading and writing to the owner, and only reading to the owner’s group and others.

Some commands that allow changing the permissions

Command Example Description
chmod mode args chmod o-w ola allows to change the permissions
chown chown nam ex.c allows the super-user to change the ower of a file

Combined modes (examples): summing the permissions

Permission in octal Description
777 everything is allowed to everyone
700 only the user has every permission
000 nothing to anyone
755 only the owner can add/remove (typical for executables)
644 only the owner can write (typical for text files)
600 only the owner can read and writer
666 everyone can read and write

chmod - set file permission

The chmod command is used to change the permissions of a file or directory. To use it, you specify the desired permission settings and the file or files that you wish to modify. There are two ways to specify the permissions. Here, we will focus on one of these, called the octal notation method.

It is easy to think of the permission settings as a series of bits (which is how the computer thinks about them). Here’s how it works:

rwx rwx rwx = 111 111 111
rw- rw- rw- = 110 110 110
rwx --- --- = 111 000 000

and so on...

rwx = 111 in binary = 7 in octal
rw- = 110 in binary = 6 in octal
r-x = 101 in binary = 5 in octal
r-- = 100 in binary = 4 in octal

Network

No time to take notes, refer to https://documentation.suse.com/sles/15-SP4/single-html/SLES-administration/#cha-network .