UNIX

“UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity.” - Dennis Ritchie

In this document, a overview of modern Unix-like systems is presented. These are my study notes when learning about Linux, they are largely based on https://documentation.suse.com/sles/15-SP4/single-html/SLES-administration/ and https://www.cl.cam.ac.uk/teaching/2122/UnixTools/.

Becoming excited

• Many elements of Unix have become part of common computer science folklore, terminology & tradition over the past 25 years and influenced many other systems (including DOS/Windows).
• Many Unix tools have been ported and become popular on other platforms.
• Unix gives us good examples of high-functionality user interfaces.

Brief History of Unix

“First Edition” was developed at AT&T Bell Labs during 1968–71 by Ken Thompson and Dennis Ritchie for a PDP 11

• Rewritten in C in 1973
• Sixth Edition (1975) first widely available version
• Seventh Edition in 1979, UNIX 32V for VAX
• During the 1980s independent continued development at AT&T (“System V Unix”) and Berkeley University (“BSD Unix”)
• Commercial variants (Solaris, SCO, HP/UX, AIX, IRIX, . . . )
• IEEE and ISO standardisation of a Portable Operating System Interface based on Unix (POSIX) in 1989, later also Single Unix Specification by X/Open, both merged in 2001
• The POSIX standard is freely available online: http://www.unix.org/ and http://pubs.opengroup.org/onlinepubs/9699919799/
• In 1983, Richard Stallman (MIT) initiates a free reimplementation of Unix called GNU (“GNU’s Not Unix”) leading to an editor (emacs), compiler (gcc), debugger (gdb), and numerous other tools.
• In 1991, Linus Torvalds (Helsinki CS undergraduate) starts the development of a free POSIX-compatible kernel, later nicknamed Linux, which was rapidly complemented by existing GNU tools and contributions from volunteers and industry to form a full Unix replacement.
• In 1991, Berkeley University releases a free version of BSD Unix, after removing the remaining proprietary AT&T code. Volunteer projects emerge to continue its development (FreeBSD, NetBSD, OpenBSD).
• In 2000, Apple releases Darwin, the now open-source core component of their OS X and iOS operating systems. Volunteer projects emerge to port many Unix tools onto Darwin (Homebrew, Fink, MacPorts, GNU Darwin, etc.).

User Interfaces and Design Philosophy

Original Unix user interfaces

VT100 terminals

Video display terminals such as the DEC VT100, all provide 80 characters-per-line fixed-width ASCII output. Their communications protocol is still used today in graphical windowing environments via “terminal emulators” (e.g., xterm, konsole).

For instance, “ESC[7m” activates inverse mode and “ESC[0m” returns to normal, where ESC is the ASCII “escape” control character (27 = 0x1B).

• http://www.vt100.net/
• http://www.ecma-international.org/publications/standards/Ecma-048.htm
• man console_codes

Unix tools design philosophy

• Compact and concise input syntax, making full use of ASCII repertoire to minimise keystrokes
• Output format should be simple and easily usable as input for other programs
• Programs can be joined together in “pipes” and “scripts” to solve more complex problems
• Each tool originally performed a simple single function
• Prefer reusing existing tools with minor extension to rewriting a new tool from scratch
• The main user-interface software (“shell”) is a normal replaceable program without special privileges
• Support for automating routine tasks from Brian W. Kernighan, Rob Pike: The Unix Programming Environment. Prentice-Hall, 1984.

Unix Documentation

man[ual] pages

Most Unix documentation can be read from the command line. Classic manual sections: user commands (1), system calls (2), library functions (3), devices (4), file formats (5).

The man tool searches for the manual page file (→ $MANPATH) and activates two further tools (nroff text formatter and more text-file viewer). Add optional section number to disambiguate:

$ man 3 printf # C subroutine, not command

Honesty in the documentation: Unix manual pages traditionally include a BUGS section.

The less command is used automatically to view one screen of information at a time while viewing man pages (it is invoked in more emulation mode). The following keys can be used with the less command:

The manual pages are organized in the following sections:

For example, entering the following display general information about mcrontab command:

man 1 crontab

Entering the following displays information about the configuration file for the crontab command (the configuration file is also named crontab):

man 5 crontab

It is especially important to know to which section a command belongs when there is more than one manual for a command.

For example, the uname command is both a user command and a system call.

Entering the following displays information about the user command:

man 1 uname

Entering the following displays information about the system call (such as name and information about the current kernel).

man 2 uname

You can display a brief description of all the available manual pages for a command or utility by using the whatis command.

If you enter man -k keyword or apropos keyword, a list of manual pages in which the keyword appears in the NAME section is displayed.

xman

X11 GUI variant offers a table of contents

info

Many programs are no longer provided with manual pages. Instead, info files are used, which can be read with the info command. It is an alternative GNU hypertext documentation system.

Invoke with info from the shell of with C-h i from emacs. Use M(enu) key to select topic or [Enter] to select hyperlink under cursor, N(ext)/P(rev)/U(p)/D(irectory) to navigate the document tree, Emacs search function (Ctrl-S), and finally Q(uit).

The following are advantages of the info format:

• A structured document setup is available.
• Specific sections can be reached directly from the table of contents
• Links between specific sections are possible.

Release Notes and White Papers

Check /usr/share/doc/ and Web for further documentation.

File System

Understanding the File System Hierarchy Standard (FHS)

• Files in the file systems can be spread out over several devices. Each file system can be “mounted” at any place in the directory hierarchy. With other file systems, each file system is placed on the same level, at the top. With Linux, the file systems can be placed at lower levels of the directory structure.
• A filename in Linux can be up to 255 characters long. It can contain any number of special characters (“_” or “%”, for example).
• Certain characters (the dollar sign “$”, the semicolon “;”, or the space, for example) have a special meaning. If you want to use one of these characters without the associated special meaning, the character must be preceded by a “\” (backslash) to mask (switch off) its special meaning.
• You can use umlauts, letters with diacritical marks, or other language-specific characters.
  NOTE: Using language-specific characters can lead to problems if you exchange data with people in other countries using other settings because these characters are not present on their keyboards.
• Linux differentiates between upper-case and lower-case letters. For example, the file names Invoice, invoice, and INVOICE refer to three different files.

The Hierarchical Structure of the File System

The file system concept of Linux involves a hierarchical file system that can be shown in the form of a tree.

This tree is not limited to a local partition. It can stretch over several partitions, which can be located on different computers in a network. It begins at the root directory (/), from which the name for the system administrator comes, and branches out like the branches of a tree.

The following shows part of a typical file system tree:

A file in this directory tree is uniquely defined by its path. A path refers to the directory names which lead to this file.

The separation character between individual directory names is the slash (“/”). The path can be specified in two ways:

• As an absolute path starting from the root of the entire file system tree.
  The absolute path always begins with a slash (“/”), the symbol for the root directory.
• As a relative path starting from the current directory.

In this example, the current position in the file system is the user’s home directory. To change to the /etc directory, you can use either one of the following commands:

• absolute path: cd /etc
• relative path: cd ../../etc

Sometimes it is necessary to specify the absolute path because certain files can only be uniquely addressed in this way. The length of the path cannot exceed 4096 characters, including the slashes.

Each directory contains two directories that allow relative path specifications.

One of these entries (“.”) points to the directory itself. The other entry (“..”) points to the entry one level higher in the hierarchy.

FHS (File System Hierarchy Standard)

The structure of the file system is described in the File System Hierarchy Standard (FHS). The FHS specifies which directories must be located on the first level after the root directory and what they contain. The current version of FHS is 2.3 (January 2004), and a description is available at http://www.pathname.com/fhs/pub/fhs- 2.3.html. The FHS does not dictate all details. In some areas, it allows for your own definitions. The FHS defines a two-layered hierarchy:

• The directories in the top layer (immediately below the root directory /).
• As a second layer, the directories under /usr and /var.

Root Directory (/)

Similar to the root of the C: drive (C:\) in Windows, the root directory refers to the highest layer of the file system tree. Normally only directories (not files) are located here. When the system is booted, the partition on which this directory is located in the first one mounted.

As the kernel cannot fulfil all the tasks of the operating system, all programs that are run at the system start must be available on this partition (they cannot be located on another partition).

The following directories always have to be on the same partition as the root directory: /bin, /dev, /etc, /lib, and /sbin.

Essential Binaries for Use by All Users (/bin)

The /bin directory contains important binaries (executable programs) that are required when no other file systems are mounted, such as all programs necessary for the system start.

These include the various shells, the most important commands for working with files, and several commands for system analysis and configuration.

Overview of the Contents of the /bin Directory

Boot Directory (/boot)

The /boot directory contains system files. Specifically, it contains

• Static files related to the boot loader GRUB (Grand Unified Bootloader). These files (except configuration files) are required for the boot process.
• The backed-up information for the Master Boot Record (MBR) and the system map files. They contain information about where exactly the kernel is located on the partition. The MBR backup file is called backup_mbr.
• The kernel has the file name vmlinuz. vmlinuz is a symbolic link to the actual kernel file. According to the FHS, however, the kernel can also be located directly in the root directory.

Device Files (/dev)

Each hardware component in the system (such as hard drive partitions, CD drives, printer, and mouse) is represented as a file in the /dev directory.

The hardware components are addressed via these files by writing to or reading from one of these files. Two kinds of device files are included:

• Character-oriented device files (for devices working sequentially, such as a printer, mouse, or tape drive)
• Block-oriented device files (such as floppy disks and hard drives).

The connection to device drivers in the kernel is implemented via numbered channels, which correspond to the number of the device driver in question. These are referred to as major device numbers.

A driver might be responsible for several devices of the same type. To distinguish between these devices, the minor device number is used.

Instead of the size of the files, these two numbers are displayed (the files do not occupy any space on the hard drive anyway):

~ls -l /dev/sda*
root disk 8, 0 20. Feb 08:40 /dev/sda
root disk 8, 1 20. Feb 05:09 /dev/sda1
root disk 8, 2 20. Feb 05:09 /dev/sda2
~

In this example, you want a long list of all SCSI and SATA hard drives in the /dev directory. You enter

ls -l /dev/sda*

• The major device number 8 is listed for all files. This refers to the driver for SCSI hard drives.
• The minor device numbers are 0, 1, and 2 (they run from 1 to 15 for SCSI hard drives).

Many device files are already available by default. Some of these, however, are never needed. If special device files are required for specific devices, you can generate these with the mknod command. The necessary parameters must be provided by the hardware manufacturer.

The null device /dev/null is also located in this directory. The null device is a special file that discards all data written to it (but reports that the write operation succeeded) and provides no data to any process that reads from it. Program output that would normally be sent to the screen can be redirected to this device (for example, using redirects). The redirected data will be deleted.

Important Device Files:

Configuration Files (/etc)

This directory and its subdirectories contain system configuration files. Almost all these files are ASCII files, which can be processed with any editor.

Normal users can read nearly all of these files, but only root can edit them. According to the FHS, no executable programs can be located here.

However, the subdirectories contain many shell scripts.

Important Configuration Files

Nearly every installed service has at least one configuration file in the /etc directory or a subdirectory.

User Directories (/home)

Every user on a Linux system has his area in which to work with files. This area is called the home directory of the user. When a user logs in, he is in his home directory.

Individual configuration files can be found in the user’s home directory. These configuration files are hidden files because they are normally not displayed by the ls command. All of these files have names that begin with a dot.

Important Files in a User’s Home Directory

If there are no special settings, the home directories of all users are located beneath the /home directory. The home directory of a user can also be addressed via the shortcut “~”, so ~/.bashrc refers to the .bashrc file in the user’s home directory.

In many cases, the /home directory is located on a different partition or can even be located on a different computer (with the central administration of home directories).

Libraries (/lib)

Many programs use specific functions that are also used by other programs. Such standard functions are removed from the actual program, stored in the system, and only called up when the program runs. They are called shared libraries.

The /lib directory contains the libraries that are used by programs in the /bin and /sbin directories. The kernel modules (hardware drivers not compiled into the kernel) are located in the /lib/modules/ directory.

You can find additional libraries below the /usr directory.

Application Directory (/opt)

Installed programs can store their static files in the /opt directory. First, a directory with the name of the application is created. The files are then stored in that directory.

Administrator’s Home Directory (/root)

The home directory of the system administrator is not located beneath /home as are the home directories of normal users. Preferably, it should be on the same partition as the root directory (/) so that it is protected from other users, whose home directories should be on a different partition. Only then is it guaranteed that the user named root can always log in without a problem and have his or her own configured environment available.

System Binaries (/sbin)

The /sbin directory contains important programs for system administration. By contrast, programs that are run by normal users are located in /bin.

Programs in the /sbin directory can also, as a rule, be run by normal users but only to display the configured values. Changes to the configuration can only be made by the user root.

Important Files in the /sbin Directory

Data Directories for Services (/srv)

The /srv directory contains subdirectories designed for containing data of various services.

This is where server-related physical storage meant to be served should be mounted.

Temporary Area (/tmp)

Various programs create temporary files that are stored in the /tmp directory until they are deleted.

The Hierarchy Below /usr

The /usr directory, in accordance with the FHS, represents a second hierarchical layer (/usr stands for Unix Specific Resources or Unix System Resources).

This is the location for all application programs, graphical interface files, additional libraries, locally installed programs, and commonly shared directories containing documentation.

Subdirectories of the /sbin Directory

Variable Files (/var)

This directory and its subdirectories contain files that will be modified while the system is running.

Important Directories Beneath /var:

Process Files (/proc)

Linux handles process information that is made available to users via the /proc directory. This directory does not contain any real files and, therefore, does not occupy any space on the hard disk.

/proc is generated dynamically when it is accessed (for example, with ls /proc). Each process has its directory. The values in these directories can be read as if they were in a file, like a “virtual” file. Some values can also be set by writing to the corresponding “files.” Changes to this virtual file system only have an effect as long as the system is running.

For example, the init process always has the process number “1”. Information about it is, therefore, found in the /proc/1/ directory. Each numbered directory corresponds to a running process.

You can view the contents of the files with the cat command, which shows the status of the process, as in the following example:

da1:~ # cat /proc/1/status
Name: init
State: S (sleeping)
Tgid: 1
Pid: 1
PPid: 0
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 256
Groups:
VmPeak: 2308 kB
VmSize: 2280 kB
VmLck: 0 kB
...
                        

In this example, a list is displayed of what the process is called (init), what state it is in (sleeping), and to which user it belongs (Uid: 0 for root).

In addition to directories for each process, /proc also includes directories and files containing information about the state of the system.

Important Files and Directories inside /proc

System Information Directory (/sys)

The /sys directory provides information in the form of a tree structure on various hardware buses, hardware devices, active devices, and their drivers. Similar to the / proc directory, /sys is a virtual directory.

Directories for Mounting Other File Systems

Other file systems such as other hard drive partitions, directories from other computers via the network, or removable media (floppy disk, CD-ROM, DVD removable hard drive) can be mounted to the file system at any point.

A directory must exist at the point where you intend to mount the file system. This directory is referred to as the mount point. The complete directory structure of the mounted file system can be found beneath this directory.

In most cases, only the user root can mount and unmount directories. Removable media, such as CDs, can be mounted by a normal user.

To mount a file system, enter the mount command, specifying the device file and the directory to which the file system should be mounted.

A file system can be removed again with the umount command. (Note that the command is NOT called unmount, but umount.) The /etc/mtab file, which is updated by the command mount, shows which file systems are currently mounted. It is possible to mount one file system at different positions.

You can mount file systems in directories that are occupied. The existing contents of these directories, however, will no longer be accessible. After the file system is removed, the data becomes available again.

You can also share certain directories with many computers. This approach is often used for the home directories of users, which are then located centrally on one machine and exported to other computers in the network.

Mount Point for Temporarily Mounted File Systems by system administrators (/mnt)

To access file systems (partitions and devices) you have to integrate or “mount” them before you can access them. You can mount files system anywhere, but the standard directory for mounting is /mnt. It should only be used for temporary purposes. For permanent mounts, you should create an appropriately named directory.

In the following example, the hard drive partition /dev/sda7 is mounted at the position /mnt in the directory tree using the mount command:

da1:~# mount /dev/sda7 /mnt All files on this partition can now be reached via the /mnt directory. To remove this partition again, you use the umount command:

da1:~# umount /mnt If you do not include any options with the mount command, the program tries out several file system formats. If you want to specify a specific file system, use the option -t.

If the file system format is not supported by the kernel, the command is aborted and you receive an error message. In this case, you either load the appropriate module manually or create a new initrd containing the module. Using an updated initrd is the preferred way.

Mount Point for Removable Media (/run/media/$USER/*)

These are auto mounts for removable media. Used to be /media.

Mount Point for Permanent Mounts by system administrators (/vol)

This is a convention, also, if you want to serve the mounted unit, then consider using /srv.

Directories That Can Be Shared

Directories That Can’t Be Shared

The following directories cannot be imported from other computers. They must always be present locally on each computer:

File Fragments (/lost+found)

This directory is created only on EXT-based file systems (not on Reiser on XFS file systems).

If you have to repair your file system, files and file fragments that can not be restored correctly are saved at /lost+found.

find – traverse directory trees

find directories expression — recursively traverse the file trees rooted at the listed directories. Evaluate the Boolean expression for each file found. Examples:

Print the relative pathname of each file below the current directory:

$ find . -print Erase each file named “core” below home directory if it was not modified in the last 10 days:

$ find ~ -name core -mtime +10 -exec rm -i {} \; The test “-mtime +10” is true for files older than 10 days, concatenation of tests means “logical and”, so “-exec” will only be executed if all earlier terms were true. The “{}” is substituted with the current filename, and “\;” terminates the list of arguments of the shell command provided to “-exec”.

File Types in the Unix System

Identifying File Types in the Unix System

Normal Files

Normal files refer to files as they are also known in other operating systems: a set of contiguous data addressed with one name. This includes files such as ASCII text files, executable programs, and graphics files.

The names for such files can be freely chosen and there is no division into the file name and file type (such as report.txt). Many file names still retain this structure, but these are requirements of the corresponding applications, such as word processing programs or compilers.

Two Special Directories (.) and (..)

Each directory contains two directories that allow relative path specifications.

One of these entries (“.”) points to the directory itself. The other entry (“..”) points to the entry one level higher in the hierarchy.

Device Files

Each piece of hardware in a Linux system is represented by a device file. These files represent links between the hardware components or the device drivers in the kernel and the applications.

Every program that wants to access hardware must access it through the corresponding device file. The programs write to or read from a device file. The kernel then ensures that the data finds its way to the hardware or can be read from the file.

Links

Links are references to files located at other points in the file system. Data maintenance is simplified through the use of such links. Changes only need to be made to the original file. The changes are then automatically valid for all links. There are two types of links: symbolic links and hard links.

Sockets

A socket refers to a special file with which data exchange can be implemented through the file system between two locally running processes.

First In, First Out (FIFO)

FIFO (first in, first out) or named pipe is a term used for files used to exchange data between processes. However, the file can only exchange data in one direction.

File Permissions and Ownership

Managing File Permissions and Ownership

Both files and processes in Unix have an access permissions system. Every file (or process belongs, always, to a user. A process (that belongs to a user) can only access a file if that file permission allows it. On the other hand, users are organized in groups. The permissions may be:

• u: owner user
• g: every user of the owner’s group
• o: other users
• a: every users (sum of the above permissions)

The permission type may be:

The permission to the user, group and others are grouped in sequences of 3 characters and match, respectively, the reading, writing and execution, and where - means the absence of given permission. For example, a file with:

rw-r–r-

Allows the reading and writing to the owner, and only reading to the owner’s group and others.

Some commands that allow changing the permissions

Combined modes (examples): summing the permissions

chmod - set file permission

The chmod command is used to change the permissions of a file or directory. To use it, you specify the desired permission settings and the file or files that you wish to modify. There are two ways to specify the permissions. Here, we will focus on one of these, called the octal notation method.

It is easy to think of the permission settings as a series of bits (which is how the computer thinks about them). Here’s how it works:

rwx rwx rwx = 111 111 111
rw- rw- rw- = 110 110 110
rwx --- --- = 111 000 000

and so on...

rwx = 111 in binary = 7 in octal
rw- = 110 in binary = 6 in octal
r-x = 101 in binary = 5 in octal
r-- = 100 in binary = 4 in octal

Network

No time to take notes, refer to https://documentation.suse.com/sles/15-SP4/single-html/SLES-administration/#cha-network .