Search for:
May 28 2019
LFCS: How to use GNU ‘sed’ Command to Create, Edit, and Manipulate files in Linux – Part 1

LFCS: How to use GNU &-8216;sed&-8217; Command to Create, Edit, and Manipulate files in Linux &-8211; Part 1 &-8211; this Article or News was published on this date:2019-05-28 18:59:15 kindly share it with friends if you find it helpful

The Linux Foundation announced the LFCS (Linux Foundation Certified Sysadmin) certification, a new program that aims at helping individuals all over the world to get certified in basic to intermediate system administration tasks for Linux systems. This includes supporting running systems and services, along with first-hand troubleshooting and analysis, and smart decision-making to escalate issues to engineering teams.

Linux Foundation Certified SysadminLinux Foundation Certified Sysadmin

Linux Foundation Certified Sysadmin – Part 1

Please watch the following video that demonstrates about The Linux Foundation Certification Program.

The series will be titled Preparation for the LFCS (Linux Foundation Certified Sysadmin) Parts 1 through 10 and cover the following topics for Ubuntu, CentOS, and openSUSE:

Part 1: How to use GNU ‘sed’ Command to Create, Edit, and Manipulate files in Linux

Important: Due to changes in the LFCS certification requirements effective Feb. 2, 2016, we are including the following necessary topics to the LFCS series published here. To prepare for this exam, your are highly encouraged to use the LFCE series as well.

This post is Part 1 of a 20-tutorial series, which will cover the necessary domains and competencies that are required for the LFCS certification exam. That being said, fire up your terminal, and let’s start.

Processing Text Streams in Linux

Linux treats the input to and the output from programs as streams (or sequences) of characters. To begin understanding redirection and pipes, we must first understand the three most important types of I/O (Input and Output) streams, which are in fact special files (by convention in UNIX and Linux, data streams and peripherals, or device files, are also treated as ordinary files).

The difference between > (redirection operator) and | (pipeline operator) is that while the first connects a command with a file, the latter connects the output of a command with another command.

- command > file
- command1 | command2

Since the redirection operator creates or overwrites files silently, we must use it with extreme caution, and never mistake it with a pipeline. One advantage of pipes on Linux and UNIX systems is that there is no intermediate file involved with a pipe – the stdout of the first command is not written to a file and then read by the second command.

For the following practice exercises we will use the poem “A happy child” (anonymous author).

Linux Foundation Certified Sysadmincat command

cat command example

Using sed

The name sed is short for stream editor. For those unfamiliar with the term, a stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).

The most basic (and popular) usage of sed is the substitution of characters. We will begin by changing every occurrence of the lowercase y to UPPERCASE Y and redirecting the output to ahappychild2.txt. The g flag indicates that sed should perform the substitution for all instances of term on every line of file. If this flag is omitted, sed will replace only the first occurrence of term on each line.

Basic syntax:
- sed ‘s/term/replacement/flag’ file
Our example:
- sed ‘s/y/Y/g’ ahappychild.txt > ahappychild2.txt
Linux Foundation Certified Sysadminsed command

sed command example

Should you want to search for or replace a special character (such as /, , &) you need to escape it, in the term or replacement strings, with a backward slash.

For example, we will substitute the word and for an ampersand. At the same time, we will replace the word I with You when the first one is found at the beginning of a line.

- sed 's/and/&/g;s/^I/You/g' ahappychild.txt
Linux Foundation Certified Sysadminsed replace string

sed replace string

In the above command, a ^ (caret sign) is a well-known regular expression that is used to represent the beginning of a line.

As you can see, we can combine two or more substitution commands (and use regular expressions inside them) by separating them with a semicolon and enclosing the set inside single quotes.

Another use of sed is showing (or deleting) a chosen portion of a file. In the following example, we will display the first 5 lines of /var/log/messages from Jun 8.

- sed -n '/^Jun  8/ p' /var/log/messages | sed -n 1,5p

Note that by default, sed prints every line. We can override this behaviour with the -n option and then tell sed to print (indicated by p) only the part of the file (or the pipe) that matches the pattern (Jun 8 at the beginning of line in the first case and lines 1 through 5 inclusive in the second case).

Finally, it can be useful while inspecting scripts or configuration files to inspect the code itself and leave out comments. The following sed one-liner deletes (d) blank lines or those starting with &-8211; (the | character indicates a boolean OR between the two regular expressions).

- sed '/^-|^$/d' apache2.conf
Linux Foundation Certified Sysadminsed match string

sed match string

uniq Command

The uniq command allows us to report or remove duplicate lines in a file, writing to stdout by default. We must note that uniq does not detect repeated lines unless they are adjacent. Thus, uniq is commonly used along with a preceding sort (which is used to sort lines of text files). By default, sort takes the first field (separated by spaces) as key field. To specify a different key field, we need to use the -k option.

Examples

The du –sch /path/to/directory/* command returns the disk space usage per subdirectories and files within the specified directory in human-readable format (also shows a total per directory), and does not order the output by size, but by subdirectory and file name. We can use the following command to sort by size.

- du -sch /var/* | sort –h
Linux Foundation Certified Sysadminsort command

sort command example

You can count the number of events in a log by date by telling uniq to perform the comparison using the first 6 characters (-w 6) of each line (where the date is specified), and prefixing each output line by the number of occurrences (-c) with the following command.

- cat /var/log/mail.log | uniq -c -w 6
Linux Foundation Certified SysadminCount Numbers in File

Count Numbers in File

Finally, you can combine sort and uniq (as they usually are). Consider the following file with a list of donors, donation date, and amount. Suppose we want to know how many unique donors there are. We will use the following command to cut the first field (fields are delimited by a colon), sort by name, and remove duplicate lines.

- cat sortuniq.txt | cut -d: -f1 | sort | uniq
Linux Foundation Certified SysadminFind Unique Records in File

Find Unique Records in File

Read Also: 13 “cat” Command Examples

grep Command

grep searches text files or (command output) for the occurrence of a specified regular expression and outputs any line containing a match to standard output.

Examples

Display the information from /etc/passwd for user gacanepa, ignoring case.

- grep -i gacanepa /etc/passwd
Linux Foundation Certified Sysadmingrep Command

grep command example

Show all the contents of /etc whose name begins with rc followed by any single number.

- ls -l /etc | grep rc[0-9]
Linux Foundation Certified SysadminList Content Using grep

List Content Using grep

Read Also: 12 “grep” Command Examples

tr Command Usage

The tr command can be used to translate (change) or delete characters from stdin, and write the result to stdout.

Examples

Change all lowercase to uppercase in sortuniq.txt file.

- cat sortuniq.txt | tr [:lower:] [:upper:]
Linux Foundation Certified SysadminSort Strings in File

Sort Strings in File

Squeeze the delimiter in the output of ls –l to only one space.

- ls -l | tr -s ' '
Linux Foundation Certified SysadminSqueeze Delimiter

Squeeze Delimiter

cut Command Usage

The cut command extracts portions of input lines (from stdin or files) and displays the result on standard output, based on number of bytes (-b option), characters (-c), or fields (-f). In this last case (based on fields), the default field separator is a tab, but a different delimiter can be specified by using the -d option.

Examples

Extract the user accounts and the default shells assigned to them from /etc/passwd (the –d option allows us to specify the field delimiter, and the –f switch indicates which field(s) will be extracted.

- cat /etc/passwd | cut -d: -f1,7
Linux Foundation Certified SysadminExtract User Accounts

Extract User Accounts

Summing up, we will create a text stream consisting of the first and third non-blank files of the output of the last command. We will use grep as a first filter to check for sessions of user gacanepa, then squeeze delimiters to only one space (tr -s ‘ ‘). Next, we’ll extract the first and third fields with cut, and finally sort by the second field (IP addresses in this case) showing unique.

- last | grep gacanepa | tr -s ' ' | cut -d' ' -f1,3 | sort -k2 | uniq
Linux Foundation Certified Sysadminlast command

last command example

The above command shows how multiple commands and pipes can be combined so as to obtain filtered data according to our desires. Feel free to also run it by parts, to help you see the output that is pipelined from one command to the next (this can be a great learning experience, by the way!).

Summary

Although this example (along with the rest of the examples in the current tutorial) may not seem very useful at first sight, they are a nice starting point to begin experimenting with commands that are used to create, edit, and manipulate files from the Linux command line. Feel free to leave your questions and comments below – they will be much appreciated!

Reference Links
  1. About the LFCS
  2. Why get a Linux Foundation Certification?
  3. Register for the LFCS exam
School Forums 0 Comments
May 28 2019
LFCS: How to Install and Use vi/vim as a Full Text Editor – Part 2

LFCS: How to Install and Use vi/vim as a Full Text Editor – Part 2 &-8211; this Article or News was published on this date:2019-05-28 18:59:10 kindly share it with friends if you find it helpful

A couple of months ago, the Linux Foundation launched the LFCS (Linux Foundation Certified Sysadmin) certification in order to help individuals from all over the world to verify they are capable of doing basic to intermediate system administration tasks on Linux systems: system support, first-hand troubleshooting and maintenance, plus intelligent decision-making to know when it’s time to raise issues to upper support teams.

Learning VI Editor in LinuxLearning VI Editor in Linux

Learning VI Editor in Linux

Please take a look at the below video that explains The Linux Foundation Certification Program.

This post is Part 2 of a 10-tutorial series, here in this part, we will cover the basic file editing operations and understanding modes in vi/m editor, that are required for the LFCS certification exam.

Perform Basic File Editing Operations Using vi/m

Vi was the first full-screen text editor written for Unix. Although it was intended to be small and simple, it can be a bit challenging for people used exclusively to GUI text editors, such as NotePad++, or gedit, to name a few examples.

To use Vi, we must first understand the 3 modes in which this powerful program operates, in order to begin learning later about the its powerful text-editing procedures.

Please note that most modern Linux distributions ship with a variant of vi known as vim (“Vi improved”), which supports more features than the original vi does. For that reason, throughout this tutorial we will use vi and vim interchangeably.

If your distribution does not have vim installed, you can install it as follows.

  1. Ubuntu and derivatives: aptitude update && aptitude install vim
  2. Red Hat-based distributions: yum update && yum install vim
  3. openSUSE: zypper update && zypper install vim

Why should I want to learn vi?

There are at least 2 good reasons to learn vi.

1. vi is always available (no matter what distribution you’re using) since it is required by POSIX.

2. vi does not consume a considerable amount of system resources and allows us to perform any imaginable tasks without lifting our fingers from the keyboard.

In addition, vi has a very extensive built-in manual, which can be launched using the :help command right after the program is started. This built-in manual contains more information than vi/m’s man page.

Learning VI Editor in Linuxvi Man Pages

vi Man Pages

Launching vi

To launch vi, type vi in your command prompt.

Learning VI Editor in LinuxStart vi Editor

Start vi Editor

Then press i to enter Insert mode, and you can start typing. Another way to launch vi/m is.

- vi filename

Which will open a new buffer (more on buffers later) named filename, which you can later save to disk.

Understanding Vi modes

1. In command mode, vi allows the user to navigate around the file and enter vi commands, which are brief, case-sensitive combinations of one or more letters. Almost all of them can be prefixed with a number to repeat the command that number of times.

For example, yy (or Y) copies the entire current line, whereas 3yy (or 3Y) copies the entire current line along with the two next lines (3 lines in total). We can always enter command mode (regardless of the mode we’re working on) by pressing the Esc key. The fact that in command mode the keyboard keys are interpreted as commands instead of text tends to be confusing to beginners.

2. In ex mode, we can manipulate files (including saving a current file and running outside programs). To enter this mode, we must type a colon (:) from command mode, directly followed by the name of the ex-mode command that needs to be used. After that, vi returns automatically to command mode.

3. In insert mode (the letter i is commonly used to enter this mode), we simply enter text. Most keystrokes result in text appearing on the screen (one important exception is the Esc key, which exits insert mode and returns to command mode).

Learning VI Editor in Linuxvi Insert Mode

vi Insert Mode

Vi Commands

The following table shows a list of commonly used vi commands. File edition commands can be enforced by appending the exclamation sign to the command (for example, b.:q! enforces quitting without saving).

 Key command  Description
 h or left arrow  Go one character to the left
 j or down arrow  Go down one line
 k or up arrow  Go up one line
 l (lowercase L) or right arrow  Go one character to the right
 H  Go to the top of the screen
 L  Go to the bottom of the screen
 G  Go to the end of the file
 w  Move one word to the right
 b  Move one word to the left
 0 (zero)  Go to the beginning of the current line
 ^  Go to the first nonblank character on the current line
 $  Go to the end of the current line
 Ctrl-B  Go back one screen
 Ctrl-F  Go forward one screen
 i  Insert at the current cursor position
 I (uppercase i)  Insert at the beginning of the current line
 J (uppercase j)  Join current line with the next one (move next line up)
 a  Append after the current cursor position
 o (lowercase O)  Creates a blank line after the current line
 O (uppercase o)  Creates a blank line before the current line
 r  Replace the character at the current cursor position
 R  Overwrite at the current cursor position
 x  Delete the character at the current cursor position
 X  Delete the character immediately before (to the left) of the current cursor position
 dd  Cut (for later pasting) the entire current line
 D  Cut from the current cursor position to the end of the line (this command is equivalent to d$)
 yX  Give a movement command X, copy (yank) the appropriate number of characters, words, or lines from the current cursor position
 yy or Y  Yank (copy) the entire current line
 p  Paste after (next line) the current cursor position
 P  Paste before (previous line) the current cursor position
 . (period)  Repeat the last command
 u  Undo the last command
 U  Undo the last command in the last line. This will work as long as the cursor is still on the line.
 n  Find the next match in a search
 N  Find the previous match in a search
 :n  Next file; when multiple files are specified for editing, this commands loads the next file.
 :e file  Load file in place of the current file.
 :r file  Insert the contents of file after (next line) the current cursor position
 :q  Quit without saving changes.
 :w file  Write the current buffer to file. To append to an existing file, use :w >> file.
 :wq  Write the contents of the current file and quit. Equivalent to x! and ZZ
 :r! command  Execute command and insert output after (next line) the current cursor position.

Vi Options

The following options can come in handy while running vim (we need to add them in our ~/.vimrc file).

- echo set number >> ~/.vimrc
- echo syntax on >> ~/.vimrc
- echo set tabstop=4 >> ~/.vimrc
- echo set autoindent >> ~/.vimrc
Learning VI Editor in Linuxvi Editor Options

vi Editor Options

  1. set number shows line numbers when vi opens an existing or a new file.
  2. syntax on turns on syntax highlighting (for multiple file extensions) in order to make code and config files more readable.
  3. set tabstop=4 sets the tab size to 4 spaces (default value is 8).
  4. set autoindent carries over previous indent to the next line.

Search and replace

vi has the ability to move the cursor to a certain location (on a single line or over an entire file) based on searches. It can also perform text replacements with or without confirmation from the user.

a). Searching within a line: the f command searches a line and moves the cursor to the next occurrence of a specified character in the current line.

For example, the command fh would move the cursor to the next instance of the letter h within the current line. Note that neither the letter f nor the character you’re searching for will appear anywhere on your screen, but the character will be highlighted after you press Enter.

For example, this is what I get after pressing f4 in command mode.

Learning VI Editor in LinuxSearch String in Vi

Search String in Vi

b). Searching an entire file: use the / command, followed by the word or phrase to be searched for. A search may be repeated using the previous search string with the n command, or the next one (using the N command). This is the result of typing /Jane in command mode.

Learning VI Editor in LinuxVi Search String in File

Vi Search String in File

c). vi uses a command (similar to sed’s) to perform substitution operations over a range of lines or an entire file. To change the word “old” to “young” for the entire file, we must enter the following command.

 :%s/old/young/g

Notice: The colon at the beginning of the command.

Learning VI Editor in LinuxVi Search and Replace

Vi Search and Replace

The colon (:) starts the ex command, s in this case (for substitution), % is a shortcut meaning from the first line to the last line (the range can also be specified as n,m which means “from line n to line m”), old is the search pattern, while young is the replacement text, and g indicates that the substitution should be performed on every occurrence of the search string in the file.

Alternatively, a c can be added to the end of the command to ask for confirmation before performing any substitution.

:%s/old/young/gc

Before replacing the original text with the new one, vi/m will present us with the following message.

Learning VI Editor in LinuxReplace String in Vi

Replace String in Vi

  1. y: perform the substitution (yes)
  2. n: skip this occurrence and go to the next one (no)
  3. a: perform the substitution in this and all subsequent instances of the pattern.
  4. q or Esc: quit substituting.
  5. l (lowercase L): perform this substitution and quit (last).
  6. Ctrl-e, Ctrl-y: Scroll down and up, respectively, to view the context of the proposed substitution.

Editing Multiple Files at a Time

Let’s type vim file1 file2 file3 in our command prompt.

- vim file1 file2 file3

First, vim will open file1. To switch to the next file (file2), we need to use the :n command. When we want to return to the previous file, :N will do the job.

In order to switch from file1 to file3.

a). The :buffers command will show a list of the file currently being edited.

:buffers
Learning VI Editor in LinuxEdit Multiple Files

Edit Multiple Files

b). The command :buffer 3 (without the s at the end) will open file3 for editing.

In the image above, a pound sign (&-8211;) indicates that the file is currently open but in the background, while %a marks the file that is currently being edited. On the other hand, a blank space after the file number (3 in the above example) indicates that the file has not yet been opened.

Temporary vi buffers

To copy a couple of consecutive lines (let’s say 4, for example) into a temporary buffer named a (not associated with a file) and place those lines in another part of the file later in the current vi section, we need to…

1. Press the ESC key to be sure we are in vi Command mode.

2. Place the cursor on the first line of the text we wish to copy.

3. Type “a4yy” to copy the current line, along with the 3 subsequent lines, into a buffer named a. We can continue editing our file – we do not need to insert the copied lines immediately.

4. When we reach the location for the copied lines, use “a before the p or P commands to insert the lines copied into the buffer named a:

  1. Type “ap to insert the lines copied into buffer a after the current line on which the cursor is resting.
  2. Type “aP to insert the lines copied into buffer a before the current line.

If we wish, we can repeat the above steps to insert the contents of buffer a in multiple places in our file. A temporary buffer, as the one in this section, is disposed when the current window is closed.

Summary

As we have seen, vi/m is a powerful and versatile text editor for the CLI. Feel free to share your own tricks and comments below.

Reference Links
  1. About the LFCS
  2. Why get a Linux Foundation Certification?
  3. Register for the LFCS exam

Update: If you want to extend your VI editor skills, then I would suggest you read following two guides that will guide you to some useful VI editor tricks and tips.

Part 1: Learn Useful ‘Vi/Vim’ Editor Tips and Tricks to Enhance Your Skills

Part 2: 8 Interesting ‘Vi/Vim’ Editor Tips and Tricks

School Forums 0 Comments
May 28 2019
LFCS: How to Archive/Compress Files & Directories, Setting File Attributes and Finding Files in Linux – Part 3

LFCS: How to Archive/Compress Files & Directories, Setting File Attributes and Finding Files in Linux – Part 3 &-8211; this Article or News was published on this date:2019-05-28 18:58:50 kindly share it with friends if you find it helpful

Recently, the Linux Foundation started the LFCS (Linux Foundation Certified Sysadmin) certification, a brand new program whose purpose is allowing individuals from all corners of the globe to have access to an exam, which if approved, certifies that the person is knowledgeable in performing basic to intermediate system administration tasks on Linux systems. This includes supporting already running systems and services, along with first-level troubleshooting and analysis, plus the ability to decide when to escalate issues to engineering teams.

Linux Foundation Certified Sysadmin – Part 3Linux Foundation Certified Sysadmin – Part 3

Linux Foundation Certified Sysadmin – Part 3

Please watch the below video that gives the idea about The Linux Foundation Certification Program.

This post is Part 3 of a 10-tutorial series, here in this part, we will cover how to archive/compress files and directories, set file attributes, and find files on the filesystem, that are required for the LFCS certification exam.

Archiving and Compression Tools

A file archiving tool groups a set of files into a single standalone file that we can backup to several types of media, transfer across a network, or send via email. The most frequently used archiving utility in Linux is tar. When an archiving utility is used along with a compression tool, it allows to reduce the disk size that is needed to store the same files and information.

The tar utility

tar bundles a group of files together into a single archive (commonly called a tar file or tarball). The name originally stood for tape archiver, but we must note that we can use this tool to archive data to any kind of writeable media (not only to tapes). Tar is normally used with a compression tool such as gzip, bzip2, or xz to produce a compressed tarball.

Basic syntax:
- tar [options] [pathname ...]

Where … represents the expression used to specify which files should be acted upon.

Most commonly used tar commands
Long option Abbreviation Description
 –create  c  Creates a tar archive
 –concatenate  A  Appends tar files to an archive
 –append  r  Appends files to the end of an archive
 –update  u  Appends files newer than copy in archive
 –diff or –compare  d  Find differences between archive and file system
 –file archive  f  Use archive file or device ARCHIVE
 –list  t  Lists the contents of a tarball
 –extract or –get  x  Extracts files from an archive
Normally used operation modifiers
Long option Abbreviation Description
 –directory dir  C  Changes to directory dir before performing operations
 –same-permissions  p  Preserves original permissions
 –verbose  v  Lists all files read or extracted. When this flag is used along with –list, the file sizes, ownership, and time stamps are displayed.
 –verify  W  Verifies the archive after writing it
 –exclude file  —  Excludes file from the archive
 –exclude=pattern  X  Exclude files, given as a PATTERN
 –gzip or –gunzip  z  Processes an archive through gzip
 –bzip2  j  Processes an archive through bzip2
 –xz  J  Processes an archive through xz

Gzip is the oldest compression tool and provides the least compression, while bzip2 provides improved compression. In addition, xz is the newest but (usually) provides the best compression. This advantages of best compression come at a price: the time it takes to complete the operation, and system resources used during the process.

Normally, tar files compressed with these utilities have .gz, .bz2, or .xz extensions, respectively. In the following examples we will be using these files: file1, file2, file3, file4, and file5.

Grouping and compressing with gzip, bzip2 and xz

Group all the files in the current working directory and compress the resulting bundle with gzip, bzip2, and xz (please note the use of a regular expression to specify which files should be included in the bundle – this is to prevent the archiving tool to group the tarballs created in previous steps).

- tar czf myfiles.tar.gz file[0-9]
- tar cjf myfiles.tar.bz2 file[0-9]
- tar cJf myfile.tar.xz file[0-9]
Linux Foundation Certified Sysadmin – Part 3Compress Multiple Files Using tar

Compress Multiple Files

Listing the contents of a tarball and updating / appending files to the bundle

List the contents of a tarball and display the same information as a long directory listing. Note that update or append operations cannot be applied to compressed files directly (if you need to update or append a file to a compressed tarball, you need to uncompress the tar file and update / append to it, then compress again).

- tar tvf [tarball]
Linux Foundation Certified Sysadmin – Part 3Check Files in tar Archive

List Archive Content

Run any of the following commands:

- gzip -d myfiles.tar.gz	[-1] 
- bzip2 -d myfiles.tar.bz2	[-2] 
- xz -d myfiles.tar.xz 		[-3]

Then

- tar --delete --file myfiles.tar file4 (deletes the file inside the tarball)
- tar --update --file myfiles.tar file4 (adds the updated file)

and

- gzip myfiles.tar		[ if you choose -1 above ]
- bzip2 myfiles.tar		[ if you choose -2 above ]
- xz myfiles.tar 		[ if you choose -3 above ]

Finally,

- tar tvf [tarball] -again

and compare the modification date and time of file4 with the same information as shown earlier.

Excluding file types

Suppose you want to perform a backup of user’s home directories. A good sysadmin practice would be (may also be specified by company policies) to exclude all video and audio files from backups.

Maybe your first approach would be to exclude from the backup all files with an .mp3 or .mp4 extension (or other extensions). What if you have a clever user who can change the extension to .txt or .bkp, your approach won’t do you much good. In order to detect an audio or video file, you need to check its file type with file. The following shell script will do the job.

-!/bin/bash
- Pass the directory to backup as first argument.
DIR=$1
- Create the tarball and compress it. Exclude files with the MPEG string in its file type.
- -If the file type contains the string mpeg, $? (the exit status of the most recently executed command) expands to 0, and the filename is redirected to the exclude option. Otherwise, it expands to 1.
- -If $? equals 0, add the file to the list of files to be backed up.
tar X (for i in $DIR/*; do file $i | grep -i mpeg; if [ $? -eq 0 ]; then echo $i; fi;done) -cjf backupfile.tar.bz2 $DIR/*
Linux Foundation Certified Sysadmin – Part 3Exclude Files in tar Archive

Exclude Files in tar

Restoring backups with tar preserving permissions

You can then restore the backup to the original user’s home directory (user_restore in this example), preserving permissions, with the following command.

- tar xjf backupfile.tar.bz2 --directory user_restore --same-permissions
Linux Foundation Certified Sysadmin – Part 3Restore Files from tar Archive

Restore Files from Archive

Read Also:

  1. 18 tar Command Examples in Linux
  2. Dtrx – An Intelligent Archive Tool for Linux

Using find Command to Search for Files

The find command is used to search recursively through directory trees for files or directories that match certain characteristics, and can then either print the matching files or directories or perform other operations on the matches.

Normally, we will search by name, owner, group, type, permissions, date, and size.

Basic syntax:

&-8211; find [directory_to_search] [expression]

Finding files recursively according to Size

Find all files (-f) in the current directory (.) and 2 subdirectories below (-maxdepth 3 includes the current working directory and 2 levels down) whose size (-size) is greater than 2 MB.

- find . -maxdepth 3 -type f -size +2M
Linux Foundation Certified Sysadmin – Part 3Find Files by Size in Linux

Find Files Based on Size

Finding and deleting files that match a certain criteria

Files with 777 permissions are sometimes considered an open door to external attackers. Either way, it is not safe to let anyone do anything with files. We will take a rather aggressive approach and delete them! (‘{}‘ + is used to “collect” the results of the search).

- find /home/user -perm 777 -exec rm '{}' +
Linux Foundation Certified Sysadmin – Part 3Find all 777 Permission Files

Find Files with 777Permission

Finding files per atime or mtime

Search for configuration files in /etc that have been accessed (-atime) or modified (-mtime) more (+180) or less (-180) than 6 months ago or exactly 6 months ago (180).

Modify the following command as per the example below:

- find /etc -iname "*.conf" -mtime -180 -print
Linux Foundation Certified Sysadmin – Part 3Find Files by Modification Time

Find Modified Files

Read Also: 35 Practical Examples of Linux ‘find’ Command

File Permissions and Basic Attributes

The first 10 characters in the output of ls -l are the file attributes. The first of these characters is used to indicate the file type:

  1. – : a regular file
  2. -d : a directory
  3. -l : a symbolic link
  4. -c : a character device (which treats data as a stream of bytes, i.e. a terminal)
  5. -b : a block device (which handles data in blocks, i.e. storage devices)

The next nine characters of the file attributes are called the file mode and represent the read (r), write (w), and execute (x) permissions of the file’s owner, the file’s group owner, and the rest of the users (commonly referred to as “the world”).

Whereas the read permission on a file allows the same to be opened and read, the same permission on a directory allows its contents to be listed if the execute permission is also set. In addition, the execute permission in a file allows it to be handled as a program and run, while in a directory it allows the same to be cd’ed into it.

File permissions are changed with the chmod command, whose basic syntax is as follows:

- chmod [new_mode] file

Where new_mode is either an octal number or an expression that specifies the new permissions.

The octal number can be converted from its binary equivalent, which is calculated from the desired file permissions for the owner, the group, and the world, as follows:

The presence of a certain permission equals a power of 2 (r=22, w=21, x=20), while its absence equates to 0. For example:

Linux Foundation Certified Sysadmin – Part 3Linux File Permissions

File Permissions

To set the file’s permissions as above in octal form, type:

- chmod 744 myfile

You can also set a file’s mode using an expression that indicates the owner’s rights with the letter u, the group owner’s rights with the letter g, and the rest with o. All of these “individuals” can be represented at the same time with the letter a. Permissions are granted (or revoked) with the + or – signs, respectively.

Revoking execute permission for a shell script to all users

As we explained earlier, we can revoke a certain permission prepending it with the minus sign and indicating whether it needs to be revoked for the owner, the group owner, or all users. The one-liner below can be interpreted as follows: Change mode for all (a) users, revoke (–) execute permission (x).

- chmod a-x backup.sh

Granting read, write, and execute permissions for a file to the owner and group owner, and read permissions for the world.

When we use a 3-digit octal number to set permissions for a file, the first digit indicates the permissions for the owner, the second digit for the group owner and the third digit for everyone else:

  1. Owner: (r=22 + w=21 + x=20 = 7)
  2. Group owner: (r=22 + w=21 + x=20 = 7)
  3. World: (r=22 + w=0 + x=0 = 4),
- chmod 774 myfile

In time, and with practice, you will be able to decide which method to change a file mode works best for you in each case. A long directory listing also shows the file’s owner and its group owner (which serve as a rudimentary yet effective access control to files in a system):

Linux Foundation Certified Sysadmin – Part 3Linux File Listing

Linux File Listing

File ownership is changed with the chown command. The owner and the group owner can be changed at the same time or separately. Its basic syntax is as follows:

- chown user:group file

Where at least user or group need to be present.

Few Examples

Changing the owner of a file to a certain user.

- chown gacanepa sent

Changing the owner and group of a file to an specific user:group pair.

- chown gacanepa:gacanepa TestFile

Changing only the group owner of a file to a certain group. Note the colon before the group’s name.

- chown :gacanepa email_body.txt

Conclusion

As a sysadmin, you need to know how to create and restore backups, how to find files in your system and change their attributes, along with a few tricks that can make your life easier and will prevent you from running into future issues.

I hope that the tips provided in the present article will help you to achieve that goal. Feel free to add your own tips and ideas in the comments section for the benefit of the community. Thanks in advance!

Reference Links
  1. About the LFCS
  2. Why get a Linux Foundation Certification?
  3. Register for the LFCS exam
School Forums 0 Comments
May 28 2019
LFCS: Partitioning Storage Devices, Formatting Filesystems and Configuring Swap Partition – Part 4

LFCS: Partitioning Storage Devices, Formatting Filesystems and Configuring Swap Partition &-8211; Part 4 &-8211; this Article or News was published on this date:2019-05-28 18:58:32 kindly share it with friends if you find it helpful

Last August, the Linux Foundation launched the LFCS certification (Linux Foundation Certified Sysadmin), a shiny chance for system administrators to show, through a performance-based exam, that they can perform overall operational support of Linux systems: system support, first-level diagnosing and monitoring, plus issue escalation – if needed – to other support teams.

Linux Foundation Certified Sysadmin – Part 4Linux Foundation Certified Sysadmin – Part 4

Linux Foundation Certified Sysadmin – Part 4

Please aware that Linux Foundation certifications are precise, totally based on performance and available through an online portal anytime, anywhere. Thus, you no longer have to travel to a examination center to get the certifications you need to establish your skills and expertise.

Please watch the below video that explains The Linux Foundation Certification Program.

This post is Part 4 of a 10-tutorial series, here in this part, we will cover the Partitioning storage devices, Formatting filesystems and Configuring swap partition, that are required for the LFCS certification exam.

Partitioning Storage Devices

Partitioning is a means to divide a single hard drive into one or more parts or “slices” called partitions. A partition is a section on a drive that is treated as an independent disk and which contains a single type of file system, whereas a partition table is an index that relates those physical sections of the hard drive to partition identifications.

In Linux, the traditional tool for managing MBR partitions (up to ~2009) in IBM PC compatible systems is fdisk. For GPT partitions (~2010 and later) we will use gdisk. Each of these tools can be invoked by typing its name followed by a device name (such as /dev/sdb).

Managing MBR Partitions with fdisk

We will cover fdisk first.

- fdisk /dev/sdb

A prompt appears asking for the next operation. If you are unsure, you can press the ‘m‘ key to display the help contents.

Linux Foundation Certified Sysadmin – Part 4fdisk Help Menu

fdisk Help Menu

In the above image, the most frequently used options are highlighted. At any moment, you can press ‘p‘ to display the current partition table.

Linux Foundation Certified Sysadmin – Part 4Check Partition Table in Linux

Show Partition Table

The Id column shows the partition type (or partition id) that has been assigned by fdisk to the partition. A partition type serves as an indicator of the file system, the partition contains or, in simple words, the way data will be accessed in that partition.

Please note that a comprehensive study of each partition type is out of the scope of this tutorial – as this series is focused on the LFCS exam, which is performance-based.

Some of the options used by fdisk as follows:

You can list all the partition types that can be managed by fdisk by pressing the ‘l‘ option (lowercase l).

Press ‘d‘ to delete an existing partition. If more than one partition is found in the drive, you will be asked which one should be deleted.

Enter the corresponding number, and then press ‘w‘ (write modifications to partition table) to apply changes.

In the following example, we will delete /dev/sdb2, and then print (p) the partition table to verify the modifications.

Linux Foundation Certified Sysadmin – Part 4fdisk Command Options

fdisk Command Options

Press ‘n‘ to create a new partition, then ‘p‘ to indicate it will be a primary partition. Finally, you can accept all the default values (in which case the partition will occupy all the available space), or specify a size as follows.

Linux Foundation Certified Sysadmin – Part 4Create New Partition in Linux

Create New Partition

If the partition Id that fdisk chose is not the right one for our setup, we can press ‘t‘ to change it.

Linux Foundation Certified Sysadmin – Part 4Change Partition Name in Linux

Change Partition Name

When you’re done setting up the partitions, press ‘w‘ to commit the changes to disk.

Linux Foundation Certified Sysadmin – Part 4Save Partition Changes

Save Partition Changes

Managing GPT Partitions with gdisk

In the following example, we will use /dev/sdb.

- gdisk /dev/sdb

We must note that gdisk can be used either to create MBR or GPT partitions.

Linux Foundation Certified Sysadmin – Part 4Create GPT Partitions in Linux

Create GPT Partitions

The advantage of using GPT partitioning is that we can create up to 128 partitions in the same disk whose size can be up to the order of petabytes, whereas the maximum size for MBR partitions is 2 TB.

Note that most of the options in fdisk are the same in gdisk. For that reason, we will not go into detail about them, but here’s a screenshot of the process.

Linux Foundation Certified Sysadmin – Part 4gdisk Command Options

gdisk Command Options

Formatting Filesystems

Once we have created all the necessary partitions, we must create filesystems. To find out the list of filesystems supported in your system, run.

- ls /sbin/mk*
Linux Foundation Certified Sysadmin – Part 4Check Filesystems Type in Linux

Check Filesystems Type

The type of filesystem that you should choose depends on your requirements. You should consider the pros and cons of each filesystem and its own set of features. Two important attributes to look for in a filesystem are.

  1. Journaling support, which allows for faster data recovery in the event of a system crash.
  2. Security Enhanced Linux (SELinux) support, as per the project wiki, “a security enhancement to Linux which allows users and administrators more control over access control”.

In our next example, we will create an ext4 filesystem (supports both journaling and SELinux) labeled sfnews on /dev/sdb1, using mkfs, whose basic syntax is.

- mkfs -t [filesystem] -L [label] device
or
- mkfs.[filesystem] -L [label] device
Linux Foundation Certified Sysadmin – Part 4Create ext4 Filesystems in Linux

Create ext4 Filesystems

Creating and Using Swap Partitions

Swap partitions are necessary if we need our Linux system to have access to virtual memory, which is a section of the hard disk designated for use as memory, when the main system memory (RAM) is all in use. For that reason, a swap partition may not be needed on systems with enough RAM to meet all its requirements; however, even in that case it’s up to the system administrator to decide whether to use a swap partition or not.

A simple rule of thumb to decide the size of a swap partition is as follows.

Swap should usually equal 2x physical RAM for up to 2 GB of physical RAM, and then an additional 1x physical RAM for any amount above 2 GB, but never less than 32 MB.

So, if:

M = Amount of RAM in GB, and S = Amount of swap in GB, then

If M  2
	S = M *2
Else
	S = M + 2

Remember this is just a formula and that only you, as a sysadmin, have the final word as to the use and size of a swap partition.

To configure a swap partition, create a regular partition as demonstrated earlier with the desired size. Next, we need to add the following entry to the /etc/fstab file (X can be either b or c).

/dev/sdX1 swap swap sw 0 0

Finally, let’s format and enable the swap partition.

- mkswap /dev/sdX1
- swapon -v /dev/sdX1

To display a snapshot of the swap partition(s).

- cat /proc/swaps

To disable the swap partition.

- swapoff /dev/sdX1

For the next example, we’ll use /dev/sdc1 (=512 MB, for a system with 256 MB of RAM) to set up a partition with fdisk that we will use as swap, following the steps detailed above. Note that we will specify a fixed size in this case.

Linux Foundation Certified Sysadmin – Part 4Create-Swap-Partition in Linux

Create Swap Partition

Linux Foundation Certified Sysadmin – Part 4Add Swap Partition in Linux

Enable Swap Partition

Conclusion

Creating partitions (including swap) and formatting filesystems are crucial in your road to Sysadminship. I hope that the tips given in this article will guide you to achieve your goals. Feel free to add your own tips & ideas in the comments section below, for the benefit of the community.

Reference Links
  1. About the LFCS
  2. Why get a Linux Foundation Certification?
  3. Register for the LFCS exam
School Forums 0 Comments
May 28 2019
LFCS: How to Mount/Unmount Local and Network (Samba & NFS) Filesystems in Linux – Part 5

LFCS: How to Mount/Unmount Local and Network (Samba & NFS) Filesystems in Linux &-8211; Part 5 &-8211; this Article or News was published on this date:2019-05-28 18:58:21 kindly share it with friends if you find it helpful

The Linux Foundation launched the LFCS certification (Linux Foundation Certified Sysadmin), a brand new program whose purpose is allowing individuals from all corners of the globe to get certified in basic to intermediate system administration tasks for Linux systems, which includes supporting running systems and services, along with overall monitoring and analysis, plus smart decision-making when it comes to raising issues to upper support teams.

Linux Foundation Certified Sysadmin – Part 5Linux Foundation Certified Sysadmin – Part 5

Linux Foundation Certified Sysadmin – Part 5

The following video shows an introduction to The Linux Foundation Certification Program.

This post is Part 5 of a 10-tutorial series, here in this part, we will explain How to mount/unmount local and network filesystems in linux, that are required for the LFCS certification exam.

Mounting Filesystems

Once a disk has been partitioned, Linux needs some way to access the data on the partitions. Unlike DOS or Windows (where this is done by assigning a drive letter to each partition), Linux uses a unified directory tree where each partition is mounted at a mount point in that tree.

A mount point is a directory that is used as a way to access the filesystem on the partition, and mounting the filesystem is the process of associating a certain filesystem (a partition, for example) with a specific directory in the directory tree.

In other words, the first step in managing a storage device is attaching the device to the file system tree. This task can be accomplished on a one-time basis by using tools such as mount (and then unmounted with umount) or persistently across reboots by editing the /etc/fstab file.

The mount command (without any options or arguments) shows the currently mounted filesystems.

- mount
Linux Foundation Certified Sysadmin – Part 5Check Mounted Filesystem in Linux

Check Mounted Filesystem

In addition, mount is used to mount filesystems into the filesystem tree. Its standard syntax is as follows.

- mount -t type device dir -o options

This command instructs the kernel to mount the filesystem found on device (a partition, for example, that has been formatted with a filesystem type) at the directory dir, using all options. In this form, mount does not look in /etc/fstab for instructions.

If only a directory or device is specified, for example.

- mount /dir -o options
or
- mount device -o options

mount tries to find a mount point and if it can’t find any, then searches for a device (both cases in the /etc/fstab file), and finally attempts to complete the mount operation (which usually succeeds, except for the case when either the directory or the device is already being used, or when the user invoking mount is not root).

You will notice that every line in the output of mount has the following format.

device on directory type (options)

For example,

/dev/mapper/debian-home on /home type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)

Reads:

dev/mapper/debian-home is mounted on /home, which has been formatted as ext4, with the following options: rw,relatime,user_xattr,barrier=1,data=ordered

Mount Options

Most frequently used mount options include.

  1. async: allows asynchronous I/O operations on the file system being mounted.
  2. auto: marks the file system as enabled to be mounted automatically using mount -a. It is the opposite of noauto.
  3. defaults: this option is an alias for async,auto,dev,exec,nouser,rw,suid. Note that multiple options must be separated by a comma without any spaces. If by accident you type a space between options, mount will interpret the subsequent text string as another argument.
  4. loop: Mounts an image (an .iso file, for example) as a loop device. This option can be used to simulate the presence of the disk’s contents in an optical media reader.
  5. noexec: prevents the execution of executable files on the particular filesystem. It is the opposite of exec.
  6. nouser: prevents any users (other than root) to mount and unmount the filesystem. It is the opposite of user.
  7. remount: mounts the filesystem again in case it is already mounted.
  8. ro: mounts the filesystem as read only.
  9. rw: mounts the file system with read and write capabilities.
  10. relatime: makes access time to files be updated only if atime is earlier than mtime.
  11. user_xattr: allow users to set and remote extended filesystem attributes.
Mounting a device with ro and noexec options
- mount -t ext4 /dev/sdg1 /mnt -o ro,noexec

In this case we can see that attempts to write a file to or to run a binary file located inside our mounting point fail with corresponding error messages.

- touch /mnt/myfile
- /mnt/bin/echo “Hi there”
Linux Foundation Certified Sysadmin – Part 5Mount Device in Read Write Mode

Mount Device Read Write

Mounting a device with default options

In the following scenario, we will try to write a file to our newly mounted device and run an executable file located within its filesystem tree using the same commands as in the previous example.

- mount -t ext4 /dev/sdg1 /mnt -o defaults
Linux Foundation Certified Sysadmin – Part 5Mount Device in Linux

Mount Device

In this last case, it works perfectly.

Unmounting Devices

Unmounting a device (with the umount command) means finish writing all the remaining “on transit” data so that it can be safely removed. Note that if you try to remove a mounted device without properly unmounting it first, you run the risk of damaging the device itself or cause data loss.

That being said, in order to unmount a device, you must be “standing outside” its block device descriptor or mount point. In other words, your current working directory must be something else other than the mounting point. Otherwise, you will get a message saying that the device is busy.

Linux Foundation Certified Sysadmin – Part 5Unmount Device in Linux

Unmount Device

An easy way to “leave” the mounting point is typing the cd command which, in lack of arguments, will take us to our current user’s home directory, as shown above.

Mounting Common Networked Filesystems

The two most frequently used network file systems are SMB (which stands for “Server Message Block”) and NFS (“Network File System”). Chances are you will use NFS if you need to set up a share for Unix-like clients only, and will opt for Samba if you need to share files with Windows-based clients and perhaps other Unix-like clients as well.

Read Also

  1. Setup Samba Server in RHEL/CentOS and Fedora
  2. Setting up NFS (Network File System) on RHEL/CentOS/Fedora and Debian/Ubuntu

The following steps assume that Samba and NFS shares have already been set up in the server with IP 192.168.0.10 (please note that setting up a NFS share is one of the competencies required for the LFCE exam, which we will cover after the present series).

Mounting a Samba share on Linux

Step 1: Install the samba-client samba-common and cifs-utils packages on Red Hat and Debian based distributions.

- yum update && yum install samba-client samba-common cifs-utils
- aptitude update && aptitude install samba-client samba-common cifs-utils

Then run the following command to look for available samba shares in the server.

- smbclient -L 192.168.0.10

And enter the password for the root account in the remote machine.

Linux Foundation Certified Sysadmin – Part 5Mount Samba Share in Linux

Mount Samba Share

In the above image we have highlighted the share that is ready for mounting on our local system. You will need a valid samba username and password on the remote server in order to access it.

Step 2: When mounting a password-protected network share, it is not a good idea to write your credentials in the /etc/fstab file. Instead, you can store them in a hidden file somewhere with permissions set to 600, like so.

- mkdir /media/samba
- echo “username=samba_username” > /media/samba/.smbcredentials
- echo “password=samba_password” >> /media/samba/.smbcredentials
- chmod 600 /media/samba/.smbcredentials

Step 3: Then add the following line to /etc/fstab file.

- //192.168.0.10/gacanepa /media/samba cifs credentials=/media/samba/.smbcredentials,defaults 0 0

Step 4: You can now mount your samba share, either manually (mount //192.168.0.10/gacanepa) or by rebooting your machine so as to apply the changes made in /etc/fstab permanently.

- mount -a
Linux Foundation Certified Sysadmin – Part 5Mount Password Protect Samba Share

Mount Password Protect Samba Share

Mounting a NFS share on Linux

Step 1: Install the nfs-common and portmap packages on Red Hat and Debian based distributions.

- yum update && yum install nfs-utils nfs-utils-lib
- aptitude update && aptitude install nfs-common

Step 2: Create a mounting point for the NFS share.

- mkdir /media/nfs

Step 3: Add the following line to /etc/fstab file.

192.168.0.10:/NFS-SHARE /media/nfs nfs defaults 0 0

Step 4: You can now mount your nfs share, either manually (mount 192.168.0.10:/NFS-SHARE) or by rebooting your machine so as to apply the changes made in /etc/fstab permanently.

Linux Foundation Certified Sysadmin – Part 5Mount NFS Share in Linux

Mount NFS Share

Mounting Filesystems Permanently

As shown in the previous two examples, the /etc/fstab file controls how Linux provides access to disk partitions and removable media devices and consists of a series of lines that contain six fields each; the fields are separated by one or more spaces or tabs. A line that begins with a hash mark (&-8211;) is a comment and is ignored.

Each line has the following format.

file system> mount point> type> options> dump> pass>

Where:

  1. file system>: The first column specifies the mount device. Most distributions now specify partitions by their labels or UUIDs. This practice can help reduce problems if partition numbers change.
  2. mount point>: The second column specifies the mount point.
  3. type>: The file system type code is the same as the type code used to mount a filesystem with the mount command. A file system type code of auto lets the kernel auto-detect the filesystem type, which can be a convenient option for removable media devices. Note that this option may not be available for all filesystems out there.
  4. options>: One (or more) mount option(s).
  5. dump>: You will most likely leave this to 0 (otherwise set it to 1) to disable the dump utility to backup the filesystem upon boot (The dump program was once a common backup tool, but it is much less popular today.)
  6. pass>: This column specifies whether the integrity of the filesystem should be checked at boot time with fsck. A 0 means that fsck should not check a filesystem. The higher the number, the lowest the priority. Thus, the root partition will most likely have a value of 1, while all others that should be checked should have a value of 2.
Mount Examples

1. To mount a partition with label TECMINT at boot time with rw and noexec attributes, you should add the following line in /etc/fstab file.

LABEL=TECMINT /mnt ext4 rw,noexec 0 0

2. If you want the contents of a disk in your DVD drive be available at boot time.

/dev/sr0    /media/cdrom0    iso9660    ro,user,noauto    0    0

Where /dev/sr0 is your DVD drive.

Summary

You can rest assured that mounting and unmounting local and network filesystems from the command line will be part of your day-to-day responsibilities as sysadmin. You will also need to master /etc/fstab. I hope that you have found this article useful to help you with those tasks. Feel free to add your comments (or ask questions) below and to share this article through your network social profiles.

Reference Links
  1. About the LFCS
  2. Why get a Linux Foundation Certification?
  3. Register for the LFCS exam
School Forums 0 Comments