Image for post
Image for post

Ops Scripting w. Bash: Frequency 3

Tracking Frequency in BASH (Bourne Again Shell): Part III

This article shows how to do a serial pipeline style with loop constructs and a a survey of popular command line tools often used with shell programming.

  1. verify if shell string is valid
  2. create an entry in the associative array with the current count
filter_valid_lines | slice_7th_column | increment_count_entry

Previous Articles

The Problem

Procedural Solutions Way

The Solutions for the Serial Pipeline Way

For these solutions, we will use a sub-shell to pre-process the lines, and then send valid list of shells directly to the while loop with <<< operator:

while read -r SHELL; do
process_shell_entry $SHELL
done <<< $( create_list_of_valid_shells )

Solution 3: Splitting with IFS and Read

In this solution, we use the internal input-field-separator IFS in conjunction with read to split each line into an array, where we slice off the shell column.

Bash read split with Grep Filter
Pure BASH Solution

Solution 4: Splitting with cut command

The cut command can remove sections form each line of a file (or files). We specify the 7th column; unlike an array, the count of columns starts at 1.

Cut with Grep Filter Solution

Solution 5: Splitting with sed

A popular tool in Linux and Unix is sed (stream editor), which has a substitute ability that we can use to only substitute the part we are interested in:

sed 's/pattern/replacement/' input_file
Sed Substitute with Grep Filter Solution
Sed Substitute with Sed Delete Filter Solution

Solution 6: Splittling with awk command

The awk tool is extremely popular for working with field separated files. With awk we can print out any column:

awk '{print $5}' input_file # print's 5th field
Awk with Grep Filter Solution
Awk with Awk Match Filter Solution

Solution 7: Splitting with grep only-match command

The grep match tool has the ability to only print out the matched content, instead of the whole string when we use the only-match option of -o. This will extract only the shell portion of the string.

Grep with Grep Filter Solution

Soluton 8: Splitting with perl command

The perl tool is a robust and fast tool to do kung-fu with strings. You can use Perl in command line mode with auto-splitting -a enabled, and then print out the column using the Perl’s built-in $F variable.

Perl with Grep Filter Solution
Perl with Perl Filter Solution
##################### awk-like #####################
# perl auto-split w/ grep filter
grep -v ':$' passwd | perl -naF':' -e 'print $F[6]'
# perl auto-split w perl match filter
perl
-naF':' -e 'print $F[6] if $F[6] !~ /^$/' passwd
############## grep-like with group-match ##############
# perl group-match w/ grep filter
grep -v ':$' passwd | perl -ne 'print $1 if /([^:]*)$/'
# perl group-match w/ perl match filter
perl -ne 'print $1 if /([^:]*)$/ and $1 !~ /^$/' passwd
##################### sed-like #####################
# perl substitute + group-match w/ grep filter
grep -v ':$' passwd | perl -pne 's/.*:([^:].*)$/$1/;'
# perl substitute + group-match and substitute to filter
perl -pne 's/.*:([^:].*)$/$1/ and s/^\s*$//' passwd

Why SubShell?

This is an excellent question. The current pattern above is to do the following:

while read -r VAR; do process $VAR; done <<< $(cmd < file | cmd)
cmd < file | cmd | while read -r VAR; do process $VAR; done

Pipe to a Loop

When you do a command | loop, you fork a sub-process with the loop, which does a fork/exec command and links the standard output of the command to the standard input of loop process. Unfortunately, variable changes in the loop are force scoped to just the loop, and thus will be lost outside of the loop.

Eval to a Loop

When you do a loop <<< $(command), all of the standard output from the command is gathered into a buffer, then the loop will use the buffer as loop input. The subshell with $(command) is run at eval and standard out is substituted in-place of the $(command), then the execution of the statement happens. Thus, your variables set inside the loop are preserved outside of the loop.

Solution Example with For Loop

Another way to use subshells if this format looks too funky, is to go back to a for loop and split the lines using this construct:

for VAR in $(cmd < file); do process $VAR; done

The Conclusion

So these solutions are essentially a survey of popular tools that can be used for processing strings as an alternative to the internal splitting mechanism with IFS.

  • limitations of command | while -r VAR construct
  • alternative loop construct with for VAR in $(command) construct
  • default IFS will filter empty lines as separator is space, tab, and newlines.
  • using splitting with an external tool: cut, awk, perl
  • using substitute with group match with an external tool: sed, perl
  • using group match or only-match with an external tool: grep, perl
  • awk can match as well as slice columns as well as other features
  • sed can substitute and delete as well as other features
  • perl can match, group-match, substitute, and slice columns.

Written by

Linux NinjaPants Automation Engineering Mutant — exploring DevOps, Kubernetes, CNI, IAC

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store