Everything for the Ubuntu Server Administrator

User Tools

Site Tools


Server Administration

Server Applications

At the Command Line


Copyright 2013 Applied Conscious Technologies, LLC

Terms of Agreement


submit to reddit


At the Command Line

Text Manipulation

As an administrator searching for meaning in log files or the output of admin commands requires reducing the output to a smaller, more meaningful subset.

The Unix philosophy includes using plain text whenever practical because there are so many excellent tools to operate on plain text. This allows us to find what we need with sorting, grep and using filters to process text.

awk one liners:

sed one-liners:

sed man page:

awk man page:

See also: Regular Expressions

Replace leading tabs with spaces for all php files in the current tree

find . -name '*.php' ! -type d -exec bash -c 'expand -i -t 4 "$0" > /tmp/e && mv /tmp/e "$0"' {} \;

Inline Text Manipulation

These are atoms that can be strung together to create a text file filter. One easy way to use one or more of these filters on a file is to use the cat (catenate) command to print the contents of a file and then use pipe characters to run the output from cat though one or more filters.

You can then, optionally, have the output of your command go into a file using the ”>” file output character.

Using Filters

Process a File and Save Output

This example takes the content of sometextfile.txt, runs it through the “Remove Trailing Whitespace from Lines” filter and puts the result in myoutputfile.txt,

cat sometextfile.txt | awk '{sub(/[ \t]+$/, "")};1' >~/myoutputfile.txt

You can then view this file with:

less ~/myoutputfile.txt

Some Filters

Search and Replace Filter

sed 's/string1/string2/g'

Remove Trailing Whitespace from Lines Filter

awk '{sub(/[ \t]+$/, "")};1'

Remove Leading Whitespace from Lines Filter

awk '{sub(/^[ \t]+/, "")};1'

Remove Leading and Trailing Whitespace from Lines Filter

awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1'

Remove Extra Whitespace Filter

awk '{$1=$1};1'


tr -s [:blank:] '_'

Remove First Line

sed -e '1d'

Case Conversion Filter

tr '[:lower:]' '[:upper:]'

Remove Non-printable Characters Filter

tr -dc '[:print:]'

text_manipulation.txt · Last modified: 2016/12/10 12:05 by andrew