I've been seeing a lot of sed lately, and I find it to be a rather confusing command.
The manpages weren't particularly helpful, but I do know that it can be used for parsing the output of other commands.
What exactly is sed and what are it's uses? I'm looking for a comprehensive answer covering what sed is, what commonly it is used for, and some basic examples/syntax.
In basic usage it is used for 'search and replace' with strings.
echo "The quick brown fox jumps over the lazy dog" | sed 's/dog/cat/'
returns
"The quick brown fox jumps over the lazy cat"
Sed really shines when regular expressions are used with it.
You might like to take a look at this article about sed, its quite comprehensive.
Definitions
Unix System V: a practical guide, book by Mark Sobell:
The sed utility is a batch (noninteractive) editor. The sed commands are usually stored in a script-file . . .although you can give simple sed commands from the command line . . .
Man page for sed (GNU sed) 4.2.2 :
Sed is a stream editor. A stream editor is used to per‐ form basic text transformations on an input stream (a file or input from a pipeline).
My informal definition:
Sed (short for stream editor) is a text-processing utility that has been developed at the time when text was processed one line at a time, but remains one of the most powerful Unix/Linux utilities; at the same time, it is a form of scripting language, designed specifically for processing text.
Uses
As the definitions suggest, sed is used for batch processing lines of text, text files, and piped streams of text. Most frequently it is used for replacing as well as deleting text:
echo "stackexchange" | sed 's/stackexchange/askubuntu/'
However, it may also be used to mimic behavior of other commands.For instance,
dmesg | head -n 3 (printing first 3 lines), we can do dmesg | sed -n 1,3p. dmesg | grep 'wlan0' (searching for a string), we can do dmesg | sed -n '/wlan0/p'The big advantage that sed has over other text-processing utilities is the -i flag, meaning we can not just output the edited text to the screen, but actually save the edit to the original file. awk flavors , by contrast, only have such feature in GNU awk version.
sed can take input on the command line, with multiple patterns separated by semicolon (;) or from a script file specified after -f flag, e.g. cat someTextfile.txt | sed -f myScript.sed
Sed applications and examples
This answer is a work in progress - it misses more examples about the susbstitute command
sed?sed = Stream EDitor
The description in the manual page for GNU sed 4.2.2 reports:
Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline). While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient. But it is sed's ability to filter text in a pipeline which particularly distinguishes it from other types of editors.
The decription in the GNU sed page at gnu.org reports:
sed (stream editor) isn't an interactive text editor. Instead, it is used to filter text, i.e., it takes text input, performs some operation (or set of operations) on it, and outputs the modified text. sed is typically used for extracting part of a file using pattern matching or substituting multiple occurrences of a string within a file.
sed used for?It can be used to perform complex modifications to streams of data (usually text, but it can be used also to modify binary data).
Among the most common cases of use there are:
These are the cases of use covered in this answer.
sed reads the input from a file stored in the filesystem if a filename is specified in the command-line arguments during its invocation, or from stdin if no filename is specified.
Minimal invocation using a file stored in the filesystem:
sed '' file
Minimal invocation using stdin:
# herestring
<<<'Hello, World!' sed ''
# heredoc
<<'EOF' sed ''
heredoc> Hello, World!
heredoc> EOF
# file
<'file' sed ''
# pipe
echo 'Hello, World!' | sed ''
sed by default reads the input file line-by-line; it reads one line, it removes the line's trailing newline and puts the processed line into a "pattern space"; finally, it executes the listed commands on the current content of the pattern space and reads a new line from the input file.
When no command is specified or when a p or a d command is specified *, sed will always print the current content of the pattern space followed by a newline at each iteration regardless:
user@debian ~ % sed '' file
Hello, world! # no command but the lines are printed
user@debian ~ % sed 'p' file
Hello, World!
Hello, World! # the p command prints the lines already printed
user@debian ~ % sed 'd' file
user@debian ~ % # the d command deletes the lines that would be printed
To prevent this one may invoke sed along with the -n switch:
user@debian ~ % sed -n '' file
user@debian ~ % sed -n 'p' file
Hello, World!
user@debian ~ % sed -n 'd' file
user@debian ~ %
* Speaking only for the p, d and s commands, which are the commands covered in this answer.
sed can process the whole input file or process only selected lines of the input file; the selection of the lines of the input file to be processed is done by specifying "addresses"; an address can be (among other things) either a line number or a pattern; ranges of lines may be selected by specifying ranges of addresses.
Possible combinations of addresses are:
<N> (where <N> is a number): the following command / commands will be executed only on line number <N>;<N>,<M> (where <N> and <M> are two numbers, <N> > <M>): the following command / commands will be executed on lines ranging from line number <N> to line number <M> inclusive;/<pattern>/ (where <pattern> is a basic or extended regular expression): the following command / commands will be executed only on lines containing an occurence of <pattern>;/<pattern1>/,/<pattern2>/ (where <pattern1> and <pattern2> are basic or extended regular expressions): the following command / commands will be executed on lines ranging from the first line containing an occurence of <pattern1> to the next line containing an occurence of <pattern2>, multiple times in case of multiple ordered <pattern1>-<pattern2> occurences;<N>,/pattern/ (where <N> is a number and <pattern> is a basic or extended regular expression): the following command / commands will be executed on lines ranging from line number <N> to the first line containing an occurence of <pattern>;/pattern/,<N> (where <pattern> is a basic or extended regular expression and <N> is a number): the following command / commands will be executed on lines ranging from the first line containing an occurence of <pattern> to line number <N>;The selection performed in order to print, delete or perform substitutions on ranges of lines will always include the lines matching the specified addresses; furthermore, the selection performed in order to print, delete or perform substitutions on ranges of lines using patterns is lazy and global (i.e., each affected range will always be the smallest as possible, and multiple ranges will be affected).
When printing ranges of lines or printing only lines on which a substitution has been performed, it's necessary to invoke sed along with the -n switch in order to prevent lines matching the criterium to be printed twice (this happens only when printing ranges of lines) and in order to prevent lines not matching the criterium to be printed regardless.
A selection of lines to be processed must be followed by a command or by multiple semicolon-separated commands grouped using braces.
The commands used to print or delete a selection are, respectively:
p: prints lines matching the specified address / range of addresses;d: deletes lines matching the specified address / range of addresses;When one of these commands is not preceded by an address / selection, the command is executed globally, i.e. on each line of the input file.
Sample file:
line1
line2
line3
line4
line5
<N>:sed -n '<N>p' file
user@debian ~ % sed -n '3p' file
line3
<N>:sed '<N>d' file
user@debian ~ % sed '3d' file
line1
line2
line4
line5
<N> to <M> inclusive:sed -n '<N>,<M>p' file
user@debian ~ % sed -n '2,4p' file
line2
line3
line4
<N> to <M> inclusive:sed '<N>,<M>d' file
user@debian ~ % sed '2,4d' file
line1
line5
Sample file:
First line
Start printing / deleting here
Random line
Random line
Random line
Stop printing / deleting here
Last line
<pattern>:sed -n '/<pattern>/p' file
user@debian ~ % sed -n '/print/p' file
Start printing / deleting here
Stop printing / deleting here
<pattern>:sed '/<pattern>/d' file
user@debian ~ % sed '/print/d' file
First line
Random line
Random line
Random line
Last line
<pattern1> to the line matching <pattern2> inclusive:sed -n '/<pattern1>/,/<pattern2>/p' file
user@debian ~ % sed -n '/Start/,/Stop/p' file
Start printing / deleting here
Random line
Random line
Random line
Stop printing / deleting here
<pattern1> to the line matching <pattern2> inclusive:sed '/<pattern1>/,/<pattern2>/d' file
user@debian ~ % sed '/Start/,/Stop/d' file
First line
Last line
The command used to perform a substitution on a selection is:
s: substitutes lines matching the specified address / range of addresses;When this command is not preceded by an address / selection, the command is executed globally, i.e. on each line of the input file.
The syntax of the s command is:
s/<pattern>/<replacement_string>/<pattern_flags>
Slashes are "delimiters"; they are used to delimit the <pattern>, <replacement_string> and <pattern_flags> sections;
The delimiter is always the character immediately following the s command; it can be set to any other character, for example, |:
s|<pattern>|<replacement_string>|<pattern_flags>
<pattern> is a basic or extended regular expression; <replacement_string> is a fixed string which may include sed-specific sequences with a special meaning; <pattern_flags> is a list of flags which modify the behavior of <pattern>.
Most common sed-specific sequences with a special meaning:
&: backreference replaced with the string matched by <pattern>;\<N> (where <N> is a number): backreference replaced with the <N> group captured in <pattern>;Most common flags:
g: forces <pattern> to match globally, i.e. multiple times in each line;i: forces <pattern> to match case-insensitively;p: prints lines on which a substitution has been performed once more (useful when using the -n switch in sed's invocation to print only the lines on which a substitution has been performed);Sample file:
A-well-a everybody's heard about the bird
B-b-b-bird, bird, bird, b-bird's the word
A-well-a bird, bird, bird, the bird is the word
A-well-a bird, bird, bird, well the bird is the word
A-well-a bird, bird, bird, b-bird's the word
A-well-a bird, bird, bird, well the bird is the word
A-well-a bird, bird, b-bird's the word
A-well-a bird, bird, bird, b-bird's the word
A-well-a bird, bird, bird, well the bird is the word
A-well-a bird, bird, b-bird's the word
A-well-a don't you know about the bird?
Well, everybody knows that the bird is the word!
A-well-a bird, bird, b-bird's the word
A-well-a...
<pattern> with <replacement_string> on each line:sed 's/<pattern>/<replacement_string>/' file
user@debian ~ % sed 's/bird/birds/' file
A-well-a everybody's heard about the birds
B-b-b-birds, bird, bird, b-bird's the word
A-well-a birds, bird, bird, the bird is the word
A-well-a birds, bird, bird, well the bird is the word
A-well-a birds, bird, bird, b-bird's the word
A-well-a birds, bird, bird, well the bird is the word
A-well-a birds, bird, b-bird's the word
A-well-a birds, bird, bird, b-bird's the word
A-well-a birds, bird, bird, well the bird is the word
A-well-a birds, bird, b-bird's the word
A-well-a don't you know about the birds?
Well, everybody knows that the birds is the word!
A-well-a birds, bird, b-bird's the word
<pattern> with <replacement_string> on each line:sed 's/<pattern>/<replacement_string>/g' file
user@debian ~ % sed 's/bird/birds/g' file
A-well-a everybody's heard about the birds
B-b-b-birds, birds, birds, b-birds's the word
A-well-a birds, birds, birds, the birds is the word
A-well-a birds, birds, birds, well the birds is the word
A-well-a birds, birds, birds, b-birds's the word
A-well-a birds, birds, birds, well the birds is the word
A-well-a birds, birds, b-birds's the word
A-well-a birds, birds, birds, b-birds's the word
A-well-a birds, birds, birds, well the birds is the word
A-well-a birds, birds, b-birds's the word
A-well-a don't you know about the birds?
Well, everybody knows that the birds is the word!
A-well-a birds, birds, b-birds's the word
A-well-a...
<pattern1> and replacing all occurences of <pattern2> with <replacement_string>:sed -n '/^<pattern1>/s/<pattern2>/<replacement_string>/pg' file
user@debian ~ % sed -n '/^A/s/bird/birds/pg' file
A-well-a everybody's heard about the birds
A-well-a birds, birds, birds, the birds is the word
A-well-a birds, birds, birds, well the birds is the word
A-well-a birds, birds, birds, b-birds's the word
A-well-a birds, birds, birds, well the birds is the word
A-well-a birds, birds, b-birds's the word
A-well-a birds, birds, birds, b-birds's the word
A-well-a birds, birds, birds, well the birds is the word
A-well-a birds, birds, b-birds's the word
A-well-a don't you know about the birds?
A-well-a birds, birds, b-birds's the word
<pattern1> and replacing all occurences of <pattern2> with <replacement_string>:sed -n '/<pattern1>$/s/<pattern2>/<replacement_string>/pg' file
user@debian ~ % sed -n '/word$/s/bird/birds/pg' file
B-b-b-birds, birds, birds, b-birds's the word
A-well-a birds, birds, birds, the birds is the word
A-well-a birds, birds, birds, well the birds is the word
A-well-a birds, birds, birds, b-birds's the word
A-well-a birds, birds, birds, well the birds is the word
A-well-a birds, birds, b-birds's the word
A-well-a birds, birds, birds, b-birds's the word
A-well-a birds, birds, birds, well the birds is the word
A-well-a birds, birds, b-birds's the word
A-well-a birds, birds, b-birds's the word
sed is a powerfull command that enables you make things (remove lines, string substitution, string filtering, etc).
I could give you a list of uses with args but internet is filled of that. Searching sed usage by examples bring me a lot of results, the cute one: http://www.thegeekstuff.com/2009/10/unix-sed-tutorial-advanced-sed-substitution-examples/
