sed – the missing manual

Table of contents

IntroductionBACK TO TOC

sed is an exceptionally powerful and often overlooked tool. Its most common use is in scripts, where we want to replace part of the string matching certain pattern. While this is the most common use, it’s far from being its only use.

In this manual I’ll try to describe sed‘s most useful features.

Why sed rules?BACK TO TOC

As with most of the Unix command line utilities, sed reads data from the standard input and writes the result into the standard output. One thing that I like about sed is that it doesn’t have many command line switches. Actually, I often use only one of them. We’ll talk about it a little later in this manual.

sed is special. Special because it turns some common and quiet complex tasks into simple. For instance, what if you want to delete last line in a file? I can think of at least four ways to achieve this, but sed is the only way to do this with one command. And how about deleting lines 5 through 10? You can write bash script that does the job of course. Yet same effect can be achieved with single sed command.

And of course its most common use. That is to find and replace a pattern with some other pattern. It is absolutely irreplaceable. The later used so widely, so I guess I have no choice but to start with it.

sed simple case – search and replaceBACK TO TOC

Let’s say we have a file with numbers. Each number occupies a single line in the file. I’ll use this file as a sample input file in this article. Obviously, you’ll have your own input information and you will do your own stuff with it, but for the sake of the demonstration, let’s think our input file contains plain numbers. By the way, you can produce the file with

$ seq 20 > file

command. This will print numbers from 1 to 20, one in a line.

sed accepts commands as command line last argument. It is common practice to place sed‘s commands into single quotes, to tell the shell it should not try to interpret and manipulate sed‘s command.

sed commands designated with single characters, followed by optional argument. Let’s see a sample sed command. Since we’re talking about search and replace, this is the command we’ll see.

$ cat file | sed 's/<search pattern>/<replace pattern>/[optional switches]'

The search and replace command is in single quotes. Both patterns are regular expressions, meaning that . will be interpreted as any character, etc. You may omit optional switches at the end. Let’s see some examples of sed‘s search and replace in action.

$ cat file | sed 's/1/x/'

This is perhaps the simplest case of search and replace. This particular command replaces all instances of 1 digit with x character. You’re right. Not very handy. Let’s try something more complicated and perhaps meaningful.

$ cat file | sed 's/1/x/'

This command replaces every instance of 1 character with x, affecting all numbers between 10 and 19 and 1. Well, this is not exactly true. This command changes 11 into x1, instead of changing it into xx. This is wrong, isn’t it?

The truth is that unless explicitly told to replace all occurrences of the search pattern with replace pattern, sed will replace only first occurance. I.e. 11 will turn into x1 instead of xx. This is where optional switches become handy. You can use g switch to tell sed to replace all occurrences of the search pattern in each line. This is how:

$ cat file | sed 's/1/x/g'

Unlike previous command, this one will cause 11 to be replaced with xx.

Now let’s see few search and replace commands that envolve regular expressions. sed supports common regular expression syntax out of the box.

Search and replace with regular expressionsBACK TO TOC

There are plenty of resources describing regular expressions on the web and I assume that you possess some degree of knowledge in regular expression syntax. Yet, here is a wonderful resource on regular expressions that I myself often use.

$ cat file | sed 's/^1./-/'

This command replaces every two digit number starting with 1, with dash character. I.e. it will replace numbers 10-19 with ‘-‘. The ‘^’ character that you see at the front of the search pattern indicates that following characters should be first characters in the line of text.

Here’s another, more complex example.

$ cat file | sed 's/^\(.\)$/1\1/'

This example is more interesting because it demonstrates one of the more powerful feature of regular expressions and that is grouping. The sample sed command above places 1 before all one character long numbers. I.e numbers 1-9 will turn to 11-19.

sed‘s other halfBACK TO TOC

sed accepts several different commands. We’ve seen only one of them, but the rest are no less useful than search and replace.

In general, sed commands consist of two parts. First part specifies what line of input to manipulate. Second part of the command specifies the actual manipulation. The command may be search and replace that we’ve already seen. Or it may be simply delete one of the input lines or print it twice. Oh and you can run several sed commands one after another for each input line.

It is true that search and replace is perhaps the most common sed command, but other commands combined with addressing features of sed, are no less useful.

Generic sed commandBACK TO TOC

Generic sed command looks like this:

sed '[address] <command>[;[address] <command>]

Note that I used square brackets for address. That is because address is optional (common designation of optional argument) – when we did search and replace we didn’t specify any address.

You can specify several sed commands one after another, delimited with semicolon. sed will apply commands on every line of input, command by command, line by line.

The additional two sed commands that I would like to explain here are ‘d’ and ‘p’ – the delete and print commands. The truth is that I rarely use any other sed command. Combined with sed‘s powerful addressing, these two and perhaps search and replace are the commands that you will use the most.

sed addressesBACK TO TOC

The idea behind addressing in sed is to let you specify what lines of input you want to alter. This is a unique feature because no other command line utility allows you to modify very specific line of text in the input with such ease. Throughout this section of this article, I’ll use the ‘d’ command to demonstrate use of addresses. This command deletes the line. So if we tell sed to modify only the 10th line and then do ‘d’ command it will delete the line from the output.

Simple addressing, by line numberBACK TO TOC

You can specify a number of line that we want to alter. As you remember, each sed command consists of address followed by a command, delimited by a space character. Let’s see few examples.

$ cat file | sed '10 d'

This command will delete 10th line in the input file, meaning that we will see all numbers but number 10.

In case you want to specify last line of the input file, you can use $ sign as address specification. Following example will print numbers 1 through 19.

$ cat file | sed '$ d'

Complex addressingBACK TO TOC

You can specify a range of lines. To do this, you specify first line of the range and the last one with comma in between.

$ cat file | sed '1,9 d'

Will print lines 10 through 20. Next example on the other hand, will print lines 1 to 9.

$ cat file | sed '10,$ d'

Now here’s something neat. You can also tell sed that address lines appear starting line X every Y lines. To do this, you specify X~Y as an address. Here’s an example.

$ cat file | sed '1~2 d'

This will print the even numbers out of 1-20 range. On the contrary, following command will print only the odd numbers.

$ cat file | sed '0~2 d'

Finally, we can specify addresses using regular expressions. You can do this by specifying regular expression in between two slash characters.

$ cat file | sed '/^1./ d'

This command will delete all two character lines starting with 1. I.e. it will print numbers 1-9 and 20. Here’s another regular expression example.

$ cat file | sed '// d'

Putting it all togetherBACK TO TOC

Last command that I would like to demonstrate you is ‘p’ command. As you could have guessed, it prints input line as is. There’s one caveat in using it. sed will print the input line (or what may have left of it) anyway, so when telling it to print the line it will do it twice. It is easy to overcome this problem is using -n command line switch – remember I mentioned a command line switch that I use from time to time, this is it :D

Let’s see few examples of the ‘p’ command in action.

$ cat file | sed 'p'

As expected, this command will simply print every line of text twice. Let’s see something more complex.

$ cat file | sed -n '1,5 p'

In this command I used the addressing feature of sed. It will print all numbers from 1 to 5. Finally, let’s try to put it all together.

$ cat file | sed -n '/^.$/ d; /^2./ d; /^1./ s/1/2/; p'

What you see here is three sed commands in a row. First one deletes all lines of text one character long. This will delete all numbers from 1 to 9. Next, it deletes all two digit numbers that start with 2. This will delete number 20. Finally, it searches for all two digit numbers starting from 1 and replaces 1 with 2. Eventually, we use ‘p’ command to print the result. Last ‘p’ command needed only because we used -n command line switch. Eventually it will print numbers 20 to 29.

Here’s another a slightly complex example, that does however exactly the same – prints numbers 20 through 29.

$ cat file | sed '1,9 d; $ d; s/1/2/g;'

This concludes this introductionary article on sed. Hope you found it interesting. If you have further questions, drop me an email or leave a comment here.

Did you know that you can receive periodical updates with the latest articles that I write right into your email box? Alternatively, you subscribe to the RSS feed!

Want to know how? Check out
Subscribe page

24 Comments

  1. rapid says:

    Hi, using cat file is useless use of cat, see
    http://partmaps.org/era/unix/award.html

    since its a sed forum i left this snippet
    assuming you have -i in your version of sed this should work
    — if you run it backup of course first :-)

    sed -i ‘s/cat file.|(.*)/1 file /’ sed-the-missing-manual.html

    ps: I found the reverse ssh blog helpful too. thanks.
    have a nice day

  2. rapid says:

    previous sed is wrong, doh :)

    sed -i ‘s/cat file.|(.*)/1 file /’ sed-the-missing-manual.html

  3. rapid says:

    the html tags were interperted last post’s
    one last try, sorry for mess

    sed -i ‘s/cat file.|(.*)()/1 file 2/’

  4. rapid says:

    ‘s/cat file.|(.*)()/1 file 2/’

  5. rapid says:

    argh, scrap it. . i’m sure you get the idea

  6. Alexander Sandler says:

    Yeah, I got the idea. Thanks for the useless use of… link – it’s refreshing :-) You are right. Cat is useless here. However I don’t understand what you need -i for. You can do ‘sed command < file' without cat and without -i.

  7. Teodora says:

    Hello

    Very interesting your sed tutorial. Newbie in sed, I would like to put you a question:

    Sometime, when I use sed to insert text into files (-i option) I’ve noticed that sed create empty files such as
    sed6OAPQ6, sedioICv3, sedOYkVW6, sedu1yKVY, etc. I believe that these files could be one of the two data buffers that sed use to do its job, but I am not sure.

    I don’t know how to fix the problem.

    thanks in advance

    rgds

    Teo

  8. rapid says:

    the sed -i is available on GNU sed and some other sed’s and does INPLACE editing without the need to redirect to a new file..

    sed ‘s/old/new/g’ newfile – (creates newfile)
    sed ‘s/old/new/g’ file > newfile – (does the same as above)

    the < to sed is not needed as sed can handle the file without a redirection

    sed -i ‘s/old/new/g’ file (sed operates DIRECTLY on the file)

    hopefully that made more sense

  9. samhill says:

    Despite reading several sources including yours, I am still unable to perform something that I -think- would be something simple with sed.

    I am using the ‘x~y’ addressing command to select every Yth line starting at line “x”. Each line is the name of a file. What I want to do is then surround the entire line with text in order to make a script file.

    For example, if the line selected is “d:\filename”, I would like to create output that looks like “copy d:\filename f:\newdirectory”.

    Seems pretty simple but I keep getting screwy results.

    Can you provide some examples of how to do this.

    Thank you.

  10. erman says:

    Hi,
    Do you know how to do this by using sed.

    “http://www.yahoo.com” change this to “http://www.google.com”

    thanks,
    erman

  11. peter says:

    The Unix ed command can also be used to delete specificied lines in files.

    http://codesnippets.joyent.com/posts/show/1608

  12. Chetan says:

    Thanks Alexander,
    I remember while at Georgetown University in 1975 in a Statistics Class during a semister final where a students’ calculator battery died, and the poor kid literally melted down before all of us that day. He failed the final because he had relied too much on the calculator and lost his confidence in doing it by hand (and using his own mind.)
    This is what has happened to Awk and Sed and Assembler and C as well.
    Most graduating “Computer Scientists” don’t have a clue what actually happens inside a computer. The high level languages are the calculators of yesterday, rendering our minds conditioned to a fantasy, leaving us dumb and frightened of doing it correctly or in the simple, easy way.

    The command line is looked upon as useless and the old way.
    I remember when the GUI began to be popular in the 70’s early 80’s. I immediately felt stripped of power and ability having no choice but to click here and there.

    Alexander, keep up the great work. You are a blessing in the computer world of ignorance.

    Why use a gun when a fly swatter is more than enough?
    Grab a hold of your computers power with the command line!
    Chetan

    • This is exactly the same as with everything else in the world. Prettier always preferred over functional. Yet when you strip off all buzzes and whistles, you need tools to do the real job. And you’re right. For the last 30-40 years these have always been same tools :-) Thanks a lot for a warm comment and please visit my web-site again! :-)

  13. Kashif says:

    hi,

    can you please let me know how can i modify a regular expression that already contains “/”. like i want to change

    http://localhost:port/DevDB with http://productionhost:port/ProdDB

    Thanks in advance.
    Kashif

  14. @Kashif
    Of course you can do that. Just make sure to use proper escape sequence(/) for ‘/’ character. Like this:

    $ echo 'http://localhost:port/' | sed 's/http://localhost:port//aaa/'
    aaa
    $ 
  15. Priyank says:

    Not sure how this will print 20 to 29
    cat file | sed ‘/.0/ d; 1,10 d; s/1/2/’
    as mentioned at the end of the article (“that does however exactly the same – prints numbers 20 through 29”). I am getting results 21 to 29.
    I guess the first sed command will delete 10,20.

  16. @Priyank
    You are right. This will print 21 to 29. My apologies. I’ll fix it.

  17. Jean says:

    @Alexander Sandler – One more fix – since your fix, it prints 22 twice – because of the /g.

    Thanks for this useful tutorial.

  18. dj says:

    Someone already mentioned this, but the `cat file |` is superfluous.

    See:
    –UNIX tips: Learn 10 good UNIX usage habits
    –http://www.ibm.com/developerworks/aix/library/au-badunixhabits.html
    –#10. Stop piping cats

    There’s an awesome sed cheat-sheet here:
    –http://www.catonmat.net/blog/sed-stream-editor-cheat-sheet

    While there, checkout the sed one-liners explained (1-3 parts):
    –Famous Sed One-Liners Explained, Part I
    –http://www.catonmat.net/blog/sed-one-liners-explained-part-one

  19. Log says:

    […] sed – the Missing Manual […]

  20. Dallas says:

    This article was fantastic. Thank you, Alex. I finally understand how to use sed!

Leave a Reply

Prove you are not a computer or die *