技术分享 | 独立思考

0%

linux常用命令备忘--sed

发表于 更新于 分类于 ，
本文字数： 10k 阅读时长 ≈ 9 分钟

Understand sed Linux Command

The sed command is a non-interactive text editor. Sed Linux command edits data based on the rules you provide, you can use it like this:

1
sed options file

You are not limited to use sed to manipulate files, you apply it to the STDIN directly like this:

1
2
echo "Welcome to peter page" | sed 's/page/website/'
Welcome to peter website

The s command replaces the first text with the second text pattern. In this case, the string “website” was replaced with the word “page”, so the result will be as shown.

The above example was a very basic example to demonstrate the tool. We can use sed Linux command to manipulate files as well.

This is our file:

1

1
2
3
sed 's/test/another test/' ./test.txt

this is a another test

The results are printed to the screen instantaneously, you don’t have to wait for processing the file to the end.

If your file is huge enough, you will see the result before the processing is finished.

Sed Linux command doesn’t update your data. It only sends the changed text to STDOUT. The file still untouched.

Using Multiple sed Linux Commands in The Command Line

To run multiple sed commands, you can use the -e option like this

1
2
3
sed -e 's/this/that/; s/test/another test/' ./test.txt

that is a another test

Reading Commands From a File

You can save your sed commands in a file and use them by specifying the file using -f option.

1
2
3
4
5
cat mycommands.txt

s/this/that/

s/test/another test/
1
2
3
sed -f ./mycommands.txt ./test.txt

that is a another test

Substituting Flags

Look at the following example carefully:

1
2
3
4
5
6
7
8
9
10
11
12
cat test.txt

this is a test
this is another test,
this is third test,more and more test

===============================================
sed 's/test/another test/' ./test.txt

this is a another test
this is another another test,
this is third another test,more and more test

The above result shows the first occurrence in each line is only replaced. To substitute all occurrences of a pattern, use one of the following substitution flags.

The flags are written like this:

1
s/pattern/replacement/flags

There are four types of substitutions:

  • g, replace all occurrences.
  • A number, the occurrence number for the new text that you want to substitute.
  • p, print the original content.
  • w file: means write the results to a file.

You can limit your replacement by specifying the occurrence number that should be replaced like this:

1
2
3
4
5
sed 's/test/another test/2' test.txt

this is a test
this is another test,
this is third test,more and more another test

As you can see, only the second occurrence on each line was replaced.

The g flag means global, which means a global replacement for all occurrences:

1
2
3
4
5
sed 's/test/another test/g' test.txt

this is a another test
this is another another test,
this is third another test,more and more another test

The p flag prints each line contains a pattern match, you can use the -n option to print the modified lines only.

1
2
3
4
5
sed -n 's/test/another test/p' test.txt

this is a another test
this is another another test,
this is third another test,more and more test

The w flag saves the output to a specified file:

1
2
3
4
5
6
7
8
9
10
sed 's/test/another test/w output.txt' test.txt

this is a another test
this is another another test,
this is third another test,more and more test
==================
cat output.txt
this is a another test
this is another another test,
this is third another test,more and more test

The output is printed on the screen, but the matching lines are saved to the output file.

Replace Characters

Suppose that you want to search for bash shell and replace it with csh shell in the /etc/passwd file using sed, well, you can do it easily:

1
sed 's/\/bin\/bash/\/bin\/csh/' /etc/passwd

Oh!! that looks terrible.

Luckily, there is another way to achieve that. You can use the exclamation mark (!) as string delimiter like this:

1
sed 's!/bin/bash!/bin/csh!' /etc/passwd

Now it’s easier to read.

Limiting sed

Sed command processes your entire file. However, you can limit the sed command to process specific lines, there are two ways:

  • A range of lines.
  • A pattern that matches a specific line.

You can type one number to limit it to a specific line:

1
2
3
4
5
6
cat test.txt

this is first test
this is second test,
this is third test,more and more test
this is fourth test,i will add one more test!
1
2
3
4
5
6
sed '2s/test/another test/' test.txt

this is first test
this is second another test,
this is third test,more and more test
this is fourth test,i will add one more test!

Only line two is modified.

What about using a range of lines:

1
2
3
4
5
6
sed '2,3s/test/another test/' test.txt

this is first test
this is second another test,
this is third another test,more and more test
this is fourth test,i will add one more test!

Also, we can start from a line to the end of the file:

1
2
3
4
5
6
sed '2,$s/test/another test/' test.txt

this is first test
this is second another test,
this is third another test,more and more test
this is fourth another test,i will add one more test!

Or you can use a pattern like this:

1
2
3
4
5
6
sed '/more/s/test/test2/' test.txt

this is first test
this is second test,
this is third test2,more and more test
this is fourth test2,i will add one more test!

Awesome!!

You can use regular expressions to write this pattern to be more generic and useful.

Delete Lines

To delete lines, the delete (d) flag is your friend.

The delete flag deletes the text from the stream, not the original file.

1
2
3
4
5
6
7
cat test.txt

this is first test
this is second test,
this is third test,more and more test
this is fourth test,i will add one more test!
this is fifth test,it's not the end.
1
2
3
4
5
6
sed '2d' test.txt

this is first test
this is third test,more and more test
this is fourth test,i will add one more test!
this is fifth test,it's not the end.
1
2
3
4
5
6
cat test.txt
this is first test
this is second test,
this is third test,more and more test
this is fourth test,i will add one more test!
this is fifth test,it's not the end.

Here we delete the second line only from myfile.

What about deleting a range of lines?

1
2
3
4
sed '2,3d' test.txt
this is first test
this is fourth test,i will add one more test!
this is fifth test,it's not the end.

Here we delete a range of lines, the second and the third.

Another type of ranges:

1
2
sed '2,$d' test.txt
this is first test

Here we delete from the third line to the end of the file.

All these examples never modify your original file.

1
2
3
4
5
sed '/fifth/d' test.txt
this is first test
this is second test,
this is third test,more and more test
this is fourth test,i will add one more test!

Here we use a pattern to delete the line if matched on the first line.

If you need to delete a range of lines, you can use two text patterns like this:

1
2
3
4
sed '/second/,/fourth/d' test.txt

this is first test
this is fifth test,it's not the end.

From the second to the fourth line are deleted.

Insert and Append Text

You can insert or append text lines using the following flags:

  • The (i) flag.
  • The (a) flag.
1
2
3
4
5
echo "Another test" | sed 'i\First test '

echo "Another test" | sed 'i\First test '
First test
Another test

这个命令在mac 下面执行会出现:sed: 1: “i\First test “: extra characters after \ at the end of i command 错误，暂时留坑，晚点加上unix和Linux -i 的区别

Here the text is added before the specified line.

1
2
3
4
echo "Another test" | sed 'a\First test '

Another test
First test

同样这个命令在mac 下面执行会出现:sed: 1: “a\First test “: extra characters after \ at the end of a command 错误，暂时留坑

Here the text is added after the specified line.

Well, what about adding text in the middle?

Easy, look at the following example:

1
2
3
4
5
6
7
8
sed '2i\This is the inserted line.' test.txt

this is first test
This is the inserted line.
this is second test,
this is third test,more and more test
this is fourth test,i will add one more test!
this is fifth test,it's not the end.

同样这个命令在mac 下面执行会出现错误，暂时留坑

And the appending works the same way, but look at the position of the appended text:

1
2
3
4
5
6
7
8
sed '2a\This is the appended line.' test.txt

this is first test
this is second test,
This is the appended line.
this is third test,more and more test
this is fourth test,i will add one more test!
this is fifth test,it's not the end.

The same flags are used but with a location of insertion or appending.

Modifying Lines

To modify a specific line, you can use the (c) flag like this:

1
2
3
4
5
6
sed '3c\This is a modified line.' test.txt
this is first test
this is second test,
This is a modified line.
this is fourth test,i will add one more test!
this is fifth test,it's not the end.

同样这个命令在mac 下面执行会出现错误，暂时留坑

You can use a regular expression pattern and all lines match that pattern will be modified.

1
2
3
4
5
6
sed '/this is/c Line updated.' test.txt
Line updated.
Line updated.
Line updated.
Line updated.
Line updated.

Transform Characters

The transform flag (y) works on characters like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cat test.txt

this is first test.1
this is second test,.2
this is third test,more and more test.3
this is fourth test,i will add one more test!.4
this is fifth test,it is not the end..5
====
sed 'y/123/567/' test.txt

this is first test.5
this is second test,.6
this is third test,more and more test.7
this is fourth test,i will add one more test!.4
this is fifth test,it is not the end..5

The transformation is applied to all data and cannot be limited to a specific occurrence.

You can print line number using the (=) sign like this:

1
2
3
4
5
6
7
8
9
10
11
12
sed '=' test.txt

1
this is first test.1
2
this is second test,.2
3
this is third test,more and more test.3
4
this is fourth test,i will add one more test!.4
5
this is fifth test,it is not the end..5

However, by using -n combined with the equal sign, the sed command displays the line number that contains matching.

1
2
3
4
5
6
7
sed -n '/test/=' test.txt

1
2
3
4
5

Read Data From a File

You can use the (r) flag to read data from a file.

You can define a line number or a text pattern for the text that you want to read.

1
2
3
4
cat test2.txt

this is fourth6 test,i will add one more test!.4
this is fifth7 test,it is not the end..5
1
2
3
4
5
6
7
8
9
sed '3r test2.txt' test.txt

this is first test.1 new life
this is second test,.2 old life
this is third test,more and more test.3
this is fourth6 test,i will add one more test!.4
this is fifth7 test,it is not the end..5
this is fourth test,i will add one more test!.4
this is fifth test,it is not the end..5

The content is just inserted after the third line as expected.

And this is using a text pattern:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
sed '/test/r test2.txt' test.txt

this is first test.1 new life
this is fourth6 test,i will add one more test!.4
this is fifth7 test,it is not the end..5
this is second test,.2 old life
this is fourth6 test,i will add one more test!.4
this is fifth7 test,it is not the end..5
this is third test,more and more test.3
this is fourth6 test,i will add one more test!.4
this is fifth7 test,it is not the end..5
this is fourth test,i will add one more test!.4
this is fourth6 test,i will add one more test!.4
this is fifth7 test,it is not the end..5
this is fifth test,it is not the end..5
this is fourth6 test,i will add one more test!.4
this is fifth7 test,it is not the end..5

Cool right?

Useful Examples

We have a file that contains text with a placeholder and we have another file that contains the data that will be filled in that placeholder.

We will use the (r) and (d) flags to do the job.

The word DATA in that file is a placeholder for a real content which is stored in another file called data.

We will replace it with the actual content:

1
2
3
4
cat newfile

The first part of data.
The second part of data.
1
2
3
4
5
6
7
cat test.txt
this is first test.1 new life
this is second test,.2 old life
this is third test,more and more test.3
DATA
this is fourth test,i will add one more test!.4
this is fifth test,it is not the end..5
1
2
3
4
5
6
7
8
9
10
11
sed '/DATA/ {
r newfile
d}' test.txt

this is first test.1 new life
this is second test,.2 old life
this is third test,more and more test.3
The first part of data.
The second part of data.
this is fourth test,i will add one more test!.4
this is fifth test,it is not the end..5

Awesome!! as you can see, the placeholder location is filled with the data from the other file.

This is just a very small intro about sed command. Actually, sed Linux command is another world by itself.

The only limitation is your imagination.

I hope you enjoy what’ve introduced today about the string manipulation using sed Linux command.

Thank you.

31+ Examples For Sed Linux Command In Text Manipulation