2

This question helped me understand how to combine stderr and stdout:

How to redirect stderr to a file

with this command: 

gunzip -vc /opt/minecraft/wonders/logs/20* 2>&1

But how may one insert a line break between the two so that the output can appear on separate lines instead of clumped together?

The stderr comes first, which is always 1 line, something like this:

/opt/minecraft/wonders/logs/2017-08-28-2.log.gz:

Whereas the stdout is usually (but not always) multiple lines, like:

06:17:05: Starting minecraft server version 1.10.2
06:18:21: Loading properties
06:18:21: Default game type: SURVIVAL
06:18:21: Generating keypair
06:18:21: Starting Minecraft server on *:25565
06:18:22: Using default channel type

But because there is no newline between them the first line from each file to be unzipped is always something like this:

/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:    06:17:05: Starting minecraft server version 1.10.2

I need these two parts on different lines so I can filter them with grep. But I also want the corresponding stdout directly following its stderr (its filename) so I know which file it comes from. Like this:

/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:    
06:17:05: Starting minecraft server version 1.10.2
06:18:21: Loading properties
06:18:21: Default game type: SURVIVAL
06:18:21: Generating keypair
06:18:21: Starting Minecraft server on *:25565
06:18:22: Using default channel type

This is the command I am piping into using with grep: 

egrep -iv "^\[.*(stopping|starting|saving|keep|spawn|joined|lost|left|: (<.*>|\/)|\/(help|forge))

Its purpose is to remove certain lines from the log files, leaving the pertinent ones.

Jesse Yishaeyn
  • 151

3 Answers3

3

Here is one simple way to do this, assume this command:

ls -d Documents/ NotExist/ /dev/null

the output is:

ls: cannot access 'NotExist/': No such file or directory
/dev/null  Documents/

the first line is stderr and the second is stdout, if I pipe the output to xargs I can print a new line after stderr and then print the stdout:

ls -d Documents/ NotExist/ /dev/null | xargs echo -e '\n'

which gives me:

ls: cannot access 'NotExist/': No such file or directory

 /dev/null Documents/
Ravexina
  • 57,256
2

It's possible with duplicating file descriptors. Long story, short: pipe stderr, where it can be processed, and send stdin to the controlling terminal ( short hand for that is /dev/tty ) so that it still shows on the screen:

$ stat /etc/passwd noexist 2>&1 > /dev/tty | while IFS= read -r line; do echo "---"; echo "$line"; echo "---"; done
  File: /etc/passwd
  Size: 2380        Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d  Inode: 937653      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-08-20 13:05:01.171247259 +0800
Modify: 2018-07-24 04:51:36.799057737 +0800
Change: 2018-07-24 04:51:36.867020035 +0800
 Birth: -
---
stat: cannot stat 'noexist': No such file or directory
---

Here I'm separating stderr with --- line, but you get the idea.

Another way, if you want to visually separate them is to colorize the output. Based on Balazs Pozar's answer, you could do:

command 2> >(while read line; do echo -e "\e[01;31m$line\e[0m" >&2; done)

enter image description here

To address the edit on the question, if you need to filter both stdout and stderr with grep (for example, if you're looking for a pattern in stdout, but another pattern in stderr ), you use multiple process substitutions >():

gunzip -vc /opt/minecraft/wonders/logs/20* > >(grep 'stdin pattern' ) 2> >(grep 'stderrpattern' )

Addressing the comments, if you only need to separate the filename /opt/minecraft/wonders/logs/2017-08-28-2.log.gz: from further content, you could use sed:

gunzip -vc /opt/minecraft/wonders/logs/20* | 
sed -r 's/^(\/opt\/minecraft\/wonders\/logs\/2018-08-18-2.log.gz:) *(.*)$/\1\n\2/'

Test:

$ echo /opt/minecraft/wonders/logs/2018-08-18-2.log.gz:    06:17:05: Starting minecraft server version 1.10.2 | sed -r 's/^(\/opt\/minecraft\/wonders\/logs\/2018-08-18-2.log.gz:) *(.*)$/\1\n\2/'
/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:
06:17:05: Starting minecraft server version 1.10.2
Sergiy Kolodyazhnyy
  • 107,582
1

What's really happening I think is that stdout is getting mixed into stderr, because the former is line-buffered and the latter unbuffered by default. To illustrate:

$ gunzip -vc ../foo.txt.gz
../foo.txt.gz:  some text here
more text here
 26.7%

(note that -v is causing two pieces of information to be emitted on stderr: the filename, and the % compression - with standard output falling in between).

If you line-buffer stderr as well, the streams will appear on separate lines:

$ stdbuf -eL gunzip -vc ../foo.txt.gz
some text here
more text here
../foo.txt.gz:   26.7%

however not necessarily in the order that you require - you might try using the sponge utility to soak up all of stdout before outputing it, so that the stderr output (filename and %) appears in the terminal first:

gunzip -vc ../foo.txt.gz | sponge
../foo.txt.gz:   26.7%
some text here
more text here
steeldriver
  • 142,475