AWK - rules on stdout and stderr

Author

rules on stdout and stderr

Laurent Deniau

2004-12-02, 3:56 pm

I would like to filter both stdout and stderr of a program and then
print the filtered messages on their respective source. So I was
wondering if there is a way to do something like:

handle == stdout && /regex/ { rule for stdout }
handle == stderr && /regex/ { rule for stderr }

and then

prog | filter.awk

would process both channels separately

-- stdout --> -- stdout -->
prog filter.awk shell
-- stderr --> -- stderr -->

Any ideas?

a+, ld.

Bob Harris

2004-12-02, 8:55 pm

In article <condsv$n6b$1@sunnews.cern.ch>,
Laurent Deniau <Laurent.Deniau@cern.ch> wrote:

> I would like to filter both stdout and stderr of a program and then
> print the filtered messages on their respective source. So I was
> wondering if there is a way to do something like:
>
> handle == stdout && /regex/ { rule for stdout }
> handle == stderr && /regex/ { rule for stderr }
>
> and then
>
> prog | filter.awk
>
> would process both channels separately
>
> -- stdout --> -- stdout -->
> prog filter.awk shell
> -- stderr --> -- stderr -->
>
> Any ideas?
>
> a+, ld.

Not as far as I know.

You can get both stdout and stderr going out stdout, but then you do not
know which lines are errors and which are stdout.

prog 2>&1 | awk '...'

But if you want to know which is which, you might try

prog 2>tmp.stderr | awk '
/regex/ { rule for stdout }
END {
while( getline <tmp.stderr ) {
if ( $0 ~ /regex/ ) {
rule for stderr
}
}
}
'

A bit ugly, and it does not intermix stderr with stdout so you don't
know which happened first or second.

Then you could also do

prog 2>tmp.stderr >tmp.stdout
awk '
FNR == NR && /regex/ { rule for stdout }
FNR != NR && /regex/ { rule for stderr }
' tmp.stdout tmp.stderr

Again, you do not have the text intermixed so you do not know the
relationship between the lines, but you can process stderr and stdout
from the same program.

Bob Harris

Kenny McCormack

2004-12-02, 8:55 pm

In article <condsv$n6b$1@sunnews.cern.ch>,
Laurent Deniau <Laurent.Deniau@cern.ch> wrote:
>I would like to filter both stdout and stderr of a program and then
>print the filtered messages on their respective source. So I was
>wondering if there is a way to do something like:
>
>handle == stdout && /regex/ { rule for stdout }
>handle == stderr && /regex/ { rule for stderr }
>
>and then
>
>prog | filter.awk

There's no way to do it with this syntax, since, by definition, this only
sends stdout to the RHS of the pipe.

In any case, this is definitely OT in comp.lang.awk, since it is a shell
problem, not an AWK one.

>would process both channels separately
>
> -- stdout --> -- stdout -->
>prog filter.awk shell
> -- stderr --> -- stderr -->
>
>Any ideas?
>
>a+, ld.

You could probably do it with ksh, using co-processes (a nifty feature
which, although seldom used, really comes in handy when it is needed.
I never understood why other Bourne-derived shells don't have it)

Failing that, you could write a small C program to do it (call pipe() twice
and redirect accordingly). This could also be done in one of the scripting
languages (not in classical AWK, alas) that grant access to low-level
functions, such as Perl.

John DuBois

2004-12-02, 8:55 pm

In article <conur8$3hk$1@yin.interaccess.com>,
Kenny McCormack <gazelle@interaccess.com> wrote:
>In article <condsv$n6b$1@sunnews.cern.ch>,
>Laurent Deniau <Laurent.Deniau@cern.ch> wrote:
>...
>You could probably do it with ksh, using co-processes

A problem with this approach is that it is subject to stalling.
You can get two pipes to connect the program to awk, but then which does it
read from? There's no select() to let it read from whichever one data is
available on. If the program writes more data than will fit in a pipe to
one pipe while your awk program is waiting to read from the other, you lose.
You would need to insert some arbitrary-amount buffering command between prog
and awk, but that's getting a little silly.

Just about as silly:

{ { prog | sed 's/^/1/' >&9;} 2>&1 | sed 's/^/2/';} 9>&1 | awk '
{
fd = substr($0,1,1)
$0 = substr($0,2)
}
fd == 1 {
print "prog stdout: " $0
}
fd == 2 {
print "prog stderr: " $0
}
'

Example output, where prog is: { echo foo; echo bar >&2; }

prog stdout: foo
prog stderr: bar

But the OP is probably better off just filtering the output with two separate
instances of awk:

{ { prog | awk '
{
print "prog stdout: " $0
}' >&9;} 2>&1 | awk '
{
print "prog stderr: " $0
}';} 9>&1

John
--
John DuBois spcecdt@armory.com KC6QKZ/AE http://www.armory.com/~spcecdt/

Kenny McCormack

2004-12-02, 8:55 pm

In article <41afa95c$0$85066$c0de7616@dsl.net>,
John DuBois <spcecdt@armory.com> wrote:
>In article <conur8$3hk$1@yin.interaccess.com>,
>Kenny McCormack <gazelle@interaccess.com> wrote:
>
>A problem with this approach is that it is subject to stalling. You can
>get two pipes to connect the program to awk, but then which does it read
>from? There's no select() to let it read from whichever one data is
>available on. If the program writes more data than will fit in a pipe to
>one pipe while your awk program is waiting to read from the other, you
>lose. You would need to insert some arbitrary-amount buffering command
>between prog and awk, but that's getting a little silly.

That's what I meant. I didn't want to go into too much detail, but the
idea is that you would run two AWK scripts. You would push stderr into one
of them via the co-process mechanism and you would push stdout into the
other one via the usual piping mechanism.

I haven't worked through all the details, nor do I care to, but that's the
basic idea. Anyway, it doesn't matter to me - let the OP work out the
details if he so chooses.

Stephane CHAZELAS

2004-12-03, 3:56 pm

2004-12-02, 20:37(+00), Kenny McCormack:
[...]
>
> You could probably do it with ksh, using co-processes (a nifty feature
> which, although seldom used, really comes in handy when it is needed.
> I never understood why other Bourne-derived shells don't have it)

You don't need co-processes,

{
{
cmd 3>&- 4>&- | awk 'stdout filter' >&3 2>&4 3>&- 4>&-
} 2>&1 | awk 'stderr filter' >&2 3>&- 4>&-
} 3>&1 4>&2

Note that zsh has co-process support (in a different way than
ksh). With any shell, you can achieve the same often less
awkwardly using named pipe.

What do you need co-processes so often for? Can't you use
process substitution instead (<(...), >(...), =(...)...)?

--
Stephane

Kenny McCormack

2004-12-03, 3:56 pm

In article <slrncr0f5d.4lb.stephane.chazelas@spam.is.invalid>,
Stephane CHAZELAS <this.address@is.invalid> wrote:
>2004-12-02, 20:37(+00), Kenny McCormack:
>[...]
>
>You don't need co-processes,

One does not *need* anything other than an assembler.
(Yes, I know there are variations on this. It could be said that all you
need are toggle switches...)

>{
> {
> cmd 3>&- 4>&- | awk 'stdout filter' >&3 2>&4 3>&- 4>&-
> } 2>&1 | awk 'stderr filter' >&2 3>&- 4>&-
>} 3>&1 4>&2

Wow. You must have stayed up all night working on that one.
Needless to say, I wouldn't exactly call that newbie-friendly.

>Note that zsh has co-process support (in a different way than
>ksh). With any shell, you can achieve the same often less
>awkwardly using named pipe.

Or with something like the above (glop).

>What do you need co-processes so often for? Can't you use
>process substitution instead (<(...), >(...), =(...)...)?

I said that I don't use them that often.

Anyway, this is all OT.

Stephane CHAZELAS

2004-12-03, 3:56 pm

2004-12-03, 13:16(+00), Kenny McCormack:
[...]
>
> Wow. You must have stayed up all night working on that one.
> Needless to say, I wouldn't exactly call that newbie-friendly.

Well, you may remove the >&- as they are there mostly to be
pedantic.

You may find it more legible (even if less clean) this way:

{
{
cmd | awk 'stdout filter' >&3
} 2>&1 | awk 'stderr filter' >&2
} 3>&1

or (less portable):

cmd 2> >(awk 'stderr filter') | awk 'stdout filter'

(beware of SIGPIPEs).

[...]
> Anyway, this is all OT.

on the awk group yes, not on the shell one.

--
Stephane

Kenny McCormack

2004-12-03, 3:56 pm

In article <slrncr0rrs.525.stephane.chazelas@spam.is.invalid>,
Stephane CHAZELAS <this.address@is.invalid> wrote:
>2004-12-03, 13:16(+00), Kenny McCormack:
>[...]
>
>Well, you may remove the >&- as they are there mostly to be
>pedantic.
>
>You may find it more legible (even if less clean) this way:
>
>{
> {
> cmd | awk 'stdout filter' >&3
> } 2>&1 | awk 'stderr filter' >&2
>} 3>&1
>
>or (less portable):
>
>cmd 2> >(awk 'stderr filter') | awk 'stdout filter'

Ah, yes. That's the one. That's the sensible way to do it.
(And, no, I'm not concerned about brain-dead shells that don't support
process substitution)

>
>on the awk group yes, not on the shell one.

True - but see my other post (in c.l.a) about "spreading mirth".

</OT>
I get a real kick out of the c.l.c wankers - when a newbie posts some dumb
question to, like 10 groups including c.l.c., some one or more of the
c.l.c.'ers will post, to all 10 groups, that the post is O/T (!)

Even though it (the post) is probably on topic for most, if not all, of the
other 9 groups...

heh heh - amuses me, anyway..

Stephane CHAZELAS

2004-12-03, 3:56 pm

2004-12-03, 14:39(+00), Kenny McCormack:
[...]
>
> Ah, yes. That's the one. That's the sensible way to do it.
> (And, no, I'm not concerned about brain-dead shells that don't support
> process substitution)
[...]

Well, actually, it was not correct.

It should have been:

cmd 2> >(awk 'stderr filter' >&2) | awk 'stdout filter'

The problem is that it's not only about shells, the operating
system also has to support the /dev/fd/<n> mechanism or named
pipes (ksh wouldn't use named pipes, though).

--
Stephane

Rufus V. Smith

2004-12-03, 3:56 pm

"Kenny McCormack" <gazelle@yin.interaccess.com> wrote in message
news:coppdc$kdh$1@yin.interaccess.com...
>
> Wow. You must have stayed up all night working on that one.
> Needless to say, I wouldn't exactly call that newbie-friendly.
>
>
> I said that I don't use them that often.
>
> Anyway, this is all OT.
>

I am a lurker on this one, and I find it very interesting and something I
might
actually use.

Where else could I have gotten this information?

If nowhere else, I feel it deserves an "OT but OK" permission slip.

Rufus

Dana French

2004-12-03, 3:56 pm

Stephane CHAZELAS <this.address@is.invalid> wrote in message news:<slrncr0f5d.4lb.stephane.chazelas@spam.is.invalid>...
> 2004-12-02, 20:37(+00), Kenny McCormack:
SNIP
>
> You don't need co-processes,
>
> {
> {
> cmd 3>&- 4>&- | awk 'stdout filter' >&3 2>&4 3>&- 4>&-
> } 2>&1 | awk 'stderr filter' >&2 3>&- 4>&-
> } 3>&1 4>&2
>

EXCELLENT "one"-liner, adding this to the King/Queen of the one-liners
page and will probably use this as a teaching tool in discussions of
file descriptors:

http://www.mtxia.com/fancyIndex/Too.../oneliners.html

--
Dana French

Sponsored Links