AWK - gawk in real-time pipelines

Author

gawk in real-time pipelines

ggrothendieck@gmail.com

2005-08-03, 9:59 pm

I am using the gawk 3.1.3 binary from sourceforge on
Windows XP:
http://gnuwin32.sourceforge.net/packages/gawk.htm

If I run a simple passthru filter with gawk then it echos
each line as expected (I entered the first abc line
and the first def line and the other 2 lines are output):

C:\>gawk 1
abc
abc
def
def
^Z

however, if I pipe two such filters together then I
don't get real-time behavior (i.e. I would have
expected the same output as above):

C:\>gawk 1 | gawk 1
abc
def
^Z
abc
def

Note that this is NOT a limitation of Windows XP since
if I write the passthru filter in vbscript then I _do_
get the desired real-time behavior:

C:\>type passthru.vbs
Do While Not WScript.StdIn.AtEndOfStream
s=WScript.StdIn.ReadLine
WScript.StdOut.WriteLine s
Loop

C:\>cscript/nologo passthru.vbs | cscript/nologo passthru.vbs
abc
abc
def
def
^Z

Since its not an intrinsic limitation of Windows XP its somehow
related to how gawk was written or built. Does gawk act like
this on other platforms too? Is there a build of gawk on
Windows that gives the sort of real-time behavior
that the corresponding vbscript program, above, exhibits?

ggrothendieck@gmail.com

2005-08-03, 9:59 pm

Reading the manual some more I discovered fflush. That solves
the problem:

C:\>gawk {print;fflush()} | gawk {print;fflush()}
abc
abc
def
def

Bob Harris

2005-08-04, 4:01 am

In article <1123120384.845863.309500@o13g2000cwo.googlegroups.com>,
ggrothendieck@gmail.com wrote:

> I am using the gawk 3.1.3 binary from sourceforge on
> Windows XP:
> http://gnuwin32.sourceforge.net/packages/gawk.htm
>
> If I run a simple passthru filter with gawk then it echos
> each line as expected (I entered the first abc line
> and the first def line and the other 2 lines are output):
>
> C:\>gawk 1
> abc
> abc
> def
> def
> ^Z
>
> however, if I pipe two such filters together then I
> don't get real-time behavior (i.e. I would have
> expected the same output as above):
>
> C:\>gawk 1 | gawk 1
> abc
> def
> ^Z
> abc
> def
>
> Note that this is NOT a limitation of Windows XP since
> if I write the passthru filter in vbscript then I _do_
> get the desired real-time behavior:
>
> C:\>type passthru.vbs
> Do While Not WScript.StdIn.AtEndOfStream
> s=WScript.StdIn.ReadLine
> WScript.StdOut.WriteLine s
> Loop
>
> C:\>cscript/nologo passthru.vbs | cscript/nologo passthru.vbs
> abc
> abc
> def
> def
> ^Z
>
> Since its not an intrinsic limitation of Windows XP its somehow
> related to how gawk was written or built. Does gawk act like
> this on other platforms too? Is there a build of gawk on
> Windows that gives the sort of real-time behavior
> that the corresponding vbscript program, above, exhibits?

Typically on a UNIX platform, the C runtime library for I/O sets its
behavior based on the type of device.

The C Runtime I/O package looks at the stdout device. If it is a
terminal, then data is buffered until the next newline character is
seen, then the buffer is flushed to the stdout device.

However, if the stdout device is NOT a terminal, then the buffer is
allowed to fill up before it is flushed to the output device. The size
of the buffer may be as small as 512 bytes, 1K, 4K, or even 8K depending
on the host operating system implementation. This is done to improve
I/O efficiency.

A program may call fflush() to force the buffer to be flushed early. A
program may call setbuf() to modify the rules for buffered I/O.

Also I/O written to stderr is different. Anything written to stderr is
always written to the output device right away. It is unbuffered. It
does not even wait for a newline.

The behavior of the C runtime library is dictated by the X/Open and
POSIX standards. Most of the information on this can be found in the
setbuf(3) man page. On-line copies of this spec or the setbuf(3) man
page can be found via Google.

Please NOTE that visual basic is NOT C and most likely does NOT use the
C runtime library. Therefore it would have totally different rules for
dealing with I/O.

Bob Harris

Patrick TJ McPhee

2005-08-04, 4:01 am

In article <1123120384.845863.309500@o13g2000cwo.googlegroups.com>,
<ggrothendieck@gmail.com> wrote:

% however, if I pipe two such filters together then I
% don't get real-time behavior (i.e. I would have
% expected the same output as above):
%
% C:\>gawk 1 | gawk 1
% abc
% def
% ^Z
% abc
% def

gawk buffers output when writing to a pipe or file. You can flush
the buffers using fflush().

I'm not sure whether there's a way to prevent it from buffering.
--

Patrick TJ McPhee
North York Canada
ptjm@interlog.com

Sponsored Links