Tcl - Can't process fileevents until comand exits

Author

Can't process fileevents until comand exits

Ron Stohl

2005-07-25, 10:10 pm

I have the following sample program that runs a command in the
background, capturing stdout and stderr in independent file handlers.
The problem is that the "readstdout" procedure is not getting called
ntil the "tar" command exits, then it is called once for each line out
output. When I run the same thing using the "ls", the procedure is
called as each file is found.

Note that, when I run the same "tar" command at the command line, the
files appear as they are restored.

Can someone help me out?
Thanks in advance

----------- cut ---------------
load /usr/lpp/storix/tcl/lib/tclx8.4/libtclx8.4.so

set cmd "ls -Rl / | grep test"
# FILES IN ABOVE LS COMMAND ALWAYS APPEAR AT ONCE
set cmd "tar -xvf/tmp/test.tar /home/makeimage /home/bbox/junk/zcat"
# ABOVE COMMAND INCICATES ONE FILE AT START OF ARCHIVE, AND ONE
# AT END OF ARCHIVE - BOTH FILES LISTED WHEN COMMAND EXITS.

proc readstderr {input} {
if {![eof $input]} {
puts stderr [gets $input]
}
}

proc readstdout {input} {
global done
if {![eof $input]} {
puts stdout [gets $input]
} else {
set done 1
close $input
}
}

pipe rerr werr
fconfigure $rerr -blocking 0 -buffering line
fileevent $rerr readable [list readstderr $rerr]

set done 0
set rout [open "|$cmd 2>@$werr" r]
fconfigure $rout -blocking 0 -buffering line
fileevent $rout readable [list readstdout $rout]

vwait done
puts stderr DONE
exit 0

Ron Stohl

2005-07-25, 10:10 pm

P.S. I didn't mention that if tar produces any stderr, it is seen
immediately, even though stdout (which actually is produced first),
comes only at the end.

Ron Stohl wrote:

> I have the following sample program that runs a command in the
> background, capturing stdout and stderr in independent file handlers.
> The problem is that the "readstdout" procedure is not getting called
> ntil the "tar" command exits, then it is called once for each line out
> output. When I run the same thing using the "ls", the procedure is
> called as each file is found.
>
> Note that, when I run the same "tar" command at the command line, the
> files appear as they are restored.
>
> Can someone help me out?
> Thanks in advance
>
> ----------- cut ---------------
> load /usr/lpp/storix/tcl/lib/tclx8.4/libtclx8.4.so
>
> set cmd "ls -Rl / | grep test"
> # FILES IN ABOVE LS COMMAND ALWAYS APPEAR AT ONCE
> set cmd "tar -xvf/tmp/test.tar /home/makeimage /home/bbox/junk/zcat"
> # ABOVE COMMAND INCICATES ONE FILE AT START OF ARCHIVE, AND ONE
> # AT END OF ARCHIVE - BOTH FILES LISTED WHEN COMMAND EXITS.
>
> proc readstderr {input} {
> if {![eof $input]} {
> puts stderr [gets $input]
> }
> }
>
> proc readstdout {input} {
> global done
> if {![eof $input]} {
> puts stdout [gets $input]
> } else {
> set done 1
> close $input
> }
> }
>
> pipe rerr werr
> fconfigure $rerr -blocking 0 -buffering line
> fileevent $rerr readable [list readstderr $rerr]
>
> set done 0
> set rout [open "|$cmd 2>@$werr" r]
> fconfigure $rout -blocking 0 -buffering line
> fileevent $rout readable [list readstdout $rout]
>
> vwait done
> puts stderr DONE
> exit 0
>

Bob Techentin

2005-07-26, 9:05 am

"Ron Stohl" <rsstohl@PRIVATE.com> wrote
>
> I have the following sample program that runs a command in the
> background, capturing stdout and stderr in independent file
> handlers. The problem is that the "readstdout" procedure is not
> getting called ntil the "tar" command exits, then it is called once
> for each line out output. When I run the same thing using the "ls",
> the procedure is called as each file is found.
....
> set cmd "ls -Rl / | grep test"
> # FILES IN ABOVE LS COMMAND ALWAYS APPEAR AT ONCE
> set cmd "tar -xvf/tmp/test.tar /home/makeimage /home/bbox/junk/zcat"
> # ABOVE COMMAND INCICATES ONE FILE AT START OF ARCHIVE, AND ONE
> # AT END OF ARCHIVE - BOTH FILES LISTED WHEN COMMAND EXITS.
....
> set rout [open "|$cmd 2>@$werr" r]
> fconfigure $rout -blocking 0 -buffering line
> fileevent $rout readable [list readstdout $rout]

It looks like you're opening a pipe to the child process without the "&"
background qualifier. So the tar command runs to completion, then control
returns to you, and the event handlers fire.

By the way, you can use [glob] instead of "exec ls" and get better (faster,
more portable) results.

Bob
--
Bob Techentin techentin.robert@NOSPAMmayo.edu
Mayo Foundation (507) 538-5495
200 First St. SW FAX (507) 284-9171
Rochester MN, 55901 USA http://www.mayo.edu/sppdg/

Ron Stohl

2005-07-26, 5:03 pm

> It looks like you're opening a pipe to the child process without the "&"
> background qualifier. So the tar command runs to completion, then control
> returns to you, and the event handlers fire.
>
> By the way, you can use [glob] instead of "exec ls" and get better (faster,
> more portable) results.
>
> Bob

I tried this with and without the & before posting, and get the same
results either way. The "ls" is just an example of a command that does
not show the same symptoms - stdout is displayed as the command is
running and not waiting until the command completes.

Any other ideas?

Ron Stohl

2005-07-26, 5:03 pm

I have also found that, if I restore a larger number of files, stdout is
being displayed as the command runs. So it seems there is some buffering
when the "|open" is used (I also tried -buffering none). The stderr sent
to the pipe does not have the same issue.

So how do I get the output from the "|open" to come through without delay?

Ron Stohl wrote:

> I have the following sample program that runs a command in the
> background, capturing stdout and stderr in independent file handlers.
> The problem is that the "readstdout" procedure is not getting called
> ntil the "tar" command exits, then it is called once for each line out
> output. When I run the same thing using the "ls", the procedure is
> called as each file is found.
>
> Note that, when I run the same "tar" command at the command line, the
> files appear as they are restored.
>
> Can someone help me out?
> Thanks in advance
>
> ----------- cut ---------------
> load /usr/lpp/storix/tcl/lib/tclx8.4/libtclx8.4.so
>
> set cmd "ls -Rl / | grep test"
> # FILES IN ABOVE LS COMMAND ALWAYS APPEAR AT ONCE
> set cmd "tar -xvf/tmp/test.tar /home/makeimage /home/bbox/junk/zcat"
> # ABOVE COMMAND INCICATES ONE FILE AT START OF ARCHIVE, AND ONE
> # AT END OF ARCHIVE - BOTH FILES LISTED WHEN COMMAND EXITS.
>
> proc readstderr {input} {
> if {![eof $input]} {
> puts stderr [gets $input]
> }
> }
>
> proc readstdout {input} {
> global done
> if {![eof $input]} {
> puts stdout [gets $input]
> } else {
> set done 1
> close $input
> }
> }
>
> pipe rerr werr
> fconfigure $rerr -blocking 0 -buffering line
> fileevent $rerr readable [list readstderr $rerr]
>
> set done 0
> set rout [open "|$cmd 2>@$werr" r]
> fconfigure $rout -blocking 0 -buffering line
> fileevent $rout readable [list readstdout $rout]
>
> vwait done
> puts stderr DONE
> exit 0
>

Glenn Jackman

2005-07-26, 5:03 pm

At 2005-07-26 12:29PM, Ron Stohl <rsstohl@PRIVATE.com> wrote:
> I have also found that, if I restore a larger number of files, stdout is
> being displayed as the command runs. So it seems there is some buffering
> when the "|open" is used (I also tried -buffering none). The stderr sent
> to the pipe does not have the same issue.
>
> So how do I get the output from the "|open" to come through without delay?

Have a look at http://wiki.tcl.tk/stderr but you probably want
http://wiki.tcl.tk/bgexec

--
Glenn Jackman
NCF Sy

min
glennj@ncf.ca

Ron Stohl

2005-07-26, 5:03 pm

My sample code is a very simple example of a much more complex
procedure, just to illustrate the issue.

My procedure does what bgexec does, in pretty much the same way, but
does a lot more, such as redirect stdout and stderr independently to
separate channels, file handlers or disk files, gets separate process
IDs and return codes for each command in the pipeline, send signals to
the processes by setting a global variable, etc. It works great barring
this issue.

What I really need is someone to help me figure out how to prevent
stdout from being buffered in a "|open" command.

Glenn Jackman wrote:

> At 2005-07-26 12:29PM, Ron Stohl <rsstohl@PRIVATE.com> wrote:
>
>
>
> Have a look at http://wiki.tcl.tk/stderr but you probably want
> http://wiki.tcl.tk/bgexec
>

Donal K. Fellows

2005-07-26, 5:03 pm

Ron Stohl wrote:
> My sample code is a very simple example of a much more complex
> procedure, just to illustrate the issue.

Yes, and the real issue is that some programs (many of them in fact)
buffer all output not sent to terminals - and especially output sent to
pipelines - with a chunk size that isn't helpful if the program isn't
producing that much output. If the program being controlled is Tcl, you
can use [fconfigure stdout -buffering line] to mitigate the problem, and
there is an equivalent incantation within C which I've forgotten, but it
is in general nasty and difficult if you do not have access to the
source of the other program. But there is a solution...

> What I really need is someone to help me figure out how to prevent
> stdout from being buffered in a "|open" command.

Try the 'unbuffer' script in the Expect distribution, or even just the
Expect extension in general. Dealing with this sort of problem is one of
the things that expect does best.

Donal.

Ron Stohl

2005-07-26, 5:03 pm

Thanks for this pointer. The unbuffer lookw like what we need.
Unfortunately, we're very limited on the size of our TCL distribution,
and changes are hard to incorporate, so adding Expect is not an option.

I understand pmany programs do their own buffering. I'm curious why,
when I run the command from the command-line it does not buffer, but
when I run it from Tcl it does. If it's because the pipe created by Tcl
is buffered, is there a way in Tcl to disable the buffering with the
"open|" command? I use "fconfigure" to disable buffering on the read
side of the pipe, but what about when writing to it?

Sorry if my understanding is limited, but I've got a very nice procedure
for executing commands and handing output, error, return codes and
signals in various ways, but this is a big kink.

Donal K. Fellows wrote:

> Try the 'unbuffer' script in the Expect distribution, or even just the
> Expect extension in general. Dealing with this sort of problem is one of
> the things that expect does best.
>
> Donal.

Donal K. Fellows

2005-07-26, 10:05 pm

Ron Stohl wrote:
> Thanks for this pointer. The unbuffer lookw like what we need.
> Unfortunately, we're very limited on the size of our TCL distribution,
> and changes are hard to incorporate, so adding Expect is not an option.
>
> I understand pmany programs do their own buffering. I'm curious why,
> when I run the command from the command-line it does not buffer, but
> when I run it from Tcl it does. If it's because the pipe created by Tcl
> is buffered, is there a way in Tcl to disable the buffering with the
> "open|" command? I use "fconfigure" to disable buffering on the read
> side of the pipe, but what about when writing to it?

Well, while pipes are themselves buffered a bit, that's not normally an
issue. The issue here is the buffer on the writing side of the pipe;
most programs have output buffering on all channels (or FILE* instances
in C, or whatever the equivalent is elsewhere) *except* for stderr
(never ever buffered) and any channel that targets a terminal (because
they're believed to be "interactive".) More often than not, this is the
right thing as it speeds up pipe-based IPC a lot by cutting the number
of context switches. But not always, and you're in one of the spots
where it sucks.

The easiest way of fixing the problem is, if you can, to ensure that the
automatic buffering is turned off. Sometimes this isn't possible, and
that's when expect (which does complicated things with virtual terminals
behind the scenes) is exactly the right tool.

Donal.

Ron Stohl

2005-07-26, 10:05 pm

Thanks, I understand the issue. I don't have source to the commands I'm
running in the background, so what I need is some way to convince the
programs they're sending output to a pty instead of a pipe. I see a lot
of references to this, but so far no practical examples. It would be
nice to know how expect does this without the whole expect library. I
see the unbuffer (expect) command, but a simple C program to do this
would work for me.

Donal K. Fellows wrote:

> Well, while pipes are themselves buffered a bit, that's not normally an
> issue. The issue here is the buffer on the writing side of the pipe;
> most programs have output buffering on all channels (or FILE* instances
> in C, or whatever the equivalent is elsewhere) *except* for stderr
> (never ever buffered) and any channel that targets a terminal (because
> they're believed to be "interactive".) More often than not, this is the
> right thing as it speeds up pipe-based IPC a lot by cutting the number
> of context switches. But not always, and you're in one of the spots
> where it sucks.
>
> The easiest way of fixing the problem is, if you can, to ensure that the
> automatic buffering is turned off. Sometimes this isn't possible, and
> that's when expect (which does complicated things with virtual terminals
> behind the scenes) is exactly the right tool.
>
> Donal.

Sponsored Links