AWK - xgawk: BEGINFILE/ENDFILE extension proposal

Author

xgawk: BEGINFILE/ENDFILE extension proposal

Manuel Collado

2006-10-06, 7:56 am

The xgawk development team is discussing about the exact meaning of
BEGINFILE/ENDFILE special patterns as core language extensions.

We are very interested in comments from experienced users about the
correctness of our approach. So please read the following summary
specification and comment about its adequacy or risk of breaking
existing awk code.

Thanks to all who respond.

--------------------------------------------------------------------
Overall flow of control:
------------------------
do-BEGIN
foreach file in arguments loop
do-BEGINFILE
foreach record in file loop
apply rules to the record
endloop
do-ENDFILE
endloop
do-END

User requested transfer of control:
-----------------------------------

'nextfile' terminates the foreach record loop
--> ENDFILE for the current file, then BEGINFILE for the next file

'exit' terminates the foreach file loop
--> END

Input under direct control of the user should not interfere with the
global flow of control:
getline --> should never give control to BEGINFILE
close --> should never give control to ENDFILE

Setting of predefined variables:
--------------------------------

FILENAME NR FNR NF, $0-$NF ERRNO
---------- ---- ----- ------------ -------
BEGIN 0 0 0 0 0
BEGINFILE set = 0 0 =
record = ++ ++ set set? (1)
ENDFILE = = = = set? (2)
END = = = = =

The symbols mean:
0 --> reset to a 0/null value
= --> unmodified
++ --> increment
set --> recomputed
set? --> set on error, unmodified otherwise

NOTES:
(1) Set on getline errors
(2) Set if an error terminates processing the current file. This occurs
in some xgawk extensions, for XML in particular.

The rationale for FNR, NF, $0, ... is that they must be kept consistent.
If FNR is reset/incremented, then NF/$0/..$NF should be
reset/recomputed. And viceversa, if FNR is kept unchanged, then also
NF/$0/..$NF should keep their values (in ENDFILE/END rules).

An open question is if BEGINFILE should be executed before or after
opening the file. The current proposal is to do-BEGINFILE before the
file is open, so to be able to set some processing mode variables before
the file is actually opened.
--------------------------------------------------------------------

Regards.
--
To reply by e-mail, please remove the extra dot
in the given address: m.collado -> mcollado

Ed Morton

2006-10-06, 7:56 am

Manuel Collado wrote:

> The xgawk development team is discussing about the exact meaning of
> BEGINFILE/ENDFILE special patterns as core language extensions.
>
> We are very interested in comments from experienced users about the
> correctness of our approach. So please read the following summary
> specification and comment about its adequacy or risk of breaking
> existing awk code.
>
> Thanks to all who respond.
>
> --------------------------------------------------------------------
> Overall flow of control:
> ------------------------
> do-BEGIN
> foreach file in arguments loop
> do-BEGINFILE
> foreach record in file loop
> apply rules to the record
> endloop
> do-ENDFILE
> endloop
> do-END
>
>
> User requested transfer of control:
> -----------------------------------
>
> 'nextfile' terminates the foreach record loop
> --> ENDFILE for the current file, then BEGINFILE for the next file
>
> 'exit' terminates the foreach file loop
> --> END
>
> Input under direct control of the user should not interfere with the
> global flow of control:
> getline --> should never give control to BEGINFILE
> close --> should never give control to ENDFILE
>
>
> Setting of predefined variables:
> --------------------------------
>
> FILENAME NR FNR NF, $0-$NF ERRNO
> ---------- ---- ----- ------------ -------
> BEGIN 0 0 0 0 0
> BEGINFILE set = 0 0 =
> record = ++ ++ set set? (1)
> ENDFILE = = = = set? (2)
> END = = = = =
>
> The symbols mean:
> 0 --> reset to a 0/null value
> = --> unmodified
> ++ --> increment
> set --> recomputed
> set? --> set on error, unmodified otherwise
>
> NOTES:
> (1) Set on getline errors
> (2) Set if an error terminates processing the current file. This occurs
> in some xgawk extensions, for XML in particular.
>
>
> The rationale for FNR, NF, $0, ... is that they must be kept consistent.
> If FNR is reset/incremented, then NF/$0/..$NF should be
> reset/recomputed. And viceversa, if FNR is kept unchanged, then also
> NF/$0/..$NF should keep their values (in ENDFILE/END rules).
>
>
> An open question is if BEGINFILE should be executed before or after
> opening the file. The current proposal is to do-BEGINFILE before the
> file is open, so to be able to set some processing mode variables before
> the file is actually opened.
> --------------------------------------------------------------------
>
> Regards.

Could you clarify why this is useful enough to warrant new language
constructs? What is it that I can't do with something like:

function foo() { ... }
FNR==1 { foo() }
END { foo() }

that I can do with

BEGINFILE { ... }
ENDFILE { ... }

The "nextfile FILENAME" thing also seems a lot like a "goto LABEL" to me
since it facilitates inversion of control, so could you also clarify why
that's a good idea?

Regards,

Ed.

Kenny McCormack

2006-10-06, 7:56 am

In article < BqydnROaeIvU1rvYnZ2dnUVZ_qCdnZ2d@comcast
.com>,
Ed Morton <morton@lsupcaemnt.com> wrote:
....
>Could you clarify why this is useful enough to warrant new language
>constructs? What is it that I can't do with something like:
>
> function foo() { ... }
> FNR==1 { foo() }
> END { foo() }

(Speaking as one who uses BEGINFILE/ENDFILE in TAWK all the time and is
quite happy with it)

I think you may be right about FNR==1 being the same as BEGINFILE. I'll
have to check to see if there are any corner cases.

But ENDFILE is different from END in that END gets run once after *all*
files have been read (or an "exit" has been executed), whereas ENDFILE
gets run once per file. I can't, offhand, think of any way to emulate
that easily in normal AWK code. (note: I say "easily" here, because I
think there is some elaborate method given in the EAP book, showing how
to emulate it in GAWK, but it looked pretty kludgey to me)

>that I can do with
>
> BEGINFILE { ... }
> ENDFILE { ... }
>
>The "nextfile FILENAME" thing also seems a lot like a "goto LABEL" to me
>since it facilitates inversion of control, so could you also clarify why
>that's a good idea?

Well, my guess is that this is all being driven by some obscure XML
based need on the part of the xgawkers. Thus, it may not make any sense
to us. However, this is the sort of thing that every once in a great
while, you find yourself wanting to do. I think I've used TAWK's
version of this, "close(FILENAME)", about twice in my life now.

Juergen Kahrs

2006-10-06, 7:56 am

Kenny McCormack wrote:

> Well, my guess is that this is all being driven by some obscure XML
> based need on the part of the xgawkers. Thus, it may not make any sense

No, if this was a problem with the XML extension,
then we would have discussed it internally. Peter
needed BEGINFILE and ENDFILE for a different gawk
extension. Since these new keywords BEGINFILE and
ENDFILE may be of interest to all gawk users, I
encouraged Peter and Manuel to post these questions
here.

Your comments have helped to elucidate the implications
of such fundamental additions to the language. I really
appreciate your comments.

Andrew Schorr

2006-10-06, 7:56 am

Ed Morton wrote:
> Could you clarify why this is useful enough to warrant new language
> constructs? What is it that I can't do with something like:
>
> function foo() { ... }
> FNR==1 { foo() }
> END { foo() }
>
> that I can do with
>
> BEGINFILE { ... }
> ENDFILE { ... }
>
> The "nextfile FILENAME" thing also seems a lot like a "goto LABEL" to me
> since it facilitates inversion of control, so could you also clarify why
> that's a good idea?

A couple of reasons for these extensions came up in our discussions.
The first
was that current awk syntax gives you no way to detect a zero-length
file. For example:

bash-3.00$ gawk '{print NR,FNR,FILENAME}' /etc/issue /dev/null
/etc/issue
1 1 /etc/issue
2 2 /etc/issue
3 3 /etc/issue
4 1 /etc/issue
5 2 /etc/issue
6 3 /etc/issue

As you can see, the gawk code never even sees that /dev/null was
processed.
Now I'm not exactly clear on why this is important, but one could
imagine that
somebody might write some code that cared about this.

Another issue is error handling. Suppose that an invalid filename is
passed
on the command line. With a BEGINFILE rule that runs before the file
is opened,
you could actually test (using the filefuncs library stat function)
whether the
file is readable, and, if not, skip it using nextfile. Or you could
use any other
logic you please to decide that you may want to skip this file.

With respect to ENDFILE: I'm not sure it's really needed, but if we
decide
to add BEGINFILE, it seems reasonable to add ENDFILE as well. While it
is true that much of this functionality can be achieved in other ways,
the
BEGINFILE and ENDFILE patterns would make it much easier to handle
this stuff in many cases. So why not improve the language's eas of
use?

As for adding an optional argument to "nextfile", it is currently true
that you can
achieve much of this functionality by manipulating the ARGV array (by
inserting
new files to process). But the argument was made that this method runs
the
risk of overflowing the ARGV array. While I agree this seems rather
unlikely, it
is true that it is rather ugly to have to manipulate the ARGV array in
that way.
And since adding an optional argument to nextfile should not break
anything
(since that syntax is currently illegal), it seems like a reasonable
extension.

Regards,
Andy

Jon LaBadie

2006-10-06, 6:56 pm

Andrew Schorr wrote:
> Ed Morton wrote:
>
> A couple of reasons for these extensions came up in our discussions.

Couple of blue-sky questions, mostly about flow control.

nextfile, as I understand it, would execute the ENDFILE block and start
the next data file from the ARGV list, starting at its BEGINFILE block.

Would nextfile "somefile" effectively inserting a file into the ARGV list,
i.e. processing of the ARGV list continuing when "somefile" was closed,
or does it end processing of ARGV

Any different consequences of a nextfile "somefile" in the BEGINFILE block
where FILENAME is set, but not yet opened? Going to the ENDFILE block for
FILENAME might be strange in that situation.

Similar question about nextfile "somefile" in an ENDFILE block. Probably
don't want to reenter the ENDFILE block for the current file.

Does close(FILENAME) have any different effect than nextfile? Might it
skip ENDFILE? Might it return execution control to the statement following
the close(FILENAME) statement rather than starting the BEGINFILE block of
the next datafile.

For some purposes variables of NEXTFILENAME and PREVFILENAME could be useful.
Generally you could play with ARGV, but that would not hold up for example
in the ENDFILE block reached after a nextfile "somefile".

Kenny McCormack

2006-10-06, 6:56 pm

In article <4on31hFf9ugaU1@individual.net>,
Juergen Kahrs <Juergen.KahrsDELETETHIS@vr-web.de> wrote:
>Kenny McCormack wrote:
>
>
>No, if this was a problem with the XML extension,
>then we would have discussed it internally. Peter
>needed BEGINFILE and ENDFILE for a different gawk
>extension. Since these new keywords BEGINFILE and
>ENDFILE may be of interest to all gawk users, I
>encouraged Peter and Manuel to post these questions
>here.

I don't see how any of what you have written contradicts anything that I
have written.

Manuel Collado

2006-10-06, 6:56 pm

Jon LaBadie escribió:
> Andrew Schorr wrote:

As said before, empty files are totally unnoticed.
[color=darkred]

It is just an efficient way of pushing a user-selected file in front of the
queue of files waiting to be processed, without tampering with ARGC, ARGV,
ARGIND.
[color=darkred]

Ditto.
[color=darkred]
>
> Couple of blue-sky questions, mostly about flow control.
>
> nextfile, as I understand it, would execute the ENDFILE block and start
> the next data file from the ARGV list, starting at its BEGINFILE block.

Yes.

>
> Would nextfile "somefile" effectively inserting a file into the ARGV list,
> i.e. processing of the ARGV list continuing when "somefile" was closed,
> or does it end processing of ARGV

Normal processing of ARGV is resumed after "somefile" is processed. But
nextfile "somefile" will only use internal variables, and not the ARGV list.

>
> Any different consequences of a nextfile "somefile" in the BEGINFILE block
> where FILENAME is set, but not yet opened? Going to the ENDFILE block for
> FILENAME might be strange in that situation.
>
> Similar question about nextfile "somefile" in an ENDFILE block. Probably
> don't want to reenter the ENDFILE block for the current file.

Sorry, don't understand your question. The proposed behaviour is:

...
processing "fileX"...
...
nextfile "somefile"
--> ENDFILE with FILENAME = "fileX"
--> BEGINFILE with FILENAME = "somefile"
...
processing "somefile"
...
at eof("somefile") --> ENDFILE with FILENAME = "somefile"

>
> Does close(FILENAME) have any different effect than nextfile? Might it
> skip ENDFILE? Might it return execution control to the statement following
> the close(FILENAME) statement rather than starting the BEGINFILE block of
> the next datafile.

Direct control of input from the user via getline/close will not cause
execution neither of the BEGINFILE nor the ENDFILE blocks. The user can
just execute the desired actions in normal rules, just before the first
getline from a given file or after closing the file.

Please consider that neither getline nor close interrupt the normal record
processing loop of the current file.

>
> For some purposes variables of NEXTFILENAME and PREVFILENAME could be
> useful.
> Generally you could play with ARGV, but that would not hold up for example
> in the ENDFILE block reached after a nextfile "somefile".

Regards.
--
Manuel Collado - http://lml.ls.fi.upm.es/~mcollado

Jon LaBadie

2006-10-06, 6:56 pm

Manuel Collado wrote:
> Jon LaBadie escribió:

>
> Sorry, don't understand your question. The proposed behaviour is:
>
> ...
> processing "fileX"...
> ...
> nextfile "somefile"
> --> ENDFILE with FILENAME = "fileX"
> --> BEGINFILE with FILENAME = "somefile"
> ...
> processing "somefile"
> ...
> at eof("somefile") --> ENDFILE with FILENAME = "somefile"
>

I was thinking of a situation something like:

BEGINFILE {
if (someflag)
nextfile "someotherfile"
...
}

You got to BEGINFILE with a new filename, but that file is not yet opened.
Will a nextfile in the BEGINFILE block still go to ENDFILE, even though
the file is not yet open and might that cause any problems?

If instead the nextfile was in the ENDFILE block, you are already ending
one file, you probably don't want to enter the ENDFILE block a second
time for the same file.

Jürgen Kahrs

2006-10-06, 6:56 pm

Kenny McCormack wrote:
> In article <4on31hFf9ugaU1@individual.net>,
> Juergen Kahrs <Juergen.KahrsDELETETHIS@vr-web.de> wrote:
>
> I don't see how any of what you have written contradicts anything that I
> have written.

You guessed "this is all being driven by some obscure XML based need".
I can assure you that these suggested changes have nothing to
do with the XML extension.

GNU AWK (and also the xgawk distribution) support
the concept of "functional extension". This means
dynamic libraries. The XML extension is a dyn lib.
Peters suggests changes that are independent of
the XML extension.

Peter V. Saveliev

2006-10-06, 6:56 pm

<skip >
> I was thinking of a situation something like:
>
> BEGINFILE {
> if (someflag)
> nextfile "someotherfile"
> ...
> }
>
> You got to BEGINFILE with a new filename, but that file is not yet opened.
> Will a nextfile in the BEGINFILE block still go to ENDFILE, even though
> the file is not yet open and might that cause any problems?

Right now (I updated the patch), nextfile, called from BEGINFILE, will
start BEGINFILE/record/ENDFILE cycle for the next file immediately. No
ENDFILE for not opened file will be interpreted, surely.

>
> If instead the nextfile was in the ENDFILE block, you are already ending
> one file, you probably don't want to enter the ENDFILE block a second
> time for the same file.

Yes, and it works right as you can expect. No ENDFILE re-entrance for
the same file will be done.

Thanks a lot for comments, I already fixed some bugs by comments in
this thread. Please, check updated patch. Small awk template for
testing is also available on the patch page.

Anton Treuenfels

2006-10-07, 3:56 am

"Andrew Schorr" <aschorr@telemetry-investments.com> wrote in message
news:1160141682.541594.314730@k70g2000cwa.googlegroups.com...
> Ed Morton wrote:
>
> A couple of reasons for these extensions came up in our discussions.
> The first
> was that current awk syntax gives you no way to detect a zero-length
> file. For example:
>
> bash-3.00$ gawk '{print NR,FNR,FILENAME}' /etc/issue /dev/null
> /etc/issue
> 1 1 /etc/issue
> 2 2 /etc/issue
> 3 3 /etc/issue
> 4 1 /etc/issue
> 5 2 /etc/issue
> 6 3 /etc/issue
>
> As you can see, the gawk code never even sees that /dev/null was
> processed.
> Now I'm not exactly clear on why this is important, but one could
> imagine that
> somebody might write some code that cared about this.
>
> Another issue is error handling. Suppose that an invalid filename is
> passed
> on the command line. With a BEGINFILE rule that runs before the file
> is opened,
> you could actually test (using the filefuncs library stat function)
> whether the
> file is readable, and, if not, skip it using nextfile. Or you could
> use any other
> logic you please to decide that you may want to skip this file.
>
> With respect to ENDFILE: I'm not sure it's really needed, but if we
> decide
> to add BEGINFILE, it seems reasonable to add ENDFILE as well. While it
> is true that much of this functionality can be achieved in other ways,
> the
> BEGINFILE and ENDFILE patterns would make it much easier to handle
> this stuff in many cases. So why not improve the language's eas of
> use?

With an ENDFILE action block (as TAWK defines it) it's possible to use any
file-oriented function on ARGV[ARGI+1] before the automatic input loop gets
a hold of it. Eg, you can check whether the filename is valid, whether it
exists, whether it has non-zero length, etc. You can't do the same thing
with a BEGINFILE action block (as TAWK defines it) because ARGV[ARGI] has
already been opened (or failed to open with an error reported directly to
the console).

You can't do the same thing with an ENDFILE action block (as the proposed
extension defines it) because you don't know if the next file will come from
ARGV[ARGI+1] or from NEXTFILE. You can do it with a BEGINFILE action block
(as the proposed extension defines it) because FILENAME is valid whether it
came from ARGV[ARGI] or NEXTFILE. But this is only because of NEXTFILE, or
more specifically NEXTFILE "somefile". Without that there'd be no ambiguity
about which file comes next.

Of course you could also do much the same thing in one fell swoop in a BEGIN
action block by stepping through the entire ARGV[] array looking for
problems at the start. But again that wouldn't work for NEXTFILE "somefile".

Still, that ambiguity comes only because NEXTFILE "somefile" doesn't alter
ARGV[]. If "somefile" was inserted in ARGV[ARGI+1] and all existing files
pushed back, then the ambiguity would go away. Also you could use NEXTFILE
within a NEXTFILE and still maintain a complete list of files used in ARGV[]
(which you could run back and forth over using ARGI). So maybe NEXTFILE
becomes NEXTFILE("somefile") and has the effect of:

for ( i = ++ARGC; i > ARGI+1; i-- )
ARGV[ i ] = ARGV[ i-1 ]
ARGV[ ARGI+1 ] = "somefile"
do_ENDFILE # the TAWK way...
close( ARGV[ARGI] )
do_ENDFILE # ...or the proposed extension way

and returns a success/fail boolean value (ie., "okay"/"not enough room in
ARGV[]").

Although maybe it would be better to have NEXTFILE() simply insert
"somefile" and have another function to trigger the ENDFILE action block
(eg., close(FILENAME) ).. That would have the advantage of allowing
ARGV[ARGI] to be read to completion or not as desired, and also the
advantage that there'd be little confusion about what NEXTFILE() in a
BEGINFILE (aka BEFOREFILE) or an ENDFILE (aka AFTERFILE) action block did
(ie., it doesn't trigger any immediate action at all). You could even add
multiple filenames at once in LIFO fashion by looping NEXTFILE().

In fact I think I could cobble together something much like this in TAWK as
it is. Not sure how I'd check for ARGV[] overflow, though, and while I've
never considered it before I wonder what would happen if I tried
close(FILENAME) in an ENDFILE action block...probably nothing, since while
FILENAME is open there are no more records to be read...hmm, but what about
in a BEGINFILE...?

Still thinking out loud...

- Anton Treuenfels

Paddy

2006-10-07, 3:56 am

> The xgawk development team is discussing about the exact meaning of
> BEGINFILE/ENDFILE special patterns as core language extensions.
>
> We are very interested in comments from experienced users about the
> correctness of our approach. So please read the following summary
> specification and comment about its adequacy or risk of breaking
> existing awk code.
>
> Thanks to all who respond.
>
> --------------------------------------------------------------------
<<SNIP>>
>
> Input under direct control of the user should not interfere with the
> global flow of control:
> getline --> should never give control to BEGINFILE
> close --> should never give control to ENDFILE
>
The above is what sticks out on first reading.
Why not Obey a rule of ALWAYS calling BEGINFILE before opening a file,
and ENDFILE just before closing a file, even when caused by a getline
or close?
If you cannot easiy modify AWK to return to just after the getline or
close call then I will understand, but I would prefer, in effect

getline ... IS EQUIVALENT TO {getline ...; ENDFILE(); BEGINFILE()}
close(...) IS EQUIVALENT TO {ENDFILE(); close(...); BEGINFILE()}

(but only when the getline gets to the end of a file).

> An open question is if BEGINFILE should be executed before or after
> opening the file. The current proposal is to do-BEGINFILE before the
> file is open, so to be able to set some processing mode variables before
> the file is actually opened.

ENDFILE before the file is actually closed.
BEGINFILE before the next is opened.

ENDFILE run before final END, but only if their has been at least one
file to process.
BEGINFILE run after all BEGIN blocks at start, but only if their is
then
a file to process.

A program of:
BEGIN{exit}BEGINFILE{...}ENDFILE{...}END{...}
would not call BEGINFILE or ENDFILE but only END{...}

This way BEGINFILE& ENDFILE will always be called in pairs for
each file. if exit is called from anywhere but a BEGIN or END block,
then ENDFILE should be called before then going to END.

- Paddy.

Peter V. Saveliev

2006-10-07, 3:56 am

<skip />
> The above is what sticks out on first reading.
> Why not Obey a rule of ALWAYS calling BEGINFILE before opening a file,
> and ENDFILE just before closing a file, even when caused by a getline
> or close?
<skip />

With getline/close one don't need BEGINFILE and ENDFILE, one can just
do something before getline and after close. In the internal maillist,
Manuel wrote:

>
> Well, global input control can be done in two differente ways:
> - Automatically, by the main core loop - file-loop/record-loop
> - Manually, via getline/close
>
> In the second case the user can easily insert code for handling file
> begin/end in normal rules, without any extension of the awk core
> language.
>
> This is not the case for the automatic loop. In this case
> BEGINFILE/ENDFILE really helps, avoiding the need for tricky
> code based on FNR==1 and END.

And I think, he's right.

Peter V. Saveliev

2006-10-07, 3:56 am

<skip />
> Still, that ambiguity comes only because NEXTFILE "somefile" doesn't alter
> ARGV[]. If "somefile" was inserted in ARGV[ARGI+1] and all existing files
> pushed back, then the ambiguity would go away. Also you could use NEXTFILE
> within a NEXTFILE and still maintain a complete list of files used in ARGV[]
> (which you could run back and forth over using ARGI). So maybe NEXTFILE
> becomes NEXTFILE("somefile") and has the effect of:
<skip />

Just think of that. I have a script, that works as a daemon. It opens
an arbitrary file, works with it, then closes it. Actually, these files
are sockets and the script is very lightweight tcp server. Must I add
"/dev/fd/x" for every connection in the ARGV array? For what? :) It is
only one of use cases. With the patch you have two ways: use ARGV with
nextfile/close and open an arbitrary file with nextfile "bala". These
ways do not interfere. You can use or not use any.

If you afraid that nextfile "bala" will break the logic of your script,
you can just don't use it. Plain nextfile and close() constructions
performs as usual, so, feel free to stick with legacy syntax. Even
more, -W posix, -W compat will abort script in case of nextfile "bala".
-W lint will print out a warning message.

Syntax 'nextfile "bala"' does the same as 'getline <file' does, but in
main input loop. So, now you have equal abilities in getline loop and
main loop:

getline - gets next file from ARGV
getline <"file" - opens an arbitrary file
nextfile - gets next file from ARGV
nextfile "file" - opens an arbitrary file

The patch adds only the last syntax construction.

Peter V. Saveliev

2006-10-07, 6:57 pm

Please, review last changes in the patch & spec, before it will be
integrated:

http://xgawk.radlinux.org/Articles/patch-fileworks/show

If it is important to have valid FILENAME in BEGINFILE (as in TAWK), we
shall rename pattern to BEFOREFILE, as it was proposed. If it is not
important, we'll leave BEGINFILE as is. ENDFILE will be as it is now,
cause there is no difference in TAWK's ENDFILE behaviour and the one
from the patch, and it will be compatible.

Thanks for posted comments, especially for nextfile-from-BEGINFILE and
nextfile-from-ENDFILE test cases. The misbehaviour of initial version
was fixed.

Sponsored Links