{ => } autocomposition

In Pugs's t/pugsbugs/map_function_return_values.t, iblech added this test:

%ret = map { $_ => uc $_ }, split "", $text;

This fails because it is parsed, undef the {=>} autocomposition rule, into:

# Fails because arg1 is not Code
%ret = map(hash($_ => uc $_), split("", $text));

Instead of the intended:

# Works correctly
%ret = map(sub ($_) { $_ => uc $_ }, split("", $text));

Is this a known issue?

Thanks,
/Autrijus/

Hi,

Autrijus Tang wrote:
>     %ret = map { $_ => uc $_ }, split "", $text;
[...]

I suppose my test is wrong.

When I clicked on reply a moment ago, I wanted to propose to change the
hash/code disambiguation rule, so that {...} is always parsed as Code
if the body contains "$_" or "$^...".

But as this change would break the following code, I think we should
leave it as it now.
my @AoH = map {
my $hashref = { $^a => ... };
do_something_with $hashref;
$hashref;
};


--Ingo

--
Linux, the choice of a GNU | Mr. Cole's Axiom: The sum of the
generation on a dual AMD   | intelligence on the planet is a constant;
Athlon!                    | the population is growing.

On Wed, Apr 20, 2005 at 05:08:53PM +0200, Ingo Blechschmidt wrote:
: Hi,
:
: Autrijus Tang wrote:
: >     %ret = map { $_ => uc $_ }, split "", $text;
: [...]
:
: I suppose my test is wrong.
:
: When I clicked on reply a moment ago, I wanted to propose to change the
: hash/code disambiguation rule, so that {...} is always parsed as Code
: if the body contains "$_" or "$^...".
:
: But as this change would break the following code, I think we should
: leave it as it now.
:   my @AoH = map {
:     my $hashref = { $^a => ... };
:     do_something_with $hashref;
:     $hashref;
:   };

Hmm, what we have said in the past is that the proper way to disambiguate
that is to require an explicit "return" if you mean the closure, but that
doesn't actually work unless there's an explicit "sub".  I suppose the
rightest answer at this point is:

%ret = map { ($_ => uc $_) }, split "", $text;

since the => is no longer the top-level operator, but the parens are, if
you believe in parens as an operator and not just a grouper.  Or

%ret = map { $_ => uc $_; }, split "", $text;

should presumably also work.  That may look like an arbitrary amount
of lookahead, but I tried to define the hash/closure rule in terms of
a semantic analysis rule rather than a syntax rule, such that it's
always parsed as a closure, but at some point in semantic analysis
you can look at the AST and see that the top-level operator is =>,
and throw an implicit "new Hash: do" on the front of the closure,
or whatever operator it is that evaluates a closure for its pairs
and builds a hash from it.

I'd really like to save the overloading of {...} if we can DWTM in most
cases and at least produce decent errors in the other cases.  We can
certainly produce a decent "Maybe you meant..." diagnostic on the
map thingy, even if it defaults counterintuitively in that case.

Larry

On Wed, Apr 20, 2005 at 08:51:24AM -0700, Larry Wall wrote:
> That may look like an arbitrary amount of lookahead, but I tried to
> define the hash/closure rule in terms of a semantic analysis rule
> rather than a syntax rule, such that it's always parsed as a closure,
> but at some point in semantic analysis you can look at the AST and see
> that the top-level operator is =>, and throw an implicit "new Hash:
> do" on the front of the closure, or whatever operator it is that
> evaluates a closure for its pairs and builds a hash from it.

Currently in Pugs, the analysis is done when the parser is done parsing a
block, i.e. when there is a Exp already formed:

-- Try to analyze Exp if the block is bare and without formal
-- arguments; extractHash merely looks at the toplevel OP, to
-- see if it matches (pair | => ) or ("," [(pair | => ), ...])
retBlock SubBlock Nothing exp
| Just hashExp <- extractHash exp = return \\{}" [hashExp]

>     %ret = map { ($_ => uc $_) }, split "", $text;
>     %ret = map { $_ => uc $_; }, split "", $text;

Sadly, both the grouping parens and the trailing semicolon are currently
ignored as semantically insignificant.

I think the latter suggestions works better, as we already make the
distinction between semicolon-separated Stmts (which contains multiple
Exps), and a simple Exp.

Adding a special form Parens that takes one Exp and simply returns
it is possible, but unless it serves to disambiguate other cases,
that approach seems more heavy-handed to me.

Thanks,
/Autrijus/

On Thu, Apr 21, 2005 at 12:09:18AM +0800, Autrijus Tang wrote:
: Adding a special form Parens that takes one Exp and simply returns
: it is possible, but unless it serves to disambiguate other cases,
: that approach seems more heavy-handed to me.

As someone who is currently trying to write a perfect p5-to-p5 [sic]
translator, you have to somehow remember the parens (and whitespace
(and comments (and constant-folded subtrees))) to have a complete AST
representation of the original text.  Or maybe it should be called
a CST instead of an AST in that case...

Though remembering doesn't have to be the default.  But I'd like
to point out that adding such annotations to the AST after you've
optimized parts of the tree away is much more difficult than reserving
a spot for them in the first place.  I speak from  experience...  :-)

Larry

On Wed, Apr 20, 2005 at 09:38:28AM -0700, Larry Wall wrote:
> As someone who is currently trying to write a perfect p5-to-p5 [sic]
> translator, you have to somehow remember the parens (and whitespace
> (and comments (and constant-folded subtrees))) to have a complete AST
> representation of the original text.  Or maybe it should be called
> a CST instead of an AST in that case...

I see.  Do you think preserving the /span/ (i.e. the character offset ranges
)
of each AST element is enough to do that?  That effectively means each
node points to a substring inside the original source string.

That way, p6-to-p6 translators in the future can, at worst, trigger a
reparse under the appropriate subrules level, and obtain the information
we optimized away.

Thanks,
/Autrijus/

On Thu, Apr 21, 2005 at 12:50:56AM +0800, Autrijus Tang wrote:
: I see.  Do you think preserving the /span/ (i.e. the character offset rang
es)
: of each AST element is enough to do that?  That effectively means each
: node points to a substring inside the original source string.

Yes, though making a copy isn't bad if it's not the default behavior.
It mostly just needs a "spot".

: That way, p6-to-p6 translators in the future can, at worst, trigger a
: reparse under the appropriate subrules level, and obtain the information
: we optimized away.

Except that you've probably thrown away the definition of "appropriate"
by then as well.  :-)

p5-to-p5 actually intercepts the op_free() calls and has a place
for all the optimized trees.  But that's not the default behavior.
By default it only costs one empty "madprops" pointer per op node.
(MAD stands for miscellaneous attribute decorations.)  Only if we
set the "madskills" option do we generate a linked list of madprops.
Then you don't have to worry about the overhead in the general case,
and you also don't have to worry about your two parses getting out
of sync in the specific case.  You just have to know in advance when
you want to remember the details, which I think is not too much of
a constraint.

If you don't want to worry about this in the Haskell version of the
compiler, that's okay, as long as we think about it more when we
bootstrap the Perl 6 version.  Certainly if you're remembering
parens in some optional data structure, you can't depend on it
for ordinary disambiguation.

Larry

On Wed, Apr 20, 2005 at 10:21:32AM -0700, Larry Wall wrote:
: Except that you've probably thrown away the definition of "appropriate"
: by then as well.  :-)

Well, maybe not, since you presumably need to keep track of the current
parser for eval.

Larry

On Wed, Apr 20, 2005 at 10:21:32AM -0700, Larry Wall wrote:
> On Thu, Apr 21, 2005 at 12:50:56AM +0800, Autrijus Tang wrote:
> : I see.  Do you think preserving the /span/ (i.e. the character offset ra
nges)
> : of each AST element is enough to do that?  That effectively means each
> : node points to a substring inside the original source string.
> 
> Yes, though making a copy isn't bad if it's not the default behavior.
> It mostly just needs a "spot".

Okay.  I'll make it happen then.  Is this something we want to make
available at the runtime language level, in addition to the compiler
error/warnings level?

> p5-to-p5 actually intercepts the op_free() calls and has a place
> for all the optimized trees.  But that's not the default behavior.
> By default it only costs one empty "madprops" pointer per op node.
> (MAD stands for miscellaneous attribute decorations.)  Only if we
> set the "madskills" option do we generate a linked list of madprops.

Wow, nice trick.  Mad props to you. :)

> Then you don't have to worry about the overhead in the general case,
> and you also don't have to worry about your two parses getting out
> of sync in the specific case.  You just have to know in advance when
> you want to remember the details, which I think is not too much of
> a constraint.

As you noted in the followup mail, I think the compiler needs to
remember the active subrule and source code range anyway, so that
performance penalty is already there.

> If you don't want to worry about this in the Haskell version of the
> compiler, that's okay, as long as we think about it more when we
> bootstrap the Perl 6 version.

I do intend to bootstrap the Perl 6 version by simply compiling
the Pugs source code from Haskell into Perl6 AST, then decompile
it back into Perl 6 code. :-)

That is, unless someone crazy enough comes around and recoded
everything in Perl 6 by hand...

Thanks,
/Autrijus/

At 10:38 PM +0800 4/20/05, Autrijus Tang wrote:
>In Pugs's t/pugsbugs/map_function_return_values.t, iblech added this test:
>     %ret = map { $_ => uc $_ }, split "", $text;
>
>This fails because it is parsed, undef the {=>} autocomposition rule, into:
>     # Fails because arg1 is not Code
>     %ret = map(hash($_ => uc $_), split("", $text));
>
>Instead of the intended:
>     # Works correctly
>     %ret = map(sub ($_) { $_ => uc $_ }, split("", $text));
>
>Is this a known issue?

A clear way to disambiguate a block from a hash-ref when using
map/grep/sort etc is to use a colon before the leading brace for a
block rather than a space, like this:

map:{ $_ => uc $_ }

I read that in the synopsis documents a month back, though I'm having
a bit of trouble finding the reference now.  Maybe it has something
to do with adverbs?  But I do explicitly remember it being used to
disambiguate.

In any event, I used that form exclusively with my Perl 6 ports to date.

So in that case, the test is wrong.

-- Darren Duncan