Home > Archive > Scheme > November 2004 > module systems: why no namespaces?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
module systems: why no namespaces?
|
|
| Nic Ferrier 2004-10-21, 8:57 pm |
| I was going to post an overview of my scheme implementation's module
system (which is based on Perl's) here. But before I do that I'd like
to ask why most Scheme module systems don't utilize namespaces.
Namespace separation is common in non-lisp 3GLs; Perl, Python, Java,
C++ all have namespace based module systems.
I can't see any reason why LISPs should not have namespace separation
embedded in their module systems.
Is there a reason why?
--
Nic Ferrier
http://www.tapsellferrier.co.uk
| |
| Pascal Costanza 2004-10-21, 8:57 pm |
|
Nic Ferrier wrote:
> I was going to post an overview of my scheme implementation's module
> system (which is based on Perl's) here. But before I do that I'd like
> to ask why most Scheme module systems don't utilize namespaces.
>
> Namespace separation is common in non-lisp 3GLs
[...]
It's common in Common Lisp. ;)
Pascal
--
Tyler: "How's that working out for you?"
Jack: "Great."
Tyler: "Keep it up, then."
| |
| Shriram Krishnamurthi 2004-10-21, 8:57 pm |
| Nic Ferrier <nferrier@tapsellferrier.co.uk> writes:
> I was going to post an overview of my scheme implementation's module
> system (which is based on Perl's) here. But before I do that I'd like
> to ask why most Scheme module systems don't utilize namespaces.
Do you actually understand the design constraints behind modules? In
particular, that the word "module" means about 4-5 different things?
Or, as Bear (I believe) will say is a reflex action around here, I
have to ask: have you read Flatt's ICFP 2002 paper?
Shriram
| |
| Brandon J. Van Every 2004-10-22, 3:57 am |
| Shriram Krishnamurthi wrote:
>
> Or, as Bear (I believe) will say is a reflex action around here, I
> have to ask: have you read Flatt's ICFP 2002 paper?
For those of us 'not in the know' about which particular paper this is, or
what it contains, could you provide a URL?
--
Cheers, www.indiegamedesign.com
Brandon Van Every Seattle, WA
20% of the world is real.
80% is gobbledygook we make up inside our own heads.
| |
| Jens Axel Søgaard 2004-10-22, 3:57 am |
| Brandon J. Van Every wrote:
> Shriram Krishnamurthi wrote:
>=20
>=20
>=20
> For those of us 'not in the know' about which particular paper this is,=
or
> what it contains, could you provide a URL?
>=20
<http://library.readscheme.org/page5.html>
--=20
Jens Axel S=F8gaard
| |
| Nic Ferrier 2004-10-22, 3:58 pm |
| Shriram Krishnamurthi <sk@cs.brown.edu> writes:
> Nic Ferrier <nferrier@tapsellferrier.co.uk> writes:
>
>
> Do you actually understand the design constraints behind modules? In
> particular, that the word "module" means about 4-5 different things?
You're probably right.
Like some people say about Art: I don't know much about it but I do
know what I like.
And I am pretty stupid.
> Or, as Bear (I believe) will say is a reflex action around here, I
> have to ask: have you read Flatt's ICFP 2002 paper?
Well, as I say, I'm stupid. But the 2002 paper is about macro
languages. I think the relevant paper is the 1998 one co-authored with
Matthias Felleisen:
http://www.ccs.neu.edu/scheme/pubs/pldi98-ff.ps.gz
I have to say, it looks jolly clever but I only got to page 3 before I
had forgotten my own name.
One of the main things I like about module systems in languages like
Perl and Python is the namespacing. If I declare a procedure:
get_address_book
in a module called:
person
then I can use it thusly:
import person
person.get_address_book
or:
import person
person::get_address_book
or:
import person
person->get_address_book
Scheme implementations don't seem to offer this simplicity. I know
many Scheme module systems can make the imported procedure appear to
be in a separate namespace but it's not just done.
I'd just like to know why this is. Why are there such complex systems
as Guile's:
(use-modules ((ice-9 popen)
:select ((open-pipe . pipe-open) close-pipe)
:renamer (symbol-prefix-proc 'unixy:)))
(unixy:pipe-open ...)
Instead of simple ones like Python's:
import os
os.popen(...)
I *think* that the Scheme community, by realizing that modules can be
complicated, is making an error in implementing totally comprehensive
solutions.
--
Nic Ferrier
http://www.tapsellferrier.co.uk
| |
| Andreas Rottmann 2004-10-22, 3:58 pm |
| Nic Ferrier <nferrier@tapsellferrier.co.uk> writes:
> I'd just like to know why this is. Why are there such complex systems
> as Guile's:
>
> (use-modules ((ice-9 popen)
> :select ((open-pipe . pipe-open) close-pipe)
> :renamer (symbol-prefix-proc 'unixy:)))
>
> (unixy:pipe-open ...)
>
> Instead of simple ones like Python's:
>
> import os
>
> os.popen(...)
>
Well, If you always want to do a namespace-like import in Guile, try
this:
(define-macro (import namespace module)
`(use-modules (,module #:renamer (symbol-prefix-proc
(string->symbol
(string-append
(symbol->string ',namespace)
":"))))))
Then you can do:
(import os (ice-9 popen))
(os:pipe-open ...)
That's not that much more complicated than Python, is it?
>
> I *think* that the Scheme community, by realizing that modules can be
> complicated, is making an error in implementing totally comprehensive
> solutions.
>
Well, you can easily layer a "simplified" module system over the
comprehensive solutions, but you can't do the other way round.
Rotty
--
Andreas Rottmann | Rotty@ICQ | 118634484@ICQ | a.rottmann@gmx.at
http://yi.org/rotty | GnuPG Key: http://yi.org/rotty/gpg.asc
Fingerprint | DFB4 4EB4 78A4 5EEE 6219 F228 F92F CFC5 01FD 5B62
Software Patents: Where do you want to stifle inovation today?
| |
|
|
| Nic Ferrier 2004-10-22, 3:58 pm |
| Andreas Rottmann <a.rottmann@gmx.at> writes:
> Well, If you always want to do a namespace-like import in Guile, try
> this:
>
> (define-macro (import namespace module)
> `(use-modules (,module #:renamer (symbol-prefix-proc
> (string->symbol
> (string-append
> (symbol->string ',namespace)
> ":"))))))
Yes. And where do I put this code?
Of course I understand that you can do anything with macros. But the
macro then becomes part of the code and makes things difficult to move
around.
> Well, you can easily layer a "simplified" module system over the
> comprehensive solutions, but you can't do the other way round.
But are these ultra-complicated systems really bringing us anything?
Why does a module system, by default, import into the current
namespace?
--
Nic Ferrier
http://www.tapsellferrier.co.uk
| |
| Shriram Krishnamurthi 2004-10-22, 3:58 pm |
| Nic Ferrier <nferrier@tapsellferrier.co.uk> writes:
> You're probably right.
>
> Like some people say about Art: I don't know much about it but I do
> know what I like.
>
> And I am pretty stupid.
1. I'm not trying to say you're stupid. I'm just making sure you
understand that there is some prior context that you should consider
acquainting yourself with.
2. Stupid can be good. "Stupid" questions like "Why can't your
modules be as simple as Python's?" are valuable, because they force
people to rethink what they've always believed.
> Well, as I say, I'm stupid. But the 2002 paper is about macro
> languages. I think the relevant paper is the 1998 one co-authored with
> Matthias Felleisen:
>
> http://www.ccs.neu.edu/scheme/pubs/pldi98-ff.ps.gz
>
> I have to say, it looks jolly clever but I only got to page 3 before I
> had forgotten my own name.
This paper is silent on the question of macros. And the problem is
that in Scheme, unlike in Python or Perl, you do have to take macros
into account explicitly (even if you only disallow them). That's why
the 2002 paper is relevant: because it explores the question of
modularity in the context of macros.
> I *think* that the Scheme community, by realizing that modules can be
> complicated, is making an error in implementing totally comprehensive
> solutions.
I'm all for simple solutions, too. I think the PLT system is (in its
essence) pretty simple, while also providing support for macros. A
solution so simple that it ignores macros is, I'm afraid, not a
*solution*.
Shriram
| |
| Andreas Rottmann 2004-10-22, 3:58 pm |
| Nic Ferrier <nferrier@tapsellferrier.co.uk> writes:
> Andreas Rottmann <a.rottmann@gmx.at> writes:
>
>
> Yes. And where do I put this code?
>
Hmm, good catch. I'd say in a module, but then you have to use
(use-modules ...) at least once.
> Of course I understand that you can do anything with macros. But the
> macro then becomes part of the code and makes things difficult to move
> around.
>
>
>
> But are these ultra-complicated systems really bringing us anything?
>
I don't think e.g. Guile's module system is ultra-complicated in any
sense. Be default, you just do (use-modules (foo bar)); not that hard
at all.
> Why does a module system, by default, import into the current
> namespace?
>
Because the Scheme language doesn't have namespaces.
Rotty
--
Andreas Rottmann | Rotty@ICQ | 118634484@ICQ | a.rottmann@gmx.at
http://yi.org/rotty | GnuPG Key: http://yi.org/rotty/gpg.asc
Fingerprint | DFB4 4EB4 78A4 5EEE 6219 F228 F92F CFC5 01FD 5B62
Life is a sexually transmitted disease.
| |
| Nic Ferrier 2004-10-22, 8:57 pm |
| Shriram Krishnamurthi <sk@cs.brown.edu> writes:
>
> I'm all for simple solutions, too. I think the PLT system is (in its
> essence) pretty simple, while also providing support for macros. A
> solution so simple that it ignores macros is, I'm afraid, not a
> *solution*.
This is another thing I don't understand. Can someone explain why
macros complicate the module system?
For my scheme impl the module system will be simple, when a module is
(import)ed the interp will:
- find the file containing the module (by scanning a path)
- load and evaluate the module file in a separate (from the main
program) top-level environment
- rename the export symbols into the importing environment and adding
the module name as a prefix to each symbol.
So if we had a module called nic.scm:
;; A module called nic
(defmacro unless (test consequent)
`(if ,test
,consequent))
(define (hello)
(display "hello"))
;; End module
And imported it:
(let ((x 1))
(import nic)
(nic::unless (< x 0) (nic::hello)))
Of course, if you're going to compile this the module has to be
available at compile time.
This is why I'm stupid: this is clearly a big deal for Scheme module
designers but I just don't get it. Why is this a problem?
--
Nic Ferrier
http://www.tapsellferrier.co.uk
| |
| Michele Simionato 2004-10-23, 3:57 am |
| Shriram Krishnamurthi <sk@cs.brown.edu> wrote in message news:<w7dr7nq3cxr.fsf@cs.brown.edu>...
> I'm all for simple solutions, too. I think the PLT system is (in its
> essence) pretty simple, while also providing support for macros. A
> solution so simple that it ignores macros is, I'm afraid, not a
> *solution*.
Well, one of the reasons why I have decided NOT to use PLT scheme was
the module system: it appeared to me cumbersome, not obvious,
over-engineered
and even plain stupid, because *by default* names were imported
without any
prefix. The issue was that I was comparing it with module systems in
other
languages. Nowadays I know more about other module system
implementations
in Scheme and I must concur that the PLT one is actually one of the
best.
The point is that all module systems I have seen in the Scheme world
are cumbersome, not obvious, over-engineered and even plain stupid :-(
Notice that by necessity a scheme module system must be more complex
than a Perl/Python module system: for instance, because of compile
time macros,
a distinction between require and require-for-syntax is needed;
however
this would not be a big deal. What I hate is un-needed complication: I
can use the module system in an easy way if I want, but I am forced to
write a macro for that (look at the Guile example in this thread; I
could
make similar examples for PLT), the job is NOT done by the
implementors for
me. And there is no good reason, IMHO, why things are made more
difficult
than needed, except history. One reason is psycological: schemers are
so
used to curbersome import syntaxes and they do not even realize they
are cumbersome. Another reason is the fragmentation of the community,
so
module systems must be over-engineered in order to be able to work
with different conventions and coding styles :-(
So there is no easy solution and I am not quite convinced that R6RS
will solve
all that. But one can always hope.
Michele Simionato
| |
| Jens Axel Søgaard 2004-10-23, 8:56 am |
| Nic Ferrier wrote:
> Shriram Krishnamurthi <sk@cs.brown.edu> writes:
[color=darkred]
[color=darkred]
> This is another thing I don't understand. Can someone explain why
> macros complicate the module system?=20
Here is my attempt, which I wrote to get a grip on the ideas in Flatt's
paper.
<http://www.scheme.dk/macros-and-modules2.txt>
(note: the example was written before begin-for-syntax went into PLT)
> For my scheme impl the module system will be simple, when a module is
> (import)ed the interp will:
>=20
> - find the file containing the module (by scanning a path)
>=20
> - load and evaluate the module file in a separate (from the main
> program) top-level environment
>=20
> - rename the export symbols into the importing environment and adding
> the module name as a prefix to each symbol.
>=20
> So if we had a module called nic.scm:
>=20
> ;; A module called nic
>=20
> (defmacro unless (test consequent)
> `(if ,test
> ,consequent))
Consider a macro that introduces an identifier into the expansion.
At the place of use of this macro, the identifier refers to the
variable that is in scope at the defition site (i.e. not the
place of the use of the macro).
Things an "ideal" module has solutions for:
What happens if the macro is defined and used in different files?
What happens if a compiler uses separate compilation?
What happens in interactive use?
Are the behaviours under interactive use and under compilation the same?
> Of course, if you're going to compile this the module has to be
> available at compile time.
Yes.
What happens if the definition of the macro it self uses another
macro from a third module?
What happens if the same module is used by both the "normal" code
and the "macro defining" code? Are variables shared? (easy for
interpreters / hard for separate compilation)
> This is why I'm stupid: this is clearly a big deal for Scheme module
> designers but I just don't get it. Why is this a problem?
It is hard because there is so many different Scheme implementations.
Designing a module system for an interpreter only is easier than
designing one that also works for separate compilation.
Designing a module system for a compiler with no macros is easier
than designing one that works with macros.
--=20
Jens Axel S=F8gaard
| |
| Marcin 'Qrczak' Kowalczyk 2004-10-23, 8:56 am |
| Nic Ferrier <nferrier@tapsellferrier.co.uk> writes:
> I'd just like to know why this is. Why are there such complex systems
> as Guile's:
>
> (use-modules ((ice-9 popen)
> :select ((open-pipe . pipe-open) close-pipe)
> :renamer (symbol-prefix-proc 'unixy:)))
>
> (unixy:pipe-open ...)
>
> Instead of simple ones like Python's:
>
> import os
>
> os.popen(...)
Because os.popen doesn't fit the traditional sexpr syntax. It would
have to be something like (get os popen), which would look ugly.
Common Lisp has lexical support for qualified names. Python doesn't
need one, it just reuses the syntax for field access. Sexprs don't
have a good syntax for field access.
(I don't like sexprs.)
(I don't like mandatory qualifications of imported names either.
Haskell's module system is more pleasant: names can be used unqualified
too, which is the norm, and conflicts are reported lazily at compile
time.)
--
__("< Marcin Kowalczyk
\__/ qrczak@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
| |
| Nic Ferrier 2004-10-23, 3:56 pm |
| Jens Axel Søgaard <usenet@soegaard.net> writes:
> Consider a macro that introduces an identifier into the expansion.
> At the place of use of this macro, the identifier refers to the
> variable that is in scope at the defition site (i.e. not the
> place of the use of the macro).
Fine. The macro therefore takes the value from the module's
environment, which is clean.
> Things an "ideal" module has solutions for:
>
> What happens if the macro is defined and used in different files?
It's different macros.
> What happens if a compiler uses separate compilation?
You can't compile macros, only the code that they expand to.
Nic said:
Jens replied:[color=darkred]
> Yes.
>
> What happens if the definition of the macro it self uses another
> macro from a third module?
Then you need that available as well (in order to compile).
In other words, I still don't really get why this is a big
deal. Macros complicate the system but surely they just require that
all the macro source be available at macro expansion time. If it isn't
then there's no way the macros are going to be able to be expanded.
I'm sure I'm just being dumb and missing something fundamental.
--
Nic Ferrier
http://www.tapsellferrier.co.uk
| |
| William D Clinger 2004-10-23, 3:56 pm |
| Nic Ferrier <nferrier@tapsellferrier.co.uk> wrote:
> I'm sure I'm just being dumb and missing something fundamental.
You aren't being dumb, but I do think you're missing something
fundamental: The semantics of a hygienic macro depends upon
the context in which it was defined. With a module system, a
macro that's defined in one module depends upon the definitions
that appear within that module (not just the macro definitions)
and upon the names that are imported into that module. It's as
though the exported macro implicitly closes over all names that
it could possibly insert during transcription. Some of these
names may themselves denote macros, so there's a transitive
closure at work.
With SYNTAX-CASE, and with other macro-defining mechanisms that
allow the macro expander to call program-specific Scheme code,
the modules must be evaluated at compile time to initialize the
variables to which the macro expander refers. This sets up a
transitive closure of compile-time evaluation.
Most of us want the module system to support some notion of
separate compilation and some degree of static analysis as well,
which constrains the design.
It isn't rocket science, but it's a little more complex than
most people would expect.
Will
| |
| Nic Ferrier 2004-10-23, 3:56 pm |
| cesuraSPAM@verizon.net (William D Clinger) writes:
> Nic Ferrier <nferrier@tapsellferrier.co.uk> wrote:
>
> You aren't being dumb, but I do think you're missing something
> fundamental: The semantics of a hygienic macro depends upon
> the context in which it was defined. With a module system, a
> macro that's defined in one module depends upon the definitions
> that appear within that module (not just the macro definitions)
> and upon the names that are imported into that module. It's as
> though the exported macro implicitly closes over all names that
> it could possibly insert during transcription. Some of these
> names may themselves denote macros, so there's a transitive
> closure at work.
I think I understand that. What I think I don't understand is how you
can ever hope to avoid having the source for the macros available at
compile time.
Macros transform syntactically correct Scheme source expressions into
syntactically correct scheme code acceptable to a translator
implementation.
Macros must be expanded at compile time.
Macros are defined in source files (possibly modular).
Therefore source files of macros must be available at compile
time for all macros. Separate compilation where there is a macro
dependancy tree isn't possible.
Clearly that statement is wrong, but I find the notion that it's wrong
quite befuddling.
> With SYNTAX-CASE, and with other macro-defining mechanisms that
> allow the macro expander to call program-specific Scheme code,
> the modules must be evaluated at compile time to initialize the
> variables to which the macro expander refers. This sets up a
> transitive closure of compile-time evaluation.
Yes.
> Most of us want the module system to support some notion of
> separate compilation and some degree of static analysis as well,
> which constrains the design.
The previous point and this one surely don't go together.
> It isn't rocket science, but it's a little more complex than
> most people would expect.
I think I understand the issue, but not the proposed solution. How can
you have a macro system without evaluating the source code at compile
time? Separate compilation just can't be done.
--
Nic Ferrier
http://www.tapsellferrier.co.uk
| |
| David Golden 2004-10-23, 3:56 pm |
| Nic Ferrier wrote:
>
> I can't see any reason why LISPs should not have namespace separation
> embedded in their module systems.
>
> Is there a reason why?
>
Well, arguably some things people call module systems and namespaces can
usefully be separate features. Agreement on what exactly module systems
are anyway can sometimes be a problem.
e.g. Common Lisp packages are namespaces, working over symbols.
"Locales" as described by Erann Gat^W^WRon Garret could be a useful
common lisp module system independently of common lisp packages,
working over the mappings of symbols to values:
http://www.flownet.com/gat/locales.pdf
(I would actually favour naming them "modules", or at least not
"locales", to avoid confusion with l10n...)
| |
| Ray Dillinger 2004-10-24, 3:56 am |
| Nic Ferrier wrote:
> Macros must be expanded at compile time.
>
> Macros are defined in source files (possibly modular).
>
> Therefore source files of macros must be available at compile
> time for all macros. Separate compilation where there is a macro
> dependancy tree isn't possible.
>
> Clearly that statement is wrong, but I find the notion that it's wrong
> quite befuddling.
>
> <snip>
>
> I think I understand the issue, but not the proposed solution. How can
> you have a macro system without evaluating the source code at compile
> time? Separate compilation just can't be done.
You are not wrong. This is true. And this is the main reason
why I have concluded that scheme is using the wrong macrology.
Traditional Lisp macros take list structure and transform it into
other list structure. They can be separately compiled, and unlike
scheme/cl macros, they're first-class entities that can be returned
from functions, stored in structures, etc. The list structure is
returned, interpreted as source code, and executed.
But then you have compilation and linking overhead at runtime,
which is why traditional LISPs were regarded as very slow.
Still, a canny implementation of it, particularly in the case
where the macro is given constant expressions to expand, can
do the overhead exactly once in a program run. And once you've
done that, you'll note that in most cases, if you're willing to
give up the first-class-ness of the macros, you can do it at
compilation (though not really in separate compilation).
Common Lisp and Scheme both push this to compile-time, an
approach which cannot be used when giving macros non-constant
data or in separate compilation.
It's my opinion that macros demand runtime expansion, compilation,
and linking in order for the full power of a LISP to be
expressible. But most of the time we don't need the full power
of a LISP. Somewhere inbetween full-on LISP macrology and
un-macro'd source code, there's a slightly less expressive
convention that can fulfill our needs most of the time, such
as scheme's define-syntax or CL's defmacro forms. My own
toy lisp uses "mu functions," an invention that can put
separately-compiled executable code in most, but not all, of
the places where scheme/CL use macros.
If you aim too high with this less expressive macrology, as
both Scheme and CL have had to do since they gave up the
ultimately-powerful traditional LISP macrology and have had
to replace it with something almost as good, you can succeed
in pushing expansion/compilation away from runtime but fail
to get separate compilation. If you aim lower (as I did with
mu functions) you get separate compilation, but expressiveness
is too limited to replace the awesome power of traditional
LISP macrology.
So, I think a good LISP needs both. There should be a
macro-like functionality that can be separately compiled and
has limited expressiveness - which would be too limited in
the event that it were all there was. And you also need the
old-school, full-on LISP macrology that demands runtime
expansion and compilation, to take up the slack and provide
the full power of LISP.
Note: several times I've mentioned "mu functions." Mu
functions are first-class entities like lambda functions,
except that they inherit dynamic rather than lexical scope
and use lazy rather than eager argument evaluation. They
can be used in about 95% of the places where modern LISPs
use macros, and they are separately compilable.
Bear
| |
| William D Clinger 2004-10-24, 3:56 am |
| Nic Ferrier <nferrier@tapsellferrier.co.uk> wrote:
> I think I understand the issue, but not the proposed solution. How can
> you have a macro system without evaluating the source code at compile
> time? Separate compilation just can't be done.
Two comments.
Suppose a program consists of modules X, Y, and Z, which may
themselves depend upon other libraries and modules M1 through
Mk. If module X depends only upon M1 through Mk, then you
might be able to compile X by looking at M1 through Mk; you
might not have to have the source code for Y and Z, even if
modules Y and Z import from X. It depends on how you design
your module system, which depends in turn on what you're
trying to accomplish with your design.
Secondly, if you knew that all of the macros that X imports
from M1 through Mk were imported from M1, then you might be
able to compile X by looking only at M1, even though those
macros may refer to variables defined in M2 through Mk. It
depends on how you design your module system, which depends
in turn on what you're trying to accomplish with your design.
I think the problem isn't one of technical difficulty so much
as a problem of evaluating the design goals and tradeoffs.
The difficulty of that can be appreciated by reading all the
stuff people have had to say about order of evaluation, where
there is as yet no consensus on the nature of the tradeoffs,
let alone a consensus on how to resolve the tradeoffs.
Will
| |
| Per Bothner 2004-10-24, 3:56 am |
| Nic Ferrier wrote:
> Macros transform syntactically correct Scheme source expressions into
> syntactically correct scheme code acceptable to a translator
> implementation.
>
> Macros must be expanded at compile time.
>
> Macros are defined in source files (possibly modular).
>
> Therefore source files of macros must be available at compile
> time for all macros. Separate compilation where there is a macro
> dependancy tree isn't possible.
A macro is a source->source mapping/transformation. With
syntax-rules that transformation can be compiled into a data
structure encoding the pattern+template; with syntax-case the
transformation is a general function - which can likewise be
compiled.
Expanding a macro is just a matter of apply a function.
Separate compilation works just fine. Kawa has done separate
compilation of syntax-rules for years, and I'm getting close
to having separate compilation of syntax-case working.
Kawa treats each source file as a separate module, and you
can specify which top-level forms are exported so they can
be imported into other modules. So what happens if an exported
macro references a non-exported value (variable/function/macro)?
In that case the latter value is marked "hidden": it is ignored
when the module is imported, but it is available when the
macro is expanded.
--
--Per Bothner
per@bothner.com http://per.bothner.com/
| |
| Ray Dillinger 2004-10-24, 3:57 pm |
| Per Bothner wrote:
> Nic Ferrier wrote:
>
>
>
> A macro is a source->source mapping/transformation. With
> syntax-rules that transformation can be compiled into a data
> structure encoding the pattern+template; with syntax-case the
> transformation is a general function - which can likewise be
> compiled.
>
> Expanding a macro is just a matter of apply a function.
> Separate compilation works just fine. Kawa has done separate
> compilation of syntax-rules for years, and I'm getting close
> to having separate compilation of syntax-case working.
I don't think you are talking about the same "separate
compilation" that he is talking about. He is absolutely
right that it is flatly impossible to macroexpand and
compile, in the absence of the macro definition.
I have seen several people use "separate compilation" as
a term for some kind of compilation that still requires
the source code for macro definitions in other files
to be available at compile time, or implements macros
in a way that pushes source-code structure and
compilation overhead into runtime, for example by
implementing in terms of an old-style lisp macrology
that directly manipulates source code and compiles
and executes (or just interprets) the resulting
expressions during runtime.
But people whose background is in other languages
expect "separate compilation" to mean avoiding both
of those cases.
Bear
| |
| Per Bothner 2004-10-24, 8:56 pm |
| Ray Dillinger wrote:
> Per Bothner wrote:
>
>
> I don't think you are talking about the same "separate
> compilation" that he is talking about.
Quite possibly. But I'm not getting what the problem is.
> He is absolutely
> right that it is flatly impossible to macroexpand and
> compile, in the absence of the macro definition.
But you don't need the *source code* for the macro definition.
You need a *compiled form* of the macro definition.
To me this seems to be no different than calling a function:
you don't need the source code for a function, but you need
its compiled form to call it.
In Kawa, a syntax-rules transformer is compiled into a data
structure that is very different from the source s-expression.
> I have seen several people use "separate compilation" as
> a term for some kind of compilation that still requires
> the source code for macro definitions in other files
> to be available at compile time,
Not needed in Kawa.
> or implements macros
> in a way that pushes source-code structure and
> compilation overhead into runtime, for example by
> implementing in terms of an old-style lisp macrology
> that directly manipulates source code and compiles
> and executes (or just interprets) the resulting
> expressions during runtime.
Not in Kawa.
> But people whose background is in other languages
> expect "separate compilation" to mean avoiding both
> of those cases.
I agree.
--
--Per Bothner
per@bothner.com http://per.bothner.com/
| |
| Ray Dillinger 2004-10-24, 8:56 pm |
| Per Bothner wrote:
> Ray Dillinger wrote:
>
>
>
> But you don't need the *source code* for the macro definition.
> You need a *compiled form* of the macro definition.
>
Ah. Here's the rub. You're using "separate compilation"
to mean compilation of files but only in the presence of
information that conveys the macro definitions from other
files. Maybe not necessarily the source code for the
macros, but it fails as separate compilation anyway,
since I can't compile a file separately (meaning, in the
absence of information from other files).
> To me this seems to be no different than calling a function:
> you don't need the source code for a function, but you need
> its compiled form to call it.
You need its compiled form to call it (at runtime, after
linking) but you don't need any form of it (source or compiled)
to compile something that calls it. That's the fundamental
difference.
Compile something in a way that doesn't need to see macro
definitions from another file, in any form, but uses those
macros when linked, and then you'll have cracked the nut.
Bear
| |
| Per Bothner 2004-10-25, 3:59 am |
| Ray Dillinger wrote:
> Ah. Here's the rub. You're using "separate compilation"
> to mean compilation of files but only in the presence of
> information that conveys the macro definitions from other
> files. Maybe not necessarily the source code for the
> macros, but it fails as separate compilation anyway,
> since I can't compile a file separately (meaning, in the
> absence of information from other files).
My attitude is I don't see why you'd ever want that ...
> Compile something in a way that doesn't need to see macro
> definitions from another file, in any form, but uses those
> macros when linked, and then you'll have cracked the nut.
But compiling something in the *same* file requires seeing
macro definions *before* code that uses the macro. Compiling
a whole program or a single module makes no essential difference.
Because Scheme has no reserved identifiers, you have see
macros before you can do anything else, at least at top level.
Trying to defer macro expansion to link time makes compilation
meaningless, since you can't do much more than reading.
Which I guess was the point people were making, but I think
they're asking for something that isn't desirable or useful.
--
--Per Bothner
per@bothner.com http://per.bothner.com/
| |
| Jens Axel Søgaard 2004-10-25, 4:02 pm |
| Per Bothner wrote:
> Ray Dillinger wrote:
>=20
>=20
> My attitude is I don't see why you'd ever want that ...
You are probably smiling as you write that, since you
know that the C people use separate compilation to keep
compile time down.
Recompiling a whole Scheme project, because I make a change
on one module, which isn't included anywhere doesn't make
much sense - unless it is neccessary.
[color=darkred]
> But compiling something in the *same* file requires seeing
> macro definions *before* code that uses the macro. Compiling
> a whole program or a single module makes no essential difference.
> Because Scheme has no reserved identifiers, you have see
> macros before you can do anything else, at least at top level.
> Trying to defer macro expansion to link time makes compilation
> meaningless, since you can't do much more than reading.
Since we are talking about properties for an "ideal" module
system, one is allowed to assume that the code is written
at the module level and not on the top level. Bending the
rules on order to achieve separate compilation is fine
with me.
But you and Ray do rise the issue whether what we (at least I)
want in terms of separate compilation may not be possible.
I am inclined to believe that you both are right.
> Which I guess was the point people were making, but I think
> they're asking for something that isn't desirable or useful.
Lowering compile times would be desirable and useful in my book.
--=20
Jens Axel S=F8gaard
| |
| Per Bothner 2004-10-27, 3:57 am |
| Jens Axel Søgaard wrote:
> Per Bothner wrote:
>
>
> You are probably smiling as you write that, since you
> know that the C people use separate compilation to keep
> compile time down.
What I meant was: I don't see why you ever want to compile
a file seperately, in the absence of information from other files.
I.e. given a module B that imports (macros, say) from module A.
Is it useful to be able to compile B without having access to A
or the compiled result of A? I don't think so.
--
--Per Bothner
per@bothner.com http://per.bothner.com/
| |
| Ray Dillinger 2004-10-27, 3:57 am |
| Per Bothner wrote:
> I.e. given a module B that imports (macros, say) from module A.
> Is it useful to be able to compile B without having access to A
> or the compiled result of A? I don't think so.
It's not so much that I want to be able to compile B in the
absence of macro information from A; it's that I don't want to
be forced to recompile B every time I touch A.
Scheme macros (and Lisps with code-transforming macros in
general) give rise to unbelievably hairy mutual-dependency
graphs.
These make it nearly impossible to maintain a makefile for a
very large program (on the order of millions of lines - for
example, an operating system), and, as a practical matter, even
when the dependency graph can be computed, cause recompilation
of very large amounts of code in the event of even relatively
trivial changes.
I don't want to commit my CPU for three days recompiling the
*&$% world just because I change some trivial function somewhere
that a code-transforming macro happens to call, so Lisps remain
exactly the wrong languages to use for very large projects.
This is frustrating, because otherwise I like Lisps very much.
Some may say that this is a nonissue because nobody is using
Lisps for such projects: I will respond by saying to them that
this is one of the reasons *why* nobody can use Lisps for such
projects.
The use of code-transforming macros, which require recompilation
of every call site in the event of changes to them or anything
they depend on, make Lisps that use them fail to scale to large
projects.
Use of linkable forms such as function calls does not have this
penalty: I can change the definition of a function and it will
not force recompilation of every file containing a call to that
function. At the call site, the caller is still evaluating the
arguments and function, putting the arguments into a call frame,
storing a return address, jumping to the function, and resuming
after the return address by reading the result out of the return
register. Even if the code at the function definition site changes,
the code at the function call site doesn't change and I don't have
to recompile.
Therefore, if we take the needs of software engineers with large
projects seriously, we should be looking for linkable forms that
allow most of the idioms we now use macros to accomplish. I don't
say we should get rid of code transforming macros completely; the
general expressiveness they allow is probably the single most
powerful strength of Lisps. But they're the tactical nukes of
the language family; they should not be used to kill bunnyrabbits.
Unless your world is so small that you don't care whether you must
destroy and rebuild it over and over, the collateral damage is
unacceptable.
Bear
| |
| George Neuner 2004-10-27, 3:57 am |
| On Wed, 27 Oct 2004 01:17:16 GMT, Per Bothner <per@bothner.com> wrote:
>I.e. given a module B that imports (macros, say) from module A.
>Is it useful to be able to compile B without having access to A
>or the compiled result of A? I don't think so.
It's definately useful not to require source to compile a dependent
module - particularly in large systems and when you are using third
party components and don't have access to source.
Java may suck as a language but one advantage it does have for
development in the large is that you don't need the source for a class
to compile new code which uses the class. All you need is the
reflection type and linkage information contained in the (binary
deliverable) class file.
Now, of course, this works because Java doesn't have macros and does
have reflection. I know many Schemers will want to fight to the death
to keep macros but I've personally never had much use for them
(complicated ones anyway). I'm intrigued by Ray's experience using mu
functions. If the result was a Scheme that could separately compile
in the same manner as Java, I would drop macros in a heartbeat.
ducking for cover ...
George
--
for email reply remove "/" from address
| |
| Daniel C. Wang 2004-10-27, 3:57 am |
| George Neuner wrote:
{stuff deleted}
> Java may suck as a language but one advantage it does have for
> development in the large is that you don't need the source for a class
> to compile new code which uses the class. All you need is the
> reflection type and linkage information contained in the (binary
> deliverable) class file.
For all practical purposes a Java class file is the source code, at least
from a native-code compiler's standpoint. If you want an incremental Scheme
system with macros, you need not change the Scheme language. You just need
the moral equivlant of a Scheme/JIT compiler that dynamically loads and
expands macros in a on demand fashion.
Reflection has nothing to do with separate compilation, and is a non-issue
in a uni-typed language.
| |
| Ray Dillinger 2004-10-27, 3:57 am |
| George Neuner wrote:
> Now, of course, this works because Java doesn't have macros and does
> have reflection. I know many Schemers will want to fight to the death
> to keep macros but I've personally never had much use for them
> (complicated ones anyway). I'm intrigued by Ray's experience using mu
> functions. If the result was a Scheme that could separately compile
> in the same manner as Java, I would drop macros in a heartbeat.
About mu functions:
I do have to know at the call site whether it's a standard
function or a mu function or a macro, because calls to lambda
functions, mu functions, and macros do different things. So
*fully* separate compilation isn't quite there.
What I have to do is define an "export" section for each module
that says whether a given exported name is a lambda function, a mu
function, or a macro. If a function (not a macro) has a definite
number of arguments, and I call it a lot, I can also say how
many arguments it has and avoid consing (a performance hack,
otherwise only available in the same module). The good part is,
as long as the export information doesn't change, changes in
the implementation do not require recompilation of stuff that
calls them.
I'm using interpreted macros, which are a linkable form but a
major performance hit. A call to an interpreted macro consists of
calling a (compiled) function with one argument - the list structure
of things passed to it - getting back (uncompiled) list structure,
calling the interpreter with the returned list structure as an
argument, and then continuing. The interpreter, unfortunately, is
slower than java, but still, I don't have to recompile a file
containing a macro call when I change the macro.
Mu functions are another linkable form; the arguments to mu
functions have to be scheme expressions in their own right (variable
references, or function or macro calls - not arbitrary list
structure, as used by macros). The compiler produces code to
evaluate these expressions (complete with correct references to the
local (call site) lexical environment), and it is pointers to
this code, rather than the results returned from calling it, that
is passed as arguments to the mu function.
And finally lambda functions; The arguments have to be scheme
expressions in their own right, as for mu functions; the compiler
produces code for evaluating these expressions, and at the call site
this code is called and the results stored as arguments to the lambda
function before the lambda function is called. This makes a third
linkable form.
So, with the caveat that macros are expanded and run (slowly!)
at runtime rather than compile time, and the caveat that I have to
declare which names have which general type of implementation, I
do get the ability to avoid recompiling every time the implementation
of a given name changes, provided it doesn't change general type.
My current front-burner project is memoizing and compiling the
results of macroexpansion at runtime, so that macros whose arguments
don't change (which is typical of source code) should only be
expanded and compiled once during each runtime and treated as
mu functions thereafter.
Bear
| |
| Per Bothner 2004-10-27, 3:57 am |
| George Neuner wrote:
> Java may suck as a language but one advantage it does have for
> development in the large is that you don't need the source for a class
> to compile new code which uses the class. All you need is the
> reflection type and linkage information contained in the (binary
> deliverable) class file.
>
> Now, of course, this works because Java doesn't have macros and does
> have reflection. ... If the result was a Scheme that could separately compile
> in the same manner as Java, I would drop macros in a heartbeat.
You don't have to: Kawa compiles a module, including macros, to JVM
class files. To compile a module B.scm that "imports" from A (including
possibly macros) you need A.class, but not A.scm.
--
--Per Bothner
per@bothner.com http://per.bothner.com/
| |
| Ray Dillinger 2004-10-27, 8:57 am |
| Per Bothner wrote:
> George Neuner wrote:
>
>
> You don't have to: Kawa compiles a module, including macros, to JVM
> class files. To compile a module B.scm that "imports" from A (including
> possibly macros) you need A.class, but not A.scm.
My problem with that is that if A changes, B has to be recompiled.
Bear
| |
| Marcin 'Qrczak' Kowalczyk 2004-10-27, 8:57 am |
| Ray Dillinger <bear@sonic.net> writes:
> It's not so much that I want to be able to compile B in the
> absence of macro information from A; it's that I don't want to
> be forced to recompile B every time I touch A.
Everything that other modules depend on at their compilation time
should be extracted to a separate file by the compiler during
processing of A.
The file should be left with its old modification time if it would be
overwritten with unchanged contents.
That's all. Other modules will not be recompiled until they have to.
Well, with a reasonable approximation.
Glasgow Haskell has some more elaborate scheme, without physical
inclusion of indirect dependences and propagating automatic version
numbers of individual definitions, but I'm not convinced that such
complexity is needed.
--
__("< Marcin Kowalczyk
\__/ qrczak@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
| |
| Tony Garnock-Jones 2004-10-27, 8:57 am |
| Ray Dillinger wrote:
> I don't want to commit my CPU for three days recompiling the
> *&$% world just because I change some trivial function somewhere
> that a code-transforming macro happens to call, so Lisps remain
> exactly the wrong languages to use for very large projects.
C and C++ suffer from the same problem, caused by both the preprocessor
macro system and C++'s function-inlining capability.
> The use of code-transforming macros, which require recompilation
> of every call site in the event of changes to them or anything
> they depend on, make Lisps that use them fail to scale to large
> projects.
>
> Use of linkable forms [...]
>
> [...] we should be looking for linkable forms that
> allow most of the idioms we now use macros to accomplish.
It is interesting to consider Self here. Self achieves remarkable
performance and yet is completely dynamic - applications are linked at
runtime, and can be reconfigured at runtime. (In fact, applications are
linked at function *call* time - the last possible moment!) The
underlying VM re-optimises the JITted code based on the changed usage of
the system.
Some kind of Self-like runtime caching and optimisation applied to a
late-bound code transformation facility could give you what you're
looking for.
(There might still be a need for staged evaluation as provided by
MzScheme's module system, of course - but in a fully dynamic environment
like Self the distinction between compile-time and runtime gets a bit
blurry anyway...)
Tony
| |
| Tony Garnock-Jones 2004-10-27, 8:57 am |
| Daniel C. Wang wrote:
> Reflection has nothing to do with separate compilation, and is a
> non-issue in a uni-typed language.
It's funny you should say this. Macros *are* reflective structures.
Reflection and a language's type system are more-or-less unrelated.
SmallTalk is reflective; Scheme is (in a limited way) reflective; and
recently, Haskell is becoming reflective (http://www.haskell.org/th/,
http://www.cs.vu.nl/boilerplate/).
Tony
| |
| Joe Marshall 2004-10-27, 3:59 pm |
|
One of my former jobs was with a company that did configuration
management. We wrote tools that dealt with complex build processes.
(The subject matter is mind-numbingly tedious: the goal was to enable
the user to click on an icon and rebuild the entire system, and have
*nothing* unusual or interesting happen at all. The engineering
required to maintain this level of dullness was itself quite
interesting.)
Ray Dillinger <bear@sonic.net> writes:
> Scheme macros (and Lisps with code-transforming macros in
> general) give rise to unbelievably hairy mutual-dependency
> graphs.
They certainly provide the capability to do so, and people often use
this capability, but it isn't the fault of the macros. For example,
this problem arises when there are large, complex, static structures
that need to be created and initialized at startup time.
> These make it nearly impossible to maintain a makefile for a
> very large program (on the order of millions of lines - for
> example, an operating system), and, as a practical matter, even
> when the dependency graph can be computed, cause recompilation
> of very large amounts of code in the event of even relatively
> trivial changes.
The dependency graph can *always* be computed (if your large system
has never run at all, the issue of dependencies are moot. If your
large system ever ran, then the dependencies are no more complex than
the entire history of what you did up to the point of running it.)
Spaghetti dependency graphs don't pop into existence out of the
ether. They arise as a result of neglect. One needs to apply some
thought at all levels of design. This is one place that is often
overlooked, and it takes a bit of practice to become good at it.
Macros are used in two different ways. `Local' uses of macros are
great for removing the minutae from repetitive forms:
(let-syntax ((define-iso-language-code ....))
(define-iso-language-code AA "Afar")
(define-iso-language-code AB "Abkhazian")
(define-iso-language-code AF "Afrikaans")
(define-iso-language-code AM "Amharic")
...)
These don't generally cause configuration problems.
The other primary use is to make fundamental changes to the syntax of
the language. The Common Lisp `LOOP' macro is an example. These
sorts of macros are the kind that cause the entire world to be
recompiled.
Non-local macros change the language in which you are programming.
This isn't something one should do lightly, nor is it something that
one should spread out incrementally in several loosely related files.
If you are going to change the language, you should *design* it as a
new language and *implement* it as a new language rather than stumble
across it ex-post-facto when your build breaks.
> I don't want to commit my CPU for three days recompiling the
> *&$% world just because I change some trivial function somewhere
> that a code-transforming macro happens to call, so Lisps remain
> exactly the wrong languages to use for very large projects.
> This is frustrating, because otherwise I like Lisps very much.
We actually found lisp to be much *better* than other languages in
this regard because Lisp has tools that other languages don't provide.
One of the nicest is the ability to `dump a world'. Part of the build
process is to start with a `virgin Lisp' from the vendor and to load
the package system changes (it was Common Lisp), the
language-transforming macros, and a few very low-level utilities.
We'd then dump this world as a `base development image'. In the
normal course of development, we'd start with a base development image
and hack away loading stuff, etc.
Now suppose a developer decides that one of these low-level macros
needs a change. There are no surprises here: this will require that
*everything* gets recompiled. (The fact is if you change something
fundamental, you are in for some serious recompilation. I don't think
anyone objects to this, it is the issue of changing something
*trivial* and then discovering afterwards how much work that causes.)
Can someone accidentally create a circular dependency? It's difficult
because the low-level macros are clearly built in the `virgin' Lisp.
Macros extend the compiler. It is, in general, possible to do
arbitrary work in a macro --- open a database, run a web browser, play
space-war. But it is, in general, *unnecessary* to do much more than
use CAR and CDR to perform code rewrites. If your macro is doing
nothing more than re-arranging your code and adding a bit of its own
code for later execution, then it doesn't need the target support code
to be runnable at expansion time. The macro may transform your code
to call the ODBC layer, but that doesn't mean that the macro needs
that layer during the rewrite phase. So the transformations for the
high-level code can themselves be placed in the virgin lisp.
> Some may say that this is a nonissue because nobody is using
> Lisps for such projects: I will respond by saying to them that
> this is one of the reasons *why* nobody can use Lisps for such
> projects.
You can (and should) use Lisp for such large projects.
> The use of code-transforming macros, which require recompilation
> of every call site in the event of changes to them or anything
> they depend on, make Lisps that use them fail to scale to large
> projects.
Yep. That they do. That's why we avoided writing new ones, or
avoided changing the existing ones. Nonetheless, we used things like
the SERIES macro package (very pervasive code transformation) and
rolled our own replacement macros (for instance, we replaced the usual
COND macro with one that bitched and moaned if it didn't have an
`else' clause. Caught more than a few bugs with that!). When we
needed a change to these macros, we were not taken by surprise that we
needed to recompile everything.
> Therefore, if we take the needs of software engineers with large
> projects seriously, we should be looking for linkable forms that
> allow most of the idioms we now use macros to accomplish. I don't
> say we should get rid of code transforming macros completely; the
> general expressiveness they allow is probably the single most
> powerful strength of Lisps. But they're the tactical nukes of
> the language family; they should not be used to kill bunnyrabbits.
Exacly. That's why we locked them up and put warning signs all over
them.
| |
| Daniel C. Wang 2004-10-27, 3:59 pm |
| Let me restate things.
The Java reflection API exists to circumvent the type system of Java.
In a uni-typed language like Scheme there is no need for a system like
Java's reflection API.
There are other notions of "reflection" i.e. meta-programming which you
speak, but by definition any Turing Complete language is reflective.
The question is about provideding better support for meta-programming, but I
can preform meta-programming in any Turing Coomplete language.
Tony Garnock-Jones wrote:
> Daniel C. Wang wrote:
>
>
>
> It's funny you should say this. Macros *are* reflective structures.
> Reflection and a language's type system are more-or-less unrelated.
> SmallTalk is reflective; Scheme is (in a limited way) reflective; and
> recently, Haskell is becoming reflective (http://www.haskell.org/th/,
> http://www.cs.vu.nl/boilerplate/).
>
> Tony
| |
| Bradd W. Szonye 2004-10-27, 3:59 pm |
| Ray Dillinger <bear@sonic.net> wrote:
> Per Bothner wrote:
>
> My problem with that is that if A changes, B has to be recompiled.
Not necessarily, if your dependency checker is sophisticated enough. I
think a large chunk of the separate compilation "problem" exists only
because 'make' is naive. It works very well for manually-generated
header files, but very poorly with automatic exports. That's
unfortunate, because auto-exporting can avoid many of the annoying
rebuild scenarios (i.e., a white space change forces a full rebuild).
--
Bradd W. Szonye
http://www.szonye.com/bradd
| |
| Per Bothner 2004-10-27, 3:59 pm |
| Daniel C. Wang wrote:
> The Java reflection API exists to circumvent the type system of Java.
Nonsense. The reflection API is to support various tools such as
IDEs and tree-walking (such as for serialization).
> In a uni-typed language like Scheme there is no need for a system like
> Java's reflection API.
No, the reason Scheme doesn't need a reflection API is because
it doesn't have structures, objects, or user-defined types.
Scheme being "uni-typed" (or rather dynamically typed) is completely
irrelevant to reflection, which is used to examine the *dynamic* types
of values.
A purely statcially typed language would not need dynamic reflection.
--
--Per Bothner
per@bothner.com http://per.bothner.com/
| |
| Daniel C. Wang 2004-10-27, 8:57 pm |
| Per Bothner wrote:
> Daniel C. Wang wrote:
>
>
>
> Nonsense. The reflection API is to support various tools such as
> IDEs and tree-walking (such as for serialization).
Which needs to circumvent the type system by turning a statically typed
objects into a uni-typed representation.
>
>
> No, the reason Scheme doesn't need a reflection API is because
> it doesn't have structures, objects, or user-defined types.
>
> Scheme being "uni-typed" (or rather dynamically typed) is completely
> irrelevant to reflection, which is used to examine the *dynamic* types
> of values.
Oh, so you'd be happier if, I said Scheme's lack of data-abstraction makes
the need for a reflection API unecessary. (Oh, I guess this is not true,
because of closures.. but it's close to true...)
> A purely statcially typed language would not need dynamic reflection.
huh? a purely statically typed language does need "dynamic reflection" if it
wishes to implement "generic" tree-walking and IDEs that need to
"circumvent" the static type system.
| |
| George Neuner 2004-10-27, 8:57 pm |
| On Wed, 27 Oct 2004 00:26:04 -0400, "Daniel C. Wang"
<danwang74@gmail.com> wrote:
>George Neuner wrote:
>{stuff deleted}
>
>For all practical purposes a Java class file is the source code, at least
>from a native-code compiler's standpoint. If you want an incremental Scheme
>system with macros, you need not change the Scheme language. You just need
>the moral equivlant of a Scheme/JIT compiler that dynamically loads and
>expands macros in a on demand fashion.
I completely disagree that the type information in the class file
qualifies as "source". It doesn't tell you anything about the
implementation behind the interface.
If you're going to argue that bytecodes are equivalent to source, then
so too is the disassembly of any deliverable in any language.
>Reflection has nothing to do with separate compilation, and is a non-issue
>in a uni-typed language.
I agree that reflection has nothing to do with separate compilation
per se ... perhaps my post was badly worded. But the inclusion of
type information within the deliverable is what allows Java to perform
both reflection and separate compilation without source.
George
--
for email reply remove "/" from address
| |
| Daniel C. Wang 2004-10-28, 3:57 am |
| George Neuner wrote:
{stuff deleted}
>
> I completely disagree that the type information in the class file
> qualifies as "source". It doesn't tell you anything about the
> implementation behind the interface.
>
> If you're going to argue that bytecodes are equivalent to source, then
> so too is the disassembly of any deliverable in any language.
Decompilers for native binaries to C are far less useful than Java
decompilers, in fact there are comerical Java obfuscators to protect
intellectual property shipped as class files.
I'm not going to "argue" about anything. I'm just stating the facts. A class
file is for the purposes of compilation and optimization the same as source
code.
| |
| George Neuner 2004-10-28, 3:57 am |
| On Wed, 27 Oct 2004 06:23:38 GMT, Per Bothner <per@bothner.com> wrote:
>George Neuner wrote:
>
>
>You don't have to: Kawa compiles a module, including macros, to JVM
>class files. To compile a module B.scm that "imports" from A (including
>possibly macros) you need A.class, but not A.scm.
I haven't used Kawa, but since you are the author I'll take your word
for it 8-)
I couldn't easily find any detail on your web site that describes how
Scheme macros are embedded in the class file. I can imagine a couple
of ways to do it: e.g., by storing the macro definition as a string
in the constant pool or creating a parser function that the compiler
can dynamically invoke on the dependent source.
However you do it, I suspect you are taking advantage of Java specific
capabilities that are not likely to be trivially replicated by other
implementations.
George
--
for email reply remove "/" from address
| |
| Per Bothner 2004-10-28, 3:57 am |
| Daniel C. Wang wrote:
>
> huh? a purely statically typed language does need "dynamic reflection"
> if it wishes to implement "generic" tree-walking and IDEs that need to
> "circumvent" the static type system.
My point is that in a purely statically typed language tree-walkers can
be generated purely statically, so you don't need run-time reflection.
What qualifies as "reflection" is a gray area. Is procedure? a
reflective function. Does (pair? x) qualify as reflection? This
is not much different from Java's (x instanceof Pair), except that
in Java the set of types is open-ended. "Real" Java 1.1-style
reflection (as in "get me the names and types of fields of this
object") is a convenience, but it doesn't get you much beyond what
you could do by the static information in a .class file. The key
difference is that you can create and load new types *on the fly*,
at run-time. Of course you can do the same in almost any Scheme that
allows you to define new classes of objects. Whether you can
introspect the in-core data structures corresponding to those classes,
or you look at a file is a convenience issue, that's all.
--
--Per Bothner
per@bothner.com http://per.bothner.com/
| |
| Per Bothner 2004-10-28, 8:58 am |
| George Neuner wrote:
> I couldn't easily find any detail on your web site that describes how
> Scheme macros are embedded in the class file. I can imagine a couple
> of ways to do it: e.g., by storing the macro definition as a string
> in the constant pool
Well, a template is encoded as a special "word code" that stored in a
String (because Strings can be stored in the constant pool), but there
is no direct correspondance with the source form of the templace.
For example pattern identifiers are replaced by integer indexes into
the array of values extracted from the pattern. And there is code
to handle hygiene.
Since I've almost finished rewriting the macro handler, and since
the details are a matter of internal data representation, I'll just
summarize he basic idea:
(define-syntax MAC TRANSFORMER)
gets compiled into bytecode corresponding to:
public [static] Macro MAC = TRANSFORMER;
(Whether the field MAC is static or not depends on compilation options.)
When a class is loaded (in the Scheme sense), a binding is created in
the top-level environment corresponding to the public fields of the
class. Thus loading the above class would add a binding for MAC.
When a class is "required" the class is loaded at compile-time as
well as run-time. Compile-time bindings are created for each public
field in the required class. Thus MAC can be used to expand a
macro for (MAC ...) in the module that did the require.
When the TRANSFORMER is called it uses bindings from class the contains
the field MAC, not the call site, to preserve proper hygiene.
> or creating a parser function that the compiler
> can dynamically invoke on the dependent source.
Yes, it does that for syntax-case transformers. (This isn't usable
in the current sources, but it is in my working tree.)
> However you do it, I suspect you are taking advantage of Java specific
> capabilities that are not likely to be trivially replicated by other
> implementations.
Java run-time generating and loading of classe, combined with a
portable "object file format", and aided by reflection certainly
make this much easier. I assume you could do similar things
with .Net.
--
--Per Bothner
per@bothner.com http://per.bothner.com/
| |
| George Neuner 2004-10-29, 3:58 pm |
| On Thu, 28 Oct 2004 07:53:33 GMT, Per Bothner <per@bothner.com> wrote:
>George Neuner wrote:
>
>
>Well, a template is encoded as a special "word code" that stored in a
>String (because Strings can be stored in the constant pool), but there
>is no direct correspondance with the source form of the templace.
>For example pattern identifiers are replaced by integer indexes into
>the array of values extracted from the pattern. And there is code
>to handle hygiene.
>
>Since I've almost finished rewriting the macro handler, and since
>the details are a matter of internal data representation, I'll just
>summarize he basic idea:
>
>(define-syntax MAC TRANSFORMER)
>
>gets compiled into bytecode corresponding to:
>
>public [static] Macro MAC = TRANSFORMER;
>
>(Whether the field MAC is static or not depends on compilation options.)
>
>When a class is loaded (in the Scheme sense), a binding is created in
>the top-level environment corresponding to the public fields of the
>class. Thus loading the above class would add a binding for MAC.
>
>When a class is "required" the class is loaded at compile-time as
>well as run-time. Compile-time bindings are created for each public
>field in the required class. Thus MAC can be used to expand a
>macro for (MAC ...) in the module that did the require.
>
>When the TRANSFORMER is called it uses bindings from class the contains
>the field MAC, not the call site, to preserve proper hygiene.
Interesting. Thanks.
George
--
for email reply remove "/" from address
| |
| David Rush 2004-11-04, 3:58 pm |
| michele.simionato@gmail.com (Michele Simionato) writes:
> The point is that all module systems I have seen in the Scheme world
> are cumbersome, not obvious, over-engineered and even plain stupid :-(
Then you have absolutely not understood just how simple a Scheme
module system can be in the absence of macros. *Everything* else is
not an apples-to-apples comparison.
david rush
--
C/asm could go either way (socialist/conservative/libertarian)
depending on who's using them -- but in general I consider having to
deal with low-level platform details to be more slavery than freedom.
-- Adrian Kubala on comp.lang.scheme
| |
| David Rush 2004-11-04, 3:58 pm |
| Nic Ferrier <nferrier@tapsellferrier.co.uk> writes:
> I *think* that the Scheme community, by realizing that modules can be
> complicated, is making an error in implementing totally comprehensive
> solutions.
Well, depending on your definition of 'module system' and 'namespace',
Scheme may *already have* a comprehensive system for the no-macro
case. Given that Scheme strives to be a right-thing community
(Matthias Blume's objections to various technical issues aside), a
comprehensive solution is really the *only* option.
david rush
--
There's a big difference between the two parties. The Republicans want
to take away all your freedom. The Democrats want to take away all
your money.
-- Bill McGarr
| |
| Bradd W. Szonye 2004-11-06, 3:56 pm |
| Ray Dillinger writes:
Marcin 'Qrczak' Kowalczyk wrote:[color=darkred]
> Everything that other modules depend on at their compilation time
> should be extracted to a separate file by the compiler during
> processing of A.
>
> The file should be left with its old modification time if it would be
> overwritten with unchanged contents.
Let's call this output file A'. I presume that you leave it with the old
mod time so that tools like "make" won't try to rebuild it. However, if
you don't update the mod time, then "make" will try to rebuild A' on
every build. In the long run, that's not much better than rebuilding all
the dependents when you touch A.
If you're relying on "make," it's much better to update A' in a way that
makes the subsequent rebuilds trivial. It sounds like the Glasgow
Haskell compiler you described does something like that.
--
Bradd W. Szonye
http://www.szonye.com/bradd
| |
| Marcin 'Qrczak' Kowalczyk 2004-11-06, 8:57 pm |
| "Bradd W. Szonye" <bradd+news@szonye.com> writes:
> Let's call this output file A'. I presume that you leave it with the
> old mod time so that tools like "make" won't try to rebuild it.
> However, if you don't update the mod time, then "make" will try to
> rebuild A' on every build.
Yes, but the dependency says something like (where .i is the extension
of A', and .o is the extension of an object file):
%.i: %.o
@true
so the rebuild is trivial, the only user-visible change is the lack of
the message that everything was up to date and a hard to notice delay.
If the compiler cares to output *.i after *.o, the trivial rebuild
happens only when someone has actually modified the source in a way
which doesn't change the interface after the previous version has been
compiled. In particular it does not happen if he builds a downloaded
package without modifying sources after a partial build.
IMHO the trivial "rebuild" is better than actual rebuild on
dependencies. It would need hundreds of builds without 'make clean'
to accumulate more time with those "rebuilds" than an actual rebuild
would take.
At least in usual make configurations. Maybe it would be worse on
a slow NFS disk. I don't know if it interferes with distributed make.
But I have no better way which is not overly complex (reimplementing
make is out of the question).
--
__("< Marcin Kowalczyk
\__/ qrczak@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
| |
| Ray Dillinger 2004-11-07, 3:56 am |
| Marcin 'Qrczak' Kowalczyk wrote:
> IMHO the trivial "rebuild" is better than actual rebuild on
> dependencies. It would need hundreds of builds without 'make clean'
> to accumulate more time with those "rebuilds" than an actual rebuild
> would take.
>
> At least in usual make configurations. Maybe it would be worse on
> a slow NFS disk. I don't know if it interferes with distributed make.
> But I have no better way which is not overly complex (reimplementing
> make is out of the question).
Not really. It's open source, after all, and the new capabilities
are a strict win where useful, with no cost for other languages or
current usage.
What's required is that make maintain a datafile in addition to the
makefile, listing the time of the last substantive change for each
file mentioned in the makefile. This may be different than the time
the file was last written.
When we change A, but don't change its interface, then A' is updated/
rebuilt, but the new A' turns out to be identical to the old A'. The
new make utility, noting this fact, writes in its datafile that A was
changed, but A' didn't get a substantive change with this update.
Things that depend on A' are therefore not rebuilt.
This same mechanism would also be useful for other languages. For
example it would prevent recompilation of dependencies in C
systems where someone changed whitespace or added comments. If
the compiled file is identical to the previous compiled file,
then the dependencies wouldn't have to be rebuilt.
So all told, it's a strict win across the board and the change can
be backward compatible. Sounds like a case for submitting an updated
make utility to the maintainers of a dozen linux distributions.
Bear
| |
| Nic Ferrier 2004-11-10, 4:00 pm |
| David Rush wrote:
> michele.simionato@gmail.com (Michele Simionato) writes:
>
>
>
> Then you have absolutely not understood just how simple a Scheme
> module system can be in the absence of macros. *Everything* else is
> not an apples-to-apples comparison.
I *have* understood that. Indeed I am implementing one and it is very,
very trivial.
But Scheme implementors don't seem to have understood this because they
still make the totally generic case the standard tool when perhaps the
simpler case needs to be the default.
The old perl maxim should hold true: make the easy things easy and make
the hard things possible.
Nic
| |
| Nic Ferrier 2004-11-10, 4:00 pm |
| Marcin 'Qrczak' Kowalczyk wrote:
> Nic Ferrier <nferrier@tapsellferrier.co.uk> writes:
>
>
>
> Because os.popen doesn't fit the traditional sexpr syntax. It would
> have to be something like (get os popen), which would look ugly.
>
I agree that it would look ugly. But I don't think it's necessary to do
that.
An import is an import from one namespace to another, so there shouldn't
be any reason why the importer could not rename the imported symbols as
they come into the new lexical environment.
Or is there?
Nic
| |
| Martin Rodgers 2004-11-10, 4:00 pm |
| Nic Ferrier wrote:
> The old perl maxim should hold true: make the easy things easy and make
> the hard things possible.
I think that's a bit older than Perl. ;)
--
http://www.wildcard.demon.co.uk You can never browse enough
Will write code that writes code that writes code for food
| |
| David Rush 2004-11-10, 4:00 pm |
| Nic Ferrier <nferrier@tapsellferrier.co.uk> writes:
> David Rush wrote:
>
>
> I *have* understood that. Indeed I am implementing one and it is very,
> very trivial.
No, this makes me think that you haven't. Scheme doesn't need one
because it is already built in. First-class higher-order functions
with lexical scope give you *everything* you need in a latently typed
language like Scheme.
> The old perl maxim should hold true: make the easy things easy and
> make the hard things possible.
Modularity in Scheme is so simple that most people don't even realize
it's there.
david rush
--
The truth may be out there, but lies are inside your head.
-- Terry Pratchett, in _Hogfather_
| |
| Rene de Visser 2004-11-10, 4:00 pm |
| "David Rush" <kumoyuki@gmail.com> wrote in message
news:cmtc6t$oh0$1@pixie.nscp.aoltw.net...
> No, this makes me think that you haven't. Scheme doesn't need one
> because it is already built in. First-class higher-order functions
> with lexical scope give you *everything* you need in a latently typed
> language like Scheme.
> Modularity in Scheme is so simple that most people don't even realize
> it's there.
Can you enlighten me (I come from a common lisp background)?
I have two libraries both of which define a function
display-matrix for example.
In my program how do I make clear which one I want to call?
How is it that the library loaded second doesn't overwrite the one loaded
first?
Rene.
| |
| Jens Axel Søgaard 2004-11-10, 4:00 pm |
| Rene de Visser wrote:
> "David Rush" <kumoyuki@gmail.com> wrote in message
> news:cmtc6t$oh0$1@pixie.nscp.aoltw.net...
>=20
>=20
>=20
>=20
> Can you enlighten me (I come from a common lisp background)?
My guess is that David is thinking of
<http://groups.google.com/groups?sel...pixie.nscp.aol=
tw.net>
and the two articles linked to in the above.
--=20
Jens Axel S=F8gaard
| |
| Michele Simionato 2004-11-11, 3:59 pm |
| David Rush <kumoyuki@gmail.com> wrote in message news:<cmtc6t$oh0$1@pixie.nscp.aoltw.net>...
> Nic Ferrier <nferrier@tapsellferrier.co.uk> writes:
>
>
> No, this makes me think that you haven't. Scheme doesn't need one
> because it is already built in. First-class higher-order functions
> with lexical scope give you *everything* you need in a latently typed
> language like Scheme.
Well, no. A module system means a STANDARDIZED way to import
names, too. Lexical scoping is only a way to avoid pollution, and this is
it only half of a module system.
Michele Simionato
| |
| Michele Simionato 2004-11-11, 3:59 pm |
| David Rush <kumoyuki@gmail.com> wrote in message news:<cmtc6t$oh0$1@pixie.nscp.aoltw.net>...
> Modularity in Scheme is so simple that most people don't even realize
> it's there.
Well, lexical scoping works and functions are first class objects so
you can return them (you pointed to me the functor trick one year
ago). But this is a
way one can implement a simple module system, it is not a module
system. As
I said to me a module system is essentially about STANDARDIZATION,
having a
common way of defining/importing/exporting functions/modules/packages.
Michele Simionato
| |
| Nic Ferrier 2004-11-16, 3:58 pm |
| michele.simionato@gmail.com (Michele Simionato) writes:
> Well, lexical scoping works and functions are first class objects so
> you can return them (you pointed to me the functor trick one year
> ago). But this is a way one can implement a simple module system, it
> is not a module system. As I said to me a module system is
> essentially about STANDARDIZATION, having a common way of
> defining/importing/exporting functions/modules/packages.
Hear, hear.
--
Nic Ferrier
http://www.tapsellferrier.co.uk
|
|
|
|
|