Release of PyPy 0.7.0

pypy-0.7.0: first PyPy-generated Python Implementations
 ========================================
======================

What was once just an idea between a few people discussing
on some nested mailing list thread and in a pub became reality ...
the PyPy development team is happy to announce its first
public release of a fully translatable self contained Python
implementation.  The 0.7 release showcases the results of our
efforts in the last few months since the 0.6 preview release
which have been partially funded by the European Union:

- whole program type inference on our Python Interpreter
implementation with full translation to two different
machine-level targets: C and LLVM

- a translation choice of using a refcounting or Boehm
garbage collectors

- the ability to translate with or without thread support

- very complete language-level compliancy with CPython 2.4.1


What is PyPy (about)?
------------------------------------------------

PyPy is a MIT-licensed research-oriented reimplementation of
Python written in Python itself, flexible and easy to
experiment with.  It translates itself to lower level
languages.  Our goals are to target a large variety of
platforms, small and large, by providing a compilation toolsuite
that can produce custom Python versions.  Platform, Memory and
Threading models are to become aspects of the translation
process - as opposed to encoding low level details into a
language implementation itself.  Eventually, dynamic
optimization techniques - implemented as another translation
aspect - should become robust against language changes.

Note that PyPy is mainly a research and development project
and does not by itself focus on getting a production-ready
Python implementation although we do hope and expect it to
become a viable contender in that area sometime next year.


Where to start?
-----------------------------

Getting started:
http://codespeak.net/pypy/dist/pypy...ng-started.html

PyPy Documentation: http://codespeak.net/pypy/dist/pypy/doc/

PyPy Homepage:      http://codespeak.net/pypy/

The interpreter and object model implementations shipped with
the 0.7 version can run on their own and implement the core
language features of Python as of CPython 2.4.  However, we still
do not recommend using PyPy for anything else than for education,
playing or research purposes.

Ongoing work and near term goals
---------------------------------

PyPy has been developed during approximately 15 coding sprints
across Europe and the US.  It continues to be a very
dynamically and incrementally evolving project with many
one-w meetings to follow.  You are invited to consider coming to
the next such meeting in Paris mid October 2005 where we intend to
plan and head for an even more intense phase of the project
involving building a JIT-Compiler and enabling unique
features not found in other Python language implementations.

PyPy has been a community effort from the start and it would
not have got that far without the coding and feedback support
from numerous people.   Please feel free to give feedback and
raise questions.

contact points: http://codespeak.net/pypy/dist/pypy/doc/contact.html

contributor list:
http://codespeak.net/pypy/dist/pypy...ontributor.html

have fun,

the pypy team, of which here is a partial snapshot
of mainly involved persons:

Armin Rigo, Samuele Pedroni,
Holger Krekel, Christian Tismer,
Carl Friedrich Bolz, Michael Hudson,
Eric van Riet Paap, Richard Emslie,
Anders Chrigstroem, Anders Lehmann,
Ludovic Aubry, Adrien Di Mascio,
Niklaus Haldimann, Jacob Hallen,
Bea During, Laura Creighton,
and many contributors ...

PyPy development and activities happen as an open source project
and with the support of a consortium partially funded by a two
year European Union IST research grant. Here is a list of
the full partners of that consortium:

Heinrich-Heine University (Germany), AB Strakt (Sweden)
merlinux GmbH (Germany), tismerysoft GmbH(Germany)
Logilab Paris (France), DFKI GmbH (Germany)
ChangeMaker (Sweden), Impara (Germany)

That's great! Congratulations!

Carl Friedrich Bolz wrote:
[[... great news ...]]

Would it be useful for people to start trying out their modules/code to see
if they work with this release, and whether they can likewise be translated
using the C/LLVM backends, or would you say this is too early? (I'm more
thinking in terms of it providing real world usecases in the hope of
finding things that don't work - rather than anything else)

Either way, this looks like a great milestone - congratulations to the
entire team. (I remember PyPy being met with skepticism as to whether
it could even be done! :-)

Best Regards,


Michael

Michael Sparks <ms@cerenity.org> wrote:

> Would it be useful for people to start trying out their modules/code to se
e
> if they work with this release, and whether they can likewise be translate
d
> using the C/LLVM backends, or would you say this is too early? (I'm more
> thinking in terms of it providing real world usecases in the hope of
> finding things that don't work - rather than anything else)

This is not how it works. Pypy doesn't translate your code to C/LLVM.
Currently PyPy can only translate a pretty simple subset of python
called RPython which has a very C-like syntax (but without memory
management code). This is needed to allow type inference inside the
interpreter code.

The code in your application is application code and can be whatever you
want, you may try to translate it to C/LLVM but it won't be that good of
course because the annotator is not that intelligent.

Just In Time compilation a-la-psyco is planned before the 1.0 release of
pypy.
Right now also the compiler/parser run in application level which means
it is rather slow because it is not translated like the rest of pypy
(which accounts for quite a bit of the slowness of the translated pypy)
[this is of course only true if they didn't manage to rewrite the
parser/compiler in RPython, which I admit I'm not sure]

More information is available on codespeak.net website :).

What's really  in PyPy is how easily you can implement a new object
space and change semantics of operations done in python.
Writing a Python interpreter that can share data structures across many
computers will be hard but possible.
There are already 4 different object spaces implemented, the standard
one, the thunk one (which is a lazy evaluation object space), the
flowgraph object space and the trace object space that traces each
operation done.

> Either way, this looks like a great milestone - congratulations to the
> entire team. (I remember PyPy being met with skepticism as to whether
> it could even be done! :-)

Indeed.

--
Valentino Volonghi aka Dialtone
Now Running MacOSX 10.4
Blog: http://vvolonghi.blogspot.com
http://weever.berlios.de

Carl Friedrich Bolz wrote:

> pypy-0.7.0: first PyPy-generated Python Implementations
>  ========================================
======================
>
> What was once just an idea between a few people discussing
> on some nested mailing list thread and in a pub became reality ...
> the PyPy development team is happy to announce its first
> public release of a fully translatable self contained Python
> implementation.  The 0.7 release showcases the results of our
> efforts in the last few months since the 0.6 preview release
> which have been partially funded by the European Union:

Cool.  I just tested EmPy's regression suite with PyPy 0.7.0, and it ran
fine (though obviously it was very slow).  Nice job so far, guys!

--
Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
It is fatal to enter any war without the will to win it.
-- Douglas MacArthur

Valentino Volonghi aka Dialtone wrote:

> Michael Sparks <ms@cerenity.org> wrote:
> 
>
> This is not how it works.

I beg to differ - it is how it can work (just not the default or currently
recommended). Currently the biggest, most interesting example is this:
(from the getting started page, compiling pypy)

cd pypy/translator/goal
python translate_pypy.py

Which has just finished executing on my machine and took about 3 hours to
complete (partly because the cc1 process reached ~780MB in terms of memory
footprint and my machine has 512Mb). However it goes on to talk about
compiling other things:

"""You can also use the translate_pypy.py script to try out several smaller
programs, e.g. a slightly changed version of Pystone:
cd pypy/translator/goal
python translate_pypy.py targetrpystone
"""

Which is pretty  of course. For those of interest running pystone with
the pypy compiled native binary has the following results for pystones on
my machine:

michaels@linux:~/pypy-0.7.0/pypy/translator/goal> ./pypy-c
debug: entry point starting
debug:  argv -> ./pypy-c
debug: importing code
debug: calling code.interact()
Python 2.4.1 (pypy 0.7.0 build) on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole) 
Pystone(1.1) time for 1000 passes = 13.97
This machine benchmarks at 71.582 pystones/second 

The same results for CPython:
michaels@linux:~/pypy-0.7.0/pypy/translator/goal> python
Python 2.4 (#1, Mar 22 2005, 21:42:42)
[GCC 3.3.5 20050117 (prerelease) (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information. 
Pystone(1.1) time for 50000 passes = 1.58
This machine benchmarks at 31645.6 pystones/second

Obviously therefore anyone sing to translate their existing code from
python to an executable directly using pypy would not be doing it for
performance reasons (again, something I'm aware of watching the
updates come out and having run svn checkouts at previous times).

(Factor of ~450 though is pretty d*^%n  though :)

I'm also well aware of the difference between python and rpython :-)

> Pypy doesn't translate your code to C/LLVM.

> Currently PyPy can only translate a pretty simple subset of python
> called RPython which has a very C-like syntax (but without memory
> management code). This is needed to allow type inference inside the
> interpreter code.
>
> The code in your application is application code and can be whatever you
> want, you may try to translate it to C/LLVM but it won't be that good of
> course because the annotator is not that intelligent.

*Shrug* Seemed pretty good at annotating and compiling a module longer than
many of mine at Europython, and then compiled using the LLVM backed.  :)

Anyway, whether it's sensible or not I'm going to play with translating some
of my modules :)

Regards,


Michael.

"Michael Sparks" <ms@cerenity.org> wrote in message
news:43121002$0$97118$ed2619ec@ptn-nntp-reader03.plus.net...
> (Factor of ~450 though is pretty d*^%n  though :)

I think the ratio was 10 or 100 times higher when first measureable.  I
have long expected success but am still impressed.

TJR

Michael Sparks <ms@cerenity.org> writes:

> Valentino Volonghi aka Dialtone wrote:
> 
>
> I beg to differ - it is how it can work (just not the default or currently
> recommended).

The chance of any random module you have written being rpython is more
or less zero, so it's not _that_ interesting for you to try to compile
them with PyPy.

> """You can also use the translate_pypy.py script to try out several smalle
r
> programs, e.g. a slightly changed version of Pystone:
> cd pypy/translator/goal
> python translate_pypy.py targetrpystone
> """
>
> Which is pretty  of course. For those of interest running pystone with
> the pypy compiled native binary has the following results for pystones on
> my machine:
>
> michaels@linux:~/pypy-0.7.0/pypy/translator/goal> ./pypy-c
> debug: entry point starting
> debug:  argv -> ./pypy-c
> debug: importing code
> debug: calling code.interact()
> Python 2.4.1 (pypy 0.7.0 build) on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> (InteractiveConsole) 
> Pystone(1.1) time for 1000 passes = 13.97
> This machine benchmarks at 71.582 pystones/second 
>
> The same results for CPython:
> michaels@linux:~/pypy-0.7.0/pypy/translator/goal> python
> Python 2.4 (#1, Mar 22 2005, 21:42:42)
> [GCC 3.3.5 20050117 (prerelease) (SUSE Linux)] on linux2
> Type "help", "copyright", "credits" or "license" for more information. 
> Pystone(1.1) time for 50000 passes = 1.58
> This machine benchmarks at 31645.6 pystones/second
>
> Obviously therefore anyone sing to translate their existing code from
> python to an executable directly using pypy would not be doing it for
> performance reasons (again, something I'm aware of watching the
> updates come out and having run svn checkouts at previous times).

No, you're still operating at the wrong level here (very easily done).
This is the _translated PyPy_ interpreting pystone.  If you run a
_translated pystone_ you'll (hopefully) get a different, faster
answer.

In expected order of execution speed:

interpreted pypy interpreting pystone
translated pypy interpreting pystone
cpython interpreting pystone
translated pystone

> Anyway, whether it's sensible or not I'm going to play with translating so
me
> of my modules :)

Whatever floats your boat :)

Cheers,
mwh

--
Ability to type on a computer terminal is no guarantee of sanity,
intelligence, or common sense.
-- Gene Spafford's Axiom #2 of Usenet

Hi Valentino, Michael, all,

On Sun, Aug 28, 2005 at 20:46 +0200, Valentino Volonghi aka Dialtone wrote:
> Michael Sparks <ms@cerenity.org> wrote: 
>
> This is not how it works. Pypy doesn't translate your code to C/LLVM.
> Currently PyPy can only translate a pretty simple subset of python
> called RPython which has a very C-like syntax (but without memory
> management code). This is needed to allow type inference inside the
> interpreter code.

This is mostly true but maybe a bit misleading.  The subset of Python
that we can translate does not involve special syntax. For example
we can translate

http://codespeak.net/pypy/dist/pypy...oal/richards.py

and gain some speed up by a factor of 70 over interpretation with
CPython.  The "RPython" limitation mainly refers to the way
how dynamically you use Python and how unambigously your values
flow across your functions and methods.  Also you cannot, for example,
create classes dynamically while creating instances is, of course,
no problem.  Also, we support all Python control flow statements.
For a more detailed list, look at:

[url]http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#rpython-definition-not[/
url]

The real culprit is that we don't currently intend to try
offering a tool that can automatically translate Python
modules to C because we are still experimenting with and
targeting our own Python interpreter and tackling the
challenges that arise out of translating this efficiently.
It's a matter of focus of the current dev group and not
one of technical feasibility.

If one or more people would want to care for producing,
testing and documenting such a tool on top of our current code
base then the next ws might be good for that as we intend
to go for some cleanup refactorings anyway.

> The code in your application is application code and can be whatever you
> want, you may try to translate it to C/LLVM but it won't be that good of
> course because the annotator is not that intelligent.

See above.  You can try to translate python programs with a bit
of effort and determination to find your way :-)

> Just In Time compilation a-la-psyco is planned before the 1.0 release of
> pypy.

We don't have a fixed version roadmap but indeed, we plan to work
on JIT compilation and other niceties rather soon now.

cheers,

holger

Michael Hudson wrote:
...
> The chance of any random module you have written being rpython is more
> or less zero, so it's not _that_ interesting for you to try to compile
> them with PyPy.

I know - the code I use contains LOTS of generators for example, which
obviously don't fit the requirements :) (It's not difficult to convert code
from being a generator though :-)

> No, you're still operating at the wrong level here (very easily done).
> This is the _translated PyPy_ interpreting pystone.  If you run a
> _translated pystone_ you'll (hopefully) get a different, faster
> answer.

Actually what I was aiming to compare was speed of pystone running on top of
CPython vs speed of pystone on top of translated pypy. ie running pystone
on top of something native that can execute python code :-)

> In expected order of execution speed:
>
> interpreted pypy interpreting pystone
> translated pypy interpreting pystone
> cpython interpreting pystone
> translated pystone

Damn, I'm now going to have to sit down and find out if that's true :-)

*shucks* :-)
 
>
> Whatever floats your boat :)

Regards,


Michael.