Jul 2012 Note: many of the things I was aiming for with Lunacy were accomplished with my new language, Irken.
Lunacy distribution:
lunacy-2000-07-30.tar.gz
here's a version hacked to run on a 64-bit platform (tested on amd64):
lunacy64-2004-02-22.tar.gz
64-bit version for Mac OS X (g5):
lunacy64-g5-osx.tar.gz
[Note: the 64-bit tarballs unfortunately are forked from the main distribution, a merge should be done soon]
Lunacy will eventually support front ends for Scheme and an extended dialect of Python. It will emphasize the use of first-class continuations (i.e, call/cc will have no overhead). I intend to use it to explore sophisticated asynchronous I/O techniques for networking (like Medusa, only using coroutines), and as a powerful development environment for Win32.
I hope for Lunacy to be fast enough to delve into some areas such as fast parsing (HTML, Python, CORBA IDL, etc...), text formatting/layout, and composable user interface components (a la Fresco/Interviews/Swing).
Lunacy began life as an x86 compiler. While putting together the runtime for x86 lunacy, I found myself spending a lot of time on nasty little bootstrapping details. (The compiler itself was easy).
Now that I've taken a full-time job, I do not have the time-luxury of building a system that is completely free of C. So for now I am building a VM version of the system; this allows me to move much more rapidly. If and when the need arises for the x86 compiler, it will be there, and usable: the runtime data structures, gc, etc... are nearly identical between then two systems. It should be possible at some point to build a hybrid system that will be able to mix bytecode and native code.
The VM is very tiny; it uses a couple of GCC extensions (lexical and inline functions) that let me put put the entire VM into one function, without having to use a lot of macros. [note: this is largely unnecessary, I can achieve much the same effect using globals and inlining.] How small? A tarball of the stripped VM and runtime image is currently weighing in at a 24KB. You could hide it in a banner ad GIF.
If the system becomes popular, then the same pressures brought to bear on the current Python implementation will motivate hackers to improve Lunacy. And there will be lots of room for improvement. Because of its grounding in Scheme, most modern compiler technology will be applicable.
The garbage collector is a simple two-space Cheney copying collector (< 200 lines of C). Somewhere down the line this will probably grow up and become a generational collector. 'generational gc' uses multiple heaps, and collects only one of them very often.
Lunacy is a continuation-based compiler. The following books are my main references:
EOPL is the place to start. The Lunacy compiler is virtually identical to the tail-recursive compiler introduced in the last chapter of the book. This book is very well written - it uses an incremental approach, introducing increasingly complex ideas with small, easily understood (and easily coded!) examples. If you want to understand Lunacy, simply read the book. Start at the front, be sure to try the code examples - you will not fully understand them until you do.
"Compiling with Continuations" is a more difficult and technical book, but picks up where EOPL leaves off in terms of compiler development. It covers "All the nitty-gritty details of compiling to really good machine code". This book will be the place to go for optimization ideas. Although it is oriented around the programming language ML, most of the concepts are applicable to Scheme.
Very comprehensive. As far as I know, this is the only book of its kind.
This is available online from ftp://ftp.cs.utexas.edu/pub/garbage/cs345/schintro-v14/schintro_toc.html , but I have a local copy here that you may find more convenient.
Good Stuff! Wish I'd found it earlier... This covers a lot of the same ground that EOPL does, but it's a bit more nuts-and-bolts. Can also serve as a good introduction to Scheme.
Lunacy will most likely be released using a Python-style copyright ('do what thou wilt'), though I may consider looking into Ghostscript-style multiple licenses.
A Quick Tour of the Compilation Process
If you have any questions, or would like to help out, drop me a line!
Yeah, a bit of a break. Fixed a couple of bugs, ported to amd64.
Roughly halved the time for a self-bootstrap by adding a few more string primitives to speed up all that byte-twiddling in the bytecode phase.
New README file
Finally self-hosting!
Lots of speed tweaks - the VM is now 'threaded' in the Forth sense, and a few opcodes were enhanced with implied-argument versions.
read-eval-print loop over a socket!
rushing@fang:~$ telnet localhost 9777 Trying 127.0.0.1... Connected to localhost.nightmare.com. Escape character is '^]'. > 234 ; compile: 5348 execute: 19 234 > ^] telnet> c Connection closed.Here's the source:
(define (make-server port)
(let ((socket (%socket 2 1)))
(print "socket:" socket)
(print "bind" (%bind socket 0 0 0 0 port))
(print "listen" (%listen socket 5))
(let ((fd (%accept socket)))
(let ((is (buffered-file fd 8192))
(os (buffered-file fd 8192)))
(repl 0 is os)
))))
Lots of very rapid progress... we're hitting a good critical mass. fleshing out the runtime: macros, support for repl-levels, returning values from errors, environment examiner, a port-parameterized repl (could probably run it over a socket with only a little work). Several tweaks to the VM to speed it up. very close to self-bootstrapping
Buffered I/O, stdin/stdout. Bare minimum set of socket funs.
A start on write.
rushing@fang:/tmp/lunacy$ ./machine test.image
read 46976 bytes
initializing runtime...
done.
Lunacy. Copyright 1998-1999 Sam Rushing
> (%socket 2 1)
4
; 40 insns
> (%connect 4 (zero-terminate "127.0.0.1") 21)
0
; 1053 insns
> (define buf (make-string 8192))
#{undefined}
; 189 insns
> (%read 4 buf 8192)
57
; 68 insns
> buf
"220 fang.nightmare.com FTP server (Version 6.00) ready.
"
; 59 insns
> (%write 4 "quit
" 5)
5
; 42 insns
> (%read 4 buf 8192)
14
; 68 insns
> buf
"221 Goodbye.
mare.com FTP server (Version 6.00) ready.
"
; 59 insns
> (%close 4)
0
; 38 insns
>
We've taken a new name, Lunacy, and a new direction. [I can hear Derek Smalls on the bass guitar] We're using a tiny VM; but the runtime environment is very similar.
Recent exciting work with coroutines in Python has rekindled my interest in this project, as a form of insurance (on the off chance that it takes a long time for the coroutine support to get into the Python distribution).
The first version of the VM is a mere 15KB (!). The runtime, reader, transformer, and compiler together make a 36KB image.
A bootstrapper has been written in Scheme. This should allow the compiler to be loaded into any Scheme system, where it can create image files 'from scratch'.
A major optimization is in place: Closures are not generated for lambda expressions when they are in the operator position. The expander turns all let-like forms into such expressions, so this has a major impact on code size (20%) and speed.
The runtime and compiler are now self-hosted. A tiny (~10k) image loader is capable of loading and starting an image. The resulting image is about 350k, very small and fast. Binaries for Linux and Win32 are available here. The next major step is to have the system compile itself and generate images (without having to dump them from a running system). This should make it possible to build Lunacy using any Scheme system.
A simple runtime, written in Scheme, is available. Dynamic environment lookup is implemented in this runtime. Relatively complex features of Scheme are being ignored for the moment (n-ary procedures, for example).
I have begun the task of rewriting the compiler in Scheme, so that it may be self-hosting. The reader and parser are finished, I am now in the process of 'borrowing' the macro expanders from scheme48 to implement the transform stage (where derived expressions are converted to the primitive expression types).
-Sam
rushing@nightmare.com