On Scheme

Thoughts on Scheme and teaching Scheme

Archive for the ‘Common Lisp’ Category

In Defense of Change

Posted by Peter on April 21, 2006

Recently there has been a burst of criticisms concerning Common Lisp. If you don’t know what I am talking about, or simply want to catch up on the current discussion before you read the rest of this piece you should visit: Steve Yegge’s blog, the usenet response, a usenet debate on CL suckage, and a usenet proposal for improving CL.

Although they have been accused of hating Lisp, my impression is that the majority of these critics love Lisp, and simply want to see it become better. For those of you who got lost in the flames let me provide a quick summary of the major complaints: 1- macros: there is no hygienic defmacro, but define-syntax may be too cumbersome for some, and also it is hard to hook some aspects of the source with macros. 2- CLOS: CLOS is becoming dated, looks too much like a hack, and is too hard to use. The object system needs to be rewritten, possibly from scratch. 3- There is no type system outside of run-time checking. 4- The Lisp community has become too fragmented, even Common Lisp implementations tend to provide features that makes them incompatible with each other. 5- Common Lisp has stagnated for too many years, and there is no way to implement changes that will be adopted outside of a single implementation, which ensures that no serious improvements are made.

In my opinion these are valid criticisms and, although they are directed primarily at Common Lisp, Scheme is not without its own flaws. I don’t believe that accepting these criticisms as valid implies that one should drop Lisp, even with its flaws I think that Lisp is still the best language, it just could become better.

However I am not writing this piece to discuss the criticisms or to propose solutions, what I want to address here is the response of the community to them, which I found appalling. Some people’s responses were scarily reminiscent of religious authorities trying to cast out heretics, which is not an appropriate attitude for a computer scientist. As scientists we should always be looking to improve our tools and methods, and this means trying new things to find out what changes constitute an improvement.

Let me discuss what makes some responses I have heard especially ridiculous.

The difficulty argument:
We should leave confusing / ill designed feature untouched because it makes Lisp harder, and thus keeps out the riff-raff. This is a flawed argument because a large volume of users is more important than a few brilliant users. Even mediocre programmers can write useful libraries / promote the language. Libraries especially make a language attractive to even more programmers, and for use in commercial applications. Basically if you want to make a living using Lisp then you should hope more programmers use Lisp, even if they aren’t as “133t” as you.

The fear of change argument:
We shouldn’t change feature X (often CLOS) because it is used so widely that changing it would force me to learn new things / update my code. This is a flawed argument because other languages (with supposedly dumber programmers) are able to change themselves without problems. If you depend on a removed feature you can always use an older compiler for the code, for example Visual Basic 6 is still used to build some older applications that aren’t compatible with Visual Basic .Net.

The perfection argument:
Feature X is perfect they way it is. This is unbelievable because perfection tends not to be humanly possible; we simply try to approximate it with a greater and greater degree of accuracy. It is possible that a change could be worse than the current way of doing things, but once we are open to the idea of change there is nothing stopping us from changing it again until we revert it to the original state or really do find something better.

The “it could be worse argument”
Sure Lisp has problems, but it is better than languages X, Y, Z and that is good enough for me. I admit that Lisp is better now, but you can’t really be sure that will always be the case. If Lisp doesn’t move forwards than we can rest assured that something better than Lisp will be invented eventually.

The diversity argument
Fragmentation gives you options. Sure it does, you just can’t have them all at once. If you want feature Y in implementation A you have to give up Z in implementation B. Experimenting with variants is important of course, but that is why people write toy versions of languages, to try out new ideas. Ideally the best features are then incorporated into the next version of the standard implementation(s). The problem with Common Lisp is that such features don’t make their way to the spec., and it looks like they never will. Also, confusing the versioning of a language with Common Lisp’s fragmentation is so ridiculous that I am not even going to address it.

The personal attack argument:
You don’t really use Common Lisp! Go out and code instead of bitching! God forbid we not be content with the status quo!

What should be done?

So what should you do if you think Lisp could use some fixing? Go out and fix it. Go write a toy version, it doesn’t have to be fast or efficient. However if you can use it to demonstrate something cool that fixes a problem people will either help you make it faster and smaller, or your feature will be adopted by a more professional implementation. Do this enough and you just might end up with a new popular Lisp dialect.


Posted in Common Lisp | 2 Comments »

Convincing Programmers They Want To Use Lisp

Posted by Peter on April 7, 2006

A reader commented yesterday that one, unmentioned, barrier to the adoption of Lisp as a universal language is convincing programmers of other languages that Lisp is a better choice. Its hard enough to convince someone to move from one language to another when they look the similar, for example from C to C++, so convincing someone that they want to use Lisp is even harder than normal.

Everyone says how elegant Lisp syntax is and how useful functional programming is. This may be true but it is definitely not convincing to the experienced programmer. The experienced programmer is not intimidated by complicated code, which can have its own kind of elegance, what they want is safety, readability, and extensibility.

For example you may, badly, try to convince them that Lisp is better by showing them how a universal map function is easier to write.
Your example may look like the following:

(define map
  (lambda (function first rest combine list)
    (if (null? list)
        (combine (function (first list)) (my-map function first rest combine (rest list))))))

You point out the elegance of this, how its operation is extendable to any kind of data structure, for example it can be called on lists as follows: (map (lambda (x) (+ 1 x)) car cdr cons '(1 2 6 7)), and expect the beauty of first order functions to win the programmer over.

However the programmer responds with the following:

interface ConCell<T>
	public T car();
	public ConsCell<T> cdr();
	static public ConsCell<T> cons(T a, ConsCell<T> b);

interface ElementFunction<A, B>
	static public A operate(B x);

class Map<A, B>
	public static ConsCell<A> doMap(ElementFunction<A, B> func, ConsCell<B> list)
		if (list)
			return ConsCell<A>.cons(func.operate(list.car()), Map<A, B>.doMap(func, list.cdr()));
			return null;

Yes, the definition is much longer, but it can be called in the same basic way: Map<int, int>.doMap(myAdditionFuncion, NumList.MakeList(1, 2, 6, 7));. The experienced programmer doesn’t care that the definition is substantially longer, since they can use it in their code in basically the same way. In fact they may prefer their definition because it is both type safe and extendable. For example if you wanted every member of the data structure to implement an index function you could add public int index(); to the cons cell definition, and either force structures to implement it or throw an exception. Yes the same extensibility, and possibly type safety as well, could be accomplished in Lisp using an object system, but then your definition of map is not much shorter or more elegant than theirs.

Now that I have demonstrated the wrong way of doing things what is the right way? The right way is to show them macros. Give the example of a networked file system, where when you open a remote file you send the in use message, and when you close it you send the finished message. Thus in the code you will see a lot of the following pattern:


All it takes is to forget to send the closed notification once and you have a hard to find bug on your hands. The experienced programmer knows that while they may always remember to do this, since they wrote the system, their colleagues may not. Now show them the following macro:

(defmacro with-net-file
  (lambda (file &rest body)
    `(progn (OpenNetFile file) ,@body (CloseNetFile file))))

Now to write their network file operations you write:

(with-net-file f

And now no one can forget to close the file, plus it’s shorter as well. There is no way to do this in a non Lisp language. The closest you can come is to create an object whose constructor sends the open command and destructor sends the closed command, but even this approach has problems. Languages such as Java don’t have reliable destructors, so this option is flat out for them. In other languages you can either create the object on the stack, which means that the file will be in use for the entire function call, which is definitely not desirable (maybe a function called within it wants to open the same file, and you only needed the file for the very beginning of the function), but the only other alternative is to dynamically create and destroy the object when you need it which has the same problem, the programmer needs to remember to dynamically destroy it.

Once you have caught the programmer’s interest with simple macros show them some other cool things they can do, like language extensions. Experienced programmers are always looking for more control, and macros give it to them. Later you can tell them that functional programming is cool too, but they probably won’t need that incentive.

On a personal note macros are what convinced me to move to Lisp. For the longest time I was a C++ programmer. Even though I had seen some Scheme and Lisp back in school it never grabbed me, probably because they didn’t mention macros. However once I read On Lisp, and saw how cool macros were I was hooked. Why I ended up favoring Scheme, with its ugly and complicated macro system instead of Common Lisp is a story for another day however.

Posted in Common Lisp, Scheme Community, Teaching Scheme | 29 Comments »

Why Isn’t Everyone Using Lisp?

Posted by Peter on April 6, 2006

You know Lisp* is great, I know Lisp is great, so why isn’t everyone using Lisp? A long time ago people might have shied away from Lisp claiming that it is too slow to be practical, but after Java took off I think we can all see that the speed of a language is not too closely correlated with its success.

I think that Lisp basically has three problems that prevent it from becoming mainstream. One is the lack of libraries, specifically libraries that handle web functionality. I know that there are many many libraries for different implementations of Lisp, and that you can do web programming in Lisp, but there is no one standardized and accepted solution to this problem. Different libraries work under different implementations of Lisp, and provide different kinds of functionality. Compare this to languages such as Python, Ruby, and Java, which have a standard ways of handling networking and the web and you can see why developers might think that Lisp is somewhat lacking in this area.

Another major problem with Lisp is its number of dialects. In one sense having a large number of dialects is a strength because it allows innovation. However because of this it is hard to find programmers who are all proficient in a single dialect, are competent, and who want to work for your business. This makes Lisp somewhat unattractive to a company who would rather be sure of having a mediocre Java team than risk not having a Lisp team, no matter how great its potential. Also, in a project with multiple programmers dialect dependant details such as how the module system is implemented, what kind of objects you are using, ect become rather important, which is why a PLT Scheme programmer can’t simply start working in Common Lisp the next day.

Finally Lisp lacks a good IDE. Yes I know many of you favor emacs, and I won’t deny that emacs has its strengths, but for a corporate environment an IDE such as Eclipse or Visual Studio is important. Such IDEs provide graphical representations of code and package structure that allow developers to easily navigate through code created by other people. Additionally it is easy to standardize on an IDE such as these, as anyone can use them with only a few minutes of instruction, unlike emacs which only exposes its powerful features to the experienced and gifted.

Obviously I wouldn’t advocate a solution to these problems that involves trying to standardize all the dialects into one language. That has already been tried with Common Lisp, and Common Lisp failed to do away with the competing dialects, and unfortunately has become stagnant as a language, with new features being added only rarely. What we really need is a set of inter-dialect guidelines, which the community actually likes and thus will implement universally. Such guidelines would need to dictate such things as how modules are declared, if dialects are lisp-1 or lisp-2, and the C foreign function interface (to allow easy porting of libraries). Obviously such a task could be as hard or harder than the creation of Common Lisp, but it is something to think about.

*When I talk about Lisp I mean Scheme, Scheme dialects, Common Lisp, ect, not just versions of Common Lisp, which is what some people seem to think it means.

Posted in Common Lisp, Scheme Community | 5 Comments »

Why Scheme Shouldn’t Have An Official Object System

Posted by Peter on April 3, 2006

Obviously object systems can be created in Scheme, even I created one, but objects are not part of the language specification itself. I don’t pretend to know the motivations behind the design of Scheme, but I think that fundamentally Scheme may lack objects because it is a minimalist language. Here however I will defend the benefits of not having an object system built into the language, whatever the designers’ real reasons are.

As you may know some Lisp languages, most notably Common Lisp, and some dialects of Scheme, such as Bigloo, do have a built in object system. Other distributions, such as PLT, come packaged with modules that implement an object system if you choose to include them. Individuals have also devised their own flavors of object systems for scheme, ranging from imitations of CLOS, to prototype based systems, to slot based systems, to message oriented systems, and many which are some mixture of these features and more. Clearly then objects seem to be desired by programmers even in Lisp family languages.

The best, and most basic, use of objects is to abstract a set of related data and operations upon this data from the rest of the program. Obviously anything an object can do could be done with a properly structured list and some operations that act on it, but the beauty of using objects is that all these functions can be conveniently tied together allowing the programmer to easily alter implementation of the object without breaking other parts of the program unintentionally. Ideally objects can result in smaller and more elegant programs, in terms of the code written, although they may have more run-time overhead. Admittedly there are drawbacks to the object oriented style of programming, which have been already covered extensively by others wiser than me. Although I recognize that the over-use of objects can be a problem, it also seems clear that objects, used in a more limited fashion, can be an excellent programming tool. Why else would developers create object systems over and over again for Scheme?

So if objects can be useful why leave them out of Scheme? There are obviously drawbacks to their omission, as object systems invented independently of each other function and look different, which can make reading code that employs an object system difficult. One advantage of not having a standard system is that it leaves out a lot of clutter from the language in its bare specification form. When object system are built into the language you end up with not only a new keyword for defining a class, but you end up with special ways of defining member functions (how else can you access the “this” reference), special functions to describe inheritance relationships, calling member functions of an object, determining the type of an object, getting and setting member variables, ect. Depending on who designs the built-in system there may be more or less added syntax, but on some level you have added a new “black box” to the language. A black box is a construct that can’t be described in terms of other constructs available in the language. For example if you consider if to be a black box construct then cond is not since it can be described in terms of the if statements. Part of the elegance of Scheme is that it only has a few of these black box constructs, and adding an object system could possibly double the number of “black boxes” in the language.

More important however is that object oriented programming is still young. Admittedly we have had object oriented systems for many years, but even so object oriented languages are still finding new ways to approach objects, both syntactically and conceptually. Compare this to functional programming, which has only a few styles of implementation, and a mathematical foundation to boot. The problem with adopting “officially” one object system into the language is that it discourages experimentation with new ideas. Consider for example CLOS in Common Lisp. Because it is already incorporated in the language, and an official standard, few Common Lisp programmers would consider using a different object system. Despite this CLOS is primitive in several ways, for example it handles name clashing in multiple inheritance poorly*, and unfortunately programmers who have become accustomed to one style of object oriented programming are unlikely to see the benefits of other styles or to dismiss them merely “syntactic sugar”, much the way programmers who are accustomed to a procedural style often react to functional programming at first.

So while Scheme’s lack of an official standard for handling objects may seem to lead to some confusion and inefficiency these shortcomings are offset by the cleanness of the language and the acceptance of the community of different approaches to the problem. Perhaps when object oriented programming matures, and it becomes clear that one way of handling objects that is superior to any others should it be incorporated officially into Scheme, but until that day I feel it is best to let homebrew solutions be the norm.

* according to a conversation concerning inheritance I had with a CLOS user, when a class inherits from multiple classes that contain variables of the same name the new class must explicitly override get and set methods for that variable so that they address a new variable internally. I don’t know how conflicts between method names are resolved. I claimed that this put an unnecessary burden to the programmer, i.e. to ensure that every time any parent class is revised no new variables have been added which might conflict with any of the other inherited classes. To defend CLOS they added that this system could handle both the case when only one copy of a parent class found more than once in the inheritance tree is needed and the case when multiple copies of such a parent class is needed. When I in turn pointed out that such problems had been solved in C++ with the distinction between virtual and non-virtual inheritance with no need to worry about conflicting variable names they seemed insulted, which I think is a case of programmers used to one system seeing any improvements as mere syntactic sugar.

Posted in Common Lisp, Exploring Scheme, OOP | 4 Comments »

An Interesting Project: Graphic Forms

Posted by Peter on March 19, 2006

I ran across an interesting project, in Common Lisp, which promises to make some cool GUI features available under windows. It doesn’t look like it is far along yet, but hey, it has promise. You can check it out here.

Posted in Common Lisp, Windows | Leave a Comment »