Disappointment...

That's the feeling you get a late night when you hit M-/ in Emacs (hippie-expand), and Emacs doesn't use the web page you are looking at in Firefox as a source to auto-complete the word you've halfway written.

I think I need to retrain myself to use the browser built into Emacs.

Some people use an IDE that semantically analyses their code and pops up a suggestion box at every opportunity with an exhaustive list of valid choices.

I program with structural templates/duck types in Python and Emacs lets me complete all sorts of stuff, in comments, from other comments, or from the documentation I'm editing. It'll sometimes suggest some weird stuff, including whole line completions. It doesn't care. I don't care. I guess we're hippies.

Hippie van by ma noelle

It won't complete stuff we haven't shared between each other recently, though. I suppose the Lisp engine would rather get into it than wade through a lot of boring code analysis.

Intuition and design

So I was writing about intution, and somehow got sidetracked into how not to do ranking. Like a review of a game where the reviewer would judge the game on 5 different aspects, game play, sound, story, graphics etc. and then compute a final score based on fixed weights. How you'd fit a smash hit like Minecraft into that, I don't know.

What I really wanted to talk about was design.

Software development is full of design decisions. How do we present the information to the user? How do we let them act on it? How much to show, how much to hide?

How do we structure the code internally? What data do we store, how do we model it, how much do we capture? What other software do we build on?

What interface do we provide to other software developers?

It's a sea of decisions. Although it can be helpful for analyzing some outcomes, you can never hope to get through this sea with logic alone. You can often make reasonable deductions that something is simultaneously a good and a bad idea.

This makes it hard to discuss and evaluate these decisions. Often, we do not have the words, and even if we had, the actual components involved may be interrelated in such a complex manner that you'd never get anything out of studying them directly. You'd have to condense this complexity into something simpler, a process which may take much longer than the design process itself. And in simplified form, you can't be sure deductions are still valid.

So we're left with the intuitions we train for ourselves.

In other words, a prime concern of a software developer should be training this intuition effectively, i.e. input good training data with valid, confirmed outcomes - this works in this situation, this doesn't.

The main road to wisdom in programming, as in other crafts, I believe, is learning by doing. So experimenting or playing around if you wish, and seeing what happens. Not just in a strictly scientific sense where the experiment is artificial and controlled. It may be as simple as recognizing a common problem, working out a possible solution and implementing it and watching it unfold.

Sometimes you can take a shortcut by learning from others. There are various ways to go about this. Reading literature, or source code, or asking questions, or getting them to comment on your work. It's mostly about basic curiosity. Typical developer chat when someone is talking about something new they've done circle around how they made it work and what problems they encountered.

But the annoying thing about learning from others is, again, that it can be hard to talk about these things. Design decisions change when the details change. Sometimes minutia can decide whether one model is better than the other. How do you communicate this effectively? The person you are talking doesn't understand your problem completely. You don't understand the situation the other person was in either. So how do you know that the advice is correct?

I think this is here analytical skills are important. You always have to be sceptical, train your intuition into recognizing discrepancies and evaluate what context causes something to be judged to be good or bad.

For instance, if someone you know to have developed a sound intuition about a certain area find an idea of yours to be cause of alarm, you'd be a fool to ignore it. But unless you possess the same intuition about the problem, it can be hard to deduce exactly what causes the alarm and whether it's a problem in the context.

Frustratingly, it can also be hard to explain the root cause on the spot. Often, there is no explanation in the intuition. And examples can be hard to think up. I've sometimes been asked for an opinion, found cause for alarm but haven't been able to figure out why until several years later. I know that because those alarms haunt me for years, making me feel like a grumpy old man seeing ghosts everywhere.

One example I have is that of software you depend upon, as I touched upon in my previous entry. Over the years, I've come to the conclusion that some dependencies serve you exceedingly well, and others serve you hell.

Probably the hardest part of design is designing for the future. We never know what happens. Still, careful design mitigates the risks.

Stimergy

I've always been attracted to the idea of self-organization by informal means, and there was an article not long ago talking about stimergy in a software development context.

The idea is that everyone somehow knows what to do without talking to each other, but just by doing the work and observing the signs left behind, implicit and explicit.

For instance, how do we add a new page to the app? By copying an existing. How do we know something is wrong? Well, either someone already removed the problematic part so we don't see it anymore, or someone added a FIXME note with the intention of doing it later. Or perhaps it just sticks out so visibly that nobody is in doubt it's bad.

Arrows

I think of it as a room full of arrows lying on the floor.

If you are bad at software development, the arrows will be pointing in random directions, and confusion ensues at every point in the process. You can never sprint, only stumble.

The goal is to make sure everything is tidied up so the arrows point in the same direction, the direction we want to go in. We need to constantly check and revise the code to align the arrows, not only align them to each other but also align them with the evershifting goals of the system. The goal of training newcomers is to teach them to read the arrows and make new arrows point in the right direction.

For instance, if a function is named after how it was conceived, not after how it is currently being used, it's pointing in the wrong direction. When you later try to reason with it, your mind will tend to go down the wrong paths and you'll reach the wrong conclusions, leading to bugs, or at least waste time having to retrace your steps.

You can add a comment to the function, but it's far more powerful to fix the name so it's pointing in the right direction, even if it's more work up front. Arrows matter.

I think this is a powerful metaphor.

Simple processes

I also think it's a powerful method.

Everything is done by engaging with the main activity, programming. It's through this activity that we learn and exchange the acquired knowledge.

Anyone can contribute to the direction. The value of the contribution is correlated with the confusion the contribution leaves in the greater system. It can be negative, but it can also be a stroke of genius, the subtle touch that makes the arrows of the system appear in greater clarity and alignment.

It is dynamic. If a bad decision is made, it's precisely as fixed as the program. The effort it takes to fix the program is the effort it takes to change the decision, no more and no less. Once something enters the program, it takes a life of its own, removed from the egos of programmers involved. The ego might point in one direction, but the code has its own signs.

Yet it's really simple. It works for ants.

Ant by Katja Schulz

Ants

Of course we're not ants. We need to and should communicate. We can pre-align our internal arrows to solve a complex problem much more efficiently than a bunch of ants that are mostly relying on evolved instincts.

But once the program starts, the communication becomes fragmented and loses some of its relevance compared to the work we actually do. You want to know what I did last week? Look at my commit log.

The unspoken truth about managing geeks

This opinion piece was funny.

Last-minute bugs

You've just completed a new big feature.

There's been ups and downs - you started thinking how hard can it be, then discovered the inherent ambiguities, the dissonances with the existing code base, got stuck, pacing the floor of the office, chatting to coworkers, slowly building, discarding and rebuilding refined models, getting stuck and unstuck again and again, then gradually transcending into a coding frenzy, with the code bending towards your goals and your will, constantly in the zone, writing tests, wrapping things up, thoroughly testing everything manually, finally letting some trusted users try it out.

And now you're ready for release. Release!

Except just now someone remarked how something is a little bit odd. Perhaps it's even yourself, the overworked brain having mentally let go of the whole thing and suddenly remembering a little forgotten detail.

Ah. But I just need to fix this. It is simple. The world is my feet. And the test works. I can do this.

Except you didn't bother running the full test suite. Or going through the full test procedure. Or full release procedure. Or checking with the users.

This is the time of the last-minute bug where the little innocent fix turns out to violate an assumption somewhere else.

My piano teacher Karen-Marie once explained to me that finales can be the most dangerous parts for a musician. When you stop thinking ahead, instead thinking about what you've accomplished, letting concentration slip.

When automated tests make sense, and when they don't

Some years ago, I stumbled upon an article about automated testing, "When should a test be automated?" (I'm linking to Google since my previous link is now broken). I found it quite interesting because it, unaffected by all the hype surrounding automated tests, provided some rational guidelines for when to write automated tests and when not to.

A main principle was that an automated test is a waste of time if it doesn't find a real bug upon being run again.

This is a good thing to keep in mind, but since we generally don't know where the bugs are or will be in the future, it's not a very practical rule. So on and off, while working on various web applications, I've been thinking about this.

Why automated tests are good
Praises of automated tests are not hard to find. But briefly: the main practical advantage is that if you change the software, you can easily rerun the tests to see if you have broken something, compared to having to retest everything manually (or just crossing your fingers). If you test the same things over and over again, having it automated can be a real time saver.

There is also a psychological advantage; writing and running the tests make you feel better. Plus you're in, which is always a good thing if you're a little low on the hype factor.

Why automated tests are bad
However, there are also some downsides.

First of all, automating tests takes time.

And unfortunately, it's usually a pretty boring task. It doesn't help that you often have to test things manually too while experimenting or as a sanity check, so you pay the cost of testing twice.

And the tests, while radiating nice comforting feelings, also have a tendency to give a false sense of security. The tests are running, so it works, right?

So you always have to keep in mind that automated tests only test part of what makes the code correct. You could argue that this is a sign of a too lazy test writer, I think that's the main driver in test-driven development, but really, you can't foresee everything (pesky user clicks the wrong button), and if you try to, you end up with so detailed tests that you can't change the program without having to the rewrite the tests completely. And then they have lost their main automated advantage.

Besides, an automated test only confirms what you already suspect, it doesn't tell you if the software is creating real value for its users. So you still need human testing.

Furthermore, even if you don't test everything, in my experience you still end up with a large amount of test code, as much or more as the actual program. There's a hidden cost in this; many changes to the program will also require adapting the tests. In other words, code maintenance take longer. People don't think about this, but imagine you show me 100 lines of your code, and I told you I could reduce it to 40 lines with no ill effects on readability or performance? Would you take that? What if those 60 lines I take away are your tests?

The trade-off
With the understanding that automated tests have good and bad sides, if you value your time, you should evaluate the situation on a scale from should-automate to should-not-waste-time-automating.

Circumstances pro automation
Hard to setup tests. If you have to go through ten steps to be able to test a small change, testing quickly becomes a tedious chore. Usual issues in web applications are getting test data prepared, or being able to intercept email sent by the system, or testing error scenarios.

Important corner cases. Testing manually that a new button does the right thing is not tedious. But it is tedious if you have to try 20 different combinations of inputs to see they all work.

Team of disparate people touching the code. Most code have some implicit assumptions that cause bugs if violated. When multiple persons are modifying it, there's a greater chance of bugs, and smaller chance that each remembers all the corner cases that need to work. Note that you don't necessarily need more than one person on the development team to fall into this category. Add a gap of a year or two in the development, and most people will have happily forgotten the pecularities of the project in the meantime.

Complex code. If the code is complex, it's likely to be needing bug fixes that don't change the expected output, which is good because then we don't have to change the tests, and it's also likely that it's harder to see what the expected output is supposed to be, which can be conveniently solved by writing it down in a test.

The code is seeing changing input data. Modifications to complex code is one source of bugs, but another is changes in the input data. You test it today, and it works fine, but tomorrow your data source decides life is too quiet and changes something that violates your assumptions about the input. Again, this requires bug fixes that usually don't change the expected output.

The code is an API with no UI. If there's no UI, you need to write code to test it, and then you might as well automate it anyway. With Python or any other language with a REPL, this is not entirely true, though. I often test small snippets I'm not yet confident are bug-free in the interpreter.

Circumstances tipping the scales toward manual tests
Simple, easy-to-follow code. If it's simple, there are fewer opportunities for bugs and thus a greater chance automation will be a waste of time. For instance, code using templates to spit out HTML is in most cases trivial, a div here, a heading there. You could add a test to find out whether the div and the heading are there, but it's trivial to see that by mere inspection, and if you've checked the way they look in the browser, there's little chance they're magically going to disappear later.

Localized changes. If changes only affect things in their near vicinity and have no impact on the rest of the software, it's easy to test them manually. For example, consider a web application made up of several independent pages. When you're working on one page, you can just reload it to check it's fine and ignore the rest of the program.

Manual tests are simple to set up. If all it takes to run the test is a reload in the browser, the relative cost of automation is high.

Hard-to-quantify issues are important. If look and feel are important, there's currently no substitute for a human. Imagine setting up a test for an animation - how do you test should look good?

The functionality sees lots of changes and experiments. If the functionality of the software keeps changing, maintaining a set of automated tests is a burden and is more likely to turn you into the grumpy nay-saying change-resistant bastard it's important not to be.

The software sees no changes. If nothing changes, there's little opportunity for bugs and little opportunity for rerunning a set of automated tests. This situation sounds strange, but actually in my experience there's a certain polarity in software maintenance; some things change all the time, others are truly write-once. Of course, this observation doesn't help a lot when you're starting on a project and don't know where it's going to end.

Only one person. Again, this is an important special case. If there's only one person working on a given piece of software (or perhaps module), that person is likely to know exactly what needs to be tested, diminishing the value of a broad test suite.

Minor errors are not critical. This sounds sloppy, but in reality with many web sites, it creates more value for the users and customers if you focus on having the major paths hit the target than being 100% correct because errors outside the major paths can be fixed quickly when discovered.

Of course, most of the above points aren't independent. For example, say you're writing a little script to convert your music collection into a standard format. In that case, you're the only developer, you're going to experiment and change the script until you're happy, an error is not critical since you're around to supervise the process, and you don't expect the script to run more than once when it's finished. Would you write automated tests for that script?

Unit testing versus acceptance testing
I have an extra remark regarding the level at which the tests are performed. There's been a lot of talk about unit testing, in fact most automated test frameworks are called unit testing frameworks, but the reality is that unit tests represent little any end-user value in themselves because they are just providing evidence that some component that nobody will ever run by itself is working if it is being run by itself in the lab.

A more appropriate level for end users is acceptance test where you are testing requirements to the system itself. For instance, the site should enable authorized users to add new products which should then show on this page or when a customer pays for some goods, an order should show up in the system. These are the kind of things that when failing will be sure to cause trouble for the users and thus you.

Of course, unit tests can add value indirectly by highlighting bugs early in the process where they are easier and thus cheaper to debug.

So where are we?
In my experience, when it comes to web development, there are some instances where unit tests make sense for a complicated component, and also some examples of vital processes where acceptance tests make sense because testing those over and over gets annoying. One example could be a sign up feature. Nobody tests that by themselves, because you only ever sign up once.

But otherwise a lot of web development fit the above criteria against automation quite well. Hence, contrary to what the hype says, I do believe automation in many cases would be a waste of effort.

Note that this is not an argument against testing. It's an argument against automating the testing.

It's also not a universal observation. For instance, if you're writing library code all day long, your world is likely looking different. Or if you're Google or another big web site, then the operating conditions are different - as a part of a roll out it's important to have easy reassurance everything is behaving as intended because even small mistakes can be painful if you have a million users. But most web sites don't need to scale.

Also, don't forget that the purpose of testing, whether it being manually or automated, is to find and fix bugs. Thus preventing the bugs from happening in the first place is perhaps the best place to focus your attention. Usually the key is easy-to-understand code with few implicit assumptions and attention to detail, making sure corner cases are handled well.

3D in the browser

We've recently done a project where we needed to do 3D visualizations in the browser. It's not incredibly fancy, mostly showing walls and floors of rooms and some stuff in them.

Before we started coding, we were discussing what platform to run it on. After some going back and forth, we ended up deciding that despite the 3D requirement, going for a web app would probably pay off in the long run because we then don't have the hassle of installed software and perhaps worse, the problem of predicting what particular platform Microsoft will declare obsolete next year.

So in the end, we settled on the web, going for HTML and canvas for the 2D part and WebGL, the new OpenGL standard for the web, for the 3D.

There's only one problem, WebGL was so new that at the time, only Chrome had a release with support. Not long after, Firefox joined the ranks. But still, it's a technology you can't rely on unless you can control the user base and ensure they install the right browser.

Similar problems kept the 2D HTML canvas back until a project like excanvas turned up. Until Internet Explorer support is there, new standards are a no-go for many.

Thus Martin has been working on and has now released JebGL, a WebGL emulation layer for older browsers and most importantly Internet Explorer, back to IE 6.

It relies on the OpenGL support in Java, translating WebGL calls into JOGL calls in a Java applet. We've had a look at other options, such as using the Flash 3D API, but we needed support for some of the fancy stuff in OpenGL so it would be a pretty huge task to try to emulate that in a completely different API (on a lower level, that's what Google is doing with the ANGLE project to try to work-around bad OpenGL drivers).

So as long as Java is in, you can now rely on WebGL (well, barring bugs and missing features in JebGL). And if Java is not there, there's at least the option of asking people to install it, most browsers seem to handle that gracefully these days, rather than requiring them to switch to a completely different browser.

Organizational patterns

We went to a seminar not too long ago in a local interest group on agile software development. One of the speakers was Jim Coplien. He's a good at it, speaking emphatically on what to do and not to do.

Also he had the interesting point that if you're going to try to change what a software development organization is doing, process improvement programmes turn out to be less effective than one thinks because processes tend to come from the underlying structures. So if you don't change them, people fall back after some time.

For instance, if you have the problem that the developers aren't good enough at coming up with what the users need, then instead of installing a process where they must fill in check lists to try to force them to listen, maybe you need to look at the fact that they aren't communicating directly with the users, but perhaps through a manager who dictates what should happen. If you change the manager's role, maybe the rest will follow automatically.

Anyway, he talked at some length about some organizational patterns he and others have come up with, which was interesting because many of the seem to describe what we're doing at IOLA. He's collected a big bunch in a book he's written (together with Neil B. Harrison), "Organizational Patterns of Agile Software Development".

If you're interested in software development and what works when you structure the work, the team and relationships to stakeholders and users, it's worth a read.

What's interesting in it is that most patterns may be profound in their effect, but are really lightweight in what you actually need to do. For instance, if you have the problem that developers are interrupted too much, you can sacrifice one person and make him a firewall that people have to go through. Simple as that. No big piles of lists and specifications and meeting minutes and memos necessary.

I think the big downside of the book is that it's not very friendly written. It's in the style of a reference in many places with only little background rationale and so many crossreferences that you need to have read most of the book to understand what's going. This probably works well for Jim Coplien when he travels around on his research and consultancy campaign to fix sick organizations, but I think the audience for a gentler style would be much higher.

Still, it has material I haven't seen elsewhere and it's actually based on research on succesful software teams rather than just handwaving.

GNOME 3 is nice

So GNOME 3 is out and I decided to give it a try with the Fedora live CD (or should I say USB stick). I must say I'm impressed. Overall it's really clean and just works. The new application chooser is downright brilliant compared to the old menu. I've tested some of the previews of GNOME Shell, and the thing has come a long way.

Of course, Alt-tab is still broken, the default (and only) theme is a bit top-heavy, window movement wasn't quite smooth on Intel GMA 950, there was the occasional crash of the new settings stuff, but all in all not bad for a .0 release. And the new direction means that there are now people working on the core user experience, something that appears to have been more or less neglected for a very long time.

Plus much of the new shell stuff is written in Javascript so should be easy to hack on in case the designers don't come to their senses any time soon (there's already an Alt-tab hack).

A take on good, reusable Java design

This is just too amusing not to pass on.

Flot 0.7 released!

I just released Flot 0.7! Highlights are multi-axis support, area charts and a pie chart plugin, and a big bunch of bug fixes, including an IE 9 fix and reworked replotting support that reuses the canvas elements to put an end to the memleak troubles. The complete list of changes is here. There are some API changes from the axis rewrite.

In related news, I've moved the source code repository to github to make it easier for other people to contribute.

It has been clear to me for some time that more of the things that are currently inside core Flot needs to move to plugins. As time goes, more specific things turn up on people's wishlist, and right now it's just not possible to satisfy them all without bloating the core. Having specific feature sets in a plugin also makes it much easier for people to customize them a bit by hacking the plugin code without having to maintain a delta with the whole of Flot.

Fortunately, some people have expressed interest in helping out, partly due to me taking a hiatus from the mailing list for some time last year, so with a bit of luck some of these things may happen in a not too distant future. Maybe we'll even get a better hyperlinked format for the documentation, something that has never really managed to climb to the top of my list of priorities so far.

Bulk inserting Django objects

I've added another Django utility member to my garden of Python, bulkops.py.

Currently, it consists of two functions, insert_many and update_many. They are intended to solve the problem of inserting a big bunch of objects in Django. If you use the standard save() or create() methods in Django, you get one SQL query per object. With thousands of objects, you'll simply drown in overhead in executing all these queries.

There are some alternatives out there, but unfortunately they are pretty complex and specific to a previous version of Django. Instead this code simple makes a list of tuples with the fields on the model and feeds that to the executemany() function in the DB API. Usage is really simple

for x in seq:
    o = SomeObject()
    o.foo = x
    o.save()

becomes

l = []
for x in seq:
    o = SomeObject()
    o.foo = x
    l.append(o)
insert_many(l)

It's tested to work with Django 1.2 and doesn't really touch the ORM internals so should be safe for at least some future versions.

Sadly, there's no support for associated many-to-many relationships. I actually needed that myself, but I don't think there's an efficient way of getting the primary ids when you do a bulk insert with the Python DB driver API. So one has to use a select query anyway afterwards, and then it was actually more convenient to do it by hand by rearranging the logic in the caller.

Cache busting in Django

One of the really annoying things in web development is dealing with caching. Browsers incorporate an elaborate caching scheme to speed up common browsing operations, which is a really great idea. But a side-effect is that when you release a change on a site, you risk that some people get the new HTML while their browser is still using an old CSS or Javascript file because their cache hasn't timed out yet.

It has bitten me several times during development too, some browsers have really aggressive caching schemes. Debugging problems caused by a stale CSS/JS file is just not fun.

One way to fix it is to use a framework to run the CSS and Javascript through a combiner/minifier that at the same time outputs a versioned filename, e.g. by hashing the contents. There are even a couple of Django projects going down this route to make it happen automatically.

I haven't found one I liked yet, though. You have to modify views or templates and fit your paths into the system, it doesn't help with changing images, if you use third-party code you have to modify that too, and the minification step makes it pretty hard to debug Javascript problems; even if you isolate it to live, problems happen on live too, sometimes.

Really, if we take a step back from the minify dream, the simple solution here is to append some kind of version number to the file. Then new HTML should have filenames with new version numbers so clients can't used their old cached files because the paths don't match.

I've found a really simple way to automate this transparently for a project by modifying one line in settings.py. The idea is to hook into Django with a middleware class, search the generated HTML for local URLs and replace them with URLs with version numbers. This way, we can catch all CSS and Javascript references as well as images no matter where they occur.

Since common file systems don't provide version numbers on the files they store, we'll use the modification timestamp instead. And instead of renaming files to contain the version number, we'll just append a ?_=timestamp to the URL:

  <img src="/media/images/logo.png">

becomes

  <img src="/media/images/logo.png?_=4321234">

Everything after the ? is ignored when looking up in the file system by the static file web servers I've tested so far. So it just works.

As a bonus, when this is running, you can safely modify the web server serving the static files to set an expiration date far out in the future for files requested with ?_=xxxxx. This allows an aggressive, yet perfectly safe caching scheme to be employed by any upstream cache, including the browser.

The above trick can be done in about 50 lines of Python. You can grab the little self-contained middleware module modtimeurls.py from my brand-new Garden of Python. It has been in production on several sites for over a year. The only problem I've found so far was with a PNG transparency fixer for IE6 that assumed .png files would end with .png rather than .png?_=xxxxxx.

Emacs learning curve

Think this is old. But still funny.

Software development and the waterfall model

As mentioned, I recently read The Design of Design by Fred Brooks.

He covers a lot of ground, albeit not in any rigid fashion. As such, it is not a handbook, but more a collection of thoughts about the design process, most grounded in personal experience. Fred Brooks knows his stuff.

An interesting point he makes is that the waterfall model of software development is completely flawed and should be abandoned.

In short, the principle in the waterfall model is reason - good solid thinking: first we figure out all what's needed in a given system, then we figure out all aspects of how to build it, then we build it.

He has several arguments for why the model is flawed. One of the simple arguments is that it doesn't model how good designers work. Naturally, if you want everyone to follow a certain mode of operation, it would seem like a good idea to copy the mode followed by the good ones.

For me, the biggest flaw of the waterfall model and its companion, the big fixed-price project, is that it requires one to make the decisions up front where the uncertainty is greatest and the consequences of each decision most remote.

This shows up even in the earliest phases, e.g. when discussing whether to include a certain requirement or not. Inclusion may mean that the project will run over budget and schedule and produce a completely impractical never-used feature. However, the budget overrun may not even be visible in the design phase, but only show up late in development, and the futility of the feature may not show up until years after deployment where it is discovered accidentally that nobody uses it.

An important point in the book is that the design process is a learning experience, a process where one learns what's needed and what's possible. And what's elegant and what's good in the particular circumstances. I'd add that the same is true of the embodiment of the design, the code itself. Often people won't understand a system properly until they sit in front of it and have to start doing what they need to do. Deployment is always a learning experience. And similarly, it can be hard as a developer to assess the consequences of a paper design. When the code is written, maybe that fancy idea everybody liked beforehand isn't that hot afterall.

If one doesn't allow for learning, both for the receivers of the system and the people who deliver it, it's really difficult to get the software right.

By this thinking, the rational model becomes a bureaucratic way of delivering what might just turn out to be the wrong system.

Javascript and Emacs Lisp

Warning: if you don't subscribe to the One True Religion, this probably won't interest you.

With my recent affair with Elisp and the development in GNOME where some influential people are dropping C in favour of Javascript for GUI programming (we may wonder how many kilolines of boiler-plate C code they've churned out to reach that conclusion), I couldn't help thinking about replacing Elisp with Javascript in Emacs.

First a bit of introduction. Emacs is an incredibly old and incredibly smart text editor with a smallish C core and most of the intelligent stuff written in a Lisp dialect called Emacs Lisp. Lisp is a very flexible language that intermingles data and code. Making a system built on it extensible is relatively straight-forward. This is probably the main reason behind the success of Emacs, the editor has been extended in literally thousand of directions over the years. If you read my previous blog post, I'm using what nowadays is dubbed aspect-oriented programming to hack a built-in Emacs Lisp module.

It's my impression that even within the Emacs community there are few people who think Emacs Lisp is a great language. It's not Common Lisp, it has some weird idioms and as a small Lisp dialect it still lacks the simplicity of Scheme.

However, weird idioms aside, in my humble opinion the big problem with Emacs Lisp is momentum. When Emacs was started, Lisp was hot. People were thinking that Lisp would be the future. Back in those days, they even had hardware that ran Lisp! The mind boggles. Lisp was an obvious choice for a dynamic extension language.



Today much has changed and few people are using Lisp. The obvious choice for a dynamic extension language today is something else. Basic, Python, Lua, etc. Or Javascript.

A die-hard Lisp fan would think the solution is another Lisp, but the official GNU Scheme embeddable language Guile looks pretty dead to me. While learning Lisp is a worthwhile goal that will definitely teach you some lessons, I think there's good reason why most programmers aren't writing parenthesized lists all day long.

By switching to Javascript, Emacs would once again be ahead of the curve. And the Emacs maintainers would be relieved of having to maintain a language interpreter with its related problems. For instance, and I don't know if things have changed recently, the garbage collector in Emacs used be pretty basic. Probably because when it just sort of works, nobody wants to mess with it anymore, people naturally want to focus on writing the best editor possible.

I don't think the idea would fly, however, for several reasons. There is an enormous body of Lisp code out there for Emacs. And since Lisp is the extension language for Emacs, everybody contributing to it is a Lisp hacker. It's hard to see how this actually pretty large group of people making up the community would accept anything else than a Lisp. And in the past, even proposals to reform Emacs Lisp or switch to other Lisp dialects have all failed.

To add to that, Javascript has a bad reputation, especially among people who haven't done any client-side web development lately.

Thus I think the only way Javascript in Emacs could happen meaningfully is by targeting the interpreter: replace the Emacs Lisp interpreter with a Javascript engine like Spidermonkey or V8 and a bridge that translates the Lisp and makes Lisp symbols available to Javascript and vice versa. So all Lisp code would run unaltered, and everyone can continue writing Lisp as they do today.

The aim would be less code to maintain in Emacs and the benefit of a well-maintained optimizing engine that is getting faster every day, with several independent free ones to choose from.

I think you could maybe sell that idea to the Emacs maintainers. A project like CEDET that replicates the code parsing engines of modern IDEs would definitely benefit from a speedup.

As a side-effect, it would then be possible to extend Emacs with pure Javascript. And a growing horde of web developers will suddenly find it much easier to extend Emacs.

I had a very brief look at the C source of Emacs, and it looks like you would have to rewrite and port maybe 10-20.000 lines of C code. So I think it's doable.

Of course, there are several open questions. How hard is it to do the translation? Can a Javascript engine actually run Lisp-translated-to-Javascript code efficiently? Is it a good idea at all or just a futile exercise in chasing the latest fad of the day? I am not sure. Javascript has been around for a decade and the web is still growing as a platform, so I don't think it qualifies as a fad. And GNOME is currently betting its future on it.

Another option would be to just embed the Javascript engine on top of the current core. As far as I know, that has been tried already with Python, and didn't work out. I don't think people will use it unless it is bundled with Emacs and sitting right in the core. After all, most customizations start out with a little hack building on existing code in Emacs.

Django shell in Emacs

Yet another recipe. One of the neat things with Python is its REPL, the interactive shell. When working with Django models, you need to setup a couple of things before you can use the Python shell. Django provides a shell command for doing this, but it makes it difficult to use the built-in Emacs Python REPL.

I was talking to Anders about it the other day when it occurred to me that it could be a quite powerful environment.

The main problem with the Python shell is that it lacks easy reload support. So you start it up, import the module you're writing and run a couple of functions to test it. Discover an error, fix it with Emacs, save, and then you're in trouble because you can't reload the module in the shell. You have to quit the shell, restart it and reimport the module. History helps but it's still silly (well, actually it's a blatantly stupid oversight, maybe in the top 3 of things to fix in Python to speed up development time).

But with a shell integrated in Emacs, I could solve this problem, right? So I set out to write the necessary Elisp. Dump the following in your .emacs, and it should magically work:

;; run Django shell when editing Django Python code

(defun get-file-in-upstream-dir (location filename)
  (let* ((dir (file-name-directory location))
         (path (concat dir filename)))
    (if (file-exists-p path)
        path
      (if (not (equal dir "/"))
        (get-file-in-upstream-dir (expand-file-name (concat dir "../")) filename)))))

(defadvice run-python (before possibly-setup-django-project-environment)
  (let* ((settings-py (get-file-in-upstream-dir buffer-file-name "settings.py"))
         (project-dir (file-name-directory settings-py)))
    (if settings-py
        (progn
          (setenv "DJANGO_SETTINGS_MODULE" "settings")
          (setenv "PYTHONPATH" project-dir)))))

When the Python interpreter is started, the hook above looks for a settings.py file in the parent directories. If one is found, it sets up Django and also sets the PYTHONPATH to the project toplevel.

Use the code with C-c C-z to show interpreter window, C-c C-c to evaluate buffer, C-c C-r to evaluate region, etc. (they're in the Python menu at top).

You'll probably want the following snippet too, it disables the warning that a Python process is still active when you quit Emacs:

(add-hook 'inferior-python-mode-hook
          (lambda ()
            (set-process-query-on-exit-flag (get-process "Python") nil)))

I think python-mode should do this by itself, but in the end I didn't have the energy to report a bug and fight with the Emacs maintainers.

So did it fix the import problem in Python? Of course not. The shell inside Emacs still can't reimport a module. Sigh. But at least it's easier to test the current module because you can easily evaluate the whole buffer (which works fine).

Flot 0.6 released!

I just released Flot 0.6! Highlights are a new plugin system, or at least the beginnings of it, lots of new features (most of them in plugins) and bug fixes, including fixes for IE 8. Check out the complete list of changes. Note that there are some API changes.

There were still some things I would have liked to do before the release, but it has been dragging along for far too long.

Otherwise the future is looking bright. The plugin system is not complete yet, but I think the general direction is OK, and will allow third parties to extend Flot considerably without going through me.

On the technological front, canvas support is gaining traction in the browsers and even support for what used to be advanced stuff like text rendering and export to image is getting more common place. I suspect we will soon have to think hard about whether to continue with the current labels-are-HTML approach or switch to rendering them directly on the canvas.

Completely unrelated to Flot: I haven't blogged for some time. It's entirely due to lack of time, I have been preoccupied, or gone crazy is probably closer to the truth, with another hobby project which is still some effort away from being useful.

Using CSS3 box-shadow with IE

At YayArt, we show a lot of images. An important part of the presentation is subtle drop shadows.

My original solution to that problem was a Javascript hack that inserts a couple of 1 pixel wide divs at the edge of any element with a shadow class. However, although I have tweaked it several times, it has always been a bit slow. For instance, for the shop front page we're currently rendering more than 60 elements with shadows which takes a couple of seconds with Firefox and Firebug on my old trusty PIII 950 MHz laptop. I'll note that this is actually an order of magnitude faster than the (more general) code I based it on from somewhere else. The really tricky part was decent-speed IE 6 support because that actually required measuring the DOM elements.

However, Firefox 3.5 was released about a month ago, and one of the good news was support for the box-shadow CSS3 property. It's a relatively simple property for adding drop shadows. So given that Safari has it, and has had for some time, and most Firefox users will soon have it, it starts getting interesting. For reference, about half of our visitors are Firefox + Safari (+ Chrome which uses the same rendering engine as Safari).

If you search for box-shadow, some are already using it to add a bit of extra embellishment. However, on YayArt, it's an integrated part of the design. So I had to figure out how to make it work on all browsers.

I ended up with a three-way solution. For recent Firefox/Safari/... I use box-shadow. For non-box-shadow supporting browsers, I fallback to the old hack. And for Internet Explorer, I'm using a new hack based on the proprietary filter stuff in IE.

Here's a quick sketch. I put this in the general stylesheet:

.shadowed {
-moz-box-shadow: 2px 2px 3px #969696;
-webkit-box-shadow: 2px 2px 3px #969696;
}

This makes it work with browsers based on the Gecko and Webkit engines. Some people recommend adding a plain box-shadow line, but I need full control over which browsers are responding, and besides it seems to defeat the purpose of the browsers using the -engine- prefix. Here's a demo:

If your browser is recent enough, you can see a shadow here.

Now for the fallback, I need to discover in Javascript whether the browser is using the CSS:

function supportsBoxShadow() {
var s = document.body.style;
return s.WebkitBoxShadow !== undefined || s.MozBoxShadow !== undefined;
}

The assumption is that the symbols will be defined only if the browser actually draws the shadow. This seems to be about right, the only exception I've found so far is the Webkit port to GTK (there's a bug report open).

Final point is Internet Explorer. As it turns out, there is a DropShadow filter. It's not terrible useful for this purpose, however. There are no soft edges, and it shadows the content rather than the containing box (it simply redraws the content with an offset in another color). So for my purpose I need to set a background color to ensure it's drawing a rectangle and not a text shadow. You could probably hack the lack of blurring with another filter, but in the end I went with a simple three-one-pixel-wide-lines approach that looks like my old hack but is much faster (updated, replaced the filter with a more appropriate one as suggested in the comments):

.shadowed {
background-color: #fff;
zoom: 1;
filter: progid:DXImageTransform.Microsoft.Shadow(color='#969696', Direction=135, Strength=3);
}

This is served for IE only. The zoom: 1 is a hack to ensure that the element gets a layout (to work around the usual IE 6 bug).

So there you have it. Practical drop shadows. I will have to adjust the code a little bit as the other browsers get support, but I can live with that.

Emacs 23...

... is out and has been so for over a week. Why didn't somebody tell me?

In any case, I upgraded to Emacs 23 on my Debian unstable laptop. This means a new font. I put Emacs.font: Bitstream Vera Sans Mono 9 in my .Xresources file and am enjoying it so far. It's a bit muddy so I wasn't sure at first, but after having stared at it for a while the old font is just too skinny. Note that configuring the font in .emacs is still broken (sigh), in that it causes the window to jerk when you start Emacs.

I've been lobbying for changing the defaults over to make copy-paste between Emacs and other X applications (from this century) work properly. Apparently, (setq x-select-enable-clipboard t) is now better, but still not according to the spec. And it's not default yet.

I noticed some Tramp updates (for accessing remote files) in the NEWS and decided to give it a spin for the first time. Try C-x C-f someserver:myfile.txt and with a little patience, you're editing a remote file locally! Very neat.

Feeling adventurous I also tried IDO, an update of iswitchb that works with file find (so overrides C-x C-f). I'm turning it on with

(ido-mode 1)
(setq ido-enable-flex-matching t)
(setq ido-use-filename-at-point t)

replacing iswitchb and ffap settings. We'll see how it goes.

Finally, Anders mentioned a new Javascript mode, js2-mode, which I'm definitely going to try out.