Luke Maurits

The NetBSD project has “gone 2.0″ recently, with the launch of a blog and twitter, uhh…channel? I’m not sure what the proper term there is. This was bought to my attention by a blog entry by well-known NetBSD advocate Hubert Feyrer (Deutsch, English).

The most recent entry in the NetBSD blog is about the project’s participation in this year’s occurrence of Google’s Summer of Code. It links to a list of projects, two of which caught my eye.

First up is Lloyd Parkes’ “Miniaturise NetBSD” project, which aims to “produce a boot image for a system with no more than 4MB Flash / 16MB RAM, run a useful NetBSD router with DHCP client/server, IPv6 route solicitation/advertisement, PPPoE, and an 802.11a/b/g WPA access point”. I like this sort of project in part because I just find small, underpowered systems kind of fun, but also because it guards against bloat and bad design. If it’s a straightforward project to strip a system down to those sorts of specifications then it says something about the system’s simplicity. I feel like having this sort of anti-bloat guard for NetBSD is particularly important just now because there seems to be a gentle trend away from the project’s traditional lean and simple approach lately – putting an unusual web server like bozohttpd into the base system is one obvious symptom of this, and I fear that the recently announced Desktop project might accelerate this trend (although so far their discussions in the mailing lists seem to have been harmless and helpful).

Secondly and even more interestingly is Nhat Minh Le’s “XML command-line utilities for NetBSD”, which “aims at providing NetBSD with a lightweight, consistent, set of stream-oriented XML utilities, inspired by traditional Unix programs such as grep, join and sed, hopefully opening the way for similar support in other mainstream systems”. I am not a huge fan of XML and I much prefer to use formats like CSV, INI and YAML where they’re appropriate, but there’s no denying that XML is now the de-facto standard format for, well, anything. This standard becomes a whole lot more palatable with equivalents to grep and friends to make working with XML from the command line nice and easy. Furthermore, this project shows that this spirit of Unix is still alive in NetBSD (although, as the project description notes, this isn’t really a NetBSD project as such, because these tools will be useful on all platforms), and that makes me very happy.

Changing topic just quickly, in my last entry I mentioned in passing that:

I’m a big fan of reusing HTTP wherever possible because it makes life easier for developers.

I recently dug up the article Why HTTP by Timothy Fitz, which played a strong role in convincing me that this was a sensible stance to take. I don’t bring this up now arbitrarily, I want to write some more about it in a later entry, so this is just to get you and I both primed). Until then!

A while ago I posted the code for a simple Unix shell written in Python. Not too long after this I expressed my delight at discovering the Python standard library’s cmd module. It turns out that these two concepts go together really well. Here’s a much nicer simple Unix shell, less than 100 lines long. It still lacks input/output redirection or pipes, but by virtue of subclassing Cmd it has gained a command line history and tab completion. The tab completion is not the smoothest you’ll ever see – this is basically the first thing I could get almost working – but I think this solidly demonstrates that Cmd makes a fine base for a Pythonic Unix shell.

from cmd import Cmd
from glob import glob
from os import access, chdir, execv, fork, getenv, getcwd, listdir, wait, X_OK
from os.path import exists, isdir
from sys import exit

class Shell(Cmd):

        def _substitute_env_var(self, token):
                if token.startswith("$"):
                        return getenv(token[1:], "")
                else:
                        return token

        def precmd(self, line):
                words = line.split()
                words = map(self._substitute_env_var, words)
                return " ".join(words)

        def _get_executables(self):
                result = []
                for dir in getenv("PATH").split(":"):
                        try:
                                result.extend(filter(lambda(x): access(dir+"/"+x,X_OK), listdir(dir)))
                        except OSError:
                                pass
                return result

        def postcmd(self, stop, line):
                self.prompt = getenv("PS1", "$ ")
                self.executables = self._get_executables()
                if stop:
                        return True

        def completenames(self, text, *ignored):
                return filter(lambda(x): x.startswith(text), self.executables)

        def completedefault(self, text, line, begidx, endidx):
                prefix = line.split()[-1]
                return map(lambda(x): x[len(prefix)-len(text):], glob(prefix+"*"))

        def do_cd(self, line):
                if not line:
                        line = getenv("HOME", "/")
                try:
                        chdir(line.split()[0])
                except OSError, (errno, strerror):
                        if errno == 2:
                                print "Directory %s does not exist." % line.split()[0]

        def complete_cd(self, text, line, begidx, endidx):
                prefix = line.split()[-1]
                return map(lambda(x): x[len(prefix)-len(text):],filter(lambda(x): isdir(x), glob(prefix+"*")))

        def do_cwd (self, line):
                print getcwd()

        def do_EOF(self, line):
                print ""
                return True

        def default(self, line):

                argv = line.split()
                command = argv[0]

                found = False
                path = getenv("PATH")
                for dir in path.split(":"):
                        if exists("/".join((dir, command))):
                                found = True
                                if(fork() == 0):
                                        execv("/".join((dir, command)), argv)
                                else:
                                        wait()
                if not found:
                        print "%s: Not found" % command

shell = Shell()
shell.prompt = getenv("PS1", "$ ")
shell.executables = shell._get_executables()
shell.cmdloop()

Lately I’ve been trying to make an effort to use Python’s functional programming features, like map and filter, instead of cumbersome piles of fors, ifs and appends, which is evident in this code.

Well known German NetBSD guy Hubert Feyrer (personal website in English or German) made an entry in his NetBSD blog the other day discussing some user-visible changes in the upcoming NetBSD 5.0. It looks like something of a mixed bag to me, although it’s mostly good.

On the good side: audit-packages and download-vulnerability-list are scheduled to become part of the base system, which I’m very pleased by. I think this functionality is fantastic and amazingly rare (only FreeBSD has something similar, to my knowledge), so I’m glad to see it made easier. /tmp will now be per-user via the use of magic symlinks, which is fairly boring but I suppose has positive security implications. The old DHCP client has been replaced by a simpler one which uses about 50% as much memory – the absolute saving is quite small (1058 vs 548 Kb), but if there is no loss of functionality or stability then it makes sense to do this. Finally, the bootloader will have a config file. I’ve never had to need to configure the bootloader (which is delightfully simple) past its defaults, but this harms nobody is actually something I had simply assumed was already there. All in all some nice, simple changes – evolution, not revolution, as the OpenBSD folks are fond of saying.

On the bad side: NetBSD will soon have a webserver in the base system. Hubert himself says “Seriously, I have no idea why this is” and I couldn’t agree more. The only other OS I’ve known which does this is OpenBSD (which comes with a heavily patched version of Apache 1.3.x), and I’ve never really enjoyed it. Frankly, a webserver has no place in the base system of an OS. It’s something that belongs squarely within a packaging system, like pkgsrc or ports.

Not only is this decision aesthetically displeasing, but the choice of webserver itself borders on the surreal. The OpenBSD folks at least have some excuse for shipping a webserver, in the sense that they are acting in keeping with their “theme” of being a high-security OS by shipping a version of Apache which they feel is more secure than the off-the-shelf version (the Apache devs have refused to put some of the OpenBSD security patches into the official Apache). NetBSD 5.0 will ship with, wait for it, the bozotic HTTP server: a small and low-featured webserver that is somewhat similar to Acme software’s thttpd, with the primary difference that about one thousandth as many people have ever heard of it. I suppose one could make the argument that NetBSD are sticking to their own “theme” by shipping a very small, simple, minimal-dependency web server that would probably be ideal for embedded devices, but I don’t quite buy that – bozohttpd is available in pkgsrc and long has been, which makes the situation different to OpenBSD because their customised Apache is not in their ports tree.

I just don’t see the point in this. Not that I have a problem with bozohttpd itself – small and simple servers like it and thttpd were an inspiration for me when I was working on comanche – but of all the servers to jam somewhere one doesn’t belong, why pick one that so few people are familiar with and will be happy with? It’s much harder to find support for bozohttpd and it’s much harder if not actually impossible to get it to do things that a lot of people used to Apache are going to want it to do, i.e. any sort of dynamic content faster than CGI. I can’t see it getting used much, if at all, so why bother putting it in? It’s such a small thing in itself that it would seem ridiculous to call it bloat, but it sets a precedent of letting in stuff that nobody needs which, if maintained for a few release cycles, could lead to genuine bloat.

Unrelated link drop:
An entry in someone’s Livejournal about “Pypes”, an interesting Python object that overloads the | operator (usually used to perform bitwise or operations on integers) to apply a function passed to the constructor. The effect is that you can mimic the familiar and much loved “pipe” syntax of Unix shells. A brief taste:

>>> double = Pype(map, lambda x : x * 2)
>>> sum = Pype(reduce, lambda x,y: x+y)
>>> range(7) | double | sum
42

Interesting and clever stuff!

Here is an absolute bare-minimum Unix shell written in just 36 lines of Python
(including a few blank lines). It doesn’t do pipelines, input/output redirection,
tab-completion or have a command history, it’s just a really light wrapper around
the fork, execv and wait system calls, using the os
module. The only internally implemented function is cd because a shell
is a painful thing to use without it.

from os import chdir, execv, fork, getenv, wait
from os.path import exists
from sys import exit

while True:

    # Get input
    prompt = getenv("PS1", "$ ")
    try:
        input = raw_input(prompt)
    except EOFError:
        exit()

    # Parse input
    argv = input.split()
    command = argv[0]

    # Do inbuilts
    if command == "cd":
        if len(argv) == 1:
           chdir(getenv("HOME", "/"))
        else:
           chdir(argv[1])
    # Do external
    else:
        found = False
        path = getenv("PATH")
        for dir in path.split(":"):
            if exists("/".join((dir, command))):
                found = True
                if(fork() == 0):
                    execv("/".join((dir, command)), argv)
                else:
                    wait()
    if not found:
        print "%s: Not found" % command

This works quite smoothly, it doesn’t feel laggy or anything when you’re
actually using it. I suppose this isn’t unusual because the Python code in use
probably just directly calls the underlying C system calls. It’s really pretty
neat, if not slightly silly.

I have a somewhat more complicated version working, but at the moment it’s still
looking a bit too ugly to post in a blog entry – it handles interpolation of
environment variables, multiple semi-colon delimited commands on a line and has the
beginnings of pipeline and redirection support.

I think I’ll end up writing an article that explains how Unix shells work,
using a Python shell as an example (because it’s much easier for the average
person to read than C), and put the complete shell up on my
software page.

On an unrelated note, here’s Super Mario World implemented in Javascript. Wow.

Following on from my post about the unavoidability of Flash, maybe a week or two ago I took a plunge back into the world of Linux after a goodly 3 years or so using NetBSD as my main desktop operating system. I decided to check out this Ubuntu thing that everybody seems so very worked up about these days. More specifically, I tried the Xubuntu flavour, because I find Xfce to be less repugnant than the bloated giants that are Gnome and KDE.

Here’s a list of things I’ve discovered about it so far which have bothered me.  I consider all of them to be fairly serious flaws, at least from an idealistic stand point:

• The “base” installation bares no resemblance to the conceptual ideal of a base installation – it contains many things which are in no reasonable way necessary for your system to work (including Perl and Python and Ruby!), yet misses out things that one would reasonably expect to be a part of any Unix installation and in many cases are necessary for the system to work, like NFS support (client or server).
• By default, the root account is disabled (i.e. you cannot login as root from the console or use su from a shell) and you are forced to use sudo to do anything with root powers. At no point whatsoever during the installation process are you told that this is happening, even though no other Unix-like operating system on Earth behaves this way.
• In a default installation, typing vi at a shell doesn’t start vi, it starts vim. I don’t like being lied to by my computer.
• Third party software that you install via apt-get gets put in /usr/bin/, the exact same place as stuff that was installed as part of the system. /usr/local remains empty and there is no separation of system from extras, as is seen in the BSD systems and like I’m pretty sure was the norm when I last used Linux.
• Whenever you install a piece of server software (like, for instance, the OpenSSH server, which, astonishingly, is not part of the titanic base install) Ubuntu immediately starts this server up, using the default configuration. You don’t get asked if you’d like to start it now. You don’t get a chance to change the default configuration to something that is appropriate for your environment. It just gets started. I think this one is particularly bad.

With all that said and done, having Flash working is pretty nice, and the package management has been an absolute dream so far, compared to pkgsrc. It doesn’t have some of pkgsrc’s cooler features, like license auditing or automated checking for security vulnerabilities, but the three simple facts that:

1. Everything you could ever want is in there,
2. Everything is available as a binary package,
3. Updating everything is easy,

make those little sacrifices entirely negligible. It’s awesome, just a shame that the rest of the system has such severe deficiencies. I suppose the obvious thing to do based on this experience is to check out plain old Debian, although I also have my eye on Arch Linux. Hopefully I can find something out there that doesn’t suck. I’m somewhat worried about the age old issue of Debian’s packages being prehistoric, but I suppose that if it’s so bad, I can live with Xubuntu – most of the issues above are things you only have to deal with once.

On an unrelated note, you’ve probably noticed the appearance of commenting on this blog – I think that this works, so feel free to have at it. The road to getting comments to work in pyBlosxom was winding and fraught with peril, and my next entry will probably be about that.

Oh, and the commencement of my PhD has been delayed by probably about a week due to various administrative complications. Sigh.