Setting up my old Mac Mini G4 as a development machine

I've spent quite a few time last week setting up my old Mac Mini G4 — a PPC 1.2GHz with 1GB of RAM running NetBSD/macppc current — as a "workstation" for the development of Kyua and other tools for NetBSD. Yes, this machine is very slow, but that's the whole point of the exercise I'm going to narrate below.

I recently got approval from the NetBSD core team to import Kyua into the NetBSD source tree and replace ATF with it... which is great, but when thinking about it objectively, I am reluctant to unnecessarily "punish" all NetBSD users by increasing the build time of the system significantly. Don't get me wrong: Kyua itself runs efficiently on old machines, but building the code — particularly the hundreds of tests that come with it — takes just too long. This slowdown is not too noticeable on a relatively-modern machine, but it's unacceptable on not-so-old machines.

Of course, I could proceed with the import right now (with the code disabled by default) and make it leaner later, but this would cause quite a first bad impression on our users. So it's better to delay the import a little bit until, at least, I have had a chance to simplify some core parts of the code. Mind you, this simplification work is already in progress and quite advanced: it consists on modularizing (as separate processes) some critical parts of the code and writing these with a much simpler style and in plain C.

But back to the original point of my post.

The first thing to mention about this experience is that with some effort and long waits, I've got a pretty decent setup for development even on this old machine. From time to time, I miss having a real Unix desktop at hand for development (no OS X, you are not one of those). The GUI behaves relatively well with a 1920x1200 display, running Fluxbox, traditional xterms, Mutt for GMail access and a bunch of other applications.

Unfortunately, too many things feel really sluggish. A few specific examples:

• Firefox 16 is barely usable. I'm not sure there are many alternatives to decent web browsing for such an old non-Intel platform. Dillo is blazing fast and allows me to access online documentation and mailing list archives (just enough for development), but it's pretty much useless for any other purpose given the "Web 2.0".
• Any operation that involves pkgsrc takes ages. Even when building the simplest packages, one can notice the system crawl through all of the pkgsrc infrastructure. Sometimes this is the fault of a bad algorithm; other times it's just sh(1) being the wrong tool for something as complex as pkgsrc internals.
• Things like basic code editing in Emacs 24 are slow at responding to typing. Disabling font lock mode makes it feel fast again, but it's just surprising to see that even color coding is slow.

I still remember my old and trusty machine from 10 years ago (a Pentium II 233 MHz): with a similar setup, it was significantly snappier. Yes, software has evolved and these packages now have many more features... but really, does editing a text file have to be sluggish?

Leaving aside sluggishness, there is also the problem of instability. NetBSD/macppc is a tier 2 port, and things don't work as well as one would like. I personally would enjoy bringing this port to tier 1... but I currently lack the time (and basic knowledge of the architecture) to do so :-/

Anyway, the result of this exercise: the new code I'm writing to modularize Kyua builds damn fast in this machine, and keeping it this way is the whole point of having such an old machine as a build environment. So I'll try to keep using it to develop these new components of Kyua.

I don't really like C++

Somebody recently tweeted me this message:

As a strong C++ dev and googler (hopefully with some #golang exposure), what's your opinion on @rob_pike post? (goo.gl/xlMi4)

The answer deserves much more than my original reply included, so here it goes.

First of all, I found Rob's article quite interesting. Basically, the authors of Go never expected Go to be more widely adopted by Python users than C++ users. In fact, their original goal was to create a replacement for C++ as a systems programming language. The rationale for this is that C++ users like the verbosity and flexibility of the language, with all of its special cases, while Python users like simplicity and switch to Go when they look for the performance bump. This is all reasonable but there is one detail I don't associate with: Rob claims that whoever is excited by the new C++11 features will not move to Go, because liking new C++ features implies that one likes all the flexibility of C++ and will not enjoy Go's simplicity.

Before I explain why, let's look at my personal scenario: I use C++ for personal projects and Python at work. I enjoy using C++ because of its expressivity and the cool hacks that can be implemented (although that generally goes against readability), and I really like Python due to its simplicity and its extensive standard library. But I dislike C++'s corner cases, lack of real modules, verbosity, huge binaries, slow compilation times... and I dislike Python's non-static typing.

So why do I use C++?  Most of my personal projects, if not all, are in some way related to NetBSD. NetBSD has support in the base system for C, C++, POSIX shell and, recently, Lua. If I want my projects to be bundled into the NetBSD base system, I need to use one of these languages; that's clear. Of these languages, shell is not scalable for large projects and I don't know Lua enough to implement large pieces of code (but from what I have learned, Lua has its own problems). Writing C is a waste of time due to its lack of support for basic high level data structures (things like, you know, strings and dictionaries) — and these are a must for any medium-large project. And C++, despite all of the problems mentioned above, is a good compromise.

In other words: the reason I chose C++ for things like ATF and Kyua is only because it simplified my work and it was supported by the NetBSD base system. It's relatively easy to write safe code (the RAII pattern is lovely) in C++ and there is an abundant amount of rich data structures in the base library. (This doesn't mean I will stick to C++ forever in Kyua; in fact, I have some ideas in mind to reduce its usage!)

And finally, this is where I can counter Rob's statement: I am excited about the new C++11 features but not because I really like C++. I will enjoy the new features because they simplify a language that is the only real alternative I have in this context.

Confession: In fact, I don't really like C++. I wish there was a better alternative within the NetBSD base system for large projects, but there is none.

Testing NetBSD's share/mk

For a long time, a pet peeve of mine has been the lack of tests for the build infrastructure files of NetBSD: i.e. those bsd.*.mk files that live under /usr/share/mk/ and on which the whole source tree depends.

One could argue that writing tests for these files is not strictly necessary because the successful build of NetBSD is the real final test of whether the files work or not. That's partly true, but unfortunately is not the whole story:

• Other projects depend on these build files and thus rely on their behavior. This can either be by just relying on NetBSD having these files installed, or by using the separate mk-files package (available e.g. in Fedora). The functionality of the build infrastructure should not regress, or it would break these third-party projects. (Think about these files as having a public API outside of NetBSD.)
• Even if the build of NetBSD is the real test of the functionality of the build infrastructure, the build infrastructure supports dozens of options and tweaks to change its behavior. One would need to rebuild NetBSD tens, if not hundreds of times, each with a different combination of build options, to ensure the files work.
• Lastly, when a new functionality is added to the build infrastructure, it is because some component of the source tree needs such functionality. It may latter happen that this component disappears or stops needing such functionality. The functionality becomes unused by the source tree, and thus can regress unexpectedly (breaking third-party packages or introducing bugs, as mentioned earlier).

With this in mind, it's clear that we should have some tests for the share/mk files. Unfortunately, that's easier said than done: the files in share/mk are extremely complex and expose hundreds, if not thousands, of different behaviors each with its own subtleties. Adding tests for these is hard. The fact that this code that was never designed to be unit-tested doesn't help either.

Regardless, I have just submitted some "placeholder" tests to the tree. The major point of these new test programs is to lower the barrier of entry to writing tests for share/mk and, therefore, maybe get other people to write some tests. To predicate with the example, I have populated these skeleton test programs with a couple of test cases each, although as you will see they are very trivial tests.

And, to conclude, why have I done this now? Well: I'm working on integrating Kyua into the source tree and, to make this happen, I need to do a couple of changes to the build infrastructure files. The changes are tricky, so I want to write tests to have some assurance that my modifications work and that, specially, they do not regress over time. Having the placeholder test programs in place makes this much easier, and the real functionality changes easier to review.

Introducing shtk

Have you ever wanted to have a collection of ready-to-use modules for shell scripts? I have, particularly because I keep reimplementing the same functions over and over and over and over again whenever I write non-trivial shell scripts, and I'm tired of doing so.

That's why I have just abstracted all the common code in the aforementioned tools and put it into a new package called the "Shell Toolkit", or shtk for short. Yeah, this name sounds very pretentious but, really, I don't intend this to be anything big. The only thing I want to do is simplify my life when implementing shell scripts, and hope that other people might find the modules useful. So far, I have taken the generic (and common!) code from sysbuild and sysupgrade, reconciled a few tiny divergences, and moved it into this new shtk package.

In reality, writing something like shtk is sin-borderline. I really should not be using shell scripting for the kind of tools I am implementing (they deserve better data structures and better error checking than what shell provides, for example). However, shell scripting is incredible convenient to get reasonably-good implementations of such tools with minimal effort, and is the only scripting language available in NetBSD's base system. (Yes, yes, there is Lua, but my limited knowledge of Lua would not let me write these tools in any decent manner nor in any reasonable time.)

So, what's in shtk? There are a few functions to deal with command lines (error/warning reporting and such things), some trivial stuff to deal with lists, a bunch of code to interact with cvs and, what I like the most, a module to implement configuration files with some kind of key/value validation.

At the moment, shtk can only be found in pkgsrc under pkgsrc/devel/shtk and I don't currently have any plans to make it more widely available. If there are enough people interested in that with real needs, I could reconsider, but the maintenance effort would be non-trivial.

To showcase the features of shtk, I have updated the sysbuild and sysupgrade packages to depend on this new toolkit while at the same time dropping all this duplicate supporting code. It's a good thing that I wrote exhaustive tests for all possible code paths, because the migration from the built-in modules to shtk was riddled with subtleties that would have impacted end users otherwise.

Now... time to really consider taking the task of rewriting pkg_comp in a more maintainable style so that I can add the features I have wished for for many years (like OS X support).

Introducing sysupgrade for NetBSD

Over the last two weeks, you might have had fun rolling your own NetBSD binary releases with sysbuild. But what fun is that if you have no trivial way of upgrading your existing NetBSD installation to a newer version?

Upgrading NetBSD to a newer version from distribution sets generally looks like the following;

1. Fetch new distribution sets (or roll your own).
2. Upgrade the kernel.
3. Unpack the distribution sets over the root directory, without fat-fingering the command and unpacking etc.tgz along the way.
4. Use etcupdate to merge new changes to configuration files.
5. Use postinstall to validate the upgraded system.

Simple? Yes. Easy? No. The above procedure is obscure to anyone new to NetBSD. (Actually, if you tell anybody that the way to upgrade your machine is by unpacking tarballs over / will stare at you thinking you are kidding. It's 2012.) "Jokes" aside, what is worse is that the procedure is quite monotonous and, therefore, it is very easy for the administrator to make a trivial mistake along the way and screw up a running system. (Been there, done that... multiple times.)

Machines are made to automate trivial and repetitive tasks like the above, and they are actually very good at that. Over the years, I have performed NetBSD updates manually and later written crappy, unreliable scripts to do the upgrades for me. These scripts have never been reusable and they haven't dealt with error conditions gracefully. Furthermore, because these scripts live in my home directory, I have to remember to carry them around every time I set up a new NetBSD box.

It was about time I sat down and rewrote my custom scripts into something more "decent". Something with documentation, with a configuration file, and with tests.

So today, and for all the reason above, I am introducing sysupgrade.

sysupgrade is a script that automates (trivializes) the whole process of upgrading an existing NetBSD installation to a newer release, be it the currently-running system or a non-live system. sysupgrade does so by following the steps outlined above and is coordinated by a configuration file. You can find the tool in pkgsrc/sysutils/sysupgrade, next to sysbuild. The bundled sysupgrade(8) and sysupgrade.conf(5) manual pages, and the default sysupgrade.conf configuration file should get you started and hopefully answer most of your questions.

For the impatient, the following command would upgrade your machine to the specified version target:

$ sysupgrade auto \
    ftp://ftp.NetBSD.org/pub/NetBSD/NetBSD-<X.Y.Z>/$(uname -m)

At the moment, please consider sysupgrade to be experimental. It works well for me on my various machines (running both NetBSD 6.0 BETA and -current), and I have been using the upgrade procedure outlined above for years without issues. However, as with any shiny-new software, be careful. If you use NetBSD on a virtual machine, take a snapshot before running this tool; I don't think your machine is going to blow up, but better safe than sorry!

Enjoy, and feedback very welcome!

PS: There is lots of room for improvement. The TODO file in the package directory includes some of the ideas I'd like to work on later.

Introducing sysbuild for NetBSD

NetBSD's build system is close to awesome: after checking a source tree out from CVS on virtually any Unix-like operating sytem, building a full NetBSD release for any of the supported platforms is as simple as running the build.sh script with the right arguments.

There are, however, a few things that would deserve automation in this process, but that are not in build.sh's domain to solve. These are:

• Fetching and keeping the source tree up to date: interacting with CVS is still the responsibility of the user.
• Simplifying the call to build.sh: this script takes a ton of arguments, and running it by hand quickly becomes "annoying". It would be nice if all the desired values to its arguments where stored elsewhere and picked up automatically. (mk.conf may not always be the right choice.)
• Plugging into your crontab(5) for easy periodic rebuilds of NetBSD, with proper reporting of failures. Upgrading a live system from source is a major supported mechanism (particularly for NetBSD-current), so having a way to keep fresh release files around is welcome.
• Performing such daily rebuilds of NetBSD as a dedicated unprivileged user.

To anybody familiar with NetBSD, it is obvious that all the above items are "simple enough" to solve one by one; however, doing them by hand gets old very quickly — machines exist to make our life easier, don't they? I know I am not the only one that has custom local wrapper scripts around build.sh, and I also know that my scripts were ad-hoc, ugly and non-reusable. (Also, of course, anybody new to NetBSD will certainly not find any of the above trivial, but that's secondary.)

To this end, I have written a script that automates all the aforementioned steps in a single tool, driven by configuration files that specify what to do. Enter sysbuild.

My setup

These days, my main NetBSD development box is a virtual machine. For a couple of years, I have been carrying around this virtual machine across machines and slowly tuning its configuration to suit my needs. The way I work is the following:

I keep a read-write copy of the sources under my home directory, which is the one I use for my development work. This copy is accompanied by a script in my home directory that, given a machine type, updates the sources and rebuilds NetBSD for that machine type, using a known directory layout within my home directory.

I also keep a read-only copy of the sources under /usr/src, checked out from anoncvs. This copy is used by the dedicated "builder" system user to perform dedicated rebuilds of NetBSD. The "builder" user also has a custom script (but different than the other one!) that updates /usr/src, rebuilds NetBSD for the machine types I am interested in, records full logs to files and places the build results into a shared network drive. Lastly, the dedicated "builder" user has a cron job that runs the previous script overnight.

The purpose of this dual setup is to always have a fresh release build somewhere in the system, built from pristine sources, while at the same time allowing me to do my development work in a tree that I could break at any time.

Unfortunately, the size of the virtual disk of my development box has become too small for my current needs, and in order to fix this I prefer to start afresh. The thought of having to set up this whole scheme all over again, by hand, is what triggered the creation of sysbuild.

Getting started

sysbuild lives in pkgsrc under the pkgsrc/sysutils/sysbuild directory. This package provides the main script, tests, sample configuration files and the extensive manual page. Installing the script is as easy as installing any other package, and I would recommend reading its documentation now.

Once installed, sysbuild should be ready to use out of the box assuming that you have write access to /usr/src. Simply typing sysbuild build from anywhere in the system will cause the source tree to be updated and a release for your current platform to be built.

If you wish to customize the behavior of sysbuild, copy /usr/pkg/etc/sysbuild/default.conf to ~/.sysbuild/default.conf and edit the latter. The tool will pick it up on the next execution and use your new configuration settings. You can, of course, manage a variety of configurations in case you want to do different builds (say -current and 6.x).

Setting up a daily cron job

The pkgsrc/sysutils/sysbuild-user package is a convenience bundle that manages a "sysbuild" unprivileged user in the system and installs a sample crontab to perform the desired periodic rebuilds of NetBSD. This crontab uses the sysbuild4cron script, which is a very simple utility to run sysbuild, store its output in a file, and send an error report any time the execution fails.

Simply installing the package will cause the creation of the "sysbuild" user and the configuration of this sample cron job. Follow the instructions provided by the package in its MESSAGE file to tune the behavior of any of these.

Concluding trivia

Yes, these scripts are extremely tied to my particular workflow and use cases, although I don't think my needs are that special. However, I have attempted to come up with a generic-enough script and configuration file to allow the addition of new features.

You may notice that sysbuild's version is 2.0 and not 1.0. While preparing the script for addition to pkgsrc, cvs add barfed that I was re-adding previously-deleted files. Uh hum, what a surprise! Upon reading the CVS log of the package's Makefile, I found that 10 years ago I already wrote this exact same script and that two years after it got deleted because it stopped working. I had completely forgotten about this! However, seeing pkg_comp's messy code (which was written around the same time), I imagine that the previous implementation of this idea was messy; I don't dare to look at the previous code.

Enjoy!

Getting rid of @public.talk.google.com GTalk contacts

If you use a native IM client to connect to Google Talk and also have a Google+ account, you probably have noticed by now that your contacts list is polluted by tons of addresses of the form annoyinghash@public.talk.google.com. Attempting to talk to these people from the native IM client does not work (maybe it does in some specific situations? I don't know.), so these contacts only add noise and annoyance.

It turns out that these "spurious" contacts are the people that you have in your Google+ circles, and I suppose allows such people to talk to you when you are logged into Google+. Me, I don't necessarily want to allow all of these people in my chat list.

After a bit of searching around, I have found that it is possible to remove them completely from your Google Talk list. Because all the answers I could find were incomplete and/or vague replies to forum questions, I'm writing the procedure down here for posterity. Do as follows:

1. Log into your Google+ account.
2. Sign into the builtin chat client on the right-hand side panel.
3. Open the drop-down menu next to your name within the chat client and select Privacy settings.
4. In the Choose who can chat with you option, select Custom.
5. Unselect all the circle names in the checkboxes that appear.
6. Click save.
7. Witness all those useless contacts vanish from your IM clients! (Both native and web based.)