Testing NetBSD's share/mk

For a long time, a pet peeve of mine has been the lack of tests for the build infrastructure files of NetBSD: i.e. those bsd.*.mk files that live under /usr/share/mk/ and on which the whole source tree depends.

One could argue that writing tests for these files is not strictly necessary because the successful build of NetBSD is the real final test of whether the files work or not. That's partly true, but unfortunately is not the whole story:

Other projects depend on these build files and thus rely on their behavior. This can either be by just relying on NetBSD having these files installed, or by using the separate mk-files package (available e.g. in Fedora). The functionality of the build infrastructure should not regress, or it would break these third-party projects. (Think about these files as having a public API outside of NetBSD.)
Even if the build of NetBSD is the real test of the functionality of the build infrastructure, the build infrastructure supports dozens of options and tweaks to change its behavior. One would need to rebuild NetBSD tens, if not hundreds of times, each with a different combination of build options, to ensure the files work.
Lastly, when a new functionality is added to the build infrastructure, it is because some component of the source tree needs such functionality. It may latter happen that this component disappears or stops needing such functionality. The functionality becomes unused by the source tree, and thus can regress unexpectedly (breaking third-party packages or introducing bugs, as mentioned earlier).

With this in mind, it's clear that we should have some tests for the share/mk files. Unfortunately, that's easier said than done: the files in share/mk are extremely complex and expose hundreds, if not thousands, of different behaviors each with its own subtleties. Adding tests for these is hard. The fact that this code that was never designed to be unit-tested doesn't help either.

Regardless, I have just submitted some "placeholder" tests to the tree. The major point of these new test programs is to lower the barrier of entry to writing tests for share/mk and, therefore, maybe get other people to write some tests. To predicate with the example, I have populated these skeleton test programs with a couple of test cases each, although as you will see they are very trivial tests.

And, to conclude, why have I done this now? Well: I'm working on integrating Kyua into the source tree and, to make this happen, I need to do a couple of changes to the build infrastructure files. The changes are tricky, so I want to write tests to have some assurance that my modifications work and that, specially, they do not regress over time. Having the placeholder test programs in place makes this much easier, and the real functionality changes easier to review.

Featured software

Featured posts