Wednesday, March 12, 2014

GSoC 2014 idea: Port FreeBSD's old-style tests to ATF

Are you a student interested in contributing to a production-quality operating system by increasing its overall quality? If so, you have come to the right place!

As you may already know, the Google Summer of Code 2014 program is on and FreeBSD has been accepted as a mentoring organization. As it so happens, I have a project idea that may sound interesting to you.

During the last few months, we have been hard at work adding a standardized test suite to the FreeBSD upstream source tree as described in the TestSuite project page. However, a test suite is of no use if it lacks a comprehensive collection of tests!

Fortunately, the FreeBSD source tree already has a reasonable collection of test programs that could be put to use... but unfortunately, these are all not part of the test suite and are broken in various ways because they have not be run for years. Here is where you come into play.

My project idea

I would like you to spend the summer working with us in converting all existing legacy tests into modern-style tests, and hooking those into the FreeBSD test suite. The obsolete tests currently live in src/tools/regression/ and src/tools/test/.

Sounds boring? Here is what you will achieve:

  • Contribute to a production-quality operating system from which you will be able to reference your work at any point in the future. If you are into systems at all, this is an invaluable experience for yourself and your résumé.
  • Gain a better understanding of Unix and FreeBSD by debugging failing tests. Trust me, debugging problems on a real-world operating system is an excellent way to gain working knowledge of systems and troubleshooting, both of which are valuable skill-sets in the industry.
  • Understand how a test suite is organized and how individual tests become useful in day-to-day operations. Not all tests are created equal and many are not as nice as they should be, so you will be using your coding skills to improve their quality.
  • Get to know ATF, Kyua and other testing technologies.

And if that's not all: the results of your project will be immediately visible to the whole world at the official continuous testing machines and your fellow developers will be infinitely grateful for having one more tool to ensure their changes don't break the system!

So, if you are interested, please navigate to the project proposal page, read the nitty-gritty details and contact us via the freebsd-testing mailing list. And, if you happen to be in Tokyo attending AsiaBSDCon 2014, stop by and talk to me!

Saturday, February 15, 2014

How to merge multiple Git repositories into one

Are you looking for a method to merge multiple Git repositories into a single one? If so, you have reached the right tutorial!

Please bear with me for a second while I provide you with background information and introduce the subject of our experiments. We'll get to the actual procedure soon and you will be able to apply it to any repository of your choice.


In the Kyua project, and with the introduction of the kyua-atf-compat component in the Summer of 2012, I decided to create independent Git repositories for each component. The rationale was that, because each component would be shipped as a standalone distfile, they ought to live in their own repositories.

Unfortunately, this approach is turning out to be a bit of an inconvenience: it is annoying to manage various repositories when the code of them all is supposed to be used in unison; it is hard to apply changes that cross component boundaries; and it is "impossible" to reuse code among the various components (e.g. share autoconf macros) in a clean manner — much less attempt to share the version number between them all.

So what if all components lived in the same repository a la BSD but were still shipped as individual, fine-grained tarballs for packaging's sake? Let's investigate.

The goal

Obviously, the goal is to get two or more Git repositories and merge them together. It's particularly important to not mangle any existing commit IDs nor tags so that history is preserved intact.

For the specifics of our example, Kyua has three repositories: one for kyua-cli (which is the default, unqualified repository in Google Code), one for kyua-atf-compat and one for kyua-testers. The idea is to end up with a single repository that contains three top-level directories, one for each component, and all independent of each other (at least initially).

Process outline

The key idea to merge Git repositories is the following:

  1. Select a repository to act as pivot. This is the one into which you will merge all others.
  2. Move the contents of the pivot repository into a single top-level directory.
  3. Set up a new remote for the secondary repository to be merged.
  4. Fetch the new remote and check it out into a new local branch.
  5. Move the contents of the secondary repository into a single top-level directory.
  6. Check out the master branch.
  7. Merge the branch for the secondary repository.
  8. Repeat from 3 for any additional repository to be merged.

Sounds good? Let's get down to the surgery!

We need to select a pivot. For Kyua, this will be the default Google Code repository in https://code.google.com/p/kyua. Let's start by checking it out and moving all of its contents into a subdirectory:

$ git clone https://code.google.com/p/kyua
$ cd kyua
$ mkdir kyua-cli
$ git mv * kyua-cli
$ git commit -a -m "Move."

We are ready to start tackling the merge of a secondary repository. I will use https://code.google.com/p/kyua.testers in this example.

The first step is to pull in that secondary repository into our pivot:

$ git remote add origin-testers https://code.google.com/p/kyua.testers
$ git fetch origin-testers

And now, check it out into a temporary branch and move all of its contents into a subdirectory:

$ git branch merge-testers origin-testers/master
$ mkdir kyua-testers
$ git mv * kyua-testers
$ git commit -a -m "Move."

Done? It's the time to merge the two repositories into one!

$ git checkout master
$ git merge merge-testers

And clean some stuff up.

$ git branch -d merge-testers
$ git remote remove origin-testers

Voilà. It wasn't that hard, was it? Just repeat the steps above for any other secondary repository you would like to merge.

Parting words

Note that this procedure achieves the goal of preserving the history of all individual repositories, the revision numbers and the tags. In other words: all previous history is left intact and all commit logs remain valid after the merge.

Do you know if there is any easier way of doing this? Would it have any differences in the actual results?

What do you think about doing the merge for Kyua? I see this as a prerequisite for the migration to GitHub.

Thursday, February 06, 2014

Moving projects to GitHub

For a couple of years or so, I have been hosting my open source projects in Google Code. The time to jump ship has come.

The major reason for this move is that Google Code stopped supporting file downloads three weeks ago. This is unfortunate given that "binary" releases are a must for proper software distribution. Sure, I could use a third-party service like Bintray to offer the downloads, but I'd rather consolidate all project data in a single location.

With this, I am moving to GitHub. "Why not <insert-your-favorite-hosting> though?" Because Google prefers us to put Google-sponsored open source projects in GitHub, and all the work I do now is sponsored by my employer as part of my 20% time. (Yes, 20% time still exists even if people out there may try to convince you otherwise.)

GitHub was already hosting some projects of my own (including sysbuild and sysupgrade), so the remaining (Lutok, ATF and Kyua) will move as well. In fact, Lutok was already moved a few days ago as an experiment and I am working on ATF as we speak.

Moving Kyua will be a little bit trickier given that Kyua is composed of more than one Git repository. I still have to decide which of the two options I have in mind is better, so stay tuned for a follow-up post detailing my thoughts.

Fortunately, because there is no VCS change with this move, the move should be pretty much transparent (unlike the move years ago out of Monotone). The biggest downside may be moving the issue tracker, and that's something I'm just starting to think about.

Stay tuned and consider the current GitHub repositories an experiment.

Wednesday, February 05, 2014

Killing the ATF deprecated tools code

The time to kill the deprecated tools —atf-report and atf-run principally— from the upstream ATF distribution file has come. Unfortunately, this is not the trivial task that it may seem.

But wait, "Why?" and "Why now?"

Because NetBSD still relies on the deprecated tools to run its test suite, they cannot just be killed. Removing them from the upstream distribution, however, is actually a good change for both ATF and NetBSD.

The main reason for removing the tools from the upstream ATF package is simplicity: with this change, ATF becomes a pure distribution of libraries to implement ATF-compliant test programs. At the moment, building upon ATF is difficult because the code of the libraries is overly complicated: the libraries include a lot of internal modules only used by the tools and this code cannot be lightened unless the tools are gone. Not putting this tools-only utility code outside of the base libraries was a bad design decision when the package was first created in 2007 and is a huge mess to untangle.

However, even if the tools are removed from the upstream ATF package, we still need to accommodate for the fact that NetBSD still requires the tools. The easiest way to do this is to just "hand the keys over" to NetBSD and transfer ownership of the tools code to them (which is a weird way to put it given that "them" includes me as well!). The actual effect of this "removal" is that the master copy of the tools code will live in the NetBSD source tree.

So, how is this going to happen?

A naive course of action would be to just remove the old code from the upstream distribution file in the next release. With this done, NetBSD would import the new release and, at the same time, keep the files for the old code without deleting them.

Unfortunately, this approach wouldn't work very well. The problem is, as mentioned above, that the tools have deep tentacles into the atf-c++ code and atf-c++, in turn, is mostly a wrapper over atf-c. If we did as mentioned above, changing the upstream code for atf-c++ and/or atf-c at a future date would be incredibly difficult: any minor change to their internals would result in the NetBSD copy of the tools not building any more on the next import, and that would be a huge maintenance nightmare (for me actually, because I'll continue to be the owner of the code regardless of where it lives).

Therefore, the process to perform the removal needs to account for this original deficiency and has to mitigate its effects. The plan is as follows:

  1. Detach the ATF tools from the libraries, which is tricky but necessary. Future changes to the internal structure of atf-c++ must not affect the code of the tools as should have been the case from day one. The way this is going to happen is by duplicating all code in atf-c++ that is required by the tools into the tools themselves and then killing the dependency of the tools on the libraries.
  2. Publish 0.19 as the final release with the deprecated tools: this is the release that will contain the standalone version of the code for use by anyone that may need it.
  3. Remove all ATF tools related code from the upstream tree.
  4. Publish 0.20 immediately after 0.19 as the first release without the deprecated tools. 0.19 should never hit any packaging system and will exist solely as a transition point for NetBSD (or others that may need it).
  5. Import 0.19 into the NetBSD source tree. This will rearrange the code currently existing in src/external/bsd/atf/ to isolate the old tools (with all of their supporting code) in a single and standalone subdirectory.
  6. Import 0.20 into the NetBSD source tree as well, right after 0.19. Because NetBSD does not import any of the autotools generated code, this double-import will result in minimal changes to the tree... yet it will be declaring the tools as owned by NetBSD.
  7. Upgrade existing ATF packages to 0.20, including those in pkgsrc, FreeBSD ports and Fedora, and simplify them significantly to remove any tools code. This may actually involve killing atf-libs and similar packages that were added in the past to cope with the duality of the contents of the atf package.
  8. Simplify the tools code imported into NetBSD to drop most of, if not all, the portability-related code. This implies removing anything that depends on the results of configure, for example, given that the indirection won't be needed any more.

OK, when?

Really soon. I have all the changes ready for 0.19 and 0.20 in my local git tree (which took two long-haul flights within the U.S. last week to finish!) and I'm now working on the integration into NetBSD to ensure all works. When this is ready, expect the new releases out and updated in your favorite packaging systems!

Anything else?

Doing this cleanup will free ATF to evolve more easily regrading new features and much needed interface cleanups. Of course, updating the deprecated tools in NetBSD will still have to happen whenever/if there are any incompatible changes, but doing so in a self-contained, OS-specific tree will be much simpler and palatable.

The ultimate goal, really, is to make the upstream atf package as lightweight and simple as possible.

What are your thoughts on this?

Wednesday, January 08, 2014

FreeBSD Test Suite tutorial at AsiaBSDCon 2014

I am pleased to announce that the tutorial on the FreeBSD Test Suite that I proposed for AsiaBSDCon 2014 has been accepted!

The conference website will soon include more details, but allow me to spoil your wait:

  • Goals: Learn how to use the test suite, how it is internally organized and how new tests can be written.
  • Stretch goal: Get attendees to contribute one or more tests to the project.
  • Audience: Mostly developers of FreeBSD that want to learn how the new test suite plugs into the system. That said, and because a major part of the tutorial will revolve around using the test suite for one's own benefit, everyone is welcome really. In particular, system administrators may get a useful tool out of this.

The main surprise in the acceptance confirmation email is that materials are due by January 20th... which is around the corner! Time to rush in getting things ready. In the meantime, you can find more details on the tutorial by reading the proposal itself.

And don't forget to arrange your trip as soon as the conference schedule is published. (Or do it right now if you know you are attending anyway.)

Sunday, January 05, 2014

FreeBSD Test Suite goals and planning

I was really impressed and delighted to see how popular my previous (and first) post on FreeBSD testing, titled Introducing the FreeBSD Test Suite, was. Looks like this project may be of great interest to developers and users out there (not unsurprisingly) so I'll try to keep you all up-to-date with any key developments.

A first question that arises from the announcement is: where are the test suite and infrastructure headed? After all, the continuous testing machines for amd64 are already up and running, so what else is there to do?

Well, my friend... a gazillion things really! I've spent some time refactoring the TestSuite project page to give you a better idea of what my vision is both for FreeBSD and Kyua. You can find there a list of verbose goals as well as a link to a more detailed project planning spreadsheet that outlines the roadmap for the project on all the different covered fronts. There are still many unknowns and it is good to explicitly surface them early.

Knowing the current state of Kyua (and ATF), I believe the set goals are pretty ambitious – and that just makes them more exciting. (But... if you pay close attention to the spreadsheet, you may miss time estimates. Sorry but if there were any, they'd be blatant lies except for the most trivial tasks. The amount of time I can spend on open source varies greatly and resolving that by improving my time-management skills sounds like a good new year's resolution.)

Before concluding, let me also answer something you may be wondering: what about NetBSD? Well, I'll save the answer to that for an upcoming post. But the short answer is that both should eventually converge into the same setup once the work in FreeBSD proves itself better.

As usual, feedback is very welcome!

Wednesday, January 01, 2014

Header files: Series wrap-up

Happy New Year!

Let me welcome 2014 with a very simple post to wrap up the header files series as is customary so that you have all relevant links in a single place for future reference:

As I mentioned a couple of posts ago, I am quite busy these days spending my limited free time on Kyua-related hacking, which means you should expect a less constant blogging pace. In particular, I do not have any more "post series" in the queue so upcoming posts will be more varied in content! Hope you will enjoy them anyway.