kvm changes in Gentoo

Since I got saddled with maintaining kvm without really having the desire (I use it but not all that often) and not really having the time to maintain it, I asked for someone to take over maintainership. Unfortunately I didn’t get any replies so the package has remained fairly unmaintained and dead-ish. Well I’ve decided today to take some time and clean it up a little bit and get it up to speed a little. However, this means that there are some changes coming.

We will use qemu-kvm releases and will no longer rename everything to make it play nicer together. External kernel modules won’t be supported and you’ll be expected to build the right settings into the kernel. If you want to use the external kernel modules, that’s up to you. qemu and qemu-kvm will be blockers since they will overlay greatly.

Really the optimal thing would be for a group of people to take over kvm and qemu and maintain them together.

NVIDIA legacy drivers update

As I previously noted, the 96.x.y and 173.x.y series had not seen any updates yet for xorg-server 1.7 compatibility, however today that has changed. They both saw updates late last night from NVIDIA and now those drivers are available in the Gentoo tree so feel free to give them a try.

MythTV 0.22 is out

As many may have already known, MythTV 0.22 is officially out. Slashdot is late to the party to report the news, however Gentoo does have ~arch ebuilds available in the tree for MythTV and all officially supported plugins. In the future I may add an ebuild as well for mythstream since it is a fairly popular plugin.

Some known issues include compilation problems on x86 as a result of PIC support, the issue is being looked into actively. For this upgrade, you must manually upgrade your database however the process is pretty painless.

If you run into any other issues, please file a bug and I’ll do my best to get to it.

nvidia-drivers and xorg-server compatiblity

As always happens when we see a new xorg-server release people start to wonder why the Gentoo ebuilds for nvidia-drivers have blockers for the new release. To help clarify this I’ll post an ASCII table showing the current support.

driver series   maximum ABI
-------------   -----------
71.86.*         2.x
96.43.*         5.x
173.14.*        5.x
190.*           6.x

xorg server     video ABI
-----------     ---------
1.7.*           6.x
1.6.*           5.x
1.5.*           2.x

As you can see from the above, the 71.86.x series has gone into very limited maintenance mode. I would be surprised to see very many updates, if any at all to it in the future. However, for users of the 96.43.x and 173.14.x series, NVIDIA has confirmed to me that there will be an update to the latest xorg-server ABI change. This update though does not have any definite dates so I can’t provide any. I recommend users of that series stay with xorg-server-1.6.x releases for the time being so they can have a smooth X experience.

Improved VDPAU abstraction

Recently, Aaron Plattner from NVIDIA announce libvdpau 0.2, which a wrapper for driver specific VDPAU implementations. You may have noticed that nvidia-drivers shipped a libvdpau.so library for a while now for apps to link against while placing their actual implementation in libvdpau_nvidia.so. Newer ebuilds of nvidia-drivers will no longer install libvdpau.so, nor any of the headers and install will rely on this being installed. Other applications, like MythTV will instead depend on x11-libs/libvdpau.

In addition to this change, there’s a new package called vdpauinfo in the tree. Some people might know it from the NVNews forums as vdpinfo. However, Aaron got the author’s permission to add it to freedesktop.org and rename it to vdpauinfo to match the library.

nvidia-drivers 190.x

NVIDIA has officially blessed the 190.x driver series to replace the 185.18.x series. Release highlights include:

• xorg-server-1.7 support
• OpenGL 3.2 support
• VDPAU decoding of MPEG-4 Part 2, DivX 4, and DivX 5 depending on your GPU
• Support additional GPUs in the GeForce GT series

For more details read their release notes, here.

Gentoo has had a few different developers maintaining “unofficial” drivers in the 190.x series. However there’s a few outstanding issues with those ebuilds that still need to be resolved before they’re unmasked. I’ll hopefully push 190.42-r2 unmasked later on this evening.

Subversion Branch Office Support aka commit-redir

Right now at work we’re currently having an issue with branch offices and SVN. We’ve got a machine with some decent power with decent disk space handling our SVN repos. We’re running a 1.6 version repo that was dumped and reloaded with 1.6 so its using the newer format fully. However we still have employees at branch offices that often complain about the poor speed of SVN. Oddly the biggest complainers were Windows based devs and people using obscure (at least to me) GUI SVN tools. Currently our branch offices are all linked back to our main office over a VPN link with a dedicated 1.5mbit up and down to each remote office, with a possibility to burst up to 80% of total VPN bandwidth. A few complaints were attributed to poorly coded SVN clients (i.e. one employee’s client would perform an ’svn log’ on the top level of the repo and filter out the correct level of messages on the client side). However the complaints have remained constant. To mitigate our guy handling our infrastructure configured SVN hotcopy repos at the branch office servers that allow everyone to get their data off those SVN servers (repo UUIDs were sync’d) and then commit back to the main office.

To remedy this I’ve created a new capability in SVN trunk. I’ve called it ‘commit-redir’. Basically a branch office employee can now checkout their code from a local SVN repo and commit as they normally would to that server instead of having to go through the time consuming steps of switching back to the main office repo. Transparent to them, the client advertises the ‘commit-redir’ capability to their local server, which then sees that its a read only mirror and replies back with the correct URL they should present the commit to. The client takes this response and re-submits the commit back to the main office. Net result, employees can enjoy the full speed of having a local mirror of SVN except when committing. They don’t have to jump through loops to use that local mirror and commit back.

Now I’ve still got some bugs in the code and some touch ups to do. We’re also not actually running the code at the office yet due to the afore mentioned bugs but I’m hoping this week to submit the code upstream and to start running it at our offices.

MythTV 0.22 & the database problem

Shortly everyone will start seeing MythTV 0.22 Release Candidate packages (you won’t see Release Candidate 1 but a newer revision) appearing in the tree available to ~arch. Gentoo users need to know that the upgrade path won’t be smooth sailing. Unfortunately, the MythTV developers incorrectly use MySQL which results in data corruption which must be manually fixed. The steps to fix this are available on their website at, http://wiki.mythtv.org/wiki/Fixing_Corrupt_Database_Encoding.

Now you might say to yourself, “hey! That page blames distros specifically Gentoo for the data corruption!” I’ve discussed the issue in length with several developers in the #mythtv-dev channel and unfortunately I can’t change their tune. So all I can do is explain the issue to you, the reader, and let you be better informed.

MythTV originated as a project created by a US based developer and such it only needed to support the needs of the basic US English language set. As time wore on the project gained overseas attention and needed to support non-English character sets and data. A wise decision was made to switch to UTF-8 string handling in all of MythTV so that it would work no matter the language. However, the change was implemented poorly. The MythTV developers implemented UTF-8 conversion within MythTV while leaving their database encoding the same. This wasn’t a problem technically at first because MySQL pre-5.0 defaulted to latin1 encoding (this change was made when MySQL 3.23 was all the rage, especially for the MythTV devs who were overwhelmingly Debian based devs at the time). Fast forward a few years when MySQL released 5.0 and finally acknowledged they need to support UTF-8 out of the box. Gentoo followed upstream’s wishes and shipped their MySQL with UTF-8 as the default and worked with many package maintainers to resolve issues. Several distros (Red Hat, Fedora, Debian, Ubuntu) chose to remain at latin1 as their default so they wouldn’t have to deal with any package issues. Fast forward a few more years (hey let’s be realistic, MythTV releases take forever to come out) and MythTV finally switches to Qt4 (cause its only been almost a year and half since Qt3 was completely end of life’d) and their data conversion code needs to go through some changes. Unfortunately, this changed the on disk representation which is where we’re left now.

The real fix would have been for the developers to read the MySQL Character Set Support page and properly use the database to store the data its suppose to store instead of assuming everyone’s MySQL database would always be hard coded to latin1. And not just the database, but the server, the client libraries, the database and the connection. When they’re in fact sending UTF-8 data.

As a follow-up for those that will ask, no forcing MySQL to be recompiled with latin1 before the MythTV 0.22 upgrade will not fix the issue. In fact it will make it worse since MythTV 0.22 will error if it detects your connection character set is UTF-8 but if you recompile MySQL with latin1, then it won’t know that your database may be corrupt. I recommend following their instructions to fix the issue.

Ideas for dm-crypt support

In an attempt to improve cross-distro support, I have been considering for some time now reworking the way the current dm-crypt setup in Gentoo works and making it work more like Debian, Ubuntu, Red Hat, Fedora, and CentOS do. Effectively the syntax they use is that of fstab, however its in a new file called /etc/crypttab. This file is formated in the following style:

/dev/VolGroup00/data    crypt-data    none           luks # Password style LUKS
/dev/VolGroup00/tmp     crypt-tmp     /dev/random    tmp  # sets up /tmp for each boot having a fresh key
/dev/VolGroup00/swap    crypt-swap    /dev/random    swap # sets up swap for each boot having a fresh key
/dev/VolGroup00/other   crypt-other   /dev/sda       luks # uses /dev/sda as the key source

The partitions above would all be referenced by /dev/mapper/crypt-NAME. As you can probably guess the above configuration is not an exhaustive iteration of all the configuration possibilities. I am merely trying to investigate if this is something that people would be interested in before I code it. I’ve had this idea kicking around in my head for over 3 months now (in fact I e-mailed the other base-system maintainers about it at the start of March) but I’m terrible at keeping up with my blog. In fact, I’ve still got some parts of my original council platform from LAST year’s election still sitting there as draft posts.

But back to dm-crypt. For more information, look at the Red Hat crypttab(5) man page. While it says there are no options for LUKS, they have added automatically added checking for LUKS and using that instead of the pre-LUKS crypt bits.  That being said, the Debian based version apppears to map bettter for us since it has a lot more options, Debian based crypto crypttab(5). Let me know what you think.

GLEP 54 Mullings

So since GLEP 54 is at the fore front of everyone’s minds I figured I’d weigh in on it. The two main issues I’ve wanted to see addressed with GLEP 54 are:

• a consistent method for exposing the revision ID of the sources that was used. This would most likely have to be exposed via a hook that the underlying scm functions (eclass) would provide
• Information how this can be integrated with Portage (will Zac code this or will be something we’ll have to find someone to code?). Additionally information/API how we could expose the revision (i.e. equery list pkg)

While I know my approach is very Portage centric there are two reasons for this. Paludis already supports this stuff and Portage is the predominate package manager in use by Gentoo users today. While many would simply chalk it up to me being another anti-Paludis developer it’s far from the case. To potentially the surprise of some, I am a paludis user, however I am not a paludis user on my Gentoo commit machine. I feel that since Portage is our predominate package manager in use by users, it’s the package manager I should be using to ensure my ebuilds work. While 99% of the time a test with paludis is good enough, there are always situations where paludis and Portage have different behaviors and it needs to be caught.

While I think that everyone involved in GLEP 54 has the best interest in mind. I see a package that I have maintained for years being used as the example package in the back and forth debates on the topic and to be honest I wish it would stop. I’ve seen a lot of assumptions made about MythTV and back and forth comments about what would be the best for the package. So, I will state the needs of Gentoo’s MythTV packages here once and for all.

MythTV upstream has stated time and time again that they prefer people to use SVN pulls of the code instead of tarballs since tarballs get stale and they’re constantly committing necessary updates without rolling a new tarball. For a long while it worked to apply patches on top of the tarball but this is no longer a viable solution since there’s a lot of changes that actually happen to binary blobs which diff can’t catch. Upstream wants to be able to find out exactly what SVN revision a user is currently using and direct the user to use a different revision very easily. MythTV has a “stable” branch (which isn’t necessarily any more or less stable then their trunk branch) which most distros use but hardly any MythTV developers use and their trunk branch where all their commits and activity takes place. This is what most developers use and they’ll be quick to tell you that it’s more solid then the “stable” branch, until one day when its not and it eats all your programming and preferences and babies. So to remedy this, beandog, myself and now joining in tanderson have implemented functionality to handle the following example ebuilds:

• mythtv-0.21_p19800
• mythtv-0.22_alpha20500
• mythtv-0.22_beta21400

In the above examples, the _p is evaluted and based on that we know that we’re on a stable branch. It then grabs the version, which is 0.21 and converts it to “release-0-21-fixes”, which is the branch name, then we take 19800 as the SVN revision to checkout. For the alpha version the following happens, the _alpha is evaluted to mean “trunk” branch, then 20500 is checked out from trunk. The fictional beta version is supported for the special cases when the developers branch the trunk version for an impending release but haven’t moved it to the normal place yet. Unfortunately over the years they’ve used different places to store these impending releases each time so I just tweak the eclass everytime for this.

There’s two features which I’m hoping will be handled nicely.

• Have a nice way to fetch the source code outside of src_unpack. How I’ve handled this is managing to get upstream to enable tarballs (well they turned on zip only, not tarballs…) for specific revisions using a specific URL. This will appear in the tree shortly. The downside of this is that we’re having upstream’s server build a tarball of each revision and every revision will require over a 30mb download.
• Have an easy way to provide a “live” ebuild for trunk. The current GLEP 54 proposal handles this with -scm, however as I stated above. It doesn’t address the storage of the SVN revision which is a must.

UPDATE

I would like to point out to everyone that is sending me mailing list comments that what’s presented to the council is NOT a mailing list comment but in fact the GLEP. The council is not taking up “GLEP 54 + message ID xxxx on gentoo-yyy ML”. The council is being asked simply about GLEP 54 and I am stating what I feel are the shortcomings of the GLEP. If you have an idea that you are suggesting that would bridge the shortfall in GLEP 54, grab the source to the GLEP and add your ideas and re-submit that as the GLEP to be approved.