Evalicious

To the best of my understanding there is a little over a week before FeatureFreeze, so now would be as good a time as ever to put together an update on the status of migration-assistant. Continuing from my last post, it took a lot longer than I expected to get the Ubiquity code up and running. I had expected to have that, email account importing, and firefox bookmark importing all done by now, but I quickly ran out of time. Even now I’m tackling some minor bugs left in that portion of my code. I’m not going to under-estimate this one: it’s going to be a race to get everything I’d like to have checked in included before the cut off. I do however have a long weekend this weekend in which I intend to use for a few all-nighters and some days during the week where my course load and work schedule is light. It’ll be a race, but I intend on getting the job done and done right.

I actually have not been able to test my Ubiquity code outside of my chroot yet. VMWare seems to choke (or rather, my 512MB of RAM) on Ubiquity off the Knot-1 CD. It will just sit there for eternity after you press the next button on the welcome screen. If anyone has any suggestions beyond, “find a faster computer” which I’m going to try to locate tomorrow, I’m all ears. I’m trying to get that code working and stable as quickly as possible, because, in addition to the quickly approaching deadline, I’d really like to get back to coding import routines.

Since Summer of Code ‘06 is officially over, I can finally accept patches to the project. If you’re interested in contributing, or you’d just like to see what progress has been made thusfar, you can check out (bzr) migration-assistant and my Ubiquity branch and the following two repositories, respectively:

• http://www.evalicious.com/projects/migration-assistant
• http://www.evalicious.com/projects/ubiquity.ma

I really don’t mind constructive criticism, so if you find something in there that you think is terribly done, by all means tell me in the comments here or to evand on Freenode and I’ll do whatever I can to fix it.

As soon as I’m certain all of the components are working properly, namely Ubiquity, I’ll put an ISO image up on here for the truly lazy. Post Feature Freeze I’m going to throw together quite a few VMWare images and stress test this as much as possible with as many different configurations as possible. But again, for now anyway, my focus is on Ubiquity, Firefox, and Outlook/Evolution.

I spent the past three days preparing and then taking two credit-by-exam tests at my university.  Having returned, I can finally get back to a language free from implied subjects and edited pictures.

Work towards 0.2 is progressing.  I intend on finishing the user page for Ubiquity today.  We’ll see how that goes.

Customers will not be able to directly upgrade Windows 2000 to Windows Vista. Rather, they will have to back up everything and format. To quote BetaNews:

While Windows 2000 Professional customers may purchase a cheaper “upgrade” copy of Microsoft next-generation operating system, Windows Vista must be “clean installed,” which means users will need to back up their files and data manually and then copy everything into place. Applications will also need to be re-installed.

Another possible use case for Ubuntu and migration-assistant. To put this into perspective, according to a random, unverified website I found on Google, Windows 2000 makes up 6.74% of the operating system market.

After fixing many bugs in the code and running countless builds of the d-i image, migration-assistant 0.1 is here. I’d like to thank Colin Watson and the Ubuntu community for their help and input with this release.

What works?

Windows XP

• User accounts are created as users in Ubuntu as requested.
• The “My Documents” folder is imported to ~/Documents.
• The “My Music” folder is imported to ~/Music and the required gconf keys are set so all of your music appears in Rhythmbox from the first time you launch it.
• The “My Pictures” folder is imported to ~/Pictures.
• The desktop wallpaper is imported to ~ (to be changed), set as the default wallpaper, and added to the list of available wallpapers.
• The user picture (the picture in the top left corner of the Windows XP style start menu) is imported and is visible when you lock the screen.
• Gaim, Yahoo, and AIM Triton accounts are all imported into Gaim.

Linux

• User accounts are created as users in Ubuntu as requested.
• Gaim accounts are imported into Gaim.

Mac OS X

• Nothing yet.

What’s next?

debian-installer

• Support for backing up during the question stage.

ubiquity

• Working ubiquity (LiveCD installer) support.

Windows XP

• Internet Explorer, Opera, and Mozilla Firefox bookmarks will import into Firefox. The work for this is partially done.
• Outlook, Outlook Express, Mozilla Thunderbird, and Opera Mail email accounts will be imported into Evolution.
• The network proxy settings will be imported into GNOME and Firefox if it doesn’t support the GNOME setting.

Linux

• The default desktop background will be imported and set as the default in Ubuntu.
• The user picture (.face) will be imported.
• Mozilla Firefox and Opera bookmarks will be imported into Mozilla Firefox.
• Mozilla Thunderbird, Opera Mail, and Evolution email accounts will be imported into Evolution.
• The rest of the home directory will be copied into ~/old-home. This is still under consideration.

Mac OS X

• Everything. Though this work will almost definitely start after the Summer of Code period.

How does it work?

Right now this function is limited to the debian-installer framework, but that will change with the addition of Ubiquity support in a few days.

The individual uses the regular debian-installer CD with the added migration-assistant and libxml2 udebs. The individual starts up the installer as they normally would. After partitioning the hard drive and creating a default user for the system, migration-assistant is run, provided that the user did not remove all other operating systems from the computer during the partitioning stage. migration-assistant checks for other operating systems (with the help of os-prober) and accounts within those operating systems. The user is then asked which operating systems they’d like to import from, followed by which users from each operating system they’d like to import, and finally followed by which items for each user they’d like to import. Upon selecting some items to import from a user account on one of the other operating systems, the user is then asked to type the username of an account they’d like to import these items into. If the account is not the one the user just created as the default in the previous step in the installer, they are then prompted for the user’s full name and password.
The installer then continues to run, coming back to migration-assistant after the base system is installed. migration-assistant then does the dirty work of merging each individual item into its Ubuntu counterpart. So if, for example, the user selected both Windows XP and Fedora to import from with Yahoo, AIM, and Gaim in Windows and Gaim in Fedora, then migration-assistant will take each individual account in each program, check to see if it’s already in the Ubuntu Gaim accounts file, and if it’s not, import it.
The rest of the installer then finishes and when the user boots up into their new Ubuntu installation they can immediately see that their background is the same, their documents are all there, all of their accounts are in each respective program, etc.

How do I get it?

I hope to get everything into Edgy soon, but if you’d like to skip the wait, you can do the following to build your own debian-installer netinst ISO with migration-assistant included:

1. Open a terminal and apt-get source debian-installer
2. Download the migration-assistant and libxml2 udebs from edgy (once it hits the archive) and save them into debian-installer/build/localudebs
3. Create the file debian-installer/build/pkg-lists/local and add migration-assistant and libxml2-udeb on lines by themselves.
4. Run fakeroot make build_netboot.
5. Map debian-installer/build/dest/netboot/mini.iso to a virtual CD-ROM device in a virtual machine. DO NOT run this on your regular installation, I do not want to hear about it destroying your dissertation unless you file a bug report.
6. Upon finishing the installer, log files are saved to /var/log/installer.

Questions

Q: Why didn’t migration-assistant show up?

A: Make sure you’ve left at least one other operating system installed after partitioning your hard drive, otherwise migration-assistant will see than there’s nothing to import from and never run.

Q: Why do I have to dual boot?

A: As was pointed out to me, partition resizing is not stable enough yet to warrant the creation of the temporary partition needed to juggle the files around of a partition that’s about to be wiped out.

Q: What about the Live-CD installer?

A: Ubiquity, the Live-CD installer, support is coming within the next few days. Stay tuned.

Q: What about Kubuntu and KDE applications?

A: My first priority was to get Ubuntu and GNOME working and because of that I have not had time to work on Kubuntu. That’s not to say I hate KDE or that this program is incapable of importing to Konqueror instead of Firefox, I just haven’t written the code for it yet. Now, I didn’t make a clean split between import from and import to in the code, but it shouldn’t be hard to make the changes necessary for Kubuntu support. If anyone is interested, I simply ask that they wait until the Summer of Code period is over (August 21st) to submit patches back to me as I should not be given credit for your work.

Q: What about Mac OS X and Windows Vista support?

A: On the way. After the Windows and Linux routines stabilize I will start work on both OS X and Vista support.

I was so close to release (I thought, as always).  I had but a few bugs, one in particular that I wanted to address: the permissions on everything I created were r-x r-x r-x.  I made some changes to the code to hopefully fix this and the other issues, and did my usual three hour wait for the installer to run.  Nothing worked.  Not just the changes I had made, but everything but Gaim and Yahoo import was failing now.  Yikes.  Now, if I was smart about this, I could see what changes I made using bzr diff and I could just roll back to better times.  But alas, I’ve been trying to only commit when I’ve reached some sort of milestone and had not sent anything to bzr in quite a while.  I’m obviously rethinking that approach for the future.

I’ve promised two things that I have not made good on.

1. I said I’d post much more often, which I’ve been neglecting to do.
2. I said I’d have the first release ready Real Soon Now.

The second part has been holding the first part back as I was hoping to save the next blog post for the release notes, but alas the release is not quite ready. Don’t get me wrong, it’s mostly there. One can boot a d-i iso, select some items to import and they magically appear once the install has completed. This after many, many builds of the installer and many tiny (stupid) bug fixes. The reason there’s no release yet is the some items part. I really wanted to have the importing of My Documents, My Pictures, My Music, the wallpaper, user image, proxy settings, and so on working for the first release as I figure that most people expect the Windows items to work before regular applications do. I thought these were low hanging fruit. Some are, but some require saving gconf keys, which I’ve only recently started working on. Again, I say “Real Soon Now” for about the fifth time, but the project is progressing, regardless of offical releases, and I hope to have the Ubiquity front-end also done soon, so the first release could be a big one.

As always, if you’re anxious you can check out the code from bzr (though as always there’s code in my local branch that I haven’t checked in yet).

• Things are progressing quite nicely at the moment.
• cdecl is a wonderful little program.
• MSN is quite tricky.  I spent a good hour trying to figure out where it stores its usernames.  I cannot find my account (Passport) in the registry or the filesystem.
• I just commmited code for the switch from Mini-XML to libxml2.  While parsing html with it is no longer an issue (what was I thinking), mxml would bail out whenever it encountered XML that it didn’t like, namely the vCards Gaim stores for Jabber accounts.

Next stop, some work towards a Firefox to Firefox importer and code cleanup so I can roll a udeb for a 0.1 release.

For all the lip service Mozilla gives about being standards compliant, they sure don’t apply the same set of rules to their bookmarks format. Now, I’m sure that it dates back to the original Netscape code and they’ve found a good reason to keep it the same way all these years, but for someone who’s trying to read the poorly formed HTML4 document, who doesn’t have access to a giant html parser, it’s a real pain in the ass.

Mini-XML chokes on it. That’s a big problem. But lets say for the sake of argument that I rewrote the code to use libxml2. I’d find, as I just did, that even though it can parse it without crashing, the nodes end up in a nonsensical hierarchy that depends mostly on what order they were added in. What’s that, you want proof? See this formatted copy of a fake bookmarks.html file. I’ve added indentation to show you the hierarchy that libxml2 creates.

Even better, Firefox cannot handle libxml2 making the links in the file standards compliant. If for example, I have a URL with an ampersand in it, libxml2 will naturally convert that to & which Firefox will leave be instead of interpreting it as &. This breaks a lot of links as you can imagine. As Daniel Veillard doesn’t want to encourage creating broken code, my only solution there seems to be a whole lot of post processing.
I’m sorry if I misunderstood something or I’ve angered any of the Mozilla developers who I will wholly admit are much smarter than me, but this puts my program in a state of flux that it really cannot be in right now. Arr.

My plans of constantly updating this blog with the status of Migration Assistant have failed to date and interested parties have been left in the dark. This is an attempt at fixing that.
m-a is progressing nicely.  Colin Watson has been an immense help.  There were a few bumps in the road, specifically working with the Windows registry and XML data, but everything appears to be in place now for the 0.1 release, which I hope to push out shortly after the new d-i installer for Edgy is released.

Right now I’m mostly tying existing code together so that I can push my local copy which should successfully detect and import AIM, Yahoo, MSN, and Gaim all into Gaim. That is, the most recent copy of all of those programs. Once m-a is completely functional I’ll worry about older versions of the applications. While finding the applications and the usernames contained within them is pretty easy, finding and decoding the saved passwords is not. This is another thing I will have to go back to in the future.
There are a lot of things that I have not committed code for yet, things that I’ve found in my own research and that of the people commenting on the Wiki page (thanks!), with an example being handling fonts globally. Addressing these will happen after I have Email, IM, and Bookmarks importing.

We had to cut our vacation to Cape Cod short yesterday as Erica’s dog had a stroke, a feat I thought only common in humans.  She obviously wasn’t going to enjoy herself while her dog died, especially figuring that it could die at any moment and she wouldn’t be there, which is completely understandable.  Still, in the time we spent up there we managed to take a few walks on the beach, eat at some really nice resturants, in my case try some new and interesting food, and walk around Provincetown.  I’m not a huge fan of shopping towns.  I’ve been to New Hope more times than I can count, and each time is more boring than the last, but I still enjoy the northernmost point on the Cape, if only for the characters and their unintentional entertainment.  “Going down on Commercial Street” indeed.