Optimizing disk access with flashcache and ssd

Flashcache is the easiest to setup of the different caching layers available for Linux (bcache, dm-cache) because it doesn’t require patching the kernel and can compile as a module; better it comes with dkms support and builds smoothly on Ubuntu 12.10.

I’ve replaced my Lenovo cdrom with this Ultrabay http://www.amazon.com/gp/product/B008MH9DFG/ref=oh_details_o02_s00_i00?ie=UTF8&psc=1 filled with a reasonably priced 64 GB solid-state disk.

I also setup laptop-mode-tools to be able to spin down the hard disk when on battery and it isn’t being accessed (like when all of the reads are cached). I’m happy to say that I’m getting 91% read cache hits today.

This cable http://www.amazon.com/gp/product/B005AKUS66/ref=oh_details_o00_s00_i00?ie=UTF8&psc=1 was not easy to find, but it will let me use the old cdrom externally, and with other systems.

Migrating 2xRAID->LVM+RAID via 2xRAID+LVM+RAID

I started with

/ = md0 (sda1, sda2) and

/home = md1 (sda2, sda3).

Then I broke the raid with

mdadm —manage /dev/md0 —fail /dev/sdb1

mdadm —managed /dev/md1 —fail /dev/sdb2.

I then validated that it was possible to use mdadm —zero-superblock /dev/sdb1 and mount e2fsck /dev/sdb1 as a single device directly (with new enough version of md metadata).

On to LVM…

I repartitioned /dev/sdb into one large sdb1 at the 1MB boundary.

mdadm —create /dev/md3 -l1 -n2 /dev/sdb1 mising

pvcreate /dev/md3

vgcreate raid1 /dev/md3

Now I wanted to migrate the existing md0 and md1 to lv root and lv home. Using mdadm -D I could get the KB sive of each md device and feed that into lvcreate.

lvcreate -nroot -L 10490304K raid1

lvcreate -nhome -L 1938828544K raid1

Note the message rounding to the nearest extent size (a few MB). This was able to work because I had removed /dev/sdb3 which was originally for swap space.

Here’s where the interesting part came in…

I now added the lv volumes to md0 and md1.

mdadm —manage /md0 —add /dev/raid1/root

mdadm —manage /md1 —add /dev/raid1/home

Wait for resync…

Now to remove the sda devices from md0 and md1, repartition sda like sdb, and add sda1 to md3.

mdadm —manage /dev/md0 —fail /dev/sda1

mdadm —manage /dev/md1 —fail /dev/sda2

sfdisk /dev/sdb —dump | sfdisk /dev/sda

mdadm —managed /dev/md3 —add /dev/sda1

Wait for resync…

Update mdadm.conf by removing the md0 and md1 lines and append md/3 outputted from this command

mdadm —examine —scan

also comment out DEVICE partitions line so that md0 and md1 don’t come up on this next reboot.

Edit /etc/fstab to change the / and /home mount points to point to the new lvs.

update-initramfs -u

update-grub

Reboot with fingers crossed. It should come up fine with the lvs mounted and only md/3.

Now clear the md0 and md1 superblocks

mdadm —zero-superblock /dev/raid1/root

mdadm —zero-superblock /dev/raid1/home

and put back the DEVICE partitions line in mdadm.conf.

Reboot once more.

Edit: The confusion grub had before rebooting required some repair with the live CD.

Now the two lv filesystems cat be grown to fit the new lv size (rounded up) with resize2fs.

LVM anti-pattern

I’ve just spent my first weekend with LVM and I’ve been researching different best practices. One practice I’ve seen suggested, and what originally made sense to me, is that when using LVM in some ideal cases (like non-booting secondary drives), one doesn’t need to use partitioning at all and should use the entire raw device as an LVM physical volume. But in deciding how I want to organize my volumes between various devices, and having to shuffle some things around and spend way to much time in GParted, I’ve learned that spanning an entire raw disk with no partitions or using only one partition is an LVM anti-pattern.

The initial draw for using LVM is the ability to logically organize volumes, which is very flexible because it allows online resizing. But focusing only on the logical organization of the data is short-sighted because eventually there may come a time where the physical data needs to be migrated to an external disk, or some of the unused space needs to be repurposed for a different volume group.

It is true that pvresize is available, but that just means one needs to go back down the path of using GParted again to make room for another pv, which will be very difficult to do in an online way.

On the other hand, the flexibility that seems to be missed in most of the documentation and discussions that I’ve come across is that LVM’s flexibility runs both ways. It is directly because LVM is an abstraction over the physical volumes that one should not care how many pv’s make up a volume group, but rather one should try to have as many pv’s as possible because that will provide the most future-proofing in the physical layer, which is the hardest to reconfigure after the fact.

Therefore, I recommend dividing a hard disk into at least 4 primary partitions, possibly with various sizes so there is a size that will be more suited to meeting a future need should you need to break off a piece of a volume group to repurpose it for something else. Maybe even better is to use extended partitions, or GPT if possible, to get more pv segments since today’s disk drives are so large. Having a couple of 50GB pieces that can be broken off of the vg will be very useful.

Case study:

Suppose you created a single partition for your LVM, but, without knowing better, didn’t use the right partition alignment to get the best performance out of your disk drive. The number of ways of correcting this are limited. You could try using GParted, but you’d need the bleeding edge version to get LVM support, and even then, would you want to wait to move all of the data on a 500GB+ partition to the right just to free up a couple of kilobytes at the start of the drive? Even so, you’d have to completely trust GParted to do this right and/or backup the entire drive to external storage. Alternatively, both of these are assuming you actually have the extra storage available, you could migrate off of the data to the external storage, wipe the drive, repartition, and copy all of the data back. (BTW This could be done entirely online by using pvmove.)

Instead, if you had created many smaller pv’s on the drive, it would be trivial to pvmove the data off of the first partition and then pvremove it so   it could be deleted and recreated with the right alignment. (Though this would probably need to be completed in stages for each pv, but I would trust LVM to managed this better than GParted.)

Git private branch pattern

I’ve frequently had to keep local changes that are not committed to our source control (svn actually), for bad or for worse. I’ve been using git-svn and update-index —assume-unchanged to hide these local changes from git status. One downside is that it is possible to accidentally commit those changes by explicitly listing a directory to commit. The bigger downside is that it can make switching between branches a bit painful because the assumed unchanged files need to be scrubbed before git will switch branches. Furthermore, the private changes I’ve made are frequently lost and I often have to re-edit those files after rebasing to master.

The private branch pattern I’m using now is much more convenient. I’ve started creating a ‘private’ branch off of master with all of the local changes that I want to make. Then I can automate rebasing those changes in and out of my working branch as necessary. This also helps make sure that my local changes aren’t lost, and it creates a nice way to catalog them in a changelog.

It works like this:

git checkout master

git checkout -b private

… make private changes and commit them

git checkout work

git rebase private

Now to merge changes to master first run this to remove the private changes

git checkout work

git rebase —onto master private

git checkout master

git merge work

git push origin … or git svn dcommit … etc

I recommend automating this with some aliases in ~/.gitconfig

Also, it is a good idea to make commit messages on the private branch with the prefix "private:" so when you see these commits in your work branch, you can easily identify them.

YouTube pair your laptop with your TV

In Chrome, open Developer Tools (found in the Tools menu), click the settings gear in the bottom right corner. Select the Overrides tab and change the User Agent to Android 4.0.2 Galaxy Nexus.

Back in the tab, visit http://youtube.com/pair. Follow the regular pairing process with your TV.

Edit: This post was written before Chromecast and the updates to the YouTube player.