November 15, 2007

procfs and preload

Two of the cool utilities that I've checked out lately have centered around /proc. /proc is a virtual filesystem mountpoint — the filesystem entities are generated on the fly by the kernel. The filesystem entities provide information about the kernel state and, consequently, the currently running processes. [*]

The utilities are preload and powertop. Both are written in C, though I think that either of them could be written more clearly in Python.

preload

Preload's premise is fascinating. Each shared library that a running process is using via MMIO can be queried via /proc/[pid]/maps, which contains entries of the form:

[vm_start_addr]-[vm_end_addr] [perms] [file_offset] [device_major_id]:[device_minor_id] [inode_num] [file_path]

Preload uses a Markov chain to decide which shared library pages to "pre-load" into the page cache by reading and analyzing these maps over time. Preload's primary goal was to reduce login times by pre-emptively warming up a cold page cache, which it was successful in doing. The catch is that running preload was shown to decrease performance once the cache was warmed up, indicating that it may have just gotten in the way of the native Linux page cache prefetch algorithm. [†]

There are a few other things in /proc that preload uses, like /proc/meminfo, but querying the maps is the meat and potatoes. I was thinking of porting it to Python so that I could understand the structure of the program better, but the fact that the daemon caused a performance decrease over a warm cache turned me off the idea.

References

Footnotes

[*]

A cool side note — all files in /proc have a file size of 0 except kcore and self.

[†]

The page_cache_readahead() function in the Linux kernel.

I look less leet around campus because I dual boot Vista

A few weeks back I got a brand new X60 tablet... to replace my relatively new X41 tablet. What can I say? I'm a sucker for high resolution screens... Lots of code and documentation needs to be viewed in parallel, and the X60 has a beautiful SXGA+ (1400x1050) configuration on its little 12" display (let's just say it's a good thing I'm near-sighted). [*] I'm planning on giving the X41 tablet to my sister for the start of her college career, since it's fully functional, in good condition, and sells for under $800 on ebay.

The Operating Systems Practicum that I'm taking (CS415, my CS project course) forces us to use Microsoft Visual Studio by having a huge, ugly, Windows API-based codebase. I tried to port the first project to make it POSIX compliant, but I couldn't figure out how to manipulate the stack pointer in gcc assembly — this was necessary to port their user-level threading library implementation. The whole mess was terribly x86 dependent to begin with.

As a result, I need to use Microsoft Visual Studio this semester, and so I decided that I would just squeeze the NTFS partition that Lenovo graciously provided to me, as opposed to obliterating it the first chance that I got. [†] I now dual boot, and many people believe that my leetness level has dropped considerably. To compensate, I did a::

% sudo apt-get install compizconfig-settings-manager

and turned my minimize effect to::

Burn @ 200ms @ (type=Normal|Dialog|ModalDialog|Utility|Unknown)

Which made all my windows go up in flames. Of course, this made me totally leet... for about five minutes, at which point I turned compiz off because I didn't want Xorg using up unnecessary resources. [‡]

Footnotes

[*]

My one gripe is that it shipped with a single pixel that's stuck bright green — single pixels aren't covered under Lenovo's pixel replacement policy. Argh!

[†]

Not-so-graciously charging me the Windows Tax for an OS that I can get for free through MSDNAA.

[‡]

More evidence that I'll probably never be a Mac OS user.

ImageMagick is amazing

Back in freshman year, not knowing any better, I scanned all of the handouts I received as JPEGs. My Automatic Document Feeder (ADF) got me a bit over-excited, so exploring the crustier portions of my filesystem, I find directories full of things like this:

CHEM211 - Prelim 1 - Practice Problems 01.jpg
CHEM211 - Prelim 1 - Practice Problems 02.jpg
CHEM211 - Prelim 1 - Practice Problems 03.jpg
...

Ad nauseam. Seeing these, I thought to myself: It be awesome if there were a utility that would take all these and bind them together in a PDF.

Meet ImageMagick's "convert" utility. It blows my mind.

convert 'CHEM211 - Prelim 1 - Practice Problems'*.jpg 'CHEM211 - Prelim 1 - Practice Problems.pdf'

Done. The resulting PDFs are pretty big, but convert can also manipulate images in just about every way imaginable — I'm just too lazy to figure out how to downsample right now.

I highly recommend checking it out. In Ubuntu's repositories you can find it using:

sudo apt-get install imagemagick

Unmet dependency solution in Synaptic

Most Linux users have an ugly Pavlovian response to the phrase "unmet dependencies". Because I use a distribution that coddles me to a (sometimes unhealthy) extreme, I had forgotten about the frightening package dependency jumbles that I used to get into when I used Fedora Core 3.

I reformatted my laptop's hard drive in order to switch to XFS. [*] As with all installations that I don't have time to automate, I end up making the same little tweaks I always make in slightly different ways. In tweaking the font clarity, however, I got a bit carried away, and started reading about code that (purportedly) improved font quality dramatically, but hadn't made it into the Feisty version of the Ubuntu distribution.

I (stupidly) installed a third party package instead of searching around for other options first. The result was some mild package mayhem.

Here's a short description of how to go about solving a problem similar to this one using Synaptic (the GUI manager for dpkg systems):

  1. Track the dependency chain down to the first conflicting package. You'll get messages like "This package [you're trying to install] depends on X, but X isn't going to be installed." X is your new target, for each X, iteratively.

  2. Try to force the package to the official version (Ctrl+E). You'll likely get more messages like in 1. If you do, GOTO 1. Otherwise, move on to 3.

  1. End up at a (most reduced) conflicting package. You'll get a message that says, "This whole mess of applications will have to be removed." If you get this, you've found one of the most reduced conflicting packages. In this case I found it to be libfreetype6. I couldn't force libfreetype6, because there were very important packages (e.g. ubuntu-desktop) that depended on upgraded packages that depended on this new version of libfreetype6. You can view dependent packages via the package properties that Synaptic displays; however, you don't see what version they are dependent on (I'll submit a feature request in a few days). As a result, your best bet is to search for dependent packages via Synaptic's search feature.

  1. Knock off 3rd party dependent packages by forcing them to the official version. Be wary of removing a lot of important things by accident in the process! If it has a slew of applications "to be removed", try forcing other dependent packages to the official version first.

  2. Work your way back down to the more shared packages by forcing versions up the dependency tree. Once you force less-shared packages to official versions or prune off unnecessary packages, you should be able to start forcing more-shared packages to official versions. This is a natural result of having less unofficial stuff to support dependencies for!

NOTE: This entry is sloppily written in a stream-of-conscious style, but I don't really have much time to edit it. Sorry!

Footnotes

[*]

I thought that the potential performance benefits of a metadata-only journaling filesystem were intriguing, and I hoped that it would alleviate some of the degradation that I encounter running a 5400rpm drive.