Honest to a Segfault

Bryan also asked me this at NodeConf last year, where I was chatting with him about the then-in-development IonMonkey:

I remembered my talk with Bryan when I went to recruit there last year and asked the same interview question that he references — except with the pointer uninitialized so candidates would have to enumerate the possibilities — to see what evidence I could collect. My thoughts on the issue haven't really changed since that chat, so I'll just repeat them here.

(And, although I do not speak for my employer, for any programmers back in Ithaca who think systems programming and stuff like Birman's class is cool beans, my team is hiring both full time and interns in the valley, and I would be delighted if you decided to apply.)

My overarching thought: bring the passion

Many of the people I'm really proud that my teams have hired out of undergrad are just "in love" with systems programming, just as a skilled artisan "cares" about their craft. They work on personal projects and steer their trajectory towards it somewhat independent of the curriculum.

Passion seems to be pretty key, along with follow-through, and ability to work well with others, in the people I've thumbs-up'd over the years. Of course I always want people who do well in their more systems-oriented curriculum and live in a solid part the current-ability curve, but I always have an eye out for the passionately interested ones.

So, I tend to wonder: if an org has a "can systems program" distribution among the candidates, can you predict the existence of the outliers at the career fair from the position of the fat part of that curve?

Anecdotally, myself and two other systems hackers on the JavaScript engine came from the same undergrad program, modulo a few years, although we took radically different paths to get to the team. They are among the best and most passionate systems programmers I've ever known, which also pushes me to think passionate interest may be a high-order bit.

Regardless, it's obviously in systems companies' best interest to try to get the most bang per buck on recruiting trips, so you can see how Bryan's point of order is relevant.

My biased take-away from my time there

I graduated less than a decade ago, so I have my own point of reference. From my time there several years ago, I got the feeling that the mentality was:

• C/C++ are horrible teaching languages, so they shouldn't really be taught in general curricula in circumstances where they can be avoided.

• Java and applications-level programming is where most of the well-paying industry jobs are. (Not sure how true this is or was, but it seemed to be the conventional wisdom at the time.)

• It's a Windows world. And, if it's not a Windows world, you've probably got a VM under you.

This didn't come from any kind of authority, it's just putting into words the "this is how things are done around here" understanding I had at the time. All of them seemed reasonable in context, though I didn't think I wanted to head down the path alluded by those rules of thumb. Of course these were, in the end, just rules of thumb: we still had things like a Linux farm used by some courses.

I feel that the "horrible for teaching" problem extends to other important real-world systems considerations as well: I learned MIPS and Alpha [*], presumably due to their clean RISC heritage, but golly do I ever wish I was taught more about specifics of x86 systems. And POSIX systems. [†]

Of course that kind of thing — picking a "real-world" ISA or compute platform — can be a tricky play for a curriculum: what do you do about the to-be SUN folks? Perhaps you've taught them all this x86-specific nonsense when they only care about SPARC. How many of the "there-be-dragons" lessons from x86 would cross-apply?

There's a balance between trade and fundamentals, and I feel I was often reminded that I was there to cultivate excellent fundamentals which could later be applied appropriately to the trends of industry and academia.

But seriously, it's just writing C...

For my graduating class, CS undergrad didn't really require writing C. The closest you were forced to get was translating C constructs (like loops and function calls) to MIPS and filling in blanks in existing programs. You note the bijection-looking relationship between C and assembly and can pretty much move on.

I tried to steer to hit as much interesting systems-level programming as possible. To summarize a path to learning a workable amount of systems programming in my school of yore, in hopes it will translate to something helpful existing today:

• You may have read K&R, but as a newbie it makes sense to beef up on fundamentals, so CS 116: Introduction to C Programming doesn't hurt (and you meet other passionate systems programming people in the process).

• CS 415: Operating Systems Practicum made you write C. Sadly, we were given a library for context switching userspace threads on top of the Win32 API in MSVC that we didn't really have to dig into. We had to write things like concurrency primitives, a scheduler, and a rudimentary filesystem that operated in terms of a soft (i.e. fake) disk model. I think there may have been some networking in there as well. The course was being revamped at the time, so I hope it's more bare-metal now with something practical like qemu.

• ECE 476: Designing with Microcontrollers was an amazing class for integrating whatever you were most passionate about from CS and ECE curricula. Though at the time we were using 8-bit Atmels on a proprietary compiler that had no dynamic allocation support, you had to write both assembly and C code and talk to your system board via I/O ports. Plus, I got to be a little sneaky and use avr-gcc.

• ECE 473: Optimizing Compilers targeted Alpha at the time, but was a great big systems project that taught a lot about machine specifics and code generation (interfacing to syscalls, executable and linkable formats).

• ECE 575: High-Performance Microprocessor Architecture made you write real and well-performing C applications for things like cache modeling with static binary translation. This was a very formative course for me.

• I did a bunch of independent projects to mess around and better understand areas where I was lacking knowledge.

• I did work with systems researchers at the university. Some were unwilling to take any undergrads as a policy, but some groups are more amenable.

I'm not a good alum in failing to keep up with the goings-ons but, if I had a recommendation based on personal experience, it'd be to do stuff like that. Unfortunately, I've also been at companies where the most basic interview question is "how does a vtable actually work" or on nuances of C++ exceptions, so for some jobs you may want to take an advanced C++ class as well.

Understanding a NULL pointer deref isn't writing C

Eh, it kind of is. On my recruiting trip, if people didn't get my uninitialized pointer dereference question, I would ask them questions about MMUs if they had taken the computer organization class. Some knew how an MMU worked (of course, some more roughly than others), but didn't realize that OSes had a policy of keeping the null page mapping invalid.

So if you understand an MMU, why don't you know what's going to happen in the NULL pointer deref? Because you've never actually written a C program and screwed it up. Or your haven't written enough assembly with pointer manipulation. If you've actually written a Java program and screwed it up you might say NullPointerException, but then you remember there are no exceptions in C, so you have to quickly come up with an answer that fits and say zero.

I think another example might help to illustrate the disconnect: the difference between protected mode and user mode is well understood among people who complete an operating systems course, but the conventions associated with them (something like "tell me about init"), or what a "traditional" physical memory space actually looks like, seem to be out of scope without outside interest.

This kind of interview scenario is usually time to fluency sensitive — wrapping your head around modern C and sane manual memory management isn't trivial, so it does require some time and experience. Plus when you're working regularly with footguns, team members want a basic level of trust in coding capability. It's not that you think the person can't do the job, it's just not the right timing if you need to find somebody who can hit the ground running. Bryan also mentions this in his email.

Thankfully for those of us concerned with the placement of the fat part of the distribution, it sounds like Professor Sirer is saying it's been moving even more in the right direction in the time since I've departed. And, for the big reveal, I did find good systems candidates on my trip, and at the same time avoided freezing to death despite going soft in California all these years.

Brain teaser

I'll round this entry off with a little brain teaser for you systems-minded folks: I contend that the following might not segfault.

// ...

int main() {
    mysterious_function();
    A *a = NULL;
    printf("%d\n", a->integer_member);
    return EXIT_SUCCESS;
}

How many reasons can you enumerate as to why? What if we eliminate the call to the mysterious function?

Footnotes

I love making lists with no title or context and then hiding them somewhere in my room. At least, that's the inevitable conclusion of my paper-purging weekend.

I actually found one fun list (under my bed) that I've rematerialized the context for: what criterion do I use, as a candidate, to evaluate a programming position in the valley?

I came up with this short list when I was interviewing at Mozilla. It's a personal list, inextricable from my personality, experiences, and my current lot in life, but some of the thoughts may apply generally and help future candidates to brainstorm.

Purpose

If you're into the whole consequentialism thing.

What's the outcome you're achieving by doing this job? Particularist (i.e. feed your family, put your kid through college) and universalist (i.e. bring education to undeveloped nations, make navigation systems that don't fail) concerns are both important to consider. [*]

Community

Who are the people you're going to be working with? Community in my mind is roughly, "Humans you will be interacting with as a part of your job." Do you have reason to believe this will be a beneficial community for you? Maybe you work best interacting with a particular disposition of person: driven, explanatory, idealistic, encouraging, expert, iconic, pub-dwelling, sky-diving?

As my friend eloquently put it, "Fuck staying in a job where people suck."

Learning

How much do you already know about the subject of your work? How much do you want to know? They say, "God is in the details." I buy it, but if the job expects and encourages you to learn new things in topics that you're passionate about on a regular basis, it sure makes this part easier.

I've also heard, "Lots of folks just try to do the same exact thing they did at their last job over again, but slightly better." Maybe you'd like to avoid that. [†]

Artistic direction

How much control do you have over the direction of the project that you'll be working on? It's quite frustrating to pour your heart and soul into your work if all your projects get cancelled or you're being metaprogrammed.

This brings up related questions of status: How valued will your input be? Is this place a meritocracy in the ways that drive excellent products and good decision making? What capability do you have to devote resources to things that you think are important? (This could be a use-20%-time-to-prove-its-worth-pursuing kind of deal.)

Or, maybe there is no meritocracy, and there's politics. That'd be important to know.

Artistic fidelity

How much craftsmanship is expected? Mandated? Tolerated?

Posed another way: how hacky will your work need to be to meet deadlines, output requirements, expectations, and so forth? How hacky do you anticipate the work of others will be? Is there some mitigating circumstance (like an important product deadline) that makes a lower level of craftsmanship acceptable?

Another consideration: does this job expect you to show appropriate respect for the product being delivered, or is it overly focused on the purity of their development methodology?

Tedium

What percentage of your time will be spent doing mundane tasks (that you have little hope of learning from)? Do you have a good strategy for coping-with/minimizing expected sources of tedium?

Legacy

How much state-that-already-exists will you have to deal with? Sometimes this is to your advantage (i.e. already-established big/important customers) and sometimes it's to your disadvantage (i.e. original code base believed the MS Visual Basic was the language of the future and wrote their application using solely the corner cases of the language semantics).

(Compensation and career advancement weren't on the list. I'm sure that everyone has a price where they'd sacrifice any one of these things, but it's a much more meaningful exercise to evaluate them on their own.)

The categories are supposed to be largely orthogonal, but the idea is simply to provoke thought for meaningful/tough questions during the interview process — if you value your time and attention, the employer should be giving you sufficiently satisfying answers!

No place is perfect, but thinking about what you're getting yourself into and why should put you squarely ahead of the game as a job candidate.

Footnotes