Over the summer, I started a group at my local Google office to read through the papers in Seminal Papers in Software Engineering: The Carnegie Mellon Canonical Collection. This is a collection of about 25 of the most influential papers (and other writings) in software engineering research, curated by the Carnegie Mellon software engineering faculty. The goal of the reading group is to get a sense for what the academic side of software engineering can teach us practitioners and what universal truths we can extract and use in our work. (As a side benefit, it’s also been a great forcing function for me to read more papers in my post-academic life.)
Having run the group for half a year now, this seemed like a good time to stop and reflect on how it has gone so far. (Disclaimer: what follows are my opinions alone; not necessarily the opinions of the group.)
How It Works
Every month we read the next paper in the list ahead of time, then we meet to discuss it. The reading order is based on this Twitter conversation with Jonathan Aldrich, which is roughly the order he and others use when they teach a class based on this list. To add some variety, every 6 months the group picks a paper not in the CMU collection (preferably one with an author from an under-represented group) and discusses that instead.
To help jump-start the discussion, we started posting our questions about the papers ahead of time on an internal tool called Dory. Dory lets everyone up-vote or down-vote each question, so it provides an aggregate signal for what the group would like to discuss. If the conversation during the meeting comes to a lull, we can use some of the top questions from Dory to keep things going.
What Has Gone Well
- Posting questions on Dory ahead of time has helped give a collective sense ahead of time of our general impression of the paper and where we got confused. It has also helped structure and guide the conversation overall.
- Reading these papers has gotten me thinking more about some fundamental questions in software development, like how to better divide programs into chunks and create clean abstraction boundaries between them. This probably affects my day-to-day work more than I realize, at least at a subconscious level.
- I’ve been exposed to a new paper (Dilemmas in a general theory of planning) that I really like. It introduces the notion of a “wicked problem”: basically a social problem that can’t easily be solved by simple math and engineering optimizations. It touches on a lot of ideas, such as how do we even define success for such an issue, how do we get feedback while working on an ambiguous problem, and how do we make plans in these situations. I’d like to read it at least one more time to really wrap my head around it.
Challenges So Far
- We usually don’t know much historical context around each paper, and without that it’s sometimes difficult to understand what makes that paper significant. For example, in the Levels and Abstraction chapter from Siewiorek et al.’s Computer Structures, we struggled to extract a more meaningful message than “organizing your system into a hierarchy helps with abstraction”. Was that less obvious to a developer back then than it is today? Or did we perhaps miss some more subtle and insightful message in that reading? The editors’ notes in the CMU tech report help with this a little, but without someone more well-versed in the area guiding us through, it’s hard to get more than a shallow reading of some papers.
- Similarly, it’s sometimes hard to see why a paper might be important to us as working software engineers, rather than to researchers in software engineering. And maybe sometimes it’s not relevant at all: after all, this list was originally intended for introducing new researchers to the field, so maybe sometimes it’s not! Tony Hoare’s Proofs of correctness of data representations was one paper where we had this difficulty. While you can generalize some of the principles there to reasoning about correctness in general, none of us in the group use formal methods day-to-day (aside from automated tools that hide the details), and we never have to do something this rigorous. Without a clear value proposition to working developers, it was hard to muster the motivation to wade through all of the formalisms in that paper
- Our group-chosen paper, Margaret Hamilton’s Universal Systems Language: Lessons Learned From Apollo turned out to be something of a dud for us. Partially this was because of the two previous points, but it also just seemed like a poorly written paper that couldn’t even explain what kind of thing the Universal Systems Language was (a programming language? A specification language? Something else?). While we want to add some variety outside of the CMU-curated list of papers, it’s also tricky for us to select what might be a good paper ahead of time without knowing the space better.
- Less significantly, some of the readings are hard to find from a legitimate source. For “Proofs of correctness of data representations” we eventually found it in a compilation of Hoare’s papers, and we ended up skipping a couple of book chapters since it didn’t make sense to have us each buy multiple textbooks just to read a chapter or two from each. I don’t think this will have a very negative effect in the end, since we can still find most of the readings, but it’s unfortunate that we might miss out on a few of them.
Conclusion
All in all, I’m happy that running this group is forcing me to read more research papers, and I’m glad that it’s exposing us to some new ideas. But at the same time, I’d like to figure out a way to extract more value and find deeper insights from these papers than we are right now.
I wonder whether tips for running Great Books programs could help here. I’ve never been through a Great Books program myself, but this feels similar in that we’re studying the original sources for big ideas and trying to wrap our heads around them. Advice for people who go through a Great Books program without a more experienced guide might be relevant.
If you have suggestions, I’m happy to hear them!
P.S. to Googlers: I’m planning to open this group up to people outside the US-CAM office. If you’re interested, check out go/cam-se-research-reading-group.