One way to do applied research

November 7, 2024

In some of the communities I hang around in¹, there’s a fondness for the kind of industrial research that happened at places like PARC and Bell Labs. Things like Smalltalk, Mesa/Cedar, and Plan 9 not only introduced new ideas but demonstrated them in practical, working systems.

If you want to actually do research like this, how should you go about it?

There are a couple of obvious strategies. One is to look for unsolved problems — say, “addressing the scale-up problem in visual programming”. Another strategy is to solve an old problem with new techniques: “X, with CRDTs”. Both of these lead to something that is obviously novel, but it might not really be useful.

Another path

But there’s another strategy, which Alex and I took with Ohm. I don’t think it’s groundbreaking, or even original, but it’s worth talking about because (imo) it’s a less obvious way to do research.

The idea is this: aim for practical utility first, by building something that addresses an actual need that you have. You don’t need an unsolved problem, but there should be something unique about your solution. In a sufficiently complex domain, there’s a good chance that will lead to publishable research.

The core ideas behind Ohm came from Alex’s experience writing grammars with OMeta (which he also created). The abstract of our first publication gives a good overview of what makes Ohm unique:

Parser generators give programmers a convenient and declarative way to write parsers and other language-processing applications, but their mechanisms for extension and code reuse often leave something to be desired. We introduce Ohm, a parser generator in which both grammars and their interpretations can be extended in safe and modular ways. Unlike many similar tools, Ohm completely separates grammars and semantic actions, avoiding the problems that arise when these two concerns are mixed.

Ideally, whatever you build is something that helps advance your broader research agenda. When Ohm was created, I was working in Alex’s research group at CDG/HARC. A lot of our research involved new programming languages and DSLs — so Ohm was valuable to us even if it hadn’t directly led to publishable work.

What can you publish?

That first Ohm publication, Modular Semantic Actions, is an example of the most likely outcome of this sort of research: a systems paper. There are many opinions on what makes a (good) systems paper, but I particularly like this one by Kayvon Fatahalian:

Good systems papers often put forth an argument of the flavor:

“We are looking to achieve the goals A, B, C, under the constraints X, Y, Z. No system existed to do so when we set out on our efforts (otherwise we probably would have used it!), so from our experiences building systems in this area, we can distill these experiences into a specific set of system requirements. These requirements may not be obvious to readers who have not spent considerable time thinking about or building applications in this space.”

(From What Makes a (Graphics) Systems Paper Beautiful.)

But there’s another interesting thing that can happen. Once you’ve built a working system and put it to authentic use — i.e., used it to solve actual problems that you have — you’ll have a totally different perspective on the problem domain.

You’ll likely uncover other interesting research questions, things that weren’t obvious to you before. Those topics will naturally be less crowded, and you’ll have insight and expertise that other people in the field don’t have. On top of that, you now have a working system that you’re an expert in, which you can use for experimentation and validation.

This is what led to our second proper publication, Incremental Packrat Parsing. It’s unlikely we’d have written it if we hadn’t spent two years building Ohm and working on live programming environments. Alex had also been using Ohm in a class he was teaching at UCLA, and incremental parsing solved a real problem that he had with the interactive course materials.

And since we had a codebase that we were intimately familiar with, it didn’t take long to prototype our solution. As I recall, the core technical contribution in that paper was conceived and implemented on a single, two-week trip I made to LA in 2016.

Indirect impact

Building something useful is also a great way to have impact in the wider research community. Patrick Rein, from the University of Potsdam, adapted Ohm into a Smalltalk library called Ohm/S, leading to at least one publication. In Daniel Jackson’s group at MIT, a number of projects have used Ohm, including Déjà Vu and Geoffrey Litt’s Wildcard. Ohm has also played a part of several of Geoffrey’s projects at Ink & Switch, including Potluck and Embark.

Some objections

It won’t work for me! Perhaps not. I’m not saying it’s easy or that it’s the best approach to take. My only claim is that it can work.

It’s obvious! Maybe it is. It certainly used to be fairly common in traditional research environments, both in industry and academia. It doesn’t seem as common these days.

I suspect many people outside these environments don’t even consider it, because it doesn’t feel like research.

What do you know? Honestly, I’m not an expert. I’ve been following the HCI and PL literature for ~20 years, but I’ve only published a few “real” papers². I’ve had a handful of workshop papers, and have reviewed papers for CHI, Onward!, and LIVE. So, take my advice for what it’s worth.

💬 Feedback? Send me an email or respond on Bluesky, Mastodon, or Twitter.

Thanks to Alex Warth for helpful feedback on this post, and for being a fantastic collaborator and co-author.