Untitled

Discuss this episode in the Muse community

Follow @MuseAppHQ on Twitter

---

Show notes

00:00:00 - Speaker 1: As software developers, maybe we go towards building a GUI with specialized inputs and forms and controls too soon, because it’s so much easier to explain to the computer what the user means if they use a specialized input tool like a button check box and so on. But if that weren’t the case, if it’s easier for the computer to understand what you mean as you’re typing in your note, then suddenly text input is the primary thing.

00:00:32 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for deep work on iPad and Mac. But this podcast isn’t about Muse product, it’s about the small team, the big ideas behind it. I’m Adam Wiggins, joined today by Jeffrey Litt. Hey. And Max Schoening, great to be here. Now, the two of you are working together with some others on an ink and Switch project we’re gonna talk about today, but first, I understand that there’s some cooking adventures going on in the lit household.

00:01:02 - Speaker 3: Yeah, so I’ve been trying to make my own stock lately. I’ve been reading this incredible book that someone recommended to me on Twitter called An Everlasting Meal by Tamar Adler, and it’s all about how to use leftovers and just like random stuff in your fridge to cook both as a way of not wasting, but also just cause it feels good. So I’ve been, you know, throwing random like carrot tops and stuff into a pot, and it feels really fun. That’s my recent cooking adventure.

00:01:29 - Speaker 1: Inspired by, we’ll get into it a little bit later with the project too, but partially by Jeffrey’s cooking. I’ve also started taking cooking maybe a little bit more seriously than before. Like I think one of the things that sort of distinguishes the amateur from someone who’s more seriously involved in something is consistency and my cooking was never all that consistent cause the loop of how frequently you repeat a dish when you’re just cooking sort of for fun is very long, right? So the learning is slow, so I’ve been getting into sous vide cooking. And just eating way more steak slash anything you can sous vide that I would like, but at least the results are getting better.

00:02:11 - Speaker 3: I’m a sous vide fan as well. It’s a major cheat code, I find. Everything is perfect every time.

00:02:17 - Speaker 1: I wish that had been my experience too.

00:02:21 - Speaker 2: Yeah, consistency and repetition, yeah, short feedback loops.

I was inspired by a book, I think it’s called the Food Lab, where basically the author does some, call it like.

Amateur science in the sense of taking common cooking claims, like should you salt meat before cooking it, or is it better if you don’t flip it, or you only flip it once versus twice or something, it would essentially just cook several side by side, varying this one thing. And then do a little informal taste test with his, you know, housemates or whatever, and sort of like try to answer that question, and many times found out that, or at least had the finding, let’s call it, that things that people swore by didn’t really actually make a huge difference in the outcome, but that idea for myself, I think even our Mutual friend and colleague Peter Van Hardenberg introduced a version of that in the Hiroku offices when he would do a little coffee workshop and essentially like brew a cup of coffee with several different approaches, you know, here’s the Chemex, here’s the French press, here’s the, and then you could taste them side by side and have new appreciation for the way these different techniques change the taste of the same source bean.

00:03:34 - Speaker 3: Yeah, I love that mindset. I think what I see as the challenge at home cooking is, you know, to bring in some of that idea of getting better and being a little rigorous without making the whole thing too overcomplicated and kind of perfectionist. There’s some aspect of amateurism and just having fun with it, that’s sort of the whole point to begin with. So I think that’s a fun balance to strike.

00:03:57 - Speaker 2: So longtime listeners of Meta Muse will know that we’re shaking up the format a little bit here.

This is our first time with two guests. It’s usually me and Mark as co-hosts along with one guest, and this is partially my theory that it’s a little hard for listeners to adapt to two new voices, but in fact, you two are not new to our guests potentially.

Max, you are one of our very first guests all the way back in episode 8 when we talked about principal products.

And Jeffrey, you joined us for somewhere around episode 34, where we talked about bring your own client. So, anyways, I thought it would be fun, especially because both of you work together on this research project to get you here together. So you can go back and listen to those episodes if you want the full backstory, but maybe you could each give a 32nd summary bio of yourself before we dive into the project itself.

00:04:51 - Speaker 3: Yeah, so I’m a grad student at MIT as well as a collaborator with the In and Switch Research lab, and my research mission is to figure out how to make software more customizable so people can edit the tools that they use and make their own software, and the project we’ll be talking about today has a lot of resonance with that theme, so excited to be here again.

00:05:12 - Speaker 1: And this is my first foray into research. I’m a software designer. I’ve worked on things like Hiroku, GitHub, cloud app, way in the past, and generally I like to summarize. My efforts as I like to make things for people who make where make is the developer build tool.

00:05:33 - Speaker 2: And our topic today will be dynamic documents, which indeed is what the potluck project that you both worked on and recently published about is all about. So we’re hoping to dive into the specifics of that project as well as some of the research process behind it. Maybe we could start out with a description, kind of the, I don’t know if elevator pitch is the right way to talk about a a research project, but a short summary.

00:05:58 - Speaker 3: We’re not raising funding, but I can give a summary, sure. So, Potluck is a substrate that we’ve been developing, where the goal is to turn regular old text documents into interactive tools that help you in your life.

And so there’s this idea that you can just start by, you know, jotting a note on your phone like you might in an app like Apple Notes, and then gradually you start enriching that note with little bits of computation and interaction.

And if you keep doing that for a while, you might end up with something that looks suspiciously like An app that you might download from the app store, but it’s not like someone else made it for you, it’s sort of organically evolved out of just a note that you started writing, and, you know, some examples of the kinds of things that we’ve thought about in the substrate are You’re writing down a recipe that your mom told you for how she makes her dumplings, and then you decide, oh, I’m gonna have a party, so I want to make 5x the recipe. What’s like 730 g times 5. That’s something that a computer should be able to help you with, right? But if your data is in a text note, how do you bring in the computer to play its role and help you out a little bit? We’ve developed these primitives where you can start injecting these little bits of computation as you need them into your text note. And so, that’s kind of the overall idea of the project.

One analogy that I think is helpful to understand the general ethos of it is spreadsheets. I’m a huge, huge fan of spreadsheets. I think they’re a really empowering medium that people interact with pretty typically on a computer these days. And the cool thing about a spreadsheet, right, is that when it starts out, it’s just a bunch of numbers in a table. It’s just data sitting there, and it’s already useful in that state. And then gradually you might add a little formula, you might add a V lookup, and if you keep doing that, by the end, you might end up with this ridiculously complicated app that’s running your whole business, but it didn’t start out that way. It wasn’t planned to happen that way, it just started out as this little bit of data that you were storing, and it naturally evolved, right? So that’s kind of the general idea.

00:08:04 - Speaker 1: Yeah, there’s the well known meme of this insert startup could have been a spreadsheet, and I think in this case, you could probably make the same argument that this app could have been a note.

And in fact, I would love to be able to do this at scale, but like if you just open the average users notes apps, what kind of notes do they take and, you know, throughout the course of this research project.

That was sort of a grounding force of, oh, what kind of notes do people keep track of and so we started looking at, you know, like Jeffrey was saying recipes. At some point we did workout tracking and plant water tracking and like collecting your favorite hikes and so on, and they all have this sort of very innocuous beginning. You’re not planning to make something big, you’re just sort of planting a little seed as a note and What was frustrating for us is, at some point, if you then want a little bit more help from the computer, you usually have to move it out of the notes app, which is a little bit sad because it’s this big drop off, and so that’s kind of what we’ve been looking at, like, how do you make that go away.

00:09:14 - Speaker 2: And I love the diversity of use cases outlined in the essay.

You focus on this cooking use case as a sort of a central one, even baked into the name of the project, but indeed all of these different kind of personal tracking stuff that tends to get scribbled down in notes and and in particular notes in your phone, text notes in your phone, that they’re not very structured, you’re trying to capture them in the moment and move on.

And certainly many of those are things I have done, but also there is a whole industry is a way to put it, category of app which is trackers. So, yeah, hike trackers and run trackers and sleep trackers. And yeah, fitness trackers, step counters, weight trackers, you know, and sometimes that’s paired with, I don’t know why, you know, Fitbit has their Wi Fi connected scale, and when you step on it every morning, it automatically records the data, but then a weakness for someone who is both curious and has some light programming capabilities is actually getting that data out or doing something with it in a more flexible tool like a spreadsheet. is often pretty difficult. Actually, Fitbit, I think even famously had a little bit of pushback for the, you had to pay for the feature to kind of like download your data as a CSV and even then it feels like this very discontinuous, OK, I’m exporting now, the data, who knows what format it’s even in, and there certainly can’t be a continuous using of the app, inputting of the data, and then also I’m gonna put it through my own, call it personal analytics.

00:10:43 - Speaker 3: Yeah, and I think another weird thing about this ecosystem of trackers is that it sort of splits up your life into these very specific categories, right? So, for example, after my workout, what if I have a nutrition shake and I’m tracking my nutrition, I need to switch from my workout tracker to my nutrition tracker, and there’s this sort of world that each app considers its space that it doesn’t go outside of, you know? This also happens when, you know, We looked a lot at recipe apps because we were thinking about cooking as one of our main domains with this tool, and a lot of cooking apps start out very simple with tracking your recipes, but then there’s sort of a natural force to bloat them with extra stuff, so, You’ll add grocery list stuff, and you’ll add menu planning, and, you know, meal planning for the week, and all these.

00:11:29 - Speaker 2: All of your friends, what are they cooking?

00:11:31 - Speaker 3: Yeah, like social, you know, and I feel like we’ve all experienced this tool starts out nice and small and grows in weird ways.

And what I think is important here is, I’m fine with it growing in the ways that I want it to be useful.

What’s annoying to me is having 100 things crammed in there that I don’t need and can’t remove from the tool. And then on the flip side, the one extra thing that I do need, I can’t add myself, right? So, we sort of like, let the developer of each tracker decide what does cooking or what does workouts mean, like, what’s the scope of that activity, and it’s really hard to permeate that kind of boundary that gets set there, and so that’s one of the problems that we were thinking about in developing potluck.

00:12:12 - Speaker 2: So some of the key concepts here, dynamic documents is obviously a spreadsheet is a dynamic document, but the idea here is taking text, plain text, which is incredibly universal.

Everybody’s phone has some kind of plain text notes app just kind of built in by default, but then you can use gradual enhancement to add some computation and make it something dynamic while keeping that same basic medium of just simple text you can manipulate.

That you also talk a little bit of the essay about personal software, which I think is precisely this concept you’re just describing here, which is rather than my run tracker being an app that I download from the app store and I’m more or less just have to use it as intended by the developers, that I can use the computational medium to build a quote unquote application that just suits my needs, is truly personal.

00:13:05 - Speaker 1: Yeah, personal software is, well, first of all, I think it’s getting more mainstream in the sense that if you look at a lot of people’s notion, usage, and all the other insert, you know, personal knowledge management tool here where people are sort of aggregating all of this stuff in their life into a personal OS and I don’t know where the appetite comes from. I don’t know if it’s tied to increased computer literacy, at least some form of computer literacy, or it’s people have been burned by their favorite app changing either by adding too many features or just being deprecated, but there seems to be a lot of energy around it and so one of the things that is surprisingly Or rather, something that you wouldn’t think about right away is when you start building these apps from scratch from a note, you never really notice that you’re actually making a big complicated thing. You’re just starting out with some text and at some point you start adorning that text with some functionality and you just keep going and going and going, and at some point you wake up and you’re like, wait, this is actually quite complicated logic.

Am I a programmer? And for us, that sort of was quite important, right, like embracing this notion of personal software that is truly yours, not from some team somewhere in Silicon Valley or wherever else deciding what’s best for you.

And I think Jeffrey, you gave this analogy early on to like imagine our homes were Just furnished completely by other people, and all the objects in there just are sort of almost immutable, like we would not have that, and we do with software, and so I think nudging at that is super interesting.

00:14:53 - Speaker 3: This is one of my favorite ways to Open up my own mind to how weird software is, is to use analogies to other parts of the world, you know, I think we sort of have gotten so familiar with these metaphors of how software is organized.

Like, in some sense, in this potluck work, what we’re doing is arguing against the idea of applications, right? Which is a really weird argument to make to a typical computer user, you know, it’s fish and water, like, what do you mean? I love apps. Apps are how we do things on computers, but It doesn’t have to be that way.

I mean, Alan Kay, who’s responsible for a lot of the metaphors we use in personal computing, has, I think, said that apps were like the biggest mistake that was made in software ever, or something like that.

You know, another analogy to bring it back to the food thing, I think, is restaurant versus home cooking. And the reason I like that analogy is that I think it gets that, I’m not trying to argue that we should ban restaurants. I love going to restaurants. It’s more like, if you imagine a world where All you can eat is restaurant food every day for every meal, and you think about what kind of society that would be, it starts to feel a little weird, right? When you go to a restaurant, you are putting a lot of trust in someone else to give you a good experience. You’re accepting kind of a restriction in choice, whether that’s like a full on oakcase, you know, meal, or even, you know, picking from a menu with 10 items is very different from going to the grocery store, right? But also, you’re acknowledging maybe that chef can do things I can’t, and maybe I’m tired today and don’t want to cook, whatever the reason may be, it’s nice.

But it’s also a certain kind of limited experience, I think.

And when I look at home cooking, I see a totally different set of trade-offs and values almost, where I’m not trying to become a professional, I’m not trying to make the best thing, I’m just trying to make something nice for myself that I like, and, you know, for my family, whatever. It’s a very different scale and and feeling, and I think that’s sort of the right way to think about, you know.

There are always going to be tons of professionals making software, and I think that’s great. I love Apple products where someone in Cupertino has thought for a year about what the width of this button should be. I’m not against that, it’s just that I think there’s also this complementary role for a different way of thinking, especially in these more personal domains.

And one last thing I’ll say about the home cooking analogy that I think is interesting is that it’s a very cultural thing. If you imagine a world where everyone always eats at restaurants and you tell someone, you know, why don’t you start cooking in your house? They might be like, well, I don’t know, that seems really hard, like, all these chefs have spent like years in school or whatever, and, you know, you can see the analogy here to, like, currently software is so professional and difficult, that it just seems unthinkable that everyone would be making this stuff themselves every day, but I think we can imagine a culture where that’s a little different, and, you know, try to promote that kind of Thinking and culture more generally, and I think that would be a good thing for software.

00:17:51 - Speaker 2: Yeah, I think the restaurant versus home cooking comparison is a great one, and also just reflects the fact that in the scheme of things, computing is just so new, and we don’t necessarily know how it fits into our lives and our society and how to relate to it, and we’ve ended up in this, you can think of it as a local minima or just a particular circumstance of time, which is that software is built by these professionals who are typically far away and building for many, many users. That’s not where we started with computing, and I certainly don’t think that’s where we’ll end up, but hence the reason to invest in research to take us in this direction.

Now one thing I think your project touches on that’s an interest of mine, obviously I would lump this under end user programming, something we’re all interested in essentially bringing programmability of computers to a wider audience, not necessarily in a professional app building context, but just in the sense of embracing the dynamic medium. But I feel one of the big unsolved problems of end user programming is really just getting it into a context where people can use it. There’s many, many really amazing research projects and prototypes and etc. where if you go into there, I don’t know what, here, launch this small talk browser and once you’re within that world, everything is malleable and composable and you have total power, but it’s not connected to anything you do in your life. And one thing I like about how your team went about this project is that You’re starting from text notes, which are on your phone. Now, it’s sort of an unanswered question is how exactly this computational medium gets into the notes app or whatever, that maybe it’s not a part that’s figured out, but very hypothetically, going from, I’ve got this text file or a series of text notes, and I wanna layer this dynamic medium on top of it, feels like a lot less of a jump than many of the other kind of programming accessibility research that I’ve seen.

00:19:47 - Speaker 1: Especially cause if you actually look, for example, Apple Notes, right, it already has hints of these data detectors. If you type a phone number and I don’t know, a few other dozen types of content, it automatically finds them for you and underlines them, and then you can, you know, tap and initiate a call and potluck just takes that notion to an extreme by saying, well, first of all, I can write my own detectors cause I don’t just want to find. A phone number I might want to find the quantity of a recipe and then it also just doesn’t limit you just to the oh I can tap and initiate a preprogrammed action. I can do something else with it, a calculation, fetch some different data and so on, but it’s a very gradual enrichment of the original note and it’s also already somewhat at home on iOS. I don’t know, Jeffrey, if you want to talk a little bit about the data detectors and the origin.

00:20:47 - Speaker 3: Yeah, totally. One of the more interesting kind of related work references that we found while we were thinking about this stuff was, what I think is the original paper describing the seed that has become these, you know, phone number recognition and stuff in modern Mac OS and iOS.

But there was a research team at Apple in the late 90s, which included Bonnie Nardi, who’s sort of well known in the end user programming space. And it’s really interesting seeing the original rationale they had for how they got to this idea of data detectors.

Their starting point was thinking about, OK, how can computers help us do stuff and be intelligent helpers, and they Sort of draw this distinction between two styles of how the computer can help you.

One is the computer just does stuff for you, or like, you know, you sort of vaguely say what you want and it doesn’t, sort of more of an assistant metaphor, you can imagine, you might not even know it’s doing stuff, it’s just behind the scenes. And I think this sort of corresponds to some of the modern ways that people think about, oh yeah, AI will just do it for you type of thinking.

But they realized that actually, both at the time that was totally infeasible. Computers weren’t good enough, weren’t smart enough to actually pull that off in a satisfactory way. And they also realized maybe it’s not quite what we want, and they went down another path, which is, let’s just have the computer find stuff for us that we care about, like, dig around in all the things on my desktop and find useful information, and then let me decide what to do with it.

And You know, as Max was saying, I think their view of what you can do with the information was relatively like straightforward. You just right click on a phone number and you hit call this number, simple interactions like that. But still, this idea that the user was in control of what to do with the information. And so, I think that’s a really nice kind of design goal for these sorts of systems is carefully balancing what are the parts that we want to be automated versus where the moments that we want to be in control, you know.

00:22:49 - Speaker 2: And data detectors is the term from, I think that paper in the potluck essay, you call it extensible searches, is this a rebrand to be a little more familiar to current audiences, or do you see that as actually, it’s because it goes beyond these more automated kind of default types, like a phone number and address?

00:23:10 - Speaker 1: With a lot of the stuff, it’s very serendipitous about how it happens during a project, and we initially didn’t even start out with potluck having these continuously running searches. It was much more of a manual process.

In fact, I think to this day, if you look at the code, it’s still like cold highlighters because we started out with this notion of, well, you have a note and then the sections that you care about, you’ll just highlight and you have different colors for highlighters to start imbuing those highlights with computation.

And someone on the project at some point sort of said, oh, well, why don’t we just run a search against it? And at the time, I think we didn’t even call it search, it was just a pattern. But if you think about how mere mortals would maybe think about this as well, I have like a Google doc open. How do I target a specific word in the Google Doc? Well I hit command F and I try and find it, right? And so that’s where this notion of search comes from, which is sort of the Most maybe human way of thinking about these detectors.

00:24:11 - Speaker 3: Another small thing I’ll add is that one of the really cool parts of the original data detector’s vision that we share, but it’s kind of been lost in the modern Mac OS version, is this extensible part of extensible searches is also really important, the idea that you can define your own.

You know, in potluck, this means that you can decide that these are the types of ingredients that I want to find in my document, and nothing else. I control the dictionary of what I consider foods, or I control the list of workouts. So if I write, you know, squat, my note will recognize that that is a kind of workout that I do, but all of it’s sort of very tailored to your life.

And the original data detectors paper had this too. They had this idea of, for example, you would teach the system, here are all the names of the conference rooms in my office. So whenever I write the name of a conference room anywhere in my OS, the computer will just know that that’s what that means. And of course that doesn’t apply to every Mac OS user, it’s more of a personal data detector that’s tailored to exactly my context.

00:25:08 - Speaker 2: Maybe like adding a word to the dictionary so that it doesn’t show up as a spelling error because it’s some nickname for something in my life that wouldn’t make sense to add to a global dictionary.

00:25:20 - Speaker 3: Yeah, exactly.

00:25:22 - Speaker 1: It’s funny to think about how frequently we have data detectors in the software that we use, right? Like on GitHub, if you want to reference an issue, you do pound 247, and that’s a data detector, famously like Twitter hashtags and at mentions were all not built into the software. They were just ways in which people invented small little microsyntaxes very fluidly.

And there is no primitive in the operating system that is not super far down to actually do something fun with those, right? And so by making it super easy right in the context of the note to write a new data detector and super easy, we can get into that a little bit later, is obviously a spectrum.

Ours still involves way too much knowledge of programming to be super easy. But the power of inventing those small microsyntaxes is super addictive.

Like you just start coming up with small things that only you are familiar with, kind of like in a notebook, you would have some sort of notation, and it’s just flabbergasting to me that operating systems haven’t embraced that at a more sort of a cross app boundary level, right? Like, I think to this day there is a class in Some SDK from Apple NS data detectors where I believe developers of apps can write data detectors for you, but you as a user have no influence over them, which is fine, like I guess some developer could build an app that lets you write your own, but partially what gets left behind there is that the same data detector should run across many applications.

If I have that, you mentioned a meeting room. If I have a meeting room name, then you should highlight it in iMessage in mail app in notes and my 23 other apps that I’ve just downloaded from the App Store, and that’s sort of lacking. If you read the paper, I highly recommend it. I’m a huge fan of BonnRD. It’s very sad that we don’t have that in our computers today.

00:27:24 - Speaker 3: And I think this brings it back to what Adam was talking about earlier around integrating with the rest of your tools, right? Sometimes. I think a really important point to make about this research project that we’ve made is that currently, just to be able to move freely, we’ve built this thing as it looks like an app.

You open this thing called Potluck, it is, you know, a web app, there’s a text box and you can do all these fancy things with the text, but the final form of this that we envision being good is not a separate app, it’s deeply integrated with all your other tools.

You could imagine any app that has text, you should be able to pull this panel of searches over and just start pointing it at any text in your system.

And we actually built a little bit of that into our prototype where we do some things where we actually interopt with.txt files on your file system. So you have a little bit of being able to open these text files in the text editor of your choice, work with them, and then when you look at them back in potluck, you see the interaction appear on top, but I think that one of the interesting open questions that we haven’t quite figured out yet is, how does it really work for this to be embedded at more of an OS platform level? Like, where do these search things live? How do you share them with other people? How do you share them across apps? It’s just sort of an interesting design challenge to think about there.

00:28:43 - Speaker 2: I feel like there is a commonality across many research projects that I was involved in and I can switch in maybe in general in the research world, which is if you could do it just in an app, you would try that, but the whole thing that makes a research is really this is something that should probably be operating system level or just cuts across.

This tech stacks or the tools you use or the devices you use in a way that isn’t really well supported by the current ways that things are divvied up or the way that we kind of compartmentalize the various elements of our computers and hence the only way to try them and see if they are plausible or good ideas in. or how they feel to use is to do them in this research context where you kind of have to hand wave and say, well, imagine this was built in your text editor or cut across all your apps or you know was there in the browser or you have a good way to share these things.

Would we want that? Would we like it? Would that help us? And that doesn’t get you all the way to what it would look like in the real world, but it certainly is a fair sight further than just sketching it out on a whiteboard.

00:29:49 - Speaker 3: The first one is figuring out what we want, prototyping the experience with enough fidelity that we actually have some idea of what platform primitives we would like to have available. And then the second part, which I think is at least as hard, is how would you actually enact that kind of change in the world. If the thing you’re doing is not trying to add another app to the app store, but totally change the structure of the app store, and all the economic incentives and the technical interfaces between things, that’s a very different shape of challenge, and so, yeah, it’s a lot.

00:30:24 - Speaker 2: Now obviously I’ll link the essay in the show notes, and there’s also a live web demo that’s pretty workable, I think, or at least in my experiments with it got pretty far, which is saying a lot for a research prototype, which tend to be, you know, focusing on the learning rather than the polished product. So I’ll link both of those in the show notes and people should certainly check them out. I’d be curious to hear briefly. On the findings and what you learned from building this and trying to use it in practice. Was there anything that stands out as surprising or unexpected?

00:30:58 - Speaker 1: I think there was this distinct moment in time where the prototype was actually good enough, that inventing your own syntax for something was very trivial.

You could just say something like find every line that starts with plant emoji, and then suddenly do something with it.

And I still remember it having the feeling of why doesn’t all software work this way. And so to me it wasn’t super obvious that personal microsyntaxes should be a thing, and the idea that the same way you can scribble personalized notations into a paper notebook, that we could bring that into software.

If you make it easy, right? If you don’t make someone go into some settings screen that’s 4 pages deep to say I’m going to change how this works, right, like usual programming, but just ad hoc, you’re like, oh, I’m just gonna start these next lines with, and, you know, famously look at markdown, like, I’m gonna start the list with asterisks. Well, I can just invent that. That to me was actually a very surprising finding is the ease of creation of a syntax and then the utility.

00:32:07 - Speaker 3: I think for me, one of the surprising things was just how nice it is to work in text. This might be sort of a bit of my programmer brain, you know, speaking, but I’m not typically one of these, you know, everything I do is in plain text kind of people, but I found that We’re so used to editing text. We have really strong muscle memory around, for example, things like, I can select some text, cut it and paste it somewhere else, or I can even paste it into another app, or I can undo, and I understand how Undo is gonna work. And all these little affordances are really mature in the systems we use, and they’re mature in our heads. They’re really strong conventions, and I think One thing we found is that when you build software on top of that really solid foundation that we all have, a lot of things just sort of fall out of that.

So, for example, when all of the state of your application and all of its UI live in a text file, you can just snip parts, move them around. If you undo your app has undo for free because it’s state is stored in the text. You get all these things out of that. And I think there’s some lesson there it feels about, I guess it’s about using the same well developed tool for many different things.

I’ve used the analogy before of, it’s like a chef’s knife, where it’s like, good at all these different things and someone put a lot of effort into making it really good and versatile. It feels like there’s something similar there going on with text, and It’s not a new insight. I think there’s lots of people out there who do Emacs or there’s all kinds of to do list apps, or, you know, budgeting apps based around text files. I think that’s a thing that some people have been experimenting with for decades, but it was surprising to us just how far you can push that into so many different domains. Of course, you can’t do everything with text. There’s lots of apps that would be ridiculous to even try making them potluck, you know, YouTube is not the target, but I think that’s fine, you know, there’s some Kernel of personal use cases that fits really well with this medium, I think.

00:34:05 - Speaker 1: Yeah, plain text, or just maybe text in general, is a surprisingly good layout engine in the sense that if you want to make personal software very frequently you’re gonna have to go and come up with your own layout.

Oh, I’d rather actually have these things at the top and not at the bottom of the screen. And I think maybe because as professional sort of software developers, maybe we go towards building a GUI with specialized inputs and forms and controls and all that stuff too soon because it’s just so much easier to explain to the computer what the user means if they use a specialized input tool like a button check box and so on.

But if that weren’t the case, if it’s just easier for the computer to understand what you mean as you’re typing in your note. Then suddenly text input is the primary thing.

And if you think about what we do on computers all day, including people who are not sort of in the industry, yes, Emacs and so on, great, but you spend most of your time writing texts to people, right? So if you can’t type on your device, then you can barely use it, which means most people who use devices spend a lot of type typing.

And I think we should encourage software designers and developers to lean into text way more than we do, and like you even see that possibly in this resurgence of the command K command lines that every app now implements, right? Like command palettes, which are also just text-based entry.

And so I think potluck maybe takes this to the extreme of saying, look, just write whatever you want, and then we’ll just teach the computer with you how to interpret what you wrote, and then you can do awesome things with it, and that’s kind of exciting to me.

00:35:47 - Speaker 3: In some ways, it’s like even one more step towards messy than spreadsheets.

Someone at the lab was computing a spreadsheet, I think, to sum up how heavy things would be in like a backpack for a hike, and At some point they realized, oh, I should just do this in potluck, because even the effort to put it in a spreadsheet table was just felt like a little bit of ceremony, like, spreadsheets are sort of clunky to edit on your phone, for example, whereas text, it just kind of, it’s one dimensional, so it resizes onto your phone, you just type characters in, it’s very low ceremony, and so if you can get the interaction you want out of such a messy data substrate, in some ways I think it’s like a good go to before you start adding too much structure.

00:36:29 - Speaker 1: Yeah, there is this, I think we link to it in the essay as well, a paper, deferred formalism, which sort of encourages you to not get into structure really early on, right? and text is great, like I can just put the cursor in the middle of a line, hit return and now I have two lines.

And if you think about some of the tools that are extremely popular, notion and so on, they always make that distinction of are you inputting pros and making a list, or do you want more structure to do some computation, which is, oh great, now you have to think like a DBA and the notion of being able to move between those two modes fluently, I think is really cool.

And at the same time, if you can afford to push the formalism as far back into the process as possible, right? Like, hopefully without the app hopping, right, of like, oh, I started thinking in Muse, and suddenly I want some more structure. Therefore we have to go get out a spreadsheet and at some point you’re like 6 level deep writing a rails app with a SQL light database and you just don’t, you know, it’s it’s not the way to go.

00:37:34 - Speaker 2: And now when it comes to structure versus free form, I do think there’s a feedback loop when you talk about microformats where you kind of are inventing your own little structure as you go, just naturally, even like writing in a notebook can be something like this.

Yeah, some of the trackers you mentioned there, one use case that came to mind for me was in the early days of parenthood, we basically had a log for things like feedings and sleeping and diaper. because it’s very useful, especially with handoff between caregivers to just at a glance, be able to look at this and see when was the last time they ate, when was the last time they slept, because that tells you a lot about trying to figure out whether they’re crying right now, what need they’re expressing when they’re crying right in this moment, as well as other maybe slightly longer term analysis in terms of like, OK, are we getting enough sleep each day, for example.

But there does tend to be a feedback loop if you do add the rigor of the computer trying to parse it, even if I’ve written that search for myself, then that is going to enforce as a strong word for it, but encourage me to use a format that can be easily parsed and to be consistent with that because I make my job on the called the programmer side or the adding the dynamic aspect to it a little bit easier. Did you see something like that in your user testing?

00:38:55 - Speaker 3: Yeah, I think there’s a really interesting tension here where it’s exactly what you’re pointing out, where on the one hand, we don’t want it to feel like programming.

So in textual programming, especially for beginners, you’re typing in characters, and there’s a very, very strict set of rules defining what’s valid and invalid, right? And if it’s valid, you get nice syntax highlighting and everything, it’s great, and if you have one comma missing, everything falls apart, and so we I thought that it was really important that you don’t start feeling that way in your text notes.

It should generally have the sense that you can just type the way you normally would. But of course, on the other hand, we still need to figure out what you meant. And so you need patterns that are accommodating enough to let you write, but also rich enough to figure out what you meant and extract the meaning. I do think one thing we realized is that there’s a big difference between applying patterns later on to some text that’s already sitting there, versus having them being applied live as you type. Because in potluck, as you type, for example, if I type 5 minutes into a note, by default, potluck has this time recognizer built in, which will search for all the durations you add to a note. And when I type 5 minutes, this underline just appears and clicks into place, and I sort of get this live feedback that I’ve typed a duration that the system has understood. And we just found that that felt really good. It feels good to have the system give you that signal that it recognized what you did. And obviously, if you were expecting it to recognize something and it didn’t, then you realize that because of the lack of feedback. And so, what we found in our experience using this thing was that if you have the searches running as you’re inputting the data, it does have this natural nudging force of making you aware of the structure a little bit and maybe being a little more mindful of where you put. New lines or things like that, but again, shouldn’t be too rigid ideally.

00:40:45 - Speaker 1: Adam, you do bring up an interesting point. I think partially why notion is so popular and such a great tool is that it does invoke a little bit like this collector mindset of I’m just going to collect, you know, whatever you’re into and make a nice table so that I can actually reason about it as a collection instead of individual items, right? And like I always joke that sort of computers are really, really good at doing stupid math and for loops. And maybe one way of thinking about this is if you look at the user interface for potluck, on the left hand side you have sort of this messy, I’m just going to type stuff out, and as you write searches that match against the document, it populates a table and that table can have arbitrary metadata, right? So I can add a new column and say actually for this timer, I’m going to add a different property and The idea of having both, both this sort of reasoning about things in collections, large, you know, all ingredients or whatever, and the idea that I don’t have to do that from the beginning, or if I change my mind, it’s not such a big deal, is really appealing to me because I think we usually switch modes from reasoning about the individual thing to the collective thing and back and forth and back and forth and software today just makes you. Sort of jump through hoops if you’re switching between one or the other, and potluck tries to, as best as possible, sort of make that fluid.

00:42:12 - Speaker 2: One thing I wanted to ask about is, in the future work section, you talk about machine learning and language models, which is a pretty hot topic among certainly the tech world broadly and also in the tools for thought space. Since you are focused so much on text as well as detection, what role do you see that as having either now or in the future?

00:42:34 - Speaker 1: It’s really interesting timing because as we were writing the paper and like doing the research project, all these big language models and like stable diffusion and a bunch of other things sort of came out and became sort of accessible to like hackers, I would say. I mean they have been for a while.

But at some point we were thinking about, well, we have these searches and the way we’ve implemented searches both sort of from a time perspective and maybe a little bit of a philosophical thing that we can get into, are all, I don’t know, like rejects, we have our own pattern language and so on, and you can if you want to write rejects. It’s obviously not super approachable to mere mortals.

So how wouldn’t it be cool if I could just have a note and say, find me all things that are quantities of food. And then GPT 3 goes off and comes back and says, here are the ones that we found.

And I think it is so obvious that the data detectors will get so much better the better machine learning gets, right? And you can kind of get a glimpse of that future in the photos app on, I forget the current photos app on on iOS, where if you take a picture of a recipe index card and it says 24 g of sugar. It’ll actually, you can tap and hold and it’ll do a unit conversion for you. Now, it won’t let you do anything else because somebody decided for you what you should do with those 24 g. That’s the part where we would hope some other, like some maybe some more extensibility, but that’s just machine learning, finding the 24 g for you and you don’t have to do it, right? And so I think it seems somewhat obvious to us that all the detectors that are currently patterns will just become much more human friendly ways to describe patterns.

00:44:16 - Speaker 3: I will add one interesting tension that we were thinking about a bit. We didn’t end up implementing AI based stuff in the project since we didn’t have time to get into it, but we thought about, you know, do you want the AI to find the stuff for you, or do you want the AI to essentially write a reject for you? And those actually end up being pretty different things because predictability and speed actually end up mattering a lot.

When I’m typing, I wanna be able to learn.

You know, if I type this string, is the computer always gonna see that as a food or not? And if you have machine learning in the loop for actually doing the detection, It’s probably pretty hard to get guarantees around, oh, you know, it depends on where it is in the sentence, or how the model’s feeling that day, whereas if you have a more deterministic pattern, that gives you something that you could learn as a human, like how it work