Justin: It was just this perfect balance of like, okay, you know, manageable enough to like write tooling for it, but realistic enough that you could actually build stuff with it. Projects that I've worked on, you know, even now, like multiple years later, I've still, targeted Elm so yeah, from a researcher's perspective, it's like lovely from that regard.

Jared: Hey folks, welcome back to Elm Town. I'm your host, Jared M. Smith. We'll be visiting with Justin Lubin today. But first, let's talk about our sponsor, Logistically. At Logistically, we make intuitive software to help logistics teams make better decisions and improve efficiency. We build the front end for all new features in Elm.

If you're interested in our mission and enjoy writing Elm, please drop us a line. ElmTown at LogisticallyInc. com.

[00:00:44] Introducing Justin

Jared: I'm going to introduce Justin Lubin. Justin is a PhD student researcher at UC Berkeley working with Sarah Chasins. He has worked with Ravi Chugh at University of Chicago on Sketch-n-Sketch, a direct manipulation programming system for creating HTML and SVG documents. He has done research on how statically typed functional programmers write code, including Elm users. He loves quote, programming languages, human computer interaction, biology, social justice, music, art, and math, and is happy to grow this list.

End quote. Justin, welcome to Elm Town.

Justin: Thanks so much! It's really an honor to be here. I appreciate it.

Jared: Just to set the scene a little bit here, I'm starting this series of episodes about the process of learning Elm and teaching Elm, and so I think this will be a great place here where we can pick Justin's brain and learn a little bit about the research that he's done in these areas.

So, uh, to get started with that, I wanted to ask you, why did you decide to become an academic researcher on programming languages and human computer interaction?

Justin: Yeah, um, Origin Story. Awesome. So, I guess it started, um, back when I was first learning about, like, um, statically type functional programming. I saw this language that was being referred to online a lot called Haskell, and I was like, ooh, like, what's that? Like, I want to see what that is. And I like looked at the website and I was like, I'm not sure I like this.

It looks so weird. Like, what do you mean? There's like no variables or like no mutation or like anything like that. I'm like, no, no, no, this is not for me. Um, but like people kept talking about it, like, so like, you know, in such high regard. Uh, so, you know, I, I looked at it again and I was like, okay, maybe there's something here.

And like, eventually I started to like really get into it and I was like, Oh, actually this is really cool. Um, so I was really excited by like programming language and programming languages and like the theory behind it. And like, you know, starting to like dabble in this sort of like category theory stuff.

And I was thinking, like, gosh, you know, um, this was maybe, like, towards the end of high school, and then when I wanted to go to college, uh, I was like, you know, maybe I want to start, like, talking to a professor who, like, does this kind of thing. so I reached out to Ravi Chugh, at University of Chicago, and, cause he was doing the Sketch n' Sketch project, and, uh, initially I was, like, coming at it from this sort of, like, you know, really keen interest in, like, you know, maybe behind the scenes a little bit of, like, how the programming languages work. I also really like math at this time, uh, still like math, um And, uh, yeah, I was just sort of like really excited about that.

Uh, but then, like, over time, like, as I was working on Sketch-n-Sketch, which, as you mentioned before, is this, uh, programming tool where you can essentially do sort of Illustrator or Photoshop like operations on, like, the output of, like, um, a program. So, you know, maybe drawing, like, a rectangle or, like, a circle or the lining things.

Um, and as you're working on that, it will actually, uh, construct a program for you, uh, in a language actually very similar to Elm. So you can do things like, you know, maybe make a ferris wheel and it has like five spokes in the ferris wheel, but then you realize actually you wanted a sixth one, and so like in Illustrator, you know, you'd have to like redraw the whole thing, compute the angles, like 360 degrees divided by six, something like that.

But in this case, you know, because you have a program, you can actually just change the five to a six. Um, so, And so this really got me thinking about like, gosh, there's this whole like world of, you know, really interesting programming language theory, um, you know, how programming is working behind the scenes, like the idea of like program synthesis, which is like generating programs, um, sort of like in this situation where we're like, you know, building a program based off of the users, um, like drawing rectangles and whatnot.

I mean, you can take all of that really awesome, interesting mathematical stuff, And actually, you know, work with people, you know, build tools that are going to be like, really interesting and useful for them. Um, that's the sort of, you know, where this kind of human computer interaction idea sort of came into play.

Um, So yeah, over time I got, you know, super interested in the human computer interaction side of it as well, um, and I kind of just had a really amazing time doing, uh, sort of like working on Sketch-n-Sketch, doing some research with Ravi, um, and so I just decided to stick with it, um, and yeah, doing similar sorts of stuff now, continuing in grad school, working with Sarah Chasins, so yeah, it's just been a really good time for me.

,

Jared: Okay, cool. Well, let's go back. So you mentioned Haskell at the beginning of this, and that's what sort of started this whole trajectory, but it sounded like you had some programming experience already when you were talking about, because you said like, you didn't see any variables and you know, this is how it was different.

So what kind of experience did you have before that?

Justin: The very first exposure that I had to programming, um, I was actually really interested in graphic design. You know, maybe that came back with Sketch-n-Sketch too, but, I was really interested in like Flash animations, you know, when that was a big thing. I was like, oh, like I want to make like a Flash animation.

I want to make like a game in Flash. Um, and so I was like, okay, how do I do that? Like, I had a technology teacher, I think like in middle school, like not programming or anything, just like, you know, how to use Microsoft Word. He was like, oh, like, I think you have to like do some programming or something like that.

Uh, so I remember I went to like the public library, um, in my, uh, town where I was growing up and I like just checked out a bunch of books on like ActionScript 3 for like, you know, Flash, um, and I had no idea really what like any of it was like doing, um, but it was like super super fun, um, and from there I got like really interested in like Javascript, um, making some like, you know, games like in Javascript or like, you know, different like, I don't know, utilities, whatever, and so I really came at it I guess from the sort of like, uh, Javascript, ActionScript kind of language originally.

and then eventually like in high school there was like a Java course, um, so I was like learning that , and Python, and those sorts of things as well.

Jared: Okay, cool. And then at some point you discovered Haskell, like you said, and that got you interested in the theory behind it, category theory, things like that. And you just, then I guess you, so you were researching for college and that's when you found Sketch-n-Sketch, is that correct? So

Justin: Yes. Yeah. So I, I just, um, reached out to a professor that was at my university. Um, there was like a couple of professors who were like doing programming languages stuff and, um, you know, the stuff that Ravi was working on really sort of spoke to me. So I kind of just reached out to him to chat about it.

Um, and then eventually the chat sort of evolved into like, Oh, like, you know, we can actually help out. And then like, I was really getting involved and I worked with him for like four years.

[00:07:07] Elm all the way down

Justin: And that was pretty much, I think exclusively, except yeah, maybe one of the projects was OCaml, but like all of that was in Elm, which was really delightful.

Jared: Okay. So yeah, let's dive into this a little bit. So Sketch-n-Sketch. You said is what part of it is an Elm? Because I know it's like an Elm like language that you use, but

Justin: yeah. It's actually all written in Elm and there's a language inside of Sketch-n-Sketch that is like Elm that you can use too. It's like, uh, there's a parser and like an interpreter and everything like that written in Elm for Sketch-n-Sketch for a language like Elm. So it's kind of like Elm all the way down.

Jared: yeah, you took the words from my mouth. Uh, okay. So Elm all the way down. Um, and so this parser was, and compiler, this was written, like differently outside of the Elm, Elm parser and compiler. Right? So these

Justin: Yep.

Jared: okay,

Justin: totally separate. Um, initially actually the language that we used was more like a Lisp like language. Um, For the sort of syntax, uh, but then we rewrote that to be more Elm like just because I think we preferred that, um, and yeah, completely sort of separate from any, like, existing tool, um, like, you know, Elm format or the compiler or anything like that, uh, yeah.

Consequently, we don't really have very good error messages, uh, because it's kind of just like a prototype, but yeah.

Jared: Oh, okay. Interesting. So yeah,

[00:08:36] What is Sketch-n-Sketch?

Jared: a little bit more about Sketch-n-Sketch, and this is more, let's kind of talk a bit about the concept and what this is, um, looking at it from the, maybe the perspective of what it's not. So how do Sketch-n-Sketch differ from other visual coding tools like Visual Studio, Delphi or Delphi and Hazel?

Justin: Yeah, yeah, totally. Um, so I think that A really interesting aspect of Sketch-n-Sketch is that we kind of wanted to push the limit of like what would be possible of being able to construct a program, uh, without actually typing any code whatsoever. Um, so you know, you mentioned the Hazel project, um, and, That's actually like one of my favorite projects ever.

I love Hazel. Cyrus is super awesome. Huge shout out there. I think that, you know, the sort of perspective on that project is that they kind of want to, like, revolutionize and, like, modernize, , what it actually means to, like, write code. And so they have this idea of a structure editor, where every single, like, sort of state, like, as you're typing, whatever your editor state is, it's going to have some sort of meaning, uh, so you can, like, run the program even if it's incomplete, you can get type information even if it's incomplete, um, they're doing things about, like, you know, sort of localizing type errors now, um, and, like, you know, new sorts of ways of, like, doing, uh, type, sort of, type based stuff, um, which is, like, super awesome.

In contrast, I would say that, like, Sketch-n-Sketch is more about, like, okay, you know, if we're sort of, you know, committed to, you know, not necessarily textual edits, uh, but something like graphical edits, um, we want to see like, you know, how far can we push just using graphical edits to construct a program, uh, that's kind of like more the direction that like Sketch-n-Sketch would take.

Uh, so for example, um, the sort of lead PhD student, so I was like an undergrad when I was working on this, um, is Brian Hempel, and one of the big projects like in his PhD thesis was basically like getting Sketch-n-Sketch to the point where you can actually like construct fully recursive programs, essentially things that are like looping, um, like building like a fractal sort of.

Using just manipulations on the output itself. Um, so you can write, yeah, just sort of like code that you would normally have to write, like using like a keyboard and like, you know, text editor and everything, solely using sort of graphical manipulations. Um, and so I'd say that that's kind of like the difference maybe between like, um, the work on like structure editing, which is like separately very cool and like, would also be like super awesome to like slot into Sketch-n-Sketch, for example.

and something like Sketch-n-Sketch, which is more like output directed, I think is like the word that we use to describe it.

Jared: Great. Yeah. Thank you for making that distinction. That, I think that makes sense. So

[00:11:03] From Parker Ziegler: Getting started in a large Elm codebase

Jared: I have a couple of questions here from your PhD lab mate, Parker Ziegler.

Justin: Okay.

Jared: And, uh, there's two separate lines of questioning. So I'll start with this, this first set here. It is quote, can you chat about your experience working on Sketch-n-Sketch while in Ravi Chugh's lab at the University of Chicago and how Elm informed the direction of that project?

What was it like getting going with Elm in a multi thousand line code base?

Justin: It's a good question. So, going into the project, uh, I actually didn't know Elm whatsoever. So, you know, I did know Haskell, um, and I was, like, really excited about it. Um, you know, I wasn't not, like, a super advanced user of Haskell, but, like, you know, I enjoyed, like, you know, monads and, like, things like that.

Uh, and so, like, I went to Ravi, and he told me it was, like, in Elm. And I was, like, okay, uh, I guess, like, you know, I can, I can learn this, because I was, like, really interested in, like, working on the project. Um, and I just remember I pulled up the, sort of, like, I think I was like on my phone like getting food somewhere or something like that and I just remember I was like looking at the Official like Elm book and it was like so readable I was just like wow, like I can just like read this like right now I'm like eating food and it was just like really sort of like easy onboarding. I did have like, you know experience doing like functional programming in the past. But like it just felt like you know, unlike, you know, maybe some other languages in the past It just felt like a very sort of like easy sort of onboarding experience for me. And then in terms of the code base, it was definitely, like, a challenge, especially as someone who had never worked in a large codebase before.

Like, I'd only ever just done small projects by myself to sort of dive into something that was, yeah, multiple thousands of lines of code. I'm in a language that I hadn't really used before. But, uh, I think, you know, I definitely credit Ravi here. He gave me, like, really small projects to start out. Stuff that was, like, very well constrained.

Um, not necessarily, like, research, you know, per se. But, like, just getting me familiar with, like, you know, lots of code and You know, how to use Elm and sort of like working up there, um, so I definitely credit like mentorship, uh, to being able to do something like that, in addition to just like how nice Elm is,

[00:13:01] The  Sketch-n-Sketch cinematic universe

Jared: Let me ask you about, kind of going back to Sketch-n-Sketch here and that work. What was kind of the, the result of that, of that work? Like, Were there findings where you thought that, okay, this will be a challenge that we're not sure if it's possible to overcome or what, like, where did it end for you?

Justin: Yeah, yeah, that's a great question. So I would say that, um, I don't know, in my mind I sort of think of it as like the Sketch-n-Sketch cinematic universe. There's a lot of sort of like related projects, um, sort of like different like lines of inquiry, um, which is maybe like sort of how like an academic kind of project goes.

There's not necessarily like a product that's at the end of it, but it's kind of like, okay, yeah, what did we learn? Exactly like you were asking. Um, and so one of the things was just this really cool, you know, push of like, okay, let's not necessarily consider whether we should be constructing recursive programs, you know, completely graphically.

Let's see, like, can we even do that? And sort of just like really pushing hard and like how far can we go without using the keyboard whatsoever? And so for like actual practical applications, you know, we really like this idea of like bi directional programming where you can like edit the code and then also edit the output.

But what if we just sort of put this limitation on ourself where we're only allowed to like edit the output and how far can we go? And so then, you know, So, that's sort of like one kind of like outcome of the project is that we really sort of investigated like what is the frontier, um, at least currently of what sorts of objects we can create, just with graphical edits. And then like Brian went on to do some like additional work for his thesis, I think it was called Maniposynth, um, that sort of like applies this to like a sort of more general domain rather than just like HTML and SVG, um, and so that's one sort of like, you know, line of work that was like coming out of this sort of Sketch-n-Sketch cinematic universe type stuff.

But there's this other line of like, okay, how do we actually sort of work with Kind of like in progress programs, like, you know, as you're building up, you know, maybe the sort of like fractal, or like as you're building up like any sort of program more generally, it's gonna be like incomplete. And so like, how do we handle those kinds of things?

Um, and so this is getting similar, you know, maybe to the sort of like, Hazel type territory. And so something that, you know, I worked on a lot when I was an undergrad, um, also, uh, with Cyrus, he was a postdoc with Rafi, um, at the time. This project is kind of like Hazel slash Sketch-n-Sketch type, uh, project, um, was the idea of like, okay, you know, let's say we have like one of these like work in progress programs.

Um, you know, maybe you're like writing some sort of like recursive function, like a map or a filter or like something like that. Um, but it's like not done yet. Um, and you want to be able to provide examples to the system and have it finish it for you. Uh, so this is the idea of, like, program synthesis. Um, and so we were working on a project like that, um, where it's like, okay, now, instead of having, like, a program and, like, an output, now we have program and examples, and we wanna sort of, like, do something, you know, to, like, bring those together, um, in a similar way to, like, you know, maybe if you have a program and a new value, and you wanna get, like, some sort of updated program.

Um, and so there's all these sort of, like, forward and backward evaluation, like, bi directional stuff, um, You know, related to like program synthesis, like um, creating like new programs to like, you know, make these like SVG type objects. Um, and yeah, I think that like a lot of that work was sort of just like exploring like, you know, what is uh, like possible to do in these sort of like, you know, new kinds of settings.

I'm not necessarily advocating that like everybody should like always just be using their mouse to like always do coding, uh, but it was like really fascinating to see like you know what can be done.

Jared: Okay, yeah, so you were pushing the limits to see where the boundary was on doing that sort of thing, but also things with that bidirectional, where you could go from the code to the mouse and back and forth. I think, just from my own personal experience, because I worked with Visual Studio back whenever that, you know, it wasn't just VS Code, it was like this full thing where on one side, you would have your code.

And then on the other side, you would have this visual representation of it. And where I worked, that was just included, you know, everybody had that product that had the license key. So it was available to me, but I found pretty quickly that I preferred to simply write the code because what would happen is if I touch this visual editor, all the code that I'd worked so hard to create over here became really ugly and like whatever it, it regenerated was just, I don't, it was just terrible. I mean, uh, no offense, I mean, I think the, the concept is great, but just it, it didn't fit. I, maybe I was a little OCD or something, but like, I, I didn't like that, so I ended up not using the visual part at all. But yeah.

Justin: Yeah, yeah, absolutely. No, so I have a similar experience of, like, you know, using Dreamweaver. Um, I don't know if you've, like, ever used that, but it's sort of like Adobe Dreamweaver. And, like, maybe it's much better now. I haven't used it in, like, I think over a decade. Uh, but it would just create the most monstrous HTML.

Like, you could, like, you know, create a beautiful, like, website. And it's like, oh, this is my header, this is my this. And then you look at the HTML and it's like, div like, all the way down, like, you know, just terrible. We didn't formally, like, evaluate this or, like, prove this in any sense, but, like, definitely a goal of, like, Sketch-n-Sketch was to, like, actually construct idiomatic programs.

And, you know, when you're given a program and you're given a new value, what's the sort of, like, minimal kind of edit that you can make to the program without, like, completely just, you know, You know, one way to do it would be, you know, essentially just like throw away the program every single time and then like give you a completely new program that was totally unrelated to your previous program, but like, you know, would produce the sort of like output that you have right now.

So we opted for more this sort of like surgical type approach where, you know, edits here, like in sort of like output can kind of like, you know, propagate back to the input and then have, you know, corresponding changes in the input to get like, you know, sort of like new like input program. I think that it, like, works better.

Um, you know, it's like a lot, um, more involved. Uh, and, you know, it definitely has, like, some limitations of, like, you know, you can't always do that. Um, whereas something, you know, that's maybe a little bit less, um, involved might be a little bit more generally applicable. But for when it does work, I thought it was pretty cool when it did work.

Um, but yeah, no, I totally share the exact same sort of like reservations of like, uh, code generation, um, from these sort of visual tools as well.

Jared: Is there something particular about using a language like Elm to do this based on the fact that, you know, it's purely functional, you know, managed effects, things like that, that made it easier than, say, a program where it could modify state in any way, anywhere, anytime?

Justin: Yeah, so I guess there's two ways of interpreting this question, which is maybe first like, you know, in implementing Sketch-n-Sketch, was it easier to use Elm? And the second way of interpreting it is, uh, was supporting a language like Elm in Sketch-n-Sketch, like, you know, easier versus like a language that could use state.

Um, I'll maybe speak to the latter one first, uh, because I'm super excited about that actually. Um, so, one of the, you know, lovely, lovely things about a language like Elm is that it is, you know, really simple. And I don't mean that, like, in a bad way. Um, you compare it to, like, some other languages, maybe, you know, sort of, like, canonically something like C++ Um, and that language is just, like, very, very complex.

And there's very, very good reasons why it's complex. And it supports, like, a lot of different things. And it's evolved over many, many years. Um, so there's nothing wrong, inherently, with being complex, but if you do have a simple language, like Elm, what that means is that the tooling can be, you know, It's a lot easier to like write tooling for a language that's simpler.

Um, so you know, we only had like a PhD student and like an undergrad, you know, we're sort of like scrappy team, um, and yet we could do some like really awesome sort of like program transformations, we could like re write refactoring tools, we could do like, you know, all these sorts of different things, in a sort of like really manageable way that did not involve like a lot of like compiler hacking or like, you know, all these sorts of things.

Um, and so the barrier to like writing sort of like Language tooling was very, very low, I would say. That being said, it was also not really like a toy language where it's like, okay, this is just a really simple sort of like prototype language that like no one would ever actually use. Like, Elm is like simple enough to like write, you know, tooling for it.

But on the other hand, it's not so simple that you can't actually build like a real system. And you know, like proof in point, like we had like, I don't know, I don't want to quote the exact number because I don't remember off the top of my head, but I think it was like 20, 000 plus lines for like Sketch-n-Sketch.

Um, and so, yeah, it was just this perfect balance of like, okay, you know, manageable enough to like write tooling for it, uh, but realistic enough that you could actually build stuff with it. Um, and so, you know, for, you know, projects that I've worked on, you know, even now, like multiple years later, uh, I've still, you know, sort of targeted Elm for like this sort of like perfect, like, you know, blend between like, I can use write tooling for it as like a single person.

Um, and also it would not be like completely useless. It's just sort of like, you know, in the ether language that I just like made up. Um, so yeah, from a researcher's perspective, it's like lovely from that regard.

Jared: Yeah. And I agree with that, simplicity is not just for beginners that for me as someone who writes Elm code and production all day, we have over a hundred thousand lines of code.

It's the, it's like my calm place, you know, like where I can go and I know that I'm able to be productive, but I don't feel overwhelmed by all of the things that I have to think about. I can just jump right back in no matter when I last left it. So, yeah, I totally agree with that.

[00:22:21] Research on how  statically-typed functional programmers write code

Jared: So then this kind of brings me to your more recent work. You mentioned the program synthesis with Sarah Chasins and, um, Then you also did some research on how statically-typed functional programmers write code. And this even mentions Elm , I believe in both the document as well as the video that you recorded for it about Elm's compiler as assistant.

So, uh, do you want to expound on that a bit?

Justin: Yeah, absolutely. Uh, this was a really fun project, I thought. Um, so, you know, after working with Ravi, um, for four years, um, you know, I decided to go to grad school, and so now, uh, I'm working with Sarah Chasins at Berkeley. And, you know, sort of going into grad school, I had this, you know, idea that was like, okay, like, I really, you know, want to build tools for statically type functional programmers, um, but in order to, like, build, you know, tools that are going to, like, really meaningfully, like, improve the lives of, like, the developers, like, working with these languages, First, I need to understand what are their existing processes.

And, you know, there's a lot of like folklore of like, you know, how statically typed functional programmers write code. Um, you know, maybe they do this or that. Uh, but like, I was like, okay, actually, like, you know, let's just sort of see if we can come up with some sort of like, um, at least somewhat, comprehensive like understanding of like, you know, what are the sort of general strategies um, or processes that like statically typed functional programmers, are employing just in their sort of like day to day practices.

So that's what led me to do this project on how statically-typed functional programmers write code. This was not just Elm, so I also looked at people who were working with Haskell, OCaml, Fsharp, these sorts of languages. But the general premise was that I had two main data sources, uh, for this project, um, one of them was where I had people come in working on programming tasks, uh, over Zoom, cause this was like, you know, really COVID era times. and, uh, the idea there was that, you know, I could sort of give them particular kinds of tasks to work on, uh, that I think that would kind of explore sorts of like questions that I had, like, as I was like watching them, um, and sort of like analyzing, you know, what they were doing.

You know, things would come up and they'd be like, oh, that's interesting, I want to probe deeper. And so I could sort of like, you know, kind of in a more targeted way, kind of see, uh, what sorts of like processes they were working on. Um, and then I also had like an interview with them afterwards, like ask them about like specific things that happened during the study. So that was kind of like, you know, a little bit more of a controlled setting, I'd say. Um, so to sort of like, counterbalance that, uh, I also looked on Twitch and YouTube and I saw a bunch of people, uh, doing like live streams of coding, which was super cool. Um, and so, you know, I was looking at these videos and seeing like, gosh, this is like really similar to like my study protocol where they're here, they're talking about what they're writing code on.

Um, they're sort of just like doing it like kind of naturally like, you know, whatever they sort of like want to work on sort of like out in the wild. And so like I don't have a say like what they get to work on, uh, but it's sort of more like, you know, ecologically valid is, you know, maybe a word that we'd like throw around, uh, for something like that.

Uh, so I sort of was like, you know, looking at these two kinds of like sources of like people writing code, um, sort of like really closely looking, you know, what are they doing like line by line? Like what processes are they doing, uh, to sort of like build up the sort of like qualitative theory of, um, some of these kinds of processes.

Uh, so for example, You know, I've been talking a little bit kind of abstract right here but if we're gonna dive into like maybe like, you know, one of the like sort of details is like You know, there's this sort of like, general like, common wisdom of like, okay, you get your types right, um, and then once you have all of your types right, the rest of your program is just gonna like, flow from that, and everything's gonna be like, you know, perfect, uh, you can like, you know, run your code, um, and it's gonna like, be great.

Um, and, you know, that is true, like in some cases for sure, like if you sit down, and you think really hard, and you get the right type, uh, then the code kind of just writes itself, um, which is a lovely, lovely experience, uh, but actually, um, in a lot of cases, what we found, is that Participants would, like, write a type signature or type definition, and then they would write some code and be like, oh actually this doesn't feel like super ergonomic or like something feels weird about this.

They would, like, think about the problem domain more, they would rewrite their type definition, they would, like, work on some code, and it was sort of like an iterative process where it wasn't, like, types first, program flows from it. Um, so that was, like, kind of, like, interesting to see. Um, another one is, like, seeing, you know, exactly, like, how these folks are, like, using their compiler, like, as an assistant.

Making an edit to, like, a type definition and then seeing, like, all of the use cases, um, of that type, you know, pop up in the compiler error messages and sort of, like, treat it as a to do list. Or other sorts of, like, things like this. It was cool to see like, yeah, some of the vocal wisdom definitely held, some of it, you know, maybe was like a little bit different than like we had thought, um, but yeah, overall, uh, it was a really like pleasant experience, I'd say.

Jared: Very cool. Okay. So yeah, this makes me, want to dive in a little bit about what you're talking about here. So one of the things I could see where, yeah, when you're doing a live coding session, but you are watching it, you kind of have that like Schrödinger's cat problem where it's like, while you're observing they may behave a bit differently than if they were doing it on their own.

So yeah, that's great that you could find multiple sources for that. But then sort of bringing that to what came out of that, like what your findings were. I think this relates to how I love Elm and like one of the reasons that it's so great is that when I'm working, I do this iterative process, like you mentioned, where it's really great with custom types, because like, let's say I'm building like, uh, some way to view errors. And so I have this custom type with different types of errors, different variants, and so maybe I start with only one and then handle that case. And then I add the second case, even though I may have sketched out and sort of commented out other variants and I kind of know where I want to go, but instead of having a whole bunch of errors and have them like partially done, I'd rather take each variant. Solve it to completion and then move on to the next one. And it just, yeah, it's a very nice way I can, you know, get these, I can make commits as I'm going along and see these logical connections between the types and the code that, uh, comes out of it. So, yeah, I think that's, that's, um, it fits with my experience anyway,

Justin: Yeah, that's really cool to hear, yeah, I have to say like one thing that I'm really happy about is that, um, A lot of times, uh, when people like hear me talk about it, or like, you know, even the reviewers for like the paper, I kind of felt like viscerally like, oh yeah, like that's, that's what I like feel like when I'm writing code.

And I feel like, you know, really proud, um, that, you know, maybe I, okay, I'm actually like sort of, you know, faithful in sort of describing how this group of people are like actually writing code. Um, and you know, you know, one sort of thing that is kind of like interesting is, you know, totally did not tackle this at all, like in any of the stuff that I've like done, uh, but it'd be like interesting to consider like, you know, okay, once we have an understanding of how experts that I think type functional program is write code.

Um, is there some way that we can actually use this to inform, you know, maybe how we will, like, onboard people onto, like, learning a new language? So, if you know that people, you know, maybe do some sort of process, like, where they're commenting out different variants in the, um, algebraic datatype or the custom type, Um, they're using their compiler as an assistant, you know, maybe they're doing this iterative process, you know, if we actually know, like, what these kinds of strategies are, like, is that, you know, maybe some way that we can begin to, like, inform, you know, beginner sort of documentation, or like, you know, when people are, like, learning this sort of language.

I don't have any, like, good answers there, um, but I think that it'd definitely be interesting to consider, um, you know, actually the sort of, like, real practices, um, and making that way into the sort of, like, more pedagogical type stuff. Um, but again, that's not my area of expertise, so I can't say for sure, uh, that that would, you know, help.

Jared: Yeah, but I think it, it starts to give us some understanding of where to start with that. So, this makes me think about when I am teaching someone new to Elm how to use the compiler, because often times they're coming from either an interpreted language, like Python, or they're coming from a compiled language where the compiler errors in the type system aren't as helpful, uh, and so particularly when, I guess in both cases, they, a lot of times folks will see that compiler error and it's instead of going to that compiler error and reading it and then trying to use it as a directive, if you will, they instead would See that error and then go back to the code and like try to figure out out on their own without using the compiler error message or make a lot of changes and then try and fix a whole bunch of unrelated compiler errors.

So it, I think that there's work that I think make, could make the experience of programming much nicer in these environments. If there's that process of learning and teaching those. ways of using it upfront versus struggling a whole bunch with like "ah the compiler's yelling at me", when really it's like no, the compiler is your friend, the compiler is your assistant.

It just wants to help you.

Um, but it, but it takes using it in a particular way of like yeah, not

And I know this is in some of your research, of like not expecting the compiler to succeed when you run the compiler. It's more of

I want the compiler to provide an error, produce an error or multiple errors, and then I can go through those and sort of fix those errors one by one. And then when it no longer compiles, I'm good.

Justin: Yeah, absolutely. and I think that, you know, nowadays, uh, so I've pivoted, like, a little bit in the sort of, like, um, research that I'm doing, and that I'm working more with people who do not have, like, a lot of background in programming, so they don't have, you know, familiarity with, like, statically typed functional programmers.

Um, you know, which maybe we can talk about if we have time. Uh, but the point that I wanted to, like, mention is that, like, a lot of times, like, novices, you know, when they get an error, they're actually, like, very scared and they feel a sort of, like, sense of shame and they're like, Oh, no, I've done something wrong.

Like, you know, uh, you know, I'm not a good programmer. Like, you know, these sorts of, like, feelings. And it's, like, quite different, you know, from, like, how you're describing it or, like, you know, from how the participants in the study where they're, like, actively seeking out, like, a sort of, like, error message from the compiler.

Um, and so, you know, one of the sort of things that, like, I'm really interested in as well is, like, okay, you know, how can we build programming systems that are going to be, you know, less intimidating for folks, that are, like, meeting them where they're at. Um, and I think that, you know, Elm is, like, you know, a fantastic example of doing that, uh, for certain, like, classes of people.

Um, I think that, you know, there's, you know, lots of work to be done in sort of, like, building programming systems that, uh, Sort of, yeah, meet people where they're at and can kind of like empower them to do uh, the kinds of computational tasks that they want to do in a way that, you know, makes sense to them.

Um, so that's like, you know, what I'm really, really excited about and, you know, I love to see projects that, you know, make people feel delightful, you know, when they're writing code instead of intimidated or scared or like ashamed. Um, so yeah, I think that that's like, you know, one really lovely aspect, um, of Elm that seems to like, uh, click for a lot of people, certainly for me.

Um, yeah. And I have to say as well, um, something that I should have like mentioned earlier is that like also the Elm community, in addition to the language itself, has been incredibly welcoming, incredibly kind to me. Um, you know, very excited to be like participating in my study, like I've gotten lots of good feedback, um, and so I definitely like have a huge shout out to like the Elm community.

I've always felt like, you know, it's a really, really lovely home, uh, for the language. Uh, so I think that like, yeah, both the language and the community, um, definitely can make an impact, um, in those sorts of things.

Jared: Yeah. Yeah. I'm so glad that you brought up the term. Or the phrase meeting people where they are, uh, because I was actually the first episode of this series of teaching and learning Elm was with Ryan Haskell. And that was one of the things that, one of the ways that I described how he writes his documentation and thinks about it from the perspective of if someone is coming from a framework like React and Vue, like they have a certain set of words that they use.

And so instead of being pedantic about it, Ryan is able to take the words that people already use and create the tutorials and guides from that, so.

Yeah, I think that that's a really great way of looking at it. And to this idea of like, yeah, the shame of errors. I don't think this only applies to novices. I think it applies to anyone who has used a compiler in other systems like, you know, like Java and C sharp. Like I had a history of doing that and I found like I would avoid the compiler, compiler error messages, maybe a sense of shame, but also just a sense of like, It's just a distraction.

Like it just, it prevents me from flowing through the code versus like with Elm where the compiler is actually helping move it forward instead of being a blocker. So

[00:35:11]  From Parker Ziegler: Ergonomics vs. hindrances

Jared: This actually ties into another question from Parker Ziegler here. This is more of a general question or line of questions about Elm and how it applies to, uh, developers, web developers in particular. So quote, from your perspective, what elements of Elm's language design and tooling make it especially ergonomic for web developers, particularly with so many other options out there by the same token, what elements do you think inhibit Elm's adoption at a larger scale?

End quote.

Justin: That's another really great question..

yeah, so I'll say, uh, one thing, um, before I maybe dive into that question specifically, is that, uh, I really believe in, uh, you know, a sort of plurality of tools. So, you know, I'm really excited by the fact that there is Elm, and there are competitors to Elm. You know, there is React, there is Vue, there's all these different things.

There's never going to be, you know, or at least I don't want there to be, one language to like rule them all. And I think that, you know, different folks really like different things and, um, you know, if you really don't like Elm, like that's okay. Um, so, you know, I think that, yeah, that's definitely like my perspective on things.

I think that we should be encouraging like more tools and more ways of like looking at the problems rather than fewer. But, that being said, I also do love Elm, so, um, yeah. Maybe this is not, specific to, like, web development. Just, like, not having to reason about, like, complex control flow, just, like, knowing exactly, like, you know, what sort of things are gonna be executed, don't need to worry about exceptions, I don't need to worry about, like, managing, like, all these different, like, effects. It's sort of, like, just an amazing, sort of, just, like, tailored experience to just, like, writing the thing that you, like, care about, like, when you're doing, like, a web app.

If you're doing like something, I don't know, maybe super low level and you like really care about like, you know, particular performance and like certain things like happening like, you know, maybe at certain times or something like that, then, you know, maybe the sort of like managed effect system that like Elm provides is like not going to be the right fit.

Uh, but you know, if you're just trying to get like an interactive like web app going, um, I think it's just like, yeah, you can focus on the things that you care about, um, and just leave the sort of like runtime and like the effect system and all of that to the computer. I think that that sort of like, you know, burden of not having to like worry about like, yeah, state or like, you know, handling, you know, different things like that is like definitely like one really nice aspect of it.

By the same token though, it's definitely a limitation. So one, I don't know if it's a secret, but it's like maybe a secret about Sketch-n-Sketch, is that actually it uses a forked version of the Elm compiler. So Let's see, I don't want to, like, misspeak here, but, um, I think that, so it's using Elm 18. Um, You know, we haven't worked on it in a while, uh, because there was some sort of, like, use of, like, native code, and there was, like, some, like, modification that we did where, um, there was, like, some deadline, and we really needed to essentially have something like an exception, uh, but we didn't want to, like, rewrite things, so, like, somebody on our team basically made this sort of, like, backdoor where we could kind of have, like, exceptions in Elm, and it's used in, like, this weird spot, and it's just, like, I, nobody really wants to, like, look at that part of the code base, but it is, like, technically there.

Um, and so, there's definitely something to be said for like, okay, you know, sometimes you kind of have to do, have these backdoors. And there's definitely that sort of like limitation of like, okay, if you don't want to take the time and like do it right, uh, then that can definitely be limiting to like be in that sort of like, sort of managed environment.

But that being said, like, I'm not sure how much of a downside like I view that because like, If we were to do this again, I don't think that I would recommend doing that. I don't think that it was maybe the best engineering trade off at that exact time. Or maybe for that exact time, when we were on a short deadline, it was okay.

So, there is always this idea of like, okay, you get a lot of benefits, you can write tooling easier, you don't have to have so much overhead thinking about mutation exceptions, all these other things. But it's like, Sometimes you just want to write an exception, you know? Um, and like, yeah, I feel that.

You know, I also work a lot with, like, OCaml, um, and I really don't like that it has mutable variables until, like, I kind of just want to write a mutable variable. And I'm like, I want to write a mutable variable, but I don't want to have to, like, read code that uses mutable variables, right? Like, I don't want to use, like, a library that has, like, mutation or is, like, doing anything like that.

But, like, you know, I want to do that. Um, and so, like, you know, I kind of say that in, like, a jest, but also, it's kind of true. Um, there's definitely, like, pros and cons. Um, and. Yeah, I would say that like, yeah, just being aware of like, you know, the benefits that the restraint and the language can give you and also the drawbacks, um, is definitely something to think about.

Um, but yeah, I'm working on a project right now or I'll soon be working on a project that's going to be like essentially sort of like front end for like a programming tool. It's going to be like a web app. Um, and even all these years later, I'm still totally going to write in Elm. Uh, so yeah, I think the trade offs are worth it.

Jared: Your preface here about, it's great that there are different types of tools out there. I totally agree with that, that sentiment. Of course, I'm, you know, a huge fan of Elm, but I think that by seeing different perspectives, we can see, yeah, what are the trade offs and like, why are they worth it for me, but they might not be for someone else.

And yeah, that's okay. It's all right.

yeah,

[00:40:25] Fork of the Elm compiler

Jared: Um, but yeah, and I knew that it was a compiled, a fork of the Elm compiler, but I, uh, I didn't know, you know, the details of why. So, yeah, I'm kind of curious, like, have it, if there weren't that time constraint, like how else would that have been done? Like, you know, cause it, could it have used ports or, you know, what other options could be available for that?

But, um, and maybe there, you know, maybe the answer is just, that's the only way or, you know, I don't know, but, um, that might be curious, uh, to, you know, if someone out there is interested in this might, you know, seek to, to find, I don't know if it is that open source that is right.

Justin: uh, yeah, yep, it's all open source, um, so people can totally look at all the, like, dirty hacks. Um, yeah, so I'm trying to remember the exact details because this was, like, a number of years ago, so I might be, like, not getting all of the details perfect. Um, but I think, let's see, so originally, I think the original reason that we had a fork is that there was this really, obscure, like, niche edge case of, like, pattern matching, where it actually was taking, like, exponentially long to compile the program in the number of branches that you had in your, uh, case expression.

Uh, which was, like, really weird, and I don't really remember the details, um, but I think that that was, like, the original reason that we had, like, some sort of fork. and so I think that, like, Brian, like, you know, fixed that, and then we were just, like, using that, um, for I don't know, we had like some big pattern matching like in the code.

It's probably fixed by now. This was like, um, maybe even like Elm 17 at this point. That was, that was like one reason why we had a fork. Um, which is kind of just like a minor reason. Uh, but the reason that I was talking about before with like the exception handling, Um, I think that basically there was a situation where We were like, deeply like, nested in some like, pure code, um, and it was possible that we, it could like, fail, or like, we needed some value kind of like, way higher, like up the stack.

Um, and so like, what would have been involved, I'm guessing, this is just, again, sort of like, you know, very hazy like, recollections, is it would be like, um, refactoring all of the intermediate code to use like an either, or like a result type, um, where it could be like this, or it was like a failure, or it was like, you know, two different things.

Um, and it was like, quite nested, um, and we were like, quite short on time. Uh, so I think that, um, basically just sort of like, hopping up the stack and like going to here, and then like getting the value that we needed, um, was kind of like the decision that was made. I was not like involved in like, that, so I don't remember exactly like what the trade offs were.

Uh, but I think that it would have been a matter of like, Yeah, like refactoring the types of these sort of like intermediate type signatures. As a side note, one sort of like really interesting area of like, you know, programming languages research is this idea of like an effect system. Um, I'm not sure if you've like come across it, but the idea is that, um, it's kind of like checked exceptions in Java, uh, but like way more ergonomic, um, where you can kind of like sort of annotate, you know, the different kinds of, effects that a function might have, like, you know, maybe failability, sort of like, you know, um, or possibly, like, returning an error, or like, using mutation, or like, being able to print out to, like, the console, or something like that.

These are all different effects, um, and so you could sort of, like, annotate, like, all of your functions, um, and, you know, if you trade the sort of, like, changing the type signature for, like, adding an effect, you've essentially, like, not really gotten anything, like, you still would have to, like, add all of these, like, intermediate effects, like, all up the stack.

But one thing that's like really cool is that people are looking into effect inference. So now all of a sudden, if you don't have to actually annotate every single, uh, sort of intermediate function, uh, but you still have it like checked statically, like if you need to make sure that something is pure, you can check that and like inference will like ensure all the way down the stack that you never need to do that, um, then that would sort of like remove a sort of like large amount of like annotation burden.

Um, so that's not an area that I'm looking, working on, um, personally, but I think it's super cool, um, because it sort of gets around this sort of thing of like what do you do when you have like all this intermediate code that you don't want to modify but you have control of the sort of like top and you have control of the bottom and you want to kind of like yeah transfer information between the two um so yeah and you get into things like effect polymorphism which is kind of like type polymorphism but for effects and like all these sorts of things to make it like more ergonomic for that kind of situation.

But we didn't have that, you know, we don't have that for Elm. It's definitely like bleeding edge programming language research, so I think that the change would have been essentially, uh, change all of these functions, uh, in the, in between.

Jared: I gotcha. Yeah, if there were some tooling that could automatically do that, that would be

Justin: that would be awesome. Yeah.

[00:45:04] Cobbler

Jared: I guess then kind of moving on to what you're up to now, um, you have this, uh, from this year, this PLDI paper, Refactoring Programs via Component Based Synthesis Using Canonicalization Based Equivalence Checking.

with Sarah Chasins. Wanna talk a little bit about that. It was recently, uh, Thanksgiving. We're recording this in early December. So, you know, talking about Cobbler, I feel like might

Justin: Yes, so the tool is called Cobbler. Um, so I was working with some undergrads on the project, um, and one of them, Jacob Yim, uh, who's now a PhD student at UCSD, uh, came up with the name Cobbler, which I really liked because I feel like it cobbles things together and like, it's like a pie, which is like really cute.

So, um, the tool is called Cobbler. Um, it's again, sort of like a research prototype, but it does work on essentially like an Elm like language. It actually uses Elm format under the hood, which was like super awesome that I could like reuse that tooling. uh, the idea is like if you have some code that is like written using something like direct recursion, um, or, you know, sort of like pattern matching, um, but you want to refactor it to use like a library, for example, the Elm standard library where you have like map and filter, um, normally you would have to sort of like manually do that rewrite.

Um, and so, you know, one of the participants, or actually, sorry, a number of the participants that I was like, um talking to essentially expressed that sometimes it's easier to just you know, pattern match on your data, and then just like bang, bang, bang, like handle each case sort of independently. And then you have your function and it works.

Uh, but actually they would prefer for like readability to have like a nice pipeline where you have like the pipeline operator It's like take the data then do some like result combinator and then do like a list dot map and then like do something like that But it can be like a lot harder for them to figure out like, okay exactly, what combinators do I need like at each stage?

Versus something like a pattern match where it's just super clear like okay. I have a nothing. What do I do? I have a Just like what do I do? Um, so So, yeah, this tool uses, uh, program synthesis, which is essentially, um, the computer, like, generating code for you, um, to give you, like, given, um, the sort of implementation of the code where it's, like, Using the pattern matching and the direct recursion, it'll give you an equivalent piece of code that uses this sort of library components, sort of any library that you want.

I sort of just use the Elm standard Library for like an example, and it'll refactor the code to use those combinators instead. It's not perfect, it doesn't always work, because yeah, a variety of like, you know, technical and theoretical reasons, but it can work with things like, you know, map and filter. And, uh, yeah, all these sorts of like combinators like that.

[00:47:36] Justin's current work

Jared: I guess this is a project that you recently, uh, finished. Is there anything else that you wanted to talk about before we move on to picks?

Justin: um, yeah, so I guess this will maybe relate a little bit to the pick that I have in mind. Um, but yeah, I mentioned before, you know, that I'm not working directly with statically typed functional programmers now. Um, I've sort of like pivoted a little bit to working more with people who don't have a lot of experience like, you know, writing code.

And in particular, I've gotten really interested with people who come from sort of like non traditional computer science background. In particular, a group of folks like biologists. Um, who are producing like a lot of data, like in the wet lab, doing a lot of really cool, awesome biological experiments, uh, but don't usually have the sort of computational background to do like all the kinds of analysis they want.

Um, and so it's been a blast to be like learning a lot about what their sort of like computational needs are, you know, how they think about problems. And, you know, what kinds of programs they need, um, and, you know, in what ways existing programming languages are sort of serving them, and in what ways they're not, um, and, yeah, I would just really encourage anyone who is, you know, really looking for some, like, really interesting, fun, and sort of, like, impactful, sort of, like, work, uh, to look, you know, beyond people who, you know, we might traditionally consider, like, programmers, or, like, software engineers, Uh, because, yeah, lots of people need programming, um, and there's this sort of, like, lot of really cool, uh, work to be done, and, you know, what would a programming system look like if we designed it, you know, with biologists, uh, for their needs, sort of thing.

Rather than, you know, sort of just saying, okay, You know, this is the system you have to use. Um, you know, that's just kind of how it goes. Um, so, yeah, I don't really know, you know, exactly what that's going to look like right now, sort of like right in the middle of that, uh, but I think it's like a really exciting area.

And, you know, I think that Elm has done this, you know, a lot, to like a large extent, with like, lots of different people, you know, it's like, what would it look like if we did functional programming, you know, for the web, um, You know, sort of like bringing these different kinds of ideas to different kinds of audiences and, and I think that's just like a really, really exciting kind of area.

Uh, so that's what I'm working on right now, um, and I definitely would encourage other folks to think about, um, kinds of problems like that.

Jared: Cool. Yeah. Well, I'm excited to find what comes out of that. And also, it sounds like you're thinking a lot about programming for folks who, yeah, are novice and, and not necessarily already knowing how to write either, you know, statically function, um, statically typed functional languages or, um, yeah, just beginners.

And so, yeah, that sounds really neat. So what kind of biolog, biology experiments are these?

Justin: Yeah, so, you know, there's like a whole world out there and I'm definitely, you know, just beginning to like learn like a lot of these sorts of things. Um, but like I've been working really closely with a lab at UC Berkeley, um, and the kinds of things that they're really interested in is, um, essentially called epigenetics, which is the idea of like, okay, we have our genome and you know, that's going to sort of like dictate, you know, uh, what kinds of like, you know, genes are like expressible by our body, like in, you know, the different kinds of like genes that we have, uh, for the proteins.

But, you know, our eyes, for example, are producing different proteins than, you know, our liver cells or like other parts of our body. And the way that that gets controlled is not necessarily like the DNA itself, but sort of like attachments to the DNA that can kind of like regulate like, okay, I want to turn this gene on now, or I don't want to turn this gene on right now.

And that can be regulated by like environmental factors or like, you know, heritable factors, and these other sorts of things. Um, so it's really about, like, the sort of, like, control of, like, what kinds of genes are getting expressed at, like, different points. Um, and they're really interested in studying, you know, how that works, basically, in humans.

Uh, can we, like, modify, you know, the sort of, like, epigenome of humans? Can we, you know, do different, like, kinds of experiments like that? Um, and, um, Yeah, essentially it involves a lot of like, you know, dealing with like physical objects, which I'm not used to, you know, I'm used to like working with computer, but there's like, you know, pipettes, and like tubes, and they're like mixing things, and they're growing cells, and they're like, you know, doing different treatments to the cells, um, and those sorts of like experiments.

Uh, but yeah, biology is like a super, super like vast world, um, and yeah, definitely really excited to, uh, learn more about it.

[00:51:45] Picks

Jared: Very cool. Okay. Well you mentioned a pick that you had, you wanna move on and see what that is.

Justin: Yes, uh, so I have a book recommendation that I am currently working through right now that's related to this idea of, um, designing with the people, uh, that you're designing for. Uh, it is called Design at Work: Cooperative Design of Computer Systems. Um, it is a really awesome book, um, it's like a whole collection of people, uh, who have thought, you know, really seriously about this idea of like, okay, what does it mean to do participatory design where, you know, our sort of participants are actually going to be actively involved in the design process.

Um, and yeah, it's not exactly clear yet what that looks like for like programming languages in particular. Um, but you know, this can be applied, you know, more broadly for computer systems. Whether that's just like people who are, you know, entering data like into a computer or like using other sorts of like applications that are not necessarily like programming based.

Um, there's this Enormously, sort of like, rich literature on like how we can involve the users in the design process, um, that I'm also still learning about and really excited to be learning about. Uh, so yeah, that would be Design at Work, uh, Cooperative Design of Computer Systems.

Jared: All right. Yeah, that sounds interesting. That something that I might like. So, Yeah, I have one pick as well, and it's again kind of, uh, you know, uh, picks for the people that, uh, I am, uh, interviewing, and so kind of, uh, You mentioned biology.

I knew that you were starting work in that. And so I've started reading this book called _Entangled Life _by Merlin Sheldrake. And it's about fungi and mycelium and mushrooms and molds and all these really interesting ways that Surround fungi and like how they interact with the world and how they interact with us even.

If you are interested in how systems work and just maybe thinking differently about what it means to think, like what, because we have such a history of thinking of, uh, like this, you know, CPU or brain based, uh, thought, but, um, the way that, uh, mycelium work in, in these, like they have this experiment that I think it really, um, makes it interesting to think about is, uh, how mold, uh, can recreate the, uh, railway network of Tokyo by placing, uh, different food in different places.

And so it finds the shortest path. And so it really takes, uh, a different set of, um, assumptions or help makes you, requires you to, uh, maybe rethink your assumptions, uh, and how you think about thinking. So, um, yeah, that's, that's my pick. Um, Entangled

Justin: That sounds super cool.

Jared: yeah,

Justin: I think I've heard of that, uh, Tokyo experiment before, um, but I didn't really know anything about it. Uh, so that sounds really fascinating.

Jared: Yeah, that's just one little bit of it. I mean, it goes a bunch of different directions. But, um, yeah, I'm only like halfway through it right now. But I will recommend at least the first half. I don't know where it goes from here.

Justin: okay, I'll only hold you responsible for the first

Jared: Yeah, thank you. But yeah, do you have anything else?

Justin: no, I think that's it. Thank you again so much. This was a really lovely conversation. Um, I really appreciate, you know, how thoughtful you've been, um, and all the questions and the preparations. Uh, so yeah, thank you so much for having me.

Jared: Yeah. Thanks, Justin. Folks out there who are interested and enjoy this show, please rate and share, and I'll put show notes for everything we've been talking about. So, yeah. Thanks, Justin. Have a good one.

Justin: You too.