san francisco ca
rjbrown at gmail
Below you'll find samples of my work, showing that I am at my best when working in that nebulous area between art, user experience and hard-core, down-in-the-weeds, hands-dirty coding. I love to invent useful and fun-to-use tools, especially those that find a unique, clever and "cognitively-optimized" way to communicate a concept graphically. You may also pick up that I am a passionate believer that everyone should learn to code, and I particularly enjoy introducing others -- children and adults alike -- to the delights of coding, as well as to technological approaches to graphics and music.
Pianoply (pee ANN uh plee) is my most ambitious home-project yet, and there is a lot to it. It's basically my answer to "why didn't you learn to play piano well as a kid, with a beautiful piano right there in your house?"
Because this didn't exist then.
Pianoply allows you to learn piano/keyboard, right in a web browser, with color-coded, "cognitively efficient" graphics synchronized with, and overlaid on top of, the original music videos on YouTube. The app actually works with a digital piano plugged into the computer, and uses the WebMIDI, WebAudio and YouTube APIs.
Pianoply is particularly inspired by my now 6 year old daughter's love of music, and yes, she loves Pianoply and enjoys all those glowy color effects as much as I do.
I've been tinkering with the technical side of color since I was a teenager with a 35mm camera and a set of color gels. In this video I show one particular thing I've invented and built, which is a way of replacing color with transparency in images, so that you can...well, do interesting, useful, and sometimes crowd-pleasing things. You can even try it yourself on the web page, with your own images or with a few samples. (and here's a link to a more technical explanation of what's going on under the hood)
Also, another quirky innovation shown briefly in the video is a unique drawing tool that allows drawing smooth curves with the sort of precision of cutting paper with an X-Acto knife (it's what you might call "directional"). Also a handy color picker I made, which better models how I think about color. And a tool for making hand printing take on the look of a glowy lightsaber, because everyone loves lightsabers.
Clocking in at around 10 minutes it isn't exactly a short video, but it is pretty information-dense without getting hyper-technical. And there is a good dose of eye-candy.
These two videos each attempt to communicate the same concept: how to use "keyboard commands" in Pianoply. (which allow you to use your computer keyboard to very efficiently change settings or jump around in the video)
The videos also give you an opportunity to see some of the features of the product that aren't shown in the intro above. The UI is admittedly a bit quirky, but also exceptionally efficient and functional.
If you are simply learning how to use Pianoply, you can watch one or the other, since they each cover mostly the same information. They are both less than 2 minutes long. The respective approaches to the videos themselves might be considered experimental (the first one showing a bit less restraint with the colorful overlaid graphics), so they might both be worth watching for that alone.
Pianoply is in beta and you can try it out at the expected place. For now, it's computer only (there is an alpha version of a mobile-device compatible one, but it isn't public yet). I suggest watching at least one of the two videos above (on keyboard commands) before trying it.
While this isn't quite as graphically interesting as the other videos, it's the one that, well, kind of freaks people out. It has all kinds of crazy implications to those who like to ponder all the intricacies of copyright law.
Basically, this shows how any Pianoply user can take any of the tens of thousands of "Synthesia videos" on YouTube (videos showing people how to play songs on piano, and do so in "pianoroll notation" as opposed to "sheet music"), and easily and accurately scrape its many megs of video data into a couple kilobytes of textual MIDI. In terms of copyright law, those videos are considered "performances", but when you add this simple tool into the mix, they are actually delivering data, i.e. "the notes." (bring on the lawyers!)
Of course, Pianoply then allows you to display that MIDI data layered top of the original music video of the song, synched perfectly. Super cool, or kinda scary, depending on which side your bread is buttered.
These two videos get deep into the design concepts of Pianoply, including an analysis of the geometry of piano keyboards and how to best represent it on screen, and how to best make it work well with the underlying YouTube video (to maximize visual feedback and visual appeal, balanced with other considerations).
Also, I talk about future approaches that might work well with such keyboards as the new LUMI from Roli, which can programmatically light up keys in colors. It even touches on music theory and how 7 colors of the rainbow and the seven letters of music (A-G) can be viewed analogously, especially as we deal with indicating sharps and flats.
While this may seem a bit overly music specific as to be relevant to other things I may work on, it hopefully sheds some light on my general approach to design, which goes further below the surface -- and deeper into the details of geometry, logic, learning and cognition -- than might be typical of a "front end developer."
This one is less about showing off the software, and more about the philosophy of learning to play piano using a more modern approach: specifically thinking of learning in a way that is optimized for digital pianos connected to computers, and as much for creating rather than just playing.
I made this video as part of an introduction of Pianoply for parents and kids who would be doing group music sessions in the pre-quarantine world.
One of the interesting things it presents is how sheet music vs. pianoroll can be seen as analogous to fractions vs. decimals. (the latter ones having large benefits especially with digital devices)
Who said Pianoply is all about music? Well, it is, but a byproduct of its development might actually affect a larger audience: per-syllable YouTube captioning. Whether or not it is a song.
Remember, this overlays on videos, and you don't have to have made the video to add the captions. Imagine parents curating and captioning videos for kids, and kids learning to read while they watch programs they enjoy. I have seen the power of this sort of "passive learning" with my daughter, and I don't think I can overstate how major a deal this could be.
Alas, I haven't had time to build it out as I'd like, but what I've experimented with so far is promising.
This is just crazy powerful, and has all kinds of uses, including as a teaching tool. I personally use it to develop new features, prior to moving the code off into proper .js and .css files and so on.
In my dreams, though, I envision a world where web apps are designed to be user-extendable in all kinds of ways. This is just a hint at that world.
This video -- which admittedly is quite long, but hey there is a lot to show! -- also demonstrates an isomorphic micro-framework called "TemplaTools" I built which allows templatizing HTML in a particularly powerful way. While perfectly good for production code, it is especially designed for teaching web development, using a hybrid string/DOM approach.
This one goes way back, but is certainly one of the things I've done that has had the most impact. AccuDraw is a very important (some say the most important) tool in the Computer Aided Design program MicroStation. It could be described as 3d T-square and triangle, and made drawing in both 2d and 3d far more intuitive, interactive, and fast than anything that preceded it. Honestly there is still nothing like it. It was and still is widely celebrated and imitated.
It is still in heavy use and very popular a quarter century after it was introduced. The first couple minutes of the video show some of the discussion of it still on the web (much of it wishing that the well known competitor AutoCAD had a similar tool), and the last part of the video I give a demo of a good bit of its features. (the video is pretty long, so feel free to skip around, as in all of these videos)
I completely conceived of and designed AccuDraw (and the other tools shown, SmartLine and the view controls) and wrote all their code, in C. It received a patent in which I am listed as the sole inventor.
Aardvark was released in 2005 and was hugely popular overnight, showing up on every web developer web site and instantly getting tens of thousands of downloads from both regular web users (who liked to use it to clean up a page prior to printing) as well as web developers (who used it for quickly analyzing other people's web pages). It influenced the Firebug Inspector which in turn influenced Chrome and Safari's Web Developer Tools. It was eventually purchased by Google.
It is still a super powerful little tool, even for being no more than a couple hundred lines of code. For what it does, it is still the most efficient tool out there.
And, it shows another micro-framework that, like TemplaTools, smooths the transition between HTML coding and web app development.
This video, like the videos below, is a bit less polished than some of the videos above (these are just me giving a demo while talking).
HyperMatch was another "widgety" invention, developed in Java when Java applets seemed new and shiny. I designed and built it in 1999-2000.
It got a lot of attention at Craigslist where it was introduced in "demo mode" prior to a planned big launch. People were very excited about it, but it never did launch at Craigslist due to their new CEO getting cold feet about it or something.
I eventually sold it to ezboard and it found a kind of awkward new home for a while. Regardless, it is worth checking out via its current home on the Wayback Machine, if only for all the excited comments from users of Craigslist (starting at 2:36), back when Craigslist was still local and edgy and cool.
Here are a few simple apps I've worked on over the years that are each innovative and/or interesting in their own way. You can play around with them here if you want.
The video at right explains why "first choice of the median voter" is something to aim for, by using a simple election where we vote for a numerical value rather than discrete candidates. You can try it and read some accompanying text here.
The one at left demonstrates another conceptual method, which is where voters pick their favorite candidate (like today), but the candidates themselves can pass their votes on to other candidates if they themselves can't win. (this video has no sound)
Finally here is one that goes back the furthest, to 2003, when I was experimenting with making a Condorcet election interactive and with it very easy to understand the results. Showing the Condorcet results as a meaningful bar graph was less than straightforward.
The first shows a typical example of me having made a custom little web app to help her learn math concepts that she was previously having trouble grasping at kindergarten prior to quarantine. The app helped immensely, and quickly -- and she found it fun. Having the web cam showing ourselves "reflected" behind the graphics was a hit. (the video is quite long, as it shows two subsequent nights worth for a total of 25 minutes. It's kind of adorable though. And no one said you had to watch the whole thing)
Other videos are mostly fun, but they also show my creative approaches to parenting and teaching. One thing you'll notice (especially in the video at right that spans several years) is that my approach to teaching is big on "scaffolding," such as the way I hold her hands or have her use a trapeze while she is learning roller sports. It keeps it fun while letting her gain confidence and learn skills quickly. I think the same concepts apply to educational software that I design.
By the way, I made the rather unconventionally designed skateboard and swing you see in the videos. The swing has been a wild success (who knew you could do those things on a swing?), and she's as good a skateboarder as I've seen for a kid her age.