Koala on the iPhone
Koala is a sampler for iOS and Android with a well thought out workflow, designed by Marek Bereza. The workflow keeps you from stumbling down a rabbit-hole of micro-editing, tweaking parameters, undoing, redoing, etc. I spoke with Marek who now lives in London to find out more about his design process and background.
A Long Line of Samplers
Koala is the latest in a long line of samplers I’ve made over the years. Over the last 20 years, I’ve been making things that record sounds and then do stuff with the audio.
Koala’s design was heavily influenced by a myth about J Dilla creating his last album, Donuts, with the very straightforward Boss SP-303 sampler.
This story of using the SP-303 to make an album distills the notion of creativity from constraints. Given this limited sampler, look at the magical music you can make with it. As a musician, I found that really inspiring. So I just wanted to make something quite simple that gives you a lot of power. One trick of the SP-303 is resampling its own output. For me, as a design concept, I think it is similar to a Spirograph. The Spirograph idea is 2 circles, you have a big circle and a little circle. Those 2 circles are very simple, but they add up to more than the sum of their parts. When you start using it, you start getting all of these complexities. That is what I like about the magic of resampling, you can keep putting things through effects and round and round until it is completely unrecognizable. You can then play a rhythm with that and put it through a delay. Then sample a little bit of that and it turns into a tone, like a synthesizer. I really like that.
Don’t be too precious with creating, just get on with it and make some music.
Boss SP-303
Interaction in Musical Instruments
The whole idea of coming up with things you never would have on your own is one thing you can really do with interaction design in musical instruments or software. You can basically stop people from going down the route they usually go down. I found this when I was younger. After a few years of opening up the piano roll and putting a bass drum here and a snare drum there it gets really dull. You can make software that forces people to do things differently. For instance, that is why Koala doesn’t have a piano roll. This forces people out of their comfort zone and to play in real-time. When you play things in real-time it tends to be more musical.
People do get upset with Koala not having a piano roll. The first prototype of Koala did have a piano roll by accident. I wanted to make a bold decision. Maybe some people think I don’t know how to program or don’t want to be bothered. It was an active decision not to have a piano roll.
Design
You have to be not too afraid to scrap something that is hurting the design.
A lot of the design came from trying it out. When there was a frustration I would try and address it. You have to be not too afraid to scrap something that is hurting the design in some way. I got it down most of the way in the paper prototype.
Koala Screenshots During Development
UI Evolution
I put a line of code in my program right at the beginning to take one screenshot after 5 seconds of running the app and just save it into a folder. I made it into a video so you can see it evolve. It is nice to see your own process as it evolves.
WWDC 2015 Designing With Animation
At WWDC 2015, I did a talk about prototyping with two other colleagues called Designing with Animation . The year before there was a talk called Fake It Till You Make It, this was the sequel to Designing with Animation.
At Apple, I did a lot of drawing and showing pictures of things to people. It saves you so much time. You don’t have to program the whole thing to find out if it will work or not.
So You Started Prototyping Koala on Paper?
I usually like to start programming early in a project, but with Koala I spent a month drawing on paper. The good thing about drawing on paper is the sky’s the limit, you are not restricted, you can try out any idea quickly. It is a good way of collating your thoughts.
Eventually, you have to start drawing simple images on the computer because you need the fidelity and accuracy. I use Sketch, it is used a lot for web and UI stuff. I’ve fallen in love with it, and it is wicked. I just started doing iterations in Sketch and working on a list of problems I had to solve with the interface. I did hundreds of evolutions of the interface. It started out looking awful, then it got slightly better. It is not good to increment too much. It is better to take a lot of u-turns and not be afraid of throwing things away and starting again. Losing some work is ok, avoid the sunk cost fallacy. I got it to the stage where I was mostly happy with but not finished and I started taking it to a few friends who do music. I talked through the designs with them or just asked them to look at it. This is a standard thing to do in UX UI design. This is before I actually made anything interactive.
The prototype screens were made in sketch then put on a phone screen. Then it becomes more realistic, to get an idea about finger size. I made a little app that shows a photo of the prototype with no chrome, so you can touch the screen without zooming or revealing the chrome. That’s something I learned from Apple.
Programming Begins
Once I was reasonably happy with it, I was ready to start programming. I already had done some of that before, it is almost like I have a virtual box of sampler parts on my hard drive I’ve written for other things. The whole of Koala is 20k lines of code. It is not massive.
Creating Your Own Library
While prototyping Koala, I wrote a library that is similar to Open Frameworks and Cinder but it is very specific to my needs. For instance, it manages all of the low-level drawing, audio routing and zips files. I keep adding features, it is almost a bucket for my good bits of code. It runs on Mac, Windows, Android and Raspberry Pi. It has a really useful feature, live coding in C++. Generally, when you program in C++ or other non-interpreted languages, you have to press the compile button, wait for one minute and then see if your change worked. In my system, I can type a bit of C++, and every time I press save, it compiles just the file I’m working on and injects it into my running program. It is primitive and has some quirks but I’ve shared it publicly and people are using it. If you are doing graphics programming, it is amazing, you can see things moving as you are typing. For sound you can very easily A/B something, you can change a variable and you can hear it right away. You can get something looking good or sounding good very quickly.
Koala Performance Effects
Performance Effects
The inspiration for the performance effects came from Beardyman and Tim Exile. They have live looping setups with loads of effects on faders and can really alter the sound quickly, and it looks so fun. In Koala, you touch the screen and an effect goes on, you take your finger off and it goes off. The idea that you can perform effects, rather than keep them on all of the time is interesting. The effects are part of the instrument.
Background and Prior Work
I initially got a BS degree in computer science, but I think I went to the wrong university. I already knew a lot about how to program and I was even working on the side doing full-stack web development in the early 2000s. I graduated and thought, what am I going to do with my life. I have this crappy web development job that pays well, but I’ve got to think of something else. I booked a flight to Japan and stayed there for 3 months. Someone persuaded me to study industrial design because I like making things.
I then went to the Royal College of Art (RCA) in London where they had a course called industrial design engineering. It is industrial design and mechanical engineering. You get 2 master’s degrees, a master’s from RCA and a master’s from Imperial College.
After graduation, I started doing freelance work. Being someone who could do design and programming, I fell into creative coding, doing arty installations for brands, in-camera stuff, and projection mappings for music videos.
What Are Popular Tools for Creative Coding?
There is a big creative coding scene, the community is really good and there’s lots of support. If you need to do something like switch a light on and off, you can find a library to do it. The main creative coding libraries are Open Frameworks , Processing , and Cinder . In the public spirit of it, everything is open source and there are many plug-ins for frameworks including some by me at OFX add-ons .
Open Frameworks is C++. The argument for C++ is if you are doing art stuff you want to be able to use cameras and talk to hardware, and that is most widely supported with C++. 5 years ago, that was more difficult in a web situation, but trivial in C++.
The problem with C++ is it is not very well designed. It started out as C, then they tacked bits and pieces on to it and now it is just, oh my god. Compared to C++, these languages have clever features that prevent bug-prone code from compiling; Swift , Kotlin , and Java. These languages are designed in ways that help you not make mistakes you’ll never find again.
What Did You Create for the U2 Music Video Invisible?
The U2 Invisible project was a 5 day job. I was there to do the programming based on art direction. Marshmallow Laser Feast asked me to put some particles on the screen and join them up with a line, doesn’t matter where the line is. What? It is really easy to do, but I don’t think it will look good. They said it looked fine.
Did You Work on the User Interface for the Google Nsynth?
I helped Google Creative Lab design the user interface of the Nsynth . It was a proof of concept and an aesthetic. With a lot of the AI stuff, it is brightly colored and not dark to be positive and show the light side.
You Created an Early Javascript Synthesizer
I think it was the first time I heard you could make audio stuff in a browser and write your own synth, instead of using building blocks or using samples, I had to try it. I wanted to build something that worked in a browser partly because I could share it so easily. It doesn’t use the modern web audio stuff, so it may not work for most people.
Did You Write All of the Audio Code?
Yes, it is attributed at the bottom of the page. I’ve developed DSP skills over time, but I’m not that great at the hard math. I have a library of stuff that has disappeared off the internet. I took a lot of C++ code I had and transliterated it to JavaScript. It is super simple. To write a saw wave oscillator in C++, there are quite a lot of ways to do it. You’d think that you just draw a triangle wave, but if you do that it doesn’t really sound that good because of aliasing. You get some strange artifacts. There are slightly more complicated ways of making a saw wave. One way is to have a series of sine wave generators that you add up in the correct harmonics to create a saw wave. Sine waves do not alias. Or you could do all sorts of clever optimized stuff that was worked out in the early days of digital synthesizers. It is really complicated. The JavaScript project was the naive approach with the fewest lines of code.
What Did You Work on at Apple?
I was on a team called HID prototyping, short for human interface devices prototyping group. Bret Victor was on the same team. We were prototyping and coming up with ideas for how people would use new sensors and hardware. I worked a lot on the front-facing 3D camera. Working out how people would want to unlock their phones, What privacy things they would like to do, for instance not showing the contents of messages until you look at your phone. I got 3 patents with other people for camera-related stuff. It was a fun job, I loved the people I worked with. There were 9 of us and 4 or 5 were British, 1 New Yorker, a very cynical design team.
Developing Koala
When Did You Build Your First iOS Apps?
I made 2 apps in 2009 and 2010. As soon as I got an iPhone I wanted to try making something on it. I like the idea of having a musical instrument in my pocket for making my own music.
How Did You Come Up With the Name Elf Audio
I like the idea that I’m the elf in the workshop since it is just me doing everything. More like a trouble-making elf or goblin.
Since You Have Built Many Samplers, What Are Some of Your Favorite Commercial Samplers?
My first sampler was an SU10 by Yamaha. The cheapest sampler you could buy at the time. I wanted to make drum & bass and trip-hop which was almost purely based on samples. With breakbeats, you have a snare sound with the tail of something else. I thought I needed to get a sampler to do that kind of music. Though the SU10 was inexpensive, it was quite difficult to sort out the financial aspect.
Later I managed to get an obsolete Mac when my dad was working in multimedia, they had loads of computers. I soon discovered you could make music on a computer and you didn’t need so much hardware. It took a while, but eventually, I had a sampler inside the computer with as many samples as I wanted.
My big criticism of every single commercial software sampler I’ve used is, none of them can sample. I used to have an Akai S2000 where you could plug your CD player in, press record and capture audio in the box. In both the early software sampler EXS24 , and Ableton , you have to record the sound elsewhere then bring the sound in. If you have a cup and want to make a noise with it you have to record it on the timeline then drag it into the sampler. It is not really that difficult but it is not really that straightforward. I like the immediate thing. This is very important for Koala.
The Capture App
I made Capture 10 years ago for the Mac. The idea is your QWERTY keyboard is the instrument keyboard or you can use a midi keyboard. The app has 2 modes, you are either playing or recording. In record mode, every key is a record button. Press “A” to record and in play mode, press the “A” key to play back. I made that and found it really fun to capture sounds. I had an old plastic guitar, and it only took a few minutes to make this multi-sample of this weird instrument. You just play it on a keyboard and it sounds magical.
The concept of slots in Koala came from the Capture app. That was the start of Koala.
Did You Use Auto Layout?
No, I’m not using the Apple interface stuff, I’m drawing with OpenGL which is deprecated in favor of metal. The layout is flexible because it has all of the design rules like padding and margin. All of the drawing is in C++.
C++ is what you used for Koala?
It is great for developing an iOS app, you can run it as a native program on your computer, it is much easier to work with and less sluggish than the simulator. It works on Linux and Windows as well.
The initial android port, to get all of the graphics on the screen and the sound working, with loads of bugs, took a day and a half. The glue code is different, setting up the window, loading files is different. Everything else is standard. It is kind of weird to think of C++ as a very portable language, but it really is.
Tools
There’s a really nice audio editor called Ocen Audio. It’s really well-considered, it is like the Sketch of audio editors. I used Sound Edit on the Mac when I was younger. It was just the best program. It turns out to be partly designed by an interaction designer from the Royal College of Art. In Sound Edit you would select waveforms like you select text in an editor. It had the same color, I really liked that shared language.
I use Max for Live to prototype audio stuff. Not always but it is definitely a handy tool.
I love Sketch and try to use it for everything. I use Pixelmator for images.
Releasing Koala
How Did You Get the Word Out When Koala Was Released?
I wrote an article about the story of why I made Koala. I saw other apps that had press packs, so I created my own press pack that included this article. I emailed the usual suspects and used their submission pages. I made sure I had a video to watch that wasn’t too long. I created an ad for Koala that just describes what it does and there’s a tutorial where I go into more detail.
What Has Been Most Rewarding About This Experience?
I think seeing so many people using the app. It has made so much music. Koala had a little hand in helping musicians create their music. The app is inexpensive, and I hear things like people say they sold their Pocket Operator or they bought an iPad so they could use Koala. That feels really good. The main thing is it is helping people make music in ways they didn’t know they could make music, and that’s really cool.
Koji Seto made a Japanese language video about Koala and the app has blown up in Japan. Koala has been the number one paid music app for weeks. I’m obsessed with Japan. Considering all of the time I’ve spent absorbing design sensibilities in Japan, it is really nice to know I might have rolled some of that into Koala, I don’t know.
Customers
A lot of times, people come with a solution. It might not be the right solution, but they identified the problem.
A lot of times, people come with a solution. It might not be the right solution, but they identified the problem. I need to look at the problem causing people to ask for a whole slew of features focused on one area of the app. It is very useful to get that feedback. I do value suggestions, please keep them coming. I keep an unofficial tally of how many times people ask for things because that identifies a problem.
Audiobus is very good for customer feedback, there is one thread that is 40 pages long.
Future Projects
I would like to collaborate a bit more on my next project. It would be cool to find someone like an illustrator in the right sort of style, something based in art and first principles.
A lot of graphic design revolves around making websites, and the designs are so similar. With Koala the aesthetic is not everyone’s cup of tea and I don’t really even like it either. I just wanted to make something that is not this or not that.
I love Skeuomorphism, I’m happy with flat stuff too. I just wanted to do something completely my own thing. I spent a while trying to make an aesthetic and set it on this orthographic thing so it didn’t look like other apps or a website. That was important to me.
How Do You Handle Many Suggestions From Customers?
I’m very aware of the design by committee issue and bloated features. There’s a lot to be said for one decision maker in design. I’ve seen so many things ruined by too many cooks.
People are good at identifying problems, but sometimes their suggestions are not the best way to fix the problem. It is really important to keep everything coherent. As soon as you start bolting things on, it falls down and ruins it for other people.
For me, the workflow is very precious, it has got to stay like that and I don’t want to pollute it. People have suggested, don’t add anything else, it is perfect, please don’t listen to people on the forums. I think that is really important.
Conclusion
My motivation for Synth Talk is to highlight the thought, effort, and philosophy that goes into designing a product. Marek has worked on really interesting projects while creating samplers of various types for years. Great to have a sampler on iOS focused on making music and fun to use.
There will be more articles to come highlighting the person and the process behind the products we love. Check back here or subscribe to the newsletter if you would like to hear more Synth Talk.
Links
If you have suggestions or topics you want covered please contact me. 🙂