The Making of Sonic Beat: Solving rhythm gaming for mobile devices

I’ve played rhythm games all my life. I’m a collector and enthusiast. I have metal dance pads, bongos, ALL of the guitars, an electronic drum kit with all the trimmings. I stream to Twitch. I host Rock Band parties at local businesses. I have dumped thousands of dollars over the years into equipment and DLC.

Knowing this about me, imagine my excitement these last 12 months. A new way to play Guitar Hero? Sign me up! My Rock Band library on new consoles? Absolutely! A growing music game scene on Youtube and Twitch? How many webcams can I plug into this computer!

But GH Live has a free-to-play model with no buy-out. Half of my Rock Band songs won’t import. I can’t play my beloved Pro Drums (until October or so). The legacy instrument adapter is laggy. I’ve been as heartbroken as I have been giddy.

Then around Christmas my boss Jason asked me something wonderful: “How would you like to make a music game?” Stars shined in my eyes and I immediately started listing the ideas that had been accumulating during our three years making apps together.

Some of them we’d talked about before, but this time it was serious. We had always stopped short for one reason or another. Certainly, it didn’t seem as if anyone else had really figured it out on mobile yet. Rhythm gaming on phones hasn’t been as fun as it should be. Fingers covering up notes. Poor performance and issue of timing accuracy. A million StepMania clones. Limited to one platform or another. Total lack of content. A bias toward one handed-ness or the other. Unskippable, insanely wordy Japanese-style “tutorials.” Forcing you to hold the phone in a way that covers up the speaker. The list goes on and on.

We could do better.

What finally convinced us was a little 3DS game called Theatrhythm. I brought it in and had Jason play it. Before long, he was as excited as I was. Not being a music gamer himself, he had been rightfully shooting down many of my ideas for fear of function or scope, as well as the fact I had not really offered any solutions to the failures of the apps that came before. But Theatrythm held the keys to not only a great rhythm game (which it is andwhy hasn’t anyone played it, what’s wrong with you), but a great mobile phonerhythm game. The “tap, hold, swipe” gameplay was uniquely mobile, and it solved so many problems. And it could be even better on a capacitive multi-touch phone.

“So just make Theatrhythm, then” I can already hear you thinking. You see, Square Enix already did that. If you’ve played the iOS version of the game, you understand it suffers from nearly all of the problems listed above. It even chugs at about 15fps on our office iPhone 6+. And if you’ve played the Android version... oh, there isn’t one. There’s only this:

But we had other ideas for the formula. Several excited whiteboard drawings later and we had a design that appeared to work! We quickly boiled down the aspects of what would eventually become Sonic Beat at a level of scope our team of two could finish in six months. We knew our game:

So let’s look at these requirements and explain the design process along the way.

GAMEPLAY and ACCESSIBILITY

The transition from the resistive stylus controls of Theatrhythm to the capacitive thumb controls of a phone game brought new challenges to consider - challenges I feel Square Enix did not handle well for their own mobile port. For instance, upon building our first playable prototype we became certain the game was meant to be played in portrait orientation. Not only does this accommodate the differences in hardware, it makes for an ambidextrous interface.

And with that first big change, more design decisions seemed to fall into place. Unlike the 3DS, we had access to multi-touch. With two thumbs tapping simultaneously, the mechanical and skill ceilings of the original format could be raised significantly. One tester gleefully likened it to learning how to alt-strum in Rock Band all over again.

Another thing Theatrhythm did differently from other DDR-style games was to not require the player to tap specific places of the screen. Unfortunately this can be a little confusing in practice, so Jason simplified the concept down to a single note lane. This felt so much easier to read and play, and the new design stuck. This decision moved the gameplay away from one that required precise aiming of inputs to one that could focus just on the fun part - rhythm. This is much more enjoyable than trying to force what works for a four-panel dance game into the confines of a five inch touchscreen.

Yet early testing showed this wasn’t enough. Players still desired to tap the middle of the note target, covering it with their fingers. We struggled with how to fix this, as we believe strongly that nobody (read: NOBODY) reads tutorial prompts. Our eventual solution was to add two pseudo tap zones that animated when the screen was touched, like pinball flippers. Behind the scenes, the player could still tap anywhere, but the added panels ensured that even if the player ignored our short tutorial and starts pressing the tap zone, they are drawn to the sides thanks to the constant visual cue.

ACCURACY

One of the biggest problems with every music game I played on mobile, even the really big ones, was accuracy. Accuracy is really four separate problems from a UX perspective. The obvious one is “lol are the notes on time.” The answer to that is more difficult to answer than you’d think.

Going into this, we had no idea how much of a challenge it would be. There are tons of factors, any one of which can go wrong and collapse the whole effort. Many mobile rhythm games rely on programmatic beat detection, but this can be finicky. Perhaps the best example I can find is Beat MP3, which is genuinely fun to play, but you can tell concessions were made regarding timing windows to accommodate for the imprecise nature of beat detection algorithms.

We chose to go with authored song charting so as to eliminate poor beat detection as a factor, and this decision ended up being a cornerstone of the project. The excellent “StepMania” dance game simulator is well known for its accuracy, and its charting format is easy to understand and tested against years of users attempting to break it in hilarious ways. While Sonic Beat does not itself use the StepMania simfile format, our choice to author songs using StepMania’s editor and convert the output for use in our game solves the problems of charting accuracy and music timing offsets as inputs for the game engine.

Then there’s gamefeel (further reading: everything by Vlambeer). A surprising number of music games do not offer calibration tools, and this comes from the mistaken belief that your target gaming hardware will always produce perfect results. It also assumes that the ‘technically perfect beat timing’ visible in a waveform would be the correct time to trigger each note. Theatrhythm makes this mistake, even on the 3DS. Without calibration tools, wearing the wrong headphones can destroy the experience (40ms delay on some noise cancelling headsets, and as much as 200ms over bluetooth). Even then, your player may subjectively prefer the beat ahead or behind. Perhaps s/he’s tired, or wired on energy drinks, or a percussionist forever cursed to be more correct than your best efforts. Offering players choice is almost always preferable to a one-size-fits-all solution.

The third factor is legibility, or how well can you read the rhythms presented on screen. We had several examples to choose from when designing our note lane. One way to help players stay in time is through animation or color shift. The osu! series employs this strategy, as does Dance Dance Revolution using its default arrow set, but it’s not something I personally endorse. DDR also has the option to color notes based on their quantization, which is closer to reading sheet music and would have worked well for Sonic Beat. We instead went with Rock Band’s approach, adding measure lines to the note track. It’s very subtle, and it allows us to keep a standard set of note gems.

Finally, repetition. I strongly feel it’s important to be able to play the same track over and over again, improving as you go, learning new rhythms by rote practice. Lack of practice-friendly gameplay has been the sole reason I’ve uninstalled several other music games over the years. And truthfully, I understand why many of these games have randomized notes - it’s easier to program and allows for virtually infinite content and levels of play. However, it also replaces the skill ceiling with a mechanical ceiling. Anyone who’s played (the excellent) Piano Tiles games has probably run into this problem.

<iframe title="Embedded content" src="//www.youtube.com/embed/fqOW84ZTL7k?wmode=transparent&amp;jqoemcache=Tq7Ld&amp;enablejsapi=1&amp;origin=https%3A%2F%2Fwww.gamedeveloper.com" height="315px" width="100%" data-testid="iframe" loading="lazy" scrolling="auto" class="optanon-category-C0004 ot-vscat-C0004 " data-gtm-yt-inspected-91172384_163="true" id="763648763" data-gtm-yt-inspected-91172384_165="true" data-gtm-yt-inspected-113="true"></iframe>

That’s not to say random is all bad. By ditching random charting and audio beat detection, we are surely limiting ourselves to the content we can create for the game by hand. We do have ideas on how to implement procedurally generated song charts that we might explore in a future version of Sonic Beat, likely as a separate mode. It’ll be like making a whole new app,though.

All of this - authored charting, timing accuracy, legibility and repetition - I feel this is the magic formula for a music game worthy of beginners and experts alike. A large portion of development time was devoted to making all these things come together, and I think it’s well worth the effort. But it does come at another kind of expense.

CONTENT and VIABILITY

We had seemingly backed ourselves into a corner. Now we needed authored content on a shoestring budget. We already had an excellent (and fast!) authoring solution using the StepMania editor, but we couldn’t afford to license more than a couple of tracks for the game. And music licensing itself seemed to be reserved for companies like Harmonix, or those friends in the music industry and weeks-long content pipelines.

At first, we played around with the content model employed by BeatX, a particularly good StepMania clone. BeatX boasts “more than 100,000 songs,” which is technically true. BeatX directly supports StepMania simfiles, and they can easily be played via a phone’s external storage. We already had the ability to import simfiles into Sonic Beat, so we expanded the option and used it for testing. We liked how the 4-panel dance charts played so well in the Sonic Beat format that we turned it into a feature that still exists in the final product.

But we knew this wasn’t enough. People would enjoy reliving their DDR Supernova tracks in a new way, but ultimately people would really want to play marathons of popular music like, um, Justin Bieber. It was Jason’s idea, and his dedicated programmatic wizardry, to do something much more bold.

He engineered a way to take music already on a player’s phone, optimize it for use in the game, and apply my authored song charts to the result. The first time I saw this work in real time I was floored! The flood gates had opened. We could make gameplay content for any songs we wished, so long as the player already had the songs!

Bingo! No more legal grey areas surrounding packaged StepMania simfiles. No more worry over licensing budget, which we had already depleted after two included tracks. Even better: using songs available across iTunes and Amazon MP3 normalized the tracks we could support. By making the game content free and encouraging players to legally download supported tracks via their favorite music stores, every song downloaded for the game directly supports the artists. And unlike discrete DLC models for games like Guitar Hero and Rock Band, when someone buys a track for Sonic Beat, they own that track.

Combining this “must own the song” requirement with our StepMania importer offered yet another benefit: user generated content. Legal user generated content. Over the lifespan of the game, we plan to support players who go the extra mile and chart their own favorite tunes into the game. The StepMania content pipeline allows anyone to easily test their work in the game without expensive dev kits or complicated mods. With only a few lines of extra metadata added to a simfile, users can share their works with each other minus any actual music files, all without breaking copyright law. And after all that, we have the option to feature community charts in the game without jumping through licensing red tape. Literally everybody involved wins.

But the best part of our new content pipeline was that it narrowed our monetization options. Other games had tried a multitude of gross strategies, none of which we