You can listen to it here.

I used my Samsung Galaxy S2 and decided to use its gyroscope in conjunction with the accelerometer and the magnetic field sensor in order to measure the user’s head rotations both, quickly and accurately. To acheive this I implemented a complementary filter to get rid of the gyro drift and the signal noise of the accelerometer and magnetometer. The following tutorial describes in detail how it’s done.

There are already several tutorials on how to get sensor data from the Android API, so I’ll skip the details on android sensor basics and focus on the sensor fusion algorithm. The Android API Reference is also a very helpful entry point regarding the acquisition of sensor data. This tutorial is based on the Android API version 10 (platform 2.3.3), by the way.

This article is divided into two parts. The first part covers the theoretical background of a complementary filter for sensor signals as described by Shane Colton here. The second part describes the implementation in the Java programming laguage. Everybody who thinks the theory is boring and wants to start programing right away can skip directly to the second part. The first part is interesting for people who develop on other platforms than Android, iOS for example, and want to get better results out of the sensors of their devices.

Update (March 22, 2012):
I’ve created a small Android project which contains the whole runnable code from this tutorial. You can download it here:
SensorFusion1.zip

Update (April 4, 2012):
Added a small bugfix in the examples GUI code.

Sensor Fusion via Complementary Filter

Before we start programming, I want to explain briefly how our sensor fusion approach works. The common way to get the attitude of an Android device is to use the SensorManager.getOrientation() method to get the three orientation angles. These two angles are based on the accelerometer and magenotmeter output. In simple terms, the acceletometer provides the gravitiy vector (the vector pointing towards the centre of the earth) and the magnetometer works as a compass. The Information from both sensors suffice to calculate the device’s orientation. However both sensor outputs are inacurate, expecially the output from the magnetic field sensor which includes a lot of noise.

The gyroscope in the device is far more accurate and has a very short response time. Its downside is the dreaded gyro drift. The gyro provides the angular rotation speeds for all three axes. To get the actual orientation those speed values need to be integrated over time.  This is done by multiplying the angular speeds with the time interval between the last and the current sensor output. This yields a rotation increment. The sum of all rotation increments yields the absolut orientation of the device. During this process small errors are introduced in each iteration. These small errors add up over time resulting in a constant slow rotation of the calculated orientation, the gyro drift.

To avoid both, gyro drift and noisy orientation, the gyroscope output is applied only for orientation changes in short time intervals, while the magnetometer/acceletometer data is used as support information over long periods of time. This is equivalent to low-pass filtering of the accelerometer and magnetic field sensor signals and high-pass filtering of the gyroscope signals. The overall sensor fusion and filtering looks like this:

So what exactly does high-pass and low-pass filtering of the sensor data mean? The sensors provide their data at (more or less) regular time intervals. Their values can be shown as signals in a graph with the time as the x-axis, similar to an audio signal. The low-pass filtering of the noisy accelerometer/magnetometer signal (accMagOrientation in the above figure) are orientation angles averaged over time within a constant time window.

Later in the implementation, this is accomplished by slowly introducing new values from the accelerometer/magnetometer to the absolute orientation:

// low-pass filtering: every time a new sensor value is available
// it is weighted with a factor and added to the absolute orientation
accMagOrientation = accMagOrientation + factor * newAccMagValue;

The high-pass filtering of the integrated gyroscope data is done by replacing the filtered high-frequency component from accMagOrientation with the corresponding gyroscope orientation values:

fusedOrientation =
    (1 - factor) * newGyroValue;    // high-frequency component
    + factor * newAccMagValue;      // low-frequency component

In fact, this is already our finished comlementary filter.

Assuming that the device is turned 90° in one direction and after a short time turned back to its initial position, the intermediate signals in the filtering process would look something like this:

Notice the gyro drift in the integrated gyroscope signal. It results from the small irregularities in the original angular speed. Those little deviations add up during the integration and cause an additional undesireable slow rotation of the gyroscope based orientation.

]]> http://www.thousand-thoughts.com/2012/03/android-sensor-fusion-tutorial/feed/ 24 http://www.thousand-thoughts.com/2012/03/master-thesis-printed/ http://www.thousand-thoughts.com/2012/03/master-thesis-printed/#comments Sat, 03 Mar 2012 17:13:16 +0000 Paul http://www.thousand-thoughts.com/?p=759 Finally, here it is. My finished thesis… …in print…

Actually…	…looking at it this way…

…	…

…Whoa!

Just a few week ago, Tobias Knoll, a freind of mine, and Ralf Zimmer started a German podcast on games. They talk about game topics on a quite regular basis and cover them more deeply in my oppinion than the mainstream formats in Germany do. Their talks are particularly interesting due to the fact that both of them have so different backgrounds. While Ralf is an independent software developer and game designer, Tobias has recently finished his thesis in study of religions and aims for a Ph.D. in his field.

The podcast is issued every Monday at 0:00 am on www.about-gamescast.de.

Big Time Monkey has been chosen for the best Fan-Adventure in the year 2011 by the German site adventure-treff.de. We are very proud of it and we want to thank all people who voted for our game. Thanks a lot, guys!

That’s what I had in mind when I previously wrote about head tracking with an Android device.

This contraption is a smaller part of the whole system (which consists mainly of software). The communication between host machine and the phone is already working. The main application running on the host is receiving sensor data from the phone’s accelerometer, magnetometer and gyroscope. The refinement of the sensor data is happening on the phone itself. That’s what I’m actually working on at the moment. The information is still quite inacurate and I’m trying to improve it with a complementary filter.

When it’s done and working, I’ll write some details about it.

I’ve struggled a long time with this. Almost everyone of my close friends (including my brother) got in possession of one of those fancy new smartphones. Half of them have an iPhone. Very early, it was quite obvious that this wasn’t just a fad. It was the future. The capabilities of those things are quickly converging towards what the clumsy boxes we used to put beneath our desks did. Nevertheless, I avoided this next generation of devices like I avoid facebook (no, you won’t finde me there). I don’t like how people show off with them. It was just another status symbol at  first. And I do not care much about status sybols.

However, as a developer I eventually had to confront those things. Sure, you could make cool games with them. Java programming became also more and more interesting for me, due to it’s shorter development cycles. For a long time I was convinced that programming with C/C++ was the real deal. Later, I realized that the end justifies the means in this regard. And then, I started with my Master Thesis and realized that head tracking will become an integral part of my work (more on this in a later post). An employee at the University refered to a talk on Sensor Fusion on Android Devices as a possible approach for this problem.

Long story short: This plus chrismas and I got myself my own Android powerded device.

I started to work on my final thesis for my master’s degree. The thesis is about “Binaural Synthesis in Interactive Media”. I discovered that the title is not self explanatory, so here come some explanatory words :

Binaural hearing is the ability of humans and animals to use both ears in order to determine from which direction a sound they are hearing originated from. Without binaural hearing your expensive Dolby 5.1 sound system would be useless to you because you couldn’t tell the difference between 5.1 sound and mono sound. Also, binaural hearing makes the cocktail party effect possible.

Binaural synthesis is the process of synthetically generating a two channel singnal (i.e. a stereo signal, one channel per ear) out of a mono signal which can be perceived by a human in a spatial manner — creating a “binaurally hearable” signal as one might say. So, you do not need 5 channels to make 3D-sound possible. Everyone has just two ears, ergo: two channels should suffice — when using headphones, of course.

The choice of this topic for my master thesis originated from an earlier interest for sound based games. I was fascinated by the idea of creating a game whith no graphics at all. But in my opinion today’s sound technology is still missing some capabilities which are necessary for a proper sound-only-experience. Well, let’s see what we can do about it . Of course there already have been many scientific investigations and approaches to this problem. I’m by far not the first one to work on this topic.

By the way: recently I found this via RockPaperShotgun. There are two developers who are working on a sound based adventure game called Blindside. It’s comforting to know that the subject is still up to date and that you’re not the only one who’s working on it.

Also, if you want to read the short version of my gibberish, scroll down until you see “Concluson” written in big letters . As the title suggests, it concludes the previously written text in a handy set of bullet points.

So, sit down, have a tea and let me answer following questions:

• What went wrong?
• What went well?

What went wrong

Just to make it clear: Big Time Monkey was a success. It even surpassed our expectations. However, there are always things which could have been made better for the sake of optimization. Those things are mentioned here.

Workload

For three people Big Time Monkey was a hell lot of work. In our first meeting in October 2010 we recognized that four months were far too short to realize all our concepts. Also, most concepts were not written, yet. It was clear that we had to cut away a lot of stuff if we wanted to finish the game. And still, it was not enough. We did not finish the final game in time. But we managed to deliver a demo version of the game and present it at the MediaNight.

Product Language

Big Time Monkey is a German game. That’s good for Germans but not so good for the rest of the world. We would have increased the number of people playing our game by making it international. If we’d had developed the game in English it would have increased the audience drastically.

Then again, we had not enough experience to write all the jokes and wordplays in English. And finding professional native speakers who fit the characters in the game and did not charge a cent for a day in the recording studio would have been impossible. So actually we had no other choice. The game would not have the same quality if it had been developed in English.

Technology

We have chosen Visionaire Studio to develop Big Time Monkey. It helped a lot and it took away much work from us. But as every technology it needs to be evaluated and understood properly before it can be useful in its whole extend. We didn’t use features of Visionaire which would have made our lifes much easier simply because we didn’t know they existed. For instance: We didn’t know about the debug console inside of Visionaire’s testplayer until I discovered it by accident… after the release.

Visionaire also had it’s own bugs and problems (like every software) we did not encounter until we used an affected feature. Some of those problems were not documented by the developers of Visionaire. Some I believe weren’t even known at all.

To mention an example: We used certain types of actions (provided by Visionaire) to trigger voice-over playbacks throughout the entire game. In the Testplayer of Visionaire  those voice-overs were played correctly. But in the release build of the game those voice-overs did not play. It was a bug in the deploy function of Visinaire Studio. This bug has been discovered by another user of Visionaire about one week before we encountered it. He reproted it in the Visionaire forum. The developers fixed it but the fix would have been applied in the next release of Visionaire which was one month in the future. But we had our release date in two weeks. So we had to sit down and apply workaround solutions in the entire game. We had to move all affected actions into another area. But the Visionaire GUI did not support copy and paste for list of actions, only for individual actions. This has worsened the entire situation. We had to copy one action at a time, navigate though the whole GUI, paste it, go back and proceed with the next action. It was quite an ordeal.

Another point was platform dependency. I’m a PC guy, it was ok for me that Visionaire was a Windows-only solution at that time. But we have a lot of Mac users walking around at our university who were quite interested in the Big Time Monkey. I ended up putting all of them off saying it would be possible to play the game on a Mac in a distant future. And I didn’t like that. I wanted to see as many people as possible playing our game.

I’m still struggling with the thought of developing an own engine for Big Time Monkey. But then, I’m sure it would have meant much more work which would have resulted in an unfinished product.

Legal Stuff

One month after we released we were contacted by a games magazine. They wanted to release our game on a DVD with their next issue. That was a big deal for us. But it included signing a license agreement with them. The problem was, I was not the only one who owned rights on Big Time Monkey. There was one musician and six voice actors who owned the rights on their individual parts because we didn’t sign any contracts with them. It was a student project and nobody had seen the necessity to do this. We had to cantact each of them in retrospect to obtain their permissions. Next time I would handle such things in advance.

What went well

Quite a lot! To begin with: the constelation of the team.

Team

I would have never expected that the three of us would complement one another so well. Frankly, I expected tensions inside of the team at some point in the project progress. Not because I thought the team members were tension prone but there are always some sorts of conflicts and misunderstandings among people. Maybe two people want to take the lead at the same time or somebody is unsatisfied with the work of another team member. But not in our team. Everybody was in charge in the same way. Every decision has been talked over with the team and in the end everybody was satisfied with the result.

Why? The first advantage we had: we were a small sized team. So this is about the upsides of a small team. You don’t have to introduce complex structures. The probability that two individuals conflict is much lower in a group of three people. It is easier to stick the heads together and sort things out. It’s easier to define milestones and come to an end with a discussion.

Last but not least we were lucky. Somehow we were a good match. Each of us is open to criticism. Nobody tried to force through his own ideas and contributed constructively to the product. This is not always the case and it is almost never controllable what type of people come together to accomplish something. During the development of Big Time Monkey I’ve heard of other student projects which suffered of problematic team constelations.

Conception

Our conception phases were fun. And that’s the point! If you have fun doing your work you most certainly to it right. It was like sitting at a round table and throwing hilarious ideas and catchphrases at each other trying to make the other laugh. And if they laughed you knew it could work. If not, you tried at least and maybe you inspired a similar but working idea. Of course, we always had to take our time management into account. But the best ideas emerged from such discussions.

Also, during work it occured that one of us stuck his head out and came up with something the others had to evaluate. If it was good, it has been included into the concept.

Beside brainstorming, we had to record and structure our output somehow. So we used a graph to visualize the scenes of the game and how they were connected. Marius who was mainly responsible for story and concepts maintained all concept documents and wrote dialogs. He used simple screenplay writing for static dialogs and flow charts for interactive ones.

Voice Acting and Music

It really paid off to take some time searching for voice talents. Our first thought was to do the voice overs by ourselves but now we are glad we didn’t do that. It would have destroyed the whole experience. The voice actors gave a fair amount of depth to the characters in Big Time Monkey. They also incorporated own ideas into the characters and made them more alive.

The first people we were casting for character voices were professionals who did moderation, ads, documentaries etc. We quickly realized that we need real actors… like in theatres! And after some searching we actually found people who were able to understand and impersonate a character. Especially Torsten who did our main character Gordo was a perfect match.

Marius knew everything about every character in the game. Even stuff which is never being mentioned in the game. He developed elaborate background stories and played with stereotypes to add depth to them. Most characters in the game have less then two pages of dialogue. But they were so well developed that those lines sufficed to transport their depth to the player. Marius’ knowledge about every single fictive individual inside of But Time Monkey made him an essential source of information and inspiration for each voice actor.

As for the music, well, that was another case of luck. I don’t know how Marius found our musician and sound designer Ingmar. But what I can tell is that the process we established between Ingmar and the rest of the project was a good one. At first, we described the moods of the different scenes in the game as good as possible and provided mood pictures to Ingmar. He then came up with some demo bits of music and athmospheric sound and we told him what to change. It was quite straightforward.

Conclusion

Here come the promised conclusive bullet points:

Workload

• Always define milestones. Having the roughest plan helps.
• Get sure to have enough people to get the work done in time or cut the concept down to the essential parts
• Try to get stuff done on a regular basis; one small finished piece a day. It’s not easy to have a constant productivity level but once established it’s one of the most precious things you can have in a project.
• Getting things done and presenting progress to the team increases overall motivation and thus productivity.
• Grant yourself breaks! You won’t come far when you’re overworked.

Product Language

• If you want an international product do it in English first.
• If your English isn’t good find somebody whose English is good and who would understand your project (and your humor if necessary).
• A bad translation of your game can destroy the whole game experience.

Technology

• Know the technology you’re using.
• Build prototypes to try out features and workflows.
• Familiarize yourself with the technologies documentation.
• Never use undocumented technology.

Legal Stuff

• When working with external contributors, always work out some sort of contract to define who owns which rights over the product (even when the product will be free of charge)
• Even if you intend to do a non-commercial product, think about the sitiuation when your product gets commercial; just to be prepared. There are many legal cases where a free product becomes commercial without the creator having thought of it (e.g. release on a print magazine’s DVD)

Team

• Try to keep your team small as long as it is able to get the work done.
• Don’t make your team too small. Recognize when you need another pair of helping hands.
• Meet your team on a daily basis.
• Try to be on a par with everyone. Show respect.
• When working, try to sit close to each other (e.g. not in seperate rooms).
• Always know and understand what the others are working on and inform them about your own work. That’s good for both motivation and organization.

Conception

• “Kill your babies”: throw away ideas which you can’t convince your team members of, even if you think “this is the best idea i’ve ever had”. Maybe your team members have a different point of view on things and see weak points you don’t see. Also, if your team is not convinced of an idea it most probably won’t be satisfied with the implementation.
• Cut parts of the concept away if you realize you won’t make it in time. Even if the parts of the concept were promising.
• Try to understand how ideas of your team members work. Listen to them! Give them a chance to convince you! You’re not the only creative mind in the universe.
• Express your doubts, but do it politely.
• Discuss details. It’s good for the product and for the motivation. But always return to the big picture.

Voice Acting and Music

• Never underestimate voice acting. Trained people are always better than laymen.
• Prefer actors. Professional speakers are ok as long as they can act (e.g. change their voice, impersonate characters)
• Choose your main character’s voice very well. An inapropriate voice can annoy players and make them quit the game after the first few bits.
• Know your characters well! You need to understand them and how they actually think.
• Know the mood in your game. You need this knowledge to choose the music. Without the music the mood will remain in your head and never reach through to the player. Music is a very important and often underestimated component.

As Asamak already pointed out in the comments of the previous post:

Big Time Monkey is being mentioned in the current issue 9/2011 of German games magazine COMPUTERBILD SPIELE. It also appears on the Silver-DVD attached to the magazine among 100 other freeware titles.

It’s also very pleasing that Gordo is printed highly visible on page 95. We all who participated in the creation of the game are glad to hear so much positive feedback on the game.  Finding our own game in one of Europe’s biggest games magazines makes our work feel particularly appreciated.