Sensor Fusion on Android Devices: A Revolution in Motion Processing

[@jianguo5228]

Even 6 years later, it seems to be the most comprehensive talk about sensors and sensor fusion I have ever watched. Thank you!

[@Fatih-Ada]

Even almost 8 years later, I totally agree :)

[@HabibRK]

Indeed ...

[@ArpanDasS]

Even almost 9 years later...

[@evilraccoon6538]

ok, 3 jun 2019 the same. I am happy to find this video.

[@iyxan23]

Welcome to Jun 26th - 10 Years ago...

[@sanjivsoni2389]

I saw this video again after three months and theres still so much to learn about this. This is so far the best resource on Motion Sensors and sensor fusion I have come across.

[@Gunner3210]

Wow, we designed the exact same thing to use for our own sensor fusion algorithm. Never knew that Google was using something exactly the same. Talk about parallel innovation!

[@saxman58b]

The best explanation of how to interpret quaternions I've seen. Thank you!

[@invensense1]

Edit: In the slide on "double integration", the total time of those data sets are actually 10 seconds, not 1 second. --- David Sachs

[@miojovege]

10 years Later. Still Amazing

[@harleydavidson845]

Checkout DARPA’s Micro-PNT (Micro-Technology for Positioning, Navigation and Timing) positioning tracking technology, it does absolute position tracking on a single chip with sensor fusion. To oversimplify it; Micro-PNT adds integrates a highly-accurate master clock (TIMU) to existing chips gyroscope / accelerometer to simultaneous measure the motion tracked and combines that with timing from the syncronixed clock, and with sensor fusion makes single chip that does absolute position tracking.

[@ishkool8664]

so did they achieve success?

[@marquesghm]

One of the best talks I have ever seen! 👏

[@dhairyamittal940]

Apple taking notes... "Ripped out the headphone jack"

[@ApocolypseZombie]

Awesome talk, even from a general sensor overview perspective

[@s4ril]

It was incredible. This video help me a lot to approach my thesis based on staft related with acc. Thanks. GreatJob

[@gnychis]

great description of why position using the sensors is challenging, thank you

[@Tadesan]

5:00 street view is run by the same technology pioneered by the video game Mist.

[@seppl5372]

how accurate is the video these days? did the sensor accuracy develop and it works better today?

[@ASelimSalman]

I have proposed a similar topic since Sept. 2009 at my MSc. I wish I had the opportunity to continue researching about!

[@AshNapToys]

I'm pretty sure that the gravity bias graph at 25 mins is wrong. If you had no gravity compensation the most you could move in a second would be 0.5g which is about 5 meters...

[@icecream296]

Can someone explain how the pan change worked at 38:00 assuming I have got the rotationMatrix from the sensor library, how to I pan the 2D image on my mobile screen?

[@kylemichaelbecker]

@ the question about a dynamic x and y reading, with a static z value, my first guess would be that x and y would reflect lat, long; wheras your altitude(z) may be fairly constant. hope this sheds light

[@tomp2309]

very good video, helps me so much writing my master thesis. additionally he has so a clear pronunciation

[@BBorn223]

sorry, I know this is inappropriate, but can I see your thesis?

You know the video will be interesting when it is 4:3

[@nyehnyi]

the best online lecture i ever watch..thank you so much sir

[@WarmWeatherGuy]

23:22 You aren't removing gravity, you're adding gravity. Accelerometers can't measure gravity. When you're sitting still on a table (for example) you're not accelerating. The accelerometer measures the force of the table pushing up but not the gravity pushing down.Once you add the gravity that the accelerometer can't sense you end up with zero for acceleration.

[@johnkruebbe4276]

This is how we could do position in VSN. Thank you!

[@TheiLame]

42:17 Hi im little bit dumb. How does he get 9 numbers from 3 axis points? How does the twist work to give 6 extra numbers? "Those axis twist around with the teapot..." i dont get it.

[@saxman58b]

+TheiLame I believe what he meant was that each of the three rows of the 3x3 matrix are the vectors describing the new x, y and z axis relative to the original xyz dimensions. For example, if you wanted to just rotate around the z axis by 90 degrees then the matrix to convert old dimensions to new would be: [0 -1 0 (vector describing the old x axis relative to the new coordinate system after rotation, i.e. x' = -y) 1 0 0 (vector describing the old y axis relative to the new coordinate system after rotation, i.e. y' = x) 0 0 1] (Z axis doesn't change so all vectors keep their same z coordinate) To test it, pick an arbitrary position in space, for example (1, 2, 3). If you multiply the matrix above by this vector you get a resultant vector of (-2, 1, 3), which is exactly what was intended, a 90 degree rotation around the z axis. This was a simple example, but it should work for any 3D rotation. Just make each row of the rotation matrix equal the vector describing the new direction of the three local x, y, z axis of the object. And unit vectors are required in the matrix, or else the space / object will expand or contract. Thus a 45 degree rotation version of the above would be: [sqrt(1/2) -sqrt(1/2) 0 sqrt(1/2) sqrt(1/2) 0 0 0 1]

[@TheiLame]

saxman58b Thank you very very much for explaining! :)

[@markchinski5071]

Search youtube 'linear algebra matrix transformations' for more info.

[@chiubobo4955]

I bought an IMU for practing embedded system and try to use integration to get the sensor displacement. It's turns out I'm a fool in the perspective of 12 years ago video.

[@PatLane4069]

Really, really excellent presentation. Thanks heaps.

at 24:24 he said it wrong. It drifts only 2cm not 20cm. As you can see later on next graph blue line is on 8,5 but green line is flat, and on first graph is at 0.020, not 0.20. If the error was 20cm then it would be obvious not flat. Or, he maybe thought 20cm after 10sec

[@belm0_]

Regarding the gravity signal, he says "it's actually mostly gyroscope data, but with the accelerometer used to correct drift" (at 17:59). I can't find any information on how this is done. The W3C Motion Sensors Explainer hints at the same thing: "On most sensor hubs, gravity is isolated from the accelerometer using the gyroscope" (www.w3.org/TR/motion-sensors/#relative-orientation), but also lacks any detail.

[@MrKarln]

25:30: The drift is 8.5 centimeters in one second, not 8.5 meters. If i drop the device it will fall (displace) 4.91 meters in one second. The integral of a small error on gravity is not larger than the integral of gravity itself.

[@sarnathk1946]

Thats nerdy stuff. The guy presented so well...

[@TheiLame]

This is 5 years old video, is it still up to date? :) anyone knows?

[@saxman58b]

+TheiLame The math and physics all still makes sense. What has changed is that most phones now have all of this hardware built in, and on the iPhone at least (I only do IOS development) the software functions are now built into the standard libraries. So no need to hack the OS or add hardware to get the gravity vector, user acceleration, attitude (via Euler, quaternion and rotation angles) etc. It's almost all there and ready to use with little or no extra processing.

[@axelriet]

Euler is 237 years old (in 2020) and he's still pretty much up-to-date yes ;-)

[@Hasan-s3q6f]

How can we remove gravity effect from raw accelerometer data? I mean how we get linear acceleration? I have seen free acceleration also in literature. Linear acceleration and free acceleration is the same?

[@syednaim2893]

Much impressive, Very informational, Wow

[@Tadesan]

So comment!

[@tim_meister]

you can really tell this comment is three years old

[@artiny16]

26:44 I doesnt get it ...why is use Kalman filter, and what was about pedometer. Someone can me explain it ,please?

[@Ateeb0Malik]

it's for the positioning of the character..for example if you want to move your character by moving your device or want to map your footsteps as input to your character's step... he was actually giving alternate to the Accelerometer (to avoid double integer errors) and proposed pedometer and Kalman filter to use for the above mention task.

[@xpeace77]

Are the presentation slides available for download ?

[@ArmendKrasniqi]

I have Samsung Galaxy S2 and I'm experiencing with an issue that's annoying me very much. When i do "auto rotate" , the rotation goes after 4-5 seconds. I've done the calibrations and all that stuffs that you will find on google or forums. Also, i've checked secret code *#0*# , and at Accelerometer Sensor i get: ACC Raw Data - X: changing numbers, y: changing numbers, z: 1024 ( z is not changing like X and Y) In this case, is my Accelerometer Sensor broke Physicaly or what?! ?!

[@FasihRana]

It is there in Android SDK. Not sure about other platforms.

[@avibank5115]

This guy is so cool

[@avibank5115]

Lol, found this video again, and was about to comment the same thing

[@bigboateng2011]

This video helped in my project. Omg thanks!!

[@shekhar2345678]

thank you..it was great explanation with demo

[@ProgramMax]

This talk is amazing. You were able to take a difficult problem and explain its solution in a simple way. Moreover, you didn't do it by dumbing it down. I would like to learn more. For example, I understand how the gyro can be used to assist in accelerometer readings. I get that we no longer need a low pass filter. But I don't understand HOW to do that. Is there a link somewhere explaining that process? Also I don't yet understand why the compass is so important. To help us find down (gravity)?

[@riddlebeats9251]

lmao the headphone jack removal years ahead of its time

[@kokopelli314]

Very well done. Simple and concise!

[@HAWXLEADER]

yet still the nexus 6p has tons of problems with its sensors. some gyro axis are not on zero when static, gravity value of the linear acceleration virtual sensor is off and there is no way to calibrate any of them...

[@gggavin]

How would I go about coding an application that outputs "roll" every time the teapot is rotated at a full 360 degrees, and not recording all other motions

[@Subhareact_01]

Hello, my mobile's gyroscope is working delay . How to solve this?

[@kayumiy]

what is the data format of 9-axis combination data?

[@roidroid]

it's an Accelerometer. "Gee" Force (or G-Force) is a measure of acceleration, 1 Gee is 9.8m/s/s (Earth's gravity).

[@chrisanderson1513]

"we ripped out the headphone jack" This was all your fault. Apple watched this video and this is how it all started.

[@stivio00]

Really Really Helpfull. Thanks David

[@axelriet]

Interesting how Anglophones call a gyrometer a "giroscope". A scope is normally something you look into and/or aim with. Microscope, telescope, periscope, endoscope, borescope... Meter means "measure" as in thermometer, dynamometer, chronometer, accelerometer, magnetometer and yes, gyrometer that measures angular velocity aka the rate of angular change, from which one can compute the rotational speed. A small mystery of science and technology :-) Maybe there was a gyro-stabilized scope someday, probably invented by Leonaro da Vinci? Modern technologies start to correct that discrepancy: docs.microsoft.com/en-us/uwp/api/windows.devices.sensors.gyrometer?view=winrt-18362

[@Sentilon]

is anywhere the current presentation to download?

[@NourSelim0]

I'm here because of something a little bit older, I wanted to get position from the sensors on some Vuzix AR glasses I have :)

[@ProgramMax]

@ballystix Thank you for the response. Unfortunately, I don't quite understand what you are telling me. Is there a friendly website to help me learn what you mean? I'll hit up wikipedia and try to piece it together. But if you have a nice tutorial or the likes I would really appreciate it.

[@iradeourum]

what is the "inegral" ??

[@DrTune]

Exceedingly interesting talk, thanks!

[@iyoutome]

Check out Ken Wheeler "Missing Secrets of Magnetism" 3rd edition.

[@Klem98]

Hi , where can i find an api to use this sensor fusion algorithm ?

[@axelriet]

Lookup TDK/InvenSense MotionApps™ Platform.

[@ballystix]

@ProgramMax I believe you would need to write out the filter as a bunch of difference equations (which is basiclly the discrete form of a differential equation describing the filter response).... alternatively you can try using fourier/s-domain analysis but i doubt that can done easily on a device like that.

[@seanpe]

Yeah, the number is wrong. I'm sure he meant 10s instead of 1s. A 1 deg error is 8.5m after 10s. His point stands though, gravity is big and double integration sucks.

[@daniyalIbrahim15]

Who is this guy? Impressive

[@rajatgupta8854]

VR in 2010, well that was something!

[@Techn0man1ac]

Молодец, знает о чём говорит.

[@ramonalisa]

euler is pronounced oiler because it is a german word. eu makes the sound oi. kind of like the strange combinations in english such as oo, ew, aw, au, ou, ow, etc....you get the picture.

[@solomon2571]

What is g-sensor then

[@TomFoolery9001]

Ha ha yep, I was just thinking that the Oculus Rift guys need to hire this guy and throw some sensor fusion in their new headset.

[@wisehomo]

The most important of all is the sensor fusion that was never talked about. lol You showed how it works from application perspective but why it works never explained by going over how sensor fusion algorithms work.

[@cavins16]

ooh all of the big words and stuff make stuff look cool

[@ignoram9us]

This is brilliant!

[@justinroughley5958]

Really useful - thanks

[@pylaochos]

Euler was Swiss, this is why his name is pronouced "Oiler".

[@F60Stunt]

very useful thanks

[@PietervandenHombergh]

Nice presentation, useful

[@Troysteele]

very informative

9-axis - it integrates all 3 types of sensors. This can't be 9DOF because you can't change your orientation for example in magnetic field without changing your position or orientation in world space.

[@AskAFreemason]

... or just use a combination to fix that problem.

[@charlesaydin2966]

Euler is pronounced "Oiler", because it's a German name. :) Regardless, GREAT talk!

[@8VT0]

Very nice.

[@moiquiregardevideo]

Interesting to compare the real world design vs the point of view of abstract University teacher who wrote the wikipedia pages about what should be the same subject. How the GPS work so well in any cell phone? The real data from the satellite is not precise enough to detect that you arrive at a street in 50 feet. But try to discover how the cell phone work from any web site or youtube video.. Nobody knows. Wikipedia just mention the expensive equipment that are mandatory in commercial airplanes. They know the cell phone in their pocket appear to match the performance of these devices. But all they can say is : thanks to Einstein, science works. The black hole in their knowledge is discretely ducted under the rug. This video published in August 2010 explain the difficulty of sensing linear displacement from the 6 sensors. A good engineer can fill the gaps, extrapolate what the device do starting from the noisy input sensors to the smooth animation that Google map and other similar application draw of the cell phone.

[@hallodaniel1312]

36:55

[@drpolishmatt]

Thank you!

[@CurtHowland]

Thank you.

[@NobleHiro]

Anyone want to give definitions for all the terminologies used here?

[@dnumdug]

Glad I got an simple explanation and understanding on those things... (!!). Is this a IQ test or what?..

[@canvin]

Thanks really helpull

[@DanHaiduc]

Google hosted this amazing talk on sensor fusion, then proceeded to cripple what could be the most useful indoor localization tool: Wi-Fi scanning. issuetracker.google.com/issues/112688545 issuetracker.google.com/issues/79906367

[@AskAFreemason]

In reality is possible to recover from a 90 degree angle on an airplane. It's possible. But if the software is using Euler angles you are giving zero chance for that to happen... so, please, do not use Euler angles on aviation please.

[@BurakCalik]

Clever!

[@brucesha7901]

youtube(google) should integrate IQ and Knowledge into thumbs-up/down. because that way when you see the ratio you'll have a better chance of correctly evaluate yourself!

[@AleksandarKospenda]

What we ripped out was the headphone jack. Apple 7 copy pasta.

[@popcat_7]

What the hell is a pedometer?:D 31:49

[@DrR1pper]

lol, you bet!

[@cactustactics]

Yeah what a noob am I right

[@lennyhome]

The demos are unimpressive and on the overall it seems to be quite useless. Good job.

[@IAmTheRealDrake]

2nd

[@TheLimonerthi]

22 people love Jesus

[@keremkambur2421]

They are 36 now.

[@EiziEizz]

Euler is pronounced that way, because he was german you geography ignoramous. :)

[@dennislange317]

Germany and Switzerland isn't the same you geography ignoramous.

[@EiziEizz]

Even though Basel is right at the border, Fair enough. I should have said the country of his origin spoke german, thus the pronounciation.

[@Fogmeister]

Eizi Eizz *pronunciation :P

[@ElTurbinado]

I'm a little upset he didn't mention the standard magnetorometer or the speedoscope. This guy is such a noob.

[@zichenliu3212]

Even 8 years later, it seems to be the most comprehensive talk about sensors and sensor fusion I have ever watched. Thank you!

[@TheiLame]

This is 5 years old video, is it still up to date in terms of what they use? :) anyone knows?