Creating an IMU-based 3D Mouse with an ESP32-S3

[@MAYERMAKES]

The FULL CODE Breakdown video is in the first link below the video! Also pictures and a blog post with more details! And as always all the files can be downloaded there!

[@danielsalloum3006]

Given a slightly elevated stand to help maintain the mouse angle and revert back to no movement between tilts, I can almost picture potential here to be less frustrating than a regular mouse. You know, because you're not running out of tracking space or knocking the coffee over.

[@MAYERMAKES]

😮

[@aviandragon1390]

In the meantime... Trackball

[@johncrombie2771]

About 25 years ago, I had a VFX-1 headgear virtual reality headset. This came with a cyberpuck 3D mouse that operated in the same way as the final mouse in this video. I believe the cyberpuck and the headgear used the earth's magnetic field to detect the changes of the devices orientation. It worked very well, and I used the cyberpuck on its own as a mouse on several occasions. Unfortunately, the VFX-1s interface card plugged into an ISA slot, so when I had to upgrade my PC and couldn't get one with an ISA slot I had to give up using the VFX-1 and the cyberpuck.

[@green64]

if you designed the bottom plate like a half ball shape, you can use it on an surface, an tip it in all directions, or just design the bottom like a spinning top. So you can use it in mid air, and on a table too.

[@MAYERMAKES]

if you look closely o nthe underside ofth the final red mouse you will see a little conical shape, that was supposed to hiold a spring and a weighted baseplate so I actually tried that idea the trouble was with return to center, it was just to jiggly to reliably move and the deadspace required was to big so I removed it again.

[@green64]

@@MAYERMAKES oh you right, if the mouse don't jump back in an neutral resting positions you can't release it with you hand. But how do I release the Air mouse? Do I need a stand to put it on a table, if I need to use the keyboard for example?

[@MAYERMAKES]

so far I just laid it on a flat surface.@@green64 when turning it own it calibrates itself to whatever position it currently is in. so i rest it on my "neutral surface" and pick it up whenever I use it.

[@bald_engineer]

13:38 "We should find an actual use case where this thing is actually useful." Uh oh, Clem is talking crazy again!

[@MAYERMAKES]

as expected

[@alenninan5524]

Hi , just to know how i could use virtual clicks (ie movement of mouse in z direction for few seconds means i had pressed /Clicked an App)intsead of buttons to press the same app on the mouse ?

[@MoisesCaster]

Try making a PS/2 to Bluetooth using esp32. Let's revive good old keyboards!

[@MAYERMAKES]

its athing....maybe

[@TheGamingDinosaurRoblox]

i like that he is making an opensource version of everything i agree that there is supposed to be an open source version of everything

[@MAYERMAKES]

😮 it does not have to be better, just working, so there is a free choice and the tech is documemented and preserved

[@Davedarko]

tried to transplant an MPU6050 once and had similar results - not sure if I threw it in the ultrasonic cleaner, but they're definitely sensitive

[@MAYERMAKES]

garstige kleine biester

[@looeee]

As a native English speaker, I have always used mouses as the plural for a computer mouse

[@MAYERMAKES]

very reassuring

[@123DragonHeart]

Great video and project, love the little face on the mouse!

[@MAYERMAKES]

thanks, its the little faces in live that count.

[@sako639]

what should I do if I want the mouse to move not only rotating it so like an acutal air mouse. should I use another sensor? can you please help me out

[@jordig3412]

thanks for the video, very interesting ;-)

[@JakubKraus0]

Doing a double integral of IMU samples wouldn't be too hard. The problem is that the IMU isn't perfect - you are measuring acceleration with a bit of error. And you're integrating the error twice as well - and that's why the device just shoots off in seconds. Similar problem needs solving for VR. There, it's universally accepted you need to correct it several times a second to get proper readings (an external system provides an absolute position, but slowly, an IMU can be sampled at hundreds of Hz, but by itself would run away. So an IMU fills in the blanks between samples of another positioning system). What an IMU can measure very well is rotation, so a trackball would be possible if the PCB was inside the ball with a battery. To get rotation, samples from a gyro get integrated - but only once - so it drifts, but at a constant speed, so it's manageable. The other sensors in an IMU can be used to correct that drift. I'd recommend madgwick's algorithms for doing so, correcting the drift is actual hard math. For a drift-free result you'll need gyro+accel+magnetometer. Magnetometers aren't great, usually needing calibration because they are influenced by metals around them, but you could provide a fake stronger north with a magnet in the casing

[@MAYERMAKES]

i like the idea of putting the imu in the ball! thats a cool concept! my math capabilities are very limited so for me it was to hard to do, but Iḿ sure some community memebers will figure it out!

[@mand6029]

thank you for posting very informative videos on electronics , we learn a lot from them

[@MAYERMAKES]

you are welcome

[@sjoer]

There are some optical sensors in stock, have not checked every single one of them but there is probably something "usable" ?

[@MAYERMAKES]

Can I haz link? 😢

[@sjoer]

@@MAYERMAKESI think optical sensors like the APDS-9500 could possibly be used? There were ADNS "LaserStream" sensors, but there might be others as well. The APDS has "cursor mode" and from what I can tell from the surface calibration some mice do... they do look at a tiny image from the optical sensor!

[@MAYERMAKES]

sadly I don´t think these sensors are usable, they cannot measure exact movement just general direction and are not meant to slide close to the ground but recognize gestures from farhter away.@@sjoer

[@azyfloof]

I'm surprised you can't get them optical mouse sensors from Mouser, of all places 😅

[@MAYERMAKES]

I was equally as surprised. could only find NOS quadrature encoders or salvaged ones.

[@wodowiesel]

thats cool with a gyro 😎 but kinda missing the middle click button or a scroll wheel for "full" functionality

[@MAYERMAKES]

The ffc header on the biard was meant for a switch joystick to provide rhat funktionality, but i ran out of time.😂

[@orbatos]

If it's already Bluetooth, why not couple time of flight with the imu?

[@MAYERMAKES]

Both to imprecise/unreliable

[@dj1NM3]

Perhaps a joystick shaped device would make it a bit more intuitive for tipping the mouse in the x/y direction, than a puck-shaped mouse?

[@MAYERMAKES]

could be, i think for these sort of devices finding the right shape could be a personal thing. not everybody might prefer the same shape, i could also see the appeal of moving the mouse only while a button is held for example for presentations.

[@dj1NM3]

@@MAYERMAKES The only commercial version which I can remember the name of is Gyration Air Mouse, which adopted a Star Trek TNG phaser shape, mostly because one mouse button is the trigger underneath and there's a control pad for your thumb on top.

[@raguaviva]

What is its battery life?

[@MAYERMAKES]

no idea but likely several hours without any sleep function, can be optimized like with any other mouse in code.

[@raguaviva]

@@MAYERMAKES what you built is called an air mouse, and their battery life is 4 months... If you are not addressing consumption, which is the elephant in the room, the rest is useless

[@sukantrai5251]

Great build !!!! I think maybe specially abled people could find good use of the mouse !!! What do you think ??

[@MAYERMAKES]

We would need someone to actually try and say if its useful to them.

[@em21701]

It looks like you have recreated the 3d space mouse that some CAD users are fond of, just a floating version. I still prefer the traditional mouse.

[@MAYERMAKES]

yes, kinda like a space mouse

[@sicko_the_ew]

For little rodents that can gnaw their way into anything, and are scared of cats, "mice" is right, yes. But for mechanical rodents (I've heard people refer to a mouse as "rodent", and was impressed by this) as far as I'm concerned the correct plural is "mouses", not mice. Mice are little rodents. Lots of them. Mice is their collective noun. Mouses are just metaphorical mouse instances. It's much more sensible to keep things simple, call them mouses, and leave whatever ancient reason there was to give the word "mouse" and irregular plural, whereas "house" plurals to just plain "houses", regularly. Hmm ... lice are the plural of louse, so it's not a perfectly irregular thing. If they make a device they decide to call a louse, go with "louses" and not "lice", in the plural, as well. Lots of words in an irrelevant matter, yes I know. What can I say? This is the Internet. This kind of thing is happening all the time.

[@sicko_the_ew]

I'd better try to be relevant. OK, so maybe what you really need is a Wild West Colt 45 pointing device with a lazer beam that changes colour when you trigger an action. So you'd fast draw it, take aim, and pull the trigger to left click. That should be nice and useful.

[@MAYERMAKES]

turns out I wasn´t so wrong after all

[@sicko_the_ew]

@@MAYERMAKES Yes. As in IMHO you were completely correct. :D

[@101picofarad]

Эх... а ведь в этом видео можно было увидеть пару часов матана на тему извлечения перемещений из многоосевых датчиков положения... Ну или хотя бы пару трюков как без высчитывания реального перемещения таки управлять курсором (дифференциальный метод).

[@tanjiro3285]

the thing us said about imu getting damages is real bro, i have been struggling with my self balancing robot. Now i am going to buy a new one ,if it makes my project work, im gonna come back with 10++ subs

[@the-THORNSPAWN]

You are not incorrect in calling them mouses, computer mouse is an acronym: Manually-Operated User-Selection Equipment the plural of which is MOUSEs. Mice refers to the animals.

[@MAYERMAKES]

vindicated after all!

[@MisterkeTube]

Somehow I get the impression that acronym was conceived after the term was already widespread, right?

[@passcod4]

No it's not. Popular myth. It was named after the rodent.

[@davidhayward1426]

Trackballs do not reqire integration like a IMU mouse would. You just need to have ball drive two slotted wheels and then count the slots as the wheels rotate.

[@MAYERMAKES]

that would require mechanical design Iḿ not getting into yet, and an optical variant would require the seemingly unobtainable sensors, but yes atrackball might be a good follow up project as it would be interesitng to see how encoder wheels work.

[@davidhayward1426]

@@MAYERMAKES for some circa 1984 documentation on how this used to be done check out page 1028…1030 of this www.weihenstephan.org/~michaste/pagetable/mac/Inside_Macintosh.pdf

[@Domspun]

​@@MAYERMAKES So a ball mouse would also be possible?

[@MAYERMAKES]

well yes, but the mechanical design is much more complicated and I thought people would not like to see such an Oldschool design for a mouse.@@Domspun

[@Domspun]

@@MAYERMAKES haha Touché!

[@aviandragon1390]

Now you must design a giant floating keyboard to go along with this. Otherwise it is just out of place.

[@MAYERMAKES]

oh swet summer child....look a few videos back I already did a BLE keyboard from scratch, granted it is not giant, but it could be used floating

[@aviandragon1390]

😄

[@chrisbutler9594]

This would make a good lightgun substitute for gaming

[@whosonedphone]

I was pretty sure you would not be able to get the original concept off the ground!

[@MAYERMAKES]

thats why we accept adapt overcome!

[@mettby9682]

Can you elaborate more why IMU mouse require complex math?

[@bartz0rt928]

I tried to do something similar in uni and basically came to the same conclusion. The IMU measures acceleration. To get from that to translation ("I have moved this far in this direction") requires a second order integral. The integral itself isn't that hard, but any error in the measurement grows exponentially. So you end up lowering the sensitivity to reduce noise at which point it can't track small movements accurately anymore, and any large movement quickly leads to the error piling up to where it becomes useless. The paper he refers to probably uses the second IMU so it can sort of average the noise out, but that only gets you so far. What you need is a way to recalibrate to a known anchor very often. That's how IMU-based position systems like the inside-out tracking in the Meta Quest VR headsets work, and also the Wii controllers.

[@MAYERMAKES]

here is a paper on that: actascientific.com/ASOR/pdf/ASOR-02-0063.pdf

[@gendragongfly]

The IMU he wanted to use is a 6 DOF IMU. It measures acceleration and rotation in 3 directions. But it has no magnetometer and doesn't output translation. This means the translation movement and position would need to be calculated from the acceleration. So, imagine you move the mouse from A to B in a straight line. It starts stationary, then accelerates. When the movement speed stabilizes the acceleration is zero again. If we want to know the position (distance traveled) we need to know how fast the mouse is moving. But in order to know how far you are moving you need to know how fast you are going. This means you need the integral of the acceleration over time (which is the change in velocity). And then you need to take integral of the velocity over time to get the distance. From the distance the change in position can be calculated. When you decelerate upon arrival at point B you need to do the same calculation but in the opposite direction (with negative values). If the calculations (and measurement data) are perfect the acceleration and deceleration, and the velocities should cancel each other out. But if you miss data or have any measurement error the acceleration and velocities won't cancel each other out and your velocity never reaches zero, so your mouse cursor would be all over the place (this is the problem Clem was running into). TLDR: this IMU doesn't output translation movement or position, so relative movement and position need to be calculated. No IMU is very accurate as a position sensor, as this is not measured directly. A 9 DOF IMU would have been better, but it would still require the calculations to be done on a computer in order for it to work as the calculations need to be performed as close to realtime as possible.. An ESP32 is simply not fast enough to even get close to real time when dealing with such complex calculations.

[@AlexandreLefaure]

It does not. It is simple integrals of integrals. IMU gives (among other things) accelereration in X, Y, and Z. If you compute the integral, it gives the speed in X, Y and Z, if you take the integral again, it gives the displacement in X, Y and Z. Respectively, the gyro of the IMU gives the angular acceleration -> angular speed -> heading in X, Y and Z. So it's not complex math, but the thing is sensors (especially this one) are not perfect and a tiny error in measurement passing twice through integration might result in very bad measurements of displacement and heading of the mouse.

[@janekkouril476]

Another problem is the existence of gravity. You can't easily distinguish an acceleration that is much smaller than gravitational acceleration from tilt.

[@antirussia_org]

I'm sorry, what again is so hard in projecting a 3d vector onto a 2d plane? (from the paper's abstract): "At first, it was required to measure how consumers’ feel about what they feel while sitting on vehicle seats." - The capitalismity of it! That paper is completely unrelated to your issue, is the link wrong?

[@crosswick]

You know that prototyping is a thing, right :P

[@MAYERMAKES]

You know that what you see is prototyping, right? :P

[@crosswick]

Well I mean making an entire custom PCB before testing out the basic functionality seems a bit overkill, but perhaps you’re just very good at it

[@crosswick]

To clarify: I don't mean to throw shade here, the video is great - I guess my point would be that seen as element14 provides informative and entertaining content by and for playful makers, it would be nice if they could offer something like $5 IMU breakout boards

[@MAYERMAKES]

I actually save time by using pcbs instead of handwiring, with a tight knit schedule airwiring modules and then stuffing it into a shell is just to erro prone, but I dod have to make bodges quite often.@@crosswick

[@alpha_pixel_]

Its already there years ago with LG motion Remote.

[@MAYERMAKES]

as far as I can tel lthis one uses angular movement not lateral movement so it works liek the thing I ended up with.

[@raguaviva]

Getting position from acceleration is taught in high school.

[@MAYERMAKES]

Doing it in practice is a different beast.Reality often disagrees with theoretic assumptions

[@raguaviva]

@@MAYERMAKES Physicist here, I've done that and it is just a couple of additions per axis ;)

[@MAYERMAKES]

Then post your code on the community page , it will surely be interesting!

[@brookerobertson2951]

You can make it with a rumble Pack inside And make the scroll wheel control the speed of the vibration. Mmmmmm that be good ❤

[@brookerobertson2951]

Make a small stash in it… just enough for a gram or 2..

[@brookerobertson2951]

That way Stacey can’t find it and snort it all and then sleep with your bf.. 😢