Trying to do 3D Localization using ESP8266's

[@brisk4t]

Here's the esp satanic ritual i was looking for.

[@Anvilshock]

1:27 You know what's also more important than not looking strange? Something that, in fact, would automatically negate any strangeness while still doing science at the same time? The fucking metric system, that's what.

[@FalcoTheImpaler]

As a born-and-raised New Yorker and American, I can say heartily, "Fuck our measurement system so bad. Please let us go to Metric. Please."

[@drkastenbrot]

Toast im wondering when people will finally make the clock decimal. 100 seconds a minute etc.

[@Anvilshock]

Ah, the old "decimal equals metric" argument.

[@goeiecool9999]

first few seconds of the video: Your camera has good bokeh

[@CNLohr]

For night shooting, I use a super cheap manual Nikon Nikkor 50mm f/1.4 I got for around $70. The f/1.4 makes it amazing, the manual focus makes focus hard.

[@petleh82]

I have used esp8266 as sudo gps tracker, scan what wifi networks that can bee seen, log them, and when finding an open network, connect and transmit them to my computer, and there i did wifi gelocation to get the position. Your esp8266 stuff is very impressive, hope you get it working !

[@derre98]

Now you can start detecting cloaked ships passing through your time of flight fence.

[@canis_majoris]

This is a really tough problem. Your "exact" timings aren't exact at all, and the moment you add RF obstructions, you need to start looking for other solutions. My friends company does this and it's a combination of accelerometers and GPS essentially. But if you're happy just having this work in line of sight, you should make a rubidium time source and then try again.

[@4.0.4]

+1 because you tried. My ballpark guess would have been that it wouldn't work, but you went and tried (which is better than guessing). I found your channel and insta-subscribe just from the titles of your other videos.

[@CNLohr]

I don't let people's opinion of what is possible or not influence my actions too much.

[@GeorgeOu]

Really cool experiment. But since you're measuring time which can be converted to distance, what you're doing is trilateration and not triangulation.

[@RicardoNapoli]

it's better to do this in small steps, start with a known setup, like a linear array of ESPs in only 1 axis spaced by 1 meter grid. Then go 2D with maybe a 4 by 4 array of 16 ESPs in 1 or 2 meter spaced grid. Third step full 3D with know grid. For every step we will know beforehand the correct answer beforehand, that makes things easier to test, and we can even train neutral networks since we have a few complete sets of input+output pairs that are know to be true.

[@gustavonunes6866]

Man, I made a project on GitHub with a "nature based" processes that moves the nodes in the simulated plane based on the distance of the nodes. The measured distance worked like a force that tries to put the node in a distance, with the measurements of the distance of the nodes i could basically make a real time calculation of the position of a node in the space gnunesmoura/simulator2 I think kk

[@FyJonas]

Esp32 s2 is going to be interesting with time of flight sensing! Looking forward to videos from you about it

[@timonix2]

I love the white text. I can't remember if it had a name. But it should

[@msurguy]

Very very interested in this topic, one thing I am thinking is that we can create a validation set of data first and then build an algorithm that will pass the new data through ML model trained on the known data. Also, another option that will be available soon is ESP32 that should be faster and also support bluetooth which is perhaps a bit more precise (I don't know the answer to that). Good luck with further experiments on this!

[@WilliamWeatherholtz]

What is the ML model supposed to do?

[@CNLohr]

I'm still waiting on getting my hands on some ESP32s! But, I really hope someone smart with this analysis thing gives a crack at my data.

[@WilliamWeatherholtz]

Let me know when/where it's up, and I can try a couple things on it.

[@CNLohr]

Github, in the toprecorder folder. I have an example file processor written in C++.

[@ilyeskml1898]

love that intro, soo unusual

[@sdrshnptl]

got engaged to your inventions, keep it up man!👍

[@thedwithalive]

another interesting project triangulating with esp8266s is named SubPos. I saw it featured on github but never had the chance to try it

[@CNLohr]

I was talking about it specifically at 4:24 - I thought I namedropped it in this version but I guess I didn't :(

[@thedwithalive]

oh ok and I think you didn't. But the Project looks interesting and gets developed for a bit now

[@drmosfet]

Some poor old lady looking out her window, probably thinks you're summoning the Mothership, it's a good thing you didn't have your soldering iron with you, they might have call the swat team. But it seem like random interrupt calls would mess up you timing, i hope that's wrong because, this is cool! It's it's a shame that the hype about $5 atomic watch from years ago never made to reality.

[@chrismr3972]

Instead of using a crystal at each ESP you could generate quadrature clocks and feed each into a separate ESP meaning that you then get 4 90 degree shifted readings at each location. Uses 4 x more ESPs but gets better time resolution. You also don't know the phase relationship between the transmit and receive (which are drifting) so there's probably 2 clocks (at 160MHZ) out.

[@CNLohr]

I was actually considering running clocks to all the ESPs, like with a cat5 cable or something, but then I started working with the ESP32. So, if I do this again, it will be with ESP32s.

[@0xPIT]

"If left running continuously, an NTP client on a fast LAN in a home or office environment can maintain synchronization nominally within one millisecond. When the ambient temperature variations are less than a degree Celsius, the clock oscillator frequency is disciplined to within one part per million (PPM), even when the clock oscillator native frequency offset is 100 PPM or more." From: www.eecis.udel.edu/~mills/ntp/html/discipline.html So maybe you can get it close enough with TCXOs and NTP?

[@0xPIT]

Or, you could attach a GPS as a clock source to each ESP. ublox NEO-7 have a programmable time pulse output that go far beyond 10Mhz and are locked to GPS.

[@CNLohr]

I was expecting to be able to use the packet times to synchronize each ESP to the other - which it seems I could do! No need for other things, but the clock jitter (not drift) seems to be what killed me.

[@MitzpatrickFitzsimmons]

very cool project you got going here... hey, maybe add some IR distance sensors and you just might be on your way to making a Star Trek Transporter Pattern Enhancement Array System!

[@AljazJelen1992]

Another thing you could try. Lets assume you want to know the distance between 2 ESPs (ESP_A and ESP_B). ESP_A sends "Where are you?" to ESP_B and marks the time. Then make interrupt service routine on ESP_B which will response immediatelly with "I am here!" when ESP_A sends querry. ESP_A receives the "I am here!" from ESP_B and marks the time. The difference divided by 2, is the distance between ESP_A and ESP_B in time. Now repeat that for any relation of ESPs and voila. You will have distance between all of them and then use the "distance to coordinate" approach. Voila :) I might give it a shot!

[@CNLohr]

Oh boy, this project has been long forgotten - I sadly won't be messing with it anymore :(.

[@AljazJelen1992]

@@CNLohr no worries, great project tho! What are you messing with now? :D

[@octavio2895]

Hey Charles, have you checked the Decawave DWM1000 module? It's specifically made for RTLS with a 10cm accuracy.

[@CNLohr]

I've heard about it and read about it, but it didn't really match up to anything I really needed and certainly not at the price it was at when I originally looked!

[@AljazJelen1992]

I might be scrolling through too fast. By the way, GREAT WORK! :) An idea, which does not require synced clocks, as long as you know their freuqency and they dont jitter too much. Have you tried a "sender-receiver-observer" approach? So that one ESP sends the message, the other ESP receive it and send the time of reception to the observer. The observer then normalizes or references the result, making assumption that chosen ESP is closest. Then based on this assumption, you substract the time of reception, which yields you an a real time of flight. Of course, the assumtion you made (that chosen ESP is closest), can be invalidated, when you take the end ESP as a sender and the rest as receivers. I am not mathematician, but this could work. You dont need synchronised clocks.

[@CNLohr]

Oh boy, I regret that I won't be working on this again - but you can discuss this on my discord server. Maybe someone else will be interested.

[@matthewprestine1974]

you will have to sync the clocks at a specified interval to get this to work at all, very interesting

[@CNLohr]

I should be able to do so randomly by arbitrarily syncing in the constellation.

[@RicardoNapoli]

trilateration not triangulation.... amazing project !!!

[@blazrepas8709]

If there would be a way to get channel state information out of esp8266 you could improve the timing accuracy significantly. Some guys were able to get a few inches of accuracy using CSI. There is symbol timing information (ofdm symbol) in channel state information vector. I just could not figure out how to get that from the ESP. I would be glad to discuss it further.

[@CNLohr]

I could believe that, you could look sub-symbol to a great degree! But, I don't think that's possible. Regardless, the biggest problem I have by a LONG shot is the clock jitter on the individual devices, so I'm pretty much tired of this project now. :-/

[@s.i.3702]

CNLohr Hi, you should have a look at this arxiv.org/pdf/1403.6697.pdf Before moving forward you should make sure to get stable reading and same timing when system is stationary (receiver in a fixed known location). If reading are consistent it's only matter of transferring the mathlab algorithm into python/C.

[@CNLohr]

Everything is already at a fixed location. And I will not be moving forward until the system is validated, but I have no interest in research papers, only people willing to try implementing them :)

[@s.i.3702]

CNLohr hi, tried your data reading in mathlab using the withepaper as guideline for algorithms and the results are not good. Timing is not as accurate as for gps clocks and there is too much inconsistency, good idea but not enough accuracy to make any use of it.

[@aerobyrdable]

Love this project. Keep it up, mate!

[@devjock]

Oh man, if only the esp8266 would have digital compass. Maybe there's a super cheap banggood/gearbest drop-in module. It would be super easy to make a "sprinkle tv". Just code up some basestation code for spitting out image pixels to wi-fi, grab a few hundred of these suckers with a big rgb LED and a small battery packaged in pingpong balls, configure some master "nodes" that handle image data segmentation, top left/bottom right framing, and coordination (possibly for moving imagery), and you can have a "screen" wherever you can throw these things. Possibly still outside of budget, but getting damn close. The idea is not mine (I'm specifically gonna namedrop Joeri Noort here, he's absolutely brilliant with finding niche applications for non-existing hardware), but the technology to achieve this kind of stuff simply wasn't there cheaply a few years ago. Thanks for sharing dude, you got my mind racing with possibilities again!

[@CNLohr]

hold yer horses, the tech still doesn't quite exist.

[@devjock]

Crap, did I miss something? I thought you had triangulation (well, atleast distance measurement) figured out? I mean, it won't have great resolution with the speeds the esp is capable of, and the jitter might throw off the pixel values a bit, but in essence, it would be feature complete, if you only add a compass module to the master nodes. Granted, the heavy lifting code would be on the basestation side, doing all the node location calculations (and re-calculation if the nodes are mobile (ea; have x-y motors and wheels), interesting possibilities to do moving displays, or crazy picture building using Dijkstra's algorythm), assigning each node onto a map of pixels, and doing network segmentation to optimise throughput to each node. But yeah, I'm just rambling now.. What exactly was it that doesn't make these things possible right now?

[@CNLohr]

Nope, def don't have triangulation working! Getting close, but no dice. Might not be possible without fixed clocks :-/ - but you do seem to start to be getting a grasp on it.

[@devjock]

Well yeah, I have the code loosely dancing around in my head as we speak :) So, what's exactly the thing that is holding back the successrate? Have you tried successive approximation tactics (ea; triangulate 3 times, take the average, and optionally, based on those results, set some calibration data pér node (master or not) to increase resolution) basically, what I'm saying is all your subscribers combined have more eyes and ears than you do, and it would be a waste of meat-processing power (no matter how dumb or smart some of us are) to not let us participate in thinktanking this thing to completion.. You want the esp to be able to do this right?

[@CNLohr]

I don't really know, I'm just very bad at math and analysis. I'm really hoping someone else can take a closer look... but I think a big problem is the jitter on the local oscillators.

[@WilliamWeatherholtz]

Good work as always. I've been thinking about this. I'd probably accept both RX power and ToF and put them in a kalmann filter to get a better estimate of position than either can provide. This gives me something to mess around with. Thanks!

[@CNLohr]

A Kalman filter is significantly outside of the scope of my expertise. Give a shot at it!

[@olivercopleston]

Hi Charles, I've been wanting to work on a similar project using multiple transmitters at different frequencies - then working out the phase difference as you move around with the receiver. Does this seem feasible?

[@Tristoo]

Basic math would tell you time of flight won't work. Light travels about a foot or 30cm in 1 nanosecond. You were talking about 6ns there, that's like half the distance than you have between those ESPs. So that's already as much precision as you're gonna get in an ideal scenario where clocks could keep sync to that degree (which they absolutely won't). Add the 25ns jitter and you're done for, maybe you'd be able to tell which side of the street you're on (again with impossible perfectly synced clocks)

[@CNLohr]

The idea is you average over tens of thousands of samples. It's just can't work here because of clock drift.

[@GeorgeOu]

25ns jitter means that you'll get an error of 25 feet ranging. That's too much error for this back yard experiment, but extremely good for long range trilateration if you don't need to deal with multipath issues.

[@miguelcamus]

That´s why GPS works and this doesn´t, because the distances are hugh the error is much smaller. Light flights at 1000 feet per nano second!!!

[@khaled752]

promising experiment...but i have a question...why you did`t use signal strength to determine the distant from each beacon instead of using the timing and light speed?

[@octavio2895]

Also, pardon me if its really stupid, but what about an external trigger or clock hardwired to all esps so you can get all modules perfectly in synced.

[@CNLohr]

I've thought about this, but I just don't have the time now to set up such a system :(

[@CNLohr]

You can actually calibrate with nothing other than reception. If A->B, ->C and ->D, and B->A, ->C, ->D, then you can know relative clock times on all the nodes and calibrate your local oscillators.

[@con-f-use]

My gut feeling would be, you need a lot of samples and analyze your time noise very carefully. With that you could get a meaningful and pretty accurate average. Even then, your noise floor will change with environment. That means, it has to be compensated for constantly. The number of samples required to get a meaningful average and compensate for noise might be too large, at least for "real-time" positioning. Did you do a noise plot and characterize the timer jitter and other noise? A model for the noise would be handy, if it's a simple, known distribution. How is the time recorded, and does the esp take roughly equal time to receive and process every packet? I'd think there is a substantial spread depending on what the receiving esp is doing at the time. Over all the esp-oscillators are likely not accurate enough for all that - at least on sub 100 yards distance. Is your idea to have a moving esp send packets to multiple stations of known locations and then triangulate its position?

[@CNLohr]

I have the raw data available. Someone's opened an issue in the project with more analysis. I /think/ it should be possible even with awful oscillators because you can continuously synchronize the ESPs with the constellation's known coordinates, but I haven't really gotten to test it.

[@Brutaltronics]

i always wanted to try this! and i thought the only way to do it was by measuring the RSSI, but i'll try this as soon as my ESP modules arrive from china! cheers

[@hubmartin]

In our company they spend lot of money on rssi localization without success. Time of flight is the only solution.

[@WilliamWeatherholtz]

Should be able to put both RSSI and ToF into a Kalmann filter to get better localization than either can provide separately.

[@MrFredericPlante]

In théory, with a huge amount of ESP, all fixed to a standard "large" mass, spread accross a very large fields, all linked with their own serial GPS and Accelerometer, could map, in some ways, the curve of space/time for alot cheaper then all these rocket scientist do. ;)

[@gabikaterzabek8647]

Hey Charles. Really awesome experiment. What exactly oscillator Is In this (I think ESP-12F module?) It will be good to know the name of the crystal or ceramic resonator and also used load capacitors (for the determining the jitter). Especially If I Think right, You overtacting the 80 MHz crystal for second harmonics (160MHz) and this may cause the higher instabillity. I really Thing , the good designed oscillator (in custom design not like built-in ESP-xxxx module) will may have verry good performance also a very high thermal stability and the very small jitter. Maybe I will do it if I find the time gap.

[@CNLohr]

I really think the right answer is to try to synchronize many ESPs by a hard wire together. Dunno if that'll happen, though.

[@s.i.3702]

CNLohr arxiv.org/pdf/1403.6697.pdf

[@CNLohr]

(See other reply, too) There's math in there. I don't do math. I hope someone else comes along who does!

[@Petex90]

Interesting.. really surprised if this is going to work at any reasonable accuracy, but well worth trying - looking forward for results :)

[@GrahamToal]

it strikes me that this is a very similar problem to ADSB MLAT but on a smaller scale. I wonder if there are any techniques from that world that would apply here?

[@CamNealie]

Probably dumb question - why not just add a fifth ESP, equidistant from each of the towers, that sends out a timer signal?

[@lb5sh]

+CNLohr can you get a rudimentary "3d view" of a room by interfacing several ESP8266s?

[@mr.0x373]

try an external clock source, or use an oven baked crystal oscillator or any kind of frequency standard like rubidium clock or something

[@CNLohr]

I have frequently wanted to try using an external wired oscillator, but never got around to it.

[@gastarbieter]

good luck with the reflections.

[@con-f-use]

What reflections? The original tcp package will always be faster and stronger than the reflections. Once a package is received, reflections of it, that take longer are ignored. Time noise is the real killer here, both from the oscillator drift and the time the esp takes to decode and record a packet (e.g. it might be doing other critical stuff and take longer for one packet while the other is dealt with immediately).

[@CNLohr]

There is a serious problem within this, though. With an encoding scheme like OFDM with low-bandwidth channels, reflections will practically induce phase shift, and will drag the bits to the right (time-wise) so reflections will certainly impact the signal. I just don't think it would impact it very much. Keep in mind, with N coding schemes, I think the symbol rate is under 1 Mbit. Once I get this working, I'm def going to want want to check the 802.11 B coding scheme at 11MSymbols/sec.

[@edism]

The speed of light in foot per second? You must be trolling. Good stuff though.

[@CNLohr]

It converts so well, though! Not like meters, with the 299 thing in there. it's about 1 ft per ns.

[@BitBerlin]

+CNLohr BULLSHIT! Use METRIC, you substandard filthy peasant! Repent your barbarism and upgrade to 2016!

[@BitBerlin]

+CNLohr Nice vid as usual.

[@edism]

lol

[@Anvilshock]

Hahaha, this guy is so full of shit. "I don't like the about-ness of [0.3] metres per [nano]second (at 0.06 % off), but I'm totally fine with 26-fold the about-ness of [1] feet per nanosecond (at 1.6 % off). Because it converts so well." Yes. Amazingly well. Can totally see that now. Oh boy.

[@onjofilms]

Maybe this is how the Chinese can spot stealth planes from 62 miles away.

[@74Gee]

That doesn't seem to add up. If light travels about about a billion ft in a second and there's a billion nano seconds in a second, surely with a 25nano second clock cycle you could at best only get accuracy to about 25ft. I'm sure there's plenty of other lag factors and also keeping all the clocks in sync. Even a cesium clock (>9Ghz) could only manage a 1ft accuracy if everything else was perfect but like with GPS, the clocks need to be exactly in sync. A really cool project though, just a little ahead of our technology :)

[@CNLohr]

The idea would be you stocastically, over time would be able to refine the estimate, even though on average it is waaay beyond the noise floor. Keep in mind that GPS operates with a 10 Mchip/s rate, but it can do a lot of tricks to recover pretty good accuracy.

[@ahsanfarooqi5663]

Hi good evening. can we measure the signal distance? in arduino using any wireless module? please replay thank you

[@CNLohr]

Modern ESPs support FTM which would let you actually measure distance with a sophisticated enough base station.

[@Siver5300]

Well i know you have stated that it didn’t work But I’m curious about how you went about synchronizing the clocks It seems to me that if they aren’t synced then there is nothing much you can deduct from the difference between them

[@CNLohr]

The crystal drift was just toooo much in this case.

[@TheMrKartman]

Very good channel! Like from Russia with love! ;)

[@teslatrooper]

10m is about 33ns. So that seems like a decent number to set as minimum resolution. How can you even keep the clocks synced at that precision? Even at 10ppm you will have 100's of meters of inaccuracy almost instantly.

[@WilliamWeatherholtz]

>How can you even keep the clocks synced at that precision? Why would the clocks need to be synced at all? I'd use an ACK instead to tell ToF, then the relative time of each ESP between send and getting an ACK back is the distance * 2 to the other ESP. By getting a couple readings, you should be able to triangulate position reasonably.

[@teslatrooper]

You'd have to know the exact moment of transmit and ACK, I don't think there's low enough level access to the radio hardware to accurately get those? I don't think you can guarantee transmit operations always take the same time either, since that could change depending on the initial state of the transmitter.

[@WilliamWeatherholtz]

That's a good point.

With 25ns precision, the inaccuracy will be 7.5m. So the resolution (imaginary grid where you can determine the location in) with 4 ESPs in a rectangle 100m away from each other will be 14x14.

[@teslatrooper]

My point is that he has nothing close to 25ns precision. He's sending timestamps to the central point assuming the clocks are synchronized while a 10ppm 26MHz crystal can drift up to 260 cycles per second. The method proposed by William could work better but I don't think there's enough low level access to the radio system to even make that accurate. I'd love to be proven wrong because the applications for this are really cool.

[@TomBurkeii]

Have ideas for you. We'll talk later.

[@peterkis4798]

You can send null data frames instead of beacons, i use this for my esp quadcopter project for remote controll. Small frame, no ack/resend perfect for hacky rc :) I can transmit about 1500-2000 frames/sec with this approach. tx: struct FRAME { uint8_t frameType = 0x48; uint8_t frameFlags = 0x81; uint16_t durId = 0x0000; uint8_t address[6] = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06 }; //bssid RC_CHANNELS rcChannels; //6rc channel 12byte sturucture (address 2, address 3) uint16_t sequenceControl = 0x0000; }; ... if (wifi_send_pkt_freedom((uint8_t*)&frame0, sizeof(struct FRAME), 0)==0) frame.sequenceControl++; //can be set to anything rx: uint8_t ap_mac[6] = {0x01, 0x02, 0x03, 0x04, 0x05, 0x06}; wifi_promiscuous_set_mac(ap_mac); wifi_set_promiscuous_rx_cb(promisc_cb);

[@CNLohr]

I hadn't tried actual null frames, but, these frames are unintelligable by normal clients, so I suppose it's a similar effect.

[@larslrs7234]

Can you please make a video about ESP8266 light sleep and its power consumption?

[@CNLohr]

There's already a couple other people who have done so. Got too many other in the backlog >.

[@larslrs7234]

I found NONE. The existing ones are all about deep sleep. No project that is demonstrating automatic light sleep or forced light sleep while maintaining the WIFI connection to the AP.

[@CNLohr]

gah bummer. I may do it. Light sleep is easy depending on your appliaction and dumb if you don't do it.

[@TomBurkeii]

Yay science!

[@diogomartins5949]

Since you can connect every esp8266 with each other. Do you think you have a way to build a digimesh?

[@CNLohr]

wouldn't be hard to do at all! Especially now that Espressif has opened up more opportunities for use with the ESPNOW framework!

[@hemanthkumarHere]

might be useful for AR gaming.

[@MrFredericPlante]

Considering the speed of light, I think you should put a little more distance between you detection nodes, so this way you give more time to your network to react.

[@matta9186]

Which ESP8826 modules did you use?

[@imignap]

Wait why not bring back the RSSI idea & use a better dipole antenna instead of the PCB antenna?

[@CNLohr]

Never got anywhere with RSSI :-/

[@imignap]

So time of flight for WiFi, in digital domain never a good idea. But using signal strength has been done already as Apple bought the company WiFiSlam for $20M. Read article: rse-lab.cs.washington.edu/postscripts/gplvm-wifi-slam-ijcai-07.pdf&ved=0ahUKEwjImtSI1e3PAhWDSiYKHfAKCDwQFggeMAA&usg=AFQjCNF6cR5_eY6BvLYbF6-7JIqDJE-7TA&sig2=7ziKgMu0cMcmB_RyCx5OMw I would say exhaust all possibilities its not the antenna before dropping the RSSI idea. Are all the broadcast nodes on different channels? Spread the nodes out further also. Screw the neighbors.

[@CNLohr]

I don't actually care /that/ much about this, but I am trying some other things.

[@andresc4]

Hey I work with VVVV, i think I can help you out to graph this,,, do you still have all that list on github ?

[@CNLohr]

List? Everything's still up there!

[@colfaxschuyler3675]

Yeay, bokah! Boo... misspelled bokeh. Eh.

[@larslrs7234]

In order to utilize (the change of) RSSI for localization, you require at each stationary module a characteristic antenna radiation pattern that you approximately know and that is at least to some degree independent from the orientation of the module to be tracked. That would be a (long) Yagi antenna (PCB version here: www.ti.com/lit/an/swra350/swra350.pdf) You should get localization data by angle, not by distance.

[@СашкаБелый-ч6м]

Bull shit!