How the Sharkbyte / HDZero video system works

[@xjet]

Given that SB is not a duplex system (ie: the VTX only transmits and the VRX only receives) how does it determine the link quality so as to adjust the bitrate. As far as I can see, the bitrate is non-adaptive and there is no change to the transmitted bitrate as the link quality falls.

[@MrBumbles2]

because it's got a dual mackie piston clutch in it bruce , who needs bitrate?? lol

[@Stefan_73]

If you look closer to Sharkbyte and old ByteFrost data, one might guess some additional insights. It seems that HDZero subdivides the image into 12x8 blocks. Those blocks seem to undergo a discrete cosine transform. Especially in older data this is good to see as it seems the system had issues to create a continuous (differenciatable) image. The latter one seems fixed meanwhile. As the datarate is fixed and compression is comparatively low compared to h.264 or DJI, there is also a comparatively lower resolution/data content. A while ago I took comparable footage of SB and DJI and calculated the power spectrum of the images. By that one can roughly compare the resolution. Doing this with current data and highly identical footage may allow to asses this. Latency is the other factor. Of course SB can be faster. The other factor is the smoothness of the video stream. It would be great if SB had an option for 120Hz (VR goes 90Hz+ to remove flickering).

[@ooHotcooleRoo]

Heh, was just mentioning the same oddity with the blocks, in a different comment - kzbin.info/www/bejne/bGTcnIKrp691iJI&lc=Ugy62dzJ3wPEzJCwyP14AaABAg.9RqDjLSFWGN9RqjYfY87xt DCT is actually more likely, but the 12x8 blocks still make little sense to me TBH. Originally to me it seemed that the blocks might have started life as 8x8 then got stretched within a single line of blocks later on, since there are some really serious issues with consistency between the 8px lines. Also there were some ideas on what might be going on with the data rate too, I too would agree that it's more likely to be constant, and by eye I'd say actual resolution is in the ballpark of 360p. Data content even for 360p seems rather low still though. The line "assembly" seems quite a bit less noticeable in motion, so 120hz would definitely help perceptual fidelity, but I suspect they do not have bandwidth for that, since efficiency of compression is probably really rather low. Have not looked at older bytefrost DVR's should be interesting to take a look, but nice to meet a fellow pixel peeper ;) P.S. Looked at some bytefrost footage (could not find any raw dvr though) but while it had more trouble actually aligning the lines when there were connection issues, I kinda think it actually had slightly more effective resolution.. Basically I do not quite see even close to the same amount of issues on the block edges which makes lines look (like that wire in the linked comment) a lot nicer and I think that upps the overall detail level quite a bit, even though the end result still is very much likely 360p or even under. I kinda think that with a good camera it would actually look quite a decent bit better than sharkbyte TBH. May be they decreased the amount of data per block since then for more range/penetration/stability? P.P.S Found some raw DVR courtesy of Mark Cocquio and I'm pretty sure the DCT/Blend/Stretch thing making lines look like macro blocking mess was not there in 2019. So that seems to be new. Cameras were utter garbage, but... and overall the whole block puzzle look is still there, but it's a lot more stable looking I'd say. I would definitely prefer the old one with a decent camera IQ wise.

[@Stefan_73]

@@ooHotcooleRoo Thx for the hint. I do indeed believe that SB is going and must be going for such high datarates. The link then becomes very susceptible to interference. This could explain the comparatively low range. SB also needs such high datarates as compression is very low even with the rather high sacrifice in spatial resolution and some trade offs in color representation. That's the price to pay for spatially local compression on 12x8; i.e. for no backchannel and constant latency.

[@pulquerofpv7209]

The reconstruction of lost packets by the receiver raises some interesting questions. It would seem the vrx could in principle make use of previous frames to better predict missing blocks. A possible way to improve the perceived quality then, would be to improve lost packet reconstruction. In the beach images, it seems a bit poor that the missing sky blocks dont have a blue that more closely matches the surrounding pixels. Maybe this is deliberate so that the noise is obvious to the pilot. One could imagine a system where the reconstruction was so good that imagine would just become less detailed or more blurry as the signal degrades. No doubt extra processing might increase latency, but this could be made configurable. At the end of the day, our brain reconstructs most of what we see. If we understood this process better, we would only need to send the parts of the image that the brain actually uses.

[@Perforator2000]

There is a phase of breakup with Shark Byte when the signal loss is moderate that looks similar to how you describe, where it gets a subtle liquidy, sparkly look, right before it starts with the rainbow colored blocks cascading down the image. It's a bit like looking through water for a moment.

[@sunsetpark_fpv]

Dude.....you are beginning to observe the arbitrary binary data packets known as "The Matrix"......Follow the white rabbit bro.....

[@dusterl1472]

I literally (and surprisingly) only found your channel because of an inquiry you posted wanting to talk to someone about OpenHD. I subscribed with the sincere hope that OpenHD will be discussed as well! But I'll stay subscribed either way because skimming your videos shows that they're awesome!

[@ChrisRosser]

Thanks! I want to do something on OpenHD as well but I am not sure what the best approach is for FPV. Is there a recommended openhd setup for FPV?

[@chrisalexthomas]

These videos are awesome, keep them coming, they're amazing! thanks a lot mate! I got my hdzeros this week and I can't wait until tomorrow to fly them!

[@juiceman1807]

Thank you! your presentations are very good. I appreciate the deep dive into how tech works. Keep up the good work!

[@ChrisRosser]

Glad you like them! I'm going to keep making them!

[@roellascano]

Thank you for a great series! It is very helpful to me to have a "side-by-side" comparison of the technologies to give me a better idea of the benefits of each technology.

[@ChrisRosser]

Side by side comparison coming up after latency testing!

[@RyanQuellet]

Thank you for this detailed video. It will help clear up confusion between DJI and HDZero. The two technologies are very different from one another

[@Perforator2000]

Yeah, I hope it does clear it up. The system is significantly better than analog in every way, but it still has some of the same limitations. It still requires the user to understand the limits of RF and make an effort, like pointing the patches the right way, staying off WiFi frequencies, and maybe not flying from inside a car.

[@RyanQuellet]

@@Perforator2000 exactly. Basic best practices for analog apply to HDZero. I feel dumb that I didn't use R8 on the quad I sent to Bardwell to avoid WiFi interference

[@Perforator2000]

@@RyanQuellet That's an easy thing to overlook. I imagine Bardwell will give it another shot on Race 8 before he does the actual review. I'm not expecting to see magical performance out of it, but I was a little surprised Bardwell didn't immediately recognize what was happening with the breakup he was getting.

[@joevo2]

Any idea how the penetration of DJI is so much better than analog? If hdzero were to have the same 1.2w like dji it’ll have similar penetration?

[@Perforator2000]

@@joevo2 It's a combination of output power and the two-way link between the goggles and VTX. DJI can slow down the link and re-transmit lost packets at the request of the video receiver. At a certain point, it hits the physical limits of RF and drops out completely.

[@tmdsparrow85]

Thank you very much for this detailed video. I am not an engineer but it realy gave me a lot of knowledge and understanding of how does the system work! The flow was in a taste of symposium and thats realy great :)

[@ChrisRosser]

You're very welcome!

[@jimmycarr3400]

Great video my man. Glad Joshua B pointed me in this direction.

[@ChrisRosser]

Welcome aboard! Happy to hear you found what you were looking for.

[@WavyHertZFPV]

Your explanation is always perfect ! Thank you for your work again ;D

[@ChrisRosser]

My pleasure!

[@vikramgupta7050]

HI Chris! Thanks so much for your phenomenal work. Ik this is off topic for this video, but the one qualm I had with the AOS 5 is... there were no specs available online to make a propeller guard/cage! It would mean a lot if you could advise me. Thanks so much!

[@ChrisRosser]

Your best bet is probably to measure up the frame once it arrives to best decide how you want to proceed.

[@DanielSingerX]

I was looking forward to this and wasn’t disappointed 😎 amazing content! I had a small idea of how this works but this really cleared it up! I fly sharkbyte and find it’s method very interesting

[@ChrisRosser]

Yes it combines the line by line nature of analogue with digital compression and encoding. A very interesting approach!

[@DanielSingerX]

@@ChrisRosser yes exactly. I feel like it’s a bit smoother looking with motion than DJI

[@LS-xb2fh]

Great series! I'm wondering about other techniques to handle longer range transmission: 1. A simple uplink that only transmits the SNR to scale the data rate but without retransmission of single packets. 2. Hierarchical modulation / coding. I.e. having a low quality base video stream that is easy to receive and than an enhancement layer that increases video quality when the reception is good. Maybe DJI's approach is superior to 1. and divimaths approach is better than 2. but I would certainly be interested why those methods are not used now.

[@Perforator2000]

It's interesting how the system tries to reconstruct missing parts of the image. That and the 4 antennas seem to really help with penetration performance. I get better video through my house and through the woods with 200 mW Shark Byte than I do with high power analog and Rapidfire. For some reason my 500 mW SB VTXs don't seem as strong as the new 200 mW boards, like the TX5R.1. One thing I've been curious about is how it uses spatial diversity to turn multipathing signals into useful data. In areas where I get multipath interference on analog, SB just has a completely stable picture.

[@ChrisRosser]

I think digital in general has much better multipath handling than analog. Both SB and DJI.

[@anthonyfpvminiquad1728]

Yeah the 500mw had defects. It's being replaced with a 1 watt with metal casing and some other features

[@_samlclarke]

Do the Hd zero goggles just receive the signal from the drone? Or does it work like DJI’s system where the goggles transmit a signal back to the drone to receive lost packets of information?

[@purusha4]

Are you going to make a video explaining OpenHD ?

[@Strelinc]

So great review! Thanks, so now i know how my system works)

[@h2o-fpv623]

What a great video like always learnig something new with the fpv community and Shark byte thank u for he work.

[@edrone_]

Been waiting for this one and definitely did not disappoint! Awesome video and great information explained perfectly I learned alot for sure very interesting 👍

[@ChrisRosser]

Thanks, I think this series is turning out to be one of my best yet!

[@GinTonicFPV]

Hey Chris, a thought came to me recently and I wanted to get your take on it. Hypothetically, would it be possible for HDZero to do a simple implementation of DJI retransmission by always retransmitting the same frame twice, and then on the goggle side wait until both frames are received and construct the best image using the data from both received frames? It will be higher but fixed latency, and given what Carl is recently doing with his new goggles (6ms glass-to-glass latency) the latency should not be overly high. Am I making sense or is there something I'm overlooking?

[@ChrisRosser]

That would be possible but only if you also halved the frame rate. Error protection is already built into the HDZero protocol at a lower level than individual frames.

[@ironlionfpv]

I see you used screenshots from one of my beach cruise videos 😁👍🏻 Really nice informative video, i definitely learned something today 🙂

[@ChrisRosser]

Thanks! Those images were reproduced from HDZero website so it's cool to know where they came from!

[@friendlynomad9840]

great video! looking forward to the next one.

[@ChrisRosser]

Thanks! Hopefully we'll have some juicy latency testing when my SB gear arrives!

[@aphinion]

This is super interesting stuff! I wish there was a 1.2W Sharkbyte VTX. The frame drop/variable latency on DJI puts me quite off a bit sometimes. Even if it got a lot better with the iFlight crystal antennas (despite the goggles always showing 28ms with both antenna types). Do you think you can somehow motivate your DJI videolink to produce these frame drop/frame recovery effects in front of that high-speed camera? Would be really cool to analyze those on an objective base.

[@flamepygmy]

From how the blocks look when there is noise to me it looks like it's using something very similar to what's used on GPUs for texture compression. Typically say a 4x4 pixel block is compressed into 128 bits. This allows for a random access of the data (well the 4x4 block) while still providing compression. There are many different texture compression formats (S3TC, ETC, ASTC, etc.) that encode the block differently, but many of them use something similar to the BTC (block truncation compression) to encode the block: en.wikipedia.org/wiki/Block_Truncation_Coding. For any given block only 2 colors can be exactly represented and 4x4 grid is expressed as differences from those base colors.

[@ChrisRosser]

If you are assessing this based on DVR footage remember that it goes through another layer of digital video encoding and compression. Standard digital video encoding will create blocks of 4x4 pixels.

[@flamepygmy]

@@ChrisRosser Yes it indeed does and it's a bit harder to look so closely as you are flying, but to me it definitely looks like a block based compression when I do get the noise on the goggles. Not saying it's 4x4 exactly, but I do believe it is block encoded.

[@grounddodger3212]

Interesting to hear the theoretical limit is 100mbit. One would assume then the CPU throughput is the potential bottleneck ?

[@ChrisRosser]

100Mbit is what Carl says the max link rate is at very close range and it falls off with distance just like DJI. The limit is the bandwidth and modulation scheme used I think rather than processing. Perhaps they could widen the channel in the future like DJI did and get more image quality?

[@dakzer55]

I am enthusiastic about hdzero shark byte. But Shark Byte receiver modules are sold out in the UK 😦

[@kgimel8532]

In theory, should it not be possible to use our analogue fpv cameras since they also turn the image into an electrical signal via the cmos sensor. If so, do you think HDzero will ever make this an option? Or will they prefer to keep us buying their hdzero fpv cameras ?

[@ChrisRosser]

The sensor is the same but analog cameras don't have the MIPI link electronics so they won't be able to interface with HDZero

[@gmivisualsjason3729]

Excellent video.... very informative

[@ChrisRosser]

Glad it was helpful!

[@RindosRides]

It seems like the image breakup would be better if they simply held the last good pixel on the display instead of whatever their current repair method is. Would be a lot less distracting than the bright white pixels.

[@Chris-xe5ts]

How can the transmission be variable based on a link quality with an unidirectional connection? there is no way to determine the link quality.

[@BartleyKleypas]

If the camera and vtx are using the same interface type between DJI and HDZero/SB, would that allow us to continue to mix and match camera and transmitter?

[@BartleyKleypas]

@@voxtelnismo frame/protocol between the camera and transmitter? The transmitter (over the air) protocol difference tracks, but the link between the camera and transmitter can be “locked up” in one ecosystem too?

[@BartleyKleypas]

@@voxtelnismo yay “progress”…

[@ChrisRosser]

Yes MIPI is a physical interface. The data that is transmitted over it is down to the software at either end. Without a standard for data protocols and formats nothing is interoperable.

[@JonE5FPV]

Very cool explanation. Just released a review of the 20x20 sharkbyte vtx, and how it performs at the most famous tracks in the world, including the Houston night spot. Curious how many racers consider the reception flyable for racing.

[@2bfpv882]

No joke after the dji range im like this sucks evertime I fly sharkbite

[@ChrisRosser]

I'm interested to try out my system when it arrives and do some latency testing.

[@grounddodger3212]

Why, what's wrong with the reception?

[@ChrisRosser]

@@grounddodger3212 Sharkbyte has lower output power than DJI or analogue which puts it at a disadvantage for range at the moment. This will change with higher power SB VTx's in the future.

[@2bfpv882]

Set output the same and see what ya thank my opinion there's no comparison between them dji is just step ahead everone gets butt hurt win ya talk about sharkbite id say there's a v2 about 2 drop hince price drop or maybe not

[@WSNFPV]

Wanna mode to send data twice, loosing sharpness and image quality in general. To reduse this annoying flickering

[@ooHotcooleRoo]

It's unlikely that they are getting anywhere near 100mbps on their 27mhz wide link. Current Wi-Fi (6 or AX) can only manage 51mbps PHY link speed at it's best 16-QAM rate on 20mhz channel. And actual transfers rarely reach 70% of that. It's likely they might try to use 64/256-QAM close range which would be quite close to 100mbps. After all - all the amazing wi-fi speeds you hear about achieved only on 1024-QAM + 160mhz channels and multiple RX TX setup. So i seriously doubt they get more than 50mbps from their 27mhz wide link in actual use (not on the bench), plus since it's very likely their encoding approach has great efficiency penalties (I doubt blocks can reference other blocks for example, at best one to the left - or more likely none at all, which probably also causes their stitching artifacts that kinda drive me nuts a bit) I also doubt they get anywhere near 720p to me it looks like always macroblocking 480p with occasional dithering/checkerboarding too.

[@ChrisRosser]

It's hard for me to verify that. Carl did say that their maximum link data rate is approx. 100Mbps and falls off with distance and as signal to noise reduces. The modulation that SB uses is quite unlike WiFi. They have 1024 channels rather than the 64 of traditional WiFi. All this info has been fact checked with Carl so I think it should be accurate.

[@ooHotcooleRoo]

@@ChrisRosser Possible, but then it probably does not really sustain that rate beyond like 50 meters I imagine. I'm no RF inclined person, but I imaigne if their RF was vastly superior to Wi-Fi based stuff, they would've found a more profitable use for it rather than FPV ;) BTW AX has increased to 256 subcarriers for each 20mhz, afaik mainly to more flexibly divide the bandwidth between participants. As a guess for why they might want 1024 subcarriers, is they use them to send each chunk in parallel, so the data sent is tiny and data lost is also tiny. Chunks seem rather odd, basically they appear on DVR as 12x8 px, but they look a lot more like 8x8 that got blended on either side (left and right), so my guess image res is something like 856x720 that gets stretched to 1280x720 after. Another possibility that nearby chunks do have some relation within an 8px line, no idea but stability of picture within a line seems to be a lot better than between the lines, since vertical aliasing (guessing that thin object getting to the border of 8px square and getting stretched within that line, yet not stretched (since it's further from the border of a chunk) a line lower) is a lot worse than horizontal. Alternatively it's some sort of weird 360p with scaling.. Dunno, but it's very far from 720p for sure, with any camera. May be chunks get to be either 8x8 or 4x4 with scaling when bandwidth is lower.. Or may be is the chunk has too much detail to fit a predetermined size it gets scaled down to 4x4 or something... dunno, but the thing mangles trees to hell and back for sure. So there's something really interesting going on with the scaling for sure. For a decent illustration of what I'm talking about here's a shot from RAW DVR's posted by Ryan Quellet - i.imgur.com/prBG0dk.png In it if you notice where the shirt meets the pants - the shirt looks quite decent, since it basically only passes two lines (and you can clearly spot where the lines change), but if you look for the nearby vertical wire to the goggles - it got messed up bad... All in all to be quite honest Sharkbyte IQ looks 360p to me with something resembling macro blocking artifacts from like 3GP video era, where if the block was too heavy with movement it basically got reduced to just being a single pixel instead of say 64.

[@RyanQuellet]

@@ooHotcooleRoo that's a lot of beating around the bush talk to me. The image quality of static frames does not hold up the same as the video in motion. I suggest you try flying it in person and see if you think it looks like 360p then

[@ooHotcooleRoo]

@@RyanQuellet I agree that it looks better in motion, I'll try to find someone with it to fly, I was kinda interested in trying it any way. Was tempted to get it with the price drop, but as I only have DJI goggles I decided it'll be a bit of an issue to get the video out of it in the goggles (could use an external display, but too much hassle). But looking at DVRs I honestly think it still looks 360p (I do not mean KZbin 360p here, just general resolution). I'm not really bashing the system, or praising DJI here, just interested in its inner workings and some observations. I do think it's mostly better than analog, and thats a good thing, even though I really dislike the macroblocky look.

[@ChrisRosser]

DVR undergoes another level of encoding and compression on top of everything else, it is not necessarily representative of the image in the goggles. It may be a different resolution and will have gone through a standard digital video compression algorithm.

[@KotNaKacu]

FIRST.. oh no... third

[@ChrisRosser]

Third! Better luck next time 💪

[@LikWeiLee]

first again!

[@ChrisRosser]

Yes you are! 🏆

[@LikWeiLee]

@@ChrisRosser This video answer my question on how HDZero achieve constant latency I asked last week. You explain it very well. Thank you.

[@fezfpv]

No way second

[@ChrisRosser]

So close, next time!

[@AHTOH4uk1979]

very poor quality of SB cameras. Make a better ISP, and image compare with DJI FPV

[@ChrisRosser]

The new Runcam HDZero looks pretty good! Not DJI quality yet but if you're a racer its a massive step up: kzbin.info/www/bejne/l4mWlWdsbdaCq6c

[@steve_beatty]

Thank You 🦾 That was very informative .Also refreshing to see someone who doesn't have enough affiliate links that it takes 1000 degrees of scroll wheel rotation to get to the comments...lol..🥴