Burrows-Wheeler Transform

[@TommyCarstensen]

Great video! 0:24 BWT 2:50 BWT implementation 9:34 Suffix arrays 12:48 Suffix array implementation 15:12 T-ranking 18:13 LF mapping 21:53 B-ranking 26:44 Reversing 32:24 Reversing implementation

[@josephc6308]

Watched a dozen video/articles, finally understand why this algo works now. This video is the best. It explains the reversing part so well!

[@mlyfall]

You got me when you made that 'blubb' at the beginning in order to show how the a is moved to the end hahaha

[@ka9dgx]

Having taken an interest in BWT yesterday, and working out my own implementation in Pascal, though not understanding all of the theory...this really clarifies a lot of it for me... thanks so much for having this out there for me to watch during this time of Stuck at Homeness.

[@alexanderkurz3621]

I believe that this the first video I have seen on youtube with no dislikes ... very well deserved.

[@ylamummo93]

This video was pure gold!

[@xinking2644]

thank for sharing ben, you are a good teacher!

[@wenruihuang3394]

Thanks Ben! Very clear and I now understood everything on the BWT

[@keith_cancel]

Just found this video. Very informative and well done.

[@cyruspang5951]

26:15 why the answer is not row 800 instead of 801? If we are looking for G100, isn't it that we should skip the first 99 rows starting with G instead of 100 rows?

[@chopstyx2009]

Thank you Ben, this is invaluable!

[@akashafif8533]

It's really helpful for me. Thank you so much.

[@BuysDB]

Thanks! It was really clear and helpful to me.

[@alexanderkuptsov6117]

Brilliant instruction, sir, thanks a lot!!

[@dragolov]

Brilliant explanation! Thank you very much!

[@danywehbe1716]

Excellent presentation, thank you very much

[@spectroxis6418]

The second video of his my engineering prof sourced us 😂

[@_mvr_]

Great video. I see how BWT relates to compression, but also how it's a form of encoding. Does anyone know of a comprehensive list of reversible functions that can applied to strings or sequences?

[@alexanderkurz3621]

Great videos, thank you so much ... one question ... "We have already talked about suffix indexes" refers to which video?

[@DrJueFish]

Thanks, Ben. Nice explanation.

[@erascarecrow2541]

Good explanation of easy decoding! (Vs sort, prepend, N times)

[@derrick20]

This is such an epic algorithm

[@guanchenzhu6652]

Correct me if wrong. But since it's 0 based, shouldn't G100 be the 101th row of G in F column so in total it's the 802th row in F?

[@commentsnipper]

How to do the B ranking? I could not understand...

[@545183202Chris]

sort according to alphabet. first $ (assuming smaller than any char, or smaller than A); then all the As, then All the Bs, etc.

[@jonathanfisher2207]

I had trouble with this initially as well. The answer is that B-ranking is performed according to the order of each letter's occurrence in the last (L) column. Then, since we know that F and L have the same rank order for a given letter, it follows that the B-rankings in F are also ordered from top to bottom.

[@545183202Chris]

BWT(T) is the same length of the orignial string T. How does BWT makes it more compressible? What infomation did you actually compress?

[@bahaaalaagmail]

BWT brings like characters together in runs for example: if we pass the string "A man has got to do what a man has got to do" to BWT algorithm we get "TOOSSNNAATTOO.MMHHH......W..AATTDDGGAAAOO..." as the output. Yes it is the same length as the original string but you will notice that BWT brought some like characters next to each other like some the character A or O or H. Yes not all the character A are next to each other but it's better than nothing. Now exaggerating this result of BWT, say there was a 100 A next to each other and a 50 G next to each other, now if you pass this - somewhat - ordered output of BWT to a compression algorithm, it will notice the 100 A next to each other, so it will - for example - output like this: A( 100), and so for any other characters that is repeated. ls it clear now?

[@545183202Chris]

@@bahaaalaagmail Yep. After thought about it for a while. I came up the exact explanation as you do. You just confirmed my guess. Thanks a lot.

[@brianwahome5789]

This has problems handling empty spaces, otherwise, it is pretty much perfect.

[@BritchesBumble57]

Can you describe how the Burrows-Wheeler-Scott transform (aka Bijective BWT) works?

[@leejh6783]

Thank you for this awesome lecture. I assume this BWT is used for exact matching. Is there a video for inexact matching as well?

[@cls5502]

Excellent presentation. I have one question. The $ is put at the end of the unique sequence. Can it be put at the beginning...I understand that the implementation would have to be changed to account for the positioning of the $. I suppose I am asking is there a particular advantage at putting the $ at the end, rather than another place. For instance, to sort 5' to 3' (assuming the left end is the 5' end) rather than going backwards.

[@erascarecrow2541]

Since the $ is just a marker, you can probably put it anywhere (so long as you consistently handle it) You can even remove it entirely, unfortunately when you do that you need to store an offset of which character then becomes your 'last character' or the offset to make the original sequence without rotations. (the offset data could take up more space than 8 bits though)

[@aryanfirouzian2529]

thanks, well-explained.

[@1637shubham]

Hi Ben at 20:30 You explain that Last column is sorted by right-context that is why we see same order as in F. But aren't middle columns sorted by right-context? Why for middle columns don't we see the same ranking order as we see for First and Last?

[@YawLingLin]

I figured that you've already figure out the answer by yourself now but I would like to add in a few lines for the newly comer later on :-) Ben gave excellent (i.e., short) explanation of the idea of the "right"-context sorted directly linked to these F-L chars list occurs in the same order in the original text. The reason is that, (1) by definition, First column reflects the sorted order of BWM, so, by eliminating the first char (these being the same char), the remaining string is remain sorted (that is, the right-context). (2) Eliminating Last column chars you get the original string (i.e. the right circular shift); then again, they are listed int the sorted order by definition of BW matrix. Thus we conclude that the F column and the L column reflects the same sorted order by definition of BW matrix! Now, for your question, these properties DO NOT hold in general. For example in the second column, the orders of a-char are in stead, the orders in first column are as shown at 16:55.

[@AakarshNair]

explained really well

[@Bob_Johnson349]

Goat

[@donconfucian]

Why to explaine the T-ranking?

[@erascarecrow2541]

to show they are in order from the left side and ride side, showing you can decode it using an easier method... or that's what i get from it.

[@anureet_kaur]

What is the need to append $ sign at the end of the string while implementing B-WT?

[@erascarecrow2541]

If you don't have an ending character with $, it isn't needed. However you will still need to store which offset which correctly decodes to the original string (without rotations). In Bzip2 it might be a 4 byte int, in a sting where $ is never used, it's 8 bits.

[@patelprachi6990]

why do take the order of a's as a3,a1,a2,a0?

[@avis9ditiu]

your code in 13:38 can simply in this way: def bwtViaSa(t): bw = [] for si in suffixArray(t): bw.append(t[si - 1]) . # list[-1] means the last element in list return ' '.join(bw) thanks :)

[@TheKyaaa]

Thank you!

[@ramsescoraspe]

code? please!!

[@mattpsu2001]

Thank you

[@akhilp7390]

i dint understood how to sort the sequences?

[@erascarecrow2541]

rotate the input and then sort it so you'd get an array of strings like: abaaba$ baaba$a aaba$ab aba$aba ba$abaa a$abaab $abaaba Afterwards sort it.

[@ramsescoraspe]

BWM???

[@selasiocansey]

BEST SOOOOOOOOOOOOO FAAAAAAARRRRRRRR

[@davidsoroko3137]

Very useful, I coded a Java implementation off this presentation ( github.com/sorokod/Burrows-Wheeler-transform ).

[@austinwelch3856]

Love the video, but I feel as though you skimmed over the lexicographic sort very quickly without much explanation. @1:27 would be great if you had an intermediate step to explain the sort. Maybe understanding a lexicographic sort is a prerequisite to learning this, but if you are trying to start from nothing, it would be great to visualize that sort process! Otherwise amazing video, and love the reversal explanation! So interesting that many other videos do not explain the reverse given that the transformation is almost useless without understanding the reversal!

[@derrylab]

Every lecturer should have youtuber skill for better explaining skill

[@markmanning2921]

i cant watch this, it goes on... and on... and on.... and on... and on.... and on.... and on.... and on... ad infinitum. you take 37 minutes to describe something that could have been described in TWO minutes without ANY loss of information. you dont need to repeat the same thing 8 or more times in a row in different ways in order to teach!