Positional Encoding in Transformers | Deep Learning | CampusX

Positional Encoding is a technique used in transformers to inject information about the position of tokens in a sequence. Since transformers lack inherent sequence order awareness, positional encodings enable the model to capture the order of words, crucial for understanding context. These encodings are added to the input embeddings, allowing the model to process and learn relationships based on token positions.
Blog link - blog.timodenk.com/linear-rela...
============================
Did you like my teaching style?
Check my affordable mentorship program at : learnwith.campusx.in
DSMP FAQ: docs.google.com/document/d/1O...
============================
📱 Grow with us:
CampusX' LinkedIn: / campusx-official
CampusX on Instagram for daily tips: / campusx.official
My LinkedIn: / nitish-singh-03412789
Discord: / discord
E-mail us at [e-mail]
✨ Hashtags✨
#datascience #positionalencoding #campusx #deeplearning
⌚Time Stamps⌚
00:00 - Intro
01:18 - Why Positional encoding is required?
07:58 - Proposing a simple solution
22:17 - The sine function as a solution
34:26 - Explaining Positional encoding
55:25 - Interesting Observations
01:04:06 - Mind blowing solution
01:10:13 - Blog for mathematical intuition
01:12:45 - Outro

[@sejalninni3076]

paid course me bhi itna explain ni karte he. usme ap yeh puchte reh jate ho ki yeh concept kyu aya, kaha se aya, kyu chahiye. Trust me i have take a paid course but its no where equivalent to Nitish sir's free content also. Yeh hoti he padane ki iccha. Thanks a ton sir.

[@datascience9020]

sir, please complete this playlist as soon as possible (we depends on it)

[@uttamkp2009]

True

[@not_amanullah]

Yes

[@rahulsbytes]

Yes sir, Please.

[@rb4754]

yes

[@ashutoshpatidar3288]

Yes sir pls

[@Ashar_Paracha]

Huge Respect for you. Without any doubt, you are one of the best Teacher. Love From Islamabad Pakistan.

[@r21061991]

Bhai Pakistan me chalta hai kya ye sab ?

[@Ashar_Paracha]

@@r21061991 Yes, you are welcome to Pakistan, especially Nitish Sir. I will sponsor everything for him in Islamabad.

[@sandysam435]

Bomb banaana sikho bhaijan, usme tmhra jada scope hai

[@Ashar_Paracha]

@@sandysam435 Bro, I am not a negative person so I don't want to give you an answer. I wish you will visit in Pakistan and see how people are.

[@vishvajeetsingh-d9v]

​@@sandysam435​bhai tune bahut mahan kam kara h na ye bat ba kar . sir ki hi respect kar leta kam se kam to unke student ko b kuch bolne se pahle soch leta

[@priyanshushukla1295]

i already purchased a course on this ml and deep learning worth 17000 still i am learning here because the depth and way of explaining is unmatchable respect 🔥

[@RashidAli-jh8zu]

Absolutely brilliant content on positional encoding! The way you've broken down complex concepts into easy-to-understand segments is truly commendable. Your teaching skills are exceptional, making advanced topics accessible and engaging. The depth of knowledge and clarity in your explanations reflect an awesome level of content. Keep up the fantastic work!🔥

[@KumR]

My Goodness Nitish. This was really Deep. Thanks for doing this. U are now not only Andrew Ng but also Andrej Karpathy of India.... Cant even imagine how much research went into this. Interpreting the paper might have been really tough. WaytoGo... I think I need to see this many more times for it to sink in..

[@sachink9102]

Hi Nitish sir, Please keep going this series of Deep Learning . Transformer and others

[@AllInOne-gn4ve]

Without any doubt, you are one of the best Teacher . sir, please complete this playlist(transformers) as soon as possible

[@ayushsrinivas]

sir also please complete playlist fast sir . Its placements time

[@trickydotworld]

Thank you very much for this master piece on Positional Encoding.

[@muhammadfarrukhshafeeq1955]

Yes, this video is an excellent effort by you which shows your deep study, research, and excellent way of explaining. Honestly. May Allah Kareem bless you more.. Dr. M, Farrukh Shafeeq

[@ShubhamGupta-zt6me]

My goodness, what an in-depth explanation about positional encoding. Huge respect for you Nitish Bhai.

[@cool12345687]

Time and again , I keep on writing about your videos. You are simply superb. No one has taught this topic the way you have done. God Bless.

[@100deep1001]

Huge respect for doing this at such depth!

[@umravsingh3998]

Simplified, curiosity Sparked, love the maths behind positional encoding. Thank you ❤🙏

[@vishalnaik5453]

Nitish bhai .. respect+++🫡

[@rahulsbytes]

Thank you, sir, for this video. The best teacher on KZbin I have ever seen. Huge Respect 🙏.

[@harsh.gupta2021]

Best ever video on Positional Encoding.... Hands Down 🙌🙌

[@kaushikmondal1772]

Sir i gave up hope of learning Deep Learning properly but seeing your content and videos brimmed confidence in me again. Thank you Sir, for these wonderful videos may God give you my life so that you continue doing this work for everyone. Huge respect for you.

[@priyankagupta2611]

Gone through a few articles but none of it explained the concept in such great details. Thank you.

[@WIN_1306]

WHICH ARTICELS CAN U SHARE THE LINK ?

[@sauravnaik6405]

Hey just completed watching this. Amazingly executed. And the last part of linear transformation was very insightful😊 thank you for making it palatable.

[@SandeepSingh-yx2si]

Superb Explanation and Hard work.

[@gauravfamily2209]

mind blowing.... your transformer videos .......👌Pls do continue its remain part....

[@vimalshrivastava6586]

Thank you for such a wonderful explanation.

[@sayamkumar7276]

Hi sir, please complete this deep learning playlist as soon as possible.. Really excited and enthusiastic to witness your impending videos in this playlist as it can help me enhance my understanding of transformers in a better way..

[@avniashkumar102]

You are great sir . The way you are teaching is mind blowing .

[@SurajPrasad-bf9qn]

Really very spark generation video

[@hashlycomedy3819]

Waited for very long time...

[@ujjawalagrawal]

Sirji..u the best mentor, guide..Please create playlist for llms, bert

[@koushik7604]

Wonderful explanation ❤

[@swapniljoshi8063]

Ultimate Video please make 20-30 min video on that matrix....................

[@3daudiomusic540]

Thank so much sir Please continue this playlist💯💯💯💯

[@VISHALSINGH-xg4ij]

huge respect for you sir

[@utkarshtripathi9118]

thank you so much for this video sir.

[@josebgeorge227]

Thank you Sir Ji!! Aap next level ho!!

[@slowedxreverb6274]

hats off to you sir.🙏

[@ayushsrinivas]

Super explination sir ( what - why - how ) type teaching making you stand out sir. Sir In the next upcomming months placements are going to start sir please make a 120 days road map for AI engineer . please sir

[@snehal.2994]

Great lecture sir

[@animatrix1631]

I remember asking Nitish to cover this in one of the older self attention videos , am not sure if it contributed at all but am happy anyway

[@MrSat001]

Thank's a lot Sir 👍

[@advaitdanade7538]

Sir, Can You tell me How many more videos are left in this series?

[@user-vd9xf9pj7p]

mind blowing explanation.

[@WIN_1306]

WHY DO U HAVE THAT BLUE STAR BESIDE YOUR PROFILE?

[@paritoshkumar7422]

Thank you so much Sir

[@oden4013]

THANK YOU SIR!

[@not_amanullah]

This is helpful 🖤

[@Amanullah-wy3ur]

this is helpful 🖤🤗

[@shrushtitripathi3873]

Thanks a lot sir. Thanks a lot🙌🙌🙌🙌🙌

[@quartz9793]

Thanks for this🙏🙏🙏...but next videos plz thoda jaldi jaldi release kariye sir🙏🙏

[@technicalhouse9820]

Thanku Sir From Gilgit baltistan Pakistan

[@gurvgupta5515]

Thanks for this video sir. Can you also make a video on Rotary Positional Embeddings (RoPE) that is used in Llama as well as other LLMs for enhanced attention.

[@mindpower1332]

sir, plz make more video on this series

[@space_ace7710]

Can you please describe what will be the topic flow for future videos??

[@rahulkumar-op7ul]

Thank you sir

[@AmitBiswas-hd3js]

sir, please complete this playlist as soon as possible

[@mirandasuryaprakash3820]

Salute 😀

[@sagarbhagwani7193]

Finally ❤ thanks sir

[@Amanullah-wy3ur]

thanks ❤

[@siddiqe-td8ts]

dear sir! please complete this playlist

[@rb4754]

1:05 Video length is not an issue at all Nitish sir, infect the longer the video the more clear will be the concept... Transformer is a complicated architecture to understand. The way you explain concepts in detail is super informative... I was waiting for this video... Looking forward for the next video on transformer series...

[@abhilashadimble7091]

Sir please NLP ka real time project ka course bhi add kriye... Free courses mein.....

[@oden4013]

kabhi kabhi research paper ko pad kar lagta he ki duniya ma itne intelligent banday kaise ha

[@ShehriarAliKhan]

Sir please complete this playlist as soon as possible

[@aiforeveryone]

Great

[@sarojrai9326]

Please continue MLOPs tutorial regularly please 🙏

[@yashshekhar538]

sir please bring next video asap

[@ImtiyazMd-sv9my]

When we have same data for every head, and same training process, how are we able to get different weights matrices for query, key and value in different heads, how did we get diversified weights matrices for every head, is it because of different weight initialization in each head or the order of tokens, in case of multi-head?

[@shubhamsharma2319]

Sir, please complete deep learning playlist

[@user-qo5qj9pv5e]

Sir, please complete the playlists ASAP.

[@JasonC-ai]

Hello🙋‍♂ Nitish Singh Sir, I noticed that the course is six months old, and I am wondering whether it is still the best option for me to purchase. Could you please let me know if there is an upcoming version of the course that might include updated content or new features like are you releasing DSMP 3.0 anytime soon? Additionally, I have a few questions regarding the current course offering: 1. If I purchase the existing course and watch the recorded videos, will there be an opportunity for me to ask questions or get my doubts clarified? 2. Is there any support provided to students who buy the course, such as Q&A sessions, discussion forums, or direct communication channels with you? Pls reply 🙏

[@K_SE__VishalRoyRoy]

sir, last portion of the video where you saw a linear transformation usually we called euler formula , I don't understand that whether are talking about absolute posiitonal embedding or relative positional embedding. in future will you come with another video where you will teach about rotatory position embedding?

[@himanshurathod4086]

Can you please upload ner and topic modeling lecture?

[@ujjawalagrawal]

Where can i get video on BERT and Varients of BERT

[@RamandeepSingh_04]

Am at RNN Reaching here soooonnn Completed DSMP❤️❤️❤️❤️

[@WIN_1306]

WHAT was in DSMP? IS IT WORTH IT WHAT U PAID FOR IT? WHAT DOES DSMP INCLUDE?

[@psldatasets2FAST]

sir, this video was superb. But I have a few questions related to positional encoding. The first question is that how does a vector define the position of a word in a sentence whereas it is consisted of multiple numbers. How do the numbers used in positional vector tells that a particular word exists in a position i.e first, second .....The second question is that after the addition of positional encoding vector with the embedding vector how does the positional and contextual embedding information is maintained using single vector in subsequent stages after this addition. I hope you understand my question.

[@ram_c]

Please make tutorial on GNN

[@barryallen5243]

I tried leaning about positional encoding myself some time ago and came across the same blog, but couldn't understand most of it. After watching this video, I will give it another try as my understanding of the topic has increased. The equation for positional encoding is so simple, but it wasn't making sense to me. Most of my doubts have been resolved, but I still have one point of concern. I get we are adding pos encoding instead of concatenating it to avoid increasing dimensions. But wouldn't adding alter the meaning of embedding, as we're changing their value at different positions by different magnitude? PS: The figure showing similarity between pos encoding, and binary encoding makes you genuinely happy as a person who loves Mathematics.

[@vishwjoshi853]

Sir can you please make videos on adversarial attacks. It is very interesting topic yet there not enough resource to understand it properly. I am currently doing a project on that and I am facing so much trouble in finding a good tutor on this subject. There are just research papers available and I have read some but yet I am not clear about some things. I would love to learn from you, I request you to make videos on this.

[@ashutoshpatidar3288]

The climax in the last 3 mins>>>

[@WIN_1306]

WHAT DO U MEAN?

[@ashutoshpatidar3288]

@@WIN_1306 all what's been taught has made absolute sense at the end

[@AllInOnekh]

Please atleast one video in a week. Highly depend on your videos..

[@harshgupta8936]

witing since 2 months. sir please complete this fast.... if possible in next two months

[@advaitdanade7538]

next video in the series please sir

[@shreeyagupta5720]

Sir we also want a playlist on big data it's a request🙏🙏

[@kashifsadiq4145]

Yes

[@WIN_1306]

@@kashifsadiq4145 CHUUP MULLE KISI MULLE SE PARHNA

[@WIN_1306]

WHAT DO U MEAN BY BIG DATA?

[@somyasingh8632]

I request you to please complete this Gen AI series Nitesh.

[@muhammadsheraz177]

Sir I'm from Pakistan and I have request you to complete this Playlist as early as possible

[@eyeonmystery]

Sir please complete transformer's playlist

[@ANAND02120]

The matrix property that you discussed at the timestamp starting at 1:05:45, how is it relevant to the Neural network training process or positional encoding? Say $M$ is the matrix so you are saying |Mv -v| is constant. So if we have position encoding say with vector say v1,v2,v3,v4,v5,v6 you want the property that |v_{n+k} - v_{n}| is always constant? If that's the case then how is M relevant?

[@muhammadikram375]

sir please do some working on MLOps series

[@hashlycomedy3819]

How you can find this much insights?? I mean how clearly you find resources?

[@hey.Sourin]

At 14:10 , why would I number two different sentences on the same document with same index? The numbering of second sentence should be start from '3' if it is in same document, right? Can you please explain it sir? Thank you?

[@Mjjjyyy]

where is the next video? sir pls krdo upload

[@varunahlawat9013]

When we add the positional and embedding vectors, the resulting vector have no way to know, for any element, that what contribution came from embedding and how much came from the positional. Both the properties should be lost and it should lead to a sort of useless thing. But of course that doesn't happen if we assume that this is what's being used. Can you explain why

[@Basit-nx3nu]

sir when you will start MLOPS playlist properly?

[@darshanv1748]

Sir you are saying in the first solution where we add position in a dimension that the problem their is it is unbounded but sir the max no of tokens we will only be able to pass is context length right ? So it is not unbounded right ?

[@ayushmanmishra4634]

I think uniqueness will not be a problem if we use sin(position) As no two natural numbers will have same value of sine even if it is periodic if sin(n) = sin(m) => n = m + 2πk => n-m = 2πk not possible for two distinct natural numbers Please clarify

[@navneetgupta4669]

Sir, at 14:40, why we have not used padding? It will make number of words in each sentence equal and problem is solved.

[@studology67]

yeh transformers ki konsi playlist hai mereko mil nahi rhi

[@ant9896]

Sir please upload video about transformers😢

[@princekhunt1]

👍🏻

[@ManpreetKaur-fo4li]

Sir ap ka data analytics wala koi course available hai ????

[@sanskaragrawal3526]

Sir Please Complete this Deep Learning Series in Faster way because this series has already taken a lot of time sir... i want to learn Deep learning only from you that's by i'm telling this..

[@AyushRaj-nt3ot]

True please complete soon, We have our placement season next semester only

[@AyushRaj-nt3ot]

True. Please complete asap as our placement season is from next semester only.

[@sanskaragrawal3526]

@@AyushRaj-nt3ot Yea Same.. ✌✌