REAL DATA: YouTube's Thumbnail Tester is Completely BROKEN

KZbin's test and compare feature appears to be totally broken. Help us keep testing the validity of this tool by running your own A/B test and control tests!
Thanks to ‪@raffitimedia‬ and ‪@Jogwheel‬ for your thoughts on all this and your help with the video! You guys rock!
Check out our KZbin System (and so much more) in Project 24: channelmakers.incomeschool.co...
Join the Channel Makers Insiders group (free!) for first dibs on data videos and in-depth audits: channelmakers.incomeschool.co...
----------------------------------------------------------------
How we can help you
🥇 Project 24 Membership: channelmakers.incomeschool.co...
🥋 One on One Coaching: incomeschool.com/private-coac...
🛠 All of our Current Recommended Tools: channelmakers.com/tools/
----------------------------------------------------------------
#youtube #testandcompare #thumbnailtool #a/b/testing #newfeature
Timestamps:
0:00 -
Project 24 is our membership program that teaches you how to work toward replacing your current income with income from KZbin channels and websites in 24 months.
Project 24 is a product of Income School LLC. Results mentioned are not typical. The trademark “Project 24” refers to the goal of some people to create a successful business in 24 months, but is not a promise or guarantee of that success. Many online businesses fail because, like any business, it requires hard work, skill, and dedication. Before making business decisions, consult financial and legal professionals licensed in your jurisdiction.

[@ChannelMakers]

Do you plan on trying these tests yourself? We're going to keep an eye on things to see how it's progressing so stay posted! Thanks again @Jogwheel and @raffitimedia for your input!

[@LearnEverythingAboutDesign]

Been using it for a little while and "inconclusive". It seems to favor text skewed at an angle and more simple layout. Haven't tried the AI generated thumbnail thing

[@MuaythaiGalaxy-Official]

Ohoh Vicious video and subject idea, nice one

[@Lord_Funk]

You guys should try/test the YT AI bot who control comment sections. Over the last few months it seems it's deleting comments randomly within seconds after posting. Maybe not totaly randomly, but I have tested to rephrase whole sentences and still they get deleted. No foul language or links being used. Comments can even get deleted within the same tree of commenst under a main comments when you exchange a discussion back and forth. My guess is that if you talk about a topic more than once and naturally use many of the same words or mention the same things, like company names etc, the bot think you are a spam bot and can't tell the díffernce between a xconversation around a topic using the same words (since it's the same toipic) in multiple comments or a spam bot typing more or less the same comment over and over again. It's so frustrating. Once one person and I who tried to exchange information around a tiopic had to start typing backwards to not get our comments deleted, so it's not just me, but happens to more people as well. You can even make a comment under one video about the topic of that video, then stumble upn another video around the same topic and all your commnets under that second video will get deleted, since you already talked about that topic before, even if write and structure it quite a bit differently. It wont matter since you have to use some of the same words, since it's around the same topic. It just seems to get worse too, not better. As if the AI bot is not learning at all over time.

[@Lord_Funk]

And now exactly what I tried to write to you about happened again, my whole original sentence was deleted by the YT AI bot within seconds after having posted it. **sigh** This one might get to stay, but my original is gone.

[@Lord_Funk]

Trying to repost but split it up in more than one comment. Maybe that will work. You guys should try/test the YT AI bot who control comment sections. Over the last few months it seems it's deleting comments randomly within seconds after posting. Maybe not totaly randomly, but I have tested to rephrase whole sentences and still they get deleted. No foul language or links being used. Comments can even get deleted within the same tree of commenst under a main comments when you exchange a discussion back and forth. My guess is that if you talk about a topic more than once and naturally use many of the same words or mention the same things, like company names etc, the bot think you are a spam bot and can't tell the díffernce between a conversation around a topic using the same words (since it's the same topic) in multiple comments or a spam bot typing more or less the same comment over and over again...to be continued...

[@YouTubeLiaison]

Thanks so much for the feedback Julia and team! Thanks to beta testers like you we made changes to reduce false positives/“control tests”. It’s not possible to eliminate them completely but the release version of Test & Compare just rolled out this month so we’re eager to get more feedback! The goal is always the best quality results and experience for creators and viewers!

[@BonytoBeastly]

What was the issue before? Smaller channels not being able to reach statistical significance?

[@ChannelMakers]

@KZbinLiaison Hey Rene! We are hopeful that the tool will be usable at some point in the future, but with the results we and other creators are getting, (Even since the official release) the tool is unreliable at best. That said, we truly hope it can be improved to a point where it can make positive impacts for all creators. We would totally be open to the possibility to have some sort of discussion with you and/or other KZbin engineers who have been working with the thumbnail tester to talk through the concerns we and many other creators have. We really appreciate what you're doing as the KZbin Liaison!

[@joyfultrails]

The feature I would love to see on KZbin is actual free speech! Stop applying warning labels/"fact check" messages to videos that simply don't put forth a left or woke point of view. Not the topic of this video, I know, but PLEASE do something about that.

[@Lord_Funk]

Hi there @KZbinLiaison, please read my comments in the *ChannelMakers* pinned comment under the video. It's about how comments gets deleted automatically for no reason. I'm 50+ years, so I do not use much foul language, especially not when typing and no links being used in comments etc. I explain more in my comments under that pinned comment above.

[@Enjoyurble]

Can you explain the logic behind using watch time as the metric to benefit creators over click through rates, because there's no correlation between the quality of the thumbnail and the quality of the video. If watch time was going to be the metric, the more logical A/B testing would be for creators to have 10-30 second plus intros for each video and then have the one with the least amount of drop off be chosen. The current format just seems far more to the benefit of KZbin's algorithm than to individual creators. Which I would expect on the backend regardless, but yeah. For a creator, it seems like being able to see things like data on the audiences of all 3 tests and the engagement in terms of likes/dislikes/comments and subscribers for each would help more on creating videos and thumbnails in the future. Right now, it seems like at best you can maybe start to see a pattern over multiple videos that "the audience likes this combination of thumbnail and video", but even that might be difficult to correlate and while you might be getting more overall watch hours to start, without more data on who is clicking and watching each test thumbnail you could be unintentionally missing out on and alienating your target audience over time. I can see the current format possibly being of use to HUGE creators who have such a large amount of data they can see stronger correlation for their videos, but it's not overly surprising in its current form that the variation would be negligible for the average creator when once a viewer clicks on something they're usually either staying or leaving based on the quality of the video alone. Like, unless your thumbnails are SO off that they're entirely missing viewer expectations, that seems reasonable.

[@bobbyrebholz2183]

Thank you for this. You might have exposed an even bigger flaw with the KZbin algorithm than just peoples' choice in thumbnails. It almost seems like thumbnails actually don't matter much, it's just who the algorithm chooses to dish the video out to. They should be doing a video title tester and not a thumbnail one.

[@TheOnlyTaps]

🎯

[@davejefferynf]

This weekend, I changed the thumbnail on my current video because it was lagging - didn't help much. Then I changed the title and description and things started picking up. You make a good point about thumbnails.

[@VeganofCourse]

100% agree. But too many of these KZbin strategists can't stop talking about thumbnails. The tool isn't broken; the mantra "Title and Thumbnails" is just BS. The algo finds audiences for you. Thumbnails don't matter.

[@DrDreadMarley]

This is actually a plausible theory, although I think thumbnails matter to a point as I click on videos subconsciously because of good thumbnails but it would make sense based on hit and miss with audience reach despite by now KZbin knowing my target audience yet some videos with great thumbnails get pushed to maybe 300 then another will get pushed to 18000k people. Strange.

[@KevinChee]

I agree. In my experience, the title actually weighs more than the thumbnail when it comes to CTR. There's just a lot of videos talking about how having a good thumbnail increases clicks but there really isn't any way to measure that, a lot of it is based on assumptions and preference. Like for me, a good thumbnail that stands out is what captures my attention first, it is what makes me stop scrolling. Then, I read the title to learn more about the video before I decide if the video is for me.

[@CaptainSauce]

They know what's wrong with A/B testing but there's no easy fix. The reason your results seem random is because the data is % watch time yet thumbnails have almost no influence on watch time. When has anyone ever watched a video for a few minutes longer because of the thumbnail they briefly saw before starting? Creators have been asking for % clickthrough rate, an analytic that actually measures thumbnail efficiency, but KZbin thinks that would incentivize bad-habit clickbait from creators who don't interpret the data. I can't really blame them because even supposed YT experts like this channel seem to completely miss any kind of interpretation. 14 minutes of presenting tests yet you guys never once explain or theorize why you get the data you get.

[@ChannelMakers]

Here's a bit of our thought process: When running experiments, it is common for the test groups to be randomized, which means there will be some variance in each test group. By running the control tests, it allows you to establish a baseline for the variance. Meaning you will know what amount of variance is normal. Hypothetically the variance group by group SHOULD be minimal if the test groups are sufficiently randomized and the sample size for each test group is large enough. In the case of all the control tests we ran, the results showed that the variance between two identical thumbnails was oddly large. (Large in a concerning way) So we tested again and again and again so we could find the average variance. After we had an average variance for the control test we could run actual A/B tests (two unique thumbnails) to see what the variance of the A/B test was for them. And since we know the average variance for the control tests we will know what result for the true A/B test would actually be significant, meaning we would know how different the watch time split would need to be for us to know if one thumbnail was really better than another. It turned out that the variance for the control tests were LARGER than the variance for the A/B tests. That means we can have no confidence in the A/B test results because the variance SHOULD have been larger between 2 totally different thumbnails. There might be a problem with the way the test groups are put together, the test groups might not be randomized well enough, the sample size might not be large enough. There are a lot of things that might be wrong, but the bottom line is, the results we are seeing cannot be trusted.

[@AndreSjoberg]

@@ChannelMakersOf course the results are oddly large when running 3 versions of the same thumbnails, because there is nothing for the system to randomize the people in the 3 groups seeing the same thumbnail from. They can’t randomize the users based on how the thumbnail looks and how people historically react to similar thumbs cause they don’t have the data - in theory the video should only be shown to people youtube think is interested in the contents of the video, bit apart from that youtube has no other way to qualify the randomization this way or that way other than picking people at random every day over time. But the sample size of people will be too small and time too short to truelly get a proper average when showing the same thumb as a/b/c If you where getting close to 33/33/33 with 3 random groups of people there would *really* be reason for concern. The chances of the same x % in each ramdom group liking a thumbnail *and* watching the video for a long time is very unlikely unless the sample size/number of people was huuuuuuge. Also, the logic for calling the tool broken because of a large discrepancy between the groups seeing the same thumbnail is flawed, because you then assume people with the same interest in the topic will respons similarily to the same thumbnail - you can easily have 3 people share the same interest in miniature painting, but react totally different to the same thumbnail based on things like personal experience, references, type of person they are, visual preferences, etc etc. Admittedly the a/b-testing should have some correlation between clickthrough-rate and watchtime, not just watchtime, and make those numbers available, since knowing which thumb got the most clicks and least watch hours vs clicks and longest watch hours etc. is key. However, the way this testing was done does not reveal the tool as broken. If for example the test was done with 2 «good» thumbnails and one that’s just wrong, and the wrong one ended up with thw worst results that would show the tool working as intended. If three thumbs are pretty good of course a/b/c will not give solid results, and also a bot random results, because you have no way of knowing the what combination of people are in the three groups. So right now the tool is only useful if one of 3 alternatives pull ahead by at least 50%, of course depending on what the spread of the remaining 50% is. If one is 50, one is 40 and one is 10 the tool is working, and has identified the «bad» alternative, and confirmed that you have 2 pretty good alternatives, but none of those 2 are much better than the other and possibly within margin of error based on sample size. And calling a/b-testing like this a «tool», with so many uncontrollable factors re: the groups composition, expecting definite answers, is naive. At best a function like this can be a useful indicator if one has a clear winner, nothing more. And if there is no clear winner it means the two closest ones, or even all three are good options, which is also a result, because it then means back to the drawing board and make something better, or, if one is certain they are actually good then one has the luxury of 3 good options to choose from. But as long as we have no clickthroughs vs exposures vs watchtime-data for each there is no way of knowing if the three similarly scoring thumbs are equally good or just equally mediocre. But the data for the video is still there, so it is possible to look at total exposure vs watch time and other numbers and see in general if the numbers are good or not, and judge the overall performance, and from that decide if a new round of thumbs is called for or not.

[@kandjlumber]

This is something that has been at the back of my mind in a different way. But you’ve clarified it for me.

[@eddieaugustin3811]

@@AndreSjobergyou must be a math major or statistic expect or something bc this was cray!!! Hahahaha! Good job brother! 🤌🏼

[@ScaleAutoGarage]

@@AndreSjoberg yuup, feel exactly the same as you do on this one. You put it into a lot of words very well. haha. I feel real testing would have thumbs that are vastly different and possibly have nothing to do with the video in question. I have a feeling that it does use some formula using CTR and watchtime to create its percentages but without knowing exatly what is being used, there is no way of knowing for sure.

[@ItsBrendaBlanco]

Guys, this is a terrific contribution to the creative community. Thank you so much!!!

[@ChannelMakers]

You are so welcome. I'm really glad we got this chance to spread the word. Thanks for watching. -Julia

[@GamerKittyEuphoric]

yeah I tried it, and it ruined my video lol. The CTR was at 9% - dropped to 2% once I started using it... it was horrible.

[@Jogwheel]

A pleasure talking with you about this, Julia - thanks for inviting me on to chat! And thanks for leading this quixotic charge to inform the public about KZbin's latest blunder 😀 I, for one, will continue to simply poll my audience and/or my wife on which thumbnail she looks better, haha - as I don't think I can trust this new tool at all.

[@ChannelMakers]

That was a really fun chat, I so appreciate your input and your behind the scenes info of what has happened over the last decade with this. Thank you so much for taking the time! Love your strategy, solid advice.

[@CineG]

Used the feature once. The three thumbnails were; a wide shot, a closeup, and something abstract. The closeup won by far with the more abstract thumbnail dead last.

[@darlenefraser3022]

That’s interesting because that’s how I would have thought it would be. Maybe it got one right? 😅

[@AliexFolgueira]

I have this weird theory that thumbnails, title, and the video itself does not matter 😅 as long as all those 3 factors are good it enough, it is up to luck.

[@NaudVanDalen]

Sometimes they give a video a lot more impressions and with everything else being the same, it gets a lot more views. Give one of my bad videos a billion impressions and they get more views than a MrBeast video if it only got a million impressions.

[@afrominded]

Honestly, same! lol

[@AndreSjoberg]

Up to a certain amount of views that is probably not wrong - however a REALLY good title and thumbnail could change things from luck to good results - an example from the advertising industry: a large company buying thousands of billboards showing a bottle of soda will get increased sales just from the exposure, no good ideas or anything necessary, just numbers. But a small company making a CRAZY COOL insane ad with a limited run and small amount of billboards *could* pop off if enough people see it and take pictures and spread the ad. This has happened several times in real life. So a video with a strangely weird or good combination of topic, title, thumbnail that sticks out so people immediately click on it and watch it for a long time *will* get more exposure than a "normal" video, which again leads to more views and more exposure - it's a simple mechanic.

[@aGameScout]

Alright here's my theory on why this feature seems so borked. It's known that the algorithm is super reactive to how much of an outlier your video is statistically to your previous videos, videos in your niche, and videos across youtube in general. On a larger channel where you get a lot of view data right away, you can know within the first 10 minutes whether your video is doing well with your core audience. So the algorithm handling a video going out with 3 different thumbnails is a conundrum. It's basically 1 video and 3 videos at the same time. So if the immediate data is that one thumbnail is great and the other two ones kinda suck, how does it handle that? Does it take the average performance between all 3 as its metric on whether to boost on the algorithm or pull back? That would be super unfair for the video if it was a banger that just needed to ditch the other two crappy thumbnails for 2/3 of the audience test in order to perform well. Does it boost based on the performance of the best thumbnail, and just drag along the other two worse thumbnails for the ride? That hurts the algorithm as a whole, because it's supposed to maximize viewer watch time and could end up having recommended pages filled a bunch of crappy thumbnails dragged into the spotlight by all the A/B testing. It's a really complex problem to treat fairly within the length of time that the user sets for the test. On top of all that, we also know the algorithm by default is already doing tons of A/B testing on videos with different groups of audiences throughout its initial days, and depending on which audiences respond the best to it, a video can end up traveling out to very different potential pockets of viewers. All that A/B testing is fairly straightforward to keep track of when you just have one video with consistent packaging the whole time. But the moment you introduce new thumbnails, especially wildly different ones, it's possible that certain ones could statistically respond better to different audiences. So how on earth does the algorithm handle its viewership A/B testing and thumbnail A/B testing simultaneously? My hunch is that there's a bunch of internal dials adjusting the percentage of A/B thumbnail tests that are applied to the similar vs more diverse audience pockets. And if the initial A/B testing is inconclusive, the dial ends up turning up the variance of audiences to see if it can find one of the thumbnails possibly working better with a specific audience. Which could actually explain why the A/A tests gave more variance than the A/B tests. By using the feature in a way that wasn't intended, it confuses and completely screws up the mechanisms designed to make it work optimally under normal conditions. Would also explain why the final feature is pretty vague in the actual data it gives you, only displaying raw watch time % share instead of a whole dropdown menu of stats like Jon described in the concept version from the KZbin employees. It gives off the vibe of "this feature interacts with the algorithm in a very complex way, we did a lot of testing and trust us, it works and showing you more data would just confuse you".

[@EpicDIZ]

Could it mean that the thumbnail might not mean as much as we thought? Or is just that tool is broken?

[@SavannaBanana205]

Yep, i used it for a few days, and then realized that KZbin was testing different thumbnails on the same people - making them think that i had posted a new video.... Which just pisses people off.

[@ChannelMakers]

Wow how did you realize that? From what we've seen I thought they always kept it separate.

[@Novakov.]

​@@ChannelMakers From what I have seen on my homepage, if I don't click on a video, later, they show me a different thumbnail. It's kind of annoying because I think it's a new video, but no, it's the same title. Sometimes, I even liked the previous version of the thumbnail better, but I just didn't have time to watch the video, so I didn't click.

[@SavannaBanana205]

@@ChannelMakers I'm obsessively checking the video to see how many new views it's gotten. When I check it, the thumbnail has changed.

[@SweetPhil]

You know what, I was talking to one of my dedicated viewers in discord and he had told me how every creator had started switching there thumbnails on videos a day or two after posting and how annoying it was all of a sudden. I bet it is For this exact reason, the A/B testing showing different thumbnails to the same person. They would think it was a new video, click on it, then realize they just watched it earlier or the day before and instantly click off. As a creator, I bet that crushes your average view duration too.

[@OffGridAdventure]

This is where channel makers can make a difference in the creator education niche. Thanks for articulating what I’ve already noticed with A/B testing on my channel. KZbin dumbed it down for their own benefit while limiting the usefulness for creators.

[@AncientCityCraftworks]

I am convinced now. Ran test and compare. A was a tn I wasn't too happy with and thought B was stronger. Results were 83% for A but CTR was .6. Not 6 but .6. I finally gave up and deleted A and went with B. As soon as I did that my CTR went way up, watch time went up. It went from my worst performing video ever to average. YT had to be showing the weak tn A disproportionately high rather than an even split.

[@davejefferynf]

Thank you for proving what I had suspected.

[@ChannelMakers]

You bet!

[@TechnoPandaMedia]

I think unless your video gets lots of impressions, this feature is not for you.

[@AgnesMann]

It makes sense. I thought KZbin was picking the worse versions for me. Also it would depend on who they show them too

[@shortreststudios]

So, I tried. It won't let me upload the same image, even if I create a copy with a different file name. It just reverts to the same thumbs in my previous test.

[@ChannelMakers]

Interesting. I tried it just today on this channel and it worked for us. I wonder if it's a glitch because you had a previous thumbnail test on that video in the past. I have noticed a couple of other bugs with it too. Maybe later it will work. Thanks for trying!

[@JamesKerLindsay]

Thanks. This was really helpful. I ran some tests and I just couldn’t make sense of the results. By the way, personally, I’d much testing for titles than thumbnails.

[@curbyvids]

I didn't know it was just an A/B testing tool. I thought it was supposed to recommend different thumbnails, to different demographics, like what Netflix does

[@TheOnlyTaps]

Great watch. I only got access to the tool on my gaming channel this month and didn't notice any difference but thought it was just me. Although my videos don't get enough views to warrant any good test results so glad to see bigger channels are able to cover it. And like some of the comments noted, i think what is ultimately going to matter more in the algorithm is WHO the video gets shown to coz no matter how good a thumbnail is, someone can still nit click on it if its just not the target market.

[@coachjohn]

Have you put up 2 thumbnails...one that is based on a thumb that is really really good and another made intentionally wrong?

[@ChannelMakers]

No, I'd like to though! It's just painful to know that part of our audience will see a really bad thumbnail in order to run that test 😅 Maybe we'll bite the bullet sometime anyway. -Julia

[@coachjohn]

@@ChannelMakers any regulars would understand. I think this could be a crucial test

[@TomleyRC]

I’ve been a Beta tester for this for ages, and honestly… I’m sure my videos where I don’t use it perform much better. But sometimes my curiosity gets the better of me and I have to test 2 or 3. I’m convinced it negatively effects the overall CTR regardless of how good your Thumbnail is. I use it way less now than i was a few months back.

[@ChannelMakers]

Wow that's wild. What a shame. I'll be interested to keep an eye on that. From some of our tests we've seen (what seems to be, without a lot of data) the same thing, that it hurts the videos. But KZbin just released a video about the feature and in many of the commenters are saying it really helped their channel. I have no clue what is going on at this point.

[@MiddleAgedSwedeGoesForAWalk]

I get the feeling that the test & compare reaches a conclusion way to quickly, especially considering that it uses time share as the deciding factor. I'm a pretty small channel and the content I create really isn't for everyone, when I release a video and check the watch time of it, it's always pretty obvious that there's a few people, probably mostly old time viewers who know what to expect, that watch the full video, while a lot of the other viewers give up very quickly. Honestly, the audience retention graphs on most of my videos are catastrophically bad. I was really surprised to see that one of my recent videos reached a conclusive result after just 170-ish views. That was a 62 minute long video, and according to the audience retention graph, after 22 minutes only 15% were watching and by the end only 9% were left, so less than 17 of those 170 people watched the full length of the video while most gave up A LOT earlier. This means that the deciding factor probably wasn't related to how good the thumbnail was, it was which of the thumbnails got presented to the most those, less than 17 people, that watched the full duration of the video. A thumbnail could have abysmally bad CTR, but as long as that thumbnail is presented to the largest share of "old time fans" of my channel that'll watch the entire video, it's almost certain to get the best time share %.

[@DoItWithPaulie]

i have found since midnight june 23rd no more impressions given on new videos with a test running.old videos with new test running seem ok

[@belyiam]

I totally agreeee!! I just uploaded a video today and I tried to test it… KZbin picked an image from the video as a thumbnail 😅

[@ThirtyMinuteMarketing]

The A/B tool, or more appropriately named the Thumbnail testing tool, is currently limited in its scope as it only tests one aspect of what contributes to the "Watch time share." The thumbnail alone cannot influence how long viewers watch a video, as it's just one of many factors. The real determinant of watch time involves a combination of the thumbnail, title, content quality, and engagement throughout the video. As one of the interviewees mentioned, having access to a full spread of analytics for each thumbnail under test would provide more meaningful insights. Unfortunately, as users of the tool, we only see the limited data that KZbin provides. Therefore, while the tool might be useful for testing significant changes like "face vs. no face" or different color schemes, it still lacks the contextual data such as who saw the thumbnail and when it was shown. A more effective tool would involve a split test of both the title and thumbnail together. Even better would be the ability to split test different parts of the video itself, such as the intro or the hook, similar to how dynamic ad elements are tested. This approach allows advertisers to switch up elements dynamically and select a winner based on performance, but it's a paid feature, as you're spending money to run the ads, also its just image and text based, not dynamically replacing parts of a video... For us as KZbin content creators using this free tool, there's a need for a more comprehensive testing mechanism that considers multiple variables, not just the thumbnail in isolation. This would provide a more accurate reflection of what drives viewer engagement and watch time.

[@alex-on-words]

Do you think it's related to what they're measuring? I might be wrong, but I'm not convinced that a thumbnail has a significant effect on watch time. Perhaps the law of averages should still give us equal results in control tests... I don't know. 🤔 I just wish we could see the CTR results to give us some comparison. If the CTR didn't balance out, it would surely prove it's broken.

[@ChannelMakers]

That's a really interesting point. Maybe a huge sample size is needed before it would level out. Hopefully they start showing us CTR soon! They've talked about updating it.

[@KennySparksYT]

I’m actually going to keep using the test but not for the data. Different people respond to different types of thumbnails and I like the idea of the randomized thumbnail per user. I typically make 2-3 thumbnails anyway so this will take away the stress of choosing. I wish you could just permanently rotate them instead of it just being 2 weeks.

[@EpicDIZ]

Just ran my first test sand I got a 33.3 percent on every thumbnail. I'll try it again on my next video but it seems like they may have listened and improved.

[@ChannelMakers]

That would be amazing! You'll have to keep us updated! :)

[@ParkMetalDetecting]

I just want to see the click through rate... That's it... Pretty simple. And shouldn't it be the A/B/C Thumbnail Tester? Also, why does it not show a running talley on views... I really hate the wait and see not enough data!

[@BoardGameOfficer]

Thank you!! The first video that I used this on was the worst performing video on my entire channel ever!

[@ChannelMakers]

Yikes. Glad this vid was helpful!

[@RodStersTV]

Thank you so much for such an informative video! What about external companies like Tubebuddy, does that mean that their A/B test feature is also non-functionnal, or could it be a good alternative ?

[@ChannelMakers]

It could be a good alternative especially because they show you more data points on their analytics. We plan to run some control tests with their software soon.

[@your_spirit_guide]

I stopped creating 2 thumbnails and went back to creating 1 thumbnail for each new video

[@georgianarundgren2385]

There is another factor about thumbnails that isn't always considered. There are many times where the thumbnail just doesn't matter. As a viewer, if I see "Channel Makers" put out a new video, I don't even care about the thumbnail. I'm going to watch. I have a whole bunch of favorites. This goes for smaller channels I watch regularly, too. Pretty sure I'm not the only one. I'm sure it matters for trying to grab new viewers.

[@weeklyfascination]

In the past, I didn't know how to make thumbnails, so as I got better, I replaced some old ones and got more views. So I was excited to see A/B thumbnail testing rolled out. I've tried a few A/B tests on my videos using the new feature. Unfortunately, I used similar thumbnails and didn't see a big difference. What I find puzzling is using the watch time metric to find the winner. Wouldn't CTR make more sense?

[@wolflover789]

I think you guys are finally getting some good attention and getting your groove on. I've been watching for a long while and I've kinda been on your journey with you. Congratulations.

[@ChannelMakers]

Thanks so much for sticking with us. Really glad to have you here.

[@Stevonjbiz]

I was NOT expecting that from this video. Just like KZbin saying "viewers are not watching this video as long" well, it's a 2 minute clip 😂. Of course they're not watching as long even though it's showing %87 watchtime. Reminds me that everything is an experiment 💯👍. Thanks for sharing.

[@ninjaproofreader8289]

Great. waiting for it for nearly a year. Got it this week. Now it's useless. Google strikes again. 😂

[@ChannelMakers]

lol pretty much

[@HayleyHadeez]

So as humans we tend to see patterns that aren't always there. I add this preface just because I may be completely wrong here and experiencing this pattern delusion for myself, but since I started using T&C my Channel has TANKED. Now I'm not going to say it is because of this. It could be a thousand other factors and I'm not making excuses if it's a case that people are bored with my content or I've killed my Channel by uploading too much/not enough/whatever. BUT I'm going to stop using it for my upcoming videos just to see if it changes anything. And it may or may not and it may or may not be due to my content and my audience... Who knows? But it wasn't a feature I was massively excited about anyway and used it because it was The Next Big Thing and I wanted to see what all the fuss was about.

[@ChannelMakers]

Love this. Good for you for being self aware enough to be weary of your own biases. But it sounds like you've also weighed the cost of using it and it isn't worth the risk that the tool IS the cause of the decline. Anything could happen with the tool at this point so I think it's really smart to give it a rest for now if you're worried.

[@HayleyHadeez]

@@ChannelMakers You're never going to achieve anything if you're unable to analyze your own actions and then accept responsibility, so it's important to recognize I could be completely wrong here. But my next upload will be using a single Thumbnail and then maybe I'll at least have something to start analyzing against the T&C option...

[@ChannelMakers]

@@HayleyHadeez So true! Let us know how it goes

[@HayleyHadeez]

@@ChannelMakers will do! Although my Channel is being so weird right now it may prove impossible. One video got 600 views then stopped. Today a video got 10 views. It's all over the place right now. Who knows what I've done. 😆

[@HayleyHadeez]

@@ChannelMakers actually now I think about it I probably just confused the hell out of KZbin by doing some shorts and live streams for the first time... 😆

[@SuburbanPhilosophy]

LoL, I love to share this! I only regret i don;t see the faces of people who was so excited about this "feature" watching it! 🤣🤣🤣🤣

[@ChannelMakers]

Hahah that's cold 😅

[@belyiam]

HAHAHA me!!! 😅until I tested it! 💔

[@TrebleKnight]

That's a crying shame. I was looking forward to using this function. I guess will have to see how to tool grows overtime.

[@ChannelMakers]

Sorry to be the bearer of bad news 😔

[@stressfrei_leben]

Thanks for this video.... Frustrating information...I'm a small channel in Germany and don't have the tool yet AND I was really waiting for it since I'm always insecure with the thumbnails... I hope they'll fix it.

[@BStride]

Uncle raff, yeah click through rate based would be a lot more efficient. Great video. This is concerning

[@sanmedina]

I got no idea why they display watch time instead of click through rate, which is literally the thing we measure thumbnails with

[@ChannelMakers]

I hear ya. They've said they're afraid of people leaning into clickbaity thumbnails. But both metrics would be really handy. It would be nice to get more data and be trusted to use it responsibly.

[@YourYouTubeCoach.]

THIS. CTR is the true meaurement on whether or not a thumbnail is effective

[@Memodeth]

I mean why not just give us CTR? I can see that there might be some sort of connection between thumbnail and watch time, but there is definitely a stronger connection between thumbnail and CTR.

[@ChannelMakers]

That would be helpful wouldn't it?

[@philscomputerlab]

It should test clicks vs imoressions...

[@juraspl93]

I think small channel with 0 views, and dead video, adding at first second thumbnail and later third, they can give video second or third chance

[@CrowContinuum]

I don't think using the same thumbnail in an A/B test is going to show a 50/50 split even if the numbers reflect just which thumbnail was shown to people. An A/B test doesn't not automatically split 50/50. It just randomly chooses one at the time a choice needs to be made.

[@ChannelMakers]

KZbin claims they won't choose a winner if the difference isn't statistically significant, yet most of our results have returned with a "winner" label even when they're control tests. They also have "inconclusive" and "preferred labels they can use if there isn't enough data. At this point it appears they aren't using those often enough.

[@TheLawnFellas]

Great information here for sure, but don’t think it will help small channels because KZbin is going to definitely recommend your video more for each thumbnail to get some sort of views!! Thoughts?? Maybe a video on this!! Thanks for the knowledge 💪🏼

[@AIKnowledge2Go]

This is far worse than I initially feared. I share new video releases with my Patrons via email, and each email contains the same thumbnail. Therefore, these numbers must be incorrect. Additionally, how does a thumbnail determine how long I will watch a video? They need to establish a coefficient with the CTR for each thumbnail somehow (I'm not a math expert).

[@FronzSchornack]

Thank you so much Channel Makers for sharing your insights on A/B thumbnails. Personnally for me , I like it. They pick pretty good thumbnails for me others I really don't know, never blurry or anything like that. I feel that both are important but hey!!! Their not that bad I.M.O. Thanks again.

[@ChannelMakers]

No problem! Do you mean you like the test and compare feature or the auto generate feature for thumbnails?

[@FronzSchornack]

@@ChannelMakers Yes but Julia has a very good point, Thumb nails are very important and many creaters do a great job at it. I believe that Y/T just wanted to give out options like this and shorts,"which I have'nt used yet" but might some day to my advantage. Thanks for your reply.

[@BlackAndBlueGarage]

Not surprised at all. I ask human beings what thumbnails are better. 😌

[@ScottJohnson2]

I totally get and agree with their decision not to just turn this into a CTR optimization tool that would almost certainly trash up the platform. But it seems the relationship to watch time is approximately nonexistent. It seems right now like someone who doesn't click on the video simply isn't part of the stats... If they want this to work without giving people access to all the stats (so they can just optimize CTR) they will have to build an algorithm that incorporates more metrics. (Maybe they already have without saying it?) CTR modulated by watch time would make a lot of sense, for example-downweight a view under 20 seconds to be the same as a non-click, etc. There aught to be enough knobs to tune there for this to function perfectly well.

[@anniegirl7152]

This feature encouraged me to experiment more with thumbnails and in every case I’ve done it with existing videos views have increased without sabotaging watch time. So thank you tube

[@the1andonly]

Aren't there other factors at play that would make it possible for the same thumbnail to get different results? I don't have time to make three thumbnails for each video, so I focus on making one good one.

[@ChannelMakers]

Yes I think there are a lot of things that could cause it. And If more than half of the time, we get a large spread with the same thumbnail, we can't be confident putting any weight to the results of a test with different thumbnails and a large spread.

[@ApocalypticRetrospective]

I’ve been doing only 2 thumbnails and each thumbnails are completely different. I’ve had good results and a clear winner each time. (On this channel and another one. 2 different niches). I think the 3 options is a case of “too many cooks” so too many options and everything gets split into the same percentage. I honestly don’t have time to create 3 thumbnails per video. Maybe it’s a case of everyone overthinking this ??

[@ChannelMakers]

Yeah, two is probably better. Although we've also tried this with two and have seen similar results. How much of a spread do you have between your loser and clear winner? For us it seems we need over a 20% spread before we can even start to give it any weight.

[@ApocalypticRetrospective]

@@ChannelMakers My clear winner was between 60-70%.

[@ChannelMakers]

@@ApocalypticRetrospective Oh nice! Yeah I'd say that is enough to know that it's preferable to the other, even with how flawed this tool is.

[@nopeoplenoproblems]

Thank you, I just put for test 3 videos yesterday, will see what will happen.

[@ChannelMakers]

Thanks for testing with us! Keep us posted

[@nopeoplenoproblems]

Will do, 13 more days until 😂 My channel do not get bunch of views, kinda brand new, so maybe nothing will happen in my case. Thank you for replying.

[@-bibliolab]

But what if I consistently see that one specific style of thumbnail is performing drastically worse? Does that still mean it's wrong? For me it was quite obvious and helpful. I hate to break your hype, but it seem to work for my purposes. It's still small sample size, but when I see 13/13 total fails of one thumbnail style, it's enough for me.

[@ChannelMakers]

Yeah, if you did 13/13 tests with two very distinct thumbnail styles and kept the thumbnail styles consistent I'd say that's a good sample size for you to know if it's working on your channel. I think it's important to run your own control, or true A/B tests and base your decisions off of what you learn for your channel specifically.

[@-bibliolab]

@@ChannelMakers That's my point. It's not that it don't work at all, rather that it's not what the people expected it to be. I'm keep running it to test some ideas and styles. It definitely works for that in my niche, but I definitely not gonna press my approach to everybody, as all channels and niches are different.

[@MattCommand1]

You guys did a major service for the YT creator community! I've only had to the tool for a week but haven't used it yet. This is probably one of the most impactful videos you guys made so far. As a solo creator, I already have a lot on my plate. I don't have the bandwidth to do everything I want to do so I constantly have to prioritize and be selective. Beta testing this tool is definitely not happening for me.

[@Lauren-xr3sz]

I used this on one video and my click through rate is SIGNIFICANTLY lower than any of my other videos. Granted, I used different thumbnails for each options. Literally a 0.5% click through rate and my average is like 4% (I'm new).

[@Money-Fast-Plan-a]

Your video is incredibly engaging! -- "Success thrives on relentless commitment."

[@CameTo]

You 2 are both great when you do your videos and content discussion in a freestyle back N forth way like this. It has more flow and i think some of your earlier stuff taking over from Nate was too heavily edited, which looked fancy but lost point of the video. This today was a great balance so thanks ❤

[@ChannelMakers]

Agreed, it took us a bit to get in the groove. Glad you're liking the content these days. Thank you so much for the support! Very kind of you. -Julia

[@MatthewGenser]

Here is the big question.... Have you or others paid for A/B Testing from other services and tested them against this? That would be the most interesting way to learn more or compare it to

[@ChannelMakers]

Yep we’re working on it now! The results so far are… interesting. More details to come once we have more tests done.

[@UnfilteredThroughOurLens]

Did you do test with just a single thumbnail to see what the results are on a separate channel? I think the only way to truly have a completely neutral test with no channel bias are 2 channels doing the exact same thing. Just a thought

[@VladimirKostek]

Hi Channel Makers Team, I just posted a video about my YT thumbnail tool being broken that you guys might find interesting. It appears to be a unique issue

[@MartinKPettersson]

This is some interesting information. Great work, thank you for sharing this!

[@ChannelMakers]

Glad you enjoyed it!

[@makeorbreakshop]

I don't know if it's an issue of the tool just not working, but maybe we need more of the data behind it. I wouldn't think KZbin isn't intentionally running an accurate test, showing 50% or 33% of the thumbnails to a potential audience and then measuring watch time. It could be more of an issue of KZbin showing the thumbnails to different portions of your audience. You mentioned at VidSummit one of the engineers showing a print out of two different columns of data under each thumbnail. That would be great to compare not only watch time but also CTR, impressions, sub/unsubed audience, and being able to see all of those at least daily to get a better idea of what's going on. The fact that your seeing a pretty high delta in identical thumbnails would mean there is a difference somewhere else in the data. I just wish we could see more than an overall percentage for watch time between the two. But KZbin could just be messing with us...wouldn't be the first time.

[@ChannelMakers]

Yes, they've said they show them to different groups so I think it's less random than would be ideal. More data would definitely solve a lot of this problem. I think the folks at KZbin are doing their best, I would be shocked it if were intentional.

[@ElectricRob]

I mean, unless the 3 thumbnails get put in front of all the same people, the results are just going to be more based upon "who" the thumbnail gets seen by, or even what "other" creators' thumbnails it is being presented alongside. Otherwise, too many random variables all lumped into one big mess!

[@cdarklock]

A/B testing is inappropriate for small or dynamic audiences. If you don't have a six-figure subscriber base, it's worthless, and even if you do, you can't run a test on your subs and then expect the results to apply when you expand to a general audience. You know, the way KZbin is DESIGNED to operate? Starting with the notify squad and your subs, then expanding to complete strangers? Most people don't even understand what this tool is FOR. They just know it's a thing professionals use, so they want to have it too.

[@JoelFoner]

Could this be exposing the randomness in their distribution algorithms, moreso than the thumbnails you're trying to test?

[@ChannelMakers]

YUP. In fact, KZbin came back and told us that they're showing the thumbnails to different audiences after we submitted the feedback. So they're doing their best, but without a random test we really can't trust that one thumbnail is any better than another.

[@JoelFoner]

@@ChannelMakers that's past absurd - a multi-variate test with a purposefully different base population? That's not just broken - it's a fully invalid test design. Wow.

[@the_eerie_faerie_tales]

do the people you interviewed have channels? if so can you link please? 😊

[@ChannelMakers]

Yes! I just linked them in the description!

[@the_eerie_faerie_tales]

@@ChannelMakers cool thanks!

[@Jogwheel]

For 18 years, in fact! 🙂

[@the_eerie_faerie_tales]

@@Jogwheel just sub'd!

[@ChannelMakers]

@@Jogwheel That's a loooonnnnggg time! 😅 I'm thinking at some point we need to hear a more extended version of your KZbin story. :)

[@kirkmangaming]

0:31 You pointed to a video that's not there. At the end you did too. 😅 2:08 Don't those numbers fall within a reasonable margin of error? I know you talked about it later, but I just don't feel like there was a real stand-out winner there in terms of numbers. I have done A/B tests on all my long-form thumbnails, and the results have been eye-opening to me. My thumbnails aren't perfect by any means, but they are better with that insight. It's not placebo, it's real data. I would love more analytics to it like the blueprint mentioned, but even just having Watch Time is helping a ton.

[@ChannelMakers]

Sorry, pointed in the wrong direction. It's on the top right at the beginning. At the end, we ended up adding that extra clip and you can't put the end screen watch next any earlier than the very end of the video. I guess it depends on what you consider a reasonable margin of error. As high as 20% seems pretty high to me. And it worries me that they put the "Winner" label on tests (even many of our control tests) which they claim means the data is statistically significant. Clearly that isn't the case. Glad to hear it's working foryou!

[@VicBarry]

REALLY interesting video! Thanks so much for digging in here! I've used it in my last two long form videos (was really excited with it) and both videos have been my worst performing videos in years. The CTRs were anywhere between 1% and 1.9% where, consistently my average across the channel is between 7%-10% CTR. Sure, that may be an audience thing for one reason or another, but I found that the videos were not being served up to my audience who had subscribed, they seem to take forever to get even half the impressions I usually would get in the first hour and even when I searched for the exact title AND my channel name, it didn't show up either. While I appreciate it's audience and NOT algortihm, I just find it highly coincidental that my 2 worst performing videos in years just happened to be using this feature. I've spoke to some bigger creators who also had similar results. Moving forward for now, I'll do what I always did, if the thumbnail isn't performing I'll swap it out manually.

[@ChannelMakers]

That's really concerning. It does sound like there's more going on than your audience just not liking the new videos. It's always risky to test out new features. I'm going to go back through all of our videos in the past 7 months and see if I can find any correlation in CTR and using the tester

[@VicBarry]

@@ChannelMakers awesome. Happy to dive in some more if you need it / wanna reach it etc

[@TheBradfords]

Already had this idea . Never even used it. I never like the thumbnails they make either on the app

[@ScaleAutoGarage]

Just to play a little devils advocate here but what if the tester is actually smart enough to know that you are uploading the same image for a/b/c and then it doesn’t push the actual difference? I’m no engineer or anything and I could be way off but since I am seeing a bigger spread in the thumbs that are the same tells me that the tester is not really putting an effort into even bothering to test further.

[@ChannelMakers]

Interesting theory! 🤔

[@ScaleAutoGarage]

@@ChannelMakers I am not sure how much testing you have done but you say that you have done extensive testing over the last 8 months, I am curious if you have tried a test using one thumb that is about the video you are posting and a second thumb that is just pure clickbait and has nothing to do with the actual video? I think this would give you some pretty good data on how the thumb checker actually works. For example, my channel builds model cars, if I did one thumb as my regular thumb and then second one a pic of Mr Beast with the text 'Mr Beast model car building challenge' then it is likely I would get a lot more clicks than normal especially if I copy a Mr Best Style of thumb. Since the tester is supposed to also check watch times and not just CTR, then the checker SHOULD come up with the Mr Beast thumb being a bum thumb because the watch time would be terrible, even if the CTR was way higher. Have you had the opportunity to do a test like this with your channel? I am not sure but I feel like a test like this could tank my channel as it is a smaller channel trying something to brazen but a channel that has more of a following may take less of a hit. Thoughts?

[@TrailTraveler]

Compare it to TubeBuddy's A/B test feature.

[@TinaHBeautifulJourney]

So disappointing. I was looking forward to this and JUST got it and used it with my latest video. Was so hopeful that as a small channel, this would help demystify some of the unknowns on creating good video packaging. I think I'll run a control test on my next video to see how it performs with a small channel. Thank you for sharing though, as it will keep me from wasting too much time coming up with alternative thumbnails. I'll put that time and effort into coming up with one good one.

[@DataRae-AIEngineer]

Ok I've been thinking about this a lot since your last video, and I obviously don't have the code they used to make this, but I don't think your results are definitive BECAUSE putting the same thumbnail in there might just be confusing the algorithm they're using. That doesn't mean it's broken for thumbnails that are very different, like how they intended it to be used. It ONLY means it's broken for thumbnails that are exactly the same. Data science and other sciences are different because of the computer code aspect. Sometimes if algorithms are used for a purpose different than what they were built for, they give unpredictable results. Since you say KZbin told you not to run the same thumbnail, it implies they didn't build the algorithm to be able to deal with that. The unpredictability might just be an artifact of that. However, this is a good video, and I hope KZbin responds because while your results don't necessarily mean the algorithm is broken, it also doesn't mean the algorithm works as they intended.

[@TorqueTestChannel]

We did an A/A/A test this week and got a 32.4 - 34.2% spread, the closest we've ever seen a thumbnail test using the feature. Not sure if data from the Beta version is as useful as what's in use now

[@ChannelMakers]

Oh very interesting, and I assume it didn't pick a clear winner right? We've had two A/A/A tests on two separate channels end this week with definitive "winner" labels and a moderate spread, so it's still happening currently. But it might not be as frequent as it was in the past! Might depend on the channel too. Thanks for trying it and letting us know your results! -Julia

[@DrDreadMarley]

I don’t get why they choose watch time for a thumbnail feature like how does that make sense it should be CTR only as watch time is based on the quality of the video 🤦‍♂️

[@ChannelMakers]

Their argument was that people would make click-bait thumbnails and believe those were better because of a high CTR despite shorter AVD. But more data would have been nice wouldn't it?

[@AlexiGiovani]

Okay without looking at your exact numbers since they flashed on screen. I can understand why the variance is so great. The net effect of the results of the controlled test is that regular users are just seeing the same thumbnail. As a viewer I’ll sometimes see a thumbnail a few times before I decide to click. Now I’m not sure if I continued this behaviour would I see a different thumbnail if the thumbs were different and not controlled. So your uptake on which thumbnail is best will be affected by users viewing patterns. What I’m trying to say is: as a viewer I see thumbnail A and I give it a miss. I see thumbnail B, I think (I’ve already seen that. Maybe I’ll watch it another time). Then third time I get thumbnail C (which is STILL the same to me as a viewer) and finally decide to opt in for it. You’ll get random results and depending on how KZbin distributes the thumbnail options. Does it rotate. Is it sticky to a user? Is it sticky to a niche. That’s all black box.

[@ChannelMakers]

KZbin claims each viewer will ever only be shown one thumbnail.

[@DarekWeber]

Guys, have you considered the time of day/week that KZbin is showing each thumbnail? Maybe the ‘winning’ thumbnail (although identical) just happened to be shown more during times when people are more likely to watch the whole video, like commute time/lunch time?

[@FelipeSeabra1]

Thumbs is showing for impression, ALL thumbs is test in same time of day, not in diferent times

[@DarekWeber]

@@FelipeSeabra1 that can’t be 100% true. Consider this, I clicked on a VidIQ video on a Monday morning with Thumbnail A, got busy and couldn’t pay attention to it so I clicked off. KZbin served me that same video again Tuesday Afternoon, this time with Thumbnail B, and this time I was able to watch the whole thing. The data should show that Thumbnail A got 20% of my watch time and thumbnail B got 100% of my watch time, skewing the results. Take that same scenario but this time imagine Thumbnail A and B are identical…

[@HISTORYUN]

Fantastic breakdown and a massive time saver 💎

[@MinimalistCarnivore]

Yup needs to be click through and watch time after click through somehow and as mentioned also the algo of showing your vids :)

[@ryanrockers]

Disappointing to hear, thanks for the heads up.

[@ChannelMakers]

Sorry to be the bearer of bad news!

[@CoolWatchesOnly]

I am super new but let's say they show the same thumbnail to 3 completely different audiences and they reacted completely different, how would that show the tool is flawed? how can you predict what will be the response of an audience to certain thumbnail? I am not saying you are wrong, I just would love to understand this better

[@ChannelMakers]

I'm glad you want to understand this in more depth! We were hesitant to go into TOO much detail in the video because attention spans are short on this platform haha. But I'm so excited to dive into it further with you. 😊 So yeah, you make a great point, there is a lot they can't control for, and I'm sure they're doing their best. I also believe the analytics (currently just watch time share percentage for each thumbnail) they're showing us are the real, accurate numbers of how much watch time share each thumbnail ended up getting. The problem is that at some point it becomes meaningless. If we run a control test on Monday which shows a spread of 23%, 31% and 46%, we of course can't conclude that the 46% thumbnail is more appealing because they're all identical. So by that same token, if we were to run a real test on Wednesday with 3 different thumbnails and saw a similar spread, let's say 25%, 30%, 45%, we can't reliably conclude that the 45% thumbnail is more visually appealing/intriguing. KZbin has said to use your own judgement and that the tool isn't perfect, which I understand. But my biggest complaint with the tool right now is the labels they give to the results which are either "Winner" "Preferred" or "Inconclusive". They have stated that "Winner" means their was statistically significant data showing one thumbnail outperformed the others, "Preferred" means they think one may be better, but the data is not statistically significant. "Inconclusive" is pretty self-explanatory. I appreciate that they have all 3 labels, and I think that's the best way to do it. The problem I'm seeing is that we frequently (like more than half of the time) get a "Winner" label smacked on our control tests. But KZbin claims that the data needs to be statistically significant for them to give it the winner label. So that should almost NEVER happen. It's hard to say what they have deemed statistical significance. the confidence interval can vary. But in the broadest terms, "statistical significance" means "unlikely to happen by chance" - and we're seeing quite the opposite. So I think the bar should be set a lot higher on how large the impressions, watch time, and spread between results should have to be before they deem a video a "winner". I believe the reason they haven't done that is because if they did, small creators would never see a winner label, and that wouldn't feel very fun for them and they'd get complaints. But that's just speculation of course. What are your thoughts? -Julia

[@CoolWatchesOnly]

@@ChannelMakers Thanks for taking the time! I understand and agree with your point better. Perhaps, for now, it would be best to select three thumbnails that you personally consider winners or very good. By using the tool, not to rely on its results, but to increase the chances of people liking one of the three. My theory is that even testing, using a bad thumbnail will negatively impact the video's performance, this is all speculation on my part as I haven't actually tried it. In my experience with Fb ads, having multiple versions of a campaign can make a difference. Of course, you wouldn't intentionally promote a bad video just to waste money; all versions should be good. However, when it comes to reach and impact, I still have a lot to learn. I only started last month, and despite nearly 200k impressions, I've only received abou 13k views. So, in case we disagree, I acknowledge I'm in the wrong. 😆

[@EdwinSchaap]

Thanks for sharing! I was disappointed I still didn't have the future but it made me think extra hard to come up with a good thumbnail and I'm super happy to say I just posted my best video so far with best ctr for me 6.7% ! 🎉 the mask😂

[@EdwinSchaap]

It seems to go viral! 5k views already, I never have that 🎉🤗

[@Alexandria_S1222]

Well that stinks!

[@DoggsLife2019]

I agree and disagree at the same time But they tried to fix shorts by adding the related video link to help push views to your long form. I think KZbin will do something like that for this. Doggs channel is so small we will not get any analytics till we get bigger

[@SupremeSkeptic]

I think it depends on the number of views for each thumbnail. If one thumbnail has 1 view, another thumbnail has 2 views and 3rd thumbnail has 3 views. Then the first one would have gotten 16.7%, second thumbnail would get 33.3% and third thumbnail 50%. Big % differences but not significantly different in the number of views. They higher the number of views, the more accurate the % data. I am not sure if there's a minimum number of views before they do the A/B test results. But they should. Let's hope they won't refuse the same thumbnails in the future because this type of videos reveal how good or badly their A/B feature works, if there are already a huge sample size.

[@ChannelMakers]

I agree, the view and impression minimums are really important. Smaller sample sizes will always be less accurate.

[@DoggsLife2019]

OK you guys have caused a domestic 😅 You really need to look at itv this way. KZbin should put the same video with 3 thumbs to the same group. That is the only way this will work. But that would take up a lot of storage and a lot of work. Each 3rd gets all 3 thumbs and the same video. Hope you understand what I am trying to explain

[@ChannelMakers]

Yep I think that would work! For some reason it appears they can't.

[@SpazzyjonesGaming]

I'm not convinced it doesn't work. The examples used in your video had less than a couple thousand views with the one exception having 45k but even that video is 8 months old (were your control tests done when the video launched or more recently on a stagnant video?) I reckon the function improves in quality the larger the channel with more impressions garnering higher statistical significance. In short, I think what you are saying may be correct for smaller channels but not so much for larger channels getting tens or even hundreds of thousand video impressions. Just my thoughts! :)

[@ChannelMakers]

I agree with you, we need more data. More channels. Big channels. small channels. Right now we are limited to the tests we've run and some additional tests run on other channels. It does seem that the test should be MORE accurate on larger channels, but even on the larger channels we ran tests on, the spread on the control (A/A) tests were concerning. Hopefully it will work for most channels, but like you, I doubt if it will ever work well for small channels since the data gets better and better the more impressions and viewers you have. PS Thank you for being a critical thinker. We need more critical thinkers :)

[@SpazzyjonesGaming]

@@ChannelMakers Appreciate that! Same goes for you. I quite honestly (naively) assumed the feature would work and when I saw your video title I HAD to click because I couldn't believe it wouldn't work and never thought to try and A/A test but now I think I just might to see for myself.

[@ChannelMakers]

@@SpazzyjonesGaming That's great. We would way rather everyone try for themselves than take what we say as the absolute truth! If you find anything interesting let us know! :)

[@SpazzyjonesGaming]

@@ChannelMakers will do! thanks for the video!

[@leosthrivwithautism]

Ehhhh, my opinion is that more research is needed. And that’s only because we don’t have the code or the inner workings of what’s happening behind the scenes. This is good data. But again, do we know if the the tool is offering the thumbnails to the same people or randomized groups? Same audience or are they testing in different audience groups. Because for example if you take the same thumbnail and you test it between two different kinds of viewers one may like it but the other won’t. So of course there is a chance that the same thumbnail may come back with different results. And that’s the part we don’t know in my opinion. I’ll give you an example. I tried it on a video of mine. It found a winner. And when I switched the thumbnail to the winner the video went from being stuck to getting views. So in my experience things have been positive. And it’s helped out some videos i did tests on. Again, I’m not saying channel makers is wrong. All I’m saying is that for me at least I need more data. I need more understanding before I can agree or disagree with anything. 👍

[@ChannelMakers]

Yeah good questions. After we submitted our feedback to KZbin they actually told us it is getting served to different groups. So it isn't broken as in giving us false statistics, but broken in the way that because it isn't going out completely randomly it can't be relied upon to know which of our thumbnails is the best.

[@leosthrivwithautism]

@@ChannelMakers Very interesting! Keep up the good work! Would love to see the follow up as you all learn more! 👍

[@ChannelMakers]

@@leosthrivwithautism Will do!

[@U_N_Owen]

Wonder what happens if you yoink Mr. Beast’s thumbnails.

[@ChannelMakers]

please elaborate..... 😁

[@ViewportPlaythrough]

dev here(not yt dev, but a prog dev) well sure.. thats why its not rolled out to everyone yet, its called beta testing for a reason. after that initial beta test, they would have to roll out to bigger sample rate to get more data. its just how feature testing works. at some point you have to release a feature to the wild to see how it goes in real life scenarios and sometimes you have to put it up as long as needed to get measurable data. formula wise, the only reasonable stat they could measure from it is click through rate. thats what static thumnbails are for. they can not measure watch time on it cause its not what thumbnails are for. thats simply unfair to expect that from a tool thats specifically meant to measure how different thumbnails are effective. running the same thumbnails on it wouldnt be beneficial to test the feature and it is kind of expected that the results wouldnt really be useful. its the devs fault for not anticipating that users wouldnt know how to use the feature on how they want it used, and not clarifying what users should expect out from it. one thing that should be added is to add an underlying function that checks if the thumbnails are different enough from each other that it would make a difference on the testing. if not, simply not accept that thumbnail. at this point they have to add a db to check audiences who have clicked the thumbnail so they could stop showing the video after. then check on audiences who have passed by the thumbnail by a threshold count but did not clicked on it, run the other thumbnails and get similar data on it till end of loop. but then that doesnt account on peoples preferences, its more of a first come first served approach. so they need to add a priority list on the users end. ie, thumbnail A is the one that the uploader trusts the most that would work the best while thumbnail C is his last choice if thumbnails A and B didnt work. this kind of feature also depends a lot on an individual channel's usage of the said feature. if you use it once or twice or even just 10 times, you really cant expect it to work properly. what more using the same thumbnail to check it. its like doing the same thing over and over and expecting different results from it. whats ironic(and contrarian to what i said above) is on its current implementation, you have to force feed it the same thumbnails till you think the data is enough to get you what you want. ie 1) use the same thumbnails 3-5 videos in a row. its to force feed the algo the data it needs with what you want from it 2) on the 4th/5th/6th time use 2 different thumbnails. 3) repeat the steps till it correctly gives you your desired data 4) use 3 thumbnails 5) repeat step3 with 3 thumbnails its honestly like training an ai model. you have to train it numerous times first before it could give you the desired results whats tricky is once they added the function to not accept similar-enough thumbnails. it would slow down the data learning process cause you cant force feed it anymore. but then, it would also teach users on how the feature should be used, and if the feature is used properly then proper data could be collected. but then theres also the question of, are they using randoms currently/previously? cause if they are, i cant see any formula that can use those random numbers. that would work properly.. if they use randoms at any point of it, their stats would really just come out as randoms.. thus, if they are, prior data should be labeled as inaccurate.. it could still be used, but just not accurate enough...

[@CameTo]

Unless you have a very small sample size (let's say less than 2k views) i don't see how this Can't NOT work - deliberate double negative. So i upload a video and thumbnail A gets CTR of 15pc and B gets CTR of 8pc. Is it really that complicated to say that A has the better thumbnail? Are we to believe that KZbin is unable to tell the difference? That it requires months of tuning here, bear in mind the data already exists! Or are you suggesting it's trying too hard to be clever with all the metrics, so it's accounting for CTR + MINUTES WATCHED + ENGAGEMENT + SUBS + SUPERTHX etc? There is also another option, which is however the system works with videos/ channels that get promoted and demoted by the algorithm that discuss certain topics that must get pushed or squashed. I always wondered how that worked when it came to stats, never happened to me as i don't make such content, but I've seen occasions where views or comments or likes or dislikes got deleted, as well as subscribers.

[@ChannelMakers]

I get what you're saying, the problem is, we ran test after test of the exact same thumbnail, and the tester claimed one was better than the other.....

[@bradley7]

I would like to hear feedback from this feature from actual large creators like Ryan Trahan, Airrack, Mr.Beast, etc. -- I'm thinking it works different when creators have a larger audience for the testing. I think you guys don't have a large enough audience (in comparison to those massive creators) for it to work as they intended because it was created to work on such a large feedback pool, that's just my assumption, also your control test seems like it would confuse the programming, so it would give you wrong results, what you were doing doesn't make sense in hindsight, because its not a control test if its not going to the same people, which it obviously isn't -- obviously we need these things to work for us smaller creators but yeah, lets hope improvements come from this -- thanks for the work you're doing

[@ChannelMakers]

I would love to see larger creators run these tests too! I agree, it is much more likely to work for them. Maybe KZbin should have only given the tool to creators with 100k baseline viewers on their videos. Or at least enough to know the test group size for each thumbnail was large enough. If that is the answer, it's sad because I think small creators are the ones who would benefit most from the tool.

[@bradley7]

@@ChannelMakers agreed! It’s the smaller channels that will be the future of KZbin. I enjoy the journey of growing our family channel but having useful tools/analytics would definitely make things better. Thank you for the research and impact that you are having in this space it will definitely have impact for the smaller creators and help to have our voices and concerns heard.

[@AgnesMann]

I want KZbin to address this!

[@davejefferynf]

We'll probably just get some form letter telling us to be patient.

[@stresseddude]

I didn’t see a problem yet. It don’t seem to affect views to damage a channel if you still have a split and has to decide which has higher percentage to take over. Am I missing something? You say terrible how it is with other people but I don’t hear it in this video. All I hear is that it’s working but isn’t growing the channel at all. That doesn’t refute that it’s it working.

[@ChannelMakers]

Here's an overview of why we think it's not working (Copied from above): When running experiments, it is common for the test groups to be randomized, which means there will be some variance in each test group. By running the control tests, it allows you to establish a baseline for the variance. Meaning you will know what amount of variance is normal. Hypothetically the variance group by group SHOULD be minimal if the test groups are sufficiently randomized and the sample size for each test group is large enough. In the case of all the control tests we ran, the results showed that the variance between two identical thumbnails was oddly large. (Large in a concerning way) So we tested again and again and again so we could find the average variance. After we had an average variance for the control test we could run actual A/B tests (two unique thumbnails) to see what the variance of the A/B test was for them. And since we know the average variance for the control tests we will know what result for the true A/B test would actually be significant, meaning we would know how different the watch time split would need to be for us to know if one thumbnail was really better than another. It turned out that the variance for the control tests were LARGER than the variance for the A/B tests. That means we can have no confidence in the A/B test results because the variance SHOULD have been larger between 2 totally different thumbnails. There might be a problem with the way the test groups are put together, the test groups might not be randomized well enough, the sample size might not be large enough. There are a lot of things that might be wrong, but the bottom line is, the results we are seeing cannot be trusted.