Does Clear Rate ACTUALLY Matter in Super Mario Maker?

WATCH ME LIVE ► / raysfire
follow me on twitter ► / raysfire
join my subreddit ► / raysfire
join my discord ► / discord
follow me on instagram ► / raysfire
In this video, I do a deep dive into the world of Mario Maker 2's level selection algorithm, with the hopes of figuring out exactly the game chooses which difficulty each level goes into. To do this, I did a series of experiments, as well as did a deep dive into some of the fringe levels in Super Mario Maker 2 that don't really make much sense on the surface.
This video would not be possible without the help of @DSteves, he helped immensely on this project, from giving me ideas for experiments to run, giving me the average clear rate and WR times, to giving me a list of the top 1,000 highest and lowest clear rate levels of each difficulty to help aid the research I did.
Please check him out here: / dsteves
versus ► bit.ly/raysfireVS
trolls ► bit.ly/raysfireTrolled
challenges ► bit.ly/raysfireSMM2Challenges
collabs ► bit.ly/raysfireCollabs
road to s+ ► bit.ly/raysfireSPlus
ALL super mario maker 2 ► bit.ly/raysfireSMM2
➡️ sub 4 more content ► bit.ly/Sub2Rays
👕 buy merch ► store.raysfi.re/
📹 edited by ►
me :)
#raysfire #Highlights #SuperMarioMaker2 #SMM2

[@raysfire]

to see the full, unedited "clear rate" experiment, check it out on my vod channel here: kzbin.info/www/bejne/gIWkYaqhmdFsg5I ✨ also, big shoutouts to dsteves for all of the help with this project! a lot of the data and experiments i used in this video wouldn't have been possible without his help!! check out dsteves' channel here: www.twitch.tv/dsteves 😎

[@Md5offical]

Nice video btw

[@torinoconnor4296]

Great vid

[@PeepPlayzSorta]

nice vid keep up the good work

[@cak5171]

It would have been cool to go back in time and track the average clear percent over time. I have a feeling that since the game released, less people are playing and the people that are still playing are really good at the game. So perhaps the levels got harder to accommodate that crowd thus the average clear percent would go down over time.

[@hisuianzoroark5726]

@@cak5171 Check around 8:59 to see that DSteves cleared K. Wonka's level, too. XD

[@charlottearanea7507]

I think the initial difficulty seeding being higher than expected makes a lot of sense. As the creator, you know all the tricks needed for a level, so it's easier for you than it would be for a new player. Having the initial seed be biased toward higher difficulty is actually pretty smart game design.

[@SirGsusFreak]

Yeah. I always wonder how one of my levels was initially placed after upload since it took me one attempt to clear check it, but found out later that most people who played it struggled and gave up.

[@rbpgamemaster]

That said, I still feel like you could get a better starting estimate by taking into accounts the # of failures before a successful attempt when clear checking it. It would make it so that harder to clear levels tend to be higher difficulty on average.

[@JusteazyGames]

And yet I've made many levels that took me dozens of tries to clear check and yet they have a 5%-10% clear rate. Many creators aren't the best or most skilled and yet make levels that test them to their limits, whereas there are significantly better players out there that are so much better and can beat the level easier. Beyond that, in my experience most levels are straight forward. No tricks, tech, or lore needed. Just standard levels. I think the point you brought up is very cogent in troll levels, little Timmy levels with random hidden blocks, and very hard levels by amazingly skilled players/creators. For the rest of levels, which might be the larger portion, it doesn't necessarily hold true.

[@Syranovara]

Initial clear check just shouldn't affect difficulty at all, actually. Bad game design.

[@LauriHarsila]

@@Syranovara If initial clear check didn't matter at all, it could be that most of the people would never get their levels in the random pool, because nobody could find them in the first place. In SMM2 you need to get at least 1 play from somebody else before you get a new level into the random pool, and as a rookie level creator finding even that first player can be tough. I'm not exactly sure if the first play needs to finish the level. Then again, when writing this it occurred to me that they could just use that first other player's clear rate to recalibrate the difficulty before putting the level into the correct random pool. They might actually do that, this video doesn't really confirm or deny anything related to that.

[@MayroSMM]

So what we've learned is that we need to throw when clear checking automatic levels to help no skip runners.

[@Fowlware]

Comment from one verified user with 9 likes placed over multiple comments of unverified users with so much more likes? Clearly, KZbin is falling apart in terms of ratings of videos.

[@literallylegendary6594]

yo

[@SJrad]

And also throw triple shell jump, mashing, and near-pixel perfect precision levels

[@frank9966]

@@Fowlware fun fact: likes are not the only thing that KZbin looks at to bump a comment.

[@Stefan2010a]

comment placement is decided based on likes and how often the user has a interaction (and maybe subscribers)

[@DSteves]

One of the things we also talked about briefly was if levels were reuploaded exactly the same way if a level would differ in initial difficulty, then after the video was in production I remembered about title screens and how I've seen the same level in easy and normal and sometimes even expert, the spike test now does show that this is the case. All in all the goal is to find some sort of algorithm that combines all the pieces together. While initial difficulty on clear check is one piece of the algorithm it's figuring out a way to combine that, possibly the clear rate, and the ratio of who flagged the level as cleared over who has not, as well as figuring out any other overlooked part of the equation that is the key here to getting a sure fire algorithm going.

[@supersexysadie]

Since the number of tries it takes the level maker comes into play, I wonder how that has affected the categorization of my own uploaded levels. I often quit a clear check and nerf my levels a little (because I don’t have the patience to beat them lol) until I end up beating them on the first or second try after failing to beat earlier versions of them. So I wonder if Nintendo factors in ALL attempts on a saved level, or if the counter starts over when the level maker quits a clear check and changes the level slightly. One of my levels like this has a 4% clear rate, but I ultimately beat its final form in one try (after failing to beat earlier versions of it). So I wonder which endless mode it ended up in initially because of that.

[@supersexysadie]

Thank-you! I’ve been on the lookout for resources that would allow me to see stats on my own levels. My curiosity can now be satiated.

[@hamsterking.8343]

Hey man! It's me, hamstercol from twitch, how are you?

[@Salzui]

I think that the algorithm might ignore any clear attempts of a person after the person beat the level for the first time. Another theory I have is that clears of different persons count differently based on some internal skill level value of a player. Somewhat similar to ELO

[@GGamerLiam]

Hi biggest maker

[@ImSumWhatMad]

I'm very curious if skips also factor into the Nintendo algorithm. For instance: take the puzzle level that was in Expert yet had a 17% clear rate, or the exceedingly long levels in Super Expert. Both would likely have very high skip rates, as many people simply do not like puzzle levels or lengthy levels. Someone may die once in a 2 minute long level and skip to the next one rather than attempt it again. This wouldn't impact the clear rate much, but it may have a bearing on the difficulty level. Just a thought!

[@Whodjathink]

I had this same question but typically a lengthy level already indicates its difficulty. So if we take that into account I surmise they may have a separate yet similar algorithm to deaths that involve skips. Since every skip would be unique to a level, if a player didn't participate in the level there must be a reason why(which I would surmised levels in Expert and Super Expert get skipped over 5/10 to 9/10 times respectively)

[@funnature8679]

If the player never clears the level like exit the software, it won't save like if you never played it, also The clear rate of 199,000 out of 400,000 is only showing every player's attempt to succeed, all the times the characters looses a life it counts as a failure point(meaning that is where the player's character fails there because of enemy, hitbox of icicle or liquid hazard),The shoe icon only shows how many people played it, and also the clear rate is everyone's attempt to reach the goal point there. Never overestimate the clear rate if the player doesn't know what to do, give it a chance if you think you can complete it.

[@Taiwannabe]

I had the same question while watching the twitch vod and I was really hoping someone would have brought it up live in twitch chat. If I had to theorize, skips wouldn't affect difficulty as strongly as deaths but it would make sense to factor them in to a degree. On one hand a skip could mean the player died once or twice and determined it was too difficult and skipped it or came across some difficult tech or some precise jump they knew they couldn't do so they skipped it before dying at all. In those cases the skip would represent the amount of difficulty the player WOULD have had if they had kept playing until they cleared the level. Either a great amount of time in the case of difficult tech without a way to die, or many many deaths in the case of precise jumps. On the other had a skipped level may not be difficult but instead be too long, boring, or super annoying. The player could easily beat it, they just can't be bothered wasting their time on something they aren't enjoying. So these skips would be the opposite; they represent a hypothetical first try clear rather than an amount of deaths or struggle clearing the level. So basically skips represent an unknown variable in the difficulty because we can't know the reason for every individual skip. In a hypothetical world where skips didn't exist and a level had to be played until cleared, a level's clear rate could potentially be drastically different than same real world level. The more I think about it, the more curious I am on just how the algorithm DOES consider skips. Does have a way of predicting the reason why a specific player skipped a specific level? Does it gauge the player's own skill level based every other level they've attempted? Does it consider the amount of spent time in the level before skipping? Does it just give all skips a blanket weight and be done with it? Or did the programmers just say "hecc it" and leave skips out of the algorithim altogether?? 🤔🤔🤔 It's so interesting to think about and we will likely never 100% understand how the algorithim works. Unless someone at Nintendo leakes it out...

[@PtylerBeats]

I think this would be grouped into the how many clear vs plays it has. Not the clear rate, but how many DIFFERENT players actually clear it. If the theory that levels that get cleared more often, even if the clear rate is low, are easier in difficulty, then skipping a level and never going back to clear it likely does increase the difficulty rating

[@raysfire]

all songs used are from the kirby 64 ost: 0:00 - Mountain Stream 1:23 - Pop Star 2:51 - Rock Star 3:36 - Aqua Star 5:15 - Above the Clouds 5:57 - Boss Theme 6:45 - Neo Star 7:54 - Bumper-Crop Bump 9:47 - Ripple Star 10:49 - Shiver Star 12:29 - Factory Inspection 13:09 - Training

[@duckisalwayswatching1192]

Thank you

[@notmagolor]

based

[@leesalee1540]

Would’ve been a sin to leave Kirby 64 out of this 😝

[@wttu2960]

Chad

[@harrybest2041]

The Kirby series has the best soundtrack, with the classic Sonic games being the second best.

[@FuzzyJeffTheory]

What would be interesting is if they factored in player experience. An experienced player dying on a level could indicate harder level. An inexperienced player 1st trying a level could be strong evidence the level is easy

[@raysfire]

That was one of the things we were looking at also, but unfortunately there wasn't an easy way to test that - that's definitely a possibility in factoring in level difficulty though!

[@Bezray]

Probably wouldn't work as I am good at smm1 (so the skills carry over) but i have barely played smm2.

[@drock1331]

Agree with this. Someone who has beaten a lot of Super Expert levels should have a different weight in how a level gets classified from a newbie and how hard the level is for them. The algorithm should be looking at the reputation of WHO beats the level.

[@FuzzyJeffTheory]

@Raysfire If the SMM2 scraped datasets contain the players who cleared each level, a correlational analysis could maybe be performed, but obviously this is well beyond the scope of a Twitch stream :)

[@FuzzyJeffTheory]

@Bezray There would be outliers, but with enough data it would average out. And they could weight players less if they haven’t beaten many levels

[@burntcookie2542]

The thing that the high clear rate super expert levels had in common is that none of them had any way of dying other than the time limit, so many people getting the level and skipping affects it (since skipping without dying doesn't count as an attempt)

[@eivarden]

I also think "skip" effects it too. Lets say ur in normal, and the level LOOKs like a normal level, the player wont skip right away.... they die... but they dont skip... they still think its within their reach. so many deaths, and a clear, or many deaths and a skip. vs 0 to a few deaths, and a skip, which means the player might just be fishing for easier levels for their record. so a low clear rate, but refusal to give up, might mean the level is considered to be in the right difficulty range. (along with likes and dislikes too)

[@PatJamma]

Something to consider: tags could also potentially have an influence. Many of your outlier examples were puzzle levels.

[@pspauloplayz4295]

Thats basically 0% chance. There are so many levels that some are going to have the same tags as other but be completely different difficulties. There have to be some easy levels with the puzzle tag

[@koye4427]

I think the issue with puzzle levels is that people just skip the level if they can't solve the puzzle, lowering its clear rate

[@bdgamer08]

my theory has always been initial clear rate determined by creator attempts, and then final clear rate determined by average attempts:clears per person. If 5 people have a 3:1 clear rate, and one has a 100:1 clear rate, then the average is closer to an easy level

[@scragar]

I figure your deaths to set the difficultly are probably reflective of the deaths per completion in general. Easy levels are ones where people generally win on the first or second try, normal on the third or fourth, and expert more. The reason the clear rates are different is because of people who skip after failing the first time or who retry for world record. Only people who go on to clear the level or who die more than the expected attempts factor into the difficulty. This would explain why some levels that force a death to respawn from a checkpoint appear in easy or normal despite having ~10% clear rates, the levels consistently get 1-2 deaths per successful attempt and almost no-one dies more often(because it'd need timeouts or similar), but a lot of people quit when they realise the level is a net negative to lives.

[@KillForAttention]

This is great Rays, your content is so good. This was super interesting, and something I've always been curious about. It's great to see some of these questions answered!

[@madkirk7431]

And now all I need is to get Mario maker 2! YOU'RE GOING INTO SUPER EXPERT, SINGULAR SPIKE! This could actually be used to create break-levels for the harder difficulties. You know, give 'em a free win.

[@pirateosu9407]

really great video! was interested throughout the whole thing, and loved the methodology used to discover things. keep up the great work :)

[@ramaicatseye]

I got that first ballon level in SE endless too! Also I lost all my lives because I kept having one coin left and the clouds left. Also thanks for this video I’ve noticed too and have been wondering! Sometimes I would get a level with one person who played it, and I don’t know how it Nintendo knew for sure that it was hard. It’s very interesting.

[@chickencoop1387]

Rays this is an amazing video that you clearly spent a lot of time making. I loved the creative style of it! Keep up the amazing work keeping everyone here entertained and informed on everything Mario maker. ❤

[@etac774]

So that's why I could play endless easy and end up on a brand new level that's incredibly difficult, only to find a dev door several attempts later.

[@duckisalwayswatching1192]

The way that all these new videos are amazing. Keep up the good work!

[@Rembrandt333]

I made the "Level 49*", thanks for playing and including this level in your experiment :) I'm so glad you like the level. Regards.

[@Derkbok]

Hey Rays. Loved the content. It's really nice to see someone put so much effort into their content. Keep up the good work Mr Shud :)

[@TiramisuSPD]

I have always wanted someone to make a video just like this, thank you Raysfire!

[@Timberkin]

Holy quality batman, this is so much different than what I remember from watching you a year or two ago. And I'm all for it!

[@DSteves]

Through the power of SCIENCE!

[@SkilledTris]

I’m glad you overcame your pc issues to upload this masterpiece! Love all the content you’ve been putting out

[@silvermalwareai4047]

This was an amazing video, as you were explaining what your experiments were and what they ment I also found my self think "well what if..." "then that means..." and its as if you answered my thoughts! Kinda creepy XD but great minds think alike though. The script was awesome and kept me intrigued and interested the whole way through. I also loved how you showed the stats, really helped me follow along. Was worth the wait, I hope this video gets into the algorithm, it deserves it!

[@lilyacedia]

Spicy video rays! 👀 And awesome editing, loved the graph :D

[@englishmuffinpizzas]

The difficulty rating of chess puzzles can be calculated using a modified form of ELO from the players who succeed or fail at it. It would make sense for Mario Maker to use a similar system

[@bluegemini3265]

These new vids have been really great, you deserve way more attention man

[@mizuotoki]

Thank goodness for the sub! I forgot about Quiz day! Awesome Vid Mr. Shud! It's really interesting that the creator in a sense can partially decide where their level will show up!

[@faseel4392]

Very insightful into the algorithm, thank you. I would like to clarify the conclusions we’ve drawn though- the only thing we’ve proven from our own experiments is that creator clear check attempts noticeably affects the algorithm’s placement. The data science studies were useful, noticing trends of lower clear rates and higher world records tending toward higher difficulty placements; however, we cannot say that the algorithm necessarily checks for these things. In order to show causation we would have to do experiments similar to the spike test but with multiple other players/accounts, keeping the creator clear checks constant.

[@HydraWerewolf]

12:00 you can also try to match the WR time for easy,medium,expert,super expert Like wait 120s to finish the lvl for super expert. Idk tho

[@ditto5041]

I’ve always wondered this. Such a good video idea and well executed

[@Nunya644]

Wow this was an amazing video! I think I have to throw when doing clear checks on my kaizo's I build. I practice them so much in the editor, that I clear check them in a couple of tries. Then they get put into categories where people won't possibly like or be able to finish them. They're dead on arrival.

[@Herbu21]

Woah very cool experiment! Love these videos, always good content and high quality!!

[@Waitwha]

raysfire's content is top notch. you should have way more subs! keep up the good work

[@bnb-gn4uo]

such a high quality vid earned my sub good stuff

[@EnderrBoi]

Hello, I am Mr. Fusion Man. There are no fusions in this video. I hope a versus challenge comes soon or I'm gonna be out of a job.

[@torinoconnor4296]

Lol

[@rorycannon7295]

Idk the twister and galoomba at 0:15 - 0:19 seem pretty fused to me

[@EnderrBoi]

@@rorycannon7295 sadly, the twister is closer than the galoomba, they are not fused.

[@rorycannon7295]

@@EnderrBoi i mean they could be shadoingled along the z axis?

[@rorycannon7295]

@@EnderrBoi but tru

[@stxcxv]

these videos just keep getting better and better

[@burntalive]

Amazing video rays! Thanks for the science!

[@professorgobbles6301]

I always look forward to your videos! Keep up the awesome content :)

[@lydia9822]

This is really neat! Thanks for the science, rayfsfire!

[@jhsonbi5914]

I wont lie Rays, Ive been loving this new type of content on your channel. Keep it up 👍. Also no test for me baby WOOOOOO

[@firered33]

Wait so does that mean it is possible to get a super expert difficulty level on easy if the creator beat it first try? My that sounds scuffed. Anyway, great job on the video. I appreciate the amount of time you've put into research.

[@MetaKoopa99]

One of the coolest raysfire videos yet! And the use of the Kirby 64 soundtrack makes it all the better!

[@dr.ravioli6654]

Great vid!!! hope this one pops off just like the last one

[@drock1331]

Love the video. The whiteboard needs to be featured more. Possibly for the Mario Party CPU tourney?

[@pristinerevolver]

Man, it’s so weird hearing you make documentary-style videos. I never thought they were your thing. But I think you do really well, I enjoy these videos!

[@CCrohny]

This is actually really interesting how you can see super expert levels within the expert category - if the creator of the level of their level practices segements before attempting to upload it they will usually clear it relatively quickly possibly allowing the level to get misassigned

[@phakepercs9599]

I love how Rays is truly just a lad living his life

[@marqimoth6987]

epic, keep up the great content. I really hope I can skip that test on thursday! I subbed, I promise!

[@lemonleproot]

Thanks for this I shall use this knowledge to make an Easy level into an Expert level as to try to get on Panga's stream.

[@fastbirb27]

Hype new rays video, hope it goes as well as the last couple

[@arishnicoff2621]

I have once encountered a very hard course with a high clear rate. It was an SMB course which was very short (one screen) and contained a giant pit filled with lots of boo rings and falling goombas and I had to jump on the goombas to the flagpole while avoiding the millions of boos, which is extremely precise. I couldn't complete it.

[@aidan7376]

Always a good day went you got content from ya boy rays

[@ichamsakkar4249]

i remember this level was used by someone who made a video about how the clear rate works, and yeah basically if you die a ton on your level while trying to beat it to upload it, it gets ranked harder

[@balls_gaming]

this was really good! We need more big brain videos

[@Bignateg]

Very good video! Keep it up💪

[@BlitzPSH]

@raysfire This seems like a good machine learning task. Have you tried grabbing a programmer who knows a bit about data science and get them to run a regression algorithm on some of the data you collected? It could be a simple linear combination of some of those variables you identified. If I were developing a game and wanted to check clear difficulty (and had lots of time) I think it'd be really smart to take a recording of the user's clear and try to mutate it by throwing in random delays / extra button presses / skipping button presses to get a sense for how "precise" the level was. I really doubt that's what they're doing though (it'd be expensive / take a long time to upload a level most likely).

[@JRENZO10]

this video is really good, and the kirby 64 ost just makes it better

[@SuperPatWorld]

Brilliant video!

[@ColasAlgo]

I remember playing a level never beaten before or two in normal mode. Always wondered how that worked

[@astral_haze]

other ideas i can think of for possible factors might be things like hazards in the level, level size, time spent creating the level, amounts of inputs on clear attemps (for example just holding right, holding right and jumping some, or having something more complex than that) also maybe the level creator's previous levels have an effect?

[@flower5185]

Great video! Super interesting

[@stephenday3901]

Outside of your mutiplayer with red and panga this type of content is some of your best. Keep it up!

[@luigitrollface]

scanning for shudheads 📡

[@failedcabbage]

shudhead located 📍

[@danksaucejeff5441]

These mario maker 2 vids r fire ngl. Idk why but i just started binging them 😩😩😩

[@failedcabbage]

BANGER VIDEO RAYSFIRE ‼️🗣

[@LucasDanielCab5]

Ty professor rays very insightful

[@InMyElement]

Worth the wait! Get this man a new chair!

[@thezestyman9159]

Thank you for the lesson, Professor Shud

[@TheTexanSCEditor]

My theory is that it starts out as the creator's clear rate, but after it's uploaded, it doesn't just determine the difficulty primarily off the level's clear rate, but instead the game might do 2 things: 1. Check for player outliers via their individual clear rate, and find the clear rate without those outliers. and/or 2. Take the average time to clear into account.

[@minbop4023]

It may be half 11 in the Queensland of England but rays has priority over sleep

[@silvermalwareai4047]

THE WHITEBOARD AND LAB COAT REVEAL! I don't know if imma pass the test on Thursday? Is there any retakes?

[@Yourmom7767]

you: a test on thursday me with a uno reverse card : no u

[@frogceo7228]

I always have great respect for Dr. Shud

[@superplushpower4649]

My top 3 hardest levels: 1.Porcupuffer's Crate Clash 2.King Bully's Mask Mayhem 3.Yoshi's Treetop Troubles It took me F.O.R.E.V.E.R to clear those 3 levels Honorable Mentions:Chaos Chasm

[@spacegamer47]

i feel like this guy had a science assignment where he could do whatever topic and chose this

[@Fa1cono]

Can't wait for the quiz on Thursday!

[@trifortay]

Okay I have studied all night for the quiz on Tuesday, I think I'm ready

[@yuriflufflover8855]

A banger vid as always

[@djtimo1490]

Very cool video Rays

[@JosepiThe13th]

You looked like EazySpeezy when you had those goggles on 😂

[@cupofapricotjam]

amazing video, as always shud!

[@DynaZ.3dsx]

I got a "First Clear" in Super Expert once. The stage was literally so easy

[@dragonomicon1788]

2:15 I look in newest and see like 15 comments saying spindel but nobody said beeBobble There you go, AComputerMouse

[@CuCuKM]

Thanks for validating me in my feeling that normal was way harder than easy.

[@sledgen.]

I love videos like this

[@wizulus]

When you see your tool being used in a video: 😇

[@slowwy]

1:36 Spindel

[@yusufh7278]

The levels I make that I rush through to make a record for people to try an beat gets dunked on in sheer amount of plays from my one level I took a leisurely stroll through and took my time to beat.

[@Taiwannabe]

Kinda related, does anyone else notice a huge difficulty spike between normal and expert? Like a "hard" difficulty between normal and expett. I'm not the greatest player but to me endless normal is a fun challenge but it's getting less fun as I improve. But when I try expert, I'm lucky to beat 2 or 3 levels with skips. Sometimes I don't beat any. It just seems like the difficulty spike is too steep where maybe having a "hard" difficulty could possibly help.

[@Tomahawk4948]

We need more professor Raysfire

[@yankeeforever101]

Damn Professor Rays at it again with another banger. Spindel

[@LockedFlea]

Oh no! I was already subscribed with all notifications turned on and liked and commented on the video!

[@luigiaiden2501]

awesome video rays :)

[@ami4705]

If you click on a users profile, the game keeps track of how many clears they've had in each respective difficulty - ie it tells you how many easy levels theyve beaten, how many normal, etc. Is it possible that these stats are taken into consideration? Like maybe if a player has beaten a ton of super expert levels before, the game considers them skilled at the game and how much of an effect they have on a levels chosen difficulty is altered.

[@dylanfitz8652]

Ngl this is a really good video

[@boxorox1964]

1:38 "Spindel" guy said to do it

[@bloodakoos]

the kirby music kinda banging