The LK-99 of AI: The Reflection-70B Controversy Full Rundown

[@bycloudAI]

for people wondering why i draw similarity with LK-99, it's because the results are not reproducible. there were too much ground to cover, so I skipped a lot of technical terms. let me know if you have any questions

[@erkinalp]

are you japanese by any chance

[@FredZhang-hd2jf]

is HyperWrite (Matt's company) also fraudulent? I copied and pasted the social proof quotes (without the names/roles) from hyperwriteai.com into gptzero.me, and I got "99% Probability AI Generated", "highly confident".

[@judgsmith]

​@erkinalp Yes, he is an anime girl

[@WoolyCow]

I'm all for 'LK-99' being used as an insult from now on...

[@vidal9747]

The problem with LK-99 is that the "researchers" were so stupid that I got the impression that they actually believed what they've talked about.

[@MiniKodjo]

Im a LK-99 believer

[@victorcadillogutierrez7282]

LK-99 will be a kind of mark of shame on robots.

[@FRareDom]

completely forgot about lk-99 lol

[@pik910]

the AGI will give us LK-99, I can feel it

[@ThePhiphler]

Holy cow an AI model that acts as a room temperature superconductor?

[@A-white-fox]

"Just train from scratch" Is something that ML developers and engineers wouldn't say. I only heard this from techbros.

[@andermendz]

Yeah, training from scratch is stupidly expensive...

[@alqash6749]

my college prof: "im gonna pretend I didnt see that"

[@andermendz]

techbros just wanted to get people to rent GPU's at their start-up, bro was cooked when he pretended to "fine tune" a model without even knowing what a LORA is.

[@victorcadillogutierrez7282]

It's the equivalent to saying just restart the computer, that will fix it, bro, training a 70B model is hella expensive and you are looking for H100 donations, somethings is off about that.

[@Geen-jv6ck]

This guy is one of the reasons is why we need new uncontaminated benchmarks.

[@luiz00estilo]

This has nothing to do with benchmark contamination. It failed all the benchmarks. The guy just lied, and provided the benchmark testers a fake model.

[@w花b]

​@@luiz00estilo that's all there is to it, the comment feels like cope or huge misunderstanding (hopefully this one)

[@rawallon]

Unless Im missing something I don't think he ever benchmaked it, he just edited the excell with the benchmark values

[@markdaga1711]

I really hope that as a community we can learn from this and not just jump to swallowing the promises of whatever the next big thing is. 🍓

[@jorge69696]

People are eager to get conned.

[@Zeni-th.]

Haha love the implication at the end

[@aykutakguen3498]

We are eager for good models, thats why me and my buddy jumped on it as well. but there is a difference between testing and paying for something id say.

[@Prathik1989]

lol

[@samcertified7178]

Well it's a day later and it seems like this new open AI model is the strawberry thing. It actually seems pretty promising so that's cool.

[@AirSandFire]

Anyone who believes them that this was a honest mistake and they need to "look into it" is incredibly naive or doesn't know all the facts. they redirected to Claude API and filtered the word. They're fraudsters. Either both are, or one is and the other is a victim. But it's more likely both are. And they're comically bad at it too.

[@human_shaped]

Yes, that was the complete deal breaker. He Theranos'd the sh*t out of it.

[@user-pt1kj5uw3b]

I hate these crypto grifter types, its unfortunate that they have become associated with AI. The malicious and incompetent ones should be called out and shunned by the community.

[@Concurr]

And worse

[@XenoCrimson-uv8uz]

Grifters be grifting

[@jonathan2847]

90% of crypto is a scam, 70% of AI is a scam.

[@doyouwantsli9680]

Ai hype people and crypto hype people are the same. Often actually physically the same people that left crypto after their predictions of world domination did not come true. And when rokos basilisk doesn't happen by 2026 they will move to something else.

[@JorgetePanete]

it's*

[@TuanPham-fc2oy]

[@human_shaped]

[@palimondo]

Matt Schumer is a scammer 🍓

[@StefanReich]

[@elyakimlev]

This was suspicious from the get go, because of the claim that they had an open-source model better than the leading closed-source ones, and that it was made in just 3 weeks and presumably didn't cost much to train. If that were true, the logical thing to do would be to bring an investor and start a billion dollar company. But nooo, they wanted to share it with the public, from the goodness of their heart. Right..

[@nocturne6320]

I automatically wouldn't trust anyone who's a CEO of an API wrapper

[@DeborahWillingham-k2f]

This video was packed with helpful information.

[@duffsdevice]

Thank you for making this video 🙏🏻

[@sasha297603ha]

Thank you for bringing light on that conflict!

[@knoopx]

there's a 1b SOTA model trained in 2 days in a home basement coming out of nowhere popping out every month. so many attention seekers.

[@maxziebell4013]

Yeah, I was in the original stream and posted my question calling out BS on this approach being revolutionary and new twice (as Anthropic used the thinking tag since sonnet at least), but it wasn’t asked by Matthew in the Q/A section. He was probably so excited and screened for positive questions.

[@DemetriusZhomir]

"Attention is not all you need", ouch 💀 Next time if they publish something truly great, they risk to get too little trust

[@human_shaped]

If the results stack up, people will be back. Results are all you need.

[@kenedos7421]

This is the same guy that about 9 months ago said he had created a framework to "allow computers to autonomously do any task in a browser".

[@muscifede]

the famous "it works in my machine" lol

[@Razumen]

Depending on what you're talking about, that's entirely possibly.

[@aaronvontell3657]

Nice breakdown!

[@IvarDaigon]

the minute they said they were "re-training" the model is the minute everyone should have realized they were lying. Why would they need to retrain a model when they could simply copy it from their private API backend which they claim still works fine?.. Also there is just no way someone would create something allegedly so groundbreaking without backing it up so there would be multiple copies of the original weights if they were a legit model builder.

[@pagadalasumanth7969]

Such high quality information

[@Arcticwhir]

great overview of the events

[@cdkw2]

you know this video was rushed because its missing the bycloud touch. Looking forward for the next upload!

[@WiseWeeabo]

The upside is that I get pretty good results from stealing their idea. I ask it to first write down all its thoughts, and then write a paragraph to reflect on those thoughts, before writing the solution. Results are great.

[@CrucialFlowResearch]

You can also just chat with it regular and respond to it saying to correct the response

[@Interpause]

thats alr a well known prompting technique long before this

[@MrDowntemp0]

Great rundown! I'm so happy I've found this channel. Unfortunate there's so many scammers in the AI space. It's like crypto and NFTs all over again.

[@tiagotiagot]

While it is possible it's just a comedy of errors; the little hints of dishonesty keeps pilling up. I don't wanna write out the few plausible explanations I came up with to not help them make up excuses in case they're dishonest. For now I guess they've landed themselves in the category of put up or shut up; until they show replicable results, nothing they say has any value.

[@Warzak77]

This has so many red flags, i thought i was in USSR

[@AaronALAI]

I spent my whole 3day weekend messing with the multiple models locally.....😢

[@sorakagodess]

Ooh man... i had hopes for this one, but i guess it was too good to be real...

[@djayjp]

I think this is possible given you can now just take Llama and go from there, releasing your own open source model.

[@raizdesamauma8607]

Besides the whole drama (great video covering all this, btw, thanks a lot), has anyone been experimenting with this thinking, reflection, output, prompt? I'm trying out different variations (adding instructions to it, to fit my use case) of this prompt and using it with a llama-3-8b to test if the model can reflect on and correct, when needed, the response from a different model. It kinda worked, but I haven't tested enough yet. But I'm still wondering how the and stuff is really important or not so much. If anyone is also experimenting with reflection prompts, I woule love to hear your thoughts about it

[@tsilikitrikis]

Good job from the community, i was excited at the start with this model, but seems like a fraud. I had also spotted a supsicious move, when he announced at start that this is the "Ref 70B" model and we we are going to release the "Ref 400b" follows, but after he started asking for trainning compute to train it!!??!! GG, waiting for his final response

[@happypeoplearereal]

it sounds comparable to the sudden language learning these models have shown. given that languages have different grammar, translation datasets could be better here. on down the line i could see models taking the time to think about mirrored text more like a human would still cool to try though

[@agenticmark]

I fine tune a lot of models, voice, agent, text, diffusion. Nothing flattens out after 2 epochs.....

[@aykutakguen3498]

This explains it, me and my buddy tested it and it did not really function.

[@GoDjMike]

Ah yes, twitter drama

[@bycloudAI]

i need to get off twitter and stay on arxiv

[@Tanvir1337x]

It's X

[@palimondo]

@@Tanvir1337x Manchildren don’t get to decide how we call the public square.

[@shravancheekati9045]

OpenAI released their o1 model that actually uses reflection to do a better job haha

[@devSero]

We may not have any more Silicon Valley episodes but we will always have plagiarism, obscurity and chaos in the AI space.

[@synx6988]

did u ever make a video on LK-99? I wonder what happened with that

[@josepibyron]

If you look at anthropics research papers, claude 3.5 sonnet has the same style of internal dialogue that reflection claimed to have, it is not revolutionary at all

[@Westlee_]

Welp now its real... thanks open ai

[@juanjesusligero391]

* Grabs popcorn *

[@Ori-lp2fm]

Hopefully will inspire someone to actually use fine tuning to work

[@CoreyJohnson193]

The AI community is finally legit! We had our first big scam within the community! We did it, guys!

[@andermendz]

They just wanted to get people to invest on their start up.

[@punk3900]

It's so funny this guy compromised his reputation so much. It seems that he might be skilled overall.

[@victorcadillogutierrez7282]

Bruh I'm not into NLP that much, but even I know what LoRA is because it's used pretty much everywhere in machine learning nowdays, big Red Flag.

[@davidli8936]

It still astounds me that people make claims about solving AI hallucination. This and prompt engineering are literally two sides of the same coin.

[@anubisai]

I seen it work and do the reflection. Problem is reflection with LLMs doesn't necessarily produce better results and can often make the final answer worse. I wouldn't say he's outright lying but doesn't ha e a clue what he is doing.

[@palimondo]

He is outright lying. Don’t be naive. His benefit of the doubt account is overdrawn until next decade.

[@telotawa]

you should talk about Act I by janus & ampdot

[@boblep0nge927]

scammer! like we need more of those.

[@MrErick1160]

Two red flags and it should be clear to anyone that has even I tiny experience with marketing and engineering: his profile says "CEO" and "Entrepreneur" 😂. I think this should be enough. He's just here to sell and doesn't know and don't want to know shit about his it's supposed to work.

[@boonkiathan]

currently the Rabbit R1 of LLM finetunes there are certainly credible arguments of grift or fraud, but let the developer/founder(s) have a chance to defend themselves, by publishing a world-defying update

[@GreatAdos]

they already had several chances though

[@Uqbquqgw]

47509 Tevin Ports

[@RichardGonzalez-v6y]

Miller Jennifer Hall Helen Thomas Elizabeth

[@OxygenGenesis]

Don't forget that Glaive AI actually managed to amass VC/investor money from this grift. BLERGHHHHHHH

[@RichardGonzalez-v6y]

Allen Kimberly Thomas Richard Williams Frank

[@urgyenrigdzin3775]

The excuses all sound like a rug pull crypto scam though 🤷

[@bestelectronicmusicfromnew5189]

that talk became too technical, especially for phrases without a lot of musicality.

[@o_o9039]

dont know why people made such a big deal about it the second i saw its output i knew it was useless

[@travisporco]

this is an overblown bunch of tripe

[@BRBS360]

Yes. Also, there is giving the benefit of the doubt and then there is this crap. Completely oblivious to some overpromising grifter and so attached to hype, they lose almost all reasoning faculties.

[@peterbabu936]

I just created synthetic ai, self learn encoding and decoding, storing data in fractal structure. Can I be famous please?

[@peterbabu936]

import matplotlib.pyplot as plt import numpy as np import torch import time from google.colab import drive device = torch.device("cuda" if torch.cuda.is_available() else "cpu") print(f"Using device: {device}") class SyntheticAI: def __init__(self): self.encoding_ability = torch.tensor([0.0], device=device) self.decoding_ability = torch.tensor([0.0], device=device) def train(self): self.encoding_ability += torch.rand(1, device=device) * 0.01 self.decoding_ability += torch.rand(1, device=device) * 0.01 def encode(self, data): return (data + self.encoding_ability) % 256 def decode(self, encoded_data): return (encoded_data - self.decoding_ability) % 256 def generate_spiral_fractal(iterations, growth_factor): points = [(0, 0)] angle = 0 radius = 1 for i in range(iterations): growth = 1 / (i + 2) # This implements the 1/n+1 growth x = points[-1][0] + radius * growth * np.cos(angle) y = points[-1][1] + radius * growth * np.sin(angle) points.append((x, y)) angle += 2 * np.pi * growth_factor radius += growth return points def plot_spiral_fractal(points, ai): fig, ax = plt.subplots(figsize=(10, 10)) x, y = zip(*points) # Plot lines ax.plot(x, y, '-', color='blue', alpha=0.5) # Plot points colors = plt.cm.viridis(np.linspace(0, 1, len(points))) ax.scatter(x, y, c=colors, s=30, zorder=2) ax.set_aspect('equal') ax.axis('off') plt.title(f"Spiral Fractal AI Evolution (Encoding Ability: {ai.encoding_ability.item():.2f})") return fig def encode_data_with_error_correction(data, points, ai): data_tensor = torch.tensor([int(b) for b in data], dtype=torch.float32, device=device) encoded_data = [] chunk_size = 3 # Store each byte 3 times for redundancy for i in range(0, len(data_tensor), chunk_size): chunk = data_tensor[i:i+chunk_size] for byte in chunk: if len(encoded_data) < len(points): encoded_value = ai.encode(byte) encoded_data.extend([(points[len(encoded_data)], encoded_value)] * 3) # Store 3 times return encoded_data def decode_data_with_error_correction(encoded_data, ai): decoded_data = [] chunk_size = 3 for i in range(0, len(encoded_data), chunk_size * 3): chunk = encoded_data[i:i+chunk_size*3] # Decode each byte (taking average of 3 repetitions) for j in range(0, len(chunk), 3): byte_values = [ai.decode(chunk[j+k][1]).item() for k in range(3) if j+k < len(chunk)] decoded_byte = int(sum(byte_values) / len(byte_values)) decoded_data.append(decoded_byte) return bytes(decoded_data) def calculate_storage_capacity(points): total_points = len(points) usable_points = total_points * 3 // 4 # 1/4 of points used for error correction byte_capacity = usable_points // 3 # Each byte stored 3 times bit_capacity = byte_capacity * 8 return byte_capacity, bit_capacity def main(): iterations = 328 # Increased for more storage capacity growth_factor = 0.618 # Golden ratio, can be adjusted start_time = time.time() ai = SyntheticAI() points = generate_spiral_fractal(iterations, growth_factor) fig = plot_spiral_fractal(points, ai) byte_capacity, bit_capacity = calculate_storage_capacity(points) sample_data = b"Hello, Spiral Fractal! This is a test of error correction using GPU acceleration with 1/n+1 growth pattern." encoded_data = encode_data_with_error_correction(sample_data, points, ai) decoded_data = decode_data_with_error_correction(encoded_data, ai) end_time = time.time() execution_time = end_time - start_time print(f"Original data: {sample_data}") print(f"Decoded data: {decoded_data}") print(f"Storage Capacity: {byte_capacity} bytes ({bit_capacity} bits)") print(f"Execution time: {execution_time:.2f} seconds") plt.show() if __name__ == "__main__": main() # To run this code, execute this cell in a GPU-enabled Colab notebook.

[@peterbabu936]

import matplotlib.pyplot as plt import random import numpy as np import math import torch import time from google.colab import drive # Mount Google Drive (optional, if you want to save results) # drive.mount('/content/drive') # Check if CUDA is available device = torch.device("cuda" if torch.cuda.is_available() else "cpu") print(f"Using device: {device}") class SyntheticAI: def __init__(self): self.encoding_ability = torch.tensor([0.0], device=device) self.decoding_ability = torch.tensor([0.0], device=device) def train(self): self.encoding_ability += torch.rand(1, device=device) * 0.01 self.decoding_ability += torch.rand(1, device=device) * 0.01 def encode(self, data): return (data + self.encoding_ability) % 256 def decode(self, encoded_data): return (encoded_data - self.decoding_ability) % 256 def midpoint(p1, p2): return ((p1[0] + p2[0]) / 2, (p1[1] + p2[1]) / 2) def sierpinski(points, degree, ai): triangles = [] encode_points = [] decode_points = [] def sierpinski_recursive(points, degree): if degree > 0: ai.train() triangles.append(points) encode_points.append(points[0]) decode_points.append(points[1]) sierpinski_recursive([points[0], midpoint(points[0], points[1]), midpoint(points[0], points[2])], degree - 1) sierpinski_recursive([points[1], midpoint(points[0], points[1]), midpoint(points[1], points[2])], degree - 1) sierpinski_recursive([points[2], midpoint(points[2], points[1]), midpoint(points[0], points[2])], degree - 1) else: triangles.append(points) sierpinski_recursive(points, degree) return triangles, encode_points, decode_points def plot_sierpinski(triangles, encode_points, decode_points): fig, ax = plt.subplots(figsize=(10, 8)) for i, t in enumerate(triangles): color = plt.cm.viridis(i / len(triangles)) ax.fill([p[0] for p in t] + [t[0][0]], [p[1] for p in t] + [t[0][1]], facecolor=color, edgecolor='none', alpha=0.5) encode_x, encode_y = zip(*encode_points) decode_x, decode_y = zip(*decode_points) ax.scatter(encode_x, encode_y, c='red', s=20, label='Encoding') ax.scatter(decode_x, decode_y, c='blue', s=20, label='Decoding') ax.set_aspect('equal') ax.axis('off') plt.legend() plt.title("Synthetic AI Evolution and Training - Sierpinski Triangle (GPU Accelerated)") return fig def calculate_checksum(data): return torch.sum(data) % 256 def encode_data_with_error_correction(data, triangles, ai): data_tensor = torch.tensor([int(b) for b in data], dtype=torch.float32, device=device) encoded_data = [] chunk_size = 3 # Store each byte 3 times for redundancy for i in range(0, len(data_tensor), chunk_size): chunk = data_tensor[i:i+chunk_size] checksum = calculate_checksum(chunk) for byte in chunk: if len(encoded_data) < len(triangles): encoded_value = ai.encode(byte) encoded_data.extend([(triangles[len(encoded_data)], encoded_value)] * 3) # Store 3 times if len(encoded_data) < len(triangles): encoded_data.append((triangles[len(encoded_data)], ai.encode(checksum))) return encoded_data def decode_data_with_error_correction(encoded_data, ai): decoded_data = [] chunk_size = 3 for i in range(0, len(encoded_data), chunk_size * 3 + 1): # 3 repetitions + 1 checksum chunk = encoded_data[i:i+chunk_size*3] checksum_encoded = encoded_data[i+chunk_size*3][1] if i+chunk_size*3 < len(encoded_data) else None # Decode each byte (taking average of 3 repetitions) for j in range(0, len(chunk), 3): byte_values = [ai.decode(chunk[j+k][1]).item() for k in range(3) if j+k < len(chunk)] decoded_byte = int(sum(byte_values) / len(byte_values)) decoded_data.append(decoded_byte) # Verify checksum if checksum_encoded is not None: calculated_checksum = calculate_checksum(torch.tensor(decoded_data[-chunk_size:], device=device)) decoded_checksum = int(ai.decode(checksum_encoded).item()) if calculated_checksum != decoded_checksum: print(f"Checksum mismatch at chunk {i//chunk_size//3}. Data might be corrupted.") return bytes(decoded_data) def calculate_storage_capacity(triangles, ai): total_triangles = len(triangles) usable_triangles = total_triangles * 3 // 4 # 1/4 of triangles used for checksums byte_capacity = usable_triangles // 3 # Each byte stored 3 times max_value = 255 * (1 + ai.encoding_ability.item()) bit_capacity = byte_capacity * math.log2(max_value) return byte_capacity, bit_capacity def main(): iterations = 9 # Increased for a more complex structure points = [(0, 0), (0.5, np.sqrt(3)/2), (1, 0)] start_time = time.time() ai = SyntheticAI() triangles, encode_points, decode_points = sierpinski(points, iterations, ai) fig = plot_sierpinski(triangles, encode_points, decode_points) byte_capacity, bit_capacity = calculate_storage_capacity(triangles, ai) plt.figtext(0.1, 0.02, f"Final Encoding Ability: {ai.encoding_ability.item():.2f}", fontsize=10) plt.figtext(0.1, 0.05, f"Final Decoding Ability: {ai.decoding_ability.item():.2f}", fontsize=10) plt.figtext(0.1, 0.08, f"Storage Capacity: {byte_capacity} bytes ({bit_capacity:.2f} bits)", fontsize=10) # Example of encoding and decoding data with error correction sample_data = b"Hello, Sierpinski! This is a test of error correction using GPU acceleration." encoded_data = encode_data_with_error_correction(sample_data, triangles, ai) decoded_data = decode_data_with_error_correction(encoded_data, ai) end_time = time.time() execution_time = end_time - start_time print(f"Original data: {sample_data}") print(f"Decoded data: {decoded_data}") print(f"Storage Capacity with Error Correction: {byte_capacity} bytes ({bit_capacity:.2f} bits)") print(f"Execution time: {execution_time:.2f} seconds") plt.show() # Optional: Save the figure to Google Drive fig.savefig('/content/drive/My Drive/sierpinski_ai_evolution.png') if __name__ == "__main__": main() # To run this code, simply execute this cell. # Make sure you've selected GPU as the runtime type in Colab: # Runtime > Change runtime type > Hardware accelerator > GPU

[@drdca8263]

You mention a fractal structure; this leads me to expect that your idea probably doesn’t work very well, because “do something with fractals” is of the kind of thing which is more emotionally compelling to work on than is justified by how promising the idea actually is? That being said, if your idea *does* work well, I don’t think there’s anything stopping you from demonstrating that, and, if you do, I imagine you will get the appropriate recognition.

[@peterbabu936]

Original data: b'Hello, Spiral Fractal! This is a test of error correction using GPU acceleration with 1/n+1 growth pattern.' Decoded data: b'Hello, Spiral Fractal! This is a test of error correction using GPU acceleration with 1/n+1 growth pattern.' Storage Capacity: 82 bytes (656 bits) Execution time: 0.04 seconds It's working very well, with error correction and triple redundancy

[@drdca8263]

@@peterbabu936 Huh, 82:105 or so ? (I miscounted a bit so that 105 is probably not quite the right length of the string. I’m in my phone, otherwise I would just copy paste to get the length.). Ok, I can be wrong. I haven’t looked much, but, for such a short string, that seems decent? I imagine there are some test suites. It may be worth testing it against those, and if it scores well, mention the scores it gets on them (and context of what competitors get) when telling people about it?

[@recursive_tv3209]

genai was better 2 years ago

[@AB-wf8ek]

Sounds like a complete bozo

[@BrianMosleyUK]

💀

[@harambe2552]

Works on my machine

[@kirigherkins]

i don't understand a gosh darn word you just said

[@spoonikle]

skill issue

[@MrErick1160]

People are hyping literal *shit*. It's like 1% better and won't be able to see any difference interacting with it at all, and everybody is losing their minds because it has a fancy name yet ultra basic modified RLHF

[@drdca8263]

“literal” shit? You mean like, for intestinal microbiome stuff?

[@MrErick1160]

@@drdca8263 exactly 💯😆😆😆

[@TheZEN2011]

I don't know I did test the model on hugging face and it worked for me! I asked some of the hardest questions and it did indeed perform. As of early in the morning on September 11th. I would have worked on it and tested it more but I was past my limit on a hugging face. I think the guy is not stupid as I think he has some good ideas. Lots of good ideas are going to fail but that's how we learn. He probably had a team and things got a little confused as people are in charge of different aspects of the project. This kind of stuff happens. That's why the openai release dates get discombobulated sometimes.

[@o_o9039]

reflection is just a token eater you can do the same thing with prompt engineering there needs to be more innovation like some sort of way to store data like thoughts that aren't in words that are cheaper then that's a big improvement.

[@aykutakguen3498]

It failed utterly for us

[@palimondo]

Do you also decode Q drops in your spare time?!

[@edwinguinea7269]

LK99 🤣

[@aaravyadav9906]

first comment

[@drdca8263]

Why would you post “first comment?” And then an hour later, not as a reply to your previous comment, but as a separate comment, post “first comment”?

[@ps3301]

Too much hype in transformer. Transformer isn't the way to go forward

[@111455]

first there's race and gender grifters, now we have tech grifters

[@mqb3gofjzkko7nzx38]

You don't remember Theranos?

[@aaravyadav9906]

first comment?

[@mattbelcher29]

Just maybe, he was told not to release it? Maybe there is a reason that a big company or covert organisation would not want him to release this?

[@BarakaAndrew]

lol 😂 bro stop it