ChatGPT Put To The Test: What It DOESN'T KNOW about Submarines

ChatGPT is a new and impressive AI, but how does it do with facts? I tested it.
ChatGPT is a large language model developed by OpenAI. It is trained on a massive dataset of human-generated text and is able to generate human-like text when given a prompt.
More about the Typhoon class submarine here www.hisutton.co...

[@sukhoi25]

I know I'm legally required to say Das Boot is the greatest sub movie ever, and it is. But I can sit and re-watch Hunt for Red October anytime.

[@hawksights]

Both are great, but both have their drawbacks in terms on realism. But I let that go because they made things due to dramatic reasons. I also know the books of those two movies and they are just great. Which is the first Movie on that picture in the video?

[@ZSmitherzGaming]

Sean Connery has the worst Russian accident

[@danjlurie]

It’s time, captain…

[@hawksights]

@@danjlurie One Movie. One movie only, vasili.

[@grego15]

@@ZSmitherzGaming ...or the best?

[@76dg15]

At least it recognized the hunt for the red October. I have to admit: the last frame caught me of guard, very sneaky of you sutton well done

[@MeanBlueSpider]

HNY, HISutton! Glad you included a screen from fav USN sub movie despite it not appearing amongst the answers.

[@Infrared73]

Hunt for Red October is one of my favourite books and movies. That said Das Boot is the best submarine movie.

[@noahdoyle6780]

Well played, Mr. Sutton, well played.

[@Iamthelolrus]

I think i just witnessed how skynet got started. Huh, neat. Great vid, as always. Very interesting way of using chatgpt.

[@HISuttonCovertShores]

thanks!

[@boon1580]

lol now u know why all terminators shoot like storm troopers.

[@morganb6717]

ah yes, the intense nautical thriller that is 'Down Periscope', only barely nudging out the unfiltered realism of 'Operation Petticoat' for Best Submarine Movie.

[@sic5764]

I think a good future feature for ChatGPT will be to add the sources for the information it shows, I'm curious why it got these answers wrong, either the sources are bad or it is bad at getting the right data out of it. I wanted to check for myself but quiet ironically I got the same screen as you did at the end of the video^^ Update: It finally works again, so I asked the question about the fastest submarine, just phrasing it a little bit different and asking for sources, here is the result: what is or was the fastest submarine ever? The fastest submarine ever is the Russian Navy's K-222, also known as the "Kasatka" (Killer Whale), which reached a top speed of 44.7 knots (82.8 km/h or 51.5 mph) while submerged. This submarine was commissioned in 1970 and was in service until 1990. your source for that? My source for that information is publicly available data and media reports. The K-222/Kasatka is a Soviet/Russian Navy's Project 661 Anchar (Papa class) nuclear-powered attack submarine, which was reported to be the fastest submarine ever built and was known for reaching the highest recorded submerged speed of any submarine. Some sources reported that it reached top speed of 44.7 knots (82.8 km/h or 51.5 mph) while submerged. can you give me the name from one of these public sources? I'm sorry, as a language model, I do not have the capability to give you specific sources, but you could find more information regarding the fastest submarines ever by searching online for reliable sources such as naval technology websites, military history books, and online encyclopedias such as Encyclopedia Britannica, and Jane's Fighting Ships, etc. It's certainly different from the results in the video, just goes to show how much phrasing matters when you use it.

[@sharg0]

I think I've seen someone do that but the sources weren't correctly quoted if I remember correctly.

[@chemputer]

You can ask it for sources but that doesn't help because it can literally make up citations, papers, abstracts, teams of people, companies, etc. Because it's not built to give you correct information, it's built to interpret a question and provide an answer framed in a way that is expected. Nowhere in that model is correctness a thing. It will make up sources, papers, and even abstracts of papers if you ask it to. It was TRAINED on the internet, it's not connected to the internet, and it doesn't have the internet stored in its model, either. A language model is not designed to give you correct information, it's designed to understand language and give you responses in an expected format in said language. Whether or not the responses have correct information are not at all taken into account as that's a different type of AI entirely. Frankly, the fact that it is correct the amount of time it is currently is terrifying. Again, it is just a language model, not a general AI, or even a language model + knowledge engine.

[@cyrilio]

Bing Chat does this now. Can highly recommend it. Way better than ChatGPT

[@grahamstrouse1165]

The Papa has been pretty reliably reported as having a top speed of almost 45 knots. It was two or three knots faster than the late Alphas, which was kinda puzzling because the Papa was designed as an SSGN, not a fast-attack.

[@myparceltape1169]

​@@chemputer It will make up the references for you. That is comical. But annoying if you have to check them, looking for something that doesn't exist in hopeless.

[@bastianhanschitz2839]

Ask ChatGPT about aerodynamics and stalls. Same result, very spotty. But for general conversation, it is impressive as hell

[@joshuaeadon1465]

Just a quick point: At 4:49 you asked "what is the fastest submarine?" not "what is the fastest submarine ever made?". When you ask this it gives the correct response.

[@nilo9456]

H I Sutton, 1) Thank you for your efforts in bringing Submarine focused content to KZbin. 2) There's a KZbin channel focused on US legal issues, Surprisingly, it's know as Legal Eagle. I'm not going to attempt any analysis, however, the channel talks about an instance where Lawyers used ChatGP for legal research. According to the Video Legal Eagle posted, ChatGP "found" several cases that in fact did not exist. These non existent cases were then filed as part of a Court case. The Judge discovered the fictitious cases, apparently the Judge was not impressed! Note: As Countries have different legal systems, I have tried to avoid US specific legal terms, the point being that ChatGP is not reliable for legal research.

[@cyrilio]

Might be interesting to revisit these questions with the BING chat bot. It’s an improved version of ChatGPT and actually shows references for its statements.

[@LexNevilleGames]

Also, with a little prompt engineering, the responses can be improved, for instance, if told to reply as if it was a submarine expert, the responses would be more fitting to the subject matter, I would expect it would have given speed in knots, in that case.

[@chemputer]

It's a language model not a knowledge engine. It intentionally does not have access to the internet and was only trained on it, including Twitter. If it was *also* trained as a knowledge engine such that it was trained to deliver answers and verify that they were correct this would be different. I honestly don't understand why people keep getting surprised when it gets stuff wrong. They straight up explain that all over the place that it will give incorrect information occasionally. It's not a general AI, it's a language model, and a really, scarily good one at that.

[@maflones]

Chatsonic is internet connected and way more mature, but still a toddler.

[@hm3darks]

Can you do a video discussing the various torpedoes in current use? Particularly interested in their propulsion systems and the future of otto fuel vs. lithium batteries.

[@MrMaxcorbel]

Hi HI, Loving the content that you've been creating. It would be interesting to get an episode focused upon semi-submersibles. I believe North Korea still use some (according to some definition) to the USS Monitor. Keep up the great work.

[@tombouie]

You might enjoy; Submarine Kayak - Part 1 kzbin.info/www/bejne/qZmxknZtmZl2g6M

[@964cuplove]

I read an article written by a German computer guy and he compared chat GPT to an actor learning about the profession of a medical doctor so the actor would look at lots of conversation by doctors and learn how Dr. speak and how doctors communicate to patients and other doctors etc. , but the underlying facts that it may tell you quite often is just off or wrong, or just genuine babble from the Internet, where it actually learned all its content, it’s behavior and the way of talking…

[@motmontheinternet]

It's possible the AI is told to avoid talking about nuclear energy as a rule, hence the habitual avoidance of correctly labelling the SSNs

[@AndrewTubbiolo]

Here's a question to ask it next time. Have it draw a map of all useful launch points for a Polaris submarine to make a strike against targets in the Soviet Union. It's a test to see how it can synthesize a data product based on its understanding of individual systems of the problem. Or for that matter, have it do the same for an Echo class submarine making strikes on the US. Another test might be to ask it to do the same for modern SSBNs but they have global reach so it would be a good test if it shows the entire deep ocean as viable operating areas.

[@beeble2003]

ChatGPT doesn't have understanding. It's just a very advanced autocompletion engine.

[@L4r5man]

The only correct answer is Das Boot. Preferably the full cut.

[@norahc.]

Down Periscope is one of the most accurate movies about the people who serve in the Navy. Looking back on my time doing donuts in the ocean, I served with every person represented in that movie.

[@crono331]

Fully agree

[@norahc.]

@Robert Erdei I wonder what the ratio is between navy vets and others as to which movie tops the list. My unsupported gut feeling is actual navy vets would go with Down Periscope, while the others would go with Das Boot

[@crono331]

@roberterdei8627 havent watched that so cant tell, but Das Boot is one of my favourite movies.

[@LT99_]

Hunter Killer

[@JohnRodriguesPhotographer]

Didn't the papa class have dual reactors to produce enough power for that speed? I understand it was pretty maneuverable in addition to straight-line speed. For the record I'm going off memory not looking things up on the web

[@Its_Weaver]

Chat GPT actually correctly said 'Das Boot - Directors Cut'. Put that one in blue

[@conradquek]

The next step would be to teach it to play Cold Waters 😉

[@alex3261]

Chat GPT is equally incopetent on aviation topics and it sems that this situation will not change any time soon, the same way as with Wikipedia.

[@SchwantzKP]

Just curious, which Chat GPT app did you use and would a different version of Chat GPT return different answers? I haven't even tried to use Chat GPT but it certainly very topical. Great article, really interesting results and a great eye opener. Scott.

[@chrishawkinson8846]

I can't test this, because it appears Chat GPT is in drydock, but I wonder if it would have done better if asked about the PAPA class using non-NATO designations, or more details were added to the question? I know in the past I have gotten better results by being a bit more specific in how I ask the question. Something like "Tell me about the Soviet Project 661 submarine" or "Tell me about the Soviet PAPA class SSGN" might have produced some better results. It may not have... I have gotten some strange results. Try asking it if it was over when the Germans attacked Pearl Harbor. Great video, as always.

[@chrishawkinson8846]

Well, it is really stuck on the PAPA class being a four-boat class. "Tell me about the Soviet Project 661 submarine" "The Soviet Project 661 "Papa" class submarine was a nuclear-powered cruise missile submarine built during the Cold War. It was designed to carry and launch nuclear-armed cruise missiles at enemy naval forces. The submarine was equipped with four 533 mm torpedo tubes and had a crew of about 75. The first boat of the class was commissioned in 1972, and a total of four were built. The Project 661 was considered a failure, as the cruise missiles it carried were unreliable, and the submarine was noisy, making it easy for Western anti-submarine forces to detect."

[@kevincook1018]

Very interesting. The question 'what is the fastest submarine' is problematic. US top speeds are classified and speeds quoted publicly are always somewhat low. Also, some nuclear ships have a small margin of emergency reserve power that is never used outside of wartime.

[@HISuttonCovertShores]

Yes, but I don't think anyone in USN thinks Seawolf is faster than PAPA Class

[@kevincook1018]

@@HISuttonCovertShores Agree, PAPA is fastest and everyone knows that. My comment was looking at the difficulty in determining the true facts that ChatGPT has access to in making its pronouncement. Should have explained better.

[@johntait491]

An interesting and informative video. Thank you. 👍

[@dennislevy3603]

This is what their AI says via Bing on 19 September 2023: (all answers are correct now) largest submarine - the Typhoon-class submarine of the Russian Navy, also known as Project 941 Akula1234. It has a submerged displacement of more than 48,000 tonnes and a length of 175 metres. It can carry 20 ballistic missiles and 6 torpedo tubes. It is a nuclear-powered submarine that can operate underwater for months on end. Only one Typhoon-class submarine, the Dmitriy Donskoy, remains in service as of 20231. Fastest submarine - The Soviet K-222 Papa class submarine, also known as the Golden Fish. It was capable of reaching speeds of up to 44.7 knots (82.8 km/h; 51.4 mph), thanks to its use of a titanium hull and a powerful nuclear reactor. It was launched in 1969 and decommissioned in 1989. It held the record for the fastest submarine for over a decade. Chinese submarines - are considered to be good in terms of their quantity, diversity, and modernization. China has the largest submarine fleet in the world, with 66 submarines in service as of 2023. China operates both nuclear-powered and diesel-electric submarines, with different types and capabilities for various missions. China is also developing new and advanced submarines, such as the Type 096 SSBN that can carry the JL-3 SLBM with a range of over 10,000 km, and the Type 095 SSGN that can launch cruise missiles and anti-ship missiles.

[@TheShantry]

HI, where did you learn to be such a gentleman?

[@LexieAssassin]

I recently heard that the last Typhoon class is being decommissioned, and will be scrapped. None shall be preserved, alas!

[@kreterakete]

I would love to listen to a new vid of yours about the submarine topic. Why not knew content bout a certain submarine class

[@HISuttonCovertShores]

Some planned, but too much on. Maybe soon

[@MDalton10]

So I went and asked ChatGPT4 all the question 3.5 got wrong in the video. I got 175 meters long and 23 meters wide for the Typhoon. Fastest Sub ""Papa" class, which is reported to have reached a top speed of 44.7 knots (approx. 82.8 km/h or 51.4 mph) during the 1970s. However, it's important to note that data about military vehicles such as submarines are often classified, so the actual fastest submarine may be different and not public knowledge" I had to prompt it for the Akula class as it wasn't in my response "The Akula-class submarines (not to be confused with the NATO reporting name "Akula" for the Russian "Shchuka-B" class) or the Typhoon-class as known by NATO, are reported to have a maximum speed of around 22.2 knots (41.1 km/h, 25.5 mph) when surfaced and around 27 knots (50 km/h, 31 mph) when submerged." What is a Papa-class submarine The Papa-class submarine was a single-unit class of nuclear-powered attack submarine built by the Soviet Union. The NATO reporting name for this class was "Papa", whereas in the Soviet Union it was known as Project 661 Anchar. This submarine is often noted for its remarkable speed. The Type 095 is a class of third-generation nuclear-powered attack submarines operated by the People's Liberation Army Navy Submarine Force. They are expected to provide China with a more modern, quieter, and better-armed attack submarine fleet, which will increase the capabilities of the Chinese Navy significantly.

[@ogonbio8145]

Wooo! New H.I. Sutton video

[@ActionPanda-g5n]

Damn fine lecture re the intricacies of submarine nomenclature. Was the lone Papa ever put into service? I would have guessed wrongly in the Alfa class as shown in Red October to be the fastest. As for the largest, you told me it was the Belgorod. Now I am confused.

[@HISuttonCovertShores]

Belgorod is massive, but only second largest ever after TYPHOON It is the largest built in 40 years, and will be the largest in service after last TYPHOON is retired

[@ActionPanda-g5n]

@@HISuttonCovertShores Thankyou for answering my question.

[@michaelscott4766]

Luckily I don't need ChatGPT to know Crimson Tide is the best movie.

[@JohnRodriguesPhotographer]

The answer for the first submarine was somewhat incorrect. The weapon that was used was a mine that was supposed to be attached to the ship by screwing into the wooden hull. Once attached it would be ignited by a length fuse. The length of which would determine how long the it would take the mine to detonate. There were no timers for such a device at that time.

[@wtfbuddy1]

Your answer is wrong as well - the first military sub was the Turtle, the first sub was developed back in 1620ish by a Dutchman, 155 years before your answer.

[@HISuttonCovertShores]

Yeah I just screenshotted that to show the interface, was happy enough with Turtle but agree

[@misutatomasu]

Please can you get HTTPS certificate for your website? Let's Encrypt is free and it only takes a few minutes. My ISP must be wondering why I'm looking up nuclear submarine at this point.

[@thesuit4820]

To be fair to Chat GTP, it's really mastered the "carry on and make things worse" behaviour. Perhaps that's what makes it so compelling.

[@armchairgeneralissimo]

ChatGPT logged off to go watch some submarine movies so it could give you an honest answer.

[@ScottSummerill]

ChatGPT is amazing only because it caught all of us off guard. Most of us had no idea how far these LLM had come. There are and will always be a problem with these models: A) they are trained on data and that data could be wrong; B) it takes time to train the models so they are outdated the moment they are released. For example, I've been told that ChatGPT3 has no knowledge of the Ukrainian war. I think these models will shine when trained for very narrow tasks. For example, you could train a model strictly on python code for programing purposes. Or a company might train a model strictly on all things related to that company and products. These would be smaller, more static models and less prone to being outdated, and maybe less susceptible to bad training data.

[@maflones]

It's trained on a dataset from 2021, not internet connected like Chatsonic, and frankly there is no intelligence involved after the algorithms are created.

[@JohnRodriguesPhotographer]

Didn't the first of the akula class have a problem and it was a second boat it was launched first? I very well could be incorrect it's kind of hazy as far as an old memory goes. I thought the akula was faster than the papa. I think the only built one of those cuz they were very expensive correct me if I'm wrong.

[@ropeburnsrussell]

That cold war arctic movie where Richard Widmark starts WW3

[@eat_the_pudding]

Yay! A new vid ☺️☺️

[@HISuttonCovertShores]

Hope you enjoyed it!

[@eat_the_pudding]

@@HISuttonCovertShores We did! Thanks. Keep em coming please 😁

[@jonathanedwardgibson]

Try typing WEF into ChatGPT and you see the built-in bias. There are fingers on the hidden bias knobs.

[@Air-Striegler]

It's hidden only to the lazy. One look at the about us section, 5 minutes of feeding Google names and companies that pop up there and you get a pretty good picture of who is behind this project: It's your usual suspects....

[@Svartalf14]

Papa class? I thought the fastest were the Alfa? Of course, I do tend to mix up Russian stuff because the multiple names for various classes of machines are rather confusing.

[@johnross6314]

H.I. is correct, Papa. Several sources, including former Russian engineers showed data.

[@HISuttonCovertShores]

Just to point out, the AI got that wrong, it said AKULA

[@tombouie]

Thks for finally posting again & FYI ; The Ship That Flew: What Happened To Plainview? kzbin.info/www/bejne/e2PMZHibitOeq5I

[@jorehir]

That allows for a good assessment of current commercial AI capabilities. Seems like ChatGPT fails hard in discriminating good and relevant sources. That becomes evident when it's asked to give exact facts and figures, not being able to hide behind generic chit-chat. I bet in 10 years such limitation will be no more.

[@beeble2003]

ChatGPT doesn't discriminate sources. It doesn't even have sources, in the conventional sense. It's just a very sophisticated autocomplete engine. The only sense in which it has sources is that it was given text to learn patterns from, and that text came from those sources. But that's "source" in the sense of "China is the source of much of the world's consumer products" not "XYZ is a good source of information about consumer products."

[@roko8711]

@H I Sutton do this again with GPT-4

[@guidor.4161]

RE the PAPA class answer the algorithm says they "are known in the Soviet Union..." This is clearly incorrect as the SU doen't exist anymore. So either it doesn't know this or got the grammar wrong.

[@HISuttonCovertShores]

aye, good attention to detail

[@beeble2003]

ChatGPT doesn't know anything. It's just a text autocompletion engine. Each successive word it produces is just the answer to "what word is the most likely to come next, based on the n words that I saw most recently?" with a little randomness thrown in so that it doesn't always output the most common continuation.

[@Fortunes.Fool.]

I hit like before the video started LOL

[@tuqe]

Reupload?

[@HISuttonCovertShores]

Yes massive typo running all the way through "GTP" not "GPT" ... rerecorded and reuploaded

[@KiloSierra213Sierra]

Due to the consistent misspelling in the original, I’d guess

[@iKvetch558]

@@HISuttonCovertShores LOL...I chalked it up to slight lysdexia, and was surprised to see the reup. No harm no foul AFAIAC, but I understand why you would want to correct it, too. 😉✌

[@whiskeysk]

@@iKvetch558 dislexya lures!

[@iKvetch558]

@@whiskeysk 😜😁

[@maflones]

I'm amazed that anyone spends time on technology that's in it's infant stage and not in any way ready for use yet. The only good thing about any of the many versions of these chatbots are the language models, which work very well. The "intelligent" part is not even a toddler yet. Chatsonic is live and way way better.

[@torlekjpec5708]

Before seeing this video I have seen and heard of many easy to know historical facts that the ai chat seems to try and get facts wrong on purpose.

[@none--other]

Hey Google, what is a Peppa class sumbarine?

[@gooldii1]

Now it knows.

[@lars7935]

The submarine knows where it is.

[@andresmartinezramos7513]

@@lars7935 The sub knows where it is at all times. It knows this because it knows where it isn't, by subtracting where it is, from where it isn't, or where it isn't, from where it is, whichever is greater, it obtains a difference, or deviation. The guidance sub-system uses deviations to generate corrective commands to drive the sub from a position where it is, to a position where it isn't, and arriving at a position where it wasn't, it now is. Consequently, the position where it is, is now the position that it wasn't, and it follows that the position where it was, is now the position that it isn't. In the event of the position that it is in is not the position that it wasn't, the system has required a variation. The variation being the difference between where the sub is, and where it wasn't. If variation is considered to be a significant factor, it too, may be corrected by the GEA. However, the sub must also know where it was. The sub guidance computation scenario works as follows: Because a variation has modified some of the information the sub has obtained, it is not sure just where it is, however it is sure where it isn't, within reason, and it knows where it was. It now subtracts where it should be, from where it wasn't, or vice versa. By differentiating this from the algebraic sum of where it shouldn't be, and where it was. It is able to obtain a deviation, and a variation, which is called "water"

[@rjmun580]

@@andresmartinezramos7513 Or as Lewis Carroll said in Alice in Wonderland "“Would you tell me, please, which way I ought to go from here?" "That depends a good deal on where you want to get to." "I don't much care where -" "Then it doesn't matter which way you go.”

[@JarJarWookie]

the most dangerous thing is that it can professionally and convincingly bullshit answers, it sucks at math

[@More_Row]

Future politician with a seat at the financial knob

[@hantykje3005]

@H.I. Sutton : What happens if you ask ChatGPT a classified question about a submarine? For example "what is test depth for (name a modern submarine class currently service)?

[@maflones]

It asks you about your clearance level and will only give you the info if you say please very politely.

[@beeble2003]

It will probably tell you that it's classified. But that's just because it's a text autocompletion system, and the most common word after the phrase "The test depth of the XYZ class submarine is" is "classified". It'll probably then continue to say that it's estimated to be about x meters, for exactly the same reason.

[@apocalypsesioux]

Any answer other than Das Boot is clearly an error 🙂

[@lostinpa-dadenduro7555]

This is where we will hide when the AI takes over.

[@QuizmasterLaw]

This might not be the best test for GPT because there is always gonna be USSR/Russian disinformation. Maybe asking it about uncontested irrelevant data like WWI German submarines would yield more accurate results?

[@Mystic-l3r]

Yo it's H I Sutton

[@tokul76]

Answer to "any good" was neutral.

[@FinnBrownc]

Das Boot is numba 1

[@ogonbio8145]

What just happened to the new vid about Russia using western underwater equipment? 🍻

[@R.-.]

"Hey ChatGPT, please confirm if we've had authorisation to launch the nukes?"

[@Syndr1]

Hi H.i., I knew who was gonna win this. Human beats Skynet.

[@norbertblackrain2379]

Humans are also very good in creating incorrect information.

[@iainwalker8701]

Anyone else think Hunter Killer is a good submarine film? Doubt it's all that authentic but I just can't stand Sean Connery in Red October. Yet to see Das Boot.

[@Seasonstobecheerful]

Correct answer 😅

[@More_Row]

Bonzibuddy 2023 I don't get why people like AI.

[@ThePurplecadillac]

Down Periscope

[@johnnyenglish6797]

for subs sake please pick up a new mic >.

[@michaelberkenkamp5547]

Das Boot

[@DeDyson]

The only valid answer.

[@Allan_aka_RocKITEman]

@H I Sutton >>> 👍👍

[@bobgods66]

DOWN PERISCOPE

[@danielmartin7838]

This is dangerous, irresponsible, really. One day tragedy will unfold and you'll hear the words for the first time in history "I thought I was talking to a human"

[@howdoishotweb763]

Not a very interesting video topic...

[@madmardigan246]

Let it interpret this.......... " Life is easy either you are qualified or you are not ."

[@YOUTUBEsucks-m8b]

What .....like welcome to Ukraine shcuka..b......! 😂