The Future of User Interfaces with A.I.

Pondering the future of user interfaces with advancements in natural language processing and artificial intelligence
Tweet in reference: / 1574448455463862274
Neural Networks from Scratch book: nnfs.io
Channel membership: / @sentdex
Discord: / discord
Reddit: / sentdex
Support the content: pythonprogramming.net/support...
Twitter: / sentdex
Instagram: / sentdex
Facebook: / pythonprogramming.net
Twitch: / sentdex

[@whatthefunction9140]

Voice to text was supposed to overtake typing. Still waiting

[@temporallabsol9531]

Not many people actually thought this basic though.

[@yerry_verse]

You are still waiting. I use it everyday, I'm developing apps that use it for my company, some workers in my company don't even use keyboards. And I just wrote this note using voice to type.

[@dinoscheidt]

Voice vs Text is like Chainsaws vs Scalpels. For some it I’ll do; but most need the (slower) precision.

[@notead]

​@@yerry_verse Absolutely missing the point. You with your sample size of, what, 5(?) completely overlooks the fact that every single consumer facing computing device comes with (or requires) a text input device.

[@xaviermagnus8310]

Talking is slower than typing, erego nfw.

[@ChrisField13]

Great thoughts. I love the way you are able to zoom out and see the big picture rather than getting caught up with the details. My perspective is definitely shifted in a more holistic direction after watching this.

[@markusbuchholz3518]

Unbelievable summarization of the current AI trends, software direction, and human interaction. Since I love C++ mostly I hope it will never disappear. Yes, the channel and Sentdex are brilliant. Have a nice day!

[@tablettablete186]

A little off-topic, but what is your opinion on Rust?

[@markusbuchholz3518]

​@@tablettablete186 II do not have any opinion since I do not use it. As we know Rust has appeared in the Linux kernel. I believe in differesifity and common sense. From my humble point of view the Carbon, and Ziglang are very promising. Good to know also about beatiful Julia. Good luck.

[@maythesciencebewithyou]

@@tablettablete186 You Rust folks sound like religious cultists trying to convert people

[@sevret313]

I think a more relevant next step for UI is context menus that can better predict what you want. (While still allow you to go directly to what you want if the UI guesses wrong)

[@inevespace]

good idea but immediately problem appears: user needs to spend time to process and understand menus because they are different. It's not like "click, and immediately click 3 row from the top without reading". I don't want to say it is impossible, I want to mention that there are so many hidden big problems.

[@noahbarrow7979]

Mind completely blown. Dude you've introduced me to so many exciting concepts and apps. Keep it going!

[@f1l4nn1m]

Beautiful video and insight. I feel I can add a very interesting and relevant anectode in the discussion. When I was a uni student, my professor of Algorithms and Data Structures told the class that when the mouse came up, tecnicians used to joke that the real reason behind it was to make computer look faster, because compared to instructions at the keyboard you needed sooo much more time. ;)

[@ramtinnazeryan]

I always tell my student to look in the past if you wanna see the potential of the future. Imagine telling a person 170 years or so ago that you would be using a video chat for the purpose of daily communication instead of telegraph and writing letters. Those who think mouse and keyboard is never going to go out of fasion should consider how much pen and paper they are using these days for their daily tasks. Amazing topic!

[@TheNaz01]

Notebook business is till thriving, a lot of people really like to have pen and paper to jot things down.

[@ramtinnazeryan]

@@TheNaz01 Good point but you may agree that it is getting less common practice though! At least in my class out of 40 few students take note on their notebooks. the rest are taking photos from the whiteboard or simply read the power point and PDF materials. Which is good in my opinion. they can fully focus on what I am saying and not worrying about transfering information to paper.

[@FlameForgedSoul]

I think you'd be surprised(I use them daily). There's quantifiable positive neurological feedback to the act of writing with pen and paper, paper also doesn't crash and isn't subject to internet outages or hackers.

[@ramtinnazeryan]

@@FlameForgedSoul Alright let's look at it like this. the probability of your message being hacked considering the amount of communication happening these days is much less than loosing your letter in a mailbox or being read by someone else. How many times of a year you are dealing with crashing while writing an email or internet outage? These are some low probability things you are pointing out and no system is perfect. About the neurolgical feedback of writing on a paper I believe it depends on the person. look at the trend and you'll find many prefer mouse and keyboard instead of pen and paper.

[@punkogo]

Sentdex, I have been learning a lot of thinks with you 3 years ago. For instances, you learnt me to use pandas. And this interesting topic I think is incredible hype, but I need to take this information with great care. Probably We can see amazing project in a year. Thanks to take the time to explain this kind of things…

[@WomboBraker]

I think it vastly depends on your cognition. By that I mean that some people think ”visually” and some ”with words”, i personally am a latter one and find verbal methods of input to be very intuitive, thus im glad you phrased it like you did. Cool ideas!

[@dreamphoenix]

Great commentary. Thank you.

[@chickenp7038]

i often have a moment where i try to describe a friend how to do something on a phone, but then do it myself as they are just to slow and words often have not very high bandwidth

[@jonathanacuna]

Amazing insight!!

[@isaacandrewdixon]

thanks sentdex, this is a very thoughtful and interesting video

[@lancemarchetti8673]

Most informative 👌🏼

[@michaelpangilinan9568]

Thank you, it works perfect!

[@flowqi212]

As someone who's developing open-source voice assistants (in Java and Javascript and just a little bit of Python 😉) I'm pretty sure graphical interfaces will always dominate because they are just way faster. Even in my mobile app (SEPIA Open Assistant) I tend to use short-cuts and suggested buttons more often than voice input. There are specific inputs like timers and search though where voice just makes sense. I think many inputs will be replaced by automations in the future similar to your car example. You will not say "do this and that" but simply define a goal like "bring me to xy". I also believe there will always be some extra manual input device (besides a touch screen) that will make input even faster, maybe not a keyboard but version 4.0 of it 😄

[@Solo2121]

I still think we could be looking at it wrong. If voice assistants can get to the point where it's more like just having a real conversation (with interruptions and real time corrections, etc) then getting info would be quicker than graphical. Think of Iron Man or Her, speaking to someone would abstract a lot of the info we think we need if we can just get really quick and personalized summaries from our "personal assistant". I do think graphical will never go away completely and will always have a large use case though (for consuming the output, not input).

[@flowqi212]

@@Solo2121 I believe conversations are actually a very inefficient way of information transfer 😅, but I agree that this type of input might dominate for complex queries or creating new automations. It's similar to the timer/reminder example. Saying things like "remind me to do X at Y o'clock or when I'm home" is a pretty quick way to input a conditional time event in a single sentence. Deleting or modifying it will probably be faster with a good UI though.

[@Solo2121]

@@flowqi212 Even if you could delete it by just saying "nah nevermind" like 30 seconds later? Or modify like "Set a timer for 50 minutes when I get home."... 30 seconds later "Actually, wait until I've been home for 10 mins before you set that timer". If it were really like just talking to a human who was just always there (so no trigger phrases) then it'd have to be something I can't think of now to make me want to deal with any other interface. Heck, when I'm in the car now with the wife I'll ask her to do something I'm 100% capable of because I'm lazy 😂. All I'm saying is that if it really gets to the point of true human level conversation then what would be something that a graphical interface would be better for than voice when dealing with a car ui? I'm not asking rhetorically, there may really be good reasons that I'm failing to see right now.

[@flowqi212]

@@Solo2121 I totally agree, there will be situations where it is very convenient to simply explain your assistant what should happen, but don't forget how incredibly fast non-verbal input can be. Think of a list of scheduled events where you want to remove multiple entries. Your eye can spot things very quickly without reading the whole list and your fingers can delete them almost instantly. Now imagine the GUI is optimized with predictive algorithms and will show you things that only need confirmation or selection etc.. Then there are the extreme cases like video games where humans have optimized input via devices to incredible speeds. It would just be a waste to not use this ability 🙂

[@peternguyen2022]

@@Solo2121 I tend to agree with you that we may be underestimating the value of vocal input. But as many have written, both will be needed. From a product design point of view (anyone interested should read Donald Norman's book, the Design of Everyday Things), telling a product (ex. car, robots, computer, etc.) what to do can be done efficiently via voice while GETTING information/indicators/metrics from the product/machine is best done via a visual interface (since we can browse and read faster than we can hear). With Meta's research on input devices using our neural signals (ex. just "intend" to move your wrist or a finger and the computer will receive your signal and do what you intend it to do), it's going to get even crazier. What wonderful times we live in!

[@Maxjoker98]

I think mice and keyboards might be going away eventually, but I do think some concepts will stay for a long while longer, for example the concept of buttons. I could totally see most of the user interface being build around natural language, but I doubt you could archive actions per second-level interaction. Maybe the future will just be like on Star Trek, you can ask the computer to do stuff, but you also have panels of buttons for doing common things quickly. I also think the "doing things manually" approach will have to live on, or civilization will fall after we've forgotten what makes the AI tick(but that's more a philosophical problem).

[@Cominotech]

Good point, Harrison! More intuitive, and probably by a large portion, predictive

[@werthersoriginal]

The problem with NLP (audio wise) is it's linear. You can ingest more visually than you can auditorily. Think about reading voicemails vs listening to voicemails. It's rather agonizingly slow.

[@Spanu96]

Nvidia already made huge progress on that aspect, it's only a matter of time until they build an A.I that can adjust it's own self speach. I give it around 5-10 years in the future for that.

[@Veptis]

but is speaking a faster input than mouse, keyboard or I guess sketching and scanning? sure a display can show you more, but you can't feed a visual model without a camera or drawing tablet that easily.

[@werthersoriginal]

@@Spanu96 It's not the speed. It's the fact that audio is linear/sequential. Example: 👩‍👩‍👦‍👦👩‍👩‍👦‍👦 There are two pairs of people in colored clothing in this emoji. Look how much faster it was to just look at the emoji and process the info (even bypassing your inner monologue) vs reading it. Now, add linear speech to the delay.

[@tylermansmann1065]

This is a great video and very educational, although I think theres two major points I partially disagree on. Firstly, the car example was a race car. Racing is a sport where winning is determined by a combination of physical specs and the drivers skill. There is such thing as AI/self driving races, but people enjoy traditional racing for the human aspect of it. AI will never replace human racing because the human endurance/skill side of racing is part of what people are watching for. When it becomes AI self driving racing, it becomes a different sport. Second, a point on programming via natural language interface. You can code via natural language, but typing will always be faster for many people than using your voice. The best programmers minimize the amount of time they spend with their hands off the keyboard, so its a natural progression that we might be able to ditch the mouse with better natural language programs, but we will not be programming with voice as efficiently as we could with a keyboard. We could get to a point where you can ask for a program via voice and have it right away, or be able to use voice to ask for complex tasks and have AI program itself to execute this task, but text input will always be faster if your job is to actually program things for a living, even if its just to enter the same prompts you would otherwise speak

[@user-qs2rw3dd1c]

been thinking a lot over the past several months. gui is something we're used to, and kind of the best working option we have right now. but that doesn't mean it will never change. i think the era of api/llm/robotic based ui is soon to come. btw, amazing video, keep it up!

[@Zebred2001]

Gives a whole new meaning to "backseat driver!" Actually, we will be interfacing with the internet/cloud etc. through an audio/visual XR experience in which we "talk" to the system possibly through the more familiar method of creating a customized avatar to represent it.

[@awesomebearaudiobooks]

I think, just as GUI didn't fully replace the terminal, the natural-language interfaces will not fully replace the GUI! I mean, think of it. Just about 30-40 years ago, most people only used the terminal. And now, most people in the world use GUI. And yet, there are still millions of people who use the terminal on a daily basis! IMHO, the same thing will happen with GUI. Billions of younger people will only be using the language interfaces, hundreds of millions will still use GUI, and millions will probably still be using the terminal.

[@shiveeshfotedar]

An effective heuristic of thinking about interfaces is to think of them as declarative interactions and imperative interactions. Driving a car today with steering wheel is an example of imperative interaction. Having an AI agent control the specific changes and Customers interact them in a managerial position is an example of declarative interaction. Today for most of the imperative interaction as humans we use our limbs , like hands and fingers and for declarative interaction we mostly use our voice.

[@angelc6643]

what do you mean by declarative interactions?

[@PaulFeakins]

Loads of good points, especially the self-driving car analogy. Something to consider though is that although we all have phones that can do voice calls and send short voice notes, most people prefer to send and receive textual messages, even if in some cases those are transcribed by an AI, perhaps for convenience of the reader?

[@eduard_soul]

Very nice video!

[@dvandamme00]

so many points to this discussion.. like the slide rule, or graphical calculator, or anything else that gets superseded. tech changes. new thinking, new tools moves into new spaces of endeavour. Our choice is to keep up, get stepped on, or move out of the way. progress does not care, and it does not stop. progress is also not linear, or any any singular direction. Your abstraction concept is the kicker in this, of course. we have the tools now that do the work we need it to do. if the work changes, the tools need to as well. trying to use a different tool for a task just means a failure is more likely. GUI's are for the kind of tasks that need them.

[@mkrichey1]

I like this summation and agree with the approach, the interesting thing will be the balance of AI inference vs having enough feedback to know that the inference is what you actually want.

[@jestempies]

Copilot is useful in the same way copy/paste is useful. I wouldn't program without it, but you can't program using only copy/paste. I'm happy to use any new tools that become available, but I find it hard to imagine they'll be able to write completely new, original code. Currently 90% of what I'm doing while coding is reading, tweaking and debugging. I'm happy to have smart tools do some of that for me, but unless they write complete, working code that I'll never have to see, I'll have to have a way of moving things around, renaming things etc. Like they say, there are two difficult problems in programming: cache invalidation, naming things, and off-by-1 errors. I expect AI to certainly help with the last one, but the other two seem like they'll still need a lot of tweaking and manual input.

[@privet20005]

Russian site for public services implemented ai assistant a few years ago and it's turned out really useful! It's incredibly fast now to get some questions answered or order some paper. You just say what kind of paper/service you want to get and it answers with big button to order that thing. Same thing with banking apps! It's very convinient and I'm looking forward to see it being implemented more and more

[@dnmurphy48]

Use of natural language is decades away except in specialised use cases. I can imagine using language for eg setting a goal (drive me to London for example). but for e.g. entering all the data required for a claims handler in insurance? I don't think so. You make lots of good points.

[@vc.moser.ferrier]

See it exaclty as you do. I envisioned this to come at university, many years ago. Great to see, it becomes more realistic now.

[@kh-en7ul]

I think that the pros and cons between GUIs and NLP/voice commands are equal and opposite, more or less. Although voice input is slow, when NLP becomes more sophisticated, it should be able to take in abstract inputs. Conversely, GUIs allow for quick interactions, but what can be done on them are rigid and specific. So I don't think natural language will end GUIs-I don't think that they ever will. Rather, I believe that they will coexist as each is good at what the other lacks.

[@user-mv3cg7hi7g]

Really fantastic thought provoking video about how the ways that technology influences us may not be in the ways we think.

[@owlmostdead9492]

Language itself is an intermediary we rely on to communicate with each other, it’s compute heavy, lossy, slow and prone to data loss. In contrast if I show you a picture of a tree or a colour, information is relayed basically as fast as our conscious brain can compute it. Visual is the key and not language based imo.

[@krunkle5136]

My main concern is the power consumption of AI technology. It's bad enough how much power is wasted on shit web apps and advertisements causing the browser to slow, and it's important to see how that all adds up throughout the day and with millions of people. Also, some ai could be nice, but I don't think humanity exists to frollic and do the scant amount of things that haven't yet been replaced by ai in the future.

[@karlfimm]

We've seen the future of UI in tv shows going back to Star Trek Classic. The captain looks at a big display (hopefully 3D) and tells an underling what he wants to happen. It's just that the underling will be an AI, not a human. Voice input (possibly combined with a vague hand-wave in a general direction) and visual feedback (perhaps with a verbal summary) seems very believable to me.

[@andybrice2711]

Yes, definitely the hand-wave thing. I've been thinking about that recently. Also eye tracking. Subtle natural gestures convey so much information instantly. Like when you're working with another person and you can say _"Pass me that bolt."_ or _"Should I increase this font size?"_ Without having to describe in words precisely what you're referring to.

[@andybrice2711]

The _Iron Man_ movies are probably a great illustration of how future UIs should work. Whereas _Minority Report_ looks cool, but would be very uncomfortable in reality.

[@JB-fh1bb]

What do you think about a distant future where the AI takes over the “decision making in response to the response”? Dense UIs are usually important because we need the information to make (a series of) secondary choices. “What’s the CPU usage?” could be followed up by deciding whether there is enough headroom to start another process. “What’s the weather?” informs what outfit we wear and what we take with us as we prep to go outside. So if AI is informed and intuitive enough to make those choices for us, do we still need dense UIs? “What time is it?” “It’s 5:55, and you still have time to grab your coat before you leave” As a personal example: I used to keep CPU, HD, and network stats on my screen at all times but I shifted my behaviour so that downloading/processing was done overnight with low QoS and further optimized with AQM. Now I don’t need those stats be visible because I don’t run in to bottlenecks. Systems have removed my need to care about the dense information on a day to day basis.

[@hewas321]

Great intuition no doubt. What he said is all on the ground of experiences.

[@cnaccio]

Honestly I've seen a similar thing take hold as a Salesforce developer and with their Flows functionality... Over time I've moved from writing a ton of code to only writing code when I absolutely have to; all other functionality is developed in a flowchart like visual environment. Just not having to write unit tests when using Flows is enough reason to shy away from writing new custom code. I never would have anticipated this change in my own behavior.

[@CrashingThunder]

Github Copilot really sped up my development, at least until the trial ended. I can't say I'm willing to shell out the money to continue to use it, but I definitely do miss having it suggest autocompletions for fairly boilerplate Django code all the time. It probably worked even better for me because of the kind of code I write for a living being primarily in that framework and there being so much of it to pull from online. One of the nicest things though was that it often would suggest the proper code for things I constantly forget how they're specifically called in Django/DRF. That time of quickly opening up a tab and clicking the first link to figure out a trivial piece of code can really add up.

[@dnmurphy48]

Until you want to use a different style to what the programmers of co-pilot thought.

[@landon9]

@@dnmurphy48 This isn’t how copilot works. Copilot is not built with a certain style in mind, it learns on code commited to public repositories on GitHub. It also takes the style of your code base into consideration. If you choose to write code in a way that has never been used before by the world, then copilot won’t be able to help you, but that’s not a good development practice. On top of all that we use tools like ESLint and Prettier to enforce code styling so your code style doesn’t matter.

[@MorebitsUK]

I tried voice interaction in Tom Clancys Endwar, and it was nearly impossible to use. If they can improve it, i'd love to see it done. A good example is Windows 11; Endwar a bad example. I studied UI at college and have done a course in WPF and have just finished learning JavaFX. So its really interesting to see. Keep it up Sentdex.

[@firstlast6796]

@2:51. He said RACE CAR, not a normal car. Obviously saying "drive me to x location" would be better or simpler with natural language, but not if you are doing something that requires lots of precise and subtle inputs in a short period of time. Also, speaking of normal cars, everyone is moving towards all touchscreen interfaces because that's the trend, but there are now studies that show older manual buttons a switches are safer and easier to operate while also using the car. Basically i get what the original tweet said, but GUI's will ALWAYS be a thing in certain contexts. Like video games just as one example. I'm not saying "okay computer, play this game for me." I'm going to want a health-bar and a minimap etc.

[@rumble1925]

The reason voice commands will probably not become the default is because the interface is not discoverable. I can click around and discover functions in a gui. Voice implies I already know what the software can do.

[@nimrodhegedus1511]

One of the main problems with communication via language is the 'lost in translation' issue, where an idea or communication is incorrectly interpreted. The human brain has only recently (that's within the context of evolutionary history) adapted and expanded to process complex language. Those parts of the human brain that do process language, when articulating thought, are much slower to process those thoughts than those thoughts that are not derived through the use of words to form them. In a nutshell, words are a layer of thought that overlies more fundamental types of thought, words were and are a way in which social creatures can interact to communicate in such a way as to organise themselves to work together, which has contributed to the success of the human species, but in some cases such communications are not as efficient or can be misinterpreted with catastrophic effects. To sum things up, one does not need to think in words, thoughts that are articulated by words are only one aspect of thought, one is able to think without words (you may want to try this yourselves), and this type of thinking is usually faster, but less easy to communicate such ideas.

[@oliverpolden]

This reminds me of the phrase: "This is what you need, not what you want." e.g. People wanted faster horses, but what they needed was a car. Paradigms as you say will completely change and we will need to shift our mindsets away. I think you're correct that a graphical representation of data will persist as it's so easy for humans to be presented with a whole bunch of data and for us to pick out what we want, after all, that's literally what the eyes and brain have evolved to do. So I think that's clear from a feedback point of view. From the point of view of data input?... that must be thought or brain impulses right? Consider how much we do by just walking. The thousands of muscles all working together, even handling our breathing and moving our eyes. So input into AI systems will probably be brain interfaces which is an abstraction above natural language, or wait is it the other way around and we will be removing the need for that extra abstraction from thoughts into language. Will this mean we will eventually be able to communicate with machines telepathically before we could do it with other people? How sensitive will these machines be? Would we need some sort of interface we wear or would they be able to pick up on our thoughts from a few feet away?!

[@justinshankle]

The "natural language" of humans is visual. Spoken language is a relatively new concept in evolution. Humans are much better at quickly processing what they see than what someone says to them. I see the near future of AI as being able to read the eyes, face, gestures, expressions, and reading between the lines of what people are trying to say in order to produce what they actually want. After that, a direct link to the brain for both input and output will be a game changer. Just think and the computer knows exactly what you want, no time wasted translating your thoughts to words or movement. Visualize something in your mind and the person your "talking" to sees the visual instantly, or the scene is projected to the world. I see spoken language as the "relic" that will go away.

[@DardanAirlines]

Think “Computer” from Star Trek. UI will be created and rendered on the fly via neural network, as directed by the user, and defaulting to whatever output makes the most sense, given a query or task. Thought-sense will radically change how we interact with software and could allow for high potential bandwidth between the user and the software. By comparison, language and physical interaction based interfaces, such as that on iPhones, digital assistants, and command line terminals, may seem absurdly slow, cumbersome, and lacking expressivity.

[@andybrice2711]

I think it will become a fusion of _"natural verbal language"_ and _"natural gestural language"._ Just like when you're working on a project with another person. You'll be able to say things like _"Change this x to y."_ whilst looking and pointing toward the _x_ you mean.

[@mannycalavera121]

An adpative UI, sam base functionality, but custom generated appearances for each individual user based on their meta data

[@helpmeget100subs]

Video editing. Video game and game developing. Homework/ typing to remember. CODEC and file storage containers. And anything based off numbers.

[@7dedlysins193]

Hey @sendex would you say that your machine learning course is enough to jump to deep learning courses and apply to robotics ?

[@WarrenParks]

Somewhat agree but there have been low/no code solutions that come out all the time that always claim to be a lot easier and faster but usually they don't pan out very well. It's normally something like they can do easy and happy path situations but once you start needing much customization or optimization then they start falling short.

[@RicoElectrico]

It seems that Google overly relies on natural language matching for its search results. Now it often returns semantically close results with no verbatim occurrence of the phrase user queried. Think closest word2vec embeddings or so. Maybe this is responsible for the degradation of Google results that is often lamented on Hacker News?

[@serta5727]

Cool Idea to replace the UIs by AI interfaces. I think neural networks will do a lot of the work that programs do today. This means that many complicated buttons and menus and functions are replaced by a simple to use neural net. :) This will make many software way more user friendly.

[@headrobotics]

Ideally nature language and GUIs can be complementary. Visual prompts are very useful, and a dual parallel system, with a robust fuzzy feature search capability.

[@DRKSTRN]

As a functional programmer. I believe that it's the best tool for the job when it comes to language. The difficulty there is having the fluency of libraries and optimizations across languages to use what when. Which in of itself would be a model. That and you can never avoid semaphores.

[@Gilotopia]

I expect future AI interfaces will be like in star trek tng: you can describe complicated commands to the computer but the basic commands are still done through the LCARS GUI.

[@ezraszandala]

I definitely think people will always like GUIs, it's just that tactile nature we have. I'm sure however that GUIs will become very advanced, whether they are holographic or touching your skin or bending different fingers.

[@andybrice2711]

Comfortable, high-res VR/AR/MR will radically change GUIs. Imagine being able to sit/stand/walk/lay in any comfortable location with all your windows just floating in space around you.

[@andybrice2711]

(And I think _windows_ are an abstraction which will stick around for most interfaces. Because a 2D rectangular sheet is the best way to contain and present many types of information in the 3D space our brains can comprehend. We figured that out millennia ago with tablets and papyrus)

[@DingoAteMeBaby]

Man ive been saying this for a long time. It's more specific than just 'natural language'. If you look at discord, the chat basically already uses low-code. The way that young people are trained to search google in "search engine syntax" is again, a sort of low-code.

[@averagedudeusa9722]

Driving relieves my stress. If I can’t be in control I would be extremely stressed out.

[@barneymattox]

I honestly feel that LCARs (conceptually, not literally) from the Star Trek movies was a really good example of AI driven interfaces. Particularly the scene in Star Trek IV where they’re writing a program to parse out the whale sound interactively with the computer and a mix of conversational, visual, touch feedback through the process. The Okudas were far ahead of their time in their meta thinking of future computer interactions. This has driven a tremendous amount of my work and research over the years since first seeing that scene. …though I think the visuals of the LCARs interface are becoming very dated, the principles and designs behind it were very well considered and insightful.

[@breakingthemasks]

I'd love ve a link to that clip, because although I'm a tng fanboy, I don't recall that scene.

[@wktodd]

We have voice controlled cars in England, they're called Taxis ;⁠-⁠)

[@inevespace]

This is the first thing that I thought about. Even if you replace AI with other human, there are a lot of problems to overcome. Taxi is maybe simple task, but take app development. Between a coder and a customer there are a lot of layers of other humans who serve as AI to translate desires of customers to the coder. And even in such scheme we constantly have tons of problems.

[@manamsetty2664]

Yes yes great video

[@87vortex87]

Why do we need direct interaction at all? If systems know everything we do, all factors, and is trained on all people. It can probably predict what we want to do, and suggest. For example: every year in September I book a skiing holiday for January, it can just suggest me in September some destinations. Then my insurance can see that and suggest, or even automatically arrange insurance. Or a kitchen appliance breaks, my smart home can see that and automatically arrange an appointment to repair. It boils down to the fact that "why" we interact with an ui is to put information in something, but we will need to do that less and less because more and more information will already be known. So we don't need to put it in, and we can use that information to predict and even lower the amount of interactions even further.

[@lakshyagoyal5560]

A great example I can think of for this is call screening menus. They always have to list out each menu option one after the other. This is clearly super slow and frustrating since you have to wait to find the one you want. Some are even worse because they make you speak the answer, for instance a number, which is much faster and easier to just type rather than say.

[@jones1618]

Good thoughts & worth discussing but I think you are conflating AI "interfaces" with two concepts: 1) Verbal input - As you said elsewhere expressing intent with speech isn't very accurate or efficient for many domains. Sure, I could ask for a piece of software with a broad architecture & components & get an AI to produce some scaffolding pretty quickly but how much extra work would be required to learn/undo the AI's assumptions & work around them? 2) Abstraction - Take current Stable Diffusion / Dall-e text-to-image AIs: Right now it is enthralling to get a credible image (along w/ a few freakish sideshow rejects) that abstractly matches concepts from my prompt ... but what's missing is ability to hone in on & refine details, style & tone. So, initial expressions & abstractions only get you so far. These tools will really make an impact when they are trained on & tuned for incremental refinements and that won't always be possible from an abstract distance. The artist/producer will want the AI "user interface" to dive into details & aid improvements. Current AI in/out-painting tools point in that direction I foresee a combination GUI & verbal interface where you paint over a section & say/type "make this shinier & more colorful" or "make the muscles of this arm more tense & pulling against the gate." In short, AI tools have to get good at more than just abstractions, roll up their sleeves & get down & dirty into the details with humans to refine the output. Then they will be true tools we can leverage to create great things.

[@assassinjohn]

I agree with you. Imagine an Advanced OS that takes into account eye tracking and voice input with minimal latency using ML models.

[@cuszco]

I don't see GUIs going away any time soon. Even just for privacy reasons I can't see it happen. Natural language is fine for use in your home, in your car, a private office, etc.. But imagine everyone on public transport or in an open office just talking out loud to their devices all the time.

[@user-cc8kb]

Interesting ideas

[@felipefairbanks]

I think where NL interfaces will shine is in stuff that people don't really know how to do easily in GUI. For instance, if you're writing a paper, I think you will edit it via GUI without any thinking about formatting. then, when you finish, you'd ask the computer to format it using the needed standard. Other area I see it flourishing is with specialized software (such as video/picture editing, 3D creation), because these softwares are too complicated. even when you know how to use it, it is not unusual to forget the path to a specific menu and that sort of thing. So it would be easir to just tell the computer "no, I want it here" or "now add a mustache to it" than doing it via GUI. but for stuff that is already easy to do via GUI, such as web browsing and stuff like that, I don't think we will see NL interfaces taking over any time soon. That might and probably will change, though, with the metaverse, when we are surrounded by our OS, instead of looking at it in a screen. might be more intuitive to use your voice than a hovering keyboard in front of you in situations like that, who knows?

[@thatotherguy4245]

I don't have enough knowledge/intelligence to have an opinion on this topic, but thank you for explaining it so well.

[@xToTaLBoReDoMx]

What about using voice for input and something like dall-e to visualize the output? Should get the point across a lot faster than a bunch of paragraphs

[@Veptis]

The more advancements are made, the more I realize how appropriate my choice 3 years ago was to do a computational linguistics course... But I kinda want to get done with it now and get beyond just language. But think of all the interfaces that currently use natural language (like talking to other humans or writing emails), it essentially builds the basis for all interactions. And anything a human can read and process in their meaty brain to then write a different email or answer ... Could be replaced by a good langauge model and retrieval network. And nowadays, Copilot helps me write homework by knowing the context of my code and notes in a separate .txt file. In a few weeks I will be helping out in teaching a python intro class, and I strongly consider to tech the basics for 13 weeks and then just show them the "cheat code" of using copilot in the last week. Brain interfaces will win in bandwidth over language and vision, but I haven't seen a good enough technical demo yet.

[@f1l4nn1m]

I ❤ computational linguistics!

[@tobias-edwards]

Honestly feels like tech has stagnated for the past few decades, things have got faster but not necessarily smarter. With that being said, once there's a breakthrough everything else will rise to that level. Keyboards will stick around for a while longer, the mouse not so much: mouse -> gesture trackpad -> eyes

[@sirynka]

Not only natural language UI would be slow but it won't work in all environments. Humans are not the only things that are making noises and all those environmental noises would interfere with you. Furthermore, it's more straightforward to create shortcuts (fast actions, macros) with current UI. And even if you would be able to compress a sentence to a single word and let ai remember it, you'd need to get used to the command. It's fine when you've made them yourself and using on a daily basis but what about the chortcuts that are available out of the box? You wouldn't even know about their existence. So, purely voice interfaces are strange idea to consider but the combination on current UI technologies with proper support of voice commands or even ability to talk to the algorithm as you're talking to the person would be interesting way to go forward. But, don't we need agi for it? GPT style conversational model that would support voice interaction with the latency of human speaker.

Where NLP and more particularly Large Language Models are required for human - machine interfaces is Robotics. As soon as humanoid robots will be a reality, they will have to understand and speak generic and even technical human languages at a level not reached today by speech to text and text to speech today. It is where models such as whisper are important.

[@WarrenParks]

Looking at latest stackoverflow survey javascript is still most popular, python is 4th but they have html/css and sql in the list before it.

[@yabdelm]

I think for user interfaces, whatever gets the job done: fastest, easiest and go furthest for the user for that specific context (depends on the application), will win. If I had to bet on the future, I think it'll be something like neuralink + all apps that exist + a transformer/some neural net that tries to predict what we want based on our past history in interacting with these apps, and then 'zooming us forward' before we even state what we want. Instead of saying "take me to destination X", the AI will get extremely good at predicting what destination we want to go to based on past events that just happened. E.g. If I say to my friend on the phone "I'll meet you at 4 at X"... and so on. So we'll go from these delayed feedback loops, to the feedback coming before the action: action .......... reward. To action ... reward. To act- reward. To a-reward. To reward.

[@gandoffboss197]

In the near future (continuing with your build on top off idea), what if our phones become our UI. For example, you could tell your phone (note natural language used here) to connect to my bank. My bank would support a computer to computer interface in which the bank app would inform my phone as to the available commands. My phone would then tell me it has connected and show me a list of available commands. I could then tell my phone to transfer $100 to my wife's account. In this scenario, the services (like the bank in my example) do not have a direct UI. The bank app simple uses an available library which provides communications with an endpoint device (like my phone). This could work in your future self driving car example as well. I get into the "taxi" pull out my phone, scan a QR code which connects my phone to the car. Then my phone receives available commands from the car and shows them to me. I select/say take me to address "X". And I am on my way. At my destination, the car asks me to submit payment (yeah I know it could be automatic but wait). The car locks the doors until payment is submitted or the authorities are called by the car. When I pay, the doors are unlocked and I am free to go on my way.

[@drizel462]

It's essentially Iron Man technology. We just need the auto prototyping but we're already working on that too.

[@LukePuplett]

Remember: someone has to build all this. Given our regime is capitalism, a force that's likely to survive or catalyse much technological change, the endeavour needs to have a return on investment. That means all the mundane interactions, apps, and all the bizarre old-fashioned ways people do things, need to be replaced by some fancy new way that generates income and benefits the buyer. And that's not even the hard part. The hard part is educating everyone that your thing even exists at all, and that their use of time or profit margins would be vastly improved by your new thing, and that they must find the thing compelling enough to take the actions to implement your new product. It's like getting your toddler to try a new food you know they'll love. "I'll give you FIVE dollars if you JUST TRY IT". And you have to convince others to invest and people to work on it with you, and that you will be able to pay them. Even then your business might fail, the idea might be too early, you might give up, not have the help network, run out of money, or have a family member that suddenly needs your help. Until you build something to sell, you don't realise how almost everything has been forced into the world in this way. All these cool ideas won't happen unless someone, perhaps you dear reader, can see a way to make money with it, and are willing to dedicate about 10 years on it. In conclusion, some dumb ways will stick around for a long time because there's no business in its disruption, it's too expensive to sell, or the brainy folks have more profitable or interesting problems to solve.

[@festro1000]

Personally I like the idea. Sure there will be situations where GUIs will be superior like the examples Rasmus gave, there will always be use cases for a GUI of some sort especially in scenarios where speech is impractical like a crowded stadium, or among sleeping family members. But there are many others where it will be a clear choice, take dictation writing for example; people naturally speak faster than the most seasoned typist, and considering how digital assistants like google work, you don't even need to use a mouse or keyboard to get the weather, date, or, or events that people usually forget, and scouring the long list of nested directories to find that file you misplaced are practically a thing of the past as well. So long as the AI is competent in contextual differentiating and actually understanding what the user says and wants there are few cases where an AI would be shadowed by a GUI.

[@scurvydog20]

I remember a study a while back that found young people generally preferred guis while older people preferred voice commands

[@adamoreilly6546]

Fear is the mind killer. Great video

[@larrybird3729]

thank googles algorithm for me forgetting sentdex for all these years

[@crackwitz]

Combine speech recog, a vision model, a large language model, and a diffusion model, and you've got an AI that you can use to make all kinds of diagrams... Flowcharts, electronic circuits, sequence diagrams, outlines of books, story boards, game levels, PowerPoint decks... Anything! Just think out loud and the AI takes it down. Change things as you stare at them. And referencing those diagrams, AI can generate the details, again with human feedback in the loop

[@kurdi_x5842]

smart talk

[@franksonjohnson]

I'm sure you had this in mind when you discussed "this is how productivity works, we build on top of things and abstract the rest away", but I think it should be said that these higher-level abstraction are (and will be great) for *solved* problems. Abstractions are optimizations right?, always trading away low-level control for conceptual clarity. So the lower levels absolutely will continue to be critical as soon as you meet the end of the road for what the abstraction can do. Say Copilot goofed and didn't know what very specific implementation I wanted them to make for some implementation reason I know but it doesn't. Either the natural language has to be sophisticated enough to know what I mean by "factory pattern" or whatever, or it's on me to type up that version and know how to write the Python/C++/C/BrankF*#& for it. My car doesn't know what I mean by "give that construction worker up ahead some breathing room", so either I rephrase it to something it understands like "move a little to the left in the lane" or I reach out and override it. Abstracting interfaces to higher levels is almost always a lossy transformation, so there will continue to be a need for the lower levels, as well as a well-designed graceful handoff between the levels in the case of failure.

[@satanistbear4388]

Not first;( Great topic! Thanks for the video :)

[@shadowdragon3521]

It will be very exciting once someone gets a non-invasive brain-computer interface working with AI

[@silberlinie]

absolutely

[@ShamusMac]

Homie never heard of rally driver assistants. Shouting left and right works.

[@meguellatiyounes8659]

I think natural language can be used like inner voice for ai. and the generated images used as imagination.

[@mfasco84]

I was wondering, Copilot uses existing codebase to develop its "predictions" but, if more and more people use this technology and stop writing new code, how do Copilot will keep himself up-to-date?

[@tobiaswegener1234]

A little more than two months later and here we are, chatGPT gives us a taste how programming could look like in the future.

[@marklonergan3898]

With what you were saying that abstraction is king and everything will become higher and higher level, i'm not by any means saying you're wrong (i think you're right to be fair), but the thought of it does make me sad. I'm not saying i use assembly or lower languages like C any more, but if i needed to write my programs in assembly, i could. I understand enough about the low level workings of a computer that i can. The problem with getting this far abstracted from the machine itself is that there's a much lower level of knowledge of how it actually works these days, and it feels like the future of people that write code to tell a computer what to do won't actually know how a computer actually does any of it.

[@inspiringeducators]

It's a thing of basic human psychology and manual physical input provides a sense of control.

[@ojussinghal2501]

Please do a QnA it's been such a long time :(

[@_edd.ie_.o.8101]

For the car one they can take the cyberpunk 2077 taxi approach