Replacing the Unix tradition

A rant about fundamental flaws in Unix userland, and a proposal for what could replace it. (Be warned: this one is quite long and rambling in parts!)

[@JeffreyGroves]

When you're working with someone over the phone and you ask them to type: ls -la and they type: Ellis Daschellie

[@mindawakening3873]

🤣🤣🤣

[@helidrones]

What‘s on the screen? A cup of coffee! Close all windows! A moment please ...

[@niewazneniewazne1890]

can't fuck it up in polish, worst he will do is ls myślnik la

[@timwasson1115]

As opposed to a gui over the phone where it's 'What do you see?' 'No, the other window' 'Yeah, it looks like a wrench' 'No below that' 'Now click "yes"' 'There's no "yes" button?' 'Try the "options"' tab.' 'No, not "settings", "options"'

[@gareginasatryan6761]

Tim Wasson i don’t see why experts have to cater to Paris Hiltons of the world. Wanna use a computer, learn how it works first. It’s not my job to educate users what the Windows key is.

[@snippletrap]

Terry Davis addressed some of these ideas in TempleOS. His shell runs C code interactively. He also argued that Linux is way too complex for single-user machines, saying that Unix-like is an 18 wheeler and all he needs is a dirt bike. As far as response-request for IPC, this is implemented by Minix. Modifying Minix might be the best way to go about building this vision of an OS.

[@PKAnon]

A fellow watcher of Down the Rabbit Hole, I see

[@tiberiusmagnificuscaeser4929]

@Patrick Keenan DTRH wasn't the first video that was made on Terry's life, and beyond that, there are a lot of people who followed Terry up to his death in the programming community.

[@csmusic6505]

RIP saint Terry. He was a tragic genius.

[@pinkguy8205]

He will forever be missed

[@scottspitlerII]

Terry and the CIA will watch this thread forever F

[@Jollyprez]

I've complained about gratuitous complexity since at least 1993. The universal response is that I'm too stupid to grasp what's necessary, clearly, and should probably revert to being a tester-or-something. Yet, somehow, I delivered over 20 shrink-wrapped projects ( when that was a thing ) on five different architectures before delving into web programming in 2005. Javascript in 2020 is insanely complex and well-nigh impossible without numerous "package managers." Aaaaand, forcing javascript to be on the server-side - ostensibly so engineers "wouldn't have to learn another language" - was crazy dumb, and the complexity of dealing with a dozen different ways to fake multi-threading makes any "wins" of using a single language on both servers and clients completely moot.

[@peterrobertson3273]

Complexity has been something I've fought against all my career. Computing is the only engineering discipline I know where "more visibly complex" is seen as "better".

[@pm71241]

We can agree that the Node.js world makes everything way more complex than it should be. Not sure you can use that as an argument to generalize.

[@rlidwka]

Javascript is running on the server-side because people want the same CODE running on both client and server. For example, if web page is generated by JS, and web browser has JS disabled, server can render that webpage instead. What dozens of different ways of fake multi-threading? There is just one, that's async runtime (with a few legacy APIs that do it, but it's all been replaced with async/await by now).

[@Jollyprez]

@@rlidwka ...all been replaced - uh, no. Do you have anything in production? You don't just yank working production code, and change things to async/await. You leave it there until it either is found to have a bug, is replaced by other code, or is refactored. And, async/await is NOT true multi-threading, and if you've done actual multi-threading you'll know that. Also, async / await is NOT what you want in many cases, particularly await - as you end-up with synchronous code that will impact user experience. I'll stick with PHP on the backend. It's GOOD to help separate tasks.

[@rlidwka]

​@@Jollyprez APIs that a developer frequently interacts with are being migrated to async/await, you can check popular opensource libraries for that. Perfectly working production code doesn't change, but who cares? If it's working, let it be working. And if it breaks, time can be allocated to rewrite it up to newer standards. Async/await allows for the underlying library to implement multithreading, which webworkers do a good job at.

[@krux02]

It's not true that computers are not shared anymore. At my university there is a computer pool for all the students to use running linux. Each user has it's own desktop and home directory. Everybody can use any computer, and I do not need to bring my laptop from home anymore.

[@scottlarson1548]

I still work on systems with over a thousand users which need to be protected from each other. Brian thinks these systems were all replaced back in the 1970s.

[@Diamonddrake]

There’s no reason why those systems couldn’t run a different os, or os feature set. “But I still use x” is not a reason why everyone must be forced to use x. If we are talking about an ideal alternative for a simplified OS, it wouldn’t replace all systems. In general it sucks to not be an admin, and if you’re admin users don’t matter.

[@smorrow]

On Plan 9, whoever logs in at the console is effectively root, and also the only user, on that machine. It's very simplifying, and no loss at all in terms of security, because in a proper Plan 9 system the terminals have no local storage.

[@eaustin2006]

And horses and buggies are still used by the Amish.

[@AmexL]

@@eaustin2006 yup and we still have libraries. Poor argument by the first two commenters

[@zzolm]

A really good talk. I don't necessarily agree with everything but its a good perspective and raises some really pressing questions, ones we should ask more.

[@sheepphic]

I think that the dichotomy of "admin software" vs "not-admin software" being your only method of permissions is a bad idea for security reasons, for the same reason that setuid root programs in Unix are encouraged to drop root privileges as soon as possible; you want software to have the least amount of permissions for the least amount of time, so that the impact of vulnerabilities in the software is minimized.

[@Mr.Nichan]

It's possible he meant them to be able to change over time. Either way, though, I think the two-way system, while it does simplify things, is actually MIMICKING on of the weaknesses of the Unix permissions system. I can agree that the focus on protecting users from other users of the same computer is somewhat archaic nowadays (even if it still applies often enough), but what's also archaic is the lack of focus on protecting users from their own software. That's why I think it might be better to start with the Android app permission system and try to improve that (e.g., generalize it, make it more fundamental in the OS if it's not already, focus it on more precise user control, maybe add OPTIONAL support for multiple users). The Android app permission system (which keeps changing -- improving I think) gives each app permissions, controlling exactly what types of things each app is able to do (rather than what files each USER is able to do what to, as in Unix). Also, since the Android app permission system always tells you exactly what permission is blocking a process and let's you fix it immediately in a pop-up (and Android 11+ lets make an exception only once, rather than permanently changing the permissions), rather than stopping and saying "Permission denies" like Linux, common Android permissions problems don't just break things and require troubleshooting using the command line like Linux ones do. Actually, I imagine a lot of my personal problems in this vein with Linux permissions could be solved within the Unix permissions framework, if I just massively switched around my settings; however, here are some problems I have with and improvements that could be made to my Linux permission: It would be nice if I could properly install software without needing full superuser privileges (I use sudo apt/apt-get install when possible). It would be nice if you could control exactly what what files, directories, or drives a particular process has permission to affect without my express permission. (That way, when I download a file that has a script that's supposed to install a new operating system on some external drive, I don't have to be worried that this script I downloaded from the internet and don't understand will mess with any other drives, like my primary hard drive. Similarly, sandboxing untrusted executables, could be trivial, and downloaded software could be used in a permanently sandboxed state without needing to use a virtual machine; also, browser scripts could be sandboxed at the operating system level rather than the browser level.) It would be nice if I could use software that can READ my partition table without having to give it permission to CHANGE my partition table. (The only way I know to check the layout of my partitions properly, or to make images of partitions, is use software that need superuser privileges whenever the run, like parted, gparted, fdisk, and partclone (actually, I think parted might be able to run without sudo, but there was something inadequate about what it did).) Although I'm not as paranoid about my Linux PCs (/laptops/etc) spying on me as my phone (mainly because it normally has no microphones, cameras, or GPS/wireless phone service/wi-fi/bluetooth/ attached), this concern definitely not unique to cell phones (and, logically, it makes more sense to worry about malware when you rely heavily on free software). Thus, being able to control the use of such devices might also be useful.

[@Mr.Nichan]

I feel like I'm showcasing a lot of my ignorance and noobiness in that last comment, even if my core points made sense. For the record, I forgot about the df command (which allows you to see partitions without needing admin privileges), but I DID already know that Android was based on Linux, though I didn't know the specifics. I was trying out puppy linux (bionicpup64) in a virtual machine, and I thought it was interesting how in fatdog and puppy, the default login is root, but you can set network applications like browsers to be run by a user called "spot", who has no admin privileges, and there's also work on a third user called "finn" or "fido" who has no admin privileges but can sudo to get them, like on Ubuntu. Someone who I guess (because I don't remember for sure) was the/a designer of fatdog and puppy compared the "spot" system to Android, which he claimed ran each process under a different randomized user. Still, maybe you can see how it might be reasonable to call this a contortion of the Unix permission system, designed for computers shared by many humans, to fit a new context, where the focus is on sandboxing many different applications from each other and from hardware, personal data, and core system programs and data, and on separately limiting the permissions of each application, rather than of each human user, to only what the one human user wants and expects that application to do. I was also interested that puppy comes with a very simple GUI "partviewer" for viewing partitions without admin privileges, though, as I mentioned, I more recently rediscovered that df does a similar thing, though without the bars to VISUALLY compare used to allotted, and with more complications, and I don't know how to get it to show all partitions (including swap) or to show start and end points on the disk.

[@ILsupereroe67]

"The notion of strangers sharing the same machine is outdated, that doesn't actually happen"... yeah right, what about fucking servers?!?

[@ILsupereroe67]

@@avinamerkur1484 i meant servers where you log in via SSH, for example

[@hasen1957]

Regarding protecting users from each other, you say it almost never happens that strangers share a computer, but what about public libraries or computer labs at universities where students have unix accounts accessible from any machine on the lab?

[@mattegeniet]

+hasen195 Not only that, we can ssh in to a shared server that typically is used by quite a few students simultaneously. In addition to your home folder, you'd typically store things in /tmp that you don't necessarily want other people to be able to access. We also have group projects where it's important to belong to a group that together have access to a folder that other people does not have access to. The concept of strangers not sharing the same computer is simply not true in university or company systems.

[@DavidPetersonAU]

+Oskar Södergren I would say yes and no regarding shared computers. Yes, people need places to store personal files. Mostly these days those files are stored elsewhere, not on the specific computer itself. Even in personal devices, we now put more stuff in Dropbox/icloud/whatever. In fact, in my university in the computer labs for my section of the school everyone's shares a single login to get onto the machine. It was simply too much maintenance overhead with no real benefit support individual logins. Everyone brings in personal storage devices for personal data, and just users the machines to run applications. Secure? Not really. But more manageable, yes. I'm not saying it's a great solution, but it is one borne out of the necessity to simplify.

[@llothar68]

I hope they have some of this hardware/software solutions where the system is reset back to the base system image after login. But the solutions for the future is not dropbox or any other opaque cloud but your own device. This convertable idea from MS. You have your virtual machine on your phone/usb stick and the host has some kind of vmware player that just runs your machine. Simple and easy, and well restrictable. Ok, there is this USB controller problem that totally kills security at the moment.

[@acmenipponair]

Well... in the beginning, when UNIX was invented, this was the typical environment and that's why all this shared stuff was implemented. But: MOST of the time nowadays you use your own devices. You don't share your home PC, your Smartphone or your console with many other people (mostly only your kids or wife, but often they have their own devices, wo even not that). You can see it especially in the case, that most of the computers nowadays have the main user profile run as Administrator (and Windows and Mac went so far to even cut away privileges from this Administrator and give it to a super administrator, that you only see in repair mode), a role, that in the old time was restricted to one type of users, the server operators, while all the others were just clients, guests on the system. Yes, in offices you often still have in the network that the PCs itself are in an environment of networked computers, where they are only clients, but remember: when Unix was invented, a "computer" didn't meant the terminal you were on, but the whole environment, and nowadays this computer is inside your box and you only network with each other. @David Peterson: In my university they have written on the screens: "Don't save on desktop or internal storage device, at it will be wiped clean after you log off", as the computers are running in a sandbox mode. We are urged to story every data that we want to save on USB devices ;)

[@PaulSpades]

We just shouldn't use multi-user systems if there's no need for them(nobody uses my linux boxes other than me). Yes, in a library or computer lab you might need multi-user system. Debate over.

[@laurencevanhelsuwe3052]

Anyone championing a reduction of all the accidental, corrupting, pointless and often insane complexity in modern IT, gets my vote. Unfortunately, this complexity sells consultancy hours, days and months... it keeps most people in their cosy jobs. It's also a power trip for most: those who understand the nonsense complexity have (lots of) power over those who do not. "Trust me, I'm a doctor" (and my invoice is in the post)

[@sless621]

(This is just gathering of lose arguments, facts and emotionsand is not the base for a thougthful - and respectful - discussion. I want to point out the grotesk of arguments given in the preceding comment(s). So take it more as a satire. Yeah.) So let's see, where do you want to start? How many storage systems are there? Read only? Spinning hardware disks? Solid State storage? Tapes for archiving? Which do you want to get rid of? Windows? *NIX? Why windows server? Why windows 10 Pro or enterprise edition? Which ot those should we remove from the market, no from universe? Hardware? RISC or CISC? I mean it seams that RISC is beginning to be more popular even in Server and desktop area. Smartphones, that's gonna be great! Are IT systems too, you know? Software, hardware. Which should we drop? MS did us a favour, no more Windows Phones? But the apple and the robot, which should it be? Two totally different operating concepts, i guess, and I am not only talking about the pesky user interface. Just get rid of all those pointless, awkward Signal Threema Pulse Telegram Skype. Just go WhatsApp. And Facebook. Btw, thanks for Google making it -1 ! All those efforts wasting in doing DIVERSITY just that some people can get jobs because they got no(t the same) clue as you playing rock music, miscalculating the next world financial crisis, lying for getting voted for being able to lye to many more people. No, you want to get rid of those IT guys because they should have learned different stuff in life. Excuse me, all those pros creating overly complex IT systems for their own benefit (money and "Trust me I'm a doctor" attitude). Where did these people start to make tech? Google founders? Facebook founder? Apple founders? These and others dedicated their lifetime, some even from childhood, to create complex, efficient, unique, diverse software and hardware - also to please you. Are these people those snobs you are calling them? What is it that you do for work? (Rhetorical questions, please don't answer) Can it be done just by watching someone else? Doing the same "moves"? Did you have to learn stuff? Would I have to learn stuff to do your work? Would it take me hours, days, weeks, months, or even years? I know you would have to to learn "IT" because you sure seem to have no clue what IT tech is, from nowhere about hardware to nothing at all about operating systems. People spend their lifes! for knowing IT and tech as doctors might spend their lifes just for doing their job, healing or research. (Gonna delete this post in a few weeks, but until then I feel a bit reliefed.)

[@jackblack4246]

You forgot to delete

[@NyscanRohid]

@@sless621 What?

[@dekippiesip]

It's a lot like biological evolution, where certain organs may be redundant, certain vains and systems take unnecesary complex paths, etc. But it's all there because of history. It's the same thing, someone starts building some very rudimentary computing device. Then instead of inventing the wheel again, someone else uses that thing to build something more complex, etc, etc. After more than 10 iterations like that you get a mess naturally. It was no ones intent but there we are, patchwork on patchwork. But now redisigning the entire thing will take a lot of work, and we would have to reschool everyone. So we just reluctantly carry on piling even more shit onto that mess, increasing the cost of redisigning the whole thing EVEN MORE, and hence we are stuck in this vicious cycle of ever decreasing efficiency where the mess just keeps getting bigger. Weather it is biological evolution, city design or software, the same mechanism explains the inefficiëny. And the bigger the history the greater these inefficiencies get. It's also called the 'law of the handicap of a headstart'. Those entering a market freshly can immediately develop anything wayyy more efficiently than established companies with a long history: en.m.wikipedia.org/wiki/Law_of_the_handicap_of_a_head_start

[@PauloConstantino167]

"(and my invoice is in the post)" lol that got me laughing good :P

[@itaborai83]

Package management on the kernel? I'd love to see you pitch it to Linus

[@abigailpatridge2948]

I'd love to read Linus' resulting tirade. I think I was tame compared to what Linus would do.

[@wiskasIO]

He would get a Nvidia-like response.🤣

[@EnergeticWaves]

@@wiskasIO what does that mean?

[@kazaamjt1901]

@@abigailpatridge2948 Linus tends to not go on tirades over these things. Fucking up userland abi, now that'll set him off.

[@thecomputergurukid]

This is kind of what containers can be used for.

[@EliteTester]

>systemd is good oh boi

[@RGD2k]

Seems all addressed by/in freeBSD. Including things like jails that they've had since forever, as well as simpler thing like insisting that man pages actually include examples. But mostly the 'who is in charge of the whole?' question in particular. Linux on the other hand: yes, it's always a mess. There is a lot here in particular I disagree quite strongly with: - that the 'terminal paradigm' needs to go (it's basically just a SMS session with the computer, with the convenience of very limited 'env' state saved and tracked for you by the computer.. The interface is in fact *so* good, that it's largely replaced both actual phone calls, as well as emails and letters, *between humans*). - that the whole unix 'story' is just 'too complicated'. For example: the 'environment' state: the command to get the complete listing of all of that, is just `env`. - that the complexity of package management is unresolvable: jails. Very old FreeBSD solution, basically the same as many newer 'container' type solutions, like a very lightweight VM. - that whole 'just a simple as copying a file' thing. Which is in fact also solved for all x86 computers (and almost all OS simultaneously) by the 'actually portable executable' file format. Apple also fixed it in OS X. - that we can just use a general language for what interactions shells are for (disagree quite strongly here - usability and also 'power user' convenience *matter*) - that naming is bad (brevity is good, and whatever names you choose, they're always a made-up id hash whatever they are, so short is good, beause usability matters). - that systemd was unequivocally a good thing. I think it's good for laptops, tablets and phones, but it's not good in general. Worse, it's OOP shoehorned into C in a fairly transparent embrace-extend-extinguish ploy, and I thought Brian would have noticed that. - that a system registry, a la Windows, is a good idea. Seriously, all eggs in one binary-format mess disaster waiting to happen? And internally it's basically identical to unix's 'one root filesystem' anyway, except you get a constant number of roots named long and meaningless things. And all meaning of every key resides hidden somewhere in source code you will never read, or maybe documented by essentially internet hearsay. - that a absolute monoculture - enforced by GUID's - is in any way a good idea. In a perfect world perhaps, but that's really asking for trouble. - that going to some binary hidden data format is a step *forwards* from unix. If you watch the old bell's labs video introduction to unix, the whole point was actually moving *away* from exactly the kind of 'database without root' type system that Brian here proposes. Unix was originally such a win, even having to coin the phrase 'unstructured data file' to refer to the simple concept of a file being merely a named array of bytes, because it made things 'simpler' by *abandoning needless complexity*. Just how many big minicomputer operating systems, of the type that lack the idea of a filesystem, that are contemporous to unix; survive today? - And the 'hierarchy thing'... to unix, the path on the front of a fully qualified filename is essentually just part of the name of the file. Searching meta data? yes, it's the `find` command. But organising things in ready-made concept of a 'container' that is a directory ('Folder' on classic OS, but same idea) is just too convenient that it'll ever die. - His take on admin / admin priviliges IS already what unix does. Set UID bit on files owned by root to have 'admin priv' for a program. Users being either 'superusers' or not. Unix already lacks the hierarchy of uncontrol that Windows has. If you *really* want a 'keep it simple, stupid' type system, Well, look into forth. It still exists, and is surprisingly widely used - even considered a 'secret weapon' in many fields. Things like controlling satellites, and spaceprobes where 'have you tried turning it off and on again?' just isn't feasible. Simple, dumb, capable, and really more an OS than just a 'language'. But if you want to work with files... or arbitrary length data, or development environments to get computer-aided work done; Unix has been the best choice for as long as Linux has existed - but only because the Linux kernel happened to release before the PC-compatible *BSD ones. Apple long realized it, but they are a company of engineers, not of 'professional programmers': an oxymoron in fact: every computer user is a programmer, whether they realise it or not. Every programmer is someone who doesn't yet know exactly how to make the machine do as they want: until they happen to achieve it. The moment they remember how, they're a programmer. If they formally record this into a file, in any way, they're just as much of a formal professional programmer as the guys at Microsoft. Since there is no legally recognized professional body prohibiting bad programmers from practising it, such as there is for actual professions, like medicine, law or engineering: The only true professional programmers have the title 'Chartered Software Engineer'. If your job title is 'programmer' and you derive some feeling of superiority from that: You really shouldn't... Any more than someone whose job title is 'writer' or 'cleaner'. Which is not to say you can't acheive some job satisfaction, just don't be confused about the delusions of grandeur: They're only there because you think that being a programmer necessariliy requires smarts. The whole point being, unix is and always was intended as a way 'for ordinary mortals' to use / benefit from computers. And sure, it's crufty and has its warts - just as C itself does: but if you throw it out, you're left with just iOS / Android type devices, which likewise 'get rid of' files and filesystems - at least from the user's perspective. Of course, it's ironic that under the surface, both are actually built on unix. But both essentially do exactly as you wish: replace the unix tradition. But they are no longer 'computers', they are appliances: smart-devices which are a bunch of 'apps', each 'for that'. They can have their place, but it isn't general-purpose computing work - unless just as a thin client for an actual computer. Unix has stood the test of time, and improved massively with age: There is a book called 'the unix haters handbook', all of whose cricisms are long fixed, and all of whose examples of 'superior operating systems' are long dead. At least give freeBSD a solid try before you dismiss 'the unix tradition' entirely.

[@insidetrip101]

You got a lot of good points in here, but this is the best in my opinion: "Every programmer is someone who doesn't yet know exactly how to make the machine do as they want: until they happen to achieve it." Using computers sucks, and Unix sucks too. The thing is that it sucks less than everything, and your whole point about smartphones being essentially what he's advocating for (minus the programmatic terminal interface) is true. However, while there are some terminals you can install on your smartphone, the phone just isn't designed to work that way. And using something like Termux on android is just a neutered general purpose programming experience. In fact, the only real use I found for termux was to ssh into an actual computer using a bluetooth keyboard on my phone. You can't lock down and simplify a computer while at the same time allowing it to be used for "any purpose." Those two things are contradictory. The reason why Unix is such a mess is because it is by far the most general use of a computer that you can get. Its got enough rules so that you're not in anarchy writing single threaded assembly programs, but enough freedom that you can legitimately do whatever you want to do that a computer can do. The problem is that there is a complex legal system to navigate. But that's the price for living in the space of freedom which is between the extremes of anarchy and tyranny. Living in this mean demands that you take responsibility for at least yourself. This is true in life as it is in computing.

[@svenyboyyt2304]

I disagree with you on the naming and shell language. How is saving a few keystrokes more important than actually knowing what a name means? Short names don't make something more usable, it's the opposite. You have to decipher all these very weird names which wastes way more time than a few keystrokes. Python is way better than any shell language, it's usually the first language non programmers learn to become programmers. Trying to make things easier for the user by removing heaps of syntax and making it less strict makes it faster, but also much harder.

[@alainzscheile6913]

14:07 package management at the kernel level seems like an ugly idea. That's too complex (and the kernel shouldn't handle such high-level things)...

[@patham9]

Indeed, separation of concerns is crucial.

[@l3p3]

Not when we talk about Microkernels!

[@jbird4478]

There's really nothing stopping you from making a "shell" exactly like you describe for current UNIX systems. There's also nothing stopping you from making a distro with just two privilege levels. Separate filespaces are also very much possible on Linux (containers). A request/response IPC that starts up programs is what systemd can do. Configuration in the form of a registry is also implemented on various systems. Of course, what you then and up with is a platform within a platform, and we already have many of those. That is part of why systems are such a mess. The GTK people have their ways of doing things, we have a variety of IPC mechanisms running alongside each other, we have various authorization protocols in place, etc. Most of what you describe as solutions are really just conventions you want to see. You can't expect everyone to follow your choices and limit everyone from making different choices. This reminds me of the old "we have both standard A and standard B in place, so we're making standard C to create a unified standard."

[@stupidburp]

Had some decent ideas at the start but this is dangerously inadequate from a security perspective. Any replacement should have more security controls not less. Actually I would go so far as to say that a new system should be built with security as the core concern with multiple measures at each level to bring it beyond any other system. User and file permissions is a weak and outdated method to place any trust in and causes all kinds of issues but the service they provide should not be discarded. A lot more should be added to a system beyond this. A registry is also particularly bad from a security standpoint since it is by nature a shared space. To adequately isolate programs from each other you wind up putting up walls that defeat the purpose of the registry. Or you just let any program have free reign to do whatever it likes to everything in your system.

[@TmOnlineMapper]

I absolutely agree with you. File security may not be an issues on systems with less than 5 users if the users know each other but everything beyond that needs strong security. Alone in the case that one account gets breached. In a system where everybody can access all user files this could cause devestating damage to all user data and something like that is plain not acceptable.

[@angrydachshund]

The Microsoft Windows WinSxS system already implements this kernel package management using UUIDs and version hashes. It guarantees that all versions of shared apps and libraries are always available to all consumers on the system. WinSxS has been shipping for ten years now.

[@alderin1]

And WinSxS is why a fresh install of Windows7 Professional needs 40+gb of disk space filled during two full days of update downloads, before Office or other programs are installed. I wouldn't point to that as a success.

[@angrydachshund]

+alderin1 - Dude, it's 2017, and you're still fretting about 40GB? In exchange for 100% backward compatibility across the entire operating system? You linux people have weird priorities.

[@alderin1]

+angrydachshund - Actually, this "priority" was handed to me by my employer getting 60gb SSD drives for workstations to speed things up: The OS taking 40 of those 60 doesn't leave enough room for user data. However, the point was WinSXS isn't a success, it is a bandaid on a foundationally flawed system, and just because storage space is getting cheaper doesn't mean my operating system should try to take up the same percentage of space. Finally, sadly, I work in an area that often has the edge cases, your "100%" is incorrect, and in my experience is closer to 90%, unless you are talking about other Microsoft products.

[@HowardCShawIII]

We did some of these suggestions before - LISP machines, Forth machines, etc. And Android Intents, SOAP, JSON-RPC, GraphQL queries... if we don't have a way to describe piping data from one small bit of code to another, we invent it. Again, and again, and again. "Those who cannot remember the past are condemned to repeat it." - George Santayana Whether you buy into his philosophy or not, we have gone from massive shared machines, to personal machines, to largely shared machines, and back again multiple times. I have customers today who have every desktop running a Terminal Services session on their server.

[@tachobrenner]

Don't forget BASIC running on home computers!

[@dabzilla05]

what are your thoughts on NixOS? Seems like Nix really tried to do this exact thing, with the added feature of utilizing immutable data structures on top

[@mskiptr]

There's also GNU Guix. It goes one step further by replacing the omnipresent shell scripts and all the bespoke config files with just Scheme code.

[@0ptikGhost]

I find it interesting that most cloud infrastructure really attempt to implement most of the proposal presented here (on top of current tech) and, for security or organization, we build all the stuff the proposal attempts to remove right back on top of the platform that doesn't have them. To me this suggests most of the pain may just be what security requires. Concretely, UNIX and other referenced initial platforms were typically developed in the absence of security and security was attached somewhat ad hoc through the years. For a proposal like this to truly work security must be worked in from the beginning. Trying to remove all those hairy parts does not bring in security. You're not just protecting against local users (who really should be protected from one another otherwise a non-administrative user can corrupt an administrative user intentionally or unintentionally and you get disaster). Minimizing the affected scope of breach is a good starting point but communication is key. We need communication for effectiveness but we must restrict it for security and stability. Finding that balance is tough no matter what the underlying platform looks like.

[@kjgoebel7098]

Warning: Statements in this video make a dangerous amount of sense.

[@brettbroadhurst4211]

You've done videos on problems with the status quo of software development like OOP and the mess of modern operating systems. You've opened my mind to ideas of what could be done to create better software. Do you think you'll do a video on the state of the World Wide Web and the overextending of technologies? I'm very eager to hear a response.

[@jbmw16]

All that is solved by having browsers adopt a well done language.

[@Spiderboydk]

@@jbmw16 How? You don't fix overextension by continuing the overextending.

[@byteme6346]

I wish, instead of WHINING, this turd would DEVELOP something, like a SOLUTION.

[@ttheno1]

@@Spiderboydk i think he means scrapping javascript for something not shit replacing, not more overextension

[@Spiderboydk]

@@ttheno1 Probably, but that wont solve it. No programming language should ever have been jammed into HTML in the first place.

[@jontymorris8200]

> *ls -la > foo* While others see bloated piles of complexity, I treat it like fine art or poetry. With just 12 simple characters, we can instruct our silicon machines to do multiple operations. Truly amazing!

[@ricardo_valerio]

same!

[@lastmanstanding5423]

yes!

[@Noname72105]

@Barry Manilowa Is that really a hard question? If it existed before the ls operation, yes. Else, no.

[@Noname72105]

@Barry Manilowa I mean, fine, but it doesn't take more than 5 minutes of experimentation to learn how it works. Or the damn manual.

[@wopmf4345FxFDxdGaa20]

​@Barry Manilowa Most shell users are naive users only for a brief period of time and after that the power of the shell becomes more valuable to them. For example, let's image the command shell was replaced by Python. How exactly is it going to make it easier to understand to naive users, who don't know Python programming at all? Maybe there would be "ls" function, and write to file functions, but I think the problems would get more complex, not less complex to a naive user who doesn't know anything about programming, functions, function calls, variables, function return values, execution order, etc. So the problem of execution order for example still remains, you just move it to a different place! I agree that Python is a nicer language than shell languages for scripting and doing anything more complex. But the current concept of working allows you to do that, it allows you to use the tool that best suits your use case. I think the concept of pipes and processes is rather simple and efficient compared to how you would do it in a programming language, where you have to think about variables and their types, and take certain amount of letters at a time, make special cases for this and that just to handle the character stream like a simple shell pipe does it.

[@IslandHermit]

One big problem lies in identifying what is part of configuration. Even the most conscientious developer will often fail to identify all the elements which comprise their application's configuration. As a result, third party packages which need to interact with that application may find it necessary to rummage around in the application's private namespace to find the configuration information they need. The alternative - petition the application developer to update their config info and hold up your release until they fix it - is rarely practical. In my experience this sort of "configuration shortfall" is responsible for 20-30% of the complexity I face in getting packages to work on my system and in creating packages of my own. All existing systems suffer from this problem. Your suggested scheme will as well, although its greater emphasis on configuration might mean that package developers are more rigourous in identifying the elements of their package's configuration.

[@cleverclover7]

in my experience the "solution" is to use containers and virtual environments. As you say, no mater what you do, configurations can change when you install some new thing or version sometimes without you knowing. Of course this absolutely adds complexity, but I don't don't subscribe to the assertion of this channel that more complexity is inherently bad. Complexity just has to be met with good abstraction. ls -al > foo.txt is a good abstraction.

[@ianhamilton350]

The straightforward solution to the majority of package management issues is to statically compile our programs

[@PeteRyland]

This is fine if you have good control over your dependencies. But if you have good control over your dependencies, it's then also unnecessary to do static linking. In other words, statically compiling everything simply proves you don't have good management of your dependencies. Perhaps you don't care about good management of dependencies. In that case, you can save a lot of time by just developing directly on production.

[@JrIcify]

@@PeteRyland People can think whatever they want about my management of dependencies.

[@PeteRyland]

@@JrIcify Should I also imply that you don't care about security either? Then I'm pretty sure I'm not going to run your software on my systems, thank you very much. :-)

[@krunkle5136]

Compiling statically just sounds like giving up. You waste more memory too and it's not elegant to have multiple versions of the same library on the system that each have to be updated by recompiling all the software, where as with shared objects you just need to update that one file.

[@ME0WMERE]

good idea, until you realise how fast your disk space is being consumed

[@substance1]

Unix was created to be a pipe and filter architecture and the shell was the means of using it. The Unix philosophy is about small programs that do one thing very well and piping them together to create an application. It is a brilliant idea for an amazing amount of productivity before a windowed environment. And Unix is meant to be a multi-user computing platform, that's why it was designed with a file and memory security. In Unix, everything is a file, this includes ports, terminals, printers, etc. So, the file security model can be used for everything on the system.

[@monad_tcp]

Those are principles, they are good. The problem is the implementation.

[@substance1]

@@monad_tcp In 1969, it was a monumental leap in that it enabled sharing "general purpose computing" on various systems.

[@monad_tcp]

@@substance1 that's history. We learn from it, not worship it as perfection.

[@substance1]

@@monad_tcp It works great for Android.

[@monad_tcp]

@@substance1 because Android removed completely the user mode. Such irony.

[@disecke]

I can't say I agree with everything here, especially the idea of dispensing with user-level protections from other users. Its important to remember that users represent separations of concern, not just human users. Ex: a mysql user or a www user. I'm also not convinced on the registry store as being better than /etc or keeping config files in a individual user's namespace in plaintext files. I think the points about package management are the strongest here.

[@jkenkane6188]

Oh good, I'm not just some crazy weirdo in thinking that users should have some "space" to themselves on a system!

[@shadowleague2486]

All shells should be replaced with the LISP REPL.

[@molewizard]

Actually yes, and use spaced lisp so there's no issues with unbalanced parentheses

[@gogl0l386]

Haskell LET'S GOO

[@humm535]

Exactly what I though: He seems to like LISP machines.

[@RealTimeArchive]

This, or Haskell

[@gjvnq]

Spotted the LISP Machine user! :P

[@BrandNewByxor]

This talk reminds me of Terry Davis' TempleOS. I think the terminal exclusively parses Holy C as opposed to some scripting language, and has the ability to improve the readability of output in small ways, e.g. printing images and 3d models to the terminal... lol. Edit: Should have guessed, people have already said this a long time ago

[@scottfranco1962]

Some comments: 1. Shell languages are certainly a mess. I am amazed that people still write complex programs in bash which was written by someone who clearly skipped language syntax classes. However, shell languages spring naturally from the idea that it is a simple increment from command line execution to scripting. This certainly means that shell languages need to be compromised somewhat (but not to the level of bash!), and that interpreted languages such as python will go their own way from shell languages. If you don't like ls and the command shell, you can always use a graphical file explorer, but they don't script and their output cannot be piped and manipulated. Ie., lets not throw out the baby with the bath water. There is a reason programmers prefer command line environments. 2. Dependency management in Linux became a serious problem due to a combination of "everyone should install from source" thinking combined with the tragic decision to include DLLs or .SO modules, aka "dynamic link loading" from Windows. This was a technique from the bad old batch days of the 1960s, brought forward into the days of demand paged virtual memory, where it makes no sense whatever. Does it save memory space? Yes, but page management makes that irrelevant, and DLL techniques actually work counter to virtual page management, which works best if you have unmolested binaries on disk. The real issue is that cross package referencing (either DLL/SO or library, or program) creates N different combinations of program run environments, and that is an inherently wrong idea. You don't want your users to be testing configurations that you haven't tested. The gain from this idea is that the user *might*, repeat *might* see better functionality via a new package upgrade unrelated to the package at hand. I would assert that it is far more likely that you will see a new bug than better functionality. The alternative to all of this is return to static program linking. The configuration will be fixed at the package creation by its developer, and that configuration will be tested and unchanging. Libraries get compiled and shipped with the product. A program that is needed and is open sourced gets shipped in a subdirectory, eliminating the need to go find it, worry if it is still compatible, etc. The binaries get bigger, but who cares? virtual memory management takes care of loading only the actively used sections of a program. 3. Registries moved the configuration data from individual files to a central system. Even if the Windows registry was organized, one of the biggest issues in WIndows is the central registry becoming corrupted, which brings the entire system down. What point did the central registry support? What point does Linux having all of its information in the /etc tree support? It does not make their configuration information look like each other. The info is divided by filename, and increasingly, by different directories as programs and devices need complex support files. My gold example here is Eclipse. This program needs no install program. It lives in one tree. You can copy a working tree from anyone to another machine and it just WORKS. All its configuration info is in that tree. Delete it and it is gone, there are no registry or information files outside. Want to be sure that you got rid of the old program before putting a new eclipse on? rm the tree and copy. Want multiple versions on your system at the same time for compatibility reasons? Make another tree. They won't fight over registries or info files elsewhere. 4. I would not be so quick to declare multiuser systems as dead. We use *lots* of shared machines, some with multiple users at the same time, some not. Even if the system does not have multiple attached users, each user has an account, and the user paradigm is valuable.

[@tibfulv]

Indeed, DLLs made a lot more sense back when hard discs were smaller, but these days, when terabytes are routine, not having a statically linked system can't be justified. And it's a lot more difficult to do multiuser if one doesn't have a multiuser-capable os. Having Unix at hand means one gets tempted to use it as a multiuser system when one can, because it's what big systems do, and big systems are inherently cooler. Thus, everywhere Unix lives, multiuser lives too.

[@sidjoosin6549]

Actually if you look a little deeper, "linking" as an idea is pretty stupid, static or dynamic. You need just executeable files - binaries (not modern 1 mb bins, but real ones - tens, hundreds Bytes) , that can call another binary. You can put all that bins to RAM, making them available for being called, and you can have full cache on disk, in case that there is no needed function in ram cache (which is smaller) and you can have remote "cache"- repo - in case if function not found in ram nor disk it just download it from your remote repo and neither disturbs your again nor or throw error - all needed package managment and linking with more efficieny and less size

[@GegoXaren]

Dlls are needed as if one library has had a security update you'll need to recompile ever peice of Software that uses it if it was statically... And that costs years for libc in a standard in cpu time. It does not work.

[@jess-sch]

@@GegoXaren years? yeah, probably for glibc. glibc isn't the the only kid on the block though.

[@GegoXaren]

@@jess-sch It is the same for ulibc too you tosser.

[@drstrangelove09]

Yes, there are times that it helps to know the complex operation of processes being forked off by the shell but it is very, very rare, in my experience. I've been working with Unix/Linus since the late 70s (Unix) and I've only needed to think about this a few times and even then it was at a surface level. I'm with you about the difficulty of installing software but I'm not sure that the OS is the cause...?

[@thought2007]

To understand 'ls -la > foo' you don't really need to know anything about EXEC or file descriptors or anything like that. Really, you just run "ls -la" and see that it spits out the result on your screen. Then, you run "ls -la > foo" and observe that it gets written to a file called 'foo'. That's all you have to know to understand that. The rest of the details are just... details (you can learn them as needed).

[@thought2007]

+MasterGhostKnight Sounds like you seriously need a new work environment if your boss is raping (or rapping??). If what I said is wrong please say how. The point is that you don't need to know all the details to get 98% of the value of shell redirection. Are there edge cases? Sure as hell. But then for those you can develop guidelines and things like static analysis/best practices, and so on. Or refactor (when your shell scripts morph into gnarly hacks -- refactor them into high-level language programs).

[@DavidRutten]

+thought2007 , unless something goes wrong. Where do you start then without any background knowledge?

[@thought2007]

David Rutten Yes. When something goes wrong the shell gives you an error report like "foo: permission denied" and so on. Try this in Python and you'll get a 30 line backtrace or something else instead.

[@jeffwells641]

+David Rutten That's the OP's point, that thought2007 is totally missing. And everything breaks at some point, especially on Linux.

[@thought2007]

Still waiting to hear why you need to know this just to use the command 'ls -l > foo'. That's the point. Going on about details is like saying you have to be a mechanic just to be able to use an automobile. Utter nonsense. If your Unix command line breaks, take it to the local guru. Not that hard.

[@ianhamilton350]

Your reasoning for why 'ls -la > foo' represents a huge amount of complexity is the equivalent of saying that your car shifting gears is a hugely complex process due to the inner workings of the transmission and the nature of how it interacts with the rest of the vehicle. Yes, it's an accurate statement, but does that complexity really affect your experience as a driver, or inhibit your ability to make use of such a mechanism?

[@ianhamilton350]

I should point that I do agree with you that the shell seems like a 'giant kludge'. It is not a clean to use interface in many cases, but it's still an incredibly powerful tool that is a lot more messy on the surface than it is outdated

[@abigailpatridge2948]

To your last: Yes, I disagree with you, and yes, but I've already been doing so for a long time before ever watching this. You seriously praised systemd unsarcastically? Really? And then suggested that we force something that is very much userland into kernel, not just backtracking all the work done recently to take cruft OUT of the kernel that belonged in userland, but going even further than Windows ever did into monolithicness? No. We should NOT put package management into the kernel. At all. Ever. Doing so means a reboot with EVERY software change. Microsoft is bad enough on that score, this would be worse. I won't disagree with you that there's a lot that needs to change. I also won't disagree with you that most package managers are bullshit. Have you ever tried Gentoo? I think it manages to get right most of what you're complaining about without introducing quite as many of the drawbacks you don't seem to have recognized to your prescribed approach.

[@saymehname]

There are OSs with the ability to edit the kernel without a reboot. Brian is right. You are stuck with the Unix state of mind and you can’t see outside of it. Learn more about other operating systems to see how the world was before and after Unix came to the world.

[@thosdot6497]

@@saymehname - no, you're missing the point. What's being suggested here is a complete re-imagining of a computing interface and OS. There was no need to actually mention Unix or Windows here at all, except that I guess some starting place was needed to push off from. The hubris on display is staggering, but that's par for the course, and there's little respect shown for the pioneers who were working within a very different environment. You also missed the point about systemd - a Unix state of mind is perfect for a Unix system. Which is why systemd is the wrong approach. This new suggested thing might be an interesting place to be but I wouldn't take a Unix mindset to it, it's not Unix. It's so different that you'd be forced to approach it with an open mind, or fail. Wills is completely disengenuous if not being decepting about 'ls -al > foo' though. The point of that syntax, like any command interpreter, is to hide the complexity, make it readable and repeatable for ordinary users. There will be huge complexity in getting any OS to do something like that. Even in this new call-response model with windows for responses and links and all that - how much complexity is under the hood? What does the kernel look like and how is it going to be any less complex than the way Unix does/did things? Hmmm?

[@SasLuca]

I would love if the shell for Unix or any os used a C like a syntax, why "ls -la > foo" and not "writeFile("foo", listDir())"

[@jakykong]

I agree with you that naming could use some updates, but generally there are two major criticisms I have of this reasoning: First is that the historical 'cruftiness' is not really a symptom of a disease; it's an effect of evolution. Designing a system to replace it means re-learning all of the various reasons why the system is the way it is, which violates the generally good assumption that you're standing on the shoulders of giants. The second, which to me is worse, is that you're mixing abstractions. Your first example - "ls -la > foo" - goes on a long twisty path talking about implementation details. Those implementation details are generally not relevant when debugging a shell script, unless you're working on severe edge cases. But - big but - edge cases like that crop up in every programmable environment whatsoever, it's just the nature of the beast that computational reality is a bit messy, and nobody has yet devised a system where that isn't true. A final consideration is that there is usually a pretty good reason for the oldest design choices in Unix environments. One of them stems from having a *very* simple process model, which is: Every process (except init) has a parent process; everything that isn't a process is a file; and text formats should be standard interchange where possible. Put those together in an environment where you don't have modern luxuries, and you wind up with terse, file and text-oriented commands a la Unix. So far, nobody that I am aware of has managed to produce a working shell model that doesn't follow this pattern and also isn't a huge mess of extra typing, or so particular to the system that it becomes a blob anyway.

[@jakykong]

Consider what you really need to know for that shell example: You need to know ls is a command, you need to know "-la" is an option, and you need to know > is the redirect to file operator. If you know those, you know everything you need to get started -- and those are really just vocabulary items. If you have some edge case you ran into, you'll get an error back from your shell describing what the error was, or, just like in any programming language, you'll go down into the rabbit hole as far as you need to go to debug, and then you won't make the same mistake again.

[@Michael88324]

TL; DR: a better thing hasn't been done; "giants" have made the bad thing so there must be a good reason; Both are weak arguments. First, they are unprovable. Second, they don't lead to any actionable conclusion but to avoid solving the problems

[@IanKjos]

What you've described is basically the Macintosh platform prior to the switch to a BSD core. Nowadays they actually solve a lot of the traditional unix-ish problems from a normal-user perspective through a strict set of conventions about where programs and user data live.

[@bobweiram6321]

Sadly, they devastated the spacial UI that was the secret sauce that made the Macintosh's UI revolutionary and unsurpassed in terms of usability and productivity. I really miss the system folder where you know which files do what and what application they belong to by just looking at the icons. Nowadays, you have hundreds of directories littered with hundreds of files with cryptic names and no clue as to what they are. It makes it very hard to troubleshoot an issue, remove, or install additional functionality. They also adopted UNIX's directory structures where its organization for the system rather than the user. Most of elegant and beautiful abstractions which hid the ugly and distracting details of the OS are gone forever.

[@tachobrenner]

​@@bobweiram6321But the stupid thing stopped being buggy and finally has multitasking, so that's a plus.

[@blarghblargh]

Some things that seem to be missing from the description, and that seem pretty important: Side-by-side versioning (because programs change and stop supporting certain kinds of messages over time). How to avoid GUIDs when figuring out which file I want to reference (e.g. module includes). Meta-facility lookup (which C++ compiler do I run, which text editor, etc). Any standardized way to capture program runtime information for debugging (e.g. log files written by stdout, etc. Or does logging to a file just work basically the same way - you just don't tend to chain things via text?). How you implement something like a system-wide database program (that has a shared state between users, but really shouldn't be running as admin).

[@IslandHermit]

The public port problem might be solved by having a service registry which maps dynamically allocated ports onto universally recognized service identifiers. That way incoming connections will only need to know the standard ID a given service, not the IDs of all the different packages which be used to provide that service. In some ways the existing port numbers already get used as a sort of service identifier, however the port number space is too small and the methods used to ensure uniqueness too ad-hoc to avoid significant conflicts. Switching to UUIDs would almost completely eliminate this problem.

[@lonjil]

You're a bit wrong about Plan 9. The idea was to make a consistent cruft-free system by applying the "everything is a file" idea for real, so every API is a set of special files controled by a server. This together with mounting remote folders locally gives network transparency. Now, the developers did of course experiment with this, but only as research on top of something more fundemental.

[@briantwill]

+lonjil Yeah, I shouldn't have said it was the main objective. I think plan 9 failed for being both too ambitious and too unambitious. The more consistent use of special files was probably an improvement, but not different enough to overcome the incumbency of existing Unixes. Meanwhile, the network transparency stuff was half-baked and of nebulous benefit.

[@cd78]

+Brian Will Hey Brian I was doing ruby on rails last year and it was fun. I started first year at University doing a software development degree and the launguage is java. It's so fucking retarded and the enjoyment is not stimulating me as much. any advice??

[@cyberpunkspike]

Brian you are wrong, we cannot abandon the entire concept of a shell, and replace it with a general purpose language. You need a way to quickly interact with a running system, and some form of shell is an ideal answer. What we actually need is a modern widely available OS built on a micro-kernal with clean lew level IPC mechanisms,... so basically FuschiaOS, then we can iterate on user-land concepts, without trashing performance or ignoring driver and platform complexity.

[@0x770x74]

Sounds a lot like Android, with the intents and all. Still some parts missing from Android though, like a real shell

[@anytimetraveler]

Termux gives you a real shell on android. It has a package manager and runs without any virtualization.

[@minhajsixbyte]

@@anytimetraveler wow!

[@patham9]

Yes and Android is really a mess, with apps having to ship with all their dependencies, leading to a lot of duplication which proper package management avoids.

[@kazaamjt1901]

@@patham9 I disagree. Statically linked libraries really are the way forward for linux too in my opinion, because otherwise packages start holding each other back. Oh can't install this version unless I install glibc version this much, but than these packages break because they require a different glibc and so on. Storage is cheap, duplication is fine.

[@tachobrenner]

​@@kazaamjt1901Yes and no. There are libraries every program uses. Core ones. They should not be duplicated. They are basically part of the OS. Others should.

[@mdashx1382]

Emacs has full-featured text editing of code with commands to execute expressions, blocks or all of the code in a buffer, with output sent to a different part of the UI, not interleaved with the code on the screen.

[@thingsiplay]

If you do not like the shell, you are free to program an alternative. There are alternatives to Bash already. So this rant is only against Bash only. In example if you do not like the namings of programs, you are free to create alternative aliases. So the terminal concept itself is not bad, its just you do not like whats provided with Bash and GNU programs. But everyone is free to create a shell.

[@smorrow]

Terminals as a deep-seated OS component are pretty bad, which is why Plan 9 doesn't do it. I agree, bash is a disgrace, but half his complaint with it seems to be the fork-dup2-setenv-exec-wait dance, which is going to be in *any* Unix program that can start other programs. Fork and friends are also generally said to be an elegance of Unix, so I don't know what his beef is or what he would replace it with.

[@thingsiplay]

@@smorrow I don't think that Bash is a disgrace. And if you do not like the Bash, you are free to change the shell. The ability to change the shell is one of the core ideas. Like with every flexible and complex system with many standards, it gets a bit ugly. But only if you are going deep. The benefit are powerful systems. In example having separate programs doing one job only needs mechanisms to connect them. So ranting on this connection system is stupid. The good thing in Linux is, you can replace the shell with one you like. In example: xon.sh/#about

[@smorrow]

@@thingsiplay *Any* shell you choose is still going to have to use the Unix process model, so changing the shell doesn't address his complaint in the slightest.

[@thingsiplay]

@@smorrow It does. And the Unix/Linux system with all its features is fantastic. No other system comes even close to its brilliance. Every other system is trash against it. Even MacOS uses the same concept. I do not agree with his complaints. He even took DOS as a better alternative. I mean holy crap Microsoft DOS and Windows are known to be the worst systems.

[@thosdot6497]

@@smorrow - well, yes. But the video's complaint was about the complexity underneath 'ls -al > foo'. All the way down. So explain how you have to know all of that to use Unix, but you suddenly don't have to know how all that works on this Utopia-OS described here? Seems to me, what with the HTML or other prettified output, 'real programming language' input, separate window areas for responses, etc, that if you're held to to the same standard, you need to know an awful lot more about more things.

[@reidpattis9478]

Absolutely love the Linux tradition, but I agree that the package management can be a hassle.

[@zyptoskid]

complexity exists for two reasons: - Backward compatibility - Outside world complexity requirements Dismissing the second for simplicity results in what's called a "toy implementation" I'm sure you know that. Your OS assumes good intent in many places (like how you said malicious users wouldn't share machines etc). UNIX model has become outdated but not entirely obsolete. The good parts should really stay. What we need isn't to dismiss outside world complexity, but to design in a way that arranges and locates it where it can be reimplemented and rearranged without affecting the semantics and the world built around them, all with modern day technology. The result isn't only a simple-where-it-matters OS, but also a relatively-secured future for the implementation. There are good solutions to specific problems popping all around to learn from. I think there's value to having many internals be (required to represent and interpret)? human-readable text. And to having hierarchy in files and processes. And I think shell languages are a great super-high-level interface. What's perhaps needed is uniformity to what these components operate on. The APIs shouldn't necessarily be plaintext, but perhaps structured plaintext, and the configs should perhaps be in a universal OS-specific format. IDK how that looks like. I'm sure it's a *hard* problem. I'm not smart enough to propose a solution that has merit. But I'm keeping my eye out because eventually the cost of designing something from scratch will be lower than maintaining this mess, and maybe then we can collectively contribute meaningfully to that.

[@xofcenter5576]

What is your opinion of the Urbit project, in this context?

[@Eugensson]

What do you think about Redox-OS? A project developed in Rust language.

[@andik70]

great presentation, just two flaws. a) who is going to do it? b) what are the incremental steps from lets say a linux to this system (you cant expect everything/all applications done from scratch)

[@adammontgomery7980]

This is an interesting topic. I've wondered why a modern, single user, hardware independent operating system hasn't been developed. Like something you can use on your phone, then dock into a workstation. You'd need a standard interface like a kernel running on each device and just switch the context between the two when docking/un-docking. Web applications try to solve this issue of a unified UX but it's turned all of our devices into web portals. It just seems like a kludge to me.

[@svenyboyyt2304]

The only difference between a phone and a computer is the screen size and user input so I don't get why everything is so different, even for Apple. Websites are just an executable on the internet, so I don't get why they're also so completely different as well. In fact they're so different, a website isn't even an executable. It's markup that can execute a script. It makes no sense whatsoever.

[@nitroflap]

@@svenyboyyt2304 Not only screen and user input. It's also the processor and the hardware.

[@tachobrenner]

ANDROID AND JAVA. Man.

[@ennio5763]

I will also lean on saying that the security view on this proposal is outdated. It seems to stem from a sense that applications are installed because user wants them, and they do what they are supposed to do, except in case of errors, which was probably accurate in the 80's or even in a modern fully controlled Linux environment, but certainly doesn't reflect what happens in a real end-user environment. In the days of smartphones, and even PC starting to feel like smartphones, we have hundreds of applications installed without user consent nor knowledge, doing untold things with accessible data, with the primary objective to invade user privacy, using the tiniest clues to rebuild a profile and sell it to highest bidder. Security and user control should be way, way more central, with default giving way more power to users as opposed to apps. The Android Permission model is a good step in this direction, and I would consider it a minimum nowadays. But only a minimum. There are still too much data that can be siphoned off by rogue applications on Android.

[@lunarolm1703]

The notion of the shell using an html-like hypertext reminds me of templeOS, which implements something like that with it's doldoc system. Temple also uses a C-like language for shell commands (and practically everything else), much as you described. If you aren't familiar with it, it's a toy OS built by a lunatic, but if you can look past that it's actually pretty brilliant in a lot of ways.

[@cebruthius]

> it's a toy OS built by a lunatic TempleOS is a temple of God! How dare you insult his Holiness Terry A Davis. You must be one of those glow in the dark dudes

[@snesmocha]

A lunatic??? Terry Davis was beyond all of us, god rest his soul

[@ahG7na4]

"every piece of text is clickable and does something" is hardly Terry's idea. Oberon was like that

[@PeteOutdoors1]

The Windows registry was created to stop piracy. In the DOS world, all the information about a program was contained in its own directory. You could copy that directory to another DOS machine and it would work. But you can't do that in Windows with a registry, because the program checks for entries it put there when it was installed. If you just copy the directory of the program to another Windows machine, it will not have the registry settings put there by the installer. The registry works to prevent copying for most people, but hackers can get around it. Never-the-less, it is a terrible idea and it should be retired.

[@alexandr0id]

I appreciate the sentiment and guts to share your thoughts! Many ideas you are describing here remind me of Android (a lot), Smalltalk VM, .NET framework and Microsoft OLE. I am sure these concepts are present in other systems as well. I agree that existing OSes are hard to learn and understand, promoting buggy programs but I suspect that complexity is not the enemy here, it's just bad design. A system can be complex and yet self-healing and easy to reason about. A well designed system is not simple, but the one which layers complexity in a sane way and keeps those layers separate. If you look at the systems which seem "simple" or try to actually build you own, you will see that they all have complexity in them, it's just with "simple to use" systems we don't need to know about it most of the time. From this regard mobile OSes are a big step forward compared to Windows, Linux or MacOS (which in many ways does good job too) in the ways they make user's interfaces "simple" and predictable compared with desktop OSes where any problem requires "popping the hood" and diving into the system details.

[@ineverknowdoyou]

the only issue that i have with android, are how hard it is to make something simple, like a simple automaton script

[@ryanshea5221]

I refer to "well designed complexity" as sophistication. Software can be sophisticated yet simple. With this definition, complexity is almost always bad.

[@cmatthews718]

Even with sync services such as Dropbox, OneDrive, and all of them -- because we want to be able to work offline and open/save files quickly, pretty much anything you're working on is going to be cached on the workstation where it needs to be protected from other users. The idea that people aren't sharing computers with personal data that needs to be protected from other non-admin users is preposterous. Additionally, the idea that out of any two users on a shared system, one of them is definitely an administrator is "not even wrong."

[@TimHayward]

I watched several of your excellent videos and wondered if you ever looked at Forth as a concept that addresses many of these concerns.

[@ruffianeo3418]

+Tim Hayward Nice trolling :) Seriously - where would forth come in? As the new shell language? The last forth I used could only write fixed sized 255 bytes blocks to disk and did not even have a notion of a file. The last I heard of forth raising from the realm of the dead is in the context of boot loaders.

[@TimHayward]

It is alive, though not well. It is in other places. It is in Postscript. You can get a new 144 core super processor with only forth, but that isn't what my comment was about. In Forth: The shell is the ide is the compiler is the loader It is painfully simple It is infinitely extensible, supports overloading, so my overabstraction will never interfere with your overabstraction. It is closely tied to the hardware and sometimes, vice-versa. What if we just made a most Turing complete architecture? What does the perfect implementation look like. Computers don't really do that much. What do they need to be told?

[@herrfriberger5]

@@TimHayward Sounds a bit like BASIC on home computers, which was equally complete and self contained (albeit far to often very slow). They also coexisted in that context (say Jupiter Ace versus Sinclair ZX-81). But frankly, I never understood why Forth had to be so wierd and untidy, compared to most algol-based languages (except the ugly c-family). One of the few popular languages I never learned in the 80/90s.

[@xcvsdxvsx]

It seems like you have this neat intersection of "the simpler the system the more it conforms to the vision" as well as "the simpler the system the easier it should be to produce". It seems like such a great idea why is there not a github project? To make it simpler start by targeting one piece of uniform hardware. Make it for raspberry pi 3 to start with since its probably the currently most accessible and widely owned piece of uniform hardware on the planet. Further the main project could be to maintain it on just the currently most widely adopted single board computer and rely on fork projects for any other hardware. That way you could focus more on the software specifically.

[@epajarjestys9981]

That sounds like a great plan.

[@jacobschmidt]

did anything ever come from this?

[@Microtardz]

@@jacobschmidt Lol, no. Ofc not. It's far easier to tear existing ideas down, and blabber on about potential replacements, than it is to actually build something to compete with existing ideas that have withstood a good few decades of people hammering on them.

[@tcroyce8128]

Read "Unix haters handbook". All this mantra of "worse is better" and the underlying crud of the system design is just crazy. So in place of fixing it, they sell it as a feature and this was back in the 80s.

[@EmanuelePaoliniMaths]

You could use python as a shell, just try it to understand why it is not a good idea.

[@neurobits]

Some good ideas. But also some that at first sounds terrifying. Like "kernel-level package management", kernel is about hard-soft interface, is a hardware abstraction, not userspace abstraction. Shell responses in HTML? I want to manage my system, not read magazines. JSON with url syntax highlight and functionality is good enough for humans, storage, processing and pipelining. Will be good to have a standard for package management OS-wide.

[@anastaziuskaejatidarjan4711]

The only thing that's really caused me any grief is the environment variables thing. I wish I could just update them and have the new values everywhere, not just in new shell instances. Proper languages like python aren't fit to be used interactively in the way that shells are used. If you encounter something you can't reasonably do in a shell, you can always hop into a python repl, though. I don't want to give up the terminal. It's not that I've learned to cope with the terminal. Rather, the terminal is too powerful a way to work, computing would suck without effortlessly piping streams of text data from one program to the next. I don't want to use a shell from a browser. If there's no terminal, how do I use vim?

[@smorrow]

Except for the very last sentence, it sounds like you don't understand the difference between a terminal and a shell.

[@anastaziuskaejatidarjan4711]

@@smorrow What gives you that impression?

[@anastaziuskaejatidarjan4711]

I don't mean any disrespect to you. Forgive me. I was responding to the video. You just gave me a copy-paste response. I assure you, I know the difference. I chose my words carefully and used the correct terms. Did you watch the video?

[@smorrow]

@@anastaziuskaejatidarjan4711 You seem to think the ability to use pipes has something to do with terminals.

[@anastaziuskaejatidarjan4711]

@@smorrow Forgive me for being unclear. The word "terminal" there is used in a certain context. The context is the terminal as "a way to work". Used in this way, terminal no longer refers strictly to the one component, but to the way the component is used, ie: in concert with a shell and with commands and everything.

[@SonnyMoonie]

The idea of using an existing kernel and writing a new userspace for it has been tried. There's Android (Linux kernel) and OSX (BSD Unix kernel) and Chromebook (Linux kernel) and probably others, maybe some game consoles or other devices that have some amount userspace apps. Every time there's a new userspace, that's one more platform that software has to be ported to, or else it's not available there or it can only be run in emulation there. Whenever there's one of these new userspace devices, the first thing that people do with it if they can is jailbreak it and get a GNU userspace and command line set up and running, so that they can install some actually good software of their own choice, instead of waiting for ports. So the proposal is just running around in a historical circle, a well beaten path. What advantage would the suggested system provide that would make it worth the hassle of porting to another platform, one that has completely different package management, files, permissions, process management, and so on, so that the software has to be mostly rewritten? I don't see one, honestly, (other than you're promising that you'll use hashes in package management, to ensure that files are what they say they are, which is something good languages on good systems can already do.) The area of security, where this proposal falls short badly, contrasts with the opposite extreme security system, GNU Hurd, which gives every program a sandboxed limited view of resources (with users and programs not being allowed to grant permission to more than their own limited view.)

[@PaulSpades]

You're confusing kernel functionality with userspace. GNU Hurd is a kernel - and it should be concerned with security, the userspace shouldn't - it should already be sandboxed by the kernel.

[@derekmcdaniel6029]

Huh, fail to see how making a new userspace be the preferred environment for new software would eliminate the ability to run legacy software. If we can do better, it's at least worthwhile to explore what it would take. If it makes it easier to write better software, that's a huge benefit that should not be ignored.

[@personaderuso3459]

Paul-Adrian Stoleriu Hurd is used to refer to both userspace as well as the kernel, where Mach is the kernel alone. Hurd CAN be ported, and there have been attempts, and personally I’m waiting for the seL4 port.

[@hgu123454321]

On any given computer, there is really only one thing of any value, and that's the users' data. The rest (programs, OS) can just be reinstalled, but the data must be protected. This is the complete opposite from UNIX, which protects the operating system first and foremost, and leaves your data at the mercy of a single mistyped rm -rf / or malicious program.

[@JaLikon65]

I know I'm 8 years late here, but I have been watching your videos about Low-level Unix stuff, and man, I'm just so appreciative. You present SO MUCH content, so concisely and well explained, it is just incredible. You and Jan Schaumann (from the 'Advanced programming in the Unix environment' series here on YT) are the ONLY people I have seen go so far into Unix with such clarity. It's just awesome. Also, I find it strangely comforting that even someone who is so clearly knowledgeable about Unix also sometimes gets the File descriptors for STDIN, STDOUT, and STDERR mixed up too. I do that all time, nice to know I'm not alone.

[@kugurerdem]

I just took a look at Jan Schaumann's Advanced Programming in The Unix Environment series - they appear to be very promising. Thank you for taking the time to point that out. I intend to look into it more closely when I find more time. :)

[@JaLikon65]

@@kugurerdem No problem! They are indeed FANTASTIC videos, I can't claim I've seen the whole series yet, but from what I have watched, they have been super informative. I'm happy to spread the word :)

[@JimHansson]

I think maybe guix package manager work somewhat in the way you describe. at least i think you can depend on a specific hashed version of a package.

[@danielbarnes3406]

It sounds like a library OS or unikernel, which is becoming more popular lately, would be a great first step. You get the application virtualization you need plus an API. There may already exist research similar to your proposal.

[@Zantorc]

I'm glad I came across your video. I've been thinking along the similar lines for some time now. I used to work on operating systems in the days before Unix, when they were a lot simpler. It seems to me Linux/Unix became complex to incorporate features in such a way as to minimise memory and processor requirements for a concurrent multi-user environment - constraints which just don't apply any longer (I'm the only user of my PC and it has 64 GiB of memory - as opposed to 16 KiB and one user or 2 MiB and 20 users). Now that I'm retired I might get round to looking at it.

[@jbird4478]

Constraints may not apply to you anymore, but that doesn't mean it applies to everyone. Our home server is a shared device. I wouldn't want my kids being able to go through my files on there. And lets not forget the developing world. Plenty of small businesses there use a multi-head setup in their offices so multiple people can work on one computer simultaneously.

[@scottanderson2871]

I love the fact that you are at least thinking about these things and looking forward to what an OS can be like. Unix, Linux, Windows, mac... these are not the pinnacle of computing. We can do better.

[@byteme6346]

Develop something or stop WHINING.

[@zubirhusein]

The HTML-command output is kinda silly IMO....half the point of the terminal is so you don't have to click buttons or follow prompts to do basic tasks.

[@TAWithiam]

+2chws exactly. I'm just imagining having a console that can display JFrames (from java) inline.

[@0xCAFEF00D]

+2chws I don't really think it's sensible to build a system around the fact that programs COULD do bad UI design. Nothing about the proposed form presentation would make you unable to just use the keyboard. For instance you'd run enter value, tab, enter value tab. Inspect the form and then tab enter on the submit button. For instance. I'm much more concerned with the fact that to make this idea realistic you'd have to make a POSIX wrapper for this message system essentially. Because there's no way people would abandon all their tools. That's a far bigger user-convenience threat than people starting to write UI that forces mouse use. Hopefully the system would encapsulate the old programs effectively making the request-response system the only interface a modern user has to deal with.

[@zubirhusein]

MrSnowman yeah but so many programs are designed to be completely non-interactive and have so many options that trying to make them interactive will be futile and not make any sense. I just don't see the point. The only thing I see this as useful for is maybe a replacement for ncurses, that's it.

[@0xCAFEF00D]

2chws I agree I don't quite see it being such a major feature that it'd be worth mentioning alongside other stuff here. Perhaps he has some plans for closing the gap between power-user and normal users somehow. I have no doubt that if you could just wrap CLI programs in a HTML form easily you could get normal users to use those programs more. If not for conveniences sake then just to make it look less scary. It's pretty clear to me that normal users can make good use of a lot of CLI applications if they wanted to explore it.

[@JeremyTaylor]

That's a really good point. For one thing, you want to be able to write a batch file that can run without user interaction. I'd assume you could simply specify "run silently" at the beginning of a file or block, but still.

[@TheEvertw]

The shell is NOT a fundamental element of Unix. It is trivial to create a Linux distribution where the default user shell is /usr/bin/python, just as easy as switching between bash, ash, csh, sh etc. You can also set it as a personal preference. But I think that using the shell is more beginner-friendly than using the python interpreter to perform the same tasks. That a lot of system sofware in Linux has been written in the shell language just shows its strenth.

[@gum8191]

Nah.

[@bazoo513]

So, we will not have the environment (a set of name-value pairs), but a configuration (a set of name-value pairs) ? We won't have a shell, but a ... shell ? The problem of protecting one user from other users is solved by ... declaring it not being an issue? Etc.

[@skepticmoderate5790]

I think the benefit of centralized configuration is that the OS can enforce that programs not step on each other. So, for instance, if my text editor stores config in ~/config/.editor, but my paint program also attempts to do so, this is an issue. With centralized configuration, this is all handled by the system, so the configuration is associated with each program in a more structured sense. I definitely don't agree with his single-user fantasy though. We should have more security, not less.

[@bazoo513]

@@skepticmoderate5790 Tell me, do you consider Windows registry a part of the problem set or the solution set?

[@AZEMBadlen]

What about just making a shell layer that will translate it self to bash or another terminal language so we can type like "list with size to foo.txt" which will translate to "ls -la > foo.txt". Think about it :)

[@tibfulv]

Indeed, for those needing readable commands, it would just about be perfect.

[@hwfjj1989]

But adds more complexity which I thought we needed less of. /endsarcasm

[@AZEMBadlen]

Oh this was 5 years ago. Why was I here 5 years ago. I found nushell, that somehow simplifies shell a lot, check it out.

[@ArtoPekkanen]

One Point: shell language has been designed so that you do not need to use parentheses when invoking a command or a method. Instead, arguments to functions or command programs are separated by white space. If said arguments would include white spaces, you quote the arguments. Very simple, very intuitive, and readable. It would be awful and stupid having to type in parentheses, commas and all the other syntactical sugar just to invoke a specific command with specific arguments, like you would do in languages like Python. You are right about the fragmented naming conventions in command names and command arguments. That was never fixed, and now it cannot be fixed. But that doesn't mean that you cannot create correct command line utilities that have intuitive name and calling conventions.

[@Gooberpatrol66]

>very intuitive Not really. You have one set of rules for one type of arbitrary data, and then another set for the other type. It's meant to save on typing, not be intuitive.

[@xXxNoisemaker]

Look at Haskell: a programming language with no parentheses but spaces around function arguments.

[@MrBebopbob]

Hi Brian, thank you for publishing so many informative videos. While I do not always agree with your arguments, I always feel like I have learned something useful from your videos. As an avid Linux fan, it seems to me that the variants of Linux are the result of an army of volunteer programmers. What you want is an entirely new OS. Wouldn't it be more productive to have a discussion from a more neutral stance. Linux is not really a "plug-and-play" operating system meant for the casual user (despite efforts to package components ala Redhat, etc..). My love of Linux is precisely the freedom to modify (through shells) my environment and to use shells as "glue" for engineering tasks involving many programs. I love the fact that no one (read Microsoft) is going to swoop in and remove features which I use. If you are talking about a "new" system, all of your comments are valid. I just don't get the point of picking on Linux, it is not the 900 lb elephant in the room. Thanks again for all your efforts. Bob

[@user-uc4ll6kx1g]

+MrBebopbob Well... at the beginning of the video he said all OSes we have are too complex, but then he talked only about Linux.

[@marcossidoruk8033]

This is fallacious, the fact that linux is an OS made by an army of developers doesn't justify the clusterfuckness of unix shells, this clusterfuckness doesn't come from the fact that linux is an open source project but from trying to maintain as it is everything about the unix tradition, wich was the point of this video and I am quite amazed you missed it. "Linux is not really a plug and play operating system meant for the casual user" yes it is, or at least it should be. Linus Torvalds himself said that one of the biggest problems with linux is lack of standarization that makes it harder to setup for common users, an operating system should be both easy to use as well as versatile and malleable when in more experienced hands, those two things are not mutually exclusive at all and you thinking they are is a direct attack towards Linux's progress as an operating system. "My love in linux is precisely the freedom to modify" again, how does this have anything to do with the fact that the unix shell conventions are outdated and overly complicated? It is completely unrelated, you can have an operating system that allows for that that doesn't have the unix nonsense. "I just don't get the point of picking on Linux, its not the 900lbs elephant in the room" exactly, because it is not the 900lbs elephant in the room it is the one we should care the most about improving, just because you are a mindless fanboy who thinks linux is utterly perfect and there is no way of improving doesn't mean the rest of us Linux users are that dumb.

[@tachobrenner]

​@@marcossidoruk8033The shell is never nice. Seriously. It really depends on what your preferences are. Bash is not the only shell too. Just the most popular.

[@TheMinustarian]

The shell experience you describe makes me think of Powershell where the output If cmdlets are actual objects, and not files à la *nix. Something like: Get-ChildItem *.png | sort Length -descending | Format-Table name, length, creationdate

[@abigailpatridge2948]

And I HATE Powershell for this among many other reasons. It's a horrid shell that's even worse than the less programmatic DOS-like command prompt it replaces.

[@donaldboughton8686]

If UNIX/Linux is over complex where does that leave windows? The inter connectivity diagram for Linux is less complex than for windows.

[@pid8307]

What you propose is not entirely clear because the complexity would require many books to explain the fine details, which are fundamental to such an endeavor (1% idea, 99% execution). But I can see what you mean. People may agree on some problems, but that doesn't mean anybody would agree on the solutions you propose. And there is a very clear vision and philosophical standpoint behind many choices on Unix. If you can't agree on 90% of what it is, just use another OS. You can't change such profound things as the very *defining* foundations of something (e.G. the filesystem theory of Unix). Doing so would mean that you have a completely different thing, and not the "same but better"! You may as well start from scratch and at that point not look at Unix at all. That would make much more sense.

[@ernestorodriguez6445]

No user isolation, well, on a personal computer maybe it is fine but when it comes to servers or super-computers they do have value. I mean, for one university class I got access to a supercomputer doing important stuff. Would have been a shame that I mess up all that important work bc I don't know how to code.

[@keplergelotte7207]

Solution to all your issues: don't open a terminal window

[@Lion_McLionhead]

Shells are here to stay. If you program microcontrollers, a shell is the easiest interface to implement.

[@RealTimeArchive]

Lisp clone is easier

[@SpTh2]

The entire terminal/shell part is a mess. It's like calling a complex mathematical equation horrible just because there's a learning curve to fully understanding it. In fact, theoretically, there isn't even an argument to be made here. If, let's say, languages like bash, bad as they are, reach a point where they hold everything back, you just come up with something more efficient. That "you" could be 100 researchers or it could be a single person. As for commands being complex, the same exact thing applies. If you want to use your computer at its outmost efficiency, there is a tradeoff. If you aren't willing to bother with any of that, this is why GUIs became a thing. And I think it's also worth mentioning that the GNU utilities are much more feature rich than they "should" be (100 extra options and whatnot), making the documentation quite enormous. If this is a huge problem, you can always try out the more "minimalist" Plan 9 or even bsd utils.

[@mactheknifeguinness]

Your opinion on gnu guix?

[@morgan0]

17:31 i think permissions shouldn’t be levels but individual options. can it access files anywhere? can it access the webcam? can it install or modify packages? can it steal focus? can it make windows? can it use network connections? (maybe both a limited mode and a free mode) etc. that way, what a program can do can be controlled more directly, with less confusion over what it does. and the more granular the better, simply make groups of them for the user to see in the normal view. if my network filter program needs full network access with the ability to run all connections thru it, it shouldn’t also get webcam access included if it doesn’t need it, and vice versa.

[@OSSMaxB]

I agree with many of the problems that you described: * Shell syntax is horrible, especially for more complex stuff than just starting programs. And yeah, having to learn how to properly escape all the arguments etc. is a pain in the ass. * Shell output could be something better than just plain text with some colors. One big improvement that could be implemented right now would be to use Markdown for the terminal output. This would be backwards compatible with current terminal emulators but would also provide nice formatting for terminal emulators that support it. * More and more programming languages have their own package managers. I think this isn't a fundamental flaw of how modern systems work, though but only with current package managers. * And some more that I don't remember. Now some questions about the architecture of your model: * If all processes can see all user data, how does the security model work? Having every process only see it's own data, I could understand that, then you could use IPC mechanisms to pass user data to the processes, but letting a process have access to all the users data seems unreasonable. * How are those Globally unique IDs assigned and managed. This sounds like it will become either unmanageable or fail in another way like some entity having controll over every ID or some failing model like the current CertificateAuthority system. * How in the world can you create files with a global UUID? And if it isn't global, what happens if you attach a USB flash drive with a file on it that has the same UUID? * Having permission groups might not be necessary in most home computing scenarios, but when you have hundreds of people working on a project you can't just give everyone access to everything. But this might then be a server side problem that lives outside of the client side computers, so this might actually work because it doesn't need to be part of the low level desing of the system. * How do you manage hardware access without a permission system. You can't just let every program access every piece of hardware, this would be disastrous from a security perspective. Btw: Why even use a global registry for configuration if you can just let every application have it's own configuration and expose it over the IPC mechanism if the configuration options are required from outside the application? Also I think that most of the problems you describe can be properly fixed on top of the current system, or rather by gradually changing it without having to throw out everything and start from scratch (just like refactoring in programming). As a sidenote: Your model has, in parts, interesting similarities to what is already done in modern webbrowsers: * The "shell" uses a modern dynamic language (javascript). * There are no files, an application can store it's data in the local storage (which also serves as a registry for configuration). The UUIDs would be the reference that programs hold to the data. Although JSON has a similiar tree structure to file paths. * There is no process hierarchy. * I could probably find more. And it even works across multiple platforms.

[@OSSMaxB]

+shevegen My whole point was that most of the problems described in this video can probably be properly solved on top of current unix systems! And if some problems keep existing, the systems can be adapted without completely throwing them away. EDIT: Well, not my whole point. But I still think this is possible and it would be much more reasonable to actually implement (in terms of effort required).

[@briantwill]

+Max Bruckner (FSMaxB) Thanks for the feedback! I think I can address most of your points: > If all processes can see all user data, how does the security model work? Having every process only see it's own data, I could understand that, then you could use IPC mechanisms to pass user data to the processes, but letting a process have access to all the users data seems unreasonable. I'm not sure the system should protect user data from installed programs. This isn't something done in Linux anyway, right? My home directory is visible to every program, e.g. any text editor can open any text files in home. Android attempts to treat certain kinds of data, like contacts, as requiring explicit privileges, but I'm skeptical that users should be protected from their own installed apps. Sure, we want to mitigate the damage a malicious user program might do, so we don't give every program superuser privileges, but specially classifying the user's data adds complexity to the system and burdens the user. If we really did want to go that route, though, the way to do it would be for each program to store user data in its own filespace. A program might make this data available through IPC requests, and it could whitelist other programs upon approval of the user. Maybe then we want some kind of system-wide way of managing these privileges...but that might just end up more complicated. This solution also complicates backing up and transferring user data. It also arguably traps user data in silos. > How are those Globally unique IDs assigned and managed. This sounds like it will become either unmanageable or fail in another way like some entity having controll over every ID or some failing model like the current CertificateAuthority system. > How in the world can you create files with a global UUID? And if it isn't global, what happens if you attach a USB flash drive with a file on it that has the same UUID? Obviously UUID's are not reliable because they can be trivially spoofed. The idea is that, anytime veracity matters, you rely on the version id (the hash), not the UUID. UUID's generally include a high-resolution timestamp. As long as the generating code is not broken or malicious, collisions between any two UUID's are highly unlikely. Still, we would want public catalogs of packages to resolve such conflicts and--more importantly--to verify hashes. Unlike with DNS, we don't need a single centralized catalog. I could use a catalog that I trust, and you could use a totally different one that you trust. This is basically what we do already with every Linux distribution package repo. (Using numeric ids instead of names also means we can sidestep politics over who gets what desirable name.) As for file UUID collisions, either from error or malice, the system should cope by just letting them live side-by-side in the same filespace. UUID's resolving to multiple files is something human users can cope with. Programs, on the other hand: 1) have complete control over their own filespace, and looking up files in your own filespace is a different syscall, such that external collisions won't interfere 2) whenever possible, programs should specify files by the version id (the hash) instead of just the UUID Annoyingly, files with the same UUID in separate filespaces may have different metadata attached. Ideally, every copy of a file across time and space would have the same label everywhere, but this is already a problem we deal with. The only new wrinkle here is that two unrelated files might erroneously share the same UUID. I think this would be an annoyance but not a real security/config problem. (BTW, there's a whole angle I glossed over about how files would, by default, be treated as if they are immutable, e.g. opening a file to write produces a new file rather than overwriting the existing one.) > Having permission groups might not be necessary in most home computing scenarios, but when you have hundreds of people working on a project you can't just give everyone access to everything. But this might then be a server side problem that lives outside of the client side computers, so this might actually work because it doesn't need to be part of the low level desing of the system. Yes, I think as I mentioned, any sort of many-user concerns belong at the application level. We have servers running webapps or services in the backroom, and users--outside increasingly rare cases--all have their own machines (multiple per person, in fact). I think administering many users at the OS level is just outdated. > How do you manage hardware access without a permission system. You can't just let every program access every piece of hardware, this would be disastrous from a security perspective. Admin/non-admin might be too simple. Perhaps the system API is split into separate dependencies, such that a package effectively states explicitly which hardware it will use. A package requiring certain system API's would require special approval upon installation, e.g. this program may use the webcam. (On the other hand, users of Android seem to have been trained to just blindly click through these permission screens. Perhaps only admin users should be able to approve packages requesting certain kinds of special access.) > Btw: Why even use a global registry for configuration if you can just let every application have it's own configuration and expose it over the IPC mechanism if the configuration options are required from outside the application? I've considered something like that. But then there's the question of where to store each user's general settings and system wide settings. We could store them in files of user space, but as previously mentioned there's zero protections on those files. Again, I'm not really clear on this area. > Your model has, in parts, interesting similarities to what is already done in modern webbrowsers: Sure, there are parallels with browsers, but of course there's a lot we just can't do in browsers, e.g. run a server or natively compiled games.

[@OSSMaxB]

+Brian Will Now I have a much more clear picture of your proposed model. Just a few additional comments: > I'm not sure the system should protect user data from installed programs. This isn't something done in Linux anyway, right? Yes, linux doesn't do that and I think it is wrong. I don't want to be forced to trust every single binary that I ever run to not do bad stuff with my data (on purpose or by accident, see the steam client deleting home directories because of a bug in a shell script for example). Also if programs are separated from user data, companies that develop them don't even get tempted to snoop around in it. >If we really did want to go that route, though, the way to do it would be for each program to store user data in its own filespace. A program might make this data available through IPC requests, and it could whitelist other programs upon approval of the user. Exactly, that's what is used (or at least planned to be used) by xdg-app. The user grants access to a certain file by selecting it via a the file explorer. This might be expanded by passing files via the command line. Shared data can be whitelisted by an applications dependency manifest. > (BTW, there's a whole angle I glossed over about how files would, by default, be treated as if they are immutable, e.g. opening a file to write produces a new file rather than overwriting the existing one.) This sounds just like a waste of disk space and confusion for users because they would have to differentiate between different versions of "the same" file. But this could be handled like regular copy on write with versioning, showing the user only the newest version and providing a backlog. Versions that are older than a certain amount could then automatically be marked obsolete so they can be overwritten when space is needed. >Admin/non-admin might be too simple. Perhaps the system API is split into separate dependencies, such that a package effectively states explicitly which hardware it will use. A package requiring certain system API's would require special approval upon installation, e.g. this program may use the webcam. (On the other hand, users of Android seem to have been trained to just blindly click through these permission screens. Perhaps only admin users should be able to approve packages requesting certain kinds of special access.) Yeah, this could be done just like android with the slight modification that access policies could be changed separately from the actual applications by the repository maintainers. This model provides the possibility for more advanced users to sanitize the kind of special access an application gets, even if the developer wants all of it. >> Btw: Why even use a global registry for configuration if you can just let every application have it's own configuration and expose it over the IPC mechanism if the configuration options are required from outside the application? > I've considered something like that. But then there's the question of where to store each user's general settings and system wide settings. We could store them in files of user space, but as previously mentioned there's zero protections on those files. Again, I'm not really clear on this area. Just store the global configuration inside the filespace of a configuration-application that allows moderated access via the IPC mechanisms. Every non-global configuration just lives in the file space of it's application. This is also how this registry you described could be implemented without having to incorporate it in the base system.

[@briantwill]

+Max Bruckner (FSMaxB) All versions of a file in a filespace share the same metadata. Generally in file listings, you only see the latest version with a column indicating the number of old versions. Users can expand a file in the list to browse and select its particular versions. Overhead from keeping a bunch of old file versions around could be mitigated by applications simply deleting the previous version as their normal 'save' operation. Better yet, applications should make the choice very clear, e.g. 'save new version and keep old' vs. 'save new version and delete old' (not sure if there's a pithier way of expressing this distinction). Applications producing large files should maybe warn users about the overhead of keeping old versions. The pseudo-immutability thing is mainly to accommodate the version hash thing: as soon as you modify a file, its version hash becomes invalid, and until the file is closed, it doesn't make sense to recompute a new hash. So it seems logical to make copy-on-write the norm and think of modifying a file as actually producing a new separate version. There do seem to be cases, though, where normal mutability might be preferable, such as with log files. Perhaps just leave it up to each program on a case-by-case basis. (Of course, while a file is being modified, it can't have a hash id, but I think it works out okay if the open file is known just by its file descriptor until it is closed. This works because programs share files through IPC by descriptors, never by names.) I like the whitelisting idea, and that could apply to files: when a program attempts access of a user file for the first time, the user gets a UAC-style prompt to authorize access. Whether the registry should be a special kernel mechanism or a standard program is something I've gone back and forth on, but I suppose as currently described there's no reason for it not to be just a program. I have a hazy notion that other system features could be exposed as service programs in the same manner, but I'm not sure how far the idea could/should be taken. Kernel modules presented as service programs that hand out device file handles? Could ioctl be replaced? Anyway, thanks again for the feedback!

[@jkbenedict]

check out archives of burroughs B5500 mainframes and MCP.

[@user-kk5qe9fj2l]

You really need two permission levels for users, one for drivers and one for the kernel. a program should not get direct access to hardware or the part of the memory where the kernel lives.

[@gruntaxeman3740]

I really don't get what is the problem here to solve. There is always requirement to have some prerequisites, something where to build software.

[@eircK]

TLDR: I've always had a funny feeling about files. They're a very basic abstraction that can do anything: hold config, share state, store data, stream data etc. While this power was key to the success of unix, I think we have reached a point where we can really split these usages apart since we've seen all of them. So by providing alternatives like key value stores for config, (real) data stores for data, reference pointers for pipes & streams what's there a file system for? What I'm really trying to get at is that we often do mistakes on irrelevant things. For example why should syntax (or even worse parsing) errors in config files be a thing? There are some benefits from using files, in the sense that you can track changes in version control or just get a nice & editable visual representation in your favourite powerful file editor but I'm sure key value stores can definitely provide these features and in better ways even. On the data store issue, there are a few categories of stuff we wanna store on computers. First there's the programs code that can be stored as described in the video with some sort of internal ID mechanism which does not concern user space at all and then there's user data. Mixing these together is obviously not benefiting you in any way and is even harmful as you probably want user data to be accessible and combinable in ways simply not provided by filesystems. Sometimes you want them temporarily, sometimes you want to query them in complex ways and sometimes you want to just store big chunks of them with failsafe redundancy mechanisms. All these features are nicely provided by databases, but sure the choice of database and actual use can be a task even experts have a hard time with. I think that proper search functions (as mentioned in the video) are a great solution to this and a great implementation example of this is gmail. This may be a rehashing of some of the ideas mentioned in the video but I hope it strengthens the argument a bit by showcasing it from a slightly different perspective. What I want to add though is that a lot of these facilities are currently provided by cloud providers such as AWS. So implementing your workload on the cloud makes your computers actually obsolete and then the only use for OSs is to power the cloud infrastructure itself where incidentally other simpler and dumbed down OS-like entities like containers do the job much better. And some disclaiming, I don't think this is the one true way of doing OSs but as the dude in the video, I'm trying to point out some of the problems we have and propose some ways that they can be approached better.

[@moczikgabor]

I think you would be in a problem, if you would facing to a non-booting system, and instead of using a whatever-found editor to fix a human readable config file, you woukd have to somehow load some html renderer to see the error message and a fancy GUI tool to be able to edit the key-value store. These things already exist on windows and I think everyone tried to manage both windows and linux agrees with me on, that it is way easier to troubleshoot linux, way easier to solve arbitrary problems, way less constrained than windows. To be simple: if windows works, it works. You don't have to know how. However, if it's not, or you want different things than somebody designed, nobody can help you, undocumented, and you'll never find any solution. By the way, not everything can be described efficiently by key-value pairs or (G)UI tools. The power of flat files, shell scrips and the command line is flexibility. With removing these, you make constraints. But sure, I agree, you remove some user-errors too, and some learning curve too (so it will be simpler eventually)... The question is, which is more valuable.

[@Mythologos]

You just described Emacs: (1) all written in one language, only used for work inside the system (2) the system itself an interactive shell (3) constant output of data sequestered on easily accessible screens (4) one package manager.

[@duncanw9901]

Unfortunately, external dependencies present a problem. Emacs + GUIX/Nix is probably pretty close to the mark.

[@ThePondermatic]

I think some of your predictions for how to make the user experience easier have come true with Android and iOS, which present users with a very flat, non-hierarchical view of their installed apps and make it easy for app developers to work in their own sandboxed file space but hard for them to interact with anything else. And some of your predictions for how to make the developer experience easier have come true with containerized platforms like Docker, which create sandboxed file and configuration spaces per container and typically require containers to communicate via networking - a form of request/response.

[@llothar68]

Mobile all sucks. Professionals have Workflows and Workflows require sharing of files. IOS is biggest shit

[@figboot]

I noticed that with this video as well. Stuff like Android bothers me because of how little control I have.

[@FabianRoling]

Doesn't Android solve many of the problems you mentioned? For example app data: Normal apps can only access their own app data, superuser apps can access all app data.

[@nazgullinux6601]

Though I fully agree with you in your stance on OOP, when it comes to *nix-based systems, I will disagree. As a programmer, the beauty of *nix systems is that YOU CAN BUILD YOUR OWN for fuck's sake. If you are fine with the Linux kernel or Unix, then by all means build your own system around them. Have your own packages. Make your own dependencies. Port over what you want, ignore what you don't. Does it seem like a headache? At first, yes. After it is done, you have an operating system that you know will work for whatever software you build on it. You know that the only thing necessary to keep maintained, as far as repos or updates, is for the kernel. This is stupid easy for a seasoned programmer that knows the linux kernel inside and out.

[@pDaleC]

I look forward to playing around with the Userland you describe. Let me know when it is ready.

[@TheSandkastenverbot]

486 people had their religious feelings hurt

[@ludwigwittgenstein1193]

Check NixOS

[@angeldude101]

It's a start. It's a very clear improvement on other systems, but it doesn't match all of his issues. It could certainly be built upon to handle the others though.

[@samsmith1580]

The main reason a shiny new operating system wont work is that it too will have new functionality and requirements kludged on top of it. Then we will be back to where we are now. Why invest all that time and effort just to go round in a big circle. I think a bigger problem is the way operating systems evolve with little or no real concern for backwards compatibility. Its crazy that my biggest concern at the moment is Ubuntu will stop supporting 12.04 LTS in 2017.

[@abigailpatridge2948]

That's mostly a problem with your Linux distribution choice than Linux distributions as a group... Gentoo for example has nigh-perfect BC. It's like 99.9% completely compatible with everything ever on a POSIX base. And for that remaining .1%, it'd just take some careful planning and a chroot to get.

[@borilboyanov5544]

Instead of a kernel and shell you can have something like a "library" (able to be "stored") and every application bundle comes as a function with other functions to be stored at the library. User is presented a REPL in a language and from then on their role is to issue calls to functions or to compose their functions for storage. Functions are organized in a namespace and served by the library daemon by a common protocol. All the other functions that are on the system level are to be performed by a modern version of UEFI with code supplied by each hardware vendor able to work with the Library of each user. Oh yeah! Also, Libraries of users can synch themselves much the same way as a git would and stay synched automatically.

[@Jalae]

Interesting stuff. My main issue with security would be for how this would work in a server setting. any process could go rogue and spill all the userdata beans. I mean, you could segment your machines so a public network facing machine only handles that. But with large datacenters handling multiple users data things get complicated. I understand that this is a "toy model" and such, but one of unixs strongest areas is it's handling of user segmenting and security (not that it is super strong, but that it is better than the rest)

[@LordOfNihil]

i just want an os i don't have to burn down periodically and reinstall. i got to burn down raspbian on my pi tablet because its 2 distros out of date. and that means i got to recompile all my software again. only reason i didn't upgrade last distro is that half my software won't compile.

[@cebruthius]

NixOS on Raspberry pi does that

[@mskiptr]

What you propose is still quite complex tbh. I also had a lot of these ideas, but the more I learn about programming language theory, the more I think we just need to decouple most of the pieces that make up our software. This way we could iteratively get rid of any unnecessary complexity and just replace that with better abstractions bit by bit. That is, we're not gonna one day come up with the perfect replacement for everything that exists today and design a replacement in one go. Here's my idea of an 'ideal OS': First of all, it's supposed to be a so-called safe-language operating system. I.e. a one that doesn't need CPU security features to ensure privilege separation. This is because there shouldn't really be a hard distinction between the kernel and user space - kinda like what microkernels have tried. Then, how privileges should be managed? Well, the whole OS should consist of dead simple abstraction layers with clear interfaces fully described in code. There'd be no concept of users, permissions (neither Unix-like nor Android-like) at this point. The lowest layers know about the hardware and can touch it directly. Everything above works with the safe abstractions the lower layers provide. First, you need a (correct-enough) model of the hardware. That is data structures that behave like or describe the device you want the OS to run on. After that, you can write thin abstraction layers that take this hardware behavior descriptions and present it as more generic models. E.g. if you know exactly how a bunch of different PCI-e network cards work you can have a piece of code that provide the Ethernet protocol device out of this. Or if you have a hard drive, you can model that as a huge array of bits or whatever, together with it's runtime behavior (how much time do reads or writes tak, what happens when you loose power at some point, etc.) Somewhere alongside that you could have a layer that can turn the abstract CPU device into threads. Or turn RAM into allocatable memory. On top of that, you can put a layer that implements a textual shell. Or alternatively a layer that runs a GUI - consumes the keyboard, mouse, display, speakers, … and uses that to drive a desktop environment. Probably it should then also provide the concept of processes - otherwise any applications would need to be built-in (like on old feature phones). Still, all of that should sit in the compile-time APIs and not necessarily in any runtime ABIs. The compiler should be allowed to optimize any of that away and the OS developers should easily be able to swap out any part of it. However, at some point you will want to run 'the userspace'. Doing so would essentially boil down to making that specific layer provide a very concrete interface to whatever will run on top of it (an ABI I guess) and validating whatever is to be run (as long as that's our security model). This isn't really that different from all the layers before, except it now requires a few more technicalities to be modeled. Looking at what I wrote, it doesn't sound convincing at all. I feel like this comment severely misrepresents this idea. But still, maybe someone will find it interesting.

[@ParticleJesus]

Urbit will save us all.

[@popsicle199]

How would a executable find it's shared libraries?

[@TheMinustarian]

poptard I might be wrong, but the exec would get them thru a system->lib->get(, ) type of functionality.

[@angrydachshund]

+Stellan Crendraven Yes, it's called DCOM, with packages managed by WinSxS. Old and reliable technology... at least on Windows.

[@abigailpatridge2948]

Old and UNreliable you mean. Windows and reliable don't belong in the same room together, let alone thought.