High Performance Rust UI

[@234_gokulsoumya3]

Timestamps: 00:00 - Introduction 00:25 - Outline 01:32 - History 05:23 - Research questions to answer 07:47 - Rust (pros and cons) 11:25 - Performance 17:20 - Xilem architecture (example code) 20:50 - Xilem architecture (why, inspiration, etc) 24:18 - Xilem core concept: View trait 28:16 - Performance problems with diffing large trees 31:29 - Xilem flow 33:53 - Xilem async 37:14 - piet-gpu 38:58 - piet-gpu pipeline 39:44 - piet-gpu scene encoding 42:26 - piet-gpu scene fragment 44:08 - piet-gpu WGSL port 45:40 - piet-gpu WGSL demo 47:00 - Signature of paint method in widget trait 48:18 - Scene fragments vs layers (compositing) 50:55 - A bit about the widget tree 52:41 - AccessKit integration 55:06 - AccessKit demo from egui 58:19 - Layered architecture 1:02:35 - View vs Widget 1:05:25 - Xilem demo 1:06:33 - Problems out of scope (for now) 1:09:12 - Current status 1:10:31 - Next steps 1:15:15 - Python demo

[@IanMcKellar]

It's exciting to see a11y baked right into the architecture.

[@NoNameAtAll2]

who's ally?

[@bernhardschmidt9844]

@@NoNameAtAll2 Stands for Accessibility (a, 11 characters, y), similar to how i18n is used for internationalization or l10n for localization.

[@Sammi84]

@@bernhardschmidt9844 horrible acronyms. Super hard to guess. Unwelcoming to newcomers.

[@SimonBuchanNz]

​@@Sammi84 Do you prefer "acby", "lczn", or "ntzn"? They are about as good as you're going to get for contractions. (But yes, they're generally overused.)

[@lycantropos]

@@SimonBuchanNz I prefer full names whenever possible

[@marschrr]

Finally! Nice! Love your work on hi perf 2d gpu graphics! Let's watch it!

[@sombrastudios]

I really enjoyed hearing these details and about your progress. Keep up the great work, these are exciting times

[@JoeTaber]

Raph got the Authentic™ twitch chat experience trying to answer troll/joke questions 😂

[@realsong-fake]

It's great to see ui frameworks put async support to priority.

[@nikitadudnik271]

I want to laugh and cry at the same time, because this is extremely cool, specifically a11y baked in, but beyond my comprehension regarding implementation. Anyway looking forward to using it as a library.

[@CalebRox]

Why go for React-style view diffing instead of an actual finer-grained reactive style of something like SolidJS? (which would also probably perform better)

[@RaphLevien]

I answered in more detail on Reddit: old.reddit.com/r/rust/comments/zce386/vector_graphics_done_entirely_in_compute_shaders/iyxkgbf/

[@tsalVlog]

I've been trying to come up with my next "weekend development" project, and I keep coming back to an ECS-based editor, particularly using Bevy.

[@rupertthurner8311]

does zed industries GPUI build on your ideas, e.g. antonio scandurra blogging about rednering a text editor like a video game?

[@anotherelvis]

Great talk.

[@forresto]

55:30 the accessibility demo is great! More interface developers should learn VoiceOver etc enough to interact with their own interfaces.

[@BboyKeny]

I'm learning Rust and building a social media post scheduler with a UI. I'm using the Iced crate at the moment which is Elm inspired. I might rewrite the UI in Xilem, because this sounds interesting.

[@JuanManuelCañabate]

Very interesting work indeed!

[@dealloc]

Funny, I was thinking about implementing a GUI library based on SwiftUI, and here we are :)

[@Speykious]

One thing that I've been waiting for a long time in Rust GUIs is a declarative animation/easing/tweening system. Edit: ah, seems like that's currently a non-goal but that would be nice to have.

[@DHorse]

Thanks. Very informative.

[@jimmyraconteur]

the only example you give is a window with a button?

[@simonfarre4907]

Are there any reasons why we aren't adopting the more "Immediate mode" for GUI's? ImGui for C++ seems to be so much faster than anything else. Is this because it's more narrow in scope or some other explanation I haven't thought of? Why build it in these abstract ways with what I assume contributes quite a lot of overhead?

[@chip2389]

There are pros and cons to immediate and retained mode. While immediate mode is easy to code, it wastes computations due to re-rendering the world over and over, and complex layout (anything with mutltiple passes) is not intuitive to implememt. Retained mode updates incrementally, which makes it more efficient, but manipulating the widget graph manually is verbose and bug-prone. So, instead people now use an imediate mode API to mutate the hidden underlying retained widget graph. I think ImGUI actually does this these days. The twist with xilem is that you can implement your own views and widgets, which is not the case for React as its widget are tied to platform widgets. Furthermore, passing the id path in events could possibly solve difficult widget communication patterns (nested collections like trees, asynchronus rendering etc.) more elegant.

[@simonfarre4907]

@@chip2389 very interesting. I think i did a very, very, simple and naive version of something like that for a very simple text editor, emulating a terminal text editor rendered using OpenGL where some elements did not get redrawn unless explicitly needed to. So i think I get what you mean. Obviously everything I did was pure hackery, but still.

[@flylord42]

So I had an idea I wanted to run past you. What if you created a graphics queue and a thread that owned the graphics memory that would just pop stuff off the queue to do memory tasks?

[@disieh]

It is a good question which is better, something like Figma exporter or doing the Electron/Tauri approach of using HTML+CSS as the UI input. To me Figma sounds nicer but HTML+CSS is probably better in the long term. That being said, since Tauri is implemented in Rust, might there be a possibility to use Xilem as backend and HTML+CSS scene building from Tauri.

[@keeslinp]

Any thoughts on mobile or is this strictly aimed at desktop?

[@RaphLevien]

We're focusing on desktop first but I am hoping that we get to mobile as well. I've been experimenting with piet-gpu (the drawing layer) on Android, and it seems to be working well.

[@davidallen5146]

Is there someplace that describes widgets and views and how they are related?

[@RaphLevien]

The best current place is xi.zulipchat.com/#narrow/stream/354396-xilem/topic/Widget.20trait/near/313688326 but I will try to make sure that gets covered in a proper writeup.

[@general9064]

Does tauri suffice?

[@AshkanKiani]

If mobile is a target, then isn't basing everything on compute shaders a risk? The driver support there isn't consistent afaik. That's why I target basic gl3 primitives myself and use SDF in my shaders and UI library I wrote. In that setup, variable length binary encoding isn't really possible (fixed size of the max instruction), which is a let down, but I can still get a similar pipeline.

[@RaphLevien]

It's a valid concern, but I think newer devices are increasingly capable. And for a fallback, we do have a plan, which is to do the early stages (including the variable size decode) in CPU, and only fine rasterization on GPU. In that setup, we'll have a lot more flexibility, including custom shaders that work specifically on low-spec GPUs.

[@JoeTaber]

Could it be possible to swap in completely different rendering strategies for the middle layers that operate at different "interaction bandwidths" than a desktop UI? Say you take your xilem app which typically operates on a desktop with full bandwidth interactivity (live mouse cursor location, wgpu output), and instead "render" it to an environment with much lower interactivity capabilities like a plain html page with forms for display and interaction. Maybe I'd characterize this idea as "extreme progressive enhancement".

[@RaphLevien]

It's an intriguing concept. On the Zulip we've been talking a bit about a View trait that can generate more than one type of element tree - Widgets and DOM nodes to be specific. Zoxc has an intriguing proposal of making the View trait generic on a dyn Trait and having several impls. I think that could work. The question is whether it's worth it and whether you want to deal with the impedance mismatch in semantics between the different element trees.

[@SimonBuchanNz]

This is basically what Flutter started with for their HTML target, they apparently found it worse than just tossing elements out and using canvas and doing everything themselves (layout, input, a11y, etc)

[@DHorse]

1:06:15 Where did my slides go? Oh no! 🤣🤣🤣

[@jaysistar2711]

All new UI architectures must work on the web. That doesn't mean "use the browser DOM", but it does mean that a full SPA style webapp should be possible, as well as a native (browserless) version. I see potential here.

[@Ergzay]

I disagree. Overly web-focused UI kits are ruining the world right now and we shouldn't be overly catering to them.

[@jaysistar2711]

@@Ergzay I didn't say "web focused" I said "work on the web". I did also say that that doesn't mean "use the browser DOM". Look at "egui" to see what I mean.

[@_general_error]

So this guy says, that Garbage Collection makes program startup slow... interesting...

[@medennis3467]

“A Bit of History” pun intended?😂🤣😂🤣 From a medical background, UI is disturbingly close to UTI. Sorry, back to our regularly scheduled program (see what I did there).

[@xtiooplus]

If you don't even manage to mention the name of the GUI pack in the title, your spirit loses its meaning for me

[@sbray-n1g]

8 years and nothing to show for it. Vaporware.