Serialization - A New Hope

[@rafaeltorres3549]

Eyyy!! Its Viktor Klang. Met him in 2016 as a cs student volunteering at scala days. Sat down with me after his talk to explain a concurrency concept further. Motivated and inspired me to keep at it with Scala. Great guy!!

[@viktor_klang]

Eyyyy!! Thank you, Rafael! 😊

[@simonmassey8850]

The way Brian speaks about tricky topics is amazing. I respect how much wisdom goes into the way he takes the pain out of talking about tough problems. I really need to practice his style when dealing with “dumpster fires” at the office… 😂

[@UnMirloBlanco]

Viktor's G-Shock (GM-110RB-2A) looks really nice under the stage lighting

[@viktor_klang]

Your comment made my evening, thank you!😊

[@prashanttendulkar]

Is Brian the greatest Java programmer ever alive. And the greatest architect of JDK community. Other names are of course James Gosling and Joshua bloch , Doug lea

[@aleksandernowak585]

"No cycles" is a huge drawback, IMHO. But nevertheless, it sounds good.

[@la-dev]

Just jumped on it right after 36 seconds of its upload.

[@shadeblackwolf1508]

Modern jackson lets you explicitly map to a constructor with annotations, and scrapes getters which can also be controlled by annotations, meaning less magic, more control, but still monolithic

[@ttcc5273]

OMG throwback alert 41:40 ... Duke The Marshall... the 1990's called, they want their IIOP back 😁

[@prdoyle]

Does this design permit high-performance implementations? Are you concerned that you're adding an extra layer of object allocation at both ends, relative to (say) using Jackson directly?

[@viktor_klang]

@@prdoyle I think we need to define ”high performance” first. It’s still early days, internal representations are not set in stone, and no wire format provider has implemented support yet so I’m expecting real world usage feedback to be important signals for judging that. Just talking about potential performance optimizations would be a couple of conference presentations-worth of material. Cheers, V

[@prdoyle]

@@viktor_klang Awesome, thanks!

[@xichenliu-w3r]

@@viktor_klang How does it works for generics? Is there a EA build that we could try it now?

[@viktor_klang]

@@xichenliu-w3r what do you mean by ” how it works for generics”?

[@xichenliu-w3r]

@@viktor_klang I thought it would be quite difficult to implement serialization and deserialization with generics, based on current given API. For example, if the pointer record is a generic record, how would schema().parameterType() be like? I guess it should be Object.class because of type erasure, if that' ture, during deserialization, developers might still need something like TypeReference to mark the real generic type. I am wondering if these problems exists in marshalling mechanism, please enlighten me. Thanks.

[@JanMichalSzulew]

21:56 so elegant

[@user-gx3ve5iu1p]

You allowed million libraries to crop up over 25 years to finally reach here. REST in peace, JACKSON, GSON, JAXB

[@deeplykind3754]

0:15 Klanguage architect 😂

[@hausi78]

Hilarious, kudos to the person that wrote the slide!

[@nipafx]

@@hausi78 That would've been Viktor himself. :D

[@hausi78]

@@nipafx Thanks Nicolai for the info. I like such subtle jokes, great to see the Java architects not being too serious.

[@ttcc5273]

Good stuff. The marshalled representations could be used to do XSL-like functionality. Wire format I/O is a close cousin to representation transformation. Edit: I'm thinking of a unmarshaller that accepts two or more marshalled representations that produces a single output.

[@aholder97]

I find it surprising that the guy tasked with implementing marshaling for Java, had never even used Java's main JSON library until recently...

[@viktor_klang]

While I haven’t used Jackson from an app-developer context, I did design Akkas serialization feature, of which Jackson was/is one of the backends. But more importantly the point I was trying to make is that it didn’t require expert-level know-how to create a proof-of-concept integration between a wire format into marshalling, even if there is no marker interface to key off of. Cheers, V

[@aholder97]

@@viktor_klang I'm sorry, that was presumptuous and rude of me, I take it back

[@viktor_klang]

@@aholder97 Thank you, and no hard feelings! I hope you get a chance to try the feature out and perhaps even share some feedback! Cheers, V

[@HonestAuntyElle]

I had the same thoughts, frankly. Glad to see some creds in the comment though.

[@musicsoftwarebyhannotify8495]

@@viktor_klang Happy to try it out and share feedback - where can we find it?

[@alontalmor]

Interesting, thanks! Question that comes into mind - what if i have a class that uses class marshalling internally. Will there be a compile time check that that the instance that needs to be marshalled is actually marshallable? (In the case of Serializable, it is force by the type)

[@viktor_klang]

A class implementing Serializable isn’t guaranteed to serialize (there could be a bug in its implementation). Currently, marshal(x) returns null if unmarshallable (tbd if it should return Optional or throw) so you’d handle that at that point. (Also, I’d personally recommend writing integration tests.) Hope this helps! 👍

[@KangoV]

I wonder if at some point an implementation could output protobuf wire format by reading a .proto file? That would be so good.

[@CYXXYC]

i wonder if you could do without intermediate marshalled representation - i mean, you have your java objects already, your serializer just has to be like "ok, time to write an object", or "ok, time to write an int"

[@askarkalykov]

So many questions of the form "how feature X of jackson should be reflected in serialization's marshalling/unmarshalling". Like, for example, I can't get past the fact that order of parameters is just hardcoded into class structure. If that needs to be changed, how is this going to be handled?

[@viktor_klang]

>Like, for example, I can't get past the fact that order of parameters is just hardcoded into class structure. I'm not sure I follow. That's exactly how Java has worked for 30 years? Even if every constructor and method has an order of parameters, it doesn't mean that must be preserved in the wire format-wire format generators and parsers can use whichever order they want, as long as they can reorder based on the schema when unmarshalling. Let me know if this helps! Cheers, √

[@jay_sensz]

Why not use the syntax `super(var px, var py)` in the Marshaller code? I think that would be much cleaner than `this instanceof Point(var px, var py)`.

[@jonieder50]

If the business class field names are different from the json representation field names would I then need to create a companion record to serialize into outbound format?

[@viktor_klang]

That completely depends on what that ”business class” represents. You could have classes that are used for many different formats at the same time, where you’d translate the ”default” names into the format-specific names. Other classes may be specific to a given format and they could themselves then be used to construct the internal ”common” instances. If you think about it, this is a rather common case of conversion which doesn’t only appear for names, but also types and in some cases even structure. By having Marshalled as the intermediate representation, you can use the embedded schema and the knowledge of what format you are producing to provide conversions. On the parsing side it’s the same but inverted, as you’re starting from the other end.

[@jonieder50]

@@viktor_klang thanks for the explanation 👍🏻

[@HonestAuntyElle]

If there's a link to additional details, mailing lists, or specs, can they be added to the video description?

[@viktor_klang]

I like your enthusiasm! It’s still early days, more information will be made available as things progress. Stay tuned 👍

[@dematrsinba4361]

Ehehe I see a lot of inspiration from Rust's serde, finally Java going for the future !

[@PietervandenHombergh]

i would like a link to Stuarts talk on cycles.

[@delabassee]

Stay tuned, it's coming...

[@georgio7759]

kzbin.info/www/bejne/rIjQq3ukqrKLfrs

[@georgio7759]

kzbin.info/www/bejne/rIjQq3ukqrKLfrs

[@georgio7759]

kzbin.info/www/bejne/nJPUcqyIf9SEhZI

[@georgio7759]

Here is the title of the talk on youtube: The Cycling Tour - Java's Fraught Relationship with Cyclic Object Graphs by Stuart Marks (comments with links are being automatically removed)

[@LA-fb9bf]

WhatsApp about classes with a lot of Fields that get set by a setter method with an arbitrary value? How can I deserialize/ reconstruct that?

[@viktor_klang]

The author of the class is in charge. Add the appropriate constructor or destructor. If you are not the class author (i.e. you cannot modify the class) you either wrap it or extend it and provide the appropriate constructor and deconstructor.

[@LA-fb9bf]

@@viktor_klang Yes. I dont know why Not to dump the Memory and restore an object out of that? An object must always be created consistently. So the Memory just can be restored and the object is then consistent as well.

[@viktor_klang]

@@LA-fb9bf I’m not sure I follow-how does this help decoupling in-memory representation from wire representation?

[@LA-fb9bf]

@@viktor_klang Why is a wire representation needed? If I Need that I can use a Library Like Jackson. But if I want to transfer from A to B or just Save it, I can stick to a Byte Array. So, I dont know why there is an effort been done here.

[@viktor_klang]

@@LA-fb9bf How do you convert a class you do not know the structure of into JSON using Jackson?

[@Rope257]

I went through the video a few times, but I can't find a JEP. Is there a captain with a JEP number out there? Would be much appreciated.

[@viktor_klang]

At the end of the presentation there’s a ”todo” for me to start drafting a JEP.

[@Rope257]

@@viktor_klang Ah great, I missed that, thanks!

[@justsignmeup911]

9:20 I think the p should be p1

[@viktor_klang]

Nice catch!

[@gudenau]

I'm not familiar with the pattern keyword.

[@HonestAuntyElle]

The idea of keeping 10 years worth of constructors in a class is extremely unpleasing to think about. The amount of constant rework I've seen in things like configuration files makes this seem like a non starter for large serializable objects. I've seen migration scripts be larger than the domain objects themselves. But i guess you could always add that as "wire format aware middleware" so i don't completely hate it. Although I'd like to see some form of surrogacy shown in a demo, or some supporting library.

[@viktor_klang]

Maintaining support for many different versions of the same thing isn’t concise in its very nature-you’re presumably going to have to provide default values, write tests that verify compatibility etc no matter what. The benefit that I see with Marshalling is that those constructors (or factories) can be private, marked as @Deprecated and will typically delegate to the current version together with default values for previously non-existing fields.

[@HonestAuntyElle]

@@viktor_klang it's mostly for code organisation purposes, and being able to change strategies based on other changes elsewhere, which is tedious for domain objects, and it's easier to have the complexity of migration in one place. I like that the class owners are responsible, but having a way to specify an external factory method instead that's responsible from that class instead would be handy. And I don't think that's a possible design if it's constrained to constructors. I'd support it, if it instead (maybe optionally? But that adds complexity) used a static factory that delegated to it's own constructor. But it breaks the symmetry somewhat. That way the class is free to dictate dynamically which constructor is called too, as long as the parameters match.

[@HonestAuntyElle]

Constructors in java have always been inflexible, it's one of the reasons why field initializers are needing to be hoisted before super calls.

[@HonestAuntyElle]

Also, marshalling might end up being one of those things that cover 80% (probably more like 40% atm) but without the complexity. There's probably still value in that. But if the other language features proposed get added, do we even need an annotation supported by the JRE, before other tools get to play with it? My suggestion is if this is the plan, to release the language features required, don't release the annotations, and just give it a year in production to see what libraries do with the new syntax, and adapt to it / support them. You can always add the accessibility/visibility tunnels later.

[@viktor_klang]

@@HonestAuntyElle static factories are already supported in the current proof-of-concept (but currently they need to be defined on the class itself (of course there’s nothing preventing delegation to other factories from within that).

[@krellin]

Anyone actually thinking about using this?

[@RonaldTetsuoMiura]

This doesn't do what Serialization does, and adds nothing that Jackson itself isn't already capable of, besides pluggable wire representation. Yes, Serialization is full of weird edge cases and downsides, it is not really appropriate for neither long term storage, nor network data transfer, but it does what no other marshalling mechanism does. If you strip it down to this, what is the advantage over Jackson, JAXB, Gson, or protobuf? Another set of lowest common denominator standard annotations? For a very interesting use of serialization (as is), look for Apache Wicket. It wouldn't be possible without the power of serialization.

[@viktor_klang]

>If you strip it down to this, what is the advantage over Jackson, JAXB, Gson, or protobuf? The advantage is that something like a standard library, or 3rd party libraries, can in a standardized way expose structure so that the user can choose Jackson, JAXB, Gson, protobuf, or all of them at once. In essence, it allows the separation of structure from wire format.

[@whiteyoghurt]

Eh, I don't think this is going the right way. Everything mentioned has already been done.

[@HonestAuntyElle]

@@whiteyoghurt I think the main difference is having official support, to aid compatibility. It's overly simplistic, but a lot of stuff can be handled in the middle layers. I just wish it was a tiny bit more flexible in regards to using factory methods instead of constructors, and had more support for named parameters then what it appears to have on the surface. Records are brilliant for that, but for classes?

[@viktor_klang]

@@HonestAuntyElle ​​⁠ static factories are supported in the prototype, and parameter names (while currently obtained through -parameters when compiling) look to be important to preserve for marshaller and unmarshallers.

[@thiagohenriquehupner1164]

@@viktor_klang Would be possible to change the behavior of the methods that contains the annotations to automatically preserve the parameter names or this would be a big change in the specification?

[@viktor_klang]

@@thiagohenriquehupner1164​​⁠ Making methods annotated with the marshalling annotations automatically preserve their parameter names (not havingb to compile with -parameters) is definitely something to consider.