Codegen: The good, the bad and the ugly

[@daniellever8441]

Use Fragments for commonly queried types, and turn on codegen for fragments! - It's great for things like article/page tile components which appear across many pages, or for standardising how you pull images/media info from your server before passing to your media components. You can use the generated type directly as prop types, and your queries and higher level components no longer need to care about the fields involved, and no need for type drilling.

[@tuvshinbayartuvshinzul8489]

How to make fragment refetchable by using client preset of codegen?

[@ordazgustavo]

For that reason I like Apollo’s codegen, it generates a separate interface for every type, it doesn’t inline everything into a huge nested type

[@yoyo26-34]

Agreed. Haven't tried it for a while but I remember that when generating react hooks to query your data (for example useQueryBooks to return the GQL query "Books"), the return type of the hooks function wasn't set in the .ts file, thus making the linter shouting when the option to force the return type of a function is set in the tssonfig file. Is it "fixed" ?

[@laesseV]

Aren’t they planing on deprecating the apollo cli tool?

[@josedev.]

@@laesseV Yes I think it doesn't even work with the latest Node versions

[@celica96]

Your TS content is really high quality. Thank you.

[@deathemperor]

Your points are so spot on. The whole video describes how my codegen lifespan has been. The small DX hit is a cheap tradeoff for correct Typescript. It improves our outputs and DX greatly. I keep looking for ways to improve it but for now it’s been great with typing the untyped with codegen.

[@ggascoigne]

I'll add that I rarely see an extracted type being as simple as: type TasksQueryTask = TasksQuery["tasks"][number]; Nulls and unknowns mean I end out with things like this: import { O, F } from 'ts-toolbelt' import { Key } from 'ts-toolbelt/out/Any/Key' import { List } from 'ts-toolbelt/out/List/List' export type GqlType = NonNullable type UsersAndProfiles = GqlType So, not too bad to use, but very awkward to work out when starting from scratch.

[@artemstratienko4842]

Thanks for listening to a community! 😉

[@mattpocockuk]

Did I? Was this requested?

[@artemstratienko4842]

@@mattpocockuk yep, I asked you a couple of videos ago to tell about a code generation and you liked my comment 😉

[@mattpocockuk]

@@artemstratienko4842 You planted the seed! Thanks!

[@Fik0n]

What I think is more interesting and that doesn't have a clear answer is what strategy to use with the types in term of the overall project. For my types I want one "source of truth". You can then either infer everything from the generated types or use your own types and stuff like Omit to use the generated API types with your project types. Both have their pros and cons and would be an interesting topic to hear your thoughts on

[@fordneild2372]

That small DX hit is completely blown out of the water by the fact that the developers no longer need to maintain the client types.

[@LarsEjaas]

You really should have mentioned THE one major frustration point with GraphQL Codegen for most developers new to the tool. And that is the fact that a lot of GraphQL API's by default will make every part of the returned dataobject optional - So basically to access every part of the data you need to use optional chaining to access the data. Plus arrays get a bit hard to access. One of the tricks here is to use fragments in the places where you need a specific type eg. for a React component. This will make the generated types more "friendly".

[@Fik0n]

This makes me go nuts using Strapi.

[@OwlArtz]

Another issue when working in larger teams with big generated schemas and types is git conflicts. Especially if the apis are also worked on frequently by some backend team. The way Ive solved this is by adding generated files to .gitIgnore and not committing generated code. Then we treat the actual .graphql file as the source of truth. The drawback is that developers are responsible for generating locally to make sure they got the latest stuff from the apis. In the ci pipelines we run a generate script which always generates from the latest versions from the apis. This is nice because then the generate step works kind of like an integration test between client and backend. The drawback is the added deployment complexity and onboarding new developers.

[@jrgalyen]

I feel your pain. Yes. When code gets checked into the server, git has no feedback mechanism to push it to local branches watching. git is not real-time. It could be. Then CI/CD & such could chain from shared source branch changes. When schema changes, we want it to break elsewhere.

[@animanaut]

commiting generated code to git is like commiting object files to git in a c project. generated code is an intermediary artifact that happens to be code

[@jrgalyen]

@@animanaut Not necessarily. There should be a two-way binding between generated code and the original document. Such that a change to the generated code can parse back into the document. With that in mind, there is nothing wrong with checking both in. And even if you change generated code and it diverges from the documentation it was generated from, that diff should be preserved.

[@animanaut]

@@jrgalyen i've been there, never doing this again. generated code is most of the time a one way street so you better treat it as intermediary by default. other languages like java put generated code into the target folder by default. modifying generated code will bite you in one way or the other in my experience. if the codegenerator cannot produce the code you need you might have to rethink its usage in the project.

[@jrgalyen]

@@animanaut Except for ORM's and Data Contracts like OpenAPI generating POCO objects. Stuff you shouldn't be writing tests for anyways. I wasn't saying code generation for anything outside boiler plate.

[@zotoavinaandriamanamihaga6661]

I have this issue on my prisma powered app where when I change the database schema and the types are updated, I need to restart ts-server to actually have the changes reflected and It's kind of frustrating

[@Tango-tt6dx]

I am currently working a lot on codegen and very much could relate to the points especially the increased complexity in typing. The main driver for codegen in TS for me is maximizing correctness in projects. Writing types or schemas manually would not cut it. But as you also stated importing types without actually knowing the generated src has a lot of pitfalls in combination with structural typing. I've seen a lot of dev time been wasted because of fixing a compiler error which was a result of importing the wrong type. Accessing nested types is actually a save guard. Also extracting Types from unions or Picking slices from types is a time saver once you get used to it. However, it is a diffrent level of working with TS. So I appreciate your emphasis for the TS complexity which come along if you care about correct types.

[@RicardoValero95]

Talking about graphql, gqty does something really nice where you do the codegen only when the schema changes and all your queries, mutations and subscriptions are written in typescript (similar to tRPC)

[@linusang]

Hi are you able to do a video on AST with the TypeScript compiler and create custom ESLint rules from it?

[@GerardoSabetta]

I've always had this question in my head. If you are working with an API that does not use OpenAPI docs. What would you do? Create the types manually with the risk of the types not beeing up to date with the real backend responses or would you put the effort in documenting the API first and then using codegen tools? Whenever I'm writing types for backend responses on a frontend repo I have the feeling that it's not being colocated where it should be and that the response will change and my types will turn missleading instead of useful.

[@mattpocockuk]

Great question, worthy of a video itself

[@GerardoSabetta]

@@mattpocockuk I'd love to know your take on this. Especially on cases where it's not a javascript monorepo and the backend is also not typescript / javascript. Because all of the cool tools out there like tRPC are not really an option for me because the backend in my project is python mostly

[@MatiasCiccone]

+10000

[@rand0mtv660]

​@@GerardoSabetta can't you use OpenAPI/Swagger with Python? Where I work, backend is done in .NET Core and backend developers add annotations to endpoints and we get a full OpenAPI spec generated from that. Then on frontend we just use a tool like "swagger-typescript-api" to generate a full API client using fetch or axios and we just update it periodically when backend changes. Yeah it's a minor inconvenience to re-generate that API client, but it's way better than just copying types from backend code or trying to type them using responses that we would get calling that API. We wouldn't be able to even cover all types easily like that. In your case, I would personally go for documenting the backend using OpenAPI because that can also generate simple documentation page that can be used for testing various endpoints without having to set up the project at all. Then use a codegen step whenever needed on frontend and that's it.

[@GerardoSabetta]

@@rand0mtv660 That's great advice, yes, we can use swagger with python, but since it's not a monorepo, running the codegen is not as easy as being in the same branch as your backend dev. We would need to publish a package with the yml for the open API and consume it on the frontend repo. It's definitely doable

[@ykhi]

Sveltekit comes to mind with it's type codegen for, for example, data loader functions

[@davidadamson1588]

I completely feel this video! We are using an OpenAPI Generator on our frontends (outbound traffic) and backends (inbound and outbound traffic) to achieve e2e typesafety with both typescript and java 🎉 Having to run the generation script manually does not bother too much since interface changes do not happen very often… Anyway I would love to give tRPC a try . The DX seems to be so great due to the minimal overhead. Unfortunately it is hard to convince my team to switch from java to typescript on our backends 😢

[@jrgalyen]

Which openapi generator? Give them a taste of deno to convince them of javascript on backend. Avoid node like the plague :) Deno was created by the author of node. one executable. Generates binaries.

[@Stmated]

I am currently working on a project to create a better developer experience for code generation. Specifically to make it easier to support more languages and to make it easier to change the generated output to look the way you want it. My initial plan was to have it semi-ready in January, but things got in the way. Right now I have support for OpenAPI, JsonSchema and OpenRPC to be translated into Java and TypeScript... but there is quite a bit left to make it easily usable as a cli. But throughout the years I've dealt with lots of code generation tools, and they all have major downsides when working in actual projects. So I know what I'd like to see in one. We'll see how it goes, if I ever get to advertise it as something public.

[@elfelipe1996]

Is that project open source ? If the code is accessible-enough maybe you could get a hand ? (from me or others interested in this)

[@Stmated]

@@elfelipe1996 it is, but I want it to be in somewhat of a shape before I tell anyone where it is. I'll get back to you one of these days 😉

[@elfelipe1996]

@@Stmated understandable. Good luck with it!

[@squrler]

@Stmated A mammoth undertaking! Hope it works out. Used OpenAPI TS generator tools for years and the dev experience wasn’t great. Would be amazing if a high quality alternative would be available. Good luck!!

[@AlvarLagerlof]

Fragments over queries in separate files! The typed-document-node package handles fragments nicely

[@snake1625b]

Our team used to have so much trouble typing when we'd pass entire entire objects or graphql responses as props. You'd have to use lookup types to dig to get the type you needed. Then we realized that it's better to not pass objects but destructure and pass primitives. Or just fetching from the Apollo cache directly instead of getting data from props

[@mattpocockuk]

Absolutely. It can be so annoying.

[@nathanfranck5822]

I'm very casually figuring out my coding ideas around my job, and I am very tempted to code in Typescript UNLESS the target doesnt support it - consoles or shaders for instance, or runtime peformance needs to be perfect / multithreaded, at which time I can use codegen to generate code for other languages/targets to create a safe interface between my main typescript code and my specialized code. I don't have enough time or energy to explore this fully though

[@GLawSomnia]

Well using a codegen with open-api schema was a breeze for me. Didn't have any (big) problems. Its true tat i had to generate the types every time the schema changed, but its a million times better than synching the types by hand, so running the generator a few times is not as bad as it might seem. But yeah in my case i got the .yaml schema from the BE and had to update the types only when it changed

[@FrontendMedia]

Thanks for explanation! 😊

[@syahrulanuar4521]

If only I found this a week ago on how to 'destructure' types generated by Codegen via indexing the type itself. Another downsize of codegen is the sheer amount of type guards that are required. I'm so used to it now that I even do it in legacy JS projects to always check if there's undefined 😝😝

[@victorlongon]

I love you content, Matt...but I don't even have to watch the video to say "YES!!!". Codegen has been one of the best tools created lately. Nonetheless I will watch the video ♥️

[@ColinRichardson]

OOOh, some good things for Lunch

[@michelecocucciocatania]

That's super useful. I love codgen. Where can I find more about the syntax TraskQuery['tasks'][number]. What's “number“ in this case? Thanks!

[@michelecocucciocatania]

@@dylan_the_wizard amazing! Thank you very very much for you reply!

[@breeeew]

For frontend (graphql queries) and backend (graphql type definitions) you should separate generated output from codegen. On frontend you should use fragments, and exclude all full typings (they are for backend use only). Codegen have options and plugins for this.

[@unlizw]

Sounds interesting, can you please share a link for that?

[@ColinRichardson]

HA... "had to wait 5 to 10 seconds"... Those are rookie numbers... I basically code blind for about 30 seconds.. We haven't setup pre-compiled types for our projects in a monorepo, so it basically compiles the whole thing all the time.... The ironic thing, we don't have time, to stop, to fix it, to speed us up... PS: thanks for the kiss though.

[@michaelbitzer7295]

27 minutes was the maximum for autocomplete here. I really hope stc becomes useable soon. Been slamming "as any" all over the codebase to speed the typechecker up.

[@ColinRichardson]

@@michaelbitzer7295 Damn, that sounds like a nightmare.. 27 minutes? might as well be turned off..

[@michaelbitzer7295]

@@ColinRichardson It was a nightmare at times, but after a ton of manual typing it now sits at around a second, so it is somewhat bearable.

[@ColinRichardson]

@@michaelbitzer7295 Ahhh.. Sadly, my slowdown is almost constant.. I type a few lines, says it doesn't know a Type, I tell it to import it, and I have to wait 30 seconds for the list of things... I have got to the point of having the "search as a panel" open all the time, so I can quickly search for somewhere it's been imported before and copy-paste... It's quicker than having the IDE do it for me.

[@fungilation]

GPT-4 and on will be the only codegen you need. "Make codegen great again"? It's already great.

[@geisterfurz007]

For the first point, I ended up writing two codegen plugins at my company. One removes all the schema type generation (it generates types you are not fetching and not using as input, so there is no real usecase for them anyway) and the other adds the input types for mutations locally to the files that need them (the plugin assumes near-operation-file-preset) and allows exporting parts of a query as separate types using a directive without the need for fragments. We are not struggling with that first bit anymore at all and the plugins act essentially as replacement for the typescript and typescript-operations plugins,

[@culi7068]

My strategy: use the actual `Task` type always. But use `Pick` always to `Pick`. Then, regardless of which query your "Task" is coming from, as long as it has the bare minimum necessary for your util/component/whatever, it will be compatible.

[@mattpocockuk]

Yep, this works well - but is more annoying when you have objects more than one level deep.

[@culi7068]

@@mattpocockuk Yes, sometimes you have to just declare it a type like `type BookCollectionForLandingPageToy` and pass that around locally. Still imo much preferred to being stuck unable to remove an expensive field that's only used in a handful of places from a query because you've used that query everywhere and this one part of the app would just go entirely red if you dared pare down how much data you're asking for unnecessarily

[@Mitsunee_]

the only codegen I've used so far is a little script that generates react components from some svg files using svgo. This also lets me add classcat in there to allow me to pass my classNames as an array with conditions, which is a really nice DX feature :)

[@aristide_F]

Thank you Matt. at least i now know i am not insane using and thinking things out with the codegen. i wish i could purchase the TTs(Total TypeScript) course but I can't for now despite the amazing discount. i hope when i am ready to grab the course there be a discount. i love your content.

[@jrgalyen]

Is the course in deno?

[@OviDB]

You should have mentioned client-preset. Makes the types so much better by using the DocumentType

[@streamoverlayprocodes6890]

"Restarting my TS Server" while I'm listening to this and I had to restart my TS Server because my VHD sucks...

[@ole.petersen]

One cool tool to mention here is slqx for Rust. It allows typesafe SQL queries and is based on macro.

[@bambang_ap]

Hi Matt, sorry for my OOT question. But i have a problem with zod const schema = z.object({ id: z.string(), name: z.string(), }); its equivalent with type schema = { id: string; name: string; } my question is, i have to make some properties optional but keep the type like this type schema = { id: string; name?: string; } if i use `.optional()`, it will make all properties optional

[@mattpocockuk]

mattpocock.com/faqs

[@82TheKnocKY]

My way around typing the TableRow props is a DeepPartial utility - it takes a type and returns an optional for any of its children and its children's children etc. This means I won't get an error because a type somewhere 4 children deep is slightly mismatched, but has the downside of not knowing what props on the object the component is ttruly expecting.

[@adambickford8720]

Yes, tight coupling has its benefits. It also has its downsides which won't be talked about until the next shiny new thing comes along.

[@jrgalyen]

you want a single source of truth for data and data contracts. You want a single source of truth here. Right? right? *cwies*

[@workflowinmind]

5:10 You can add it to your bundler, vite has a plugin for instance.

[@deathemperor]

Would you mind to elaborate?

[@DevduttaBain]

hey Matt! what is your thought about drizzle-orm?

[@БронетемкинПоносец-х9ч]

but what about code-first approach like TypeGraphQL utilises?

[@rafaeltab]

Lovely video, I always experience this same frustration when using openapi sdk generation. It really makes me want to smash my monitor. I do agree that it is a necessity though, and it just makes me wonder if there is a better way.

[@hynekss8618]

One note regarding tRPC: While I am all for it, there IS an autogenerated schema living in node_modules/.prisma/client, which is regenerated on every schema update. And yes, I need to manually restart the TS server in VSCode to catch up with the changes…

[@mattpocockuk]

Sounds like you're describing Prisma, not tRPC?

[@hynekss8618]

​@@mattpocockukOh, yeah. It's Prisma. My bad. I've mistaken bare tRPC for create-t3-app.

[@ShaharHarshuv]

I'm disapointed, I expected some solutions. I wonder if it's possible to write a webpack plugin that reads graphql files to generate ts. If it can generate only files that were touched, it might also be faster for bigger projects

[@nomadshiba]

prisma also has the same problem its almost like they generate the types to use with their api only internally

[@nomadshiba]

tho not gonna lie, at least prisma is trying to be better

[@nikolakikanovic9740]

What if you have graphql on BE written in Java and 'hidden' behind API Gateway. Is there a way to auto generate schema.ts on FE whenever BE do change on graphql schema?

[@deathemperor]

BE can generate the schema.ts then release a new package version for FE

[@quintencabo]

I use zeus petsonally and it gives nice readable types but I still use zod to verify the response. Sometimes on big queries though it does take like 10 secondes for it to check the types

[@weirdwordcombo]

Typescript inference will only get you so far. For example if you need to transform function signatures in extrem ways (e.g. for react) and those function signatures happen to use generics. I believe you need higher kinded types or something like that for that, which Typescript does not support. Then the ONLY way do it, is doing manual codegen with something like ts-morph. Of course if you use tRPC or something, then that problem goes away, but then you also dont have generic function signatures.

[@petervanholland6667]

I think the task component should be considered a presentational component that defines the interface it requires for its props. It should not be using a type from an upstream API. Separation of concerns. I think if you do it this way then this problem becomes less confusing. And as for the VS Code typescript compiler being slow to infer updates, I have two words for you. Apple Silicon

[@cicerotcv]

"what am I doing with my hands?" hahahahaah

[@chawkichalladia1812]

I used codegen as a substitute for Amplify codegen, and I liked what it offered a lot. with that being said, I still needed a huge ass file for type extraction lol

[@Glinkis]

I'm fairly sure it should be possible to infer GQL types based on the schema and the query definition, by using literal types and inferrence. I mean, someone created a JSON parser that could infer an entire structure from any JSON literal, so this should definitely be possible. Might be super heavy on TS though.

[@culi7068]

yes that's what the --watch flag does. As pointed out in the video, in very large projects, this starts to really slow down

[@albertulio]

3:33 you can use Fragments and it's solved

[@DaKruzze]

First issue could be resolved by Fragment

[@lautarodapin]

Matt, in problem number one, the solution is using fragments

[@mattpocockuk]

Yes - but other codegen tools suffer the same problem and don't have a built-in abstraction like fragments to solve it.

[@denisbessa-tx1dv]

Make a video about Drizzle ORM

[@alexeinunez778]

I have a 12 Prisma models and use zod and trpc generation and it takes minutes for tsserver to provide any useful type information, it's infuriating

[@dmitryponyatov2158]

what about a _real_ modern projects which includes dozen of programming languages and frameworks for native/mobile/web/etc ? There should be some metaLanguage to describe shared parts of it, and generate code in multiple languages in a programmer-tunable way

[@Stmated]

That is exactly what I am working on as a side-project. The steps it takes for each generation is: 1. Read input schema 2. Convert into common model 3. Convert common model into an Abstract Syntax Tree (like "class/type" with "properties/accessors", not bound to output language specifics) 4. Convert Abstract Syntax Tree into a Concrete Syntax Tree (now locking it into the form of the output language) 5. Convert CST into output strings by running a chosen renderer over it Each conversion between the different steps is done with appendable transformers that can be added/removed depending on your need. This way I can easily build a "convert fields into getters and setters" for multiple languages on the AST level, et cetera. Reuse as much functionality for as many different languages as possible. It works surprisingly well so far. I just need to find more time to actually make it usable and not just a Proof of Concept running test cases.

[@culi7068]

> dozen of programming languages and frameworks What the hell

[@MrHands-pp6dr]

This feels like a personal attack on me as the author of a C++ code generation library. 😅

[@hyperprotagonist]

Big fan of codegen but the more complex the graphql schema (some headless CMS are overly complex) the more overly complex the discriminated unions become. Useless runtime guards everywhere. 😢

[@j3bb9z]

Can't imagine myself not using code generator and typing the client code by hand. IMO these issues you've mentioned are negligible.

[@jrgalyen]

RapidYou only did graphql. You left out Quicktype & ngswag. And history. SOAP and wcf. And Application Development, RAD, around for decade. New term being API first. Write doc & generate data contract code (any language). And writing documentation that produces code is the fundamental principle of how AI helps us write code. Another video with more detail. Also. What is a typescript server. Seriously. Asking for a deno user. I don’t know what that is.

[@damon8541]

I disagree with both points being made here. If the generated code is bad then that's a problem with that particular generator, not the concept as a whole. Also, if your entire stack is written in a single language, then code generation is quite obviously the wrong tool for the job, there's no point comparing the speed of code generation to the speed of importing a type from your backend.

[@joergbaier]

Your first example is flawed. The whole reason for having accurately typed query data, is that typescript will complain when your sub-components need data that you missed in the query. The props to TaskRow should be { task: Pick } or { name: Task["name"] } or { name: string }.

[@cas818028]

Sounds like y’all f’d up creating a codegen tool. Codegen a have been around for ever for a wide array of uses. Not just GraphQL. Perhaps use or create better tooling. If the tooling is more work than just creating types yourself. Then do it by hand. Tooling is is supposed to automate and streamline processes not get in the way

[@hicoop]

Do any of you know if this kind of library exists: Prisma but completely in typescript. The same type of prisma.users.findMany syntax but with pure typescript schema file like a zod object. Prisma's cold starts are killing me and make it unusable in serverless but I haven't found a different tool with better performance but the same ease of development.

[@OviDB]

You should have mentioned client-preset. Makes the types so much better by using the DocumentType