How To Create A Complete Internal Developer Platform (IDP)?

It's time to build an internal developer platform (IDO) with Crossplane, Argo CD, SchemaHero, External Secrets Operator (ESO), GitHub Actions, Port, and a few others.
#internaldeveloperplatform #idp #crossplane #argocd #schemahero #externalsecretsoperator #githubactions #port
Consider joining the channel: / devopstoolkit
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Gist with the commands: gist.github.com/vfarcic/78c1d...
🎬 DevOps MUST Build Internal Developer Platform (IDP): • DevOps MUST Build Inte...
🎬 How To Create A "Proper" CLI With Shell And Charm Gum: • How To Create A "Prope...
🎬 Crossplane - GitOps-based Infrastructure as Code through Kubernetes API: • Crossplane - GitOps-ba...
🎬 How To Shift Left Infrastructure Management Using Crossplane Compositions: • How To Shift Left Infr...
🎬 Argo CD - Applying GitOps Principles To Manage A Production Environment In Kubernetes: • Argo CD - Applying Git...
🎬 How To Apply GitOps To Everything - Combining Argo CD And Crossplane: • How To Apply GitOps To...
🎬 SchemaHero - Database Schema Migrations Inside Kubernetes: • SchemaHero - Database ...
🎬 Manage Kubernetes Secrets With External Secrets Operator (ESO): • Manage Kubernetes Secr...
🎬 Github Actions Review And Tutorial: • Github Actions Review ...
🎬 GitHub CLI (gh) - How to manage repositories more efficiently: • GitHub CLI (gh) - How ...
🎬 How To Build A UI For An Internal Developer Platform (IDP) With Port?: • How To Build A UI For ...
▬▬▬▬▬▬ 💰 Sponsoships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please use calendly.com/vfarcic/meet to book a timeslot that suits you, and we'll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: / vfarcic
➡ LinkedIn: / viktorfarcic
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
06:03 Control Plane
07:52 User-Friendly Interfaces
11:00 Synchronization From Git With GitOps
12:45 Schema Management
14:40 Secrets Management
16:36 Graphical User Interface (GUI)
19:29 CI/CD Pipelines
21:44 The End Result
26:00 What Else Is Missing?

[@DevOpsToolkit]

What do you think about the presented IDP? Is it a good start?

[@abtris]

At least basic observability is useful if app really running ;-)

[@DevOpsToolkit]

@@abtris You can ingest any information into Port so "basic" observability should not be a problem. It does not replace Grafana or a similar solution but those are typically for people dedicated to observability. For a developer, Port is more than capable of showing relevant info.

[@kazwalker764]

I'd like to know if the self-hosted operator Port is promising will be OSS or not. If not, Backstage is probably the option to go with if you want an IDP using CNCF projects. Great video!

[@DevOpsToolkit]

@@kazwalker764 I think I know the answer but I'll let Port folks answer that one.

[@johnramsko4535]

Awesome! Can't wait to see more!

[@MasonGelosotech]

You have a great talent of explaining complex things. Thank you my man.

[@natachinhas]

I love this so much! I was implementing something similar so that I can have more free time to work on other things. You've given me a lot of cool ideas and I think I can save a lot of time by doing some of the steps you did here.

[@andrewnhien9714]

Thank you, Viktor! Please share more your idea about IDP, crossplane and Port, fully build on it.

[@2stoon]

This is awesome, I'd definitely love to see you take it further!

[@amitgadhia8831]

This is amazing i've been waiting for this video so long. All the tools used in the setup make sense and we can add more stuff no doubt. Overall it's a good start for initial setup of IDP. Thank you very much.

[@sohansingh2022]

Legendary!! No words for the effort you make for the community.

[@gustavofigueiredo280]

I am new in the area, but I found this video and it is amazing! Great Work!! You have explain a lot of things in a really easy way, I can point what I don't know. When I study, it is possible to come back and see my knowledge grow. Great work in video dynamic, structure too.

[@cowgod77]

Wow, meaty video with lots of great info. Going to take my time and dig through this, because it looks very applicable to my environment. Thanks!

[@onemanops]

Victor your style of Dev Ops is so great man, very different from what I'm doing, I do cloud operations, learning alot from your videos. Now you give me a use case for crossplane and this IDP sounds really cool. Defeinitely awesome tips in. this video like udate control plane with git. truly awesome bro. I will try to implement it and let you know how it goes.

[@hEAVEN_Beyond]

Enjoyed the video a lot! Very interesting to see your way of doing this. I would be interested in seeing your approach to preview environments for PRs. 👍

[@DevOpsToolkit]

I have it on my to-do list. Can't say when i will do it except to say that it is coming.

[@GneralTsao]

I spent the weekend reading up on all of the things you mentioned (and some comparable alternatives). You've convinced me that I've been thinking about Kubernetes in a very limited way. It's not just for scaling containerized apps. CDRs and operators allow it to be so much more. It can be a general purpose thermostat for infrastructure, SaaS products, and more. I've seen organizations take years to try and build something like this (and ultimately fail). It's amazing to see what the right combination of off-the-shelf tooling can do.

[@autohmae]

I think it's good to understand a lot of these off-the-shelf tools didn't exist some years ago.

[@absolutejam]

This is really great, and I love how it rolls a lot of your previous videos into one implementation. I'm a recent covert for all things control-plane focused (GitOps, cross plane, etc.) and I love this approach at providing a standard API, coupled with the IDP aspect. Really amazing work!

[@jameshouston3907]

this was such a great video! I can definitely see this being the new normal!

[@giorgosfiotakis4803]

great value please keep up the awesome content !

[@salborough2]

This is amazing and I would love to see more videos and options on this. The one thing I also wanted to mention is that I have watched the cross plane videos and checked out the documentation previously but I found it wasn't clear enough to pick up and understand easily so I put it down. I really want to get into cross plane and so maybe a suggestion might be too do a deep dive cross plane course perhaps?

[@DevOpsToolkit]

yeah. I should do that. Can't promise when, but it is coming...

[@gulyasbendeguz2033]

This is very interesting! Thanks for putting in that much effort to build such tooling! Got my imagination going, seems like endless possibilities on enhancing this. Although i would be really curious how certain changes could take place in an IDP like this (changing schemas/custom definitions/services) and also how preview enviromnents would work with PR integration.

[@DevOpsToolkit]

Preview environments will probably be the topic of the next video whenever I manage to find time to continue down the IDP path in this channel.

[@TheGriffender]

Thank you Viktor for this great video ! I would be very interested to see the Crossplane Composite Resource Defintion you used 😀

[@DevOpsToolkit]

You'll find references to everything in the gist. The link is in the description. Now, that gist is referencing composition packages in a registry. The source code of compositions themselves is in github.com/vfarcic/devops-toolkit-crossplane

[@TimothyLaurent]

This is great -- As long time gitops user, I really resonate with your style! It's also much easier to direct colleagues to your videos instead of getting all bent out of shape explaining why I hate git-flow or something :) I'd like some way to declaratively add blueprints to Port other than that, it looks great!

[@DevOpsToolkit]

That was my main complaint to port folks. I want it to discover CRDs in my cluster and convert them to blueprints automatically. Later on, i might tweak them though.

[@KnThSelf2ThSelfBTrue]

SO excited to watch this one.

[@vangelkrstevski566]

Thanks! Keep up the good work.

[@DevOpsToolkit]

Thanks a ton Vangel.

[@roguepithlit]

Thank you Viktor!

[@MarkusEicher70]

A bummer! I just love it.

[@rmkohlman]

This is an amazing video, and I fully believe it would be great for you to expand on this. Personally I would love to see integration where the database is in cluster with a Postgres operator to manage it and the ability to have staging, and prod etc. Would be great to see a mini IDP for local development leveraging rancherdesktop. As well what about a service mesh? You really could do a full blown long series on this IDP 😊

[@DevOpsToolkit]

That's great to hear. I already started working on the next IDP video and, if it continues to be useful, more will come afterwards.

[@MarkusEicher70]

@@DevOpsToolkit Sir, that's fantastic. Can't wait to see more of your IDP prototype. This has potential to skyrocket. You should sell it. 🚀

[@MarkusEicher70]

This for small scale local environments built on a few VPS or VMs or in a K8S is a burner. Just gonna try this, but I'm a noob compared to Viktor, so this might be a lifetime job. 😋@rmkohlman

[@mzimmerman1988]

great video. thanks!

[@FernandoRipoll]

Hey Viktor, thanks for the video. How long does it take you to create the whole setup? I see it is quite powerful thought it takes time to build it and each company could have a different view on how steps are done or configured.

[@DevOpsToolkit]

I can't answer that question since it depends on the experience and user needs. I prepared all the material (code) for that video in less than a day, but I spent years working with the tech i used.

[@cotyhamilton]

Now we just need an external DP for us to use to create our IDPs 😂 great video!

[@robertdeheer3365]

it is a good foundation. it would be cool to see more abstractions around other primitive resources and operations on types of primitives. the goal being to split infrastructure deployment from app deployment, standardize operations that can be performed on the primitives or groups of primitives... | that through a rigorous specification process, and you have yourself a new IDP. One challenge with creating many workflows that use multiple crossplane or terraform primitives is finding and running every automation that uses a primitive after it is modified. it would be very difficult to search for resources across repos, across tools, and track updates. thanks for all of your videos.

[@DevOpsToolkit]

The key is in building APIs. If you do that, some of the problem like searching for resources becomes trivial. It's all about creating services and exposing them as APIs. From there on it becomes easy to expose them through a GUI, from command line, in vs code, or anything else. More often than not, that is not what we are doing. That is a lesson Amazon learned long time ago and that lesson gave birth to AWS.

[@robertdeheer3365]

@@DevOpsToolkit building an api is a good solution. brilliant

[@user-rs5do8yl5p]

I want it now! Give us more more more!

[@DevOpsToolkit]

More is coming in 4 weeks approx.

[@tw1ce723]

Beautiful! But how to track the actual deploy status in gitops/pull model? Also In general, we pushed manifest from pipeline, what the actual result? Is the pipeline green? How to know when things deployed and ready?

[@DevOpsToolkit]

Every kubernetes resources has statuses so it should not be a problem to track them and know when they are ready. As a matter of fact, events from those statuses might be the only way to know. Pipelines will not give you that info simply because statuses change over time and are not directly related to kubectl apply which only sends a request to kube api and receives ack that the request is received not that the rollout of all resources is finished nor the future changes of those resources.

[@SanityAwry]

@DevOpsToolkit love this solution. You mentioned about preview environments / PR e2e testing, are you able to expand a little on your thinking on how we could achieve this? Would it still involve using Port?

[@DevOpsToolkit]

Yeah. It would involve port since it is just a frontend for whatever is happening in the backend (a cluster). What's missing is a way to make preview environments cheap and for that we need one more thing. Stay tuned... That "thing" is coming in a month or so. I just started working on the material.

[@maxgio]

Great, interesting and entertaining video! You've been able to made simple, a complex architecture like an IDP. Thank you. About CI, did you leverage ArgoCD Image Updater in order to update apps container image tags on new published image versions? I think the solution makes sense. It would be cool to dig into collaboration on Git, with the current GitOps operator-big players - e.g. pull requests over direct commits on long-living branches. As a feedback, I found not so clear the CI about the application bootstrap (git repository creation and link of the manifests to the parent manifests already pulled by ArgoCD) - anyway, the video is is already pretty dense of informations and it would have been too long to dig also into it :-)

[@DevOpsToolkit]

I did not use argo CD image updater. Instead, in GitHub actions, i just executed yq to modify the manifest and pushed it back to git. You're right. I rushed through explanations in order to keep it relatively short and digestible. There is a link to the gist with all the commands in the description. That should allow you to reproduce it and through that go deeper into it.

[@maxgio]

@@DevOpsToolkit That's very good as so. Thank you! I'll definitely go into it :)

[@kirkpabk]

Good stuff. My concern is that I find very few resources speaking of an IDP within a hybrid or pure on-premises perspective. Some of us do not have Github and use Azure DevOps GIT or another internal provider. Would love to see an IDP perspective from an internal or hybrid perspective.

[@DevOpsToolkit]

The logic is still the same. Create APIs and/or data models, use them to generate UIs, push data to git, sync git with the system, feed the data from the system to the UI for observability purposes.

[@andrewnhien9714]

Devstream is cool. If can make a short video for this please. Thank you!

[@DevOpsToolkit]

Will do...

[@scottmolinari]

Nice stuff. Quite inspiring for me. One thing I'd like to see added is how you'd give devs a development environment, which allows them to run and test their code within the system. The dev may have the code on her local computer, but she would need to have it running in the actual environment (k8s with a lot of different services running already, including database, cache, ingress, event system, et. al.) and even better, actually coding in the environment too. Or, coding locally, but able to connect to all the services in a highly complex application (i.e. not just an app with a database). I'm working on exactly that right now, but much more opinionated in the app platform (i.e. specific language and database). I already have my solution for remote development. I'd be interested in seeing yours. 😊 Ah, and where can I find the manifests used to build your IDP?

[@DevOpsToolkit]

Adding it to my to-do list... You will find the link to the gist with all the commands (including clone of the repo) i executed in the description of the video.

[@scottamolinari]

@Scott Thanks for the reply. What is "very quickly" for you all?

[@felipe88alves]

You make very good comments about exposing a simplified interface to users, but a problem I've seen with Crossplane is the difficulty in deploying only the CRDs that are needed by the IDP. Do you know of any easy options for deploying only the required CRDs?

[@DevOpsToolkit]

If you are referring to CRDs installed through crossplane providers, they will soon be broken into smaller groups.

[@thibaultdeheurles4561]

@felipe88alves it s now released in crossplane, just select appropriate provider (lambda, ec2, etc)

[@Ztall0880]

amazing, thank you for the work... please add more stuof tothis IDP setup...also on how can we swipe one tool for another..

[@DevOpsToolkit]

Already working on the next one. Coming in a few weeks...

[@ArmandoCollazo]

Great video! I only have some questions... what would be the strategy to manage changes to the control plane? What if the platform teams need to suport a new abstraction which requires installing new crossplane providers? How will they test the whole flow before making live for everyone?

[@DevOpsToolkit]

I think that would follow the same principles as any other testing. You might want to start with unit tests (search for kuttl on this channel) abd then proceed with functional tests in a test environment.

[@starmh82]

Amazing topic and details about full scenario about IDP❤, can we achive this with change getport with open source backstage

[@DevOpsToolkit]

Yes you can. The principles and the flow would be the same but the specific implementation in backstage would differ.

[@IvanRizzante]

Great and inspiring video as always, is it possible to share the code you used (port blueprints, github actions...)?

[@DevOpsToolkit]

There is the link to the Gist with all the commands and references in the description of the video (under "Additional Info").

[@kdietz65]

From the POV of an actual developer writing code, I would like to see more things that help the developer find critical technical information in a timely manner. That means the IDP needs to support docs-as-code style documentation and code search and other things along those lines. Coding takes a lot of research and I'd like the IDP to help me do that research.

[@DevOpsToolkit]

That is the goal of IDP (helps devs do things or find into in an easy way with right levels of abstractions). Now, specifically for coding, GitHub copilot is the closest one. It's not there yet but it is on the path of becoming a good coding helper.

[@absolutejam]

Port can extract data from your CR and display it in the UI, so you could have a field for repo, docs, etc. and they become links in Port. These can also be propagated to the underlying workloads as annotations

[@DevOpsToolkit]

@@absolutejam True, but that applies to entities. What's missing, from my perspective, is a way to auto-generate blueprints from CRDs. That's probably the feature I miss the most.

[@absolutejam]

@@DevOpsToolkit ah yeah, I forgot about the mismatch there. Definitely would benefit from a convertor! I've only just begun exploring port so I wonder if there's some kind of middleware or plugin system 🤔

[@DevOpsToolkit]

@@absolutejam There is no plugin system as far as I know.

[@justcloudnative]

Great

[@lukasmrtvy]

Good. Thanks. You should cover some GitOps CLIs, devstream is looking promising but does not seem to be in active development. Something that can scaffold an app, template it, create a repository, PR or commit, and set up Flux / ArgoCD integration. Sca(ffold)GitOps ?

[@DevOpsToolkit]

For scaffolding a repo i feel that devstream does the job. There's not much for it to do anyways but create a repo based on a template repo and change templated files. That's a relatively easy job that could be done even with a simple script so i don't see a bit need for devstream to do much more. From there on, there is nothing else to do except to push a manifest with the reference to that repo into the repo that contains argo CD apps. In my case, that is done through GitHub actions that were created through that templated repo. That does not mean that there is not a more elegant solution, but I'm not actively looking for it since it's all about creating that repo that contains everything i need.

[@fpvclub7256]

I know Backstage is a pain to work with - but .. if you could swap out getport ( security concerns with having Saas able to literally do anything in our control plane ) with Backstage so we can run it on-prem? I've heard rumours port will be releasing a self-hosted version soon - but I don't want to wait until then.. Also, would it be much different if you used HashiCorp Vault for secret management? I love this series btw, I have been wanting to put together an IDP for ages now to enable our agile teams to be more self-sufficient and reduce DevOps/SRE efforts.

[@DevOpsToolkit]

It would look more or less the same with backstage. As for vault... Nothing would change. ESO works with Vault as well.

[@kdietz65]

Excellent ideas. In terms of templates or starter projects, I've had a hard time making that work in the past. They get too monolithic too quickly, and with everyone using different technology stacks, they either have a short lifespan or they have too limited use cases to be useful. The developer has to wade through hundreds of templates trying to find the one that will work for them, and eventually they just have to give up and start from scratch.

[@danielhd6719]

TBH this works only for "hello world" projects and not anything that will be used for many many years and evolve over time... Sounds like every other "no-code" solution in the past, but now you will need someone who is able to operate all of those tools because at some point they will stop working and nobody will know why.

[@DevOpsToolkit]

The benefits of creating services others can consume are proportional to a number of users of such services. If there is something that only one team needs, it makes no sense to converts that something into a service. That team should do it alone.

[@DevOpsToolkit]

It's not really no code. There is code behind everything I used in that video and all of it is stored in git.

[@swagatochatterjee7104]

🔥🔥🔥🔥

[@telecomprofi]

Yeah, please add another maintenance external cluster that will: 1) observe it (who watches the watcher btw?) 2) track component's changes/sec. vulnerabilities in every used tool/dependency and update them automatically in IDP cluster ;-).

[@hugolopes5604]

ok some feedback based on difficulties we are facing building our idp: - observability... we are trying to include otel for auto instrumentation and colecting decoupled from observability backends. that means workloads must be defined also with observability details that after map to the otel configs... we are going purist with all obs data in otel, including logs, for later log semantic standarization, etc. -we desire a single pane of glass for observability, we have chosen the loki-tempo-prom-grafana stack for this... -on the ui of the observability we have a topology issue, do we want a observability per enviroment or for the whole app? or both? -resources that we add to the app with crossplane, such as aws RDS db, should automatically configure grafana dashboards... this means a crossplane composotion should also orchestrate the observability UI -flexibility... this model focus on developer apps that are standalone, but often its more confusing... we have vendor apps that come with their own helm, and the developed application is often an add on to the vendor app in the same cluster. so, the idp should after allow still people to have k8 access to add whatever else they they need, such as a vendor app helm chart or a small oss component like keycloak, etc. not sure how these cases fit your idp -for the reasons above, it would be nice to have "idp" as a k8 controller, one could just add to the cluster as a CR in the gitops repo.... again topology questions, like multitenancy, granularity of the app defenitions in the enterprise, which usually is all over the place, etc. -service meshes.... we are trying to do zero trust and as such, workloads need to be wrapped in service mesh... be it istio, linkerd or whatever.... so the abstracted workcload CR needs to add also the sidecar or configure the ambient mesh or whatever and lifecycle it HA -HA topics... should the application teams also define the application slo so that more replicated workloads with proper afinity rules be set? -testing... aplications need to be tested automatically... the best practice is a test pyramid of unit/integration/load tests/E2E and even tests in production and chaos tests to test HA resilience? as such there can be a lot of diferent test technologies application teams choose, selenium, cypress, gatling, etc, etc. where is this test infrastructure hosted? we have chosen testkube so that tests are defined together with the aplication as k8 gitops objects... this also needs integration with the CI/CD steps so its getting complicated -canary releases, we use argorollouts but its full of details because observability data must feed back to the canary "proxy", be it ALB or envoy or whatever. -compliance and policy... you made a video about it... we are following a similar aproach with a mix of opa validating low level k8 objects and CR objects but there are real life details... for example, we have to comply to cisbenchmarks, but many of the default rules have to be adapted to our platform, because if we use service mesh, we don't really need to define network policies... but the service mesh envoy needs privileges that is against cisbenchmarks, so we need rules exeptions, uff... complicated. - in observability there are also alerts, and in enterprise enviroment we need picket teams and use diferent ways of alerting team members, as such, ops genie... -inventory issues... we have to stay aligned with enterprise inventory, so we went with define application in code as a CR and controllers update the enterprise inventory systems. - lifecycle... we have a lot of operators in the platform per app cluster... and they need to be lifecycled... we are exploring OLM for this so we can automatically upgrade minor versions of operators - other topics, autoscaling, orchastrating other rss like vms and lambda functions(which we plan to orchestrate from k8 ofcourse), E2E enterprise observability integration such as cross application tracing... service mesh integrated with event driven architecture with zero trust. -there are a lot of stuff, but in general the intuition more is done with controllers in k8, the easier it is to fit things together... so i question why some of your idp components like port are not controllers? maybe they are, but I saw a lot of bash script...

[@DevOpsToolkit]

Few answers... > observability... we are trying to include otel for auto instrumentation and colecting decoupled from observability backends. that means workloads must be defined also with observability details that after map to the otel configs... we are going purist with all obs data in otel, including logs, for later log semantic standarization, etc. The part of getting OTel data from outside apps is relatively easy (depends on languages used to code apps). The challenge is to get "more detailed" info and, for that, people writing code needs to adopt OTel instrumentation. > We desire a single pane of glass for observability, we have chosen the loki-tempo-prom-grafana stack for this... That's a great default choice, as long as you make dashboards for "experts" and dashboards for each team. Lately, I've been adopting more and more Pixie (with optional Grafana). You might explore that one as well (a video is coming in a few weeks). > on the ui of the observability we have a topology issue, do we want a observability per enviroment or for the whole app? or both? I guess that depends on team structures. If teams own their apps fully, than it's per app for them and per environment for those observing the system as a whole. > resources that we add to the app with crossplane, such as aws RDS db, should automatically configure grafana dashboards... this means a crossplane composotion should also orchestrate the observability UI True. Once you figure out dashboards without Crossplane, adding them into compositions should not be an issue. > flexibility... this model focus on developer apps that are standalone, but often its more confusing... we have vendor apps that come with their own helm, and the developed application is often an add on to the vendor app in the same cluster. so, the idp should after allow still people to have k8 access to add whatever else they they need, such as a vendor app helm chart or a small oss component like keycloak, etc. not sure how these cases fit your idp Tycally (even though I did not add that to the video), I would create a generic blueprint for adding any third-party app. In such a case, there would be a generic field that represents Helm values. Now, if there is a third-party app that is commonly used, that would become its own blueprint with some values exposed as separate fields and others that are always the same hard-coded. > for the reasons above, it would be nice to have "idp" as a k8 controller, one could just add to the cluster as a CR in the gitops repo.... again topology questions, like multitenancy, granularity of the app defenitions in the enterprise, which usually is all over the place, etc. That's what I did. I just added a CR for the DB to a repo. Port was used as the interface to do that and Argo CD to sync it with the control plane cluster. The same logic would apply to a cluster or anything else. > service meshes.... we are trying to do zero trust and as such, workloads need to be wrapped in service mesh... be it istio, linkerd or whatever.... so the abstracted workcload CR needs to add also the sidecar or configure the ambient mesh or whatever and lifecycle it HA ... or you can do that on Namespace level so that sidecars are added automatically. > HA topics... should the application teams also define the application slo so that more replicated workloads with proper afinity rules be set? That depends on your customers (application teams). You expose what they need and hide the things that they don't care about but are necessary. > testing... aplications need to be tested automatically... the best practice is a test pyramid of unit/integration/load tests/E2E and even tests in production and chaos tests to test HA resilience? as such there can be a lot of diferent test technologies application teams choose, selenium, cypress, gatling, etc, etc. where is this test infrastructure hosted? we have chosen testkube so that tests are defined together with the aplication as k8 gitops objects... this also needs integration with the CI/CD steps so its getting complicated Unless working with a very small system, I would host all that in a cluster separate from production. > canary releases, we use argorollouts but its full of details because observability data must feed back to the canary "proxy", be it ALB or envoy or whatever. This is a similar case like others. Expose things people need, hide those they don't care about but are needed. Some might want to control queries used to make decisions whether to move forward or roll back while others might not care about it. > compliance and policy... you made a video about it... we are following a similar aproach with a mix of opa validating low level k8 objects and CR objects but there are real life details... for example, we have to comply to cisbenchmarks, but many of the default rules have to be adapted to our platform, because if we use service mesh, we don't really need to define network policies... but the service mesh envoy needs privileges that is against cisbenchmarks, so we need rules exeptions, uff... complicated. Yep. It's not easy. > inventory issues... we have to stay aligned with enterprise inventory, so we went with define application in code as a CR and controllers update the enterprise inventory systems. That's the way to go. > there are a lot of stuff, but in general the intuition more is done with controllers in k8, the easier it is to fit things together... I agree 100% > so i question why some of your idp components like port are not controllers? maybe they are, but I saw a lot of bash script... That was my main complaint to the Port team. I want it to a) run as a controller backed by CRDs and b) to auto-detect CRDs in a cluster and convert them to Port Blueprints. I think that they are working on it. As for scripts... I created one only so that the initial setup for the video/demo is easier. Later on (after the setup), it's all GitOps (push to Git and forget about it). Or, to be more precise, it's GitOps for continuous part and GitHub Actions for one-shot actions like, for example, building images or running tests.

[@hugolopes5604]

@@DevOpsToolkit so many thanks for your detailed feedback :) ... I will split my feedback in multiple answers... first, you made no feedback to the OLM part.... IDPs are living things, the operators need lifecycle, how are you dealing with this? is OLM a good choice?

[@hugolopes5604]

@@DevOpsToolkit >.. or you can do that on Namespace level so that sidecars are added automatically. we are still debating if we should do a opt in or opt out strategy in the service mesh... another thing is the service mesh of the infra... should we use service mesh on otel? for now we decided not "wrap" in service mesh the "infra" in the cluster, such as otel, argocd, crossplane, etc

[@hugolopes5604]

@@DevOpsToolkit >Unless working with a very small system, I would host all that in a cluster separate from production. testkube works better in the same cluster as the application. we would have to expose all app components in ingress for them to be accessible to the external testing tool. also some tests do run in production as production probes, so it feels its part of the application. anyway, it was just to mention on your idp how you deal with test automation... it is kinda missing and all applications need testing

[@hugolopes5604]

@@DevOpsToolkit >Lately, I've been adopting more and more Pixie (with optional Grafana). You might explore that one as well (a video is coming in a few weeks). for what I understand, pixie is its own collecting and visualization layer... that would be a big no from us... all observability data must be colected with otel, for all kinds of reasons, like decoupoling observability colecting from obs backends and to reuse the standarization work of otel... and we still want the single pane of glass for observability... the features of pixie seem fine, but imo they should be plugins in grafana.

[@jirityr]

I like these videos about end-to-end solutions. All components make sense but I'm still not sure about Crossplane. Does Crossplane already have some way of checking what change is gonna happen on the infrastructure prior the actual change being applied? Something like triggering/waiting for a webhook or waiting for a Approval CR to be created to progress with the change would be nice. Also how do all the hundreds of Crossplane CRDs affect the Kubernetest API performance?

[@DevOpsToolkit]

By default, crossplane will apply whatever you tell it to apply. Seeing what will happen before it happens is close to impossible with kubernetes (that's unrelated to crossplane) since the scheduler makes decisions all the time and after a resource is created. For example, if you create a deployment, you cannot know in advance where will the pods run. They might be in one node, than one might be moved to another, another might fail and be created somewhere else, etc. When using kubernetes, we accept that the scheduler is making decisions (constantly). As for performance... When using one of the big three providers, kubernetes might struggle with too many Cards (depends on the size of the control plane nodes). Those providers will soon be broken into smaller groups so that you can pick and choose what you need and reduce the number of CRDs. That should be available in a matter of weeks. At the same time, kubernetes project is working on better optimizing the way it handles CRDs and with each version of kubernetes it is working better.

[@jirityr]

@@DevOpsToolkit When it comes to infrastructure, I like to know if the change that I'm applying will only update the resource or it will replace it. Sometimes a change of a single parameter can lead to the resource replacement. Unnecessary to mention that the type of the change might have a different time and/or financial implication. That's why I believe that knowing what will most likely happen with my infrastructure before the change is applied is important part of the infrastructure management.

[@DevOpsToolkit]

@@jirityr That's true, but not necessarily for Crossplane. Most Crossplane users are creating compositions which is a way to create a service (e.g., a DB). In those cases, people test compositions and, if it works as expected, users create any number of claims of that composition. It's a different approach with work split between service providers (e.g., you) and consumers (e.g., devs that work with you).

[@jirityr]

​@@DevOpsToolkit I understand. But I can still imagine a situation where the composition accepts a parameter that can trigger replacement of a resource. It's the same like with Terraform modules - it all depends what you allow the user to change. I still prefer to use Hashicorp's Terraform Cloud Operator (hashicorp/terraform-cloud-operator) or WeaveWork's Terraform Controller (weaveworks/tf-controller) to trigger Terraform runs in the Terraform Cloud. Like that I don't eat much of the resources on the actual cluster (Terraform runs outside of the cluster) and I'm leveraging all the features provided by the Terraform Cloud (policy enforcement, drift detection, cost estimation, ...). Perhaps you could try to use such combo next time? ;o)

[@DevOpsToolkit]

@@jirityr Terraform is great or, to be more precise, it was great. I'm committed now to Kubernetes and Crossplane (so much that I chose to join the project). From my perspective, the next generation of IaC (we call it control planes) is certainly going to be based on Kubernetes. That can be seen by the direction GCP, Azure, AWS, and VMWare are taking. Now, to be clear, it does not have to be Crossplane (there are others with more to come), but it does mean that it has to be Kubernetes. Terraform cannot work well in Kubernetes due to its architecture not being done to leverage it. For example, the idea that there is a drift detection based on a project instead of individual resources is bad news. As for policies, cost estimation, observability, logging, etc... That's one of the reasons why I bet on Kubernetes. Kubernetes-native apps can work seamesly with other Kubernetes-native apps. That means that there is no reason to, for example, create a whole ecosystem around Crossplane since it works with Kyverno, Prometheus, ArgoCD., Flux, or almost anything else designed for Kubernetes. CNCF alone is the biggest and fastest growing foundation, mostly focused on Kubernetes with most (if not all) projects working seamesly with all others. In other words, while I do recognize Terraform as an amazing tool with a lot of other tools and service that work well with it, it is a previous-gen tool just as Ansible was before it. P.S. Terraform is more mature than Crossplane (it exists for much longer) and I'm in no way saying that the latter is better than the former in all aspects.

[@danielacosta9662]

why not add eclipse che or similar for developing?

[@DevOpsToolkit]

Do you mean eclipse as the UI for "stuff"?

[@sarawutklangphukhiao5063]

Can I deploy Port using self-hosted?

[@DevOpsToolkit]

Right now no. I think that they are working on a self-managed solution but I don't have details nor the date. You should ask them on Slack (they are very responsive).

[@zohareini843]

Zohar, CEO of Port here :) We are working on a k8s operator to self-host Port. ETA is ~4 months 💪🏼

[@dirien]

@@zohareini843 Can' wait to test this operator! 😍

[@Khaan-Saaheb]

This video is more on hand movements

[@onemanops]

Its funny I use a product called Victor Ops

[@SergejJevsejev]

> now all you need is to click that button And support all 3rd party tools integrations, updates, and etc. :)

[@DevOpsToolkit]

I don't think i introduced anything others are not using in some form or another. There are pipelines (GitHub actions), infra as code (crossplane), schema management (schemahero), and a frontend (port). Others might make different choices but hardly anyone skips one of those categories except, maybe, frontend. So there's nothing new here except a way how to combine them all (and not even that).

[@davidcourtey1243]

Shouldn't project git repository be create by a custom kubernetes CRD ? Creating them in the pipeline seem to be wrong based on the logic of your IDP Also IMHO database schema and structure should be managed by the backend application using migration script/code, for me it's the developer responsibility and not the infra responsibility. Ps: thanks for your videos I really enjoy the work you have done

[@DevOpsToolkit]

You're right about Git. Ideally, it should be managed by a controller in Kubernetes but I did not yet find one that does that and did not have time to create it myself. As for DB schemas... The manifests to manage the schema are bundled with the rest of the manifests of the app so I guess that counts as "managed by the backend application" unless you meant that it should be managed by the code of the app.

[@joebowbeer]

Exactly. Why not hoist it by its own petard? I'm watching the new video prompted by this comment now.

[@DevOpsToolkit]

The new video was inspired by this comment 🙂

[@julianomoraisbarbosa]

# til

[@horsethi3f]

I heard IDPs are all the rage now. What does this mean to me as a game developer.

[@DevOpsToolkit]

It means that the tasks others were doing for you will be available to you as a service so that you can do them.

[@kostiscodefresh]

It is a real shame that you simply pasted json in the Port UI for your blueprints. Port blueprints should also be handled with GitOps/Crossplane 🙂

[@DevOpsToolkit]

Oh yeah. I'm waiting for it to be k8s CRD