How To Shift Left Infrastructure Management Using Crossplane Compositions

[@DevOpsToolkit]

What do you think about Composites? Could they help in shifting left?

[@barefeg]

I didn’t understand what it means shifting left. But I like composition. Please do an in depth video

[@DevOpsToolkit]

@@barefeg Typically, we say that devs (people writing code and tests of end-user apps) are to the left, and people managing apps, infra, security are to the right. The expression left-right references steps in (traditional) app lifecycle. If we simplify it as write code, test, build, deploy, manage in prod, code/test are on the left and build/deploy/prod on the right. Those tasks are traditionally given to different groups (devs, testers, ops, etc.). As a result, everyone waits for those on the right. Devs wait for testers to test, testers wait for ops to deploy, etc. Shifting left means enabling devs (those who write code of apps) to do everything. However, the cannot know everything, so enabling typically means "here's a service that allows you to do the tasks we do, but in a much easier way that we do it). What do you think? Does that make sense?

[@barefeg]

@@DevOpsToolkit yea thanks for the explanation!

[@dro5484]

Nice video as always! It would be nice to have a public git repo to share crossplane compositions across the community. Like terraform registry for modules. And it definitely helps a lot, abstracting infra complexity so that it is consumed by the devs quickly and easily.

[@DevOpsToolkit]

@@dro5484 github.com/crossplane-contrib oontains contributions which, later on, will be moved to the "official" repo. There is also the conformance program (github.com/cncf/crossplane-conformance).

[@barefeg]

Congrats on the new job. And yes crossplane is awesome!

[@DevOpsToolkit]

Thanks!

[@rodoherty1]

Excellent video, Victor! Very much like your explanation of Left and Right and the relevance of ShiftLeft.

[@SoccerOneOps]

13:13 Yes, Please. A deep Dive into Compositions would be awesome. Great Great Demo. 👏

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@SoccerOneOps]

@@DevOpsToolkit Thanks you and keep up the great work.

[@MarkusEicher70]

Hello Viktor and many thanks for this eye opener. Exactly the way to do it. It is near genius to shift left. I love the quote "unopinionated way to create opinions" 💪Sir, thanks again. I will concentrate my effort towards Crossplane now. I just needed to join this channel. Looking forward to learning much more from you. And btw, I don't care if you joined upbound or not, what you say makes sense to me.

[@cristianocolangelo9920]

Very well done and nice introduction.

[@mk-md4py]

This is awesome. It would be nice to have instructions for digital ocean as well.

[@Peter1215]

Thanks for awesome vid. I've been using terraform for ages and lack of automated drift detection (and potential correction) was always a feature I was missing. With terraform, we have modules and specialized post-provisioning configuration steps to get different kinds of clusters with pre-installed tools and addons, but it's pretty complicated and is bound to the infrastructure, in our case Azure services. So it is hard to have this kind of "provider-consumer" relation between devs and platform team. Crossplane seems to solve this problem with Composites, something similar like DAPR. I like crossplane's philosophy and tooling, especially the ease of combining it with GitOps model. I hope it will get wider adoption, it's very hard to compete with terraform. As for shifting left it hugely depends on company culture and unfortunately I can also see things actively shifting right and happily creating new silos, which is pretty sad to see in this day and age of cloud computing. As you mentioned platform Teams are the way to go (key finding in DevOps Report 2020). I could rumble about this whole day, probably material for a blog ;)

[@santosharakere]

Great explanation. Amazing video. Thank you very much.

[@bappy776]

As always, you videos are rocks. Thanks

[@asawarijoshi9941]

You are super cool man!!! great video!

[@UtkarshMishra-it4oc]

If you can make one video on deep dive in compositions, how we can use Crossplane as a universal control plane defining Gitops, CI/CD and Clusters with crossplane (Crossplane + ArgoCD + ArgoWorkflows), that'd be great.

[@roguepithlit]

Viktor, than you for your videos. Would it be possible to add vsphere and oci(oracle cloud) to your list )? A composition deep dive would be amazing!

[@DevOpsToolkit]

I rarely work with vSphere and/or OCI so it would be a little challenging for me to work on that. Would you like to help? You can get me up-to-speed with vSphere and OCI and I can get you up to speed with compositions and even creating Crossplane providers for those.

[@roguepithlit]

@@DevOpsToolkit I would love to help.

[@DevOpsToolkit]

@@roguepithlit I did a quick search and could not find Crossplane providers for those two. That does not mean that they do not exist though. It's open source so someone could have made it, but I did not find it. Assuming that there are no providers for those, and that Terraform already supports them, they can be generated fairly quickly through github.com/crossplane/terrajet. Maybe that could be the first task if you'd like to contribute to the project. We can continue the conversation on Crossplane Slack workspace. I can hook you up with other contributors over there. My user is vfarcic.

[@roguepithlit]

@@DevOpsToolkit I appreciate it. I reached out to you there.

[@chriwas]

Hi Viktor, I‘m researching automation options in an on-premise, somewhat virtualized, safety relevant and distributed application environment still with explicit hardware dependencies in the application software (i.e. only tested and validated hardware is allowed to run this software). Especially the network design and the configuration and deployment of network components is a very manual process. You present Crossplane as an automation platform to automate basically anything, but all examples, I have found, are in cloud environments. Do you know of an application of Crossplane to automate physical network environments? I would appreciate some hints. My research on network automation leads to Ansible, Nornir or vendor specific solutions, maybe with Netbox or Nautobot as network source of truth. I have not found a solution that uses Git as a network source of truth and applies true GitOps principles to network automation.

[@DevOpsToolkit]

Crossplane depends on providers that extend it with additional resource types. Someone (you?) would need to create a new provider for that. As long as there is an API to talk to, that should not be a problem.

[@srgpip]

Thanks for another excellent video Viktor. Regarding Crossplane: Am i correct in assuming it is designed to only function on the cloud providers managed kubernetes offerings? I ask as our use case found the managed EKS offering too limited and oppininated partly due to limitations with pod density caused by AWS-CNI / ENI / Instance Type. We reverted to KOPS so we had more control over the control-plane and CNI. Would be great option if Crossplane could employ KOPS / Terraform providers if does not already! ? Thanks again.

[@DevOpsToolkit]

Crossplane itself can run in any Kubernetes, not only in EKS/AKS/GKE. As for KOPS... As far as I know, there is no provider in Crossplane that implements KOPS. Would you volunteer to work on it (I can help)? As for the Terraform provider... I did a video with Nic about it a week ago on the Crossplane channel. You might want to check it out.

[@srgpip]

@@DevOpsToolkit Thanks for your reply Victor. I will definitely delve a little deeper into Crossplane along with checking out the Terraform provider. Regards to committing I would love too but sadly I am in the process of building a startup and currently pulling crazy hours, if I take on another project at this time the lady of the house will be serving me with divorce papers (at best) ;-) But if I do find some time I will gladly revisit the idea :-)

[@DevOpsToolkit]

@@srgpip I have the same problem. There's always too much to do and too little time. I'd love to know a bit more about your startup. Would you like to share for the idea is? Maybe as a private message on Twitter or LinkedIn?

[@SV-tc8cu]

Great explanation, thank you Victor. I am wondering what value would Developer get when they use Compositions instead of Terraform modules provided by Ops team? Both compositions and TF modules could be used as an interface to a complex set of infrastructure resources.

[@DevOpsToolkit]

The major difference is in API. Crossplane Compositions are fully qualified Kubernetes CRDs with all the benefits that brings. Terraform modules, on the other hand, are just "libraries" (of sorts) that you can invoke in your manifests. On top of that (e.g., API), given that they are all k8s resources, you can combine them with (almost) any CNCF tools. For example, you can use them with Argo CD or Flux. At the end of the day, the most important question is how important it is to have an API and a control plane. If those things do not matter, than Terraform is a better option.

[@andreykaliazin4852]

Composites are great! Would love to see how are they used within the GitOps framework with ArgoCD. On a separate subject - what's your take on the Brainboards IO? They offer a gui for the Terraform. What I'd really like is to have some way to create a diagram automatically from the rolled out k8s landscape. Not necessarily using TF. An using Composites in those diagrams would be just perfect - we do not really want to look at the low-level details all the time.

[@DevOpsToolkit]

I haven't (yet) used Brainboards. Is it brainboard.co? Having diagrams auto-generated would be a great addition to Crossplane. How about you start a conversation about it in the community? The Crossplane Slack workspace could be a good place.

[@andreykaliazin4852]

@@DevOpsToolkit yes, thats the web site. I did not want to quote it myself, being bitten before. :-) If you could add the BB overview to your TODO list, that would be great! Automatic documentation generation with diagrams like swagger does for APIs would be extremely helpful for the k8s world.

[@DevOpsToolkit]

@@andreykaliazin4852 Adding it to my TODO list... :)

[@ashleymail4u]

It would be awesome if crossplane would have a visual composition editor or vscode extension

[@DevOpsToolkit]

We are getting there. The next step is VS Code extension which is available at github.com/upbound/vscode-up. I believe it will be officially announced very soon. Apart from that, the work is starting on creating a (more potent) visual editor in Upbound Cloud but I do not yet have more info I can share on that one.

[@rotemjackoby3112]

Thank you Viktor ( again and again..(: ) for the great video. I have a small question: In the XRD (definition.yaml) you specified in the claimNames field that the claim kind will be equal to "KubernetesCluster" But when you wrote the claim in cluster.yaml: 1. You specified a the kind of "CompositeKubernetesCluster". 2. You've added the "compositionRef" field which specifed the Composition that this claim is "using" (or "refering", not sure about the correct word here). Is this a different approach then what we see here in the Upbound example: github.com/upbound/platform-ref-aws In the "Cluster" resource in: github.com/upbound/platform-ref-aws/blob/main/cluster/definition.yaml#L85 They are specifying the name of the claim an this is what is used in the claim in here: github.com/upbound/platform-ref-aws/blob/main/examples/cluster.yaml#L2 Are these two different approaces for exposing claims or am I missing a lot of things here?

[@DevOpsToolkit]

When you create an XRD, it can be used directly (as in my video) or through the claim. When used directly, you're creating a cluster resource, while claim allows you to create namespaced resources. The latter (claims) is arguably a better option than what I did in the video. I will probably do another video explaining the differences between using XRDs directly and through claims.

[@rotemjackoby3112]

@@DevOpsToolkit Thank you Viktor for the clear answer. I think composite resources is one of THE biggest strengths of Crossplaplane but its not quiet trivial to understand all the options of how to use it and the different use cases. I think that a video that explains the 3 sections below will help a lot: crossplane.io/docs/v1.3/concepts/composition.html#using-composite-resources crossplane.io/docs/v0.8/services-guide.html#dynamic-provisioning-with-claims-and-classes crossplane.io/docs/v0.3/concepts.html#resource-claims-and-resource-classes (*) Please notice the different Versions - I'm not sure that some of those topices are even relevant anymore (also a source of confusion in Crossplane's Offical docs).

[@DevOpsToolkit]

I will do my best to cover those soon. I will probably do that in the upbound or the crossplane channel because a) that might be a better place and b) to keep this channel objective (not to make it crossplane-specific). Does that make sense?

[@rotemjackoby3112]

@@DevOpsToolkit Sure Viktor! Thank you (:

[@eamonnmccudden1070]

How would this work from the testing of the XRD? For example, when making custom modules (in TF) that might wrap some logic up, I can spin up some local tests as part of the PR - In this is there a method to test out values? I.e if you typo the available version of K8s to a 2.21 or something, would that only be captured at that state (unless caught by a human in the PR process?)

[@DevOpsToolkit]

There is no crossplane-specific testing framework. If you do test it (and you should), normally you'd spin up infra in parallel with production, test it, and then apply the same to prod. It would be ideal if some form of mocks are created, but, as far as I know, no one did that (yet).

[@zhechozhechev118]

Only one question: How do you upgrade statefull infrastructure (i.e. RDS, DocumentDB, etc) described as crossplane composition ? Suppose you forgot to set some options tags, policies...etc) ? Creating new composition will not update current instance options... you'll need to move data/traffic from old ones to new ones somehow?

[@DevOpsToolkit]

You should edit the composition, not create a new one. Now, if the changes should apply to all instances of something managed by Crossplane, than an update of the composition will make sure that all managed resources are updated. On the other hand, if you want to control the update for each instance individually, extend the CRD of that composition so that you can control that change for each managed resource separately.

[@zhechozhechev118]

suppose you have to change definition and composition, because you forget to add tags or other option and you want to change that option via claim. How do you proceed with that if you have only 1 RDS instance that has data already in place? setting tags is not working as expected or I missed something? Do we have documentation about similar cases ? I've tried that scenario several times and didn't find a way to update RDS with tags.. besides of recreating it ..

[@DevOpsToolkit]

@@zhechozhechev118 Generally speaking, changing a composite resource definition (XRD) results in those changes being propagated to composite resources (XRs). Now, I cannot guarantee that some specific case does not work (doesn't have bugs or something ommitted). Also, some providers (e.g., AWS, Azure, etc.) do not allow updates of some resources so that might also be the issue you're experiencing. If setting tags does not work as expected, and the issue is not that the provider (e.g., AWS) does not allow changes to those properties, you probably encountered a bug and, in that case, the best option is to open an issue. Ideally, since it's open source, an issue with a PR would be ideal, but issue alone is great as well.

[@TechStory5]

Hello and thank you for this very in sightful video I have a question. Currently i'm working on a web app that allows users to digitally sign pdf docs but i don't want to restrict end users to this app, i want them to use documents let's say stored in Google Drive to sign them using this app. The idea is to convert my app to a service that can be used by other apps hence the name of my project "non repudiation as a service". Is there a devops toolkit that allows this and is it even possible or is there another close alternative ?

[@DevOpsToolkit]

There are many thing in that description so I'm not sure which one you're referring to when you say "toolkit that allows this". Do you mean toolkit that allows us to create a service consumed by other arrives?

[@TechStory5]

@@DevOpsToolkiti'm sorry i'm not really into devops but i just want my app to be a service for other applications that do document exchange and other stuff but don't include digital signature

[@DevOpsToolkit]

@@TechStory5 I'm not sure about digital signature... As for the rest, you would normally expose the API of that service and, from there on, other services can invoke that service and either get the file directly or get a temporary URL to the file residing in some storage.

[@TAICHI1SCO]

Help please, f I create a cluster with a provider aws-account-a and then try to create another cluster with another provider aws-account-b, kubectl does not create the new cluster but instead does a configure. So am left with 2 clusters in different accounts but kubernete is only aware of one clusters.

[@DevOpsToolkit]

It's hard to answer without having additional info. It could be an issue in manifests, AWS, or almost anywhere else. Can you open an issue in the Crossplane project and post the steps to reproduce it? That would help us help you.

[@CloudNativeJanitor]

I belive you ar Awesome, thank you, and I belive crossplane is awesome tech and mostly people

[@ElvenSpellmaker]

Going back to my previous comment on your channel and something this video doesn't seem to address is, how do you bootstrap the initial cluster so people can use these definitions in the first place? You're telling developers to just kubectl apply on the custom resource but there must be some underlying cluster already present. I'm liking the idea but I feel the chicken and egg in this case is much harder to solve. How does Crossplane recommend you solve this, use Terraform for the base cluster? Also would you say it's best to have a small cluster for Crossplane only and then applications use that to spin up a new cluster (or new clusters) to run the apps on?

[@DevOpsToolkit]

It is, in a way, the "chicken and egg" problem. You need the initial k8s cluster to run Crossplane. But, once you do get that cluster, no matter how you create it, you can tell Crossplane to manage it. So, from that moment on, Crossplane will not only manage everything else, but also the cluster where it's running. The alternative is to use Upbound cloud (cloud.upbound.io/). Over there, you can get a hosted control plane (a k8s cluster with Control Plane running in it). Bear in mind that something similar is true for Terraform as well. If you're self-hosting, you need storage for Terraform state and you are likely going to create it manually, or you might use Terraform Cloud for that in a similar way you'd use Upbound Cloud. For most of the companies I'm working with, that initial "investment" into the cluster with Crossplane tends to pay off quickly since they tend to manage resources at scale (e.g., tens, hundreds, or thousands of clusters and other types of resources).

[@ElvenSpellmaker]

@@DevOpsToolkit I suppose with Terraform it's just a storage account (or also dynamo for AWS) so it seems far less of an issue than an entire k8s cluster be it AKS/EKS or otherwise. I suppose you could use a local K8s cluster such as Docker for Windows or k3s (through Rancher perhaps) and then transfer the definitions to the cluster once it's up and running.

[@DevOpsToolkit]

@@ElvenSpellmaker Oh yeah. Creating the initial cluster (for Crossplane) is not as simple as creating the initial storage (for Terraform). So there is definitely bigger initial investment for Crossplane. It's not a huge one though since almost all the providers have a simple and easy way to create the initial cluster, but it's still an investment. Now, that initial cluster can be any k8s cluster, including local clusters like k3d, minikube, kind, etc.

[@sergei5879]

This is all cool with shifting, but... Can you make an introduction in Crossplane for someone for who kuber is more of proof of concepts as containerisation on its own. I know its kind of a question in 2021, but yet. Is there something Crossplane and kube can afford to more of an Ops guy, who used to terraform IaC only stepping in kube realm? I really liked that Crossplane giving you an immutable (ansible-like) state, but they's hardly a big demand for managing kuber itself. Cheers!

[@DevOpsToolkit]

Apart from composites, what distinguishes Crossplane from Terraform is that it uses Kube API allowing you to have a single API for everything (infra, apps, services, etc.), constant drift detection and reconciliation and that it follows GitOps principles and, hence, it works with the tools like Argo CD and Flux. You might want to check kzbin.info/www/bejne/pGmum4mjqq2gbaM and kzbin.info/www/bejne/r6PNZZ-jiMh-frM. These provide an overview of those features.

[@sergei5879]

@@DevOpsToolkit I guess my main concern that in big organizations hardly anyone has really "unlimited permissons". As for us using azure for example from documentation. you need to grant admin permissions on the Azure Active Directory to the service principal because it will need to create other service principals for your AKSCluster Note: You might need Global Administrator role to Grant admin consent for Default Directory. Please contact the administrator of your Azure subscription. compared to terraform, where I can have a service principal that has contributor to a particular resource rather than godlike permissions across subscription.

[@DevOpsToolkit]

@@sergei5879 I think that's one of the strong points of Crossplane. You can (and should) give Crossplane wide permissions but, since it's running in Kubernetes, you can use k8s RBAC to control who can create which k8s resources, including those managed by Crossplane controllers. In other words, people do not need Service Principals. They do not interact with Azure. They interact with Kubernetes API.

[@sergei5879]

@@DevOpsToolkit well, totally fair point for the way of design for some cases but not all. For big/ government organizations its quite often case when you actually not owner (global admin in AAD as example for Azure) in cloud. And I`m not talking only the way it just another department, but it might be CSP - meaning its owned/managed by 3d party. So as example - I can be an owner of anything inside "subscription" but not an owner of subscription. So in my case - I would like to use APi of cluster as a single source of control for infrastructure automation, but I in a nutshell can't authorize Crossplane.

[@DevOpsToolkit]

@@sergei5879 You're right. If you can't authorize Crossplane, it's a no-go.

[@alissonr.deoliveira6567]

What happen if I, as an infrastructure engineer, change the default values? What happen with the in-place clusters? Imagine I change the minNodes from 1 to 3... My cluster will be touched or not?

[@DevOpsToolkit]

If you change the default value without changing the version, all those that did not specify that value will be updated. If that's not the desired behavior, you should also change the version of the xrd.

[@alissonr.deoliveira6567]

@@DevOpsToolkit makes sense!

[@AntonioSy-A-Foek]

I am evaluating crossplane. When I create an s3 bucket I get warnings that acl are not enabled and that policies should be used. Can you help out what should be configured to let this warning/error disappear

[@DevOpsToolkit]

Not sure from the top of my head. That message is coming from aws itself though and is not directly related to crossplane.

[@chengjohnny5228]

This might be a dump question. I tried to create a cluster with all EC2s, but for some reason, I can only create one EC2 instance even if I define multiple blocks in Composition. Could you answer this for me?

[@DevOpsToolkit]

You should be able to combine any number of resources in a composition, including multiple EC2 instances. Can you send me the link to the manifests you're using? I might be able to provide a more concrete answer as to why it's not working in your case if I have a closer look and try to reproduce the issue.

[@chengjohnny5228]

@@DevOpsToolkit That's what I think too, otherwise this tool is too bad :) I just created a github repo for you to review it. github.com/Johnny00520/infra/blob/main/aws.yaml branch: main Please let me know if you have any questions, or some other way you'd like to DM :) Thank you for your time, and a great video you provided. Very helpful

[@DevOpsToolkit]

@@chengjohnny5228 I don't see anything (obvious) that would make me thing that would not work and create, among other things, 3 ec2 instances. How about we do a screen-sharing session and try to debug it together? If that sounds like a good idea, please pick any time that works for you from calendly.com/vfarcic/meet.

[@chengjohnny5228]

@@DevOpsToolkit Thank you, sir. Just book a time with you :) Have you tried to run it based on the repo? If you haven't, you will see only one instance will have been running. Unfortunately, there's not many research I can do for this problems

[@chengjohnny5228]

@@DevOpsToolkit Hello Viktor, I found out where the problem is, I have the same code for each EC2's FromCompositeFieldPath and it will overwrite the metadata.name.... Cannot believe I made this mistake. I will cancel the meeting with you :) Thank you for your time!! And hopefully I can learn more from your videos :)

[@NeverDoubtMe23]

So lost :( a XRD is a composition of different resources that make up a service. A composition is a combination of one or more resources done in a certain way...???...???

[@DevOpsToolkit]

XRD is sort of a Kubernetes CRD and it can have one or more compositions associated with it. Each composition is a collection of kubernetes resources or XRs.

[@NeverDoubtMe23]

@DevOpsToolkit thank you sir, you rock for responding and doing it so fast, love your content.

[@MuratKeremOzcan]

Can you just paste the script between minutes 2-3 so I can quote it everywhere?

[@DevOpsToolkit]

I'd do that if I'd have a script. My firsts videos had a script (text that I was reading). But, after a while, I stopped doing that and just talk to the camera. I feel that's more genuine and natural.

[@valour.se47]

Combine it with ArgoCD and do nothing 😀

[@denisrazumnyi6456]

Such an incendiary story, but ... The developer just wants and needs to write the code. He doesn't care about cluster. He wants it was tested and deployed and then tested in cluster. They want just get the logs or traces + metrics. This tool is useless for the developer. It's kind of division for DevOps team but it's abstraction on top of abstraction.

[@DevOpsToolkit]

I think it depends from one team/company to another. In some places, devs are not looking at metrics, in others they are. In some, they like the idea of deploying their own apps, in others don't. The same can be said for infra. It truly depends on how much control each team wants to have.