The NEED TO KNOW Info On Amazon's Software Development

[@rao180677]

One comment I'd like to highlight is that all of amazon's approach is based on CD principles and not CD processes. We don't see them talking about testing piramids for example. What they test and how they test is a result of a system analysis and identification of critical points that and how they are mitigated. These are, and will always be, business specific. For me that is one of the biggest take aways of this. Thanks for this content Dave!

[@mirekrusin]

Amazon has 1mln workforce, 70k tech employees out of which more than half are purely software devs. They serve more than 300mln active users. What you'll learn from Amazon is how to make progress at this scale (of people, interconnected projects, users). Most startups and projects will learn more from smaller scale shops like ie. hey, basecamp, stackoverflow (there are many). All of those microservice shops were running on monoliths at the moment of gaining hypergrowth (github, airbnb, twitter, amazon, linkedin... I mean everybody, no point iterating names). Shouldn't we study what tech they were using at the time when they were achieving this hypergrowth? Ie. RoR is weirdly over-represented in those companies as early tech, why? 99.99% of your visitors don't deal with current scale of Amazon. Amazon doesn'tt use microservices-and-friends because they want to but because they have to at this scale.

[@blaiseutube]

You're very close. It's not the tech they used that matters, it's the problems they chose to solve.

[@mirekrusin]

@@blaiseutube why RoR is overrepresented in successful ones then?

[@kevinfleischer2049]

You are completely wrong here. ;-) The lessons learned of Amazon where not (only) learned now or in the last 2-5years. Amazon solved some really fundamental problems when they where young (~2000-2006) When I read the old blog posts that started Amazons journey down the line of Microservices/2-Pizza-Teams, their situation is nearly 98% identical with what lots of big enterprise IT systems face. Huge slow code bases with 30-100 Devs committing to the same repo without any automatic CI/CD pipeline. Painful integrations, slow tests etc. Thus: Amazons is a prime example for companies that find themselves maintaining a not so majestic monolith.

[@ryanfestag5128]

I would argue that RoR was overly represented specifically because it sped up the process of developing new features. Anyone who has used that framework knows that it's very easy to add new capabilities very quickly, so they got to MVP quickly and were able to build on it quickly. Early on (when you have fewer developers, fewer features, and fewer dependencies), you can just crank out code. There are fewer inherent sources of contention. As you grow, you have more features you can break, more people contributing to the work, and thus more opportunities for things to go wrong. This slows down your development. The speed at which you add features becomes more process bound (how well you verify correctness, how well you decouple, and how independently developers can work without interfering with each other, just like the good old days) than framework bound (how easy the framework makes it to build new capabilities as an individual). And thus team processes become more and more important. This doesn't mean process doesn't matter at smaller scale. It's just more forgiving because you'll hit the contention points less often. It isn't the limiting factor in your ability to meet customer needs (yet). But if you achieve success, it eventually will be. That's why there's value in looking at what the big players do - it lets you look around the corner and see solutions to challenges you will likely face eventually, and you can proactively work towards solutions. Nothing here is, strictly speaking, only applicable at large scale, and arguably would only make small scale development even faster than it already is (it just may not be the dominant factor in increased speed of release in those early days)

[@ContinuousDelivery]

Yup, Microservices *is* a scalability tool.

[@blaiseutube]

As someone who was in the room when it happened, I think most people have the wrong impression of what makes the unicorns successful. If one were to look at Google when they were at a similar stage of development, the differences would be more relevant. Here is the fundamental question that unicorns answer differently from donkeys: "How much better am I at doing this today than when I started?"

[@tongobong1]

Yes Dave learn from Google, Amazon and Facebook! They are all writing classical unit tests with very few mocks and NOT the London style with lots of mocks everywhere.

[@aaronzhong]

Awesome video. I particularly like the idea that it's not all one size fits all - there are lesson to learn from one another and opportunities to take what may work in your context. Imo, this is what makes it fun and exciting! One idea which I haven't gotten my head around with trunk based development is if we use a remote source control repository for the commit phase, how do we prevent that the build isn't constantly being broken as people would push code possibly many times a day?

[@ContinuousDelivery]

TBD is pretty well synonymous with Continuous Integration, so when anyone commits, their commits are evaluated, and if they break the build they will be notified immediately and it is their job to correct the mistake by fixing it, or reverting it immediately. It works fine! Here is my take on the "10 rules that you need to practice CI": kzbin.info/www/bejne/jp2ZY5qHpaafZ9k

[@adrianojordao4634]

I have a question. How good was Pirate Bay software in relation of the objectives of pirate bay? Is small beatifull? Sory for the bad english. Great channel

[@BryonLape]

I'd say must businesses ignore it because it doesn't fit into Scrum.

[@airman122469]

Blindly chasing unicorns is also really stupid. You’ll likely chase them off a cliff.

[@Action-s1v]

Is it not a change in production every 2.4 seconds as a day has 86400 seconds?

[@calunsagrenejr]

First! Great video as always :)

[@sneibarg]

I am a huge fan of the new switch expressions in Java 14. While working the backlog rotation, I had a bit of free time to experiment, so I played around with C-style for loops. It was interesting to see how the latency behaved after submitting enough iterations in a warmup method to not invoke the c1 compiler. It seemed clear to me that is likely a cache hit.

[@takacsot]

hmmm... what an unfortunate fact the officially no amazonian could comment how it is working in practice :)

[@ContinuousDelivery]

Not sure what you mean, this is based on publications from several Amazon employees, links are in the description to the video. “Werner Vogels Talk from 2007” ➡ www.infoq.com/news/2007/08/we... “How Amazon do CD” ➡ www.infoq.com/news/2020/07/co... “Automating Safe, Hands-off, Deployments at Amazon” ➡ aws.amazon.com/builders-libra... “Going Faster With Continuous Delivery”, Amazon ➡ aws.amazon.com/builders-libra...

[@fringefringe7282]

Google doesnt use Agile. Thats my main lesson.

[@ContinuousDelivery]

Well, if so it is the wrong lesson to take. Google is a big firm these days, so I am sure that there is a lot of variance, but Google own DORA which is the home of the "State of DevOps" report, DevOps is a 2nd gen agile discipline and close cousin to Continuous Delivery. Read this abseil.io/resources/swe-book/html/ch11.html#the_story_of_google_web_server Which talks about Google's approach to automated testing. Continuous Integration, and the importance of making progress quickly, in many small steps. All of these ideas are foundational to agile thinking in general, and to Continuous Delivery in particular. So if that really is the lesson you took from this, I am afraid that you are mistaken. Google is a VERY agile company, even if pockets and individual developers may not be and may say otherwise.

[@fringefringe7282]

@@ContinuousDelivery I think the question is "what is the relation between DevOps and Agile development?". I dont have anything against DevOps, Automation, etc. I do have against "Agile development" meant as "this needs to bring business value". And of course to certain degree it does, however middle managers dont know what brings business value, however in 100% of times I've witnessed they caputre the whole process and this turns into Command and Control kind of development and I hate it. Overall this turns into No Value, No Engineering.

[@ContinuousDelivery]

@@fringefringe7282 I completely agree that the idea of "agile" has been diluted to the point of irrelevance in many quarters, and that largely came at the point when it got taken over by project management thinking rather than engineering thinking, but the principles in the agile manifesto are still correct, and very much at the core of how Google operate as far as I can tell, and the fundamentals of Extreme Programming, the real genesis of agile, rather than Scrum, to my mind just work, and so they are pretty well reflected in what I understand of Google's approach. SRE is a very agile way of organising a process of "inspect & adapt".

[@fringefringe7282]

​@@ContinuousDelivery Agile has changed the nautre of our profession. With Waterfall it was an engineering profession, like civil engineering. Whats the nature of Software Development nowadays? I don't know really. The nature seems to be "whatever is in the head of a middle manager". "The Core" has been lost, there is no spine.

[@BryanFinster]

@@fringefringe7282Two things. 1. DevOps isn’t automation. DevOps is the union of people, process, and products to enable the continuous delivery of value to our end users. Minimizing it to “automation” is the same as saying CD == Jenkins. 2. No, waterfall didn’t apply engineering discipline to software. Even if it did, building a bridge is deterministic. Building software is exploratory. It’s product development, not product construction. The tools to do that effectively are different. Show me a Gannt chart for inventing something new.

[@eco8user]

Google software has horrible bugs.

[@plartoo]

So does Amazon's. This guy (the one who made this video) read an article published by an Amazon engineer, who obviously wants to put out a beautiful facade, which doesn't really explain how the sausage is actually made, for her employer (probably for her promotion and career if I am being very cynical), but he hasn't seen/used Pipeline and Brazil that are used in Amazon and how things are tested (or not). :D Everything is pretty/tidy and rosey when you want to sell books/write articles. But in reality (according to my 13+ years of career in software), things are ugly/messy at best.

[@russellsanders4535]

@@plartoo Brazil 😓

[@tongobong1]

@@plartooyes huge systems are complex but when you have good unit tests then you can extend even the most complex systems rather easily and fast.

[@airman122469]

@@tongobong1That’s a product of design, not unit tests.