AWS Kinesis Tutorial for Beginners [FULL COURSE in 65 mins]

ℹ️ johnnychivers....
☕www.buymeacoff...
📁 github.com/joh...
☕ www.buymeacoff...
00:09 - What the course will cover
00:54 - Set Up Work
05:43 - Kinesis Streams Theory
09:01 - SDK Vs KPL Theory
10:31 - Kinesis Data Streams Practical
12:03 - Kinesis SDK
15:54 - KPL Practical
22:26 - Lambda Consumer Theory
23:19 - Lambda Consumer Practical
31:45 - KCL Theory
32:43 - KCL Practical
37:24 - Studio Note Book Theory
38:19 - Studio Note Book Practical
44:22 - Kinesis Firehose Theory
45:17 - Kinesis Firehose Practical
53:27 - Kinesis Analytics Theory
54:03 - Kinesis Analytics Practical
In this course we take a look at AWS Kinesis in full using the console. In under 65mins we go from zero to hero. The course covers the AWS Kinesis Data Streams, AWS Kinesis Data Analytics and AWS Kinesis Firehose. Over the course of 65mins we create a fully functioning kinesis streaming application.
😎 About me
I have spent the last decade being immersed in the world of big data working as a consultant for some the globe's biggest companies.My journey into the world of data was not the most conventional. I started my career working as performance analyst in professional sport at the top level's of both rugby and football. I then transitioned into a career in data and computing. This journey culminated in the study of a Masters degree in Software development. Alongside many a professional certification in AWS and MS SQL Server.

[@xeroxeroxeroxeroxeroxeroxero]

Things to clean up when you're done: - Delete the CloudFormation Template - Empty and Delete the S3 Buckets housing your data - Delete the lambda function created in lab 6

[@picassoofai4061]

yeah, AWS billed me 300$. I use AWS SAM/Terraform now.

[@sany2k8]

This guy is awesome 😎, deserves more subscribers. Keep up your good work on data engineering. I am a full time software engineer planning to move on data engineering.

[@JohnnyChivers]

Thanks for watching the channel Sany!

[@shoooozzzz]

This video took me from Kinesis zero to hero! Thanks, mate. I now feel comfortable producing and consuming data from Kinesis

[@book_beats]

Hello from Ukraine, Johnny! Thanks for your videos! At this point i`ve watched all videos on aws glue and kinesis, they are really great.

[@JohnnyChivers]

Glad you like them!

[@akshayhs2766]

Hi Johnny. Your AWS Glue video was super helpful. Happy to be one of the first viewers of this video !!!

[@JohnnyChivers]

Glad the AWS Glue video was useful - and thanks for the on going support.

[@jeffgoodrich278]

You're awesome Johnny C! I'm transitioning into data engineering in the next 6 months and your videos are great! I would like to study for the AWS solutions architect certs, and I will pay you for a custom project and learning plan. Let me know if you would be open for that, I am an engineer by training, so I do not need to be spoon fed everything, but I do want to take the most effective and efficient path. Thank you!

[@andrewaylward7654]

Love the Belfast accent! Good work Johnny!

[@LeeLeeCode]

Thank you so much!! Very nice and detailed instructions. Much appreciated!

[@OttoTtt]

Guys, don't forget to shut down everything on Zeppelin (Johnny says about this in the video). I missed one query and got a $16 bill after one day. Better delete the whole Cloud Formation right after you finish this video.

[@DakFour]

I deleted the cloudformation template an hour later after finishing the course. i dont remember shutting down Zeppelin. i got a bill of $35.In Billing , Kinesis analytics, Opensearch and ec2 were active( but nowhere to delete them in the console and CLI) it did stop after 24h which meant some applications were still sending data. How would i prevent that in the future. i thought deleting the cloudformation would do the job

[@atulkumar7030]

Awesome Going! Thanks for all your hard work in putting up these videos (y)

[@skviknesh]

Thanks so much! This was awesome!!!! Do more....

[@Chuukwudi]

Why did you not use Kinesis Data Generator ? Why did you write your own custom data generator? Is the data generator presently broken? I have been trying for the last 48 hours to get KDG to work but without success.

[@yinwong667]

Thank you for your awesome tutorial! But can you also teach us how to clean up?

[@shayankabasi160]

Hi Johnny - Thanks for the nice tutorial, could you please check provide some work-through of the yml script that you are using in cloudformation section for resource creation.

[@Chuukwudi]

I wanted to request the same!

[@magneat]

thanks a lot!

[@vishnupandey3390]

19:56 mvn clean compile package will not work we need to change pom.xml file with the following text

[@rakeshmali1727]

one question: do we need to mention which shard a consumer can listen/read to? or the customer automatically discovers the shard to read from?

[@milogodo100pre]

hI, I've been trying to do an exercise which consist on ingest data from an website (currencies), store them, and then show in a graphic the data collected, that's very simple to say but very difficult for me to do it, do you have any information I will really appreciate it. I have the API key from the source of data

[@vipulkumar6082]

Bookmark 48:00

[@databloke2755]

I dont see the jar file in kinesis-producer-library-examples-master folder

[@barisdnmz]

nice tutorial, by the way how to clean up the environment that all set up?

[@TheSuperJCN]

Whenever I launch the CF template it fails with this error: Resource handler returned message: "ServiceExecutionRole has insufficient permission to perform glue:GetDatabase. (Service: KinesisAnalyticsV2, Status Code: 400, Request ID: 0543627e-528b-453e-8e23-9781a5525d9b) ServiceExecutionRole is a role within the template and it does have permissions for glue:GetDatabase (as these are specified in a policy also created with the template) Haven't been able to get around this error to do the tutorial. Has anyone faced this issue? Note: I have Lake Formation enabled in the region I am doing the tutorial, but the error does not exactly mention missing LF permissions, but an IAM action, plusI did try using broad LF permissions but still got the same error

[@lingxiankong4791]

Thank you for guiding this learning!

[@JohnnyChivers]

Thanks for watching!

[@leamon9024]

Hi, is it possible to enable "dynamic partitioning" in kinesis firehose when integrating with cloudwatch subscription filter? I tried it, but it reports an error saying "Only UTF-8 encoded data is supported for dynamic partitioning records.". So now my workaround is Cloudwatch -> subscription filter -> lambda -> kinesis firehose -> S3, which I'm not sure if it's a best practice.

[@smokecracy5831]

bravo

[@jessemcmullen-crummey3158]

was going great until the maven compilation failed for me around 20:00. Got an error: " Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacPro..." :(

[@JohnnyChivers]

Hi Jesse there’s been changes since I did the video. You need to update the POM file to have the latest version of Lombok and that should solve the problem.

[@geoffreyli552]

btw, fixed the 2nd POM issue by changing the java version to lower one, by #sudo update-alternatives --config java and #sudo update-alternatives --config javac.

[@ayaanquadri5203]

same.

[@vishnupandey3390]

change pom.xml file with the following text

[@erickjhormanromero6905]

KPL Practical is throwing the next exception Could not copy native binaries to temp directory

[@parikshithshivaprakash5523]

how to do streaming for large files

[@srinivastadinada3571]

Gem!

[@JohnnyChivers]

Thanks for watching.

[@anshikagupta4931]

niceee

[@undesinyvalley]

thank you Jack black

[@arupde6320]

be regular

[@bruh_1283]

Is this legit

[@xeroxeroxeroxeroxeroxeroxero]

So far, great tutorial. I wanted to point out the Java does not compile. No big deal, as this part can be skipped because the stream already has data on it from the previous step [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project amazon-kinesis-replay: Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors com.sun.tools.javac.processing.JavacProcessingEnvironment.discoveredProcs accessible: module jdk.compiler does not "opens com.sun.tools.javac.processing" to unnamed module @6b350309 -> [Help 1] Sorry, I don't know Java. Otherwise I'd make a PR to fix it.

[@christian112]

I managed to get it running with these updates to the pom.xml file: Old:************************* [...] 8 2.15.66 0.14.6 [...] New: ************************* [...] 11 2.19.33 0.15.7 [...] Edit: Later, for the kcl-app, you will need to update the amazon-kinesis-client version to 2.4.8 to avoid a similar error: software.amazon.kinesis amazon-kinesis-client 2.4.8

[@vipul7010]

@@christian112 thanks it works!

[@kaizen9554]

@@christian112 very helpful. thanks

[@aaronvillegas7900]

@@christian112 This worked! Thank you!

[@madhurnayak55]

@@christian112 Thanks for helping out.

[@nitinsaini05]

Thanks for your video & efforts!! YAML file kinesis-immersion-day-cfn.yaml errored while uploading to create Stack. Something got changed over the period ?

[@JohnnyChivers]

Hi Nitin, I just managed to create the stack successfully in the North Virgina region with my Admin user. What region did you run it in? It took 20mins to create in full.

[@nageshwarburman8819]

I have been getting this error when I ran "mvn clean compile package" : Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project amazon-kinesis-replay: Fatal error compiling: java.lang.ExceptionInInitializerError: Unable to make field private com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors com.sun.tools.javac.processing.JavacProcessingEnvironment.discoveredProcs accessible: module jdk.compiler does not "opens com.sun.tools.javac.processing" to unnamed module @6b350309 -> [Help 1] Did something change by the time when this tutorial was created to now?

[@parakhtayal1254]

same, now in progress to solve that :(

[@halloweex]

same