Free Data Engineering Course with AWS, TDengine, Docker and Grafana

▶︎ Link to the GitHub with source code and links: github.com/and...
▶︎ Learn Data Engineering at my Academy: learndataengin...
▶︎ Jump Marks:
01:26 Project Architecture
04:18 Time series data explained
04:58 Time series database vs relational database
06:39 Big pros of time series databases
08:45 TDengine adoption
11:07 Weather API explained and implementation
14:52 TDengine database setup
20:04 How to build the Docker image
21:37 How to setup AWS ECR
21:42 How to upload your image to ECR
23:07 Create Lambda function using Docker image
24:12 How to schedule your AWS Lambda function with EventBridge
25:38 Monitoring your Lambda with Cloudwatch
26:45 Grafana setup with TDengine data source and Dashboard creation
#dataengineering #bigdata #course

[@jefftao2358]

Very good tutorial for time series database and TDengine.

[@andreaskayy]

Thanks Jeff!

[@daniellupercio7173]

Awesome video Andreas. I have been meaning to do this project in Microsoft Azure, using Azure Container Registry, Azure Function App and Azure Logic Apps. The docker file with the python source code and the following docker image is not working well in the Azure Function App. Can someone point me to the proper documentation or similar project done in Azure? I can upload the docker image to Azure ECR, but I am failing in the (@23:07) Create Lambda function using Docker image and (@24:12) How to schedule your AWS Lambda function with EventBridge parts of the video with the Azure services.

[@andreaskayy]

Hey Daniel, did you find a way? I'm sure there's a option to use a docker image as base for an Azure function.

[@daniellupercio7173]

Hey @@andreaskayy. I haven't touched the project since last summer. I remember getting stuck on programming the Azure Function. I'll revisit this sometime this week

[@andreaskayy]

​@@daniellupercio7173 Would love to know if you made it :) Maybe ask chatGPT if it can create you a boiler code for Azure? 😅

[@kpicsoffice4246]

I noticed you haven’t been making new courses on your academy learndataengineering. No new courses or no courses on productionizing data pipelines. Please don’t abandon folks who have spent their hard earned money for your subscriptions.

[@andreaskayy]

We actually do. Last release was in May (dbt for data engineers) There are 4 courses in the making right now. A GCP course, Data Modeling 2, platform & Pipeline Design 2 and a beginner SQL course.

[@Bastlovebast-fu7yr]

Everyone can change the world.

[@andreaskayy]

100%

[@climbersisyphus]

Have to pay a lot to use TDEngine wow..

[@andreaskayy]

Look at AWS Timestream and your head will start spinning 😂