As usual, excellent job Yusuf. Could you please do a video for a roadmap to transition into a Data Engineering role, and if possible, a BootCamp on it

Thanks you so mush, Your Channel is very helpful and affective. I am learning a lot from you.

sir can you please make one project on real time data processing using kafka by solving use case with help of machine learning or deep learning

am learning a lot from you.thanks my brother

why i caught this error while running dbt snapshot Failure to initialize configuration for storage accoutt : Invalid configuration value detected for fs.azure.account.key

thanks 🎉🎉🎉 a real deal😊

Awesome video as always! I learned so much about the ELK stack

Good evening. I have changed the connection as you said but still cannot run the dag. Do you know any other reasons? ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 767, in _execute_task result = _execute_callable(context=context, **execute_callable_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 733, in _execute_callable return ExecutionCallableRunner( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py", line 252, in run return self.func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 422, in wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/apache/spark/operators/spark_submit.py", line 177, in execute self._hook.submit(self.application) File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/apache/spark/hooks/spark_submit.py", line 567, in submit raise AirflowException( airflow.exceptions.AirflowException: Cannot execute: spark-submit --master spark://spark-master-1:7077 --name arrow-spark --deploy-mode client jobs/python/wordcountjob.py. Error code is: 1.

Would love to see this for AWS

Hi , Im facing issue on creating the requirements.txt file It gets created as a directory and not a file

Few things to note in the profiles.yml file if you are following this tutorial. 1. Also update the method: oauth to method: service-account. Great tutorial

Thanks for this video!!! Great content!! I am trying those on my windows system, and I am getting error : Error response from daemon: unable to find user appuser: no matching entries in passwd file. I know this is because in this OS, the Username is returned not the uid . Can you help me fix this?? Thanks Yusuf.

@CodeWithYu sir pls say can we do this in windows?

sir, can we implement this in windows ?

Hi Sir, Thank you for such great video! Im facing an issue, the container is exited right away, localhost is not accessible. docker exec -it redditdataengineering-airflow-init-1 bash Error response from daemon: Container cc4365612e9634f313c75ff7f56bc7dd2c0ac5d0b57a4974ae5304fb546efe9c is not running ERROR: You need to initialize the database. Please run `airflow db init`. Make sure the command is run using Airflow version 2.7.1. redditdataengineering-airflow-worker-1 exited with code 1 redditdataengineering-airflow-scheduler-1 exited with code 1 ERROR: Could not open requirements file: [Errno 1] Operation not permitted: '/opt/airflow/requirements.txt'. checked the permisssions, everything looks fine then I changed the path and provided the full path for requirements.txt Removed the version of Airflow, then changed to 2.7.2 version Would you be able to assist? Thank you!

Please upload an exhaustive video for production scale system using Kubernetes!!!

what is the system requirements? can i make this project in my personel laptop?

Hello bro, I got a problem, the web server image keep saying this: 2025-01-17 13:58:51 exec /opt/airflow/script/entrypoint.sh: no such file or directory I just copy paste the docker file you got on the github repository, could you plis help me?

Hi sir I got the error with large frame below: java.lang.IllegalArgumentException: Too large frame: 5135603447292250188

Good job🎉

This is a really neat demo. You might break it out into a few other demos such as Elastic Search and its dashboards, Configuring and Hooking it all up(with no code), etc. Thanks

It would be nice if you use Terraform to setup the cloud services! and again great video!

The video is super. Sadly, I haven't been able to complete the project, kept getting conflicting packages version errors when trying to run the containers. I'll check another tutorial

I appreciate the work. But is it for me or everyone seeing same - the screen is getting stuck in between , the audio is fine and tried with multiple devices.

trying to run docker compose it gives error version is obselete on vs terminal cna yoiu help me out

Apache flink or Kafka streams with golamg. Would be really great

Very nice!

Could you do a java version of this, JK. But, probably shouldn't have your customers watch you type in DDL. Maybe use a constant typed in once, if you must.

Thanks you for this work. Please can you explain how to create bucket policy ?

Nice!

I want to retrieve metrics from spark, but i only retrieve jvm's metrics, not spark related. Maybe there are some configurations to set in spark application?

provide the datset please

Nice one Yu!

you are 100% spot on))) if everything works you feel like you are a god but if not you know)

Great tutorial. please do the second video of this with real-time dashboard

Nice video. Thanks. But i still don't get it, what the postgres DB for? it was just past through Cassandra.

Great Video

Thanks for your tutorials, is there a particular playlist to start from as a beginner?

Excallent Content 🎉❤.