Link to AWS Playlist: kzbin.info/www/bejne/mXeyY2CdqJppmpo
@GiovanniDeCillis2 жыл бұрын
This was extremely helpful! I really like that you are able to compress such valuable information in just 8 mins! I think it would be really useful to see how to build an ETL pipeline in a IaC framework. Haven't see many on the web! Thanks!
@calvinbutler551710 ай бұрын
You're a hero for the well explained content and then answering everyone's comments. :)
@BiInsightsInc10 ай бұрын
Thanks for the motivation!
@satishmajji4812 жыл бұрын
Subscribed!!! Thank you so much for the great content!! Can you please make dedicated videos on how to use AWS Glue, Triggers, Lambda functions and Athena for ETL pipeline?
@kofio7581 Жыл бұрын
Thanks great video! Other examples I have seen used a crawler to write the schema of the redshift table to the data catalog before loading using a Glue Job. If I just wanted to do this using only a Visual Glue Job and without a crawler, is it possible?
@BiInsightsInc Жыл бұрын
I am not exactly sure what are you trying to ask. The crawler, crawls the data and infer schema from it. Do you mean to say you want to infer schema without the crawler? Here is a good read on how a crawler infer schema from AWS. repost.aws/knowledge-center/glue-crawler-detect-schema
@joegenshlea6827 Жыл бұрын
Hi - thanks for such concise content! I noticed that you deployed to S3 without debugging locally. Suppose i wanted to test the etl script before deploying it? is there a way to execute the etl.py script on the local host using aws_cli?
@BiInsightsInc Жыл бұрын
Yes, you can set up a local development environment to test your work prior to deploying it to AWS. I haven’t covered it but here is an article to get you started. medium.com/@bezdelev/how-to-test-a-python-aws-lambda-function-locally-with-pycharm-run-configurations-6de8efc4b206
@joegenshlea6827 Жыл бұрын
@@BiInsightsInc Thank you. In your view, what is the best practice? Using a terminal, lambda console or the procedure in that link you posted? I'm a big time NB
@BiInsightsInc Жыл бұрын
@@joegenshlea6827 I would go with the CLI, I have used the AWS CLI to test code locally. Along with AWS CLI AWS provides SAM CLI to test the AWS Lambda function locally. The Lambda developer guide also advocates for this approach.
@joegenshlea6827 Жыл бұрын
@@BiInsightsInc Thank you again! I think I understand. Maybe a video idea for you! I like your videos because you skip all the superfluous nonsense and get right to the meat and potatoes. Good work.
@koyalmudi007 Жыл бұрын
Hi, how can we read the credentials from connections or secrets in aws glue python shell, it not working for me
@BiInsightsInc Жыл бұрын
Hi KK, you can get the Glue embedded connection details in Python with boto3. Hope this helps. glue = boto3.client('glue', region_name='us-east-1') # get the connection response = glue.get_connection( Name='name-of-embedded-connection', HidePassword=False ) # get specific connection args response['Connection']['ConnectionProperties']['USERNAME'] response['Connection']['ConnectionProperties']['PASSWORD']
@PawanKumar-gl4yw Жыл бұрын
Hi, Can we transfer 1Tb data from s3 to Redshift using Glue or Lambda +Glue ?
@BiInsightsInc Жыл бұрын
Hi Pawan, you can transfer large datasets with AWS Glue. It’s is a distributed platform, uses Spark behind the scene, to process big data. Lambda is for small to medium size datasets.