How to build AWS Glue ETL with Python shell | Data pipeline | Read data from S3 and load Redshift

[@BiInsightsInc]

Link to AWS Playlist: kzbin.info/www/bejne/mXeyY2CdqJppmpo

[@GiovanniDeCillis]

This was extremely helpful! I really like that you are able to compress such valuable information in just 8 mins! I think it would be really useful to see how to build an ETL pipeline in a IaC framework. Haven't see many on the web! Thanks!

[@calvinbutler5517]

You're a hero for the well explained content and then answering everyone's comments. :)

[@BiInsightsInc]

Thanks for the motivation!

[@satishmajji481]

Subscribed!!! Thank you so much for the great content!! Can you please make dedicated videos on how to use AWS Glue, Triggers, Lambda functions and Athena for ETL pipeline?

[@kofio7581]

Thanks great video! Other examples I have seen used a crawler to write the schema of the redshift table to the data catalog before loading using a Glue Job. If I just wanted to do this using only a Visual Glue Job and without a crawler, is it possible?

[@BiInsightsInc]

I am not exactly sure what are you trying to ask. The crawler, crawls the data and infer schema from it. Do you mean to say you want to infer schema without the crawler? Here is a good read on how a crawler infer schema from AWS. repost.aws/knowledge-center/glue-crawler-detect-schema

[@joegenshlea6827]

Hi - thanks for such concise content! I noticed that you deployed to S3 without debugging locally. Suppose i wanted to test the etl script before deploying it? is there a way to execute the etl.py script on the local host using aws_cli?

[@BiInsightsInc]

Yes, you can set up a local development environment to test your work prior to deploying it to AWS. I haven’t covered it but here is an article to get you started. medium.com/@bezdelev/how-to-test-a-python-aws-lambda-function-locally-with-pycharm-run-configurations-6de8efc4b206

[@joegenshlea6827]

@@BiInsightsInc Thank you. In your view, what is the best practice? Using a terminal, lambda console or the procedure in that link you posted? I'm a big time NB

[@BiInsightsInc]

@@joegenshlea6827 I would go with the CLI, I have used the AWS CLI to test code locally. Along with AWS CLI AWS provides SAM CLI to test the AWS Lambda function locally. The Lambda developer guide also advocates for this approach.

[@joegenshlea6827]

@@BiInsightsInc Thank you again! I think I understand. Maybe a video idea for you! I like your videos because you skip all the superfluous nonsense and get right to the meat and potatoes. Good work.

[@koyalmudi007]

Hi, how can we read the credentials from connections or secrets in aws glue python shell, it not working for me

[@BiInsightsInc]

Hi KK, you can get the Glue embedded connection details in Python with boto3. Hope this helps. glue = boto3.client('glue', region_name='us-east-1') # get the connection response = glue.get_connection( Name='name-of-embedded-connection', HidePassword=False ) # get specific connection args response['Connection']['ConnectionProperties']['USERNAME'] response['Connection']['ConnectionProperties']['PASSWORD']

[@PawanKumar-gl4yw]

Hi, Can we transfer 1Tb data from s3 to Redshift using Glue or Lambda +Glue ?

[@BiInsightsInc]

Hi Pawan, you can transfer large datasets with AWS Glue. It’s is a distributed platform, uses Spark behind the scene, to process big data. Lambda is for small to medium size datasets.