AWS Glue - Access On Premise Database and Transfer Data

[@GabrielRato-m3f]

Tks a lot man, been watching some other vids about the theme and a lot of them people were not going direct to the point, you did! save me a lot of time, tks again

[@TEACHDATASCIENCE]

I am excited to use this kind of way to upload s3 in glue.

[@FlaggedStar]

How are you getting this to work without putting any Connections in the definition of the Glue Job? I thought they were mandatory.

[@marcoap]

Perfect video! I don't know if you will see my question. Did you create a "Connection" in AWS Glue, or is the on-premises database IP exposed on the internet? In this case, should I use something like a VPN from AWS to create a secure way to connect to it? Thanks a lot!

[@francismagnusson378]

THANK U FOR THIS 😭🙏 im currently working as a data engineer and man glue feels kinda limited (probably because im new to it). ive been trying to connect to our DB using only the python shell glue job, but the thing is, i have to setup a driver for it. theres a workaround using this library called pymssql but sometimes it doesnt work. i hope u can add tutorial on how to add drivers and use it on python shell jobs

[@akhilvarma6648]

where do i get my host details from, since mysql in my local machine(mac) would the host be localhost? please help me on establishing this connection, i'm working on it from past 2 days.

[@AY1986R]

thank you for this great video, but I do not see where you have created the connection, an example with JDBC oracle will be great

[@arunnair1392]

In this case, I guess , no incoming ip restrictions are in place on the server side . Mysql server port must be open for all traffic . So no connection is required. But for enterprise implementations , we must create a secured connection

[@krishnareddy4948]

Great content. Please do a demo on Oracle source system in Glue

[@codingwithlu]

Will do!

[@xroydacute]

@@codingwithlu Hi Lu, any chance you have done a video for Oracle source system?

[@sunilmakwana2898]

Great Information! Please make a video on how to load multiple table to the S3 bucket from on prem server?

[@codingwithlu]

Sure will make a video of that

[@DevTipsForYou]

This is why i like azure data factory... it can connect to to anything without any code/config...

[@rafaelfpk2079]

Does your glue applications takes long time to run for the first time? I'm having a huge problem with this here, with 2 DPUs, just connecting to AWS RDS and WRITING into S3, like your application, it took 30minutes and gave timeout.

[@codingwithlu]

It shouldn't take too long. I have a few glue jobs, and some of them are querying a few million row of data, and it takes just a few minutes. How huge is the data that it's querying?

[@rafaelfpk2079]

@@codingwithlu Very small, 50 rows...

[@peterpan9361]

Great content !! Subscribed :)

[@codingwithlu]

thanks for the sub! move content coming!

[@andrezzo80]

what about using coalesce to make sure the data stays in 1 partition?

[@nizor_]

Awesome content. How can i connect to a firebase realtime db?

[@codingwithlu]

Glad you liked the content. I would need to do some testing to see how to connect to firebase db. I would let you know what I find.

[@kshitijbansal3672]

How to make a pipeline with Aws Glue to move data from On premises SQL Server to S3? Pls reply in detail sir.

[@codingwithlu]

I will make video that is specific for On Premise SQL Server to S3

[@poojabharathi4826]

what all service access must be included in iam role to load on premises to s3

[@ashutoshsharma4525]

Thanks for nice video. Could you please provide the link for your code. I am getting some parenthesis missing errors while running it.

[@achyuthvaleti3639]

How can I get the Mysql Connection details

[@codingwithlu]

If it is a MySql instance in AWS RDS you can get it from the AWS console. If it is a on-premise instance, you can connect via IP address.

[@tintin1678]

How to connect to Oracle Db in AWS Glue ? I'm getting error using cx_Oracle lib

[@codingwithlu]

I will make a video over this

[@sweetysweetyvghb]

Hi, did u make a video on this

[@sweetysweetyvghb]

Please make a video on how to connect on premise oracle database from lambda

[@appurvagnihotri2218]

how to get mysql database info to add in code?

[@kshitijbansal3672]

How to make a pipeline with Aws Glue to move data from On premises SQL Server to S3? Pls reply in detail sir.

[@kshitijbansal3672]

How to make a pipeline with Aws Glue to move data from On premises SQL Server to S3? Pls reply in detail sir.

[@akarshba3052]

Hey, were you able to do it?