AWS Tutorials - Data Ingestion Services in AWS

In data platform, data is ingested from various sources such as databases, APIs, ERPs or File Systems. There are many AWS Services which can ingest data from these sources. Learn about selecting the right AWS service for the different data sources.

[@Skandawin78]

very good presentation

[@sandeepmodaliar8042]

This channel deserves more than 1 lakh subscribers for sure.

[@AWSTutorialsOnline]

Thanks

[@ranjanasingh7950]

Thank you for this extremely informative Session

[@anchalgupta5674]

Thank you, content is quite helpful and would like to request you please share video with step by step setup of data ingestion pipeline of Glue/Lamda/DMS

[@AWSTutorialsOnline]

I have already provided pipeline configuration using Glue Workflow and DMS. Please search in my channel.

[@Shwetahosur1220]

its very informative. Thank you

[@AWSTutorialsOnline]

You are welcome

[@hsz7338]

Thank you, it is extremely informative, and it implies data ingestion decision making. As an observation, the suggested services in the tutorial are AWS native services. Would you consider AWS MSK or AWS managed RabbitMQ if the data sources are social media and web traffic/front-end client event data?

[@AWSTutorialsOnline]

You are right. The scope of tutorial was aws native services and S3 as the data location. However, since AWS Lake House architecture, support multiple databases registered with various data types of data sources, it is perfectly ok if you use MSK or RebbitMQ for the ingestion,

[@bharathikannan5772]

Very good explanation

[@AWSTutorialsOnline]

Thanks and welcome

[@avishekghosh4879]

Quite Insightful...

[@AWSTutorialsOnline]

Thanks

[@anasjemaa9138]

Thank you, great content! I love your video and you are excellent to synthesize the information. If I use AWS Glue, can I ingest multiple tables in one job?

[@AWSTutorialsOnline]

Yes, absolutely

[@adityamishra7438]

@@AWSTutorialsOnlinehow can we please help me in this, how can we make it automated? like table name changes, can we use for loop for table list ?

[@AWSTutorialsOnline]

@@adityamishra7438 You can pass comma separated tables names in job parameter a the time of job execution. Then loop in the tables and do ingestion one by one.

[@adityamishra7438]

@@AWSTutorialsOnline Thanks, can you please show me the code.

[@vivekjacobalex]

in DATA SOURCE- file system, we can also use direct file transfer from source location to s3 using programmatic way (aws cli, python boto3 ) OR user longing in s3 console .

[@AWSTutorialsOnline]

Technically you can do. Think about security especially if you are using AWS Access Key / Secret Key. Also think about scalability, performance when handling big files especially when using code.

[@vivekjacobalex]

@@AWSTutorialsOnline ok

[@imsanjaya]

Hi It was a excellent comparison. Can we compare the licensing cost also ,plz answer which ingestion tool costs more licensing cost

[@AWSTutorialsOnline]

There is no license cost. Being cloud, it is pay as you go model. You can see pricing of the services online. For instance, here is the lambda and glue pricing. aws.amazon.com/lambda/pricing/ aws.amazon.com/glue/pricing/

[@SumitSharma-zp2sh]

Can you comment on getting data from SaaS applications which only provide APIs, and something getting data of large volume may take 5-6hours. Is glue the right approach?

[@tracyding4906]

AWS EMR is not mentioned, EMR can do some streaming data ingestion, am I right?

[@AWSTutorialsOnline]

EMR can process streaming data. But the ingestion is done using services like Kinesis. So Kinesis ingests data and then hands over to EMR for processing using Kinesis Firehose.