Пікірлер
@calvinharris4771 6 сағат бұрын
25:12 Could you pls specify more on the Remove duplicate option in the query (to prevent data-redundancy if we run the pipeline twice ) ?
@sivaprasanth5961 Күн бұрын
This video is really amazing and next level.
@cloudquicklabs 22 сағат бұрын
Thank you for watching my videos. Glad that it helped you.
@realamrutpatil Күн бұрын
Great explanation! Thanks.
@cloudquicklabs Күн бұрын
Thank you for watching my videos. Glad that it helped you.
@ulrich1081 2 күн бұрын
I'm having this error: py4j.protocol.Py4JJavaError: An error occurred while calling o191.getCatalogSource. : com.amazonaws.services.glue.util.NonFatalException: Formats not supported for SparkSQL data sources. Got json at.... Does someone may know what could be happening.
@sandeepreddy9689 2 күн бұрын
Hi, @cloudquicklabs What was the use of second crawler. Is it only run once to get schema of redshift in to temp database and later it won't be running any time?
@noushadktnoushad7383 2 күн бұрын
Good explanation !
@cloudquicklabs Күн бұрын
Thank you for watching my videos. Glad that it helped you.
@aslanbaharloo8150 3 күн бұрын
great video, thanks, I'd like to use the data on s3 via athena, do you have a solution for it?
@cloudquicklabs 3 күн бұрын
Thank you for watching my videos. Please find the link kzbin.info/www/bejne/a3iTmmirhrt6f9k I have one more video on this but using csv file.
@hafizadeelarif3415 3 күн бұрын
that's great explained
@cloudquicklabs 3 күн бұрын
Thank you for watching my videos. Glad that it helped you.
@hafizadeelarif3415 3 күн бұрын
❤@@cloudquicklabs
@karthickmuthiah7241 4 күн бұрын
Very Good session, i have an automation SSM document which needs to call in my terraform code, would you help me with the procedure or documents.
@cloudquicklabs 3 күн бұрын
Thank you for watching my videos. You can use terraform null_resource something like below and call AWS CLI to invoke the AWS SSM document resource "null_resource" "example" { provisioner "local-exec" { command = "echo This command will execute whenever the configuration changes" } }
@suneelkumarparvatala7527 4 күн бұрын
Hi Thank you for the informational videos, would you clarify this doubt I have, here gluecrawler created and ran only once I believe instead of creating glue crawler can it be possible to migrate data directly from s3source crawler to redshift table . My intention of asking this query is since we are running gluecrawler only once or only at a time when we would like to see records in redshift table since the functionally of accessing records in redshift is possible by quering in the query editor
@cloudquicklabs 3 күн бұрын
Thank you for watching my videos. We migrating data from source s3 to Amazon redshift which is destination here. Crawlers can be scheduled or invoked on demand.
@Lab-0420 4 күн бұрын
thank you ok haha
@cloudquicklabs 3 күн бұрын
Thank you for watching my videos.
@tjhsiao06 4 күн бұрын
ok? yah?😁
@cloudquicklabs 3 күн бұрын
Thank you for watching my videos.
@yogithakakarla1716 5 күн бұрын
Wonderful scanerios. Instead of just giving brief . The way you are explaining different scenarios is amazing
@cloudquicklabs 5 күн бұрын
Thank you for watching my videos. Glad that it helped you.
@yogithakakarla1716 5 күн бұрын
Thankyou . Need more such videos
@cloudquicklabs 5 күн бұрын
Thank you for watching my videos. Glad that it helped you.
@yogithakakarla1716 5 күн бұрын
@@cloudquicklabs I have doubt on how it provisions the new VM. Is it based on provisioner file configurations where we give instance types. Or just dyanamically any instance type and instance size
@user-fn8oc2dw4m 6 күн бұрын
Hi Sir, Thanks for making the things easier. I have a doubt from where you got that result.json and Helppowershell file before updating the cluster? Thanks!
@cloudquicklabs 5 күн бұрын
Thank you for watching my videos. Please check the video at 5:23 minute you see that I am executing a powershell script which gives the output and stored in results.json.
@hafizadeelarif3415 7 күн бұрын
Hi sir what is the difference between EMR Cluster vs EKS Cluster?
@cloudquicklabs 7 күн бұрын
Thank you for watching my videos. EMR : is Elastic Map Reduce which is managed big data processing and analytics service. EKS: is Elastic Kubernetes Service which is basically managed Kubernetes (container orchestration) Service. Both are different.
@hafizadeelarif3415 7 күн бұрын
@@cloudquicklabs Okay, hum yahan kah sakte hain ki EMR jo hai, wo Spark aur PySpark ke jobs run karta hai aur EKS jo hai, wo Python ke jobs run karta hai.
@anjanianju2408 7 күн бұрын
I want copy from dynamodb in one table data to paste it in couchbase in one collection.
@cloudquicklabs 7 күн бұрын
Thank you for watching my videos. We have solutions for it I shall creat a videos on this soon.
@SheikhIrfan-qi2if 8 күн бұрын
thanks bro you save my time
@cloudquicklabs 8 күн бұрын
Thank you for watching my videos. Glad that it helped you
@WolfmaninKannada 8 күн бұрын
Excellent sir thank u so much
@cloudquicklabs 8 күн бұрын
Thank you for watching my videos. Glad that it helped you.
@DhanunjayaTalari-g4c 9 күн бұрын
Bro do video on how does interrupt queue working with example.
@cloudquicklabs 7 күн бұрын
Thank you for watching my videos. Indeed I shall create video on this scenario as well soon. It's quite interesting scenario.
@mhamadnemer1230 11 күн бұрын
Thank you for this richfull video , i want to ask you what type of ec2 instance you have used ? i have a similar project and i need to identify the cost metric and hardware requirements, so what you suggest for similar project that host a simple web application using kubernetes cluster and prometheus monitoring?
@cloudquicklabs 10 күн бұрын
Thank you for watching my videos. Cost of EKS cluster depends on multiple choices 1. Size of Node 2.Disk usage 3.Networking Resources. etc. Did you AWS calculator for this. And also explore my KZbin channel to find correct videos for your use case (as there are many here)
@shubhamgarde6830 11 күн бұрын
I need to send different messages to n different users, Can we implement API Gateway triggers lambda and the lambda triggers SNS or SQS, to which we send a custom mobile number and a custom message. Please let us know if we can implement this. Thanks in advance.
@user-nf1nj9bc7h 11 күн бұрын
Let me know when you find the answer to this.
@cloudquicklabs 10 күн бұрын
Thank you for watching my videos. Indeed you can do it and it's workable solution. I would create new video on this topic which mimic your scenario, keep watching my channel.
@cloudquicklabs 10 күн бұрын
I would sharing a video on this requirements soon.
@user-nf1nj9bc7h 10 күн бұрын
@@cloudquicklabs thank you so much. This means a lot🙏🏽
@techconcepts4you850 12 күн бұрын
Appreciate and subscribed!
@cloudquicklabs 11 күн бұрын
Thank you for watching my videos. Glad that you liked it.
@accountfortesting6321 12 күн бұрын
HI, can you suggest, if we have nginx ingress with controller using in EKS then, in this senario do we have to use istio service mesh with nginx ingress, controller, how we can verify whether istio is using or not in my EKS cluster?
@cloudquicklabs 10 күн бұрын
Thank you for watching my videos. transitioning from nginx-controlled Ingress to Istio service mesh involves replacing the Ingress controller with Istio's ingress gateway and using Istio's custom resources for advanced traffic management, security, and observability within the Kubernetes cluster. It requires changes to your Kubernetes YAML files and deployment strategies to fully utilize Istio's capabilities.
@rutujabadre4372 13 күн бұрын
How you added aws access key id , secret id region name during aws configure command ?
@cloudquicklabs 13 күн бұрын
Thank you for watching my videos. Indeed I have used 'aws configure' and provided required secret and key and region.
@user-bo9ls6fu6p 15 күн бұрын
this video save my Final topic what a wonderful function
@cloudquicklabs 15 күн бұрын
Thank you for watching my videos. Glad that it helped you.
@rajash1819 15 күн бұрын
I will buy a coffee for sure 😅
@cloudquicklabs 14 күн бұрын
Thank you for watching my videos. Appreciate your time here.
@rajash1819 15 күн бұрын
Please help me with out - thanks so much
@cloudquicklabs 14 күн бұрын
Happy to help you, please find response below.
@rajash1819 15 күн бұрын
How batch jobs work like informatica workflows. Migrating informatica workflows and sql jobs from oracle to Postgres using lambda , glue, S3, DMS
@cloudquicklabs 14 күн бұрын
I did not get requirements correctly here. Do you want migrate Oracle Database to Postgresql here ?
@rajash1819 15 күн бұрын
Hi brother need some information
@cloudquicklabs 14 күн бұрын
Please provide more details here to help you.
@FIBONACCIVEGA 15 күн бұрын
great video . I just have a question abut how I can send info from my python code to the bucket on google. I create the bucket previously on google cloud but im not sure what's the info or key I have to copy in my code to send this info to the bucket. (key_path = '' project_id = '' dataset_id = ''') Thanks in advance
@praveenbadagi4285 15 күн бұрын
Can I do this work in free tire time?
@cloudquicklabs 14 күн бұрын
Thank you for watching my videos. I believe there is no free SMS here. May be can check better by going through below link : aws.amazon.com/sns/sms-pricing/
@devops_jaswanth 16 күн бұрын
Any option to the contents of the file instead of just file name over the mail
@cloudquicklabs 16 күн бұрын
Thank you for watching my videos. Indeed we also pass the content of the file as well but that you would need to read the file programmatically and then use in sending the Notification.
@RohitTiwari-mo8mi 16 күн бұрын
Is there a way to iterate over the provider in the module resource instead of creating separate resources and module per region?
@cloudquicklabs 16 күн бұрын
Thank you for watching my videos. I believe separate providers tf per region but single source file makes it meaningful. May I know what is your requirements in more details.
@RohitTiwari-mo8mi 16 күн бұрын
@@cloudquicklabs I am trying to create multiple vpc across the region using single module having single resource defined that dynamically takes the variables values for vpc cidr and provider alias. Don't want to create multiple modules for separate regions instead trying to use a single module having a single resource defined, want to keep it DRY
@aniketjain4615 17 күн бұрын
How to get the logs with last modified time stamp in their name? Because after exporting the logs to s3 we do not have time stamp, so its very difficult to differentiate which log it is
@cloudquicklabs 14 күн бұрын
Thank you for watching my videos. You might need log your workflows with using custom modules.. like below. import logging logger = logging.getLogger() logger.setLevel(logging.DEBUG) logging.debug("hello?")
@strangemate 18 күн бұрын
How are you able to get the selection of data files viz. covid.json file state.csv from within the Visual ETL section. Where did you place them initially?
@cloudquicklabs 17 күн бұрын
Thank you for watching my videos. I shared those source data files, can be kept in your s3 bucket and then you would need to create crawler to Extract data from same and store it in data catalog table as shown in video.
@jaideep1222 18 күн бұрын
Do we need to run crawler every time when ever there is a new data that comes into S3 ?
@cloudquicklabs 18 күн бұрын
Thank you for watching my videos. You mean to add lambda trigger when objects created in s3 bucket or just schedule the lambda.
@jaideep1222 18 күн бұрын
@@cloudquicklabs In this video at 33 minute for to fetch the incremental data the crawler has ran again. Do we really need to run the crawler if there is no schema change but only has the new data?
@cloudquicklabs 18 күн бұрын
Indeed I run it for second to fetch data from source but update only incremental data at destination side.
@awsconsole-se2vi 19 күн бұрын
How to upload 100+ object files in a bucket from python code?
@cloudquicklabs 18 күн бұрын
Thank you for watching my videos. When you have 100+ files in folder you can use S3 Sync method.
@awsconsole-se2vi 19 күн бұрын
If we have 100+ object files in a bucket, what is the logic to store in bucket from python program?
@cloudquicklabs 18 күн бұрын
Thank you for watching my videos. When you have 100+ files in folder you can use S3 Sync method.
@vijaykumarreddyt3287 19 күн бұрын
I am getting "Account *** is denied access" while creating crawlers. I tried signing in afresh. It's not helping. Also, while modifying inbound rules for security groups i selected allow all for the types.
@cloudquicklabs 18 күн бұрын
Thank you for watching my videos. May be AWS might have disabled crawler or ETL job in your account , just raise service quota limit or raise support case should help you.
@ABHISHEKKUMAR-sf1is 20 күн бұрын
wonderful explanation, crisp and clear!
@cloudquicklabs 19 күн бұрын
Thank you for watching my videos. Glad that it helped you.
@GaneshBabu-vr2lg 21 күн бұрын
InvalidInputException - VPC S3 endpoint validation failed for SubnetId: subnet-0b09b6cb378f40056. VPC: vpc-0e27491d28971d6f9. Reason: Could not find S3 endpoint or NAT gateway for subnetId: subnet-0b09b6cb378f40056 in Vpc vpc-0e27491d28971d6f9 (Service: AWSGlueJobExecutor; Status Code: 400; Error Code: InvalidInputException; Request ID: 5a5a41c8-f74e-48c9-9fc0-d5a8a0574d00; Proxy: null) I face the issues how can i slove it i try it but i can't slove the error. my data source is an s3 and i replicated to the redshift and this is a incremental data how. can i slove it
@phmeeeee 21 күн бұрын
currently I have EKS cluster provisioned from Terraform before, when I follow the helpfile.txt to describe the Service Account it seem token not be used, so I think the problem is here and that why the log in EKS not sending to my OpenSearch.
@aiwithvanshikaa 21 күн бұрын
Nice. Please improve audio quality. Informative video. Thanks
@cloudquicklabs 21 күн бұрын
Thank you for watching my videos. Glad that it helped you.
@ashwinreddy2866 22 күн бұрын
Thanks a lot for explaining it in very detailed way,
@cloudquicklabs 21 күн бұрын
Thank you for watching my videos. Glad that it helped you.
@snakhil90 23 күн бұрын
Incase of SCD, how we can define the SCD logic for merge and load? which option will have this option?
@cloudquicklabs 17 күн бұрын
Thank you for watching my videos. In terms of SCD , I believe the process would remain same untill the schema of table remains same.
@adityachaubey3965 27 күн бұрын
Amazing I have a question on Redshift... 1. How companies deal with redshift as a data warehouse because in order to perform ETL from S3 we need to keep the cluster up and running all the time for loading a new batch of data ...coming from S3 to the data catalog to Redshift tables so ... Do we need to keep running the redshift cluster in order to perform such ETL where data updates are done in the indefinite time period. Wouldn't this be a costly method to go for...and what companies do in such cases to avoid cost or they keep the redshift cluster up and running 24*7 ?? Can you or anyone help me give a company's perspective for this question? What companies do in such cases ?
@cloudquicklabs 27 күн бұрын
Thank you for watching my videos. I believe it depends on multiple factors here. 1. You need to go for Redshift Datawarehouse when this workloads has good value to your business. 2. There should multiple Consumer of this workload like application, analytics, Reporting etc. 3. Here there are chances of cost optimization is via removing the cluster in required time ( while snapshot of cluster is taken) In total there should be good tested startegy to use this service. I shall cover these points in my new video soon.
@girishgirijan9900 28 күн бұрын
I am really sorry to say. You are literally doing theory class. You are reading codes and explain like theory class in college class rooms.
@cloudquicklabs 27 күн бұрын
Thank you for watching my videos. Apologies that I it does not meets your expectations. I have plan to create very creative videos on coding soon.
@nawazuddin2178 28 күн бұрын
Nice tutorial. Do you also provide online training ? if so, please share the contact details
@cloudquicklabs 27 күн бұрын
Thank you for watching my videos. Currently I am not providing any online courses but when I start it I shall announce to the community.
@GermanSavloff 29 күн бұрын
Hi! Thank you for the video! I'd like to ask, what is the difference between this and the PartiQL option integrated in DynamoDB? Thanks
@cloudquicklabs 28 күн бұрын
Thank you for watching my videos. Query option integrated on DynamoDB provides your limited capabilities. At Athena you can run all possible on your data.