AWS Tutorial - Series Introduction
2:06
Snowflake Stream and Task
28:25
What is Snowflake Snowpipe?
18:11
2 ай бұрын
Пікірлер
@JagadishAekula
@JagadishAekula 11 күн бұрын
how to establish connection to snowflake from pyspark
@natarajbeelagi569
@natarajbeelagi569 14 күн бұрын
In the same way, I'm trying to connect with local sql server but getting TCP/IP error. Can you please help?
@TheiTTechSolutions
@TheiTTechSolutions 13 күн бұрын
You may have to check if TCP/IP is enabled in SQL Server Configuration. A screenshot of the error will help to understand the issue better
@TheiTTechSolutions
@TheiTTechSolutions 21 күн бұрын
To know how to create a Free Tier Account please watch this video below: kzbin.info/www/bejne/qoeQpYumo7GZg68si=djx1ZM0ga7HQ2aDl
@TheiTTechSolutions
@TheiTTechSolutions 21 күн бұрын
IMPORTANT NOTE: Also watch below video to complete two very important setup before starting the hands-on. This will protect you from unwanted costs by setting up Budget Alerts and also create an IAM user to protect your root account kzbin.info/www/bejne/iYTJiGVpeJahgLssi=nxRF15A8BDwC17fX
@TheiTTechSolutions
@TheiTTechSolutions 26 күн бұрын
Let me know in comment box any specific AWS service you are willing to learn. I'll be happy to present it
@TheiTTechSolutions
@TheiTTechSolutions 26 күн бұрын
Here is the first tutorial of this series kzbin.info/www/bejne/qoeQpYumo7GZg68si=pQFPaylXD8YGW740
@TheiTTechSolutions
@TheiTTechSolutions Ай бұрын
Watch this to learn about realtime data processing using Snowflake Stream and Task kzbin.info/www/bejne/eJnMoHh_hNaYacUsi=bCoJ8qlNjyvMOTEL
@TheiTTechSolutions
@TheiTTechSolutions Ай бұрын
Aslo watch this to know about Storage Integration, a better and recommended way to connect to AWS s3 as external stage kzbin.info/www/bejne/aZirmoaQatushJo
@HiransameiraAmarasinghe
@HiransameiraAmarasinghe Ай бұрын
Nicely presented and easy to absorb. Simple and useful video. Thanks a lot
@TheiTTechSolutions
@TheiTTechSolutions Ай бұрын
You are most welcome
@AkkiSolutions
@AkkiSolutions Ай бұрын
But if you do this all your code will be in one container, now when ever any lambda function is triggered the image containing all the functionality will be loaded to memory unnecessarly because we need the functionality of only one lambda. imagine what the impact would be if i have more than 100 lambdas
@TheiTTechSolutions
@TheiTTechSolutions Ай бұрын
Yes I agree, this model should be applied only if we have simple and less number of lambdas, for large and complex multiple lambdas we should always consider creating heirarchy of folders with separate dockers for each lambda. Thanks for bringing it up, really appreciate 👍
@shantanudixit306
@shantanudixit306 2 ай бұрын
can you add state name (Lambda invoke) with the existing resultPath to next state i.e Lambda Invoke (1) ?
@TheiTTechSolutions
@TheiTTechSolutions Ай бұрын
Hey there, sorry for a little late response. Yes, you can do that by capturing result form $ using RestulPath in first state and InputPath in second state, something like below: { "StartAt": "LambdaInvoke", "States": { "LambdaInvoke": { "Type": "Task", "Resource": "arn:aws:lambda:region:account-id:function:LambdaInvoke", "ResultPath": "$.lambdaResult", "Next": "LambdaInvoke1" }, "LambdaInvoke1": { "Type": "Task", "Resource": "arn:aws:lambda:region:account-id:function:LambdaInvoke1", "InputPath": "$.lambdaResult", "End": true } } } Try this way, it should work
@harshadocp
@harshadocp 2 ай бұрын
Detailed and to the point.
@TheiTTechSolutions
@TheiTTechSolutions 2 ай бұрын
Glad that it helped
@deepakkakkar7698
@deepakkakkar7698 2 ай бұрын
thanks for the video, helpful. But in my case i put Retry [ErrorEquals] MaxAttempts = 3, but issue is that if some error occurs then step functions executing in loop. never ending still after 3 attempts. Please let me know how i should fix that, my all step function goes in infinity loop if some error comes.
@TheiTTechSolutions
@TheiTTechSolutions 2 ай бұрын
I hope you are adding the catcher with correct error to catch and using a pass state as fallback. I will be able to check if I can see your step function code
@MuhammadNisar-c6m
@MuhammadNisar-c6m 2 ай бұрын
Thank you so much sir.
@umeshranasinghe
@umeshranasinghe 2 ай бұрын
Nice demo. Thank you! <3
@TheiTTechSolutions
@TheiTTechSolutions 2 ай бұрын
Glad you liked it!
@__sjduebsj
@__sjduebsj 3 ай бұрын
can you do a similar video from Sagemaker Code Editor? I'm trying from there but Docker is not installed
@TheiTTechSolutions
@TheiTTechSolutions 3 ай бұрын
Sure, I will give it a try, but any specific issue that you are facing?
@__sjduebsj
@__sjduebsj 3 ай бұрын
great video, thanks
@TheiTTechSolutions
@TheiTTechSolutions 3 ай бұрын
Thanks you
@Mohamed-n5d8z
@Mohamed-n5d8z 3 ай бұрын
Hello, in my case I need to write the spark df into Postgres. I’m currently using kubernetes so what would you suggest me on where to add the jdbc jar file? Confused with that
@TheiTTechSolutions
@TheiTTechSolutions 3 ай бұрын
What I can suggest is to create a Dockerfile that includes the PostgreSQL JDBC driver JAR in the Spark image. Something like this: FROM bitnami/spark:latest # Or any other Spark base image you are using ADD postgresql-<version>.jar /opt/spark/jars/ And then build the Docker image and push it to your Docker registry. Then, use the Custom Image in your kubernetes job. Another option would be to include the JDBC driver JAR within your Spark code spark = SparkSession.builder \ .appName("YourAppName") \ .config("spark.jars", "path/to/postgresql-<version>.jar") \ .getOrCreate() df = spark.read.format("jdbc").option("url", "jdbc:postgresql://your-db-url").option("dbtable", "your-table").option("user", "your-username").option("password", "your-password").load()
@Mohamed-n5d8z
@Mohamed-n5d8z 3 ай бұрын
@@TheiTTechSolutions I’ve used the first method and it worked. Thanks for your guidance 😊
@dharamrajvadiya7856
@dharamrajvadiya7856 4 ай бұрын
i follow the all steps but i am getting the error ...Failure using stage area. Cause: [Access Denied (Status Code: 403; Error Code: AccessDenied)] like this ...
@TheiTTechSolutions
@TheiTTechSolutions 3 ай бұрын
Apologies for responding late, 403 means your snowflake is not able to communicate with the aws. There is some issue in key. Try re-creating the stage again and give the check key in the IAM user that you would have created for Snowflake.
@afsaraf1533
@afsaraf1533 4 ай бұрын
Thanks that was useful ❤
@techaisolution
@techaisolution 4 ай бұрын
Hi, this setup spike my billing very high, The setup was to build lambda function to read the latest file from the s3 dir and make transformation then finally to s3 target dir, So this all setup with the python script has to run once the s3 notification to lambda function that an file just came to s3. But it went into a loop and made the s3 and lambda billing spike Let me knew what is the issue in my setup that i didn't noticed at first while running this python script in lambda
@TheiTTechSolutions
@TheiTTechSolutions 3 ай бұрын
Were you writing the files back to same directory? You can have event notification on a specific key and write the processed file to a different key on which there is no event notification otherwise it will trigger the infinite loop. Be careful!
@halfengineerr
@halfengineerr 4 ай бұрын
Getting error in last step
@TheiTTechSolutions
@TheiTTechSolutions 3 ай бұрын
Can you share the error please?
@balakrishna61
@balakrishna61 6 ай бұрын
Event Bridge task creation is missing.Didn't show from starting means from the console.Anyhow good video.
@balakrishna61
@balakrishna61 6 ай бұрын
Nice explanation. Thank you very much. Keep it up the good work.
@pradeepkumarht4782
@pradeepkumarht4782 6 ай бұрын
Thanks. If you want to execute Lambda 3 with the values from Lambda1 or Lambda2 (based on choiuce state), how can you read the values in lambda 3?
@TheiTTechSolutions
@TheiTTechSolutions 6 ай бұрын
Hello Pradeep, that can be achieved using ResultPath and InputPath step function variables. Check this if this helps: kzbin.info/www/bejne/b6PdepuIl5ekgc0
@loulou4799
@loulou4799 7 ай бұрын
You explained it well! But I have a question : how can I provide the path to the apikey if it is stored in a .txt file not .env?
@TheiTTechSolutions
@TheiTTechSolutions 7 ай бұрын
Well, in that case you will have to read the file. Something as below: # Open and read the .txt file with open('env_variables.txt', 'r') as file: lines = file.readlines() # Parse each line and store the variables in a dictionary env_vars = {} for line in lines: key, value = line.strip().split('=') env_vars[key] = value # Now you can use the environment variables in your code api_key = env_vars.get('API_KEY') print("API Key:", api_key) But, keep in mind that storing sensitive information like passwords or API keys in plaintext files is not recommended for production use
@csec0565
@csec0565 7 ай бұрын
Is this free or paid. I'm asked to get subscription now. Can somebody help me.
@TheiTTechSolutions
@TheiTTechSolutions 7 ай бұрын
Hi, No its not free. To activate or create account you don't need to pay anything as such, but you will be charged for each api call that your program will make. You pay for what you use. As per their website new users get free $5 worth of free tokens. after that it is charged @ $0.0010 / 1K tokens for inputs and $0.0020 / 1K tokens for outputs. Tokens can be thought of as pieces of words. Before the API processes the prompts, the input is broken down into tokens. 1 token ~= 4 chars in English 1 token ~= ¾ words 100 tokens ~= 75 words More about these can be found at: openai.com/pricing help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
@vasutke1187
@vasutke1187 8 ай бұрын
Sir, High clarity. Professional teaching. Useful videos.
@TheiTTechSolutions
@TheiTTechSolutions 8 ай бұрын
Glad that it is helpful
@shravanuchil
@shravanuchil 8 ай бұрын
One free comment 😊
@qiratwajahat
@qiratwajahat 8 ай бұрын
Thanks! for this informative guide!🙂
@TheiTTechSolutions
@TheiTTechSolutions 8 ай бұрын
Glad it was helpful!
@davidkempton2894
@davidkempton2894 9 ай бұрын
Thanks for this tutorial!
@wolfmanwatcher5699
@wolfmanwatcher5699 9 ай бұрын
do we have to purchase or pay any thing to activate the api of gpt 3.5 turbo?
@TheiTTechSolutions
@TheiTTechSolutions 9 ай бұрын
Hi, To activate or create account you don't need to pay anything as such, but you will be charged for each api call that your program will make. You pay for what you use. As per their website new users get free $5 worth of free tokens. after that it is charged @ $0.0010 / 1K tokens for inputs and $0.0020 / 1K tokens for outputs. Tokens can be thought of as pieces of words. Before the API processes the prompts, the input is broken down into tokens. 1 token ~= 4 chars in English 1 token ~= ¾ words 100 tokens ~= 75 words More about these can be found at: openai.com/pricing help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
@wolfmanwatcher5699
@wolfmanwatcher5699 9 ай бұрын
@@TheiTTechSolutions is there any free api of chart gpt type AI? I am student , and I only need it,to build small personal project .
@TheiTTechSolutions
@TheiTTechSolutions 9 ай бұрын
No, calling api is not free, you'll get $5 credits as a new user.
@wolfmanwatcher5699
@wolfmanwatcher5699 9 ай бұрын
@@TheiTTechSolutions i am facing some problem in this, can u send your insta id where i can send u my problem photo.
@TheiTTechSolutions
@TheiTTechSolutions 9 ай бұрын
Still facing issue? U can post me on insta @the_it_tech_sol, I will try to help
@zweis
@zweis 9 ай бұрын
Thanks for the updated guide! I've been looking for something to showcase the new method using OpenAI() function
@TheiTTechSolutions
@TheiTTechSolutions 9 ай бұрын
I am glad that it helped!
@call-me-potato.
@call-me-potato. 9 ай бұрын
your screen is so small. cant watch it properly.
@TheiTTechSolutions
@TheiTTechSolutions 9 ай бұрын
My apologies, try to see it in a laptop or PC, I hope that will be helpful. But I will try to zoom it a little next time so that its clearly visible.
@RichardChai
@RichardChai Жыл бұрын
very clear instructions and to the point! Thank you for creating this tutorial!
@TheiTTechSolutions
@TheiTTechSolutions Жыл бұрын
Thanks for watching!
@VaibhavPatel-f4r
@VaibhavPatel-f4r Жыл бұрын
good keep it up!
@TheiTTechSolutions
@TheiTTechSolutions Жыл бұрын
Thanks!
@SafeRoom-dsd
@SafeRoom-dsd Жыл бұрын
Thanks :)
@TheiTTechSolutions
@TheiTTechSolutions Жыл бұрын
Welcome!