how to establish connection to snowflake from pyspark
@natarajbeelagi56914 күн бұрын
In the same way, I'm trying to connect with local sql server but getting TCP/IP error. Can you please help?
@TheiTTechSolutions13 күн бұрын
You may have to check if TCP/IP is enabled in SQL Server Configuration. A screenshot of the error will help to understand the issue better
@TheiTTechSolutions21 күн бұрын
To know how to create a Free Tier Account please watch this video below: kzbin.info/www/bejne/qoeQpYumo7GZg68si=djx1ZM0ga7HQ2aDl
@TheiTTechSolutions21 күн бұрын
IMPORTANT NOTE: Also watch below video to complete two very important setup before starting the hands-on. This will protect you from unwanted costs by setting up Budget Alerts and also create an IAM user to protect your root account kzbin.info/www/bejne/iYTJiGVpeJahgLssi=nxRF15A8BDwC17fX
@TheiTTechSolutions26 күн бұрын
Let me know in comment box any specific AWS service you are willing to learn. I'll be happy to present it
@TheiTTechSolutions26 күн бұрын
Here is the first tutorial of this series kzbin.info/www/bejne/qoeQpYumo7GZg68si=pQFPaylXD8YGW740
@TheiTTechSolutionsАй бұрын
Watch this to learn about realtime data processing using Snowflake Stream and Task kzbin.info/www/bejne/eJnMoHh_hNaYacUsi=bCoJ8qlNjyvMOTEL
@TheiTTechSolutionsАй бұрын
Aslo watch this to know about Storage Integration, a better and recommended way to connect to AWS s3 as external stage kzbin.info/www/bejne/aZirmoaQatushJo
@HiransameiraAmarasingheАй бұрын
Nicely presented and easy to absorb. Simple and useful video. Thanks a lot
@TheiTTechSolutionsАй бұрын
You are most welcome
@AkkiSolutionsАй бұрын
But if you do this all your code will be in one container, now when ever any lambda function is triggered the image containing all the functionality will be loaded to memory unnecessarly because we need the functionality of only one lambda. imagine what the impact would be if i have more than 100 lambdas
@TheiTTechSolutionsАй бұрын
Yes I agree, this model should be applied only if we have simple and less number of lambdas, for large and complex multiple lambdas we should always consider creating heirarchy of folders with separate dockers for each lambda. Thanks for bringing it up, really appreciate 👍
@shantanudixit3062 ай бұрын
can you add state name (Lambda invoke) with the existing resultPath to next state i.e Lambda Invoke (1) ?
@TheiTTechSolutionsАй бұрын
Hey there, sorry for a little late response. Yes, you can do that by capturing result form $ using RestulPath in first state and InputPath in second state, something like below: { "StartAt": "LambdaInvoke", "States": { "LambdaInvoke": { "Type": "Task", "Resource": "arn:aws:lambda:region:account-id:function:LambdaInvoke", "ResultPath": "$.lambdaResult", "Next": "LambdaInvoke1" }, "LambdaInvoke1": { "Type": "Task", "Resource": "arn:aws:lambda:region:account-id:function:LambdaInvoke1", "InputPath": "$.lambdaResult", "End": true } } } Try this way, it should work
@harshadocp2 ай бұрын
Detailed and to the point.
@TheiTTechSolutions2 ай бұрын
Glad that it helped
@deepakkakkar76982 ай бұрын
thanks for the video, helpful. But in my case i put Retry [ErrorEquals] MaxAttempts = 3, but issue is that if some error occurs then step functions executing in loop. never ending still after 3 attempts. Please let me know how i should fix that, my all step function goes in infinity loop if some error comes.
@TheiTTechSolutions2 ай бұрын
I hope you are adding the catcher with correct error to catch and using a pass state as fallback. I will be able to check if I can see your step function code
@MuhammadNisar-c6m2 ай бұрын
Thank you so much sir.
@umeshranasinghe2 ай бұрын
Nice demo. Thank you! <3
@TheiTTechSolutions2 ай бұрын
Glad you liked it!
@__sjduebsj3 ай бұрын
can you do a similar video from Sagemaker Code Editor? I'm trying from there but Docker is not installed
@TheiTTechSolutions3 ай бұрын
Sure, I will give it a try, but any specific issue that you are facing?
@__sjduebsj3 ай бұрын
great video, thanks
@TheiTTechSolutions3 ай бұрын
Thanks you
@Mohamed-n5d8z3 ай бұрын
Hello, in my case I need to write the spark df into Postgres. I’m currently using kubernetes so what would you suggest me on where to add the jdbc jar file? Confused with that
@TheiTTechSolutions3 ай бұрын
What I can suggest is to create a Dockerfile that includes the PostgreSQL JDBC driver JAR in the Spark image. Something like this: FROM bitnami/spark:latest # Or any other Spark base image you are using ADD postgresql-<version>.jar /opt/spark/jars/ And then build the Docker image and push it to your Docker registry. Then, use the Custom Image in your kubernetes job. Another option would be to include the JDBC driver JAR within your Spark code spark = SparkSession.builder \ .appName("YourAppName") \ .config("spark.jars", "path/to/postgresql-<version>.jar") \ .getOrCreate() df = spark.read.format("jdbc").option("url", "jdbc:postgresql://your-db-url").option("dbtable", "your-table").option("user", "your-username").option("password", "your-password").load()
@Mohamed-n5d8z3 ай бұрын
@@TheiTTechSolutions I’ve used the first method and it worked. Thanks for your guidance 😊
@dharamrajvadiya78564 ай бұрын
i follow the all steps but i am getting the error ...Failure using stage area. Cause: [Access Denied (Status Code: 403; Error Code: AccessDenied)] like this ...
@TheiTTechSolutions3 ай бұрын
Apologies for responding late, 403 means your snowflake is not able to communicate with the aws. There is some issue in key. Try re-creating the stage again and give the check key in the IAM user that you would have created for Snowflake.
@afsaraf15334 ай бұрын
Thanks that was useful ❤
@techaisolution4 ай бұрын
Hi, this setup spike my billing very high, The setup was to build lambda function to read the latest file from the s3 dir and make transformation then finally to s3 target dir, So this all setup with the python script has to run once the s3 notification to lambda function that an file just came to s3. But it went into a loop and made the s3 and lambda billing spike Let me knew what is the issue in my setup that i didn't noticed at first while running this python script in lambda
@TheiTTechSolutions3 ай бұрын
Were you writing the files back to same directory? You can have event notification on a specific key and write the processed file to a different key on which there is no event notification otherwise it will trigger the infinite loop. Be careful!
@halfengineerr4 ай бұрын
Getting error in last step
@TheiTTechSolutions3 ай бұрын
Can you share the error please?
@balakrishna616 ай бұрын
Event Bridge task creation is missing.Didn't show from starting means from the console.Anyhow good video.
@balakrishna616 ай бұрын
Nice explanation. Thank you very much. Keep it up the good work.
@pradeepkumarht47826 ай бұрын
Thanks. If you want to execute Lambda 3 with the values from Lambda1 or Lambda2 (based on choiuce state), how can you read the values in lambda 3?
@TheiTTechSolutions6 ай бұрын
Hello Pradeep, that can be achieved using ResultPath and InputPath step function variables. Check this if this helps: kzbin.info/www/bejne/b6PdepuIl5ekgc0
@loulou47997 ай бұрын
You explained it well! But I have a question : how can I provide the path to the apikey if it is stored in a .txt file not .env?
@TheiTTechSolutions7 ай бұрын
Well, in that case you will have to read the file. Something as below: # Open and read the .txt file with open('env_variables.txt', 'r') as file: lines = file.readlines() # Parse each line and store the variables in a dictionary env_vars = {} for line in lines: key, value = line.strip().split('=') env_vars[key] = value # Now you can use the environment variables in your code api_key = env_vars.get('API_KEY') print("API Key:", api_key) But, keep in mind that storing sensitive information like passwords or API keys in plaintext files is not recommended for production use
@csec05657 ай бұрын
Is this free or paid. I'm asked to get subscription now. Can somebody help me.
@TheiTTechSolutions7 ай бұрын
Hi, No its not free. To activate or create account you don't need to pay anything as such, but you will be charged for each api call that your program will make. You pay for what you use. As per their website new users get free $5 worth of free tokens. after that it is charged @ $0.0010 / 1K tokens for inputs and $0.0020 / 1K tokens for outputs. Tokens can be thought of as pieces of words. Before the API processes the prompts, the input is broken down into tokens. 1 token ~= 4 chars in English 1 token ~= ¾ words 100 tokens ~= 75 words More about these can be found at: openai.com/pricing help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
@vasutke11878 ай бұрын
Sir, High clarity. Professional teaching. Useful videos.
@TheiTTechSolutions8 ай бұрын
Glad that it is helpful
@shravanuchil8 ай бұрын
One free comment 😊
@qiratwajahat8 ай бұрын
Thanks! for this informative guide!🙂
@TheiTTechSolutions8 ай бұрын
Glad it was helpful!
@davidkempton28949 ай бұрын
Thanks for this tutorial!
@wolfmanwatcher56999 ай бұрын
do we have to purchase or pay any thing to activate the api of gpt 3.5 turbo?
@TheiTTechSolutions9 ай бұрын
Hi, To activate or create account you don't need to pay anything as such, but you will be charged for each api call that your program will make. You pay for what you use. As per their website new users get free $5 worth of free tokens. after that it is charged @ $0.0010 / 1K tokens for inputs and $0.0020 / 1K tokens for outputs. Tokens can be thought of as pieces of words. Before the API processes the prompts, the input is broken down into tokens. 1 token ~= 4 chars in English 1 token ~= ¾ words 100 tokens ~= 75 words More about these can be found at: openai.com/pricing help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
@wolfmanwatcher56999 ай бұрын
@@TheiTTechSolutions is there any free api of chart gpt type AI? I am student , and I only need it,to build small personal project .
@TheiTTechSolutions9 ай бұрын
No, calling api is not free, you'll get $5 credits as a new user.
@wolfmanwatcher56999 ай бұрын
@@TheiTTechSolutions i am facing some problem in this, can u send your insta id where i can send u my problem photo.
@TheiTTechSolutions9 ай бұрын
Still facing issue? U can post me on insta @the_it_tech_sol, I will try to help
@zweis9 ай бұрын
Thanks for the updated guide! I've been looking for something to showcase the new method using OpenAI() function
@TheiTTechSolutions9 ай бұрын
I am glad that it helped!
@call-me-potato.9 ай бұрын
your screen is so small. cant watch it properly.
@TheiTTechSolutions9 ай бұрын
My apologies, try to see it in a laptop or PC, I hope that will be helpful. But I will try to zoom it a little next time so that its clearly visible.
@RichardChai Жыл бұрын
very clear instructions and to the point! Thank you for creating this tutorial!