How to Validate Millions of Record in ETL testing?

How to Validate Millions of Record in ETL testing?
We have two different scenarios that I have explained where we have to validate the data -
1- When Source and Target both are in same server or database
2- When Source or Target may present in different Database or File or Azure NoSQL
#etltesting
#interviewquestions
Blog - www.testerhoon.com

[@dastageerpinjar5065]

Only thing i want to tell u ..i will wait for your videos.....and its worth of my time....thanks a lot for ur big help.... These are very imp questions.... Please dont stop.....people like me have life in this....

[@testerhoon28]

thank u so much for ur motivational words... it really help me to make new vdo... 🙏🙏😇

[@Mehala_Vision]

​@@testerhoon28 mam please may I add subtitles in English

[@chethanmotebennur7538]

How to Validate Millions of Record in ETL testing this is the only vido over YT which contains relevant explanation with respect to caption.

[@user-rx3vl2en5i]

Grate one you are explained amazing

[@BillionDollerDream]

Wow... Was waiting for this

[@deveshpadgelwar8895]

Thank you Mam... Really appreciate your efforts to share some real time scenario

[@testerhoon28]

okay 🙏

[@deveshpadgelwar8895]

How to coordinate with dev Team in case there is not mapping document ?

[@kirankalliguddi3617]

Thanks a lot for KT!!

[@testerhoon28]

thank u 🙏

[@ramaantony3849]

Hi Sissy.. Thank you very much.. For this video..As a new joiner for ETL testing.. Very useful for me...

[@testerhoon28]

thank you 🙏😊

[@RaviKumar-ii6wo]

Thank u so much for good explanation

[@testerhoon28]

🙏

[@lokeshankamreddi1626]

Thank you sister 🙏

[@tusharsonwane8468]

Thank you so much... explain very nicely thank you mam

[@testerhoon28]

🙏🙏✌😇

[@mytoprecipes6759]

Super mam.. I am wrkng as a qa engineer.. Its really helpful for me.. Plz do more videos..

[@testerhoon28]

thank you 🙏🙂

[@Falcon-dt6li]

Best video for etl testing in KZbin

[@testerhoon28]

thank you so much 🙂🙏

[@Falcon-dt6li]

@@testerhoon28 not able to open your web page

[@testerhoon28]

@@Falcon-dt6li try once - www.testerhoon.com

[@pushkardafre319]

Same question asked in my recent interview but can't ,so can you elaborate 2 nd scenario in more detail with python

[@shreesha1111]

Thank you so much for vdo Please make complete sql from beginning to advance level course

[@testerhoon28]

okay 👍

[@gds3535]

After getting defect we follow following procedure: 1.check is reproducible 2.email to developer 3.then respond defect resolution type 4.then we raise defect into hpal Is this right process ?

[@user-di1mc6us3l]

Hi Mam, it was really helpful, can you please make a video on ETL Testing using Automation using python or Java.

[@gds3535]

How to get srs document or build from server?

[@chethanmotebennur7538]

lot of helpful video request to pls ans for following Request to Please provide answer for below scenarios frequently getting these queries @interview *) how to validate flat file in landing area. 1.How to validate data and quality between landing and stage area if source is flat file and stag is db with out converting flat file to any other format. 2.If json file has millions of record and this is source file and target is oracle table then what is your approach to validate and how to validate 3.If oracle database is source and Teradata is Target and millions of record then how to compare without using any automation tool and not with sample data, need to validate full load data

[@testerhoon28]

Yes, in this vdo i have explained that to validate the cross db data we have to use automation... bcoz it wont be possible to run sql query to validate 2 different server data..

[@umakasibatla6389]

Thanks a lot excellent explanation can you go over python basic for testers.

[@testerhoon28]

okay 👍

[@shekharpawar2536]

I’m getting frustrated learning SQL. Can you tell me how to master SQL and which topics of SQL needed for a etl tester.

[@testerhoon28]

No need to get frustrated .. just focus on joins first, later learn subquery and some date, sub str functions... plssee my sql interview vdo and learn only those topics.. 👍

[@kanchankumar-pt2lp]

can you pls help how can we compare production data with test data,if not using any automated tool

[@testerhoon28]

Okay 👍

[@My-Research]

Nice video thanks sis.... how to explain to interviewer this -- what is your testing approach if source have 1000 recods and out of this 900 loaded ??

[@testerhoon28]

Approach - First validate all the 900 records are same in source or target. Second, find the missing records and reason for not loaded in target...

[@motherschannelforvbac5673]

Asked for me same question

[@shraddhadhakad1154]

Hi, Can you tell me how to perform ETL testing on unstructured & semi-structured data

[@testerhoon28]

okay👍

[@SANJAYYADAV-hm2bs]

Ma'am please make videos on ETL testing using cloud - Azure, AWS, GCP.

[@testerhoon28]

Sure will make 👍

[@a1geet]

Thank you! Please share the sheet link python/c# sample code for cross db validation

[@testerhoon28]

okay will soon post the selenium code that used to validate the report... 🙏

[@rohith3294]

In the second scenario can we use Informatica tool and validate the data?

[@testerhoon28]

yes u can...

[@neerajmittal6060]

Thanks a lot mam, can you please end to end automation video?

[@testerhoon28]

okay 🙏🙂

[@himanshumaggu6562]

Can you make video on E2E flat file to database validation.Ifs very helpful for us.

[@testerhoon28]

i already made that vdo... pls chk my old vdos 🙏

[@rameshairaddi1647]

please make brief video on different db validation

[@testerhoon28]

pls check out all my vdos.. 🙏

[@gds3535]

Does except query is possible between SQL server and bigdata?

[@testerhoon28]

no

[@gds3535]

Can only count, duplicate and null validation done between SQL server and bigdata Hive?

[@harshitagrwal9975]

can u provide the script mam?? please

[@gds3535]

Can we handle million of record if database is same ? Sql server to sql server Using SQL server

[@testerhoon28]

yes

[@azharmirza2013]

mam can you please take practicle example of cross DB valiadtion

[@testerhoon28]

ok will make 👍

[@SivaKumar-vm6iq]

Thank you for your effort. Can you please explain in English or you can add captions under the videos.

[@testerhoon28]

okay

[@akshayghadge299]

for millions of records for cross db in excel what query we have to write for source and target ...it will take except ?

[@testerhoon28]

No, we could not use sql query in cross db... to overcome this issue, we have to use automation

[@akshayghadge299]

@@testerhoon28 I am requesting you please make video on automation with end to end etl testing it will really help

[@vinaykale5053]

How we can validate data from different Databases manually? (without python)

[@testerhoon28]

Manually we have to do the sampling of data and validate.... it wont be possible to test millions of data manually... 🙏

[@chethanmotebennur7538]

and which is the commonly used automation tool for etl

[@testerhoon28]

You could use any automation tool, in my organization we have build c# script and python script to validate the data..

[@irfanmulla3801]

how to check this in mongo db

[@uttambongarde6558]

Suppose if you have 20GB text file data for the validation. How you are validated this data without using automation tool.

[@testerhoon28]

To test we have to do the sampling of data and test it

[@uttambongarde6558]

Thank you @@testerhoon28

[@prajakta887]

please mam .. make video on how to do automation using python script like how to check null, count .it will be helpfull for us

[@testerhoon28]

okay 👍

[@rohitchavan3246]

Can you explain about flat file to database .

[@testerhoon28]

I already cover that in my below vdo link - kzbin.info/www/bejne/gZnHpJJtf6-KkK8

[@gds3535]

Can automation possible between Oracle and big data?

[@testerhoon28]

yes

[@gds3535]

Thank you

[@ud4747]

Mam please make video on end to end project on real time project

[@testerhoon28]

pls check out all my vdos.. 🙏

[@Dope007]

mam how to do Source target mapping in excel, can u help?

[@testerhoon28]

pls see me data validation vdo... their i have explained it... 🙏🙏

[@sakshinaik05]

Plz explain 1 end to end project on etl testing

[@testerhoon28]

I had already explained pls go thru my end to end vdo... 🙏

[@Freakouts_and_found_Insane]

Can you please share your code How that works. That really helpful please madam share how that code is prepared

[@testerhoon28]

i have already shown that in my 2nd recent vdo.. pls chkout 🙏

[@aniketkadam4167]

Hi mam,tableau tool pe ek vedio banado.

[@testerhoon28]

okay 👍

[@thelayer5211]

This is not enough for us we need more details video on cross db testing as well as we need test closure report

[@testerhoon28]

okay 👍

[@manasenamanakarthavyam4150]

Hi..thanks for sharing practical things like this can you share one more video to count check and minus check Null checks If any error record comes then how to debug..

[@testerhoon28]

thank u for liking my content... i already made a vdo on that... pls chk my all end to end vdos..,🙏

[@manasenamanakarthavyam4150]

@@testerhoon28 Can you please explain one or 2 business logics like direct mapping and derived mapping logic ../ How firm a script by those logic Show practically plz....

[@dpinched]

English or not English? I can't decide😮

[@prabhatgupta6415]

Mam we really need if u can provide us the script

[@testerhoon28]

even if I provide you the script... you would not be able to execute... it needs some environment to run 🙏

[@prabhatgupta6415]

Mam i dont waant to run ur script. I want to know the logic , will create my script. I know we need environment to run all this. @@testerhoon28

[@pravinkashid3647]

Mam pls csharp ka code explain karo na hoga to..

[@testerhoon28]

wo code nahi show kar sakti 😰

[@pravinkashid3647]

I have written code for automating count ,null,duplicate validation in python using pandas ...but the difference is that both the table present in same DB..

[@durbandiary508]

can you please explain in english ? that will be really helpful please

[@testerhoon28]

okay

[@KiranTrends]

Kaha hai Automation madam..

[@testerhoon28]

wo is vdo me nahi hai... that vdo is yet to make.. 🙏

[@prabhatgupta6415]

Mam can u give us to script which u prepared to validate the data.. so that we can replicate the thinngs@@testerhoon28

[@riyavenkat2637]

Pls mak it in English r else try to give English caption

[@testerhoon28]

Sure 👍

[@user-th4tt1oq2e]

Hi Mam, Please explain in English that would be great help😊

[@testerhoon28]

yes i have started making vdos in english only... thatwas an old vdo 🙂

[@pravinkashid3647]

Mam pls automation code show karo na

[@testerhoon28]

code mai show ni kar sakti... 🤐

[@Dope007]

mam, can u connect and help me if u have time i can pay u 😇. whenever u r free...its just excel sheet data source target mapping I, pls give an indian time if u r free

[@Dope007]

it will not take more than an hour i guess pls

[@testerhoon28]

Hello, i have already made vdo on source to target data validation in excel... pls see that vdo.. 🙏

[@Dope007]

@@testerhoon28 mam , Mere palle nahi pad raha ... 😅 Pls

[@Dope007]

@@testerhoon28 plzzzzzzzzzzzzzzzzzzzzzzz

[@testerhoon28]

@@Dope007 sorry am not providing 1-1 support..

[@thirumaleshkumarreddykalak7188]

Please do videos only in English

[@testerhoon28]

okay.. will try to add subtitle..

[@thirumaleshkumarreddykalak7188]

@@testerhoon28 no subtitles, please do videos in English only so that people can understand easily and improve their communication as well

[@vishalkoundal5008]

Hello mam I mail you few things to understand but didnt get any reply could you please let me know how can I contact you?

[@testerhoon28]

pls email me... iwill respond ..

[@vishalkoundal5008]

​@@testerhoon28.. Hi I mail kindly reply Ns thxs for the response

[@vishalkoundal5008]

Hello I have few question related to etl I mailed you plz reply on that