Practical use case for fork and pipe in C

[@ziga1122]

Perhaps it would be useful to mention that the pipe has a built in "wait function" if we want to write to the pipe and the pipe isn't empty the OS waits for the pipe to be read out. It goes the same way for the read end, if we want to read from the pipe and the pipe doesn't have any data in it, it waits for the data to get there and then reads it. Great series btw, the forking process was fantastically explained

[@CodeVault]

I have clarified this misunderstanding many times in the comments below (either here or on other pipe-related videos). In retrospect I should have mentioned that it automatically waits, it's something I didn't think would cause so much confusion. Thanks for the feedback!

[@just_patricia280]

That is exactly what I wanted to ask thanks!!

[@kingkai5115]

I don't know if you see these comments this late on, but I just wanted to say thank you thank you thank you. You're explanations, demonstrations are so clear and easy to understand. I struggle when watching my university lecturer explain the same concepts, but when watching you, I actually find it interesting and enjoy it. Thank you so much!

[@CodeVault]

Glad I could be of help! I try to read all the comments

[@kingkai5115]

@@CodeVault Thank you sir!

[@robertgabrielzaharie5405]

I paused the video and did the whole thing then compared it to you... it feels good to get things right - you re a great teacher my boy, thank you for these videos - I'd say you saved a guy from failing his course ( probably not, I should ve started way sooner ) but that s not true, what is true though is that you ve finally made someone too lazy to start working actually start doing something.. and I think imma finally get my life back on track - I wrote this here more like a self reminder - keep it going Rob, you ll eventually be happy.

[@spreadingpeace2425]

MashAllah brother very good explained video and your typing speed is fabulous

[@chaitanyagadgil3614]

Hi. This playlist helped me in understanding unix processes clearly. Your are doing a great effort in explaining in a way that makes understanding easy with a practical approach.

[@batuhan4347]

THANK YOU SO MUCH HOMIE! YOU HELP ME READ THE SLIDES OF MY PROFESSOR MUCH QUICKER!!!

[@sipo9333]

Thank you so much for your content, every video of yours is straight forward and easy to understand even for a beginner like myself. It is a blessing to have you making these videos.

[@hannav7125]

I was able to understand fork and pipe after watching this example. Great video. Keep it up!

[@pavelj5808]

Bro these videos are great you are helping me understand my material in my OS class much better! Thank you!

[@samarthtandale9121]

I wrote this program for dividing the task of sum into 4 arrays (i.e. 2 fork() calls) and it works --> #include #include #include #include #include int main() { int fd1[2], fd2[2]; pipe(fd1); pipe(fd2); int arr[]={1,2,3,4,5,6,7,8,9,10}; int sizeOfArr=sizeof(arr)/sizeof(int); int id1, id2; id1=fork(); id2=fork(); if(id1==-1) { printf("fork 1 caused error! "); return 1; } if(id2==-1) { printf("fork 2 caused error! "); return 2; } int start, end; int sum=0; if(id1!=0 && id2!=0) //parent process { start=0; end=start+sizeOfArr/4; } else if(id1==0 && id2!=0) //process x { start=sizeOfArr/4; end=start+sizeOfArr/4; } else if(id1!=0 && id2==0) //process y { start=(sizeOfArr/4)*2; end=start+sizeOfArr/4; } else //process z { start=(sizeOfArr/4)*3; end=sizeOfArr; } // we set the start and end pointers for all processes for(int i=start; i

[@CodeVault]

Great job! Yes, for sure you can divide into however many processes you want (not necessarily powers of 2). You might just have processes that get a different amount of numbers to process

[@aishwaryam3497]

@@CodeVault Hi.. Firstly thanks a lot for your videos. In every video you have explained the concepts clearly in a way which we could all understand and apply. In this particular video, you provided an assignment to use multiple processes to find the sum of the array with the use of pipes. Could you please provide a detailed solution for the same as I find it a little tricky. Thanks for your time and effort.

[@散华-l9m]

omg you are really teaching the ppl abt the knowledge...

[@nicolasarcelin3220]

Thank you for this video. Straight forward and easy to understand.

[@antoinecombremont6929]

Awesome, really helped understanding basics of inter process communications in C !

[@vikramsinghyadav5972]

Love from india❤...your videos are really helpful, thank you so much !

[@roguz9426]

Thanks, this video really helps me a lot to understand how pipe works.

[@azv10]

Hi, Great video. I have a question: shouldn't the 'wait' func be at the beginning of the last 'else' block? what if the parent process is scheduled before the child process put the sum in the pipe and the pipe is empty?

[@CodeVault]

The read() function will wait until there is something in that pipe to be read. So the call won't return until the other process calls write()

[@mrcorbak6793]

had the same question in mind the whole episode :p

[@uttkarshs]

@@CodeVault So, it is not necessary to have a wait for this specific logic, right?

[@martinhuang3469]

@@uttkarshs I think so.

[@jeffsherin9564]

Yeah I don’t think the wait is needed in this case, because the read function just waits anyway

[@amillion7582]

Perfect. I just recommend at the end of your videos you do a quick recap of what you did so that we all get to see the big picture. By recap i really mean like in a few seconds explain the whole function. Otherwise all is perfect. Keep up with the good work.

[@CodeVault]

Good idea! Usually I do a recap when it comes to more complex topics... If it's just a 5-10 mins lesson I don't think it's too important.

[@togosakutaro5882]

Beginner here: how does “forking” impact performance? In what way is the fork method more appropriate than a single for loop to sum all elements of an array? Pros and cons? All the best, Sergiu

[@CodeVault]

Usually, when improving performance, we use multiple threads instead of processes (see my other playlist regarding threads kzbin.info/www/bejne/mmrWkJdobd59p5I). But, in both cases, there is a little overhead for creating the thread (or process) and communicating with it. For small arrays like I show in the video, the time save is probably negligible. For arrays with millions of elements you are starting to get a benefit from using multiple lines of execution (be it with processes using fork() or threads using pthread). You can experiment this on your own. Try creating larger arrays. Compare a simple for loop with using the code in the video. Maybe even with different operations, not just the sum of all elements. Something like matrix multiplications would greatly benefit from multi-thread and multi-process implementations for example

[@eduardoparejalema]

Thank you so much for your help! You make all of these very easy to understand!

[@albanec4702]

Your videos are soooo neat and eazy - thank you much and wish all the best!👍

[@AlanKao-fc4km]

Thank you so much. Great explanation.

[@mariolim96]

best videos on linux programming

[@mariolim96]

i passed my exam watching ur video ,ty

[@robinranabhat3125]

Excellent explanation series you have put here. :) Just one question : Why don't we add `wait` call above the line `int sumFromChild;` Wouldn't it ensure that child process has written the partial sum. Before we initiate reading this sum from parent process. Okay. I experimented. It seems read operation will block until it get's some data. maybe that's why it won't matter.

[@CodeVault]

Exactly. Similarly, the write operation waits for space in the pipe's buffer

[@robinranabhat3125]

@@CodeVault thanks for taking your time :) appreciate it

[@tejasanand9311]

what if the buffer had some garbage and write thought the buffer is full, and the child hadn't ended yet, I think we should wait before reading for correctness @@CodeVault

[@johndoesson]

Thanks for providing such an example, I wondered if you could do this and out of luck stumbled upon this vid through recommendation.

[@tarangranpara5913]

You're a great teacher I must say! I just have one question though: In parent process, shouldn't we wait for child process to complete before we read sumFromChild var?

[@CodeVault]

No. We read whenever the child process wrote something to the pipe. We don't have to wait for it to finish its execution

[@dopetag]

@@CodeVault So the read() is already waiting for the input when it is executed before child's write(); This is a very confusing part of the parent-child processes relation.

[@halilibrahimustun50]

Hey, I want to thank you for the time you spent on these videos and, more importantly, the time you're spending answering questions. Having said that, I have a couple of simple questions: 1- When you write "sum" into fd[1] inside the child process, which "sum" will be written inside fd[1]? Can we know that? (without printing it for the test purpose) 2- After we read this "sum" into "sumFromChild" in the parent process and do "totalSum = sum + sumFromChild" how is this second "sum" is different from "sumFromChild"?

[@CodeVault]

1) Yes, the sum that is calculated is based on that "start" variable so you can check if start is 0 then it's writing the left hand sum, if it's arrSize / 2 then it's writing the right hand sum 2) Well, don't forget that each process has its own memory. So, while, the variable "sum" is technically the same in both the parent and child process they don't have the same address where they are stored in (therefore they also don't have the same value). Here's an example: int x; int pid = fork(); if (pid == 0) { x = 1; } else { x = 5; } In this case you'd have two x variables, one for each process. One that has the value 1 in the child process and another one that has the value 5 in the parent process. They are completely different. It's similar to having two different C programs with a variable with the same name. They don't share the same space in memory. Hope it was clear enough.

[@halilibrahimustun50]

@@CodeVault thanks :)

[@shanthgaitonde]

Thank you so much for creating a treasure trove of content on OS. I work at Big tech and these concepts are really helpful as an SWE working on Embedded Systems!

[@torresaguilarnancymayek5340]

Thank you so much all the way from Mexico!

[@vladimir_khlghatyan]

This was very useful. Thank you, man ;)

[@georgegeorge2751]

if we have more than one child-processes with the same parent-process .... with the wait(NULL) the parent-process will wait all children or just one of them ?

[@CodeVault]

Just one. You'll have to call wait multiple times. Also, it'll basically be random the child process wait will wait for so you'd have to use waitpid if you want to wait for them in a specific order

[@georgegeorge2751]

@@CodeVault thanks

[@maxim1748]

Very informative video, tank you very much!!

[@johanliebert8637]

First of all, I want to thank you for your kind efforts in making this amazing playlist. I have a little question. Why the line for "wait(NULL)" is the last line in the else statement (the parent's code), and not the first line? shouldn't we wait for the child process to calculate its sum first, and then read its sum and calculate the totalSum afterwards?

[@CodeVault]

No, the read() call automatically waits for something to exist in the pipe before returning. If we wait(NULL) right after creating the child process then the execution would be sequential (and it's not something that you always want)

[@mishaaskar]

@@CodeVault explaining it for every comment gotta be some fun xD

[@suncrafterspielt9479]

Great videos. But why is it beneficial to use multiple processes? And not simply use multiple threads? In fact in what use case are multiple processes more useful than multiple threads?

[@CodeVault]

Usually you want multiple threads. They are easier to control, they share memory, if the process it resides in terminates, all threads terminate. But in cases you want completely separate execution lines (that have NOTHING to do with each other), processes might be better (but they also have a bigger overhead). There's this video on the topic: code-vault.net/course/6q6s9eerd0:1609007479575/lesson/18ec1942c2da46840693efe9b51da486

[@suncrafterspielt9479]

Thanks

[@bharathhegde8526]

If the newly created process doesn't execute on a different core, this wouldn't actually improve performance right? Since if they execute on the same core they are not technically executing parallelly (only context switching). So does fork() create it on a diff core or it's left to the OS to handle it?

[@CodeVault]

The OS should handle this. But usually you expect that the OS scheduler schedules processes in such a way that would be most efficient and would, most likely, put that newly created process on a core that isn't doing much at the moment. So, while what you say is true, because we know that the OS scheduler does things right, we assume that parallelization (for the most part) improves performance

[@bethlehemteshome3660]

Thank you so much for this video

[@navguest1740]

Clean explanation... is there any video about the thread pool in this Vault? thanks...

[@CodeVault]

Not yet! But a series on Unix threads is in the works and we'll probably cover that too

[@navguest1740]

@@CodeVault Thank you...:)

[@ihoussem3367]

Best tutorial! Thanks

[@vivi-gl3vk]

dude ure a god sent

[@xinking2644]

thanks for your video! what should we do if we want to let children process do different jobs?

[@CodeVault]

There's a video on that topic: code-vault.net/course/46qpfr4tkz:1603732431896/lesson/as15alvu0a:1603732432433 Basically you can use the exec family of functions to run any executable you want from your own program

[@xinking2644]

@@CodeVault thank you for your reply!

[@xinking2644]

@@CodeVaultDo you have video which talk about how to debug for c++ program in vscode remotely for program written in high performance compute?

[@xinking2644]

@@CodeVault In addition, I have a question about threads, if i open a new thread using thread library ,and i call another multi - thread program within this thread, if my computer(Linux) could assign enough thread for this multi-thread program that called within new thread ? thank you for you time and answer!!

[@CodeVault]

Related to debugging, maybe this video helps: code-vault.net/lesson/tv93s6yw9r:1610568955955 I just VSCode (with gdb) to debug programs. Nothing special honestly Regarding the threads question, yea, of course you can do anything in any of the threads

[@abbaskareem5281]

Machine learning libraries using c or c++ for doing the matrix operations .. Are they *these libraries* using this method or multi-thread method? Thank you very much sir. I will completely this series and move on to multi-thread series 🖤

[@CodeVault]

Usually they just use a multi-threaded approach combined with some special batch operations that exist either on the GPU or the CPU

[@narendraparmar1631]

Thanks a lot bro😁

[@georgioziozas9031]

if we want multiple processes with multiple forks() to make them , then on the if statements we work with the 2 id's ? like we did in "calling fork multiple times" video ?

[@CodeVault]

Exactly

[@jy76312]

Hey shouldn't the wait(Null) be at the starting of the else ?

[@CodeVault]

Technically it would work either way. The only difference is that they would no longer run concurrently. 1) With wait(NULL) at the end of else: both the read from child process and write from parent process could be called concurrently. 2) With wait(NULL) at the beginning: the read from the parent will only run AFTER the child has finished its execution (after writing to the pipe). Do note here that pipes have an internal buffer so writing to them does not necessarily mean you have to have someone already waiting to read from them.

[@jy76312]

@@CodeVault But the read is in the parent. If parent runs before the child wouldn't it give an error since there is nothing written to the write end of the pipe ?

[@CodeVault]

No. The read call would just block the parent and wait until there's something to read. Similar to how scanf waits for the input from your keyboard.

[@duanedsilva]

@@CodeVault if that's the case then wait is technically not required. Am I right ?

[@CodeVault]

@@duanedsilva Not quite. While the program might work the same, it would cause certain resource leaks. Whenever you create the a process, that gets added to an internal table and this only gets removed when some other process calls wait on it.

[@sup3rs566]

Should i use more pipes if i need to communicate with more processes?

[@CodeVault]

Yes, usually you want a pipe for each direction between each 2 processes

[@stack.1]

Awesome!!!

[@jalalshah185]

The teacher i want!🤩

[@nom9d825]

thanks for videos! really helps

[@voidrust8336]

What if read happens before right ? Since both processes are in parallel ?

[@CodeVault]

read() always waits for something to be written in the pipe before returning

[@劉紀霆-j3d]

Great Video !!!

[@abhishekrai4677]

Good One....

[@leftenanalim]

6:55 Does it really matter to close the pipe that we don't use?

[@CodeVault]

Yes. Otherwise you might have read calls that are waiting for EOL but never receive it

[@fadieid5638]

What is the limit where creating more distributes computing process will stop making the computation faster? My PC has 2 CPU cores, does that mean that having more that 2 processes is not useful?

[@CodeVault]

Usually you don't want to create more threads/processes than you have CPU cores. If your CPU has multi-threading then that limit doubles although... if you have processes that use 100% of each core it won't make a difference.

[@fadieid5638]

@@CodeVault understood, thank you

[@misterammaiu9157]

goated

[@sreyamathew327]

Why do we sometimes add 1 to the value sizeof() sometimes?

[@CodeVault]

If we are working with strings we have to add 1 to make space for the NULL terminator

[@ShivamSharma-nb6gh]

Family of exec function is not working in visual studio on Windows connected to WSL UBUNTU Linux . Fork() is working fine but exec family is not working neither it is showing any error it's just ignored in program 😦 Please help is, there anything with directory @CodeVault

[@weydans]

Hello and thak you for the course! I'm asking me about somethings, becouse sometimes we have to run programs that is necessary say "yes" in the middle of execution. Lets pretend I want to automate a task like upgrade a linux system, the command "apt upgrade" has a flag "-y", but I want to create a program to automate it and I Ialso want to be able to say "yes" to my program during the excecution, how can I do this? Is it possible?

[@CodeVault]

Huh, I never tried making such a program. I think I would look into stdin manipulation and send that "yes" through it. The problem, I guess, is when you need to send it. I was thinking you could have the stdout routed through your program using a pipe and do some sort of string manipulation there. Maybe worth asking the question in discord.code-vault.net as this is a bit complex and there might be better ways of doing it

[@IsmailElbakkouchi]

thanksss !!

[@Murat-Kaynar]

thank youuuuuuuuu :)

[@selsabil2117]

I have un exercice that says two sons and one father the fisrt son read a string and send it to the second son this son extract vowels and send the vowels and the string to father and the father will poster them can i have a solution !??

[@CodeVault]

You just need to open 2 pipes, one from child to child and another from that child to parent. Each doing their own processing.

[@selsabil2117]

Yes i did this but ot doesn't work i dont know where is the probleme should i do two id1 and id2 So i did that If (id1==0)//child one ....pipe1 If (id2==0)// child two {Pipe1 from childe one And send it in pipe2} Else //parent { ....from pipe 1 And from pipe 2}

[@CodeVault]

I don't know what fd1 and fd2 are. Your process IDs?

[@selsabil2117]

Yes the process ids

[@selsabil2117]

Please can i have a solution cuz i have exam in this programing with tube of communication

[@tens0r884]

Are you sure man? Wouldnt this incur a huge overhead?

[@CodeVault]

Yeah, it will... probably because of using pipes. Sharing memory makes more sense in this case. I guess it's as practical as I could make it for a 12 minute tutorial for beginners. A better practical example was the video about simulation of the pipe operator

[@reydavid7300]

Is it possible to create a dynamic number of pipes?

[@CodeVault]

Of course. Just have an array of pipe file descriptors to save them in

[@reydavid7300]

@@CodeVault Nice, it worked with malloc() and pipe(). I'm just finishing the game of life with multiprocessing, each child process runs the game on a chunk of the wholw world!

[@CodeVault]

@@reydavid7300 Wow, that's impressive! You should share it on our discord server: discord.code-vault.net

[@olawaleolatunjibright6388]

I have a question, how is the variable sumFromChild the sum read from the Child? I have a little difficulty understanding that.

[@CodeVault]

The sum is basically calculated in each process (child and parent). Due to the if (id == 0) we make sure that we write the sum from within the child process (since id is only 0 in the child's process) to the parent process. This is why, whatever we read() into is the sum from the child's process

[@olawaleolatunjibright6388]

@@CodeVault well explained. Thanks man.

[@rajshah9129]

how the file descriptor work exactly not sure need to confirm if child write in so it offset will move, does the write pointer of parent will also move ahead n if yes why it is not case with read n no then it should overwrite/replace the data written by another process and for read both will have independent according to the data each process has read, read offset will be that much??

[@CodeVault]

The pipe fds are duplicated on fork() so they would have different cursor positions

[@rajshah9129]

​@@CodeVault so both read, write pointer will be different ?? so means both write to different location ??

[@demonboy7777]

I'm running the program in visual studio and I'm having issues with my ide being unable to find unistd.h.

[@CodeVault]

unistd is a Linux library. You can't find it on Windows

[@charlessnachez4248]

I don't understand what is the diffrence between putting wait() at the end or at the beginning of the parent block? is it the same thing?

[@CodeVault]

No, it's not. wait() basically causes the program to pause its execution until a child process finishes its execution if there are any child processes. So if you put it at the beginning of the block then, the subsequent lines of code are always executed after the child process has finished its execution

[@charlessnachez4248]

@@CodeVault So why you put it at the end? What is its purpose here in this code? thanks for your reply.

[@CodeVault]

@@charlessnachez4248 Ahh... that is for resource cleanup. Basically, in Unix, you want the child processes to finish execution before the parent processes. That is good practice. Without the wait() at the end, the parent process can exit before the child process. If the parent process would finish its execution first then, the child process will become a zombie process and will have to be cleaned up by the operating system (usually by allocating it to some predefined parent process and having it clean up after us). Hope that's more clear now

[@rickywong9525]

Just wondering why the wait(NULL) is placed at the end of the parent's procedure? If we want to wait for all the child to complete its execution why not just placed at the top of the parent procedure?

[@CodeVault]

Because then the processes would execute in series. You will lose the performance advantage that comes from parallel execution

[@rickywong9525]

@@CodeVault Thanks for the reply. May I know what are the performance advantage if we place it at the end of the parent's procedure? Is there anything I can read further on this? Thanks!

[@CodeVault]

In this example should be about a 100% increase in performance (in execution time). You're calculating both half-sums at the same time. If you wait(NULL) for the child process at the beginning of the parent process then you would calculate the sum for the second half only after the first half has been calculated It's not something generally applicable. In some cases you might want to wait for the child process to finish before continuing execution of the parent process. And, if you want to optimize performance even more multi-thread programming is a better fit for that

[@rickywong9525]

@@CodeVault Ahh i see! I think I get the point now. Make sense! Thank you sir!

[@harshsharma57]

saying they are executing in parallel is conceptually incorrect actually, instead they are executing concurrently on the same cpu, nice video though :)

[@CodeVault]

Yeah, you're right. Parallelism vs concurrency, I always mistakenly use these terms interchangeably

[@maxmustermann9654]

Do I really need 2 pipes for the program with 2 child processes? I wrote a program with 2 child processes and 1 pipe and it works but I don't know if this is the right approach: int fd1[2]; if (pipe(fd1) == -1) { return 1; } int arr[] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}; int sizeOfArr = sizeof(arr) / sizeof(int); int id1, id2; id1 = fork(); if (id1 == -1) { return 2; } int start, end; int sum = 0; if (id1 == 0) { start = 0; end = (sizeOfArr / 3); } else { id2 = fork(); if (id2 == -1) { return 3; } if (id2 == 0) { start = sizeOfArr / 3; end = 2 * (sizeOfArr / 3); } else { start = 2 * (sizeOfArr / 3); end = sizeOfArr; } } for (int i = start; i < end; i++) { sum += arr[i]; } printf("Partial sum is: %d ", sum); if (id1 == 0) { close(fd1[0]); write(fd1[1], &sum, sizeof(int)); close(fd1[1]); } else if (id2 == 0) { close(fd1[0]); write(fd1[1], &sum, sizeof(int)); close(fd1[1]); } else { int partialSum1, partialSum2; close(fd1[1]); read(fd1[0], &partialSum1, sizeof(int)); read(fd1[0], &partialSum2, sizeof(int)); close(fd1[0]); close(fd1[0]); int totalSum = sum + partialSum1 + partialSum2; printf("Totalsum is: %d ", totalSum); }

[@CodeVault]

Yes, for this example I think you can use just one pipe. Really, as long as you don't need to know which sum is from which process then you can just use one pipe

[@maxmustermann9654]

@@CodeVault Allright, thanks!

[@cesaltinofelix]

i was here

[@yassinemakhlouf9907]

doesn't the main process have to wait for the childprocess before reading from the pipe? i.e. this block becomes like this else { wait(NULL); int sumFromChild; close(fd[1]); read(fd[0], &sumFromChild, sizeof(sumFromChild)); close(fd[0]); int totalSum = sum + sumFromChild; printf("total sum is: %d ", totalSum); }

[@CodeVault]

That would make the whole program run sequentially. Remember that the read() call always waits for there to be data in the pipe it reads from

[@VultureGamerPL]

What if we want to divide the work between more than two processes? How can we extrapolate this method so it works with a process tree?

[@CodeVault]

Hmm, you just have to divide the array into even more parts from the parent process

[@kevinzebb]

Il y a 4 ans

[@aelthalyste6692]

That's the most bullshit argument I have ever heard in months. There is literally no practical reasons to use fork() to "optimize anything.". Baggage you are pulling by creating a new process, doing pipe communication is enormous compared to what you are trying to achive, threads would solve your problem by introducing a lot less than %10 percentage of weight fork() does. It's delusional to think any kind of optimization is done at that level. It's even crazier to taught people (let alone think!), that fork() is logical way to utilize cpu cores.