Inside TensorFlow: tf.data + tf.distribute

[@bapireddy5790]

Great talk on tf.data kudos to the engineers for making these high level api's

[@zytriesthings4540]

My mind just exploded

[@haneulkim4902]

This is an amazing talk! Thank you. I have two questions. 1. @9:22 I understand that it reads all elements tfrecords files sequentially. but since map function is applied before batch, how many elements are loaded? -> one by one? 2. @27:25 so after gradients from different GPU have been for example averaged it then gets passed back to each GPU to update the model? and this process is repeated?

[@prathameshdinkar2966]

Is synchronous training with 2 devices is similar to just doubling the batch size on single device?

[@shaunsun9076]

In fact, a little confused about the pipeline. Sometimes we consider it a kind of inputting data method, but sometimes it shows a way of training acceleration in distributed parallel strategies. How to think about it? Thx

[@RobertLugg]

I like to think about the purpose: Read data from somewhere. Perhaps perform some transformation. Make the data available to keras.fit(). The "transform" is going from the original format to the correct format that the model needs. One common example is to convert RGB images, where each pixel is represented by three numbers with a range of 0 to 255. The transform might be a division by 255 which will convert each pixel to be three numbers with a range of 0 and 1. This function is specified within the pipeline (the function .map) calls. Now, when reading and making the data available there is a lot of wasted CPU time. So, instead, while the system is waiting, it does the transform for another batch. It doesn't really accelerate training. However, it makes sure that the data in process isn't delaying the training process. I think the best way to think about it is to call it the data input pipeline. Also during inference, you run data through the same input data pipeline so that the same transforms are applied.

[@tanyachutani7272]

Informative talk

[@jijie133]

Great!