More details on Apache Beam windowing in Google Cloud Dataflow: kzbin.info/aero/PLIivdWyY5sqIEiHGunZXg_yoS7unlHNJt

Just an FYI you actually cant use apt install, because the Beam base image as one of its last steps wipes out all sources lists from apt.

This talk was very helpful. The use of examples made the concepts like per-entity training easy to understand and how Beam is a natural fit for this type of work.

IMO a very nice contribution. Well explained.

Thanks for introducing this. Well explained with code

You have to pay for the badge

I am pretty clear about how an object is shared across DoFns and threads in a single process, my question is that if I cache a set object then will it be shared across VCPUs as well? Full disclaimer: I am working on a beam code which works on dynamic schema, i.e, there is a possibility of a new column addition.

is there any git repo for the code shown in this demo ?

The deduplicate can cause data loss unfortunately. I don't know exactly why

💪 eбаш еще!!!

Orderedliststate is not supported in Runner v2, correct?

Great talk! I'm sorry I didn't get to see it live.

great..

❤

Nice talk, keep it up 👍

Amazing talk. It helped me to understand general process of apache beam. Thanks to both of you.

Excellent talk 🎉

Wow , truly insightful. Proud of you Shafiqa

Wow so cooooool

Wonderful session!

very well explained

Excellent explanation Zeeshan.

Superficial, platitudes, waste of time...

Hi Ragy, While running the dataflow job via flex template from Cloud Build, I am getting the following error "Sandbox, launcher-, stopped.". The pipeline graph is created but the dataflow doesn't read from the source. Kindly help. Regards, Arijit Bose

where is the source code ................

thanks for sharing

would be good to show more code

what about avro to row where the avro has nested object types?

Is there some source code available?

Can I get the notebook link in description

Hello first of all I want to congratulate you for your work, and tell you that I have been developing python pipelines with apache beam for almost two years and I am in the process of migrating to scala, I hope to adopt it completely soon.

Quality of audio is not great but better from 5:00

Guess I need to schema my stuff, good job guys!

So cool. Understood zilch. Okay, not completely zilch. It was in English, after all.

At 16:19 you're mentioning that you're using a single SDK worker and a single thread to avoid the complication of dealing with multiple threads trying to access the GPU. We just came across that pain point. The downside of the solution proposed here is that you can't do parallel file I/O. Is there a way to control the number of worker threads on a per-pipeline-step basis for a single DoFn so that you can do still do parallel I/O for file reading and batch queuing?

Que crack el de la derecha

The mentioned track by Joe Smooth - Promised Land: kzbin.info/www/bejne/j6uUqYCfrteZqdk

Thanks😃

I think the topic is so interesting, but low quality audio

Good job @Mazloum ! Proudly

can you share the code on github which is shown in video

Horrible presentation. Did the presenter even prepare before recording the video?

MUCHAS GRACIAS POR LA INFORMACION

I need Hop to prepare pipeline, Beam to build pipeline in Flink format and Flink to run it, right? Is there any tutorial how to do some simple HOP pipeline executed on Flink? If I'm processing large Excel files (merge data, sort, search...) - Flink will speed up my job?

That is great however is there any more basic explanation how to integrate Hop with Beam? Some step by step tuto?

Extremely informative. Well presented by Zeeshan. Where Can we get the ppt shared in the video?

Hi Thanks for the tutorial, While running SqlTransform I am getting an error No such file or directory 'java':'java'. can someone please help me resolve this . I am running the py script inside a docker container

How can we store checkpoints on already processed events (like offset.storage) - so that our Beam app doesn't process all records in all MySQL binlog files when the Beam app/process is restarted?

UPDATE (January 2022): If you are running on Cloud Dataflow, it has now builtin support for using the Google Cloud Profiler with Python pipelines. I strongly recommend trying out that if you are using Dataflow, rather than following the instructions given here. Find more details at cloud.google.com/dataflow/docs/guides/profiling-a-pipeline#python

Great session