PixInsight: FastIntegration

FastIntegration is a wonderful new process in PixInsight that dramatically decreases the processing time of large data sets (hundreds of exposures). It is a simpler registration and integration routine that is multi-threaded and very cool to use. This video summarizes some aspects of the process- the full explanation is now a part of my initial course on my site: FastTrackTraining (look at that..so appropriate!).
REVIEWS!
www.adamblocks...
More at:
AdamBlockStudio...
Those just starting out in PixInsight will enjoy FastTrack Training!
www.adamblocks...
(see the video @www.youtube.co...)
#PixInsight #AdamBlockstudios #Integration #FastIntegration

[@billblanshan3021]

Thanks for covering this Adam!!

[@somethingclever1234]

Just for comparison, in this example, Adam had 700 large subs and it took 24 hours to process. I had a large frame camera 750 sub run (WBPP) on an AMD Ryzen 9 5950X running at 4.3ghz, dual top of the line SSD's (not Raid) and 128GB memory, Win 10Pro. The run took a little less than 4 hours. We need to get Adam a faster computer.

[@Chiclets1]

Apples to cantaloupes comparison

[@Phillyo118]

This is awesome for large data sets with current CMOS sensors allowing people to use shorter sub exposures without then having to wait days to pre-process all the data. Adam, do you think this is a feature that might be added as an option within WBPP in the near future?

[@AdamBlock]

Seems to be a common question... so I guess. My opinion...it misses the point of WBPP and will bloat it. It is just two steps. Calibrate then FastIntegration.

[@trevorgreen2232]

Thank you Adam for another excellent presentation. Do you know if this fast integration is planned to be added to WBPP ? That would make an even faster process I would say.

[@AdamBlock]

Seems to be a common question... so I guess. My opinion...it misses the point of WBPP and will bloat it. It is just two steps. Calibrate then FastIntegration

[@Nico.75]

Wow, what a great option!! Is it an option to set a small tolerance and exclude the failed frames in order to get a integrated "best of" stack that can be used for taking its stars only later? And using a full stack with highest SNR for the rest?

[@AdamBlock]

@@Nico.75 Such creativity PixInsight permits.

[@Microtonal_Cats]

Is Drizzle check box removed in this? I can't tell if we should just check "Rejection maps / drizzle files" in Calibration? It's not clear in the documentation if that runs Drizzle, or just prepares for it

[@AdamBlock]

No drizzle ...

[@johnadastra1754]

Wondering if the subframes could be sorted by the pier side in the metadata, E or W, using a Keyword in WBPP. Then the sorted data could be rotated 180 using an Image Container so they have the same orientation prior to FastIntegration.

[@julianzoller8237]

So I did get it to run with "small" numbers of images (1-2k). There is the problem of star detection when using short exposures with large focal lengths, but tweeking the settings does work. However, I am not able to integrate large lucky-imaging sets with >20.000 images. So for the moment I have to stick to the classical stacking softwares for this...

[@julianzoller8237]

I have to correct myself. The processing of large numbers of images works. However, it takes a long time for the process to start.

[@f.k.6151]

Thanks for the video. I am curious as to why we don't let WBPP go through Image Registration process as well and why we should do the registration manually before using the FastIntegration? Also, what about Normalization that can be optionally chosen in WBPP, can it be chosen with FI? I don't recall seeing that feature.

[@AdamBlock]

Registration in WBPP (the full frame type with distortion correction) takes a lot longer! Fast Integration really speeds things up. You should not do any manual registration before FastIntegration. Normalization is not part of FastIntegration... because it takes too long! This is the point... you compromise to gain speed. BUT... with enough frames this can be an excellent compromise as I showed.

[@f.k.6151]

@@AdamBlockThanks a lot.

[@junktrunk909]

@adam I think this is why people are suggesting that WBPP should just offer this faster integration itself as part of a modified workflow. PI in general and WBPP are very powerful but not well documented so it's hard to even understand what each step does, why it matters, and when it should be used vs these faster options. Some UI simplification to help users understand what paths are best for them would go a long way. Having to do a couple steps in WBPP and a couple in FI is fine if you're really used to these tools but very hard for newer folks to remember what does what.

[@SpaceCityAstro]

Wouldn't it be nice if WBPP included some of these elements of FastIntegration to help speed things up?

[@AdamBlock]

I think this misses the point. WBPP is a pipeline designed to do everything. This is like a modification of the pipeline plus a very specific method of processing. So you could add the functionality to WBPP... but it would unnecessarily bloat it in my opinion. Calibrate the data (WBPP) then run FastIntegration for the rest quickly. Doesn't seem to bad.

[@BoeingBrian]

What is the recommended way to pre-flip the images?

[@AdamBlock]

I want to stress you do not need to do this. My data set was a worse case scenario. But, if it only takes a moment to pre-flip... it certainly couldn't hurt. I just used an ImageContainer with FastRotation (180 degrees). This took care of things. Then on to FastIntegration.

[@nicolassavard6860]

How do I add my flats/bias/and darks in fast integration?

[@AdamBlock]

You don't. FastIntegration is a post processing tool. You calibrate the data first before using it. FastIntegration is now available in WBPP... or you can use WBPP to calibrate first and then manually run FastIntegration.

[@MegaMichaeltodd]

There is still no substitution for two xeons 96 GB of RAM, an NVME drive and siril

[@andrevanzegveld]

Where can I find fastintegration ????????

[@AdamBlock]

It is a process in PixInsight. (In the latest version.)