Normalize Scale Gradient Script Update 2.1.6 (PixInsight)

[@jeffdutton1910]

Every time you show a new technique I feel compelled to go back to revisit an old dataset...probably just as well in view of how few clear nights we get over my part of the globe. Actually I've learned a great deal from my subscription to your "fundamentals" series on your web site. Not ready to move on to "horizons" yet as I don't feel I've mastered fundamentals, but I keep surprising myself with the improvements I can make over previous attempts, even with less than ideal data.

[@AdamBlock]

Excellent... keep at it!

[@astrophotocologne]

Hello Adam, what got me the most ist the fact that you mentioned to do more videos on KZbin not only PixInsight related. I am looking forward very much to that. Wish you all the best. Frank

[@AdamBlock]

Thanks! Yes... this is upcoming....

[@davidleejenkins]

Your PixInsight tutorials are, by far, the best out there. No one else even comes close. Thank you for contributing so much to this amazing hobby!

[@AdamBlock]

Thank you!

[@bobc3144L]

As usual, very clear and informative presentation. Mostly through luck, I have been using the new version of NSG correctly but had no idea that you could view the amount of correction. Thank you.

[@AdamBlock]

Thanks for watching!

[@davidleejenkins]

Same here. Thank you so much, Adam, for taking the time to thoroughly explain how to use this new update. Much appreciated!

[@IcedReaver]

I was looking forward to seeing this video given the recent update to Pixinsight. Well explained Adam, thanks for sharing.

[@AdamBlock]

Thank you! I am planning on more...this is just the opening salvo. :)

[@mrrockenrock]

I really like the visual flow chart that tells us what you are going to do. Then you do it. For persons new to PI this is very helpful as it puts the whole calibration sequence into one chart. Of course the rest of the video is your excellent, and understandable explanations of what you are doing and why. All together my comment tells everyone they should subscribe to your video series!

[@AdamBlock]

Thanks Roger!

[@SunJao]

Wow, especially great video for those of us that have been using NSG for a while and perhaps completely wrong! @18:10 you demonstrate a new goal for how to choose the gradient smoothness factor. Based on previous demonstrations (e.g. PIX TV) and NSG documentation, I was selecting smoothness factor to "Apply sufficient smoothing to follow gradient trend without following the noise." I had it completely reversed (err backwards) based on that information. I interpreted gradient trend to be the macro sense of curve fitting and noise to be the micro bumps along the way. Hence, make a nice smooth curve without the bumps and my results were underwhelming. Your demo here implies the opposite. I will give NSG another go with this new perspective. Thanks!

[@SunJao]

Also, just found the new NSG updates in Adam Block Studios Fundamentals. Suppose I am about to learn even more!

[@AdamBlock]

The original concern was the a too high aggressiveness will cause artifacts. However, now that you can visualize the results in advance- you can see if there are any . It appears it is very difficult to force artifacts...so higher aggressiveness is OK.

[@Endymion4242]

Thanks for the great video. Look forward to a discussion of NSG versus local normalization approaches from you!

[@AdamBlock]

COming up.. in some form. Everything will be discussed in detail on my site..but I do expect to highlight some things here on YT.

[@AstroPills]

Hi Adam! Great video as always, thanks! I think the previous comment was removed by KZbin because it had an URL in it..however I wanted to tell you and other people using NSG, that I created and hosted on my server, in collaboration with John Murphy, a repository for it. So you can have it always up to date without doing it manually that can be tedious sometimes. 😊 The repo URL can be found on its official website, as I said I can't post it here.. Anyway NSG still is far superior than other normalization processes available in PixInsight.

[@AdamBlock]

Thank you for that. I will include the repo both here and on my site. Regarding strengths of normalization and weighting available in NSG and the official PixInsight processes- I will be explaining how each works and the differences between them. Then a user can decide which methods they like- rather than just making a coin-flip based on a method's name or tastemaker's "I like this one... " statement.

[@AstroPills]

@@AdamBlock thank you! I really appreciate it and I hope it will help NSG to spread even more among the PixInsight community. :)

[@AstroPills]

@@AdamBlock I do believe LocalNormalization has improved a lot in the latest release and in many cases the end result matches NSG in terms of SNR and gradient. However I've seen a couple of cases in which LN had some bad artifacts which NSG didn't have, I talk about dark halos and rings. It might be that there have been an improper use of the process, but I feel more confident with NSG nowadays. That said we're talking about edge cases from heavily light polluted skies, but still..

[@mrrockenrock]

Great video! I am revisiting it after 1 year and still learning more details. I do want to ask... Since you had a combination of exposure durations, did NSG punish the weighting of the shorter exposure images? Thanks!

[@AdamBlock]

A perfect 300sec exposure compared to a 600 second exposure will be given a weight of 0.5. If the measurement of the signal (and noise) warrants further reduction in weight it goes from there. Is this what you mean?

[@montekristum]

Thanx for the tutorial, Adam! I'm a big fan of assigning the weights with SubframeSelector tool, so I can play with different coefficients for eccentricity, FWHM, etc of the images. All this weighting strategy, proposed by the traditional LightVortex tutorial, is lost if we now use the NSG weighting. Do you think the NSG benefit is all that way better to use it over the other? Wouldn't it be useful to somehow combine both weighings? Say, an image with low SSWEIGHT could have good gradient, or an image with high NSGWEIGHT could have fat or oval stars - some combined criteria would rate it low or in the middle, while either of them would rate high an image with low quality on one of the criteria.

[@AdamBlock]

Pavel... this is part of a larger discussion. I can say that based on YOUR preference, you should like (and enjoy) PixInsight's PSF Signal Weight method that takes into account these metrics. NSG as well as PSF SNR and especially PSF Scale SNR follow the signal only.

[@jean-marclemoine9636]

Very great video! Thank’s a lot Adam. You will have a new subscriber on your website. JM

[@AdamBlock]

Thanks Jean-Marc!

[@trevorgreen2232]

Thank you Adam another excellent presentation. My understanding is if the best quality frame is used as the reference there would be no truncation of files as the best one is the comparison so all other frames would be less than 100% anyway. Is that correct ?

[@AdamBlock]

No... I do not think that is correct. If a target frame is half as good as the reference- by multiplying everything in that frame by 2- many near "1" values will become more than 1. Then when these are saved to disk... truncation occurs. Remember it is the normalized images that are saved to disk. So it is the worst frames, in terms of stars, that generally suffer the truncation issue. Am I wrong?

[@trevorgreen2232]

@@AdamBlockThanks for your reply . I will do some more testing and let you know my results

[@asd171029]

Thanks for video and a new technique .

[@AdamBlock]

Thank for watching!

[@stephen2615]

Thanks for this video. I am still to see any marked difference with NSG adjusted output considering the computing power and time it uses. The adjustments are always so small (generally less than 2% if I am reading it correctly) that I don't see much benefit with NSG.

[@AdamBlock]

I am not sure how you are judging the benefit. If you have data (images) that show little variability- the differences between normalization and weighting methods (of any sort) will also be small. So the statement isn't a good generalization for any normalization or weighting algorithm. It depends on the data you feed it. I don't know what the 2% value is you are referring to. In my video when I refer to a weight- it refers to the relative value between the reference and the frame of interest. So 0.02 means this frame is 50 times worse than the reference. That is pretty bad. I only keep frames that are better than 0.5 (one half) the value of the reference. Less than this is usually too poor to keep... more noise than useful signal.

[@volens31]

Thankyou Adam, very much 😀

[@AdamBlock]

Thank you for watching.

[@paths1111]

Thanks for the video, great job as always. I look forward to hearing your thoughts on the differences between NSG 2 and PI's revamped native LN process. This got me wondering something though - instead of normalizing to a reference frame still with a gradient, any reason not to clean up the gradient on the reference, say with DBE, then use that DBE'd frame as the reference for NSG (or LN)? Pros / cons?

[@ideaslinger]

I came here to ask this, too (the first part, given that PI is somewhat promoting their native LN process over NSG)

[@AdamBlock]

@@ideaslinger This isn't the way NSG works. On my site I explain in detail- but basically NSG is a strictly normalizing algorithm (which is robust and very difficult to somehow have artifacts). The reference frame is used for the measurements of the scaling factors and noise. If you give it another "clean" image as a reference- these measurements are all bogus and likely things will not go as expected. It is done differently in PixInsight- where the normalization and weighting are DECOUPLED from one-another. *This* requires some explanation of pros-cons. And you know where you will find that discussion. :)

[@ideaslinger]

@@AdamBlock .. thanks for taking the time to reply, Adam. I suppose I just meant that, from what I've seen, "they" believe you don't need NSG if you're using the built in processes. I'm just trying to wrap my head around that. I saw, above, that you're thinking of doing a pros-cons about them so I'll look forward to seeing that (I'm not sure where you're referring to about 'knowing where to find that discussion', I thought that original discussion had been locked off.)

[@rossc383]

I watched your videos about NSG and didn't get what is a difference with "usual" workflow on actual data? How big is it? How big is the difference on actual data with free and paid version?

[@AdamBlock]

I do not know what you mean by "usual" workflow. But the difference between the paid and unpaid version deals with truncation of value (since you have to write out the normalized files to disk). The paid version uses normalization files...so there is no truncation of values. This is can help with bright stars and bright features not getting clipped just due to math.

[@rossc383]

@@AdamBlock I'm interested in comparison with pixinsight's normalisation on actual data. I'm new to pxi, all this math is distant for me(sorry) so I understand things on practical side

[@hottokatrazi]

again a very informative video on this topic, thank you very much. I suppose it is possible to add future imaging sessions by choosing the same reference frame? maybe if you cover the new Local Normalization routine you can show how that is possible with LN?

[@AdamBlock]

If you add more data, I would just run NSG again and recompute normalizations and weights. If your original reference is still good, yes using it again would be fine.

[@hottokatrazi]

@@AdamBlock to clarify.: run NSG just with the new data and the old reference frame. Then integrate the New and old files?

[@AdamBlock]

@@hottokatrazi No... I would run NSG on the complete set to get the proper weighting. When you add new files.. .some might be better..some worse than the original (older) data...so a redistribution of weights makes sense.

[@paulwilson8367]

I realize that you made great suggestions in the development of NSG, and I do have the new, paid version. But some people of course point out that WBPP itself now has a new normalization check box and also a check box to create drizzle files. How different are the algorithms for these 2 programs that ostensibly do the same thing? Do you believe that NSG is still superior? I don't know definitively if my observing rig is "undersampled", but I was drizzling prior to NSG. Now when I attempt to return to drizzling, PixInsight crashes (not Windows, just PixInsight). This is happening not only to me. I checked my computer CPU and memory, both reported good. My computer has a SSD, i9 water cooled CPU, and 32GB RAM. Personally, I will need to re-watch this video where you ID the best frame (doesn't WBPP do this for us?) but also highlight the worst frame. This isn't something I was doing. Thanks for your updates, you are a very important resource for PixInsight.

[@AdamBlock]

By some reports- I also made good suggestions for processes in PixInsight and many Scripts (including WBPP where I am also credited). I am a disciple of no method or software and I make up my own mind. At the time NSG was released the official version of PixInsight did not capture the weighting of images in an optimal way. PixInsight has been updated and does things differently today- which is what I am explaining in videos on my site right now. The assignment of a best reference in WBPP is different than what is done in NSG. The measures are different and the algorithms are looking at the data differently. This isn't an apples to apples comparison.

[@davidemancini7853]

Hi Adam,if i use NSG after WBPP do i have to still click Local Normalization in WBPP? when i calibrate and register my files?

[@AdamBlock]

NSG will set up ImageIntegration for you when you use the template icon. There are two paths, with the free version you (NSG will load) the normalized image files. This means NO NORMALIZATION is configured. If you are using the paid-for module...then NSG will create XNML files... which means Local Normalization will be configured. I try to explain this in my video. :)

[@davidemancini7853]

@@AdamBlock Yes i got that, but usually i run WBPP to calibrate and register, there is one section of WBPP where say Local Normalization, inbelive when we loading the light, do i have to click on that if later i am using NSG?

[@AdamBlock]

@@davidemancini7853 Ah... you need to pick one method of Normalization. So either you use NSG or LN as implemented through the Local Normalization process. You will make a mess by doing both. I will be making videos on this... I just completed the weighting aspect... normalization is next.

[@davidemancini7853]

@@AdamBlock Thx a lot Adam, i am asking because yesterday i have processed 200 frames but i click LN in WBPP and then i put the registerd and calibrated frame in NSG, the free version and thr photo come out horrible 😂😂, i did whatch the last video you have make on your website, very interesting👍

[@AstroPhotoFacts]

Great video Adam, just try today NSG and work fantastic/amazing , my teste was in 30 frames of M101, about 15 was with clouds and was possible to recover/remove the clouds, in the pass was trash ..