How to pre-process your spectra for research (SNV, MSC, Derivatives, etc.)

In this webinar, graduate student Edwin Caballero offers an introduction on what are unwanted spectral variations and what methods can be used to reduce them.
Data preprocessing (DP) methods are mathematical algorithms used to reduce unwanted spectral variations in spectral data. They are highly versatile however they should be used sparingly, specially when using spectral data to create models.
----------------------------------------------------
CHAPTERS
00:00 Intro
02:44 Artefacts
03:21 Baseline Artefact
05:20 Scattering Artefact
09:30 Noise Artefact
11:24 Data Preprocessing Methods
12:45 Reducing baseline (detrending, assymetric least squares, derivatives)
21:30 Reducing scattering (SNV, RNV, MSC, normalization)
30:00 Reducing noise (SG smoothing, moving average)
33:39 Strategies for DP
37:59 Programs where you can use DP methods
----------------------------------------------------
More information from the presenter
👉 www.uprm.edu/ccs-cicsa/dt_tea...
Want to know more of our group? Visit our webpage!
👉 www.uprm.edu/ccs-cicsa/
See our webinar/events
👉 www.uprm.edu/ccs-cicsa/news/
See our projects
👉 www.uprm.edu/ccs-cicsa/our-work/
See our facilities
👉 www.uprm.edu/ccs-cicsa/facili...
Data preprocessing is a set of techniques that can be applied to data obtained with infrared and Raman spectroscopy to improve the quality and usefulness of the data. Data preprocessing can help to remove noise and other unwanted features from the data, and to correct for systematic errors or biases. It can also be used to align and combine data from multiple spectra, and to transform the data into a more useful form. For example, data preprocessing can be used to normalize the data, to apply baseline correction, or to perform peak picking. Overall, data preprocessing is an important step in the analysis of infrared and Raman spectroscopy data, as it can help to improve the accuracy and reliability of the results.
----------------------------------------------------
#DP #datapreprocessing #SNV #RNV #MSC #Normalization #spectroscopy #algorithms #baseline #chemistry #analyticalchemistry #college #webinar #research #instrumentalanalysis

[@ccscisac5607]

Thank you for your attention! Comments? Suggestions? Recommendations? All options are welcomed!

[@MIZRAIM1984]

I also mention Orange. It is a Python-based free software and it has a Spectroscopy add-on, which is excellent for the spectra visualization & preprocessing. The respective tutorials are encountered in KZbin too.

[@mainathuku9360]

Man. I just found out that I need to preprocess my data for MSC, this channel has really helped me. Thank you for the good job!!

[@pedramporbaha]

Wow that is awesome!!! The only point is the voice quality but your content and your presentation was awesome. Thanks a lot

[@DarshanaGopal]

Good quality content

[@marineduperat52]

Thank you so much ! I've been looking for these types of information for weeks. This is so far the mos informative stuff I watched. I just need to get how to apply it with R now hahaha. (Forest scientist trying to work with spectral data here).

[@ccscisac5607]

Always makes me happy when it helps someone! In the following link I started writing R scripts for SNV, MSC, and normalization. May not be optimal but they will give you a good start. www.uprm.edu/ccs-cicsa/files-info-for-research/r-language-resources/ Have a great day! - Edwin

[@rishabhjain6285]

you dont need R that much.There's a Git link in which the codes for all these are already given

[@monaallam130]

Hello , thank you for this nice video , for the scattering , you mentioned that one of reason for scattering is the molecule not at the same distance , so how can i make the molecule at the same distance if i prepared as example solution contain dye dissolved in water , i prepared different concentrations then i measured spectra for them and i got scattered for the data, thank you.

[@iam_Simbiat]

Thank you so much for this informative and well-explained session. I have been trying to write some code in MATLAB to perform MSC, could you please help with a script to execute that? Thanks

[@MrEcoscience]

Hello! Thank you! The best video in the net on data pre-processing I found so far! In Unscrambler, when you run the PCA based on the original data, you have the choice to select mean centering, which I think is meant to get rid off scattering. Would you in all,cases preprocess data with SNV and maximum centering?

[@ccscisac5607]

Hello! Thank you for your kind words. The choice depends on what I (Edwin) want to study with my spectra. For presentation I would use baseline correction and SNV to maintain as much as the original shape of the spectra as possible. For creating models, I usually use 12 different combinations of DP methods. Use SNV, MSC, SG1, and SG2 separately and then combine them in different orders (SNV+SG1, SG1+SNV, MSC+SG1, etc.). Once I have a matrix with each different DP method, I create a model for each different matrix. This way I can determine which combination gave the most optimal results. However if the artifact can be seen clear as day on the data you can simply use the DP method that best reduces the variation. Hope this helps! We are welcomed to any suggestions and/or corrections. Hope you have a great day!

[@oscarjmaytzuc2818]

Hello, an excellent video. The best explanation of spectral preprocessing techniques I've seen in years. Taking advantage of the occasion, I have a question that I would appreciate if you would help me by clarifying it. All these current techniques are on the spectra individually (rows), but in the literature, I find other techniques, such as mean centering, auto-scale, and variance scale, among others, that act on the variables (columns). In some manuals, I found the latter necessary because several multivariate algorithms compute results driven by variance patterns in the independent variables. Specifically, my question is: When to use tools such as mean centering, auto-scale, and variance scale, or are they already integrated into techniques such as PLS?

[@ccscisac5607]

Thank you for your kind words. Hope you are doing well. Mean centering is used to remove the common information on your data. Chemometrics assumes that variation implies information, hence why mean centering is so useful. When using PCA or PLS, mean centering the data allows the average variation between samples to be placed on the origin (0,0) of the scores plot. Auto scaling is usually used to leave the mean at zero and the standard deviation at one. From what I've understood, you scale variance when the variables that are being analyzed have very different magnitudes. This make some variables overshadow others, hence dividing the std allows them to play on an even field. Hope this gave some insight. Regards! - Edwin

[@muhammadharsanto2024]

Professor, which one from the programs you've mentioned on the video that can "mass" preprocess spectra? like can do multiple spectra in one touch Thank you

[@fabianenriquequinterotorde6038]

Buenas tardes seria bueno que esots videos tambien los compartieras en español

[@vivasjimmy]

can anyone point for me places to download spectral data for my research?? i am lokking for bearings spectrum data

[@ashenafibelihu1123]

am ok with data normalization (i.e., scaling), however I have a doubt concerning to the importance of DP methods for modeling!