Splunk Configuration Files : Search time field extraction

[@hamidbk7164]

This is a perfect content. I bought a Udemy class and this one so far is the best.

[@splunk_ml]

Thank you Hamid.

[@almesa7099]

Excellent Detail, this explanation is priceless. Thanks and keep the amazing splunk videos coming!

[@jaireddy1258]

Hey Sid, thanks for sharing your wonderful knowledge. It is helping thousands of people. You're awesome man. Can you also do a video for extracting Json data in Splunk, no body have did that until today.

[@splunk_ml]

I discussed similar stuff in below video kzbin.info/www/bejne/gKTLemmAiZtjiNU

[@CHAMP_GUY]

splunk watch this guy. He is the right one.

[@dionrivera1163]

Hi Sid, does this video apply to splunk cloud and events being sent over sc4s?

[@hemnaathgovartan3668]

Hi Siddhartha, the videos which you are posting is good and easy to understand and in this video the concepts were explained well, thanks Siddhartha. I have a suggestion for you, If all your videos are placed in sequence, it will help to get aligned with the next topics and also if you can create a videos on Splunk clustering it will be really helpful. Looking for more videos Keep going !!!

[@splunk_ml]

Hi Hemnaath, Many thanks for your valuable feedback. The videos are properly sequenced in my splunk how to playlist. So please follow that. Regarding clustering I have a plan to cover splunk admin as well. I will definitely try to cover that. Have a nice day.

[@chenna8989]

Splunk & Machine Learning hi Sidhartha, looking forward to your admin series

[@splunk_ml]

Hi Kartiga, I will be starting from next week 🙂. Already planned for it. Sid

[@biznesless4672]

great i realy like your videos it helpful

[@rotimiakinbobola2327]

Hello Sid! Thank you for doing a fantastic work. Please if this extraction is done at search time which obviously it is. Do you get to still edit the props.conf? coz I can see that you change the sourcetype to demo,demo3 etc. I am a little bit confused...have you already load those files separately?

[@splunk_ml]

Hi Rotimi, Yes I already created those stanzas in props.conf to save time on multiple Splunk restart. Sid

[@pankajkapadiya]

very nice explained I have one query if you can help me on it. Two types of events is coming in one log files with , separated. its have only values without header. I want to make it two regex and field mapping based on starting string of event For example: one file will have below two rows start,id,type,category stop,date,time,id,type,category,reason how can I achieve this field extraction based on types of rows? thanks in advance

[@kumarodela]

hi sir, i have one doubt, is there any specific scenario where we have to use REPORT-className option Or EXTRACT-className option? i got the difference, however i just want to know scenario where we should go with REPORT option Or Extract option. thanks in advance...

[@JiaLiang-cg3up]

Per my understanding, correct me if I'm wrong. REPORT is like function, and you declare how the function works in transforms.conf. Then, in props.conf, you can always call to the function you defined in transforms.conf. So, write once, can be used in different stanza(s) in props.conf. Then, whenever you want to change, you just make changes once at transforms.conf that particular stanza. Instead of going through every EXTRACT functions to change in props.conf under every applicable stanza.

[@prammod123]

Hi Sid, Is splunk index-time extraction video yet to be published... I couldn't find it under play list or videos.

[@splunk_ml]

Hi Prammod, Yes it's in pipeline. I will be posting it soon. Sid

[@ebq8281]

Hi Siddu, thanks for all your videos.. they are much helpful... I have one question for you... Let's say I have extracted fields like A B C.. Now I want to merge any two fields by creating new field.. Like D = A+B, How can we do this??? I have done this eval report = A. "-" .B and which is giving result... But I need to run SPL query every time.. So can you tell me to merge the existing two fields?

[@splunk_ml]

eval is the best way to achieve this. Otherwise you need to go for index time field extraction (INGEST EVAL config) but thats not recommended. Another way could be through data model. If you have datamodel setup for your app then in data model you can do eval so that whenever you query the datamodel the new field will be available.

[@DunstanVavasour]

@@splunk_ml Could you not do that using a calculated field? If A and B have been created using an EVAL or REPORT field extraction then calculated field should follow?

[@splunk_ml]

Yes even we can use calculated field here as well. Behind the scene they are eval like statement only so it will be calculated at the search time, so performance wise it will be similar I think.

[@riptideflute3552]

Hi Siddhartha, I have been following you wonderful videos. Never hit an error with your exact same git repo, until this time:/. Starting from xmlExtraction with demo, I didn't get the fields to extract in my search and "WARN SearchOperator:kv - Invalid key-value parser, ignoring it, transform_name='xmlExtraction'." in my splunkd.log while line breakers set in props.conf work properly. I'm also working on props.conf and transforms.conf in Splunk\etc\apps\search\local folder on Windows. Could yout think of any reasons for this error? Thanks a lot!

[@splunk_ml]

Hi Yilun, I just tried...its working..are you uploading data3.txt for xmlExtraction (sourcetyrpe = demo)? Sid

[@riptideflute3552]

@@splunk_ml Thank u for your reply! Yes data3.txt except that I changed all timestamps to 2019 ones so as to avoid _time parsing "out of ... range" error

[@riptideflute3552]

@@splunk_ml and not only 'xmlExtraction', same error for parameterExtraction and activity_report also

[@splunk_ml]

Can you send me the updated file. I will take a look into that.

[@Sharukhkhan-br7pw]

Awesome👍👍 i think you working in splunk...

[@splunk_ml]

I am not working in Splunk ☺️

[@peace336-n8g]

Very helpful. Can I fork your repo?

[@splunk_ml]

Thanks...Yes of course you can fork..It's open to public.

[@CHAMP_GUY]

full of knowledge

[@tibuuso]

How can I apply regex for multiple sourcetypes with different names?

[@splunk_ml]

you will be creating seperate stanza for different sourcetypes in your props.conf. Please remeber you can also apply props settings for source or host level as well. So based on your situation you need to create corresponding stanzas.

[@adrianboom6879]

regex101 has change since 2019 and we now have two PHP options but neither seems to want to work ...both of them tell me that my regular expressions does not match the subject spring

[@adrianboom6879]

string even

[@naveengade423]

where we need to configure this props & transforms for search time extraction? in app level?

[@splunk_ml]

You can either keep it app level or system level, at run time splunk always combine app and system level configs to create the final version of the props and transforms.conf The practice I used to follow is when I am keeping those configs at app level I keep it in default, for system level I keep it in local.

[@dhanabalanrangasamy9462]

What is the difference between field extraction and field transformation in fields section..

[@splunk_ml]

Fields extractions are related to props.conf entries and field transforms related to transforms.conf.

[@dhanabalanrangasamy9462]

@@splunk_ml thank you..we want to create fields fields extractions field transformation calculated fields in Dev environment same like prod.. We can copy props and transform conf from production? That should be good approach right instead of creating manually in dev environment

[@splunk_ml]

@@dhanabalanrangasamy9462 yes that should be enough.

[@RamuChittiprolu]

Great session.

[@sangeethsuseelan2380]

Sid, please share the link of Regex Video.

[@splunk_ml]

Hi Sangeeth, Please find the links below kzbin.info/www/bejne/rGLLe6SAadmMic0 kzbin.info/www/bejne/gnW9YYOBiM6opdk kzbin.info/www/bejne/pqG2qaOwgJemoJo

[@Good-Enuff-Garage]

On a Scale of 1 (low) to 5 (high) Presenters Knowledge = 5 Presentation = 1

[@TheSharkdba]

Not sure why you complain on the presentation. You can clearly see the presenter did a lot of preparation with appropriate data, configuration and examples to be able to present a somewhat complex subject in a concise manner and following a clear logical path. Every piece of information builds nicely on previous so as long as you pay close attention everything makes sense. You can learn a lot in short time. You can clearly see everything that is being shown, and sound quality is good as well. According to your scale the presentation is if not 5, then certainly a very solid 4.