Spring Boot With Splunk Integration | Realtime logs analysis using Splunk | HEC | JavaTechie

In this tutorial I will walk you through step by step how to utilise Splunk's Http Event Collector (HEC) feature to index customised log events via a Spring boot application and a log4j2 .
Splunk provides the common Universal Forwarder utility to push log automatically from servers to index but this technique will make your application capable of logging into Splunk directly without having a properly configured server infrastructure in place
Spring boot microservice Live course Just started (Recordings available)
Hurry-up & Register today itself!
COURSE LINK : javatechie5246.ongraphy.com/
PROMO CODE : Java40
Splunk download link :
www.splunk.com/en_us/download
GitHub:
github.com/Java-Techie-jt/spl...
Blogs:
/ javatechie
Facebook:
/ javatechie
guys if you like this video please do subscribe now and press the bell icon to not miss any update from Java Techie
Disclaimer/Policy:
--------------------------------
Note : All uploaded content in this channel is mine and its not copied from any community ,
you are free to use source code from above mentioned GitHub account

[@tejachowdary309]

Never seen a splunk integration to application in any channel. We use it in our application, but donot know the backend configuration. Thank you so much for your efforts and keep going.

[@vamsyr1863]

Great Video!. Please explain about distrubuted tracing using Splunk in your future video.

[@msdinfo2871]

As usual crystal clear explanation on this topic! Would love to see more detailed video/ crash course on splunk for java developers!thank you

[@samirmujawar9703]

As always most useful video !! To the point, clearcut explanation !!!

[@abhijitsengupta6666]

A really very detailed walkthrough, very helping and prominent 👋👍

[@amitmalge1707]

Ohh my God, I am waiting since long time spring boot with splunk integration, Thanks Basant for the great concept

[@rohannova8268]

you are a best Teacher Bro 👨‍🏫, keep up the good work

[@ashishsahu1310]

Thanks Basant for giving such nice explanation.

[@phanimc11211]

Thanks for providing an excellent session

[@itsnarahari]

Yes bro more video we want. bcoz these is very easy for microservices also

[@gopalkrushnamohankuda1177]

Thanks Basant bhai for this wonderful session👏

[@cipahraoul2748]

Thank you for the splunk explanation

[@akumarsingh85]

Another superb video. The topic covered is useful irl

[@ranitdey5829]

This was really helpful. Thanks!

[@markosimic3732]

Nice intro. Keep up the good work.

[@gopalkrushnamohankuda1177]

Excellent tutorial👏👏

[@nitesh85a]

Eagerly whaitin for this.

[@mrinalsharma4137]

Awesome.. please create more content on splunk..

[@abulsyed4851]

For anyone running on docker. You need to expose 2 ports. 8000 is for the UI & 8088 which is the HTTP event collector.

[@robinrajlouis9183]

I have same doubt

[@karuppiahkaruppiah8803]

For docker we changed aport 8080:5000(or any other ports we want sooo) it will not contradict..docker and splunk we will integrate it together

[@deeproy2719]

Would like to see more videos on splunk search. Keep up the good work Basant

[@Javatechie]

Will do

[@akshaygavhande5070]

Good Video.. Please make one video on splunk with ECS AWS

[@rahul-java-dev]

Thank you so much Sir!!

[@dowlathbashag65]

Bro...thanks awesome . Have a query difference b/w Splunk & ELK(filebeat) which one is good and what is the scenario will these. Any Pro's & Cons . Kindly share the same .

[@hari.5375]

Hi basanth, please make a tutorial on how to implement logger for multithreaded application in java, and CONGRACTS FOR 100K SOON🥳🥳🥳

[@asishkumarsatapathy3466]

Perfect bhai .. in upcoming videos if you could please please add some tutorials about the ELK stack, that will be very very helpful. Projects are largely moving to ELK and Graphana with Microservices observability.

[@Javatechie]

Hi Ashish , please take a look at the tutorial 👇 . Already i covered ELK kzbin.info/www/bejne/a6ScoYVvirqnd7c

[@ritamsarkar3156]

excellent

[@sathiskumarp]

Thanks for providing an excellent session. I have been following many KZbin channels to learn the concepts. I feel you are able to explain very simply and it's really useful for me. In my project we were creating a user provided service (splunk) in PCF and binding that service with the application. Any idea how the index was created internally in PCF?

[@aniketsukhija9916]

Can you help me out, I am unable to see the logs on the splunk, although those logs appear on my terminal

[@rr3107]

Excellent bro

[@akshaysilgari2097]

Nice video

[@blessycheriyan5870]

Great resources

[@HakanOzcan-yp4kn]

Thank you

[@sumitjain3526]

Thanks Sir

[@yusufsaifi3108]

thanks a lot to upload the video , I need one video on how to handle exception in production environment using logs.

[@Javatechie]

Irrespective of any environment you will find logs directly in splunk dashboard just you need to filter and analyse further buddy

[@yusufsaifi3108]

Thanks for update me.

[@myselfshrikantkale]

Hi bro, believe logging not the only facility provided by splunk. Can you please provide how it helps to analyze systems behavior by analyzing logs. Also clear by Environment (is there option beside index ?). Also reporting using gui possible?

[@punyabikash3624]

Thank you so much Sir 🙏for the Splunk explanation this video is very useful great job....just one request please make video on end to end full stack application with angular Springboot....

[@Javatechie]

Already i uploaded a spring boot crud application with angular please check

[@Javatechie]

Refer this kzbin.info/www/bejne/h4eanJhqlpx_rMU

[@hrithik_raina24]

Such an amazing explanation. But sir I'm facing too many problems while implementing. Errors are regarding jakarta, splunk dependancy added there and many more things which I think happened because of spring boot 3.0. Can you please tell how to resolve all those errors?

[@hazarathreddy5239]

Recently I encounter the question how we can search logs using 2 indexes same time for single search. Could please make a video in this use case for the Transaction

[@beinspired9063]

What is the difference between Spunk, ELK, Appdynamics and Promethius? Can you please make a video on this topic? Please

[@muddeynagaraju644]

nice

[@AbdoAli-dr7ky]

Why the controller and service class logs are combined under single entry. In our app we have the different entry for each log entry we do in code. Which setting is responsible for that?

[@dejarvis8407]

Azure App Insight Log Analytics vs Splunk ? Which is preferred and efficient plus simple way bhai?

[@debasishmahapatra681]

Hi sir Your videos are truly next level. And helped me alot. I too want to create content like this for python and aws. Can you please tell me what all tools do you use to create videos and how do you create the display page please?

[@Javatechie]

Please connect with me on [e-mail]

[@hungcao1342]

nice !

[@yogeshchaudhari9971]

Excellent session.. Starting from setup to adding and searching logs.... Can you please add more videos on splunk for more searching tips... Setting alerts... And preparing dashboard in splunk(like no of request on a Api etc)

[@Javatechie]

Yes i will do that 🙂

[@tejachowdary309]

@@Javatechie Yes, that we be helpful as in most of the real time applications, alerts will trigger for specific conditions

[@vineethgaddam6137]

@@Javatechie can you please make a video on what are the tools or frameworks used in the real time. Like now you expained for splunk which is used for logging in realtime. For service discovery in your sessions you have explained it using Eureka but in the production level they might be using different tool or framework to achieve this serice discovery right. simillarly loadbalncing,resiliance,API Gateway and declarative client they might using some realtime tools.can you please please make a vidoe on this.

[@yogeshchaudhari9971]

@@Javatechie thank you so much

[@krishnan6201]

what level type logs are configured while during the production? please clarify my query?

[@praveenpatel7148]

Hi everyone, Can i use splunk along with sleuth and zipkin in springboot project for logging as splunk and zipkin for distributed tracing.

[@VijayKumar-sv3zd]

More videos on splunk sir... In my current project it is getting used and I am new to this tool

[@VijayKumar-sv3zd]

Thank you so much for this video

[@vinayakjamadar986]

Logs are printed correctly in the console without any Error, but not getting pushed to the Splunk server. Please help

[@pallavisinha9047]

The logs are not getting reflected in splunk . Please help

[@Javatechie]

Please copy paste the logback.xml correctly

[@pallavisinha9047]

I have copy pasted it correctly but still it is not reflected in splunk

[@LeetCodeWithAK-im1nn]

How to configure log2.spring.xml file for different environment. As of now we are configuring for dev index what about other environments like qa, prod etc..

[@vaibhavpatil2693]

Can u make video how we send pipeline logs on AWS Cloudwatch

[@mahadevaswamygn4216]

i did the same, evrything is fine , but in dashboard it showing ( No results found. Try expanding the time range.) , i dont know why, anyone help me

[@ranikanojiya7394]

Hi Sir, If we want Splunk Logs back in ourJava Application. Is it possible?

[@abhishekkeshari47]

Hi, Please can you cover ELK stack with spring-boot microservice application in details including logging agent, It will be very vert helpful. Thanks in Advance

[@faizulislam5096]

Nice explanation! But this demonstration is for monolithic architecture. Could you please also explain the log aggregation for microservices architecture using splunk?

[@Javatechie]

Sure i will give it a try

[@faizulislam5096]

@@Javatechie Thankyou!

[@sandeshmagar9670]

How do we remove the initial unnecessary lines e.g. {“host”:”localhost………. I want to see only app logs The other texts makes hard to read.

[@user-bt6zq3tn3g]

How can we make each log in new line for readability instead of having whole flow logs in one record in splunk? Can any one help me with this.

[@ezhilarasikrishnan5408]

❤❤❤

[@pardeeppathak3394]

bro i am not getting logs in Splunk i fallowed the same step but not getting

[@Javatechie]

Please validate your index

[@lllingardium]

splunk enterprise system requriements please tell us that also

[@vikaskumarmishra4353]

I have been working on POC of it , how did you get to know the respository URL

[@Javatechie]

Which repository url ?

[@noorahameds8]

Do we need to name the logger file in the same format as specified in the video? And we we have multiple logger files (for dev and other for QA and the logging level of each environment is different ?)

[@Javatechie]

No you can customise it as per your needs. It's just a demo

[@noorahameds8]

@@Javatechie can we have multiple files also? With different indexes?

[@Javatechie]

Yes we can have

[@nethanchowdary4657]

Need video on how to use new elasticsearch java client

[@Javatechie]

Already uploaded please search with keyword elastic search in my channel

[@nirmesh44]

CAN YOU CREATE video for generating alert?

[@Javatechie]

Yes working on it

[@alexlinux904]

Hi, I tried to follow your tutorial but when I searched the index in splunk is not displaying anything :(

[@Javatechie]

Did you configured index , source and source type correctly in log4j.xml . Make sure to define splunk http port 8088

[@aniketsukhija9916]

stuck on the same problem. were you able to solve it?

[@jeetendra53]

Yes...same problem...no logs displaying.

[@kunalsharma-zc2ho]

Hi sir , Any specific advantages does it can provide as compared to kibana ...please suggest ... I am looking for a scope more of a distributed request tracing .

[@manoharbomma467]

Hi Basant How to post sleuth traceId to splunk?

[@Javatechie]

I have not done any similar integration will check and update you

[@aniketsukhija9916]

Can someone help, I am unable to see any logs in the splunk enterprise after following the tutorial

[@abulsyed4851]

Facing same problem. Were you able to find a solution?

[@ShivamKumar-eh6mg]

Could you please zoom while showing code or configuration it is difficult to see

[@Javatechie]

Sure buddy going forward i will take care this

[@ShivamKumar-eh6mg]

@@Javatechie Thank you

[@adriansmith38]

feels like it's not needed to put the classname into the log message as the logging framework already knows that and passes it for you.

[@sahilpatel2885]

Can I format logs for better readability ?

[@Javatechie]

Yes you can do that just specify your log pattern in log4j2.xml which i created in resources folder

[@sahilpatel2885]

@@Javatechie okay thanks for your support.

[@pallavisinha9047]

How to generate separate trace I'd for logs

[@Javatechie]

Traceid or index ?

[@pallavisinha9047]

TraceId

[@nilanjansarkar100]

I do not see logs on Splunk, but I see no error on Spring application log as well

[@Javatechie]

Copy paste my log4j.xml file

[@nilanjansarkar100]

@@Javatechie thanks for your reply. I figured out the issue. Basically, the splunk logger library does not work with Splunk 9 which I was trying with, falling back to Splunk 8 solved the issue

[@Javatechie]

Cool , can you share the dependency on what you used

[@nilanjansarkar100]

@@Javatechie I used your code. Did not changed anything. I just downloaded the Splunk version to 8.2. FYI, I used the docker for splunk

[@Javatechie]

@@nilanjansarkar100 got it

[@manirsg9641]

React +spring webflux + gcp/aws + monitor grafana trending combination. It would be more helpful if we get video with mini project

[@ZtowhyA]

seems like elasticsearch

[@Javatechie]

Non comparable buddy both are different context

[@TaufikHidayat-oo6ko]

i get this error Sir when i do search the log with index="order_api_dev", "Configuration initialization for C:\Program Files\Splunk\etc took longer than expected (4343ms) when dispatching a search with search ID 1663750394.31. This usually indicates problems with underlying storage performance."

[@Javatechie]

Shutdown and restart your splunk server again please

[@naidu12341]

Great Video I am getting the below error while running the above code ,Please advice me to resolve the below 2022-12-21 01:06:46,480 main ERROR Unable to invoke factory method in class com.splunk.logging.HttpEventCollectorLog4jAppender for element SplunkHttp: java.lang.NoClassDefFoundError: okhttp3/Callback java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:138) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1122) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1047) a