CentOS 7: Set up Centralized Logging with Rsyslog

In this video, I will be showing how to set up centralized logging with rsyslog service on CentOS 7 servers.
This demonstration will be carried out on Virtual machines.
I have also been doing videos on more modern technologies for logging like ELK stack and EFK stack. If you are interested, please watch the following playlist.
• [ ElasticSearch 1 ] In...
For any questions/issues/feedback, please leave me a comment and I will get back to you.
Thanks for your time watching this video. If you liked it, please share it with your friends and don't forget to subscribe to my channel.
Thanks,
Venkat

[@patrickgosselin9756]

Great video! Good production quality and right to the point. thanks!

[@justmeandopensource]

Hi Patrick Gosselin, Many thanks for watching this video. Hope you found it useful. Thanks, Venkat

[@yfs9035]

Thanks, it's about time I set this up.

[@justmeandopensource]

Thanks for watching. Cheers.

[@thelebbies]

This was fantastic man!!

[@justmeandopensource]

Hi Sahr Lebbie, Many thanks for watching this video. Hope you found it useful. Thanks, Venkat

[@iammrchetan]

That's great! Very easy & crisp, learnt this. Thanks! Although I've issue like, %HOSTNAME% creates directory name as localhost of remote server

[@coreycox4802]

Great job explaining this

[@justmeandopensource]

Hi Corey Cox, Many thanks for watching this video. Hope you found it useful. Thanks, Venkat

[@SAlexandert8]

Really made this easy, thanks!

[@justmeandopensource]

Hi Stephen Alexander, Many thanks for watching this video. Hope you found it useful. Thanks, Venkat

[@Waris-bv7nu]

Great video. Thanks for sharing.

[@justmeandopensource]

Thanks Waris

[@nadzeyahutsko1094]

It is so helpful! Thank you very much!!!

[@justmeandopensource]

Hi Nadzya, thanks for watching. Cheers.

[@Pingao2012]

Nice!

[@georgeradu3759]

great work,man

[@justmeandopensource]

Thanks for watching

[@hamzadev8311]

Thank you man , very interesting (y)

[@justmeandopensource]

Hi Hamza, thanks for watching. Cheers.

[@santhosh933]

God bless you...

[@justmeandopensource]

Hi Santhosh S T, Many thanks for watching this video. Hope you found it useful. Thanks, Venkat

[@brainio1762]

Great tutorial :-)

[@justmeandopensource]

Hi Brain I/O, Many thanks for watching this video. Hope you found it useful. Thanks, Venkat

[@bodhisattwaghosh2673]

Very Good One.

[@justmeandopensource]

Hi Bodhisattwa Ghosh, Many thanks for watching this video. Hope you found it useful. Thanks, Venkat

[@Mac-ew1gv]

Great work!!! I was stuck for about 2 hours trying to figure out why my switch could not contact my linux server.

[@justmeandopensource]

Glad that it helped you. Thanks for watching!!

[@flavioreis9280]

Thank you for your video! o/

[@justmeandopensource]

Thanks for watching.

[@georgesmiley3334]

Thank you for this nice video. I tried to follow this to configure my own rsyslog file. However, my template won't create a new remotehosts directory for me. Did you do many other configure for the rsyslog.conf file? What is the unmask for your rsyslog.conf file?

[@justmeandopensource]

Umask doesn't matter. Are you sure you followed the video and rsyslog.conf configuration line by line. Please give it another try from "5:30". If you couldn't see the configuration clearly, it is the below three lines that you need to add to rsyslog.conf and don't forget to restart the rsyslogd service. $template RemoteLogsTesting,"/var/log/remotehosts/%HOSTNAME%/%now%.log" if $fromhost-ip != '127.0.0.1' then -?RemoteLogsTesting & stop Thanks

[@SonnetGomes]

Thank you so much for this tutorial. It's great to be able to use a log server to aggregate all the logs. But would it be possible to access the local logs if Network or the log server was down (or overwhelmed by 1000s of server logs)? Is it possible to scale or have any redundant log server?

[@justmeandopensource]

Hi Sonnet, Thats definitely possible. I just read an article from RedHat knowledge Base. IMPORTANT: Only works if you configure TCP (reliable guaranteed delivery of packets) forwarding with "@@" and not UDP (not reliable protocol) Configure two central log servers in the client's /etc/rsyslog.conf as below *.* @@central-log-1 $ActionExecOnlyWhenPreviousIsSuspended on &@@central-log-2 This will forward logs to central-log-1 server and if it can't be reached or it is powered down, the logs will be forwarded to central-log-2. IMPORTANT: On both the central log servers, where you collect clients logs (for eg: /var/log/remotehosts), it has to be NFS and mounted readwrite on both the central log servers. Otherwise if you use local filesystem then you will end up having log files in two places. Hope this makes sense Thanks

[@b_h_a_v_i_k]

Hi, Thank you for the video. I am facing an issue though. My configuration is not creating the remotehosts directory and continues to write to var/log/messages Do you know what can be the issue?

[@justmeandopensource]

After following my video from "5:30", did you restart rsyslogd service? The configuration you need to add as mentioned in the video is $template RemoteLogsTesting,"/var/log/remotehosts/%HOSTNAME%/%now%.log" if $fromhost-ip != '127.0.0.1' then -?RemoteLogsTesting & stop Thanks

[@panneerselvam3785]

Great tutorial Venkat. Do you have any reference to config rsyslog for application log files???

[@justmeandopensource]

Hi Panneer, You can do that. But its not straight forward. For example, if you wanted to forward Apache logs to a central log server via rsyslog forwarding, you will have to do something like below. Configure Apache to log using rsyslog. Have a look at below documentation on how to use the "pipe" argument on CustomLog directive httpd.apache.org/docs/current/mod/mod_log_config.html#customlog For eg; CustomLog "|/usr/bin/logger -t apache -p local6.info" combined ErrorLog "|/usr/bin/logger -t apache -p local6.err" That should be it. Apache after restart, should start logging to syslog facility and rsyslog will forward to centralzed log server. Hope this gives you some direction to explore further. Thanks.

[@JoseBarbosa-gv2mr]

This is an awesome video and it's working for me....however, I'd like to know if it at all possible to have the files merged together, when I'm trying to look at the big picture? Thanks in advance.

[@justmeandopensource]

Hi Jose, I guess you mean, on the server side you wanted all the clients to write to a single file instead of one per machine. You can do that. In the video at "5:45" I demonstrated how to configure rsyslog.conf to allow clients to log into separate file under /var/log/remotehosts//.log If you want all clients to write to same file, then just use the below line $template RemoteLogsTesting,"/var/log/remotehosts/allclients.log" And follow rest of my video. The more ideal way would be to use Splunk for these. If your organization has Splunk server, you can install Splunk Forwarder on each client or just on the centralized log server and forward logs to Splunk Server. In this case you can look at the bigger picture. Splunk is vast and apt for logging,searching,visualizing. Hope this was useful. Thanks.

[@faizi80]

Do you know if it is possible to just "copy" already created log files instead of online logging on remote server. I mean, copy the files to remote server when logrotate happens ?

[@justmeandopensource]

Hi Muhammad, the idea of centralized logging is to allow a system (client) to forward its logs **realtime** to a centralized server. This is useful since you can have one stop shop for all your client logs instead of logging into each of them. Not just that, it is also useful when security of client machine is compromised and intruder gets hold of the system. The intruder can do whatever he wants and manipulate the log files to make you believe that nothing happened. If you had centralized logging enabled, it is always recorded. Also you can control logrotation for all clients in one place on the centralized server. Given all these usecases, I don't understand why you want to copy rotated log files to another server. May be you wanted to backup and retain those logs. You can do that via a cron job on the client that rsyncs the pattern matching log files to a remote server. Hope this makes sense. Thanks

[@lanhnguyenthi1205]

I followed your steps, restared rsyslogd service after configured. Both servers can ping to gether but the log is still written into `var/log/messages`. Do you know the reason? Do we need to set Selinux to Permisive mode?

[@priyasudhirpriya]

Hi Venkat, I did the exact configuration but logs are not displayed on the server. logs are not recorded in the server. Any idea? Thanks for your video. It gave me an idea of how central logs work. Thank you.

[@justmeandopensource]

Hi Sudhir, thanks for watching. I did this video more than 5 years ago. I am glad that it is still relevant. I need to revisit this to be able to help you. Let me see if I can get some time for this.

[@priyasudhirpriya]

@@justmeandopensource thank you very much Venkat.

[@aronb.acostagarcia9784]

Nice video! qq how can I configure this to remove all logs older than one week?

[@justmeandopensource]

Hi Aron, thanks for watching this video. As per the setup shown in this video, you will see one directory per client machine in /var/log/remotehosts directory. Under each of those directory you will see log files per day. You can configure log rotation to do what you want. You can create a config file under /etc/logrotate.d directory. Thanks

[@aronb.acostagarcia9784]

@@justmeandopensource Thanks for your responce, I am trying with this but not works, vi /etc/logrotate.conf /var/log/remotehosts/* { ifempty size 0 postrotate /usr/bin/find /var/log/remotehosts/ -name "*.log.*" -type f -mtime +7 -exec rm -f {} \; endscript } Could you please provide the commands to do that? Thanks a lot!!

[@justmeandopensource]

@@aronb.acostagarcia9784 just noticed in you postrotate block, in your find command you have "*.log.*" which won't match anything. Try changing that to "*.log". But there are other elegant ways without using the hacked way of using postrotate block. That block is to do something when the log has been rotated.

[@TheNielsAlmelo]

can i also log windows machines to my centOS Rsyslog?

[@justmeandopensource]

Hi, thanks for watching this video. I believe its possible but not out of the box. You have to install some third party softwares for that. Check the below blog post which looks promising with datagram-syslog agent. yallalabs.com/windows/how-to-forward-windows-system-event-logs-to-a-linux-syslog-server/ There are also other softwares that can do the same thing. But I haven't tried any of them. Cheers.

[@faizi80]

Good short tutorail, It could have been better if you had used server/client words instead of cenvm01 02, it kind of confuses.

[@justmeandopensource]

Yeah that makes sense. I should have thought about it from viewers point of view. Thanks for your feedback and I will make sure I use appropriate conventions for naming.

[@erickwakye4491]

Hi how do u push windows servers or cisco logs to rsyslog a video could help please

[@justmeandopensource]

Hi Eric, Thanks for watching my video. I wasn't sure whether we could forward Windows event logs to a Linux rsyslog server. But when I searched, I found the below serverfault link where someone already asked the same question. And there seems to exist few options. I have got many videos in the pipeline that I need to work on. And if I get some time, I will check this and if something seems to be working, I will definitely make a video of it. Thanks, Venkat

[@erickwakye4491]

@@justmeandopensource i appreciate your response, also doing checks if i get something on it i would push the link to you. You know centralized log system is a big thing now so it would really help if rsyslog could very flexible with other operating systems or devices such as windows, CISCO, and others to push logs.

[@justmeandopensource]

Hi Eric, I don't think I pasted the link in my previous comment. Here it is. serverfault.com/questions/422800/forward-windows-events-logs-to-rsyslog/427036 Yes Centralized Logging is a big thing now a days. Just wondering whether you thought about Splunk or Elastic Search which are widely used. ELK stack especially is a good one. You can install an agent and forward logs/metrics from any device/OS. I did a video on ELK stack recently. If you are interested, you can check it out at kzbin.info/www/bejne/eZfFkqGfpJh_l9E and see if thats helpful. Thanks, Venkat

[@erickwakye4491]

@@justmeandopensource okay sir, will check and revert

[@aaammm1888]

can this be config to send log to my remote tor server?

[@justmeandopensource]

Hi, thanks for watching this video. You want the logs to be sent to a tor server? Is that a syslog service? Or do you want your tor server logs to be forwarded to a central syslog server?

[@aaammm1888]

@@justmeandopensource Hi thanks for the reply first thing first what a great video you made thanks well Im looking into security/hardening my centos servers I will have a few each server running different thing like one will be running nginx and php one will be running mysql so on i would like to make a central logserver where all my other servers will send logs to this server so i can monitor all log from one server but this server will be hidden within tor so the central server will be hosted on tor

[@justmeandopensource]

@@aaammm1888 If your client machines can talk to the central log server in a consistent way, then there shouldn't be any problem. What is the exact issue you are facing? Is your central log server not accessible? Is its IP constantly changing?