SOC Open Source, ELK- TheHive- Cortex- MISP Complete Setup Guide, Part 1

[@BlackPerl]

Soc Open Source is a Project Designed for Security Analysts and all SOC audiences who wants to play with implementation and explore the Modern SOC architecture. All of the components are used based on Open Source Projects(Available at the time of first commit). This is Part-1, we will show the base of the model with ELK, TheHive- Cortex-MISP and we will use some dummy data to ingest in ELK. In coming up episodes, we will include more data sources to ELK- Wazuh, Snort, Honeypot and Also we will integrate Atomic Red Team to ELK for Attack Simulation. We will also show how can you automate your flows with Shuffle. So watch this space out! This Project serves below usecases: 👉Collect Data to a Single Place. 👉Normalize and Parse Data 👉Visualize Data and prepare meaningful Security Analytics 👉Create Incidents/Cases out of Security Alerts identified based on collected data/logs 👉Automate process of Threat Hunt, Creation of actionable Playbooks, SOC data Analytics 👉Automate the process of analysis observables they have collected, at scale, by querying a single tool instead of several 👉Actively respond to threats and interact with the constituency and other teams 👉Enrich Data feeds with Open Source Threat Intelligence Platform In this episode, I will cover from scratch how can you install all of the components- Elastic Stack, TheHive, Cortex, MISP and will also show how can you integrate all of these components with each other. This Project can be used to any small/big organizations who wants to create their SOC Set up using Open Source Tools, also by any Security Analysts, Engineers who wants to build a SOC Lab which has all of the components- SIEM, Case Management, Threat Intel Platform, Threat Hunt & Analytics capability and lot more. You will find similar kind of projects online but this is the FIRST TIME we are showing everything bundled up and with full working condition. Just follow along the tutorial to get a high level overview of the end product and get started from the Git Repo Below. 🔗LINKs for your requirements- ------------------------------------------------------------------------------------------------------------------------- 1. Project- github.com/archanchoudhury/SO... WATCH BELOW Playlists as well, if you want to make your career in DFIR and Security Operations!! ------------------------------------------------------------------------------------------------------------------------- INCIDENT RESPONSE TRAINING Full Course 👉kzbin.info/aero/PLj... DFIR Free Tools and Techniques 👉 kzbin.info/aero/PLj... Windows and Memory Forensics 👉 kzbin.info/aero/PLj... Malware Analysis 👉 kzbin.info/aero/PLj... SIEM Tutorial 👉 kzbin.info/aero/PLj... Threat Hunt & Threat Intelligence 👉 kzbin.info/aero/PLj... ⌚ Timelines ------------------------------------------------------------------------------------------------------------------------- 0:00 ⏩ Introduction 1:28 ⏩ Architecture Overview 8:40 ⏩ Overview of the full setup 22:12 ⏩ Install the components 41:30 ⏩ Integrate the components 48:01 ⏩ Summarize 📞📲 FOLLOW ME EVERYWHERE- ------------------------------------------------------------------------------------------------------------------------- ✔ LinkedIn: www.linkedin.com/company/blac... ✔ You can reach out to me personally in LinkedIn as well- bit.ly/38ze4L5 ✔ Twitter: @blackperl_dfir ✔ Git: github.com/archanchoudhury ✔ Insta: (blackperl_dfir)instagram.com/blackperl_d... ✔ Can be reached via [e-mail] SUPPORT BLACKPERL ------------------------------------------------------------------------------------------------------------------------- ╔═╦╗╔╦╗╔═╦═╦╦╦╦╗╔═╗ ║╚╣║║║╚╣╚╣╔╣╔╣║╚╣═╣ ╠╗║╚╝║║╠╗║╚╣║║║║║═╣ ╚═╩══╩═╩═╩═╩╝╚╩═╩═╝ ➡ SUBSCRIBE, Share, Like, Comment ☕ Buy me a Coffee 👉 www.buymeacoffee.com/BlackPerl 📧 Sponsorship Inquiries: [e-mail] ------------------------------------------------------------------------------------------------------------------------- 🙏 Thanks for watching!! Be CyberAware!! 🤞

[@CosmicSoundMDB]

when is continuation coming?

[@AliRaza-s3x4f]

Can you please tell me how can I integrate MISP and ELK Security so I can get MISP feeds on ELastic Security?

[@rajkumarkumawat3236]

Great guru dev ... Esa content koi. Nhi deta but aap aache se share kr dete ho ...💥💥🙌🙌🙌🙌🙌🙌🙌🙌🙌🙌🙌🙌🙌🙌 thanks 😊

[@BlackPerl]

Thank you

[@palevelmode]

Finally someone take the initiative!!!

[@TrusteestDesiChhora]

Thank you so much vai bahat helpfull video oll in soliusation thank you so much brother

[@yalghaar9936]

Can we install the same setup in local machine in virtual box or VMware ?

[@BlackPerl]

Yes, absolutely.

[@ses8105]

What VM requierements do i need?

[@BlackPerl]

You can check out the git repo. Link is in the description. It has the system requirements

[@PaulMisner]

Can I get thoughts on doing this versus something like Security Onion. Hive and Cortex used to be part of SO, until Hive changed their licensing.

[@numanmaavia8575]

Thanks excellent ,informative lecture

[@BlackPerl]

Thank you. Keep watching the space, more enhancement are planned

[@haitran3765]

I have set up as documented but when start service thehive is still running but no port 9000 appears

[@BlackPerl]

Not sure, what issue you are facing. Have use checked the app logs in hive?

[@haitran3765]

@@BlackPerl I have checked but can't see that it's not running port 9000, or because I installed the ELK 8X and thehive 4x version, so it's not compatible

[@BlackPerl]

@@haitran3765 You can try out a stable version of elk. Also did you verify if hive is running, may be it took diff port. But is the service up?

[@md4six]

Your video is fantastic, but with the end of the thehive open source project, and with your new stragebee licensing for version 5 of Thehive where you have limitations on community license, what alternative and recommendation for a soc project. Tell me about this.

[@BlackPerl]

I haven't tested it out to understand how the limitations will affect our usecase and functionality. Will check on it.

[@handyplazt]

Great video tutorial and works perfectly, unfortunately for webhook connector in elk you need a subscription to activate it or do you have another tutorial to activate it without subscription?

[@kaushalpatel748]

at the connector stage of elasticsearch-thehive webhook gives the error [404] Not Found: /api/cases please help me

[@BlackPerl]

Make sure your both instance are reachable from each other and also you have filled the other details properly at the connector page

[@kaushalpatel748]

@@BlackPerl i installed both in linux vm locally and the all information i filled correctly and both are reachable in browser still i got same error

[@kaushalpatel748]

now it shows The following error was found: error calling webhook, invalid response Details: [400] Bad Request: [Attribute title is missing][Attribute description is missing]

[@BlackPerl]

@@kaushalpatel748 There must be indentation error or syntax error

[@ShantaNaha-yq4tb]

can you please make a video how can i install cortex on ubuntu. I tried so many times and i installed it on my machine but when i run it on browser on port 9001 than it show connection closed. Can you please help?

[@BlackPerl]

We do it almost everyday in our Live Certification Course. If you are installing it on aws ec2 on ubuntu, please make sure you have a port open on security group for 9001 to your IP. And for local installation, make sure nothing is blocking on local firewall on port 9001

[@cyriljohns]

Thank you very much for sharing this!

[@nirikshatk3150]

Thanks for the great content. I have a question , we can create alerts and rules only with paid version of elasticstack, any way to use it for free even after free trial ends?

[@BlackPerl]

Thanks for reaching out. Yes, you just need to redeploy the Elastic pod after the lisence gets expired

[@1UniverseGames]

Hi, Is there any video how to install SIEM tool on windows or ubuntu from scratch? I can't find a good video to install it, anyone would like to help. Thank you

[@BlackPerl]

You can check this video- kzbin.info/www/bejne/bHWznq2Ojbuila8 I have explained some SIEM installations. Also, if you are looking for any specific SIEM tool, let me know.

[@1UniverseGames]

@@BlackPerl sorry for get back late. Yes this video seems cover elastic siem ibm one. by any chance can you share a video link which shows splunk one recently. That's would help. Thanks again

[@BlackPerl]

@@1UniverseGames OK. Will try to make one

[@1UniverseGames]

@@BlackPerl thank you 😊

[@vimukthiperera4993]

sir i want to connect filebeat to misp . sir is there are possible way or tutorial to archived that target

[@BlackPerl]

You need to use elasticsearch to get logs from filebeat and elasticsearch can be integrated with misp

[@rofiko7056]

What about the hardware if I want to build for enterprise 500+ users

[@BlackPerl]

It's preferred to use autoscaling to load balance with Elastic running on 5 to 6 nodes. Remaining you can still try to get for 3 to 4 nodes. The highest limit should be as per your need.

[@raony6175]

do you think this project is a good open-source alternative to splunk?

[@BlackPerl]

Elastic SIEM is the alternative of splunk. This project has further capability which SOC needs and SIEM is only part of it

[@bhabyzhark4603]

can you achieve this setup on a local machine using oracle virtual box instead of aws?

[@BlackPerl]

Sure, we can do that. The vms will be on local machine and you should have a higher end machine to configure all.

[@bhabyzhark4603]

@@BlackPerl 8 cores 16t cpu & 32gb ram can run this setup?

[@BlackPerl]

@@bhabyzhark4603 Yes, it should run, but will work slow. Since Elastic needs 4gb ram, TheHive is also same, cortex and MISP can manage with 2 gb each.

[@bhabyzhark4603]

@@BlackPerl what ubuntu os should i use? is it the desktop image or the server image?

[@BlackPerl]

@@bhabyzhark4603 Anything should work. Ubuntu 18 is preferred

[@bataviaproductions6537]

can i replace elastic siem with wazuh?

[@BlackPerl]

Yes, you can

[@bataviaproductions6537]

@@BlackPerl thanks for reply sir

[@1231super]

The docker compost file is not available anymore! Could you upload it again?

[@BlackPerl]

Yes the file has been removed since it was having our custom codes. You can still get it from Elastic git repo

[@1231super]

@@BlackPerl will it still work when I follow the tutorial if I use the one from the Elastic repo?

[@BlackPerl]

@@1231super Yes it will work. Because, it's just yo create the Elastic stack. It's the same config. Nothing extra. You can alternatively install Kibana and elasticsearch separately. That is also easy. Steps are given in Elastic Repo

[@1231super]

@@BlackPerl thanks! I used the elk-stack compose from another user on git. I will follow the tutorial now. Thanks for your work on this project :)

[@bdcirt6125]

H, Archan! To create connectors for the hive, a license is required for the elasticsearch, right?

[@BlackPerl]

Yes correct

[@GoGoStitch]

Hello, i have a problem for bringing the hive instance up. There is a weird error that i cant solve can you help me with this problem?

[@BlackPerl]

What is the problem or error you see?

[@GoGoStitch]

@@BlackPerl i cannot download the hive because it says the certificate is expired a secure handshake couldnt be done.

[@BlackPerl]

@@GoGoStitch Try a diff version of hive. Either the latest version or the stable version of hive4 should work. You can get this from TheHive official website

[@GoGoStitch]

@@BlackPerl yes thanks! but i have now a different error now with the hive service. I cant get the service up, because it says that there is a bad setting in the service which why it wont start. do u have encounter this error before?

[@BlackPerl]

@@GoGoStitch This seems odd. I didn't face this error

[@hibaoueslati1481]

Have you changed the link of ELK compose it is 404 not found

[@hibaoueslati1481]

here is the link that you have put it raw.githubusercontent.com/archanchoudhury/SOC-OpenSource/main/codes/elk/docker-compose.yml?token=AMFWN76WO6EJP3LVF5DVHNLBWN7KQ

[@BlackPerl]

Yes, it has been deprecated

[@usmanshah260]

hey, how much RAM and HDD are required for each of the components ?

[@BlackPerl]

Please check the git repo. Everything is mentioned

[@adiyavmani1742]

in my elasticsearchnyml file securty is enabled, after that i am not able to open cortex web interface . It is showing elasticsearch connection refused

[@TheSplash1983]

Hi BlackPerl. Thanks for the great content. I have a question for you. Can I run this project only on one instance in AWS? Let's say on t3a.xlarge?

[@BlackPerl]

Thank you. There might be little problem since some of the components uses elasticsearch at their backend. So it might clash with our Elastic siem. But if you can compile the full environment in docker/kubernetes, it will be awesome and can be deployed on a single host.

[@abhishekchaurasia1060]

Thanks for this amazing video, just a quick help required, not able to find the ELK yml code on your git repository

[@BlackPerl]

Thanks for the feedback. Yes, I have removed our custom code from git, and enabled a freelance service. So if you are interested, let me know. Basic Elastic code can be found in Elastic git repo, I believe.

[@ameer526690]

Hello, Do you have a latest version of GitHub for open source soc solution ?

[@rohanpanchal4232]

Hi Sir, I have stuck between cortex and the hive setup can you pls send us a document for implementing this server.

[@BlackPerl]

Hey, have you checked the git documents? What exact error you are facing?

[@kader8815]

how many ressource i need to run this project ( 16 RAM good ? )

[@raonyjose532]

link to download elasticsearch and kibana docker is not working, error 404

[@BlackPerl]

The custom code has been removed from repo. You can find the same code in Elastic official repo.

[@horijanrai1640]

sir is it offline or online ????

[@devashishsingh1]

How do you manage the upgrades and least storage consumption without impacting performance? Another thing is since Elasticsearch has to be a subscribed one, it's no longer open source anymore.

[@k-beauty06]

is there any part where we need credit card details or pay for something

[@VictorOliveira-gf8fr]

Hi Archan, I'm trying to assemble the Elastic SIEM solution in my local environment, but I'm having difficulties making the integrations because the opensource version doesn't support connectors. Is there any way I can generate these alerts and integrations for free?

[@BlackPerl]

Hi Victor, Thanks for reaching out. Unfortunately, Elastic SIEM doesn't offer connectors and integration on free tier. You can still try with ElastAlert which might be helpful.

[@BlackPerl]

Also you can try Wazuh in the place of Elastic SIEM.

[@lautaronahuel6413]

Excellent thanks! Could you activate the automatic subtitles option in this video?

[@BlackPerl]

Thank you. Sure, will do it

[@magueritemichima6818]

Interestting topic but it would be interessting to include an open source NGFW ( next generation firewall)

[@BlackPerl]

It's just a prototype. Feel free to contribute and raise PR

[@rohanpanchal4232]

Sir can you let us know how to configure cortex?

[@BlackPerl]

I believe I have explained the steps in this video. You can refer the git repo too for a written steps. Are you looking for anything else in specific?

[@madhankumar7847]

Hi BlackPerl, this video is amazing.. and i installed and configured the ELK, thehive, cortex, and MISP, now i can able to create automatic ticket in thehive but cortex and MISP part is not working.. i am not get alert.. i am struggling more than 20days..Kindly help me ASAP

[@BlackPerl]

What issue you are facing on Cortex and misp?

[@madhankumar7847]

@@BlackPerl "worker cannot be run" this error showing in cortex.. Any possible to contact you with screen share..

[@BlackPerl]

@@madhankumar7847 Sorry, don't have bandwidth for screenshare session. But will see if this error can be generated from my end to troubleshoot the issue

[@madhankumar7847]

@@BlackPerl any update??

[@nafeeskhan007]

nice tutorial. deeply described. Why dont you include Wazuh in it??

[@BlackPerl]

Thanks, Will do for next tutorial..

[@emrea1570]

@@BlackPerl im a noob on this. one question, what is alienvault in this archicture. Is wazuh and alienvault doing the same thing?

[@BlackPerl]

@@emrea1570 Alienvault is basically kind of siem but we are not using that here. We are using Elastic siem. Alternatively, wazuh can also be used as open idps solution in replace of Elastic siem

[@emrea1570]

@@BlackPerl okej, Thanks for the answer🙏! is it possible to do your solution as a virtuel SOC and a In-house SOC.?

[@BlackPerl]

@@emrea1570 It can be done for both

[@mohomedarfath4780]

how enable ssl in cortex

[@BlackPerl]

You can use openssl for self signed cert or buy one from a CA.

[@nabiladouani7781]

I recommend using a reverse proxy in front of it to handle SSL

[@BlackPerl]

@@nabiladouani7781 good idea

[@brunoduquenoy3362]

Great content ! Many thanks !

[@JaeVoris]

Nice video, how do we install it all? Step by step instructions?

[@BlackPerl]

Thanks. You will find them in github. Link is there in the description area

[@577Pradeep]

buddy watched the whole video now ..can you make another video where u ingest malicious test data to elastic and show the flow among other products...please

[@BlackPerl]

Yes, we will do. It's just part1. More episodes are planned. We will utilise Elastic and Thehive from now on. I hope you got the idea of MISP and Cortex. Let me know otherwise

[@577Pradeep]

@@BlackPerl thanks buddy

[@vimukthiperera4993]

sir can i send theHive alerts to opencti

[@julionasmon]

bro, why Installation Guide pages is different with this videos? 😄

[@anthonymukwaya]

Great content. Thank you. I keep getting a " ERROR 404: Not Found." error everytime i try to download the "docker-compose.yml" file.

[@BlackPerl]

Yes, the custom code has been removed from git. It's just to keep our propitory information intact and we take it as freelance project. You can still check Elastic official repo for base docker code. Otherwise, you can install elasticsearch and kibana seperately. If you need installation help, do reach out.

[@anthonymukwaya]

@@BlackPerl understood. Thank you

[@vasudevanayak4439]

i believe this is outdated. any chance to update this ?

[@577Pradeep]

Good one mate

[@BlackPerl]

Thank you

[@vinyldown8490]

I was waiting for a full tutorial on Elastic and installing all of this. Too bad you are skipping this part

[@BlackPerl]

Thanks for your feedback. Detailed tutorial would have been a very long episode. You can check out the step by step process for the same in my github anyway.

[@vinyldown8490]

@@BlackPerl Ι can understand it, but that would be helpful. I am trying to build my lab and I find it hard...

[@BlackPerl]

@@vinyldown8490 Understood. Let me know what issue you are facing. We can connect over LinkedIn. Please reach out to Archan Choudhury.

[@vinyldown8490]

@@BlackPerl Thank you, but I am not really interested in hiring someone for it. I prefer doing it by myself :)

[@BlackPerl]

@@vinyldown8490 Not an issue. It's not about hiring, but since he has done these kind of many projects, if you face difficulties, he might help solving them. Has done for same for many.

[@meryemb1324]

to create rules in ELK it is not free

[@sumit2308]

Archan you are champ!!!!

[@BlackPerl]

Thank you!!

[@jimneshjimnesh2106]

please share the github link

[@ctnguyenvn2178]

please enable subtitle @@

[@BlackPerl]

Sure, will do

[@ctnguyenvn2178]

@@BlackPerl thank bro.

[@pauloramos40]

nice

[@aseemk1605]

great!

[@BlackPerl]

Thank you

[@wizcactus2223]

Make a video of obj/streams in pdf in Hindi.

[@BlackPerl]

Thanks for the suggestion. But doing video in hindi appears difficult for now.

[@MoonSlayer007]

👍👍👍👍

[@CCap-ir4ik]

Honestly whatever you have shown in the video is not there in the repository. Its dubious and misleading. Sorry to say this.

[@BlackPerl]

Sorry to hear this from you. The repository private codes and steps have been removed to maintain proprietary information.

[@meryemb1324]

ELK isn't 💯 open source