Port Mirroring

[@clabretro]

mxshift over in The Serial Port Discord made a great observation: "cache engine using a FD:FD:10 MAC address seems wrong. That's an unallocated OUI and marked as multicast. That feels like an autogenerated MAC used when the NVRAM is trashed. That would also explain the 'is a .' as the model designation would also be in NVRAM" Perhaps some corrupt NVRAM? More investigation to do on the Cache Engine for sure!

[@RaStrNL]

That's exactly what I'v wanted to tell you. You must fix your NVRAM or CMOS RAM or whatever is in there. Arp has no response because of bad source address and then nothing else could work. But you have to fix that NVRAM issue first since otherwise it will never boot from anything if there is no model number stored there.

[@jfbeam]

Not NVRAM... IDPROM. And that will be "impossible" to fix. This is obvious from it's lack of hardware identification. Most of their things made in that old PIX form factor use an I2C EEPROM (flash) for the hardware details. It's either failed, or something scrambled it.

[@AureliusR]

@@jfbeam How do you know it's EEPROM and not a battery-backed SRAM, for example?

[@moffix]

Most cisco gear used sdcard for flash. Does not look like this cache engine has one. The chip might be cooked.

[@WedgeStratos]

I'm reminded of the Sun Microsystems issue. Can't even boot the machine until you've run through safe boot processes for an agonizing 2 minutes, then you get to tell the machine what it is and what it's MAC is, before you can finally try to boot. And that's before you discover that the replacement battery costs $100 both new or "gently discharged"

[@andyg8888]

I’ve been unemployed and depressed for 7 months now. Your videos bring me unspeakable joy. Thank you Mr. Retro.

[@Thiesi]

It's incredible how much KZbin(rs) can uplift you, right?

[@1nico517]

I'm out of work due to chronic pain from an injury years ago. I'm dealing with my depression through the company of my pets and a few KZbinrs whose videos brighten my day whenever they upload. Every time I see one of clabretros videos I think "ah just what I need" no hyperactive editing,clear and thoughtful communication of information and an obvious true love for what he's doing. Adrian's digital basement and the backlogs (very different channel topic) are my comfortable places on the internet, amongst a few others. I wish I was in a financial position to be supportive of the people who provide all of this to us, maybe one day, but I hope our engagement and joy for their content can be enough to keep them doing what they do for a long time to come. Cheers to you all, take care and happy holidays.

[@cmfrancis1]

Hang in there guys, better days are ahead.

[@FI-TECH-250]

its currently like midnight in germany, and this is exacly what i needed. thank you.

[@kazuyachan8212]

Same here lol, watching this in bed XD

[@Moretah]

Ye 23:53 In czechia RN and just got home from party. Just the content I need to fill rest of my friday night

[@Kabodanki]

A huge terr attack just hapened in germany

[@twei__]

​@@KabodankiI wouldn't call it huge, still horrible though

[@FI-TECH-250]

@@Kabodanki Where???? How tf do i live in germany and not know any of this????

[@gmailpson]

Working at a mixed vendor ISP back in ancient times we had quite a few problems when connecting Ciscos to other switch vendors. Mostly this was solved by disabling auto-negotiation on switchports and set them hard to 100/100 FDX or 1000/1000 FDX (newer equipment). Mostly we needed this when adding HP or 3Com switches. Sometimes we needed to do the same for some 3rd party server equipment too. That could be the problem with old Cisco talking to your Ubiquti switch.

[@uendarkarplips7263]

With these old machines, I always set the time and check for any batteries that may have dried up decades ago. You never know how a machine will behave if its time is wrong.

[@jfbeam]

"portfast"... I'm pretty sure the cache engine is not going to wait 30s for spanning-tree to complete loop detection. It's a VERY common issue. Watch the front of the switch. The light will flash orange while it's blocked. Until it goes green, no traffic will be forwarded.

[@clabretro]

very good observation, someone else mentioned this as well! i'm definitely going to play around with that

[@ax14pz107]

Yeah most likely. Portfast and bpdu guard all the client facing ports. It's super annoying and some devices really really hate it. Edit: after finishing watching the video, it's not the port being down because then you wouldn't see any packets with the port span.

[@fcosmin327]

Yes! Portfast! . Seeing the ARP requests and no response I thought "the problem is the switch" Might be the agonizingly long STP wait time on the cisco switch. This brings me back to my CCNA days, some guy accidentally whipped a class router and the teacher assigned us to bootstrap it and install it over the network. Long story short, we wasted hours banging our heads trying to figure it out. It was the STP...

[@Consequator]

I'm so used to immediately turning on portfast on switches that I managed that I totally forgot about this one lol

[@cfabz2023]

@clabretro at 21:08 you can see the LED as orange, which on that era of Cisco switches means that spanning tree hasn't gone fully active yet and is not passing traffic. It then switches to green a few second later. I would definitely want to rule that out. It would explain why the ARP requests not receiving a response. I would set the interface parameters on both the device and monitor port to "switchport mode access" and then "spanning-tree portfast"

[@JTrickZ]

ITS NOT EVEN BEEN 7 WHOLE DAYS MY HEART CANT TAKE THIS MUCH CLABRETRO!!

[@MrSunDevil23]

Oh those big GBIC slots takes me back!!! Thanks for the memories.

[@dodegkr]

I recall buying big blue non Cisco ones for my home Cisco meddling back in 2003 together with sun sparc happy meal cards

[@itsnik4705]

Its really intersting learning about hardware from when i wasnt even planned, keep up the great work!!!

[@bududdy]

Love seeing all the older equipment being setup and configured! Great job as always!

[@_vilepenguin]

"Come to bed Clab!" "Be up soon Honey, I am port mirroring." :)

[@Deraco1]

LOLZ!

[@kopspijker3515]

Some uses for port mirroring is for IDS systems. Intrusion Detection System. Some examples at work are monitoring industrial networks. Actively doing stuff in those networks can be very dangerous. From monetary, lost production, to life threatening, things going sideways in the field.

[@nilswegner2881]

Exactly. Vectra is one of these systems. Absolute amazing to see what happens across a large company network.

[@spvillano]

We used it for IDS mostly, DoD used it for both IDS and full packet capturing of traffic. We also used it during a campus wide network issue, around 150 C2950's hanging off of a pair of C6509's that were fully populated and a C4506, for that one, turned out to be a misconfiguration by the boss that triggered a spanning tree storm base wide. Of course, taking longer to fix than to diagnose.

[@renatofp]

By the way, I bought 2 Cisco WAVEs.. they have a VGA connector hidden under the disks. They will be 2 excellent machines for me to install VMWARE ESXi (and cheap, here they ask for 80 dollars or less)... As a cache, I have no way to test them, they are unlicensed. Thank goodness sellers haven't discovered that these machines work VERY WELL as a PFSENSE/OPNSENSE firewall or virtualizer, for laboratories.

[@clabretro]

very interesting! let me know how they work out

[@nated4wgy]

Love this Cisco stuff! Keep it up, I’ll keep watching!

[@clabretro]

glad to hear it!

[@ergosteur]

Wow I had no idea those 2950s came in 12 port but full 19” width. So weird seeing that huge blank space between the RJ45s and GBICs

[@DanielTekmyster]

Love it!! I can watch your retro videos for hours on end!

[@legarishere]

Never missed a single clabretro vid since March 🔥🔥

[@clabretro]

niiiice

[@yannickberrios]

Yeah!!! a new clabretro video, watching while coming home from work

[@subloc]

It’s cool to watch the skills sharpen.

[@clabretro]

🔪

[@stealth210]

In later Cisco switches, monitoring was/is? called port spanning or span. Not sure what it is today. Just FYI. Used it a lot back in the 2000s

[@ToxicwasteProductions]

I need to try portmirroring on my cisco gear to. Looks like fun

[@LeeZhiWei8219]

Man dude. Your troubleshooting videos are awesome man! You're so cool! Awesome job again dude!

[@clabretro]

thank you!

[@nicholascraig8349]

Great content as always brother!

[@clabretro]

thank you!

[@ergosteur]

WCCP oh boy, flashbacks to setting up Websense 7.5 with a Catalyst 6509.

[@jdkap201]

Never heard of port mirroring before, but it seems like it turns part of your switch into a hub of sorts ^^ On a different note, when playing with older gear that didn't behave nicely with modern network gear I'd grab an old laptop and connect it directly for tftp. The laptop itself has Debian 12 on it, but I can chroot into a Debian Etch environment and start a tftp server from there. I really hope for you to be able to get the cache running again, despite the probable NVRAM issue. Wishing you best of luck :)

[@der.Schtefan]

Cool feature. In lieu of such a thing, you can also simply get a cheap 100mbits rated HUB from 1997, and connect sniffer and Device under Test to it. You will have the same effect.

[@OzeCovers]

Let’s go!!!! 6am here, but it don’t matter.

[@bfg_9001]

YEEES A NEW CLAB VIDEO FOR MY SATURDAY

[@TeeEllohwhydee]

Ah back in the days before HTTPS was a thing. Good times, good times.

[@ora2j251]

You should put the image on internet archive if it was hard to find.

[@criggie]

Try pinging .40 from .200. Run tcpdump on .200 while the cache boots, make sure you see an ARP reply being sent. Check your 2950 config for any vlan config left over - arp is a layer 2 function, which sits "beside" vlans. A "factory reset" may not have cleared them. Also check that portfast is turned on, or that spanning tree is turned off. The standard delay is 30 or 45 seconds from memory.

[@evertp]

Even if I'm not super into the topic I enjoy your reverse-valley-girl cadence if that makes sense

[@clabretro]

I take that as a compliment

[@evertp]

@@clabretro it was!

[@Consequator]

Port mirroring is super useful. You can even do this with virtual switches. This issue is rather sketchy though lol You have a link between the switch and the cache device, but then the uplink to the other network goes down ? You get no responses to ARP requests coming from the cache. You can tell that it doesn't work in wireshark because there was no response :P I am unfamiliar with the cache device from cisco although I have used cache devices in the past. There is a chance the 5500 is acting as a switch and your home lan is going 'nope' when it sees another switch being plugged in to a lead switch, this is BPDU guard or root guard I think it was called. That might explain the uplink to the network going down. But this does not always make the link light go off. Open up a telnet to the switch and turn on console logging before turning on the rest to get a sense of what that's doing when the link becomes active. Another thing that might explain this is vlan configuration but I don't think you have those configured ? I would expect to see vlan oriented broadcasts. It's also possible that something created an ACL on the switch ? Like a default config thing ? Then there's you saying the mac address isn't right probably, is it possible the cache device is actually binding to fastethernet1 instead of 0 ? That's the only thing I can think of that would cause that beyond some sort of hard coded thing or there being a boot menu item on the cache device that has been misconfigured. In regards to the invisible cache, the way this used to work is that on the gateway (L3 or NAT) you would use traffic rules to intercept port 80 traffic and redirect it to the cache server which then just acts like a regular proxy server and serves its own content or fetches it from the internet. Port 443 gets a little more tricky due to certificates and 'you're basically doing a man in the middle attack on your own network'. It's been a while since I messed with it but the new L4 and up firewalls might be able to do encrypted traffic. If you want a laugh on this topic, look up 'Upside down ternet' which is basically this concept in a nutshell but then to taunt people stealing your wifi🙂 (edit) And the portfast thing like the other guys said in the comments, I totally forgot about that as it's one of the first things I turn on and then forget about lol

[@RedShift5]

Enable spanning-tree portfast on the port that's going to the cache server. You can see it's not forwarding traffic yet when the port LED is orange.

[@MattArmstrong-c9m]

Clabretro, I went down the SAME rabbit hole trying to get a VOIP system set up. In order for your TFTP to work you need to have a DHCP server running on the same box as the TFTP. It doesn't matter that you've hard coded an ip address into the Cisco box. Something has to tell the Cisco box where the TFTP server is. In the DHCP config you will need an entry for your TFTP address. With the two working together, your ARPs will be answered and the Cisco box should be able to contact the TFTP box. You *might* have to stuff your DHCP server address into the gateway section on the Cisco box ... only experimenting will tell. GOOD LUCK ... and great content as always!!

[@mattym8]

This doesn’t make sense. He is specifying the tftp server. Why do you think the cache box will use tftp config from dhcp ? Because your voip phone did? I don’t follow.

[@MattArmstrong-c9m]

@@mattym8 I know it sounds odd, but when I was setting up the VOIP system the only way to get the ARPs responded to was to also set up a DHCP server. Once DHCP was set up and running it responded to the ""who has xx.xx.xx.xx" requests. I scratched my head for days before trying this ... and it worked.

[@clabretro]

Interesting! Certainly no harm in trying that out

[@ickipoo]

DHCP is NOT necessary for TFTP. You can absolutely TFTP without DHCP. What DHCP gives you (aside from IP config) is extensions to specify stuff like the initial TFTP boot image name. (TFTP is a much older protocol than DHCP).

[@andreashammargren751]

I think you are thinking about booting from a tftp image automatically, like PXE-ish. Then you need a set of DHCP/BOOTP options set to point the machine to the correct source to find the image, but that is not necessary when you just want to store/get a file from a tftp like in the video.

[@2008spoonman]

@4:39 It is called : apipa 😊

[@smpstech]

I have an OS/2 Warp 4 pc that I have been having a hard time getting to connect to my network. I'll have to try mirroring to see what its actually doing!

[@andresbravo2003]

Quite needed to get the cables working.

[@SilentServiceCode]

Your port Fe12 to your other network shut down probably because it's access by default and received a BPDU from your Ubiquiti gear and shutdown to prevent a loop

[@I_Am_Your_Problem]

Such occurrences are logged. Not the issue.

[@seanrikard3163]

Those old 2950's did not have auto mdx - You need a cross over ethernet cable between that switch and your modern network equipment.

[@wardtj]

@clabretro Duplex. That's 100M. It could be a duplex issue. In the olden golden days, 100M never did autonegociate very well. If you try forcing both sides to 100M full, see if that solves it. A common symptom of duplex mismatch is that you can send, but, you cannot receive.

[@clabretro]

good idea!

[@locusf2]

There was a list of flags in the boot message that the firmware can be configured to use, did you try setting the flags along the boot message instructions?

[@JMassengill]

Have you tried a crossover cable between the switch and cache?

[@SilentServiceCode]

Not since like 1994

[@seanlynchbp]

With some of the Cisco systems I have found that I have to rename the file it's looking for to what the device has on it. Try that and see what happens.~bp

[@samuelhulme8347]

Doesn’t the TFTP server need to ARP to find the MAC address for the cache engine to?

[@TomStorey96]

Two things I always have on my work laptop: serial terminal software and Wireshark Indispensable, both of them!

[@KelvinMiddleton]

Love the channel Clab! Noob question, why cant the cache just be directly connected to your workstation and you wire shark it that way, why do you need the switch inbetween?

[@clabretro]

it's more fun this way 😂

[@KelvinMiddleton]

😂😂😂 love it ​@clabretro

[@studioxxswe]

tftp is really tricky, not sure how many times I've used port mirroring for that. normally you get exactly these kinds of strange error messages. having said that did you try the other ethernet port (1) - I guess the ports should somehow be configured for what side of the caching you have it connected. how does the inside looks like?

[@djtomoy]

have you tried turning it off and on again? that normally fixes most stuff

[@Luzgar]

What do you have against maximising windows ?

[@JonneBackhaus]

Isnt the eth0 the "wan" port and eth1 the "lan" port. Why would it try to fetch tftp stuff from the "wan" port

[@jwSawmillNL]

Whats the difference with just running wireshark on the network?

[@racingmars]

Network switches only send packets out ports that need it (they keep a MAC address table of every MAC address they've seen on every switch port, and only send packets addressed to a particular MAC address out the port that it knows that recipient is on), except for broadcast packets. So let's say you have computers plugged into ports 1, 2, and 3 of a switch. If you run Wireshark on computer 1, it will not see the traffic between computers 2 and 3 because the switch knows it doesn't need to send the packets to computer 1. With port mirroring, you are able to tell the switch to send all of those packets to the port computer 1 is plugged into. Back before switches were common, ethernet hubs repeated *all* packets on *all* ports, so you didn't need port mirroring. This was closer to how original Ethernet (where every computer was connected to a single shared wire) worked, where all devices saw all traffic and could collide with each other. Overall network throughput potential was way lower since only one computer could send a packet at a time, and there was all sort of collision detection and backoff and retry stuff to try to coordinate sharing a single collision domain.

[@jwSawmillNL]

@ Thank you for the clear answer!

[@deadreaver666]

Hey Clab, when you show the "New FTP Server" at 22:05, is that an old "U3 Enabled" Geeksquad USB drive in the back? If it is, I haven't seen one of those in forever! Were you an Agent back in the day? Also, it LOOKS like its not getting ARP response like you suspected. Is the MAC address of your server FF:FF:FF:FF:FF? If so, its likely causing an issue. Per Google LABS "A Cisco device cannot resolve an ARP request when the MAC address is set to "FF:FF:FF:FF:FF:FF" because this is the broadcast MAC address, meaning it sends the ARP request to every device on the network, and no single device will respond as the intended target, causing the ARP resolution to fail."

[@ax14pz107]

I'm pretty sure the default Mac address for arp is the broadcast address of all Fs so it would be weird for Cisco devices to drop them unless they were configured to do that

[@deadreaver666]

@ax14pz107 I couldn't tell if it was a broadcast or if it was reaching out to his server using that Mac address. I kind of hoped it was just broadcast but couldn't be bothered to rewind and scour the video LOL

[@ax14pz107]

@@deadreaver666 lol I haven't watched the full video yet so I'm not sure either. On the initial arp frame the dst should be broadcast but the src needs to be the device.

[@mikehensley78]

i wonder.... if that cache worked would it cache HTTPS or only HTTP??

[@clabretro]

I'd imagine HTTP

[@thelockzgaming]

just a dumb thought, Try a / in front of of the file name when attempting to boot from net.

[@stonent]

booting from ata0 indicates it's trying to reach an IDE device, maybe a CF card inside?

[@morofry]

Sanity check: are the ports you're attached to adminstratively down?

[@fo0barDE]

Is your stack of linksys devices growing from video to video?

[@clabretro]

occasionally lol

[@halitimes2]

Does that thing require RARP (reverse ARP)?

[@betonmischer_86]

One problem was already found, but otherwise my hunch would be to port-mirror the device on the other end too.

[@knightcrusader]

Port mirroring is cool, but I'm surprised you didn't go even more old school and just use a hub.

[@jakint0sh]

Hey uhh… are you sure that the traffic coming from the cache is reaching any other ports on the switch? I’d recommend checking by firing up wireshark on that linux machine that’s acting as a TFTP server as well, see if you have any incoming ARP traffic. If not, the cisco network switch is causing a problem.

[@UpLateGeek]

What a coincidence, just the other day I used a monitor session to prove to a service provider that they setup a new link incorrectly. But it wasn't using another port as the destination. We just got a new 10G interstate link between a new office and our head office, I was seeing packets sent and received, but it couldn't resolve an ARP for the router at the other end of the link, same for the other end. I figured there must be something wrong with the packets they're sending, but what do you do when you've got a 10G link that you have to capture the traffic from, but no device with a 10G port to mirror to and capture? Easy, you set the destination to a file! Obviously you can't do it on these old switches, and it's mostly just supported on more modern Cisco router platforms. However, it is supported on the NCS 540 router that I was using. So I configured a monitor span session with the source set to the port with the new link and the destination set to a file, started the session and pinged from the other end of the link, then stopped the session and downloaded the file. Opening it in wireshark, lo and behold I see the packets have an 802.1q header with a VLAN ID, they were tagging the packets with a VLAN ID, even though it was supposed to be untagged. Once I told them what I saw, eventually they figured out where the extra tagging was coming from and fixed it, and then I could ping from the router at both ends of the link to the one on the other side. And that's why the concept of port mirroring is still an important and useful tool for troubleshooting, even so many years after the gear you're using it on came out.

[@slawekkrolik]

U could use any hub and you would get same result without any config

[@slawekkrolik]

Also u could setup bridge on 2network interfaces on Linux and would get same 😊result without

[@buba1234]

isn't that a hub with extra steps?

[@poweredbyWatts]

Return of the Spudger….

[@MetaversaicCat]

Cache Cache Money

[@majoryoshi]

I think part of your problem was that ports Fa0/11 and Fa0/12 seemed notably absent on your switch when you ran `show ip int brief` at 2:31 . Even if the local tftp server errored out, that certainly wouldn’t have helped things

[@nathangreer4685]

He stopped the printout of the sho command before it got to 11 and 12. The port could have been shutdown or he may need a crossover cable. Gigabit can auto-cross but 10/100 ports did (do?) not.

[@ax14pz107]

​@@nathangreer4685I think it depends on the switch. Older ones don't but newer ones I think have auto mdix.

[@majoryoshi]

@@nathangreer4685 from my experience, it’s not so much if 10/100 ports support MDIX as it does if the actual switch does. 2960 and 3750 switches definitely do, I haven’t worked with Cisco switches older than that, but I’m gonna hazard a guess and say that a cross over would’ve helped here

[@SilentServiceCode]

If you listen he broke out of it

[@bborkzilla]

These days caching isn't much use when everything uses TLS.

[@hokuspokus8570]

Ciekawe czemu zabrakło rekordzisty na 1 miejscu: Bernie Madoff ?

[@progdotgay]

me, having just finished a comptia network+ and a+ class: "hey, i've heard of this!"

[@RT-qd8yl]

I'm just starting mine at 36, wish me luck 🫡

[@ToxicwasteProductions]

12:09 errno 🤣🤣🤣 cisco your drunk my guy...

[@doq]

"errno" is a common shortening for "error number". Not strictly a Cisco-ism and you can see this term used widely in the software world.

[@BurkenProductions]

well everyone knows how to setup port mirroring on simple cisco like that.

[@donwald3436]

Just use a hub lol.

[@Jerrec]

Finally Linux again. Cant stand that crappy Windows machine! ;-) Reminds me of the old videos.

[@Ski4974]

What? 25 mins and no solution 😭