Data Center Rebuild

Decommission of a pair of 6509's. The environment was broken up to Nexus 7004's for the cores, Catalyst 3750's for the end user edge, and Nexus 5k/2k for the server edge.
(update 1/2/2019)
Answering some of my most common critiques up front:
· "That's not a data center" - Not all data centers look like they're in a co-lo space. That room contains all the servers, networking, and other essential gear necessary to run the organization. Because of the equipment it housed we called it the data center.
· "The job isn't finished" - I agree. Unfortunately my career took me elsewhere before I could make more progress.
· "You should have done X instead of Y" - Possibly. The goal of the project was to decommission the old equipment and bring in new equipment. My approach was to bring in different equipment that allowed the environment to be managed more naturally through shorter cables.
I'm certain there are some other critiques. I'll update this over time.
(/update 1/2/2019)
Here is what I was doing at roughly when I was doing it:
00:00 - 01:01 Configuration documentation
01:02 - 02:20 Prepping racks for new user edge switches
02:21 - 04:09 Installing new switches
04:10 - 04:21 Unwrapping new cables
04:22 - 06:56 Cutting over to new switches
06:57 - 09:17 Removing old cables (user edge)
09:18 - 09:41 Removing old cables (server edge)
09:42 - 09:51 Prepping racks for new core
09:52 - 10:27 Relocating new core to new home
10:28 - 11:50 Recabling new core
11:51 - 15:26 Cutting service over to new core
15:27 - 16:31 Prepping to remove old core
16:32 - 16:36 Removing old core
16:37 - 19:14 General cable cleaning
19:15 - 19:39 SAN cut over

[@mattmurphy1065]

It may not have ended up pristine, but as someone who often goes into rooms like this, it’s a hell of a lot better. I can appreciate the work for sure.

[@SystemMTUOne]

Appreciate the kind words!

[@richardwarfield7386]

I do this for a living. As a data contractor the worlds greatest invention of all time - the 1 foot patch cord !

[@BlueDippy]

What’s the pay range

[@theanteaterparadox344]

pry off the insulation, peel back the silver foily thing, carefully remove a tad of the inner silicone insulation to expose the copper soul, attach COAX plug, do the same at the other side, throw cable in box cut another 100 cm length, repeat process about 20 times /hour for 8 hours, no less than 120 lengths/day for 3 months My summer job in 1984....

[@henrythompson7595]

I too, did this for a living, except all router and switch ports were PERMANENTLY wired to 110 blocks on a backboard, all work stations and PBX/phone jack wired to backboard, when changes needed, just change or add a cat5 twisted pair cross connect. Never ever had a rats nest like this again once we started doing this. The data center was very large, several 7000 series Cisco switches and routers. Patch cords not allowed....ever! Our data center had more that one main frame computer, and several hundred file servers.

[@thomasshi9138]

@@henrythompson7595 cat5? In a data center?

[@cyclopsvision6370]

If it was one foot total, then it would have been out of spec. Those short blue cables you see are connected to the 110 block, which I'm assuming lead somewhere else that is longer than 2 feet.

[@dismanca]

Oh the challenges we all face in networking. Too often the wrong lengths of cables are purchased, available and used. Too often there is not enough smart cable management trays both horizontal and vertical. Too often almost anyone is granted access to the room to "just run one patch cable" to connect "X" device. If feel your pain and have spent many long hours doing exactly what this video demonstrates. The best environment I have been inside is an IBM "lights out" datacener. All of the racks where 4 post secure cabinets controlled by RFID card readers and numeric touch pad. If you were not granted access to the cabinet you couldn't open it. If you were granted access it was time controlled and you had to use coloured cables they provided. You were required to labled each end with a number and the number had to be recorded on a chart that told the next person in what exactly that number was for and where it was connected. If you didn't follow those rules all of your work was removed and you had to schedule it again.

[@ananomaus]

Nothing more annoying than "Just one patch cable"

[@daven4005]

> If you didn't follow those rules all of your work was removed and you had to schedule it again. But but that cable was for the CEO's new personal laser printer, it can't have downtime. The company's very existence depends on those documents being printed and posted on time! (/s)

[@davepickering997]

So satisfying watching someone else doing it! Good video

[@officialak47buda63]

Someone would say, Ethernet cables don’t matter what interfaces you use. But with these many, there are Mac tables, which could be so many entries por the table to collapse and causing a failure in network, truly experts these fellas

[@lylestavast7652]

Well Chuck Kelly and Rob Windsor, you did a stellar job for us back in 1997 with something that looked this bad (or worse) and added up-time and stability along the way. Thanks for letting me help as your manager.

[@lowvoltagedude7837]

I love work like this. Normally a job like this we would do over a weekend and do first day of service just in case a few things aren't up monday.

[@tpgoat]

When the video started I thought it was going to be cleaned up by the end, glad I skipped most of the video, it was quit disappointing.

[@guhanv7388]

Me too.

[@Angel-yi5ov]

This comment should be pinned.

[@travislenker9994]

Typical half assed job, likely at a school district. See it every day. Plan accordingly, schedule the maintenance window accordingly and do the job right or gtfo. Don't get me wrong, its a good start but why spend all that effort and not finish. And god, people putting their cores on the floor rubs me the wrong way.

[@trs-80fanclub12]

No pride, just looking for the check. What a way to display your talent. Don't apply for a job in my department. I don't hire slackers

[@lucaslagarto2569]

No one would ever watch this shit without skipping...

[@johnqsak]

At first I thought this was a video of a Italian restaurant showing how they hang dry their homemade noodles.

[@hanro50]

I'll admit... It looks like the Italian restaurant next door had a "accident"

[@notactive9473]

ah, yes. the mystical blue, red, and green spaghet noodles, just like mama mia used to make.

[@jrcogburn]

BWAHAHAHAHAHAAHHA -thank you bro -fukin IT humor made my day better

[@francescomattiussi]

Do Italian restaurants have noodles? In italy we hate them

[@granttaylor1790]

Glad I read your post skiiiiiiiiiip!

[@DJaquithFL]

I own a Data Center .. this is a Server Room. Here's a quote from one of Amazon's engineers, so if for some reason you disagree take it up with them. "A data center is a facility where the entire structure’s function is primarily to house network equipment. A large server room inside a corporate building may have all the same equipment, right down to cooling and electrical, but the purpose of the building is something different and has a large server room inside of it."

[@indeepjable]

So This Is A Data-Center Type Server-Room, Essentially A Mini-Data-Center?

[@FainTMako]

Okay, I missed the part where its standard to build rats nests.

[@Jon.......]

Yes, I did read the FAQ's. Assuming they do use labels ... I hope that there usage is at the opposite end of the quality scale as compared to cable management. I'm guessing this is a colo cage where the actual client visits buts once a year (if that) and simply has vendors and contractors (that both have a high turnover rates) do all of manual work. If it's the onsite data center team "magic hands" crew doing the work, I'm speechless!

[@SystemMTUOne]

Not a colo. On prem at their site. All staff that touched it were direct full time employees.

[@Moppup]

Fantastic job! Looks way better!

[@smilingipad3044]

Every time I eat spaghetti, I arrange it in straight lines because I am inspired by this video.

[@majito0727]

me 2

[@GothGuy885]

before you guys did the cable management and equipment upgrades, this rack looked like "Explosion at the spaghetti factory" great work guys! 😀👍

[@Penfold8]

That paper flapping around on the cabinet side drove me nuts!!!!

[@suharijabon1869]

Wooo...very talented skillfull tehnician, able to fix messy cable in very fast movement 😀😀😀

[@Eclipce051]

Every time I need inspiration I come to this video

[@kerbygator]

Legend has it that the guy's still working on it.

[@JanEringa8k]

Well done guys. It's always satisfying to see a cable monster tamed... At least for a while :) How long did it stay tidy!

[@SystemMTUOne]

Hard to say for sure. I actually left that position before I was totally finished with the clean-up job. The far left rack still had more work to be done. That was supposed to be the internet zone but there were a few other devices in there that I wanted to relocate to the server row, which would have been to my back. A few more months and I would have had that area totally cleaned but sometimes you go where the jobs take you.

[@paulomenezes3400]

Technicians work hard in the data center congratulations

[@vertujoe2886]

My back sores just by looking at it. This guy must have a heavy duty spine.

[@mukinha]

This guy needs a medal!

[@kothaloas2179]

this is who we are and this what we do... you should see telecom termination panels in India. great job... for the people who think it can done better its not so easy as we speak. apparently this person did it without any down time and thats very important otherwise we may require down time of 2 to 3 days.

[@jamesgaudio466]

Exactly!!

[@elb4neado]

No commentary, just perfect

[@Sojuzkarta_82]

Una tarea de mucho tiempo y mas que nada mucha paciencia y empeño, TI de Centro de de Datos ya veo que no es nada fácil, buen trabajo mi estimado.

[@retardedguy9847]

Looks better than the previous cabling. ❤️

[@cheif10thumbs]

How I spent 1993-2003. Doing EXACTLY what's in this video. Intel, HP, Weyerhauser, Boeing, General Dynamics, LANL, LLNL, Pacificorp, USBank, and MANY more........

[@artangjeli]

my impression is that it looks to be a real mess with cables instead of putting all equipments side by side. what kind of work is that?

[@valrabellkeys9867]

I wish there were many more of these.

[@jonakers704]

I have had to deal with similar monsters like this, and I still have one rack that, while I would love to re-cable the lower half of it, we cannot justify the cost of doing so. The problem with that rack is that the guy that installed the cabling used 5m infiniband cables where 1m and 2m cables would have been sufficient, so there is a ton of extra cable just sort of stuffed wherever. Unfortunately the cost of replacing those cables for just sixteen nodes is in the range of $1-2k, and it just isn't worth it.

[@surftec]

Jon Akers I've never understood the logic of such long cables. If it's being lazy when ordering it really makes the job far longer and harder during install.

[@jonakers704]

It was a case of a decision to move the servers to a different rack after we had already received them, and using what we had on hand to wire them up. The distances changed, and the cost for getting the right cables was a bit steep. Since we were able to keep the cables contained within the rack doors, we have basically said it will be there until we retire those nodes.

[@gwhunter07]

Is it not possible to just cut the ends and put new connectors on? Or is that not allowed? While it would be a bit tedious. The cost for a box of RJ-45 connectors is pretty cheap and the crimpers aren't to bad either. Surely alot cheaper than $1-2k

[@dylandrouin3085]

its infiniband, they`re not rj-45 connectors

[@udirt]

FYI: I ordered my IB cables from a store in Rotterdam (major shipping port for north europe?) and they ran like $40 a piece. Cutting out a few intermediaries seems to have helped. Would still leave you at a few $$$ but better. also, the cable routing matters. one place I was at initially had HVD SCSI cables, 25m long, and thicker than IB. There was many dozen of those, and _zero_ mess nonetheless, it was set up by some good techs by EMC^2 and they just spiralled them in the right place.

[@mrrobertwolfiii1079]

Awards for that type of show room's.

[@joemarz2264]

This must be one of the most fascinating jobs that a human being can do!!!

[@FainTMako]

Its really not. Looking at the old cables is depressing for many reasons if you actually take the time to consider how it got to that state, one cable at a time and how that was allowed. And then there is a video of a person "fixing" the problem using pretty much the same paradigms as what started the problem. Its almost impossible to believe people get paid for work like this but they do... Yeah its not fascinating, its like saying wiring all the fuses in a car yourself must be fascinating'

[@aravindsankaranaorist]

Reminded me of the haricut I had after college.

[@Red1Wollip]

May the force be with you......

[@LDrumsOhio]

A master at work.

[@lookatthis4701]

Wow that cable managment do 🤯🤯💥💥💥💥

[@mindasb]

... moms spaghetti, but on the surface he looks calm and ready.

[@sameetpatil_5]

Underrated comment

[@Solidfire]

Holy Cable Monsters!

[@throughthestorm3852]

You forgot to end with, “Batman!”

[@jonasresilva]

Roy? Moss? 😂🤣👏🏼👏🏼👏🏼 Beautiful Job

[@plyzmaz]

Tanto exforço, o video acaba e ainda não tem um fim satisfatório.

[@IgnaceLaplaese]

very satisfying

[@jairparreiras7649]

Cara, esse aí conseguiu fazer mais gambiarra do que eu!🤩

[@lQuadXl]

*Looks like somebody's having spaghetti for dinner in the server room!*

[@wassaa3739]

wtf it's not finish

[@saturn0660]

thats what i was thinking

[@melmels1337]

they left. resigned.

[@AhmedHassan-if3iw]

I want to ask a question , is that the final build of essential server ? if yes , I'm really hate this shape of cables unarranged because it suppose to have with container for every cell, it's really very ugly Synapse cables without arranged, and also it should to hide every cell on it's own container.

[@srish249]

It looks much better. However, it is not done. That would not fly in my MDF...

[@SystemMTUOne]

Sadly I left the company before I was able to truly finish it up. I was able to clean a few more things up after the end of this video, but nothing as large as the initial clean-up.

[@aguyandhiscomputer]

stunod7 Fired for changing the cables from yellow to blue?

[@udirt]

guessing from initial state it was probably for the wiser to just leave than try fighting windmills :-)

[@ClaytonCannon]

What is that tool that you are using on each of the cables during the configuration documentation portion?

[@NeoTheOne41]

What a turnaround

[@marksnexus]

Would have been nice to see it clean and finished

[@ahnilatedahnilated7703]

When this was "done", I think it still needed a lot more wire routing cleanup.

[@SystemMTUOne]

I don't disagree with you. A few more months with the company and I would have been able to polish it up a bit more.

[@QuantumLocksmith]

The Network Masters!

[@aguyandhiscomputer]

Some hedge trimmers would have been good.

[@hamoudanachat6452]

Good job

[@mrrobertwolfiii1079]

Nice really nice team's.

[@rdengin]

One, not knowing what are they doing, at the beginning could think they are fighting with rack or they are being shocked by elektricity.

[@mrrobertwolfiii1079]

Getting really worked with upgrades

[@richarddoran4378]

looks like fun

[@BixbyConsequence]

Pastafarian school of cable management.

[@locklock8845]

Respect! Professional! But normal people never knows.

[@Technosupportdriversolution]

Woooooow wonderful your video guys

[@DanA-yb5uu]

How does one get into the field? I do server desk IT but I’ve grown an interest and curiosity of cable management for servers I would like to know how does one go down this path? Thank you

[@whistlinturbo]

When your parents tell you to clean your room so you make a path through the mess from the door to your bed.

[@mrrobertwolfiii1079]

With clean design

[@theprocrastinator2232]

Saw the comment from 4y ago w/ 1k likes, as the title states... REBUILD.. and not cable management aesthetic video... but clearly the ones who made that comment have no idea what goes around the data center and thats fine... if they only knew what the IT section has to go through even if one switch has to be replaced with a new one.. but yk, no one knows it all, nor anyone is perfect... gotta love it tho, working all those hours and allat, thumbs up and kudos to you!!

[@frankperdomo102]

this video with Benny Hills theme song in the background would have been the bomb xD

[@drawingtheways3601]

LOL

[@CodeAsm]

open this in another tab and enjoy XD kzbin.info/www/bejne/gKnShqCeZ6t2jpY

[@KeivanDjafarizad]

This is my kind of ASMR

[@jaumakis]

give a medal to that man

[@abelmendezuh]

Y Proto HW & Tec mira bro, y tú quejandote por los del gabinete jeje saludos. Excelente gestión de cables.

[@evdiegoth]

Excelente? No terminó

[@BCTAHbKA]

هذا أفضل فيديو على الإطلاق

[@roykeng6976]

we were working on it once.9 persons pulled 2 days and nights efforts on 27 IDF, 4509 x12, 6513 x2. only slept for 6 hours.

[@jonharson]

Ho gawd! I will no longer complain about the state of our server room after having seen this disgusting mess. The giant spaghetti monster is real folks.

[@ThierryC2373]

Even at the end, there is some mess to deal with!!

[@michaelbyrd4004]

It is a shame that this room was this way. I have seen and worked with several functioning disasters like this. I certainly appreciate the cleanup. Thanks for the video.

[@maritodesbordes4883]

This DC makes me feel so comfortable with my out of norm DC

[@SystemMTUOne]

Yeah. If you scroll around you'll see some people giving me a fair amount of hell for this "data center" not being a perfect co-lo environment with properly raised floors, or tidy hidden cables, or any other thing you'd have in a "real data center". Not every environment is ready to appear in advertisements, but they still gotta house equipment.

[@oleksandrlytvyn532]

Good job 👍

[@derekscheltmeyer2992]

Can I come apprentice under you guys? I'm a CCNA, and have a degree in applied science

[@damianadorno890]

is there a part 2 cause it still looks bad but way better

[@marty5300]

Lol that spaghetti monster looks like our network closets at my work. There's nothing like not being able to live 100lbs of patch cables high enough to see if port 34 on blade 7 is active or not.

[@pradeepsingha9733]

Oh they changed the yellow cable to blue??? But why? Are all yellow cables are damaged?(

[@brianlee1012]

Oh~ pasta! I like it!

[@robertheffernan2756]

Prom a practical perspective this is about as good as you could get it cleaning up that mess. It's a damn sight better that it was to begin with but with all those services coming up and down as each patch cable is swapped out there was enough disruption going on. It would take a VERY tolerant employer to allow you to tidy that up any further.

[@jimshaw597]

I don't buy it, nothing more than an excuse. Plenty of time to plan before a core upgrade. Clean the cables up in advance & get the room RIGHT. Reduces time & risk during the upgrade by at least 50%.

[@rodrigoribeiro6632]

If hell exists, it looks like this. And when you're done and look back, the mess comes back and you have to clean up again. (FOREVER)

[@gustavopedro525]

Espero um dia trabalhar com isso :D

[@bg73]

That piece of paper flapping in the wind would have pissed me off in the first 5 minutes. Stick it down! :)

[@SystemMTUOne]

Ha. You know what, it was so loud in there I didn't even notice it except for when I would be walking past.

[@n10cities]

Exactly stunod7, In most data centers you have great big Liebert AC unit(s) roaring, as well as a bunch of blade servers running (sound like jet engines almost), so you can't hear squat.

[@jankockv]

Root password

[@manuelarnoldoaldanaalegria1584]

Que bien ordenados. Exelente

[@jeffmaloughney3485]

I thought that was all SM fibre... Wow good job

[@fredflinstoner9530]

Wow, way better

[@J-TrainRivas]

Nailed it!

[@Angel-VTek]

what kind of engineer do this type of working? electronic and telecommunications? or systems administrator? I'd like to get a job like the one in this video, it's so relaxing for me and enjoyable

[@Angel-VTek]

@maverickM249 i never said it was easy. I mean, it's maybe hard in a certain way, but i like it, doesnt matter how hard it is

[@frietpan8917]

Looks like the cable management in my pc.

[@charadremur333]

Which one?

[@charadremur333]

Oh nvm

[@marshallshelton6262]

Gotta love still having to wear "business casual" while doing essentially manual work inside a data center.

[@daltoniks]

what' that stuff that they holding in their hands?

[@MusicWorldNoCopyright]

How many days will take complete this process

[@PoTTerBeanBagHa]

I have a few questions about a gig like this. What is the job title? Or are you self employed? What are the hours like, average? What kinda money do you make from a job like this? I too would be interested in doing something like this on a professional level. Possibly part time? I have a lot of free time and only work 3-4 days a week. I’m also a night owl so this would be perfect. I have been working with my church for about two years now, all voluntarily of course and making some extra income would be great.

[@joshllena9112]

Awesome

[@augurseer]

Certainly cleaner. Been there done that.

[@paulmoadibe9321]

been there, done that

[@tornadokat]

Me too for single racks at multiple locations. It was like pulling teeth to convince them they needed to spend $400 or $500 per rack for half foot and one foot cables to fix it , even after I told them they pay us more than $500 in time wasted tracking down cables (they would activate and deactivate wall jacks as people moved around because they would not buy enough switches to make every jack live ). They also did not want unused jacks live for security reasons. It would have been much more time efficient to have all jacks connected and just activate/deactivate ports on the switches remotely which is even better when a tech is sick or on vacation and another tech in another city is covering.

[@n10cities]

Same here, not a fun chore that is for sure. In my old Data Center gig, we always used pre-made patch cabling so you had to guess what length cord you needed. Sometimes the server was two cabinets away from the switch and other times the server was across the room! I was lazy, so I always erred on using a cable that was too long rather than too short and having to pull it back out and try another one. And if you didn't use snagless cables, oh Jesus help me now! Would have been nice if we could have terminated the cabling ourselves and avoid the rat's nest that this guy started with.

[@cokeacolasucks]

An ex co-worker I had used to cable EVERYTHING with 7' patch cables. Upon asking why, he responded "because that's what we've always done." Thinking was not his strong point...

[@majstealth]

you are here to work, not to think! god damned jimmy, learn that already....

[@robertpurvis6106]

Lol. I don't even discuss with the clients anymore how they want the layout in the rack. I just install patch panel, space for switch, patch panel, repeat. Leave a box of 300 6 inch patch cables for the IT person. Leave slack in the rack management in case they want to change the layout. Everyone just goes with my plan in the end. It makes to much sense.

[@jankockv]

Thats culture, tha all ways done...

[@rhdtv2002]

Try 10'. I have about 120 of them that I need to RIP apart

[@mikeE0055]

Just the indication of the air flow in there make my ears ring.

[@duniaintermezzo]

Wooww.. Amazing.. 😯😯😯