[Panel] Sarah troubleshoots Incoming #30

Play along at home as Sarah works out some bugs in Incoming selector #30. This video is longer than usual, as I wanted to leave the camera rolling as much as possible.
One of the issues I have with myself is that I want to be a perfectionist and release really well-made professional videos, but I don't have the money or the time to do that. Instead, here's something that's a little rough around the edges, but is hopefully satisfying. You'll see me screw up, make mistakes, ramble a little bit, but it eventually works out in the end :)
P.S. Here's a scan that explains how sequence switches work, and what the symbols on the drawings mean: archive.org/st...

[@artnavan6860]

I'm a former ESS switchman for N.Y. Telephone. I've seen many retired panel offices, but never a working one. This is very interesting.

[@kenunix1863]

Us Panel guys used to hate ESS because when a call would fail ESS would put a shunt on the trunk. That caused the GCO incoming to constantly cycle to go to tell tail downdrive and do it again. This caused excessive ware on the clutches and cork roller ..

[@americanspirit8932]

I work for Western Electric starting in February 1963, worked on installation of number one ESS, in New York Long Island, Brooklyn and Manhattan. Also number four ESS, tsps, number 5 ESS. Retired after 36 years service. Today September 27th 2022

[@Laguna_Jim]

You MUST visit the museum in-person some day, unless in retirement you really hate CO's. They are on the top two floors of a working CO in Seattle, and they get power and talk-batt from the working CO downstairs. They have Step, Panel, CrossBar 1, CrossBar 5, and a 3ESS. ALL ARE UP and completing calls :) They even have a computer-based application that randomly places test calls every few seconds on all the switches using a very nifty digital-to-analog multiplexer setup. There is a video of the cold startup sequence of the 3ESS that is truly amazing.. kzbin.info/www/bejne/oWmZZmCMn7yIoZo

[@stuffnva]

I still remember a trouble with a sequence switch almost 50 years later. A wire clipping embedded in the bear grease was causing a trunk failure in the #1ESS where I worked. Also, we used to have a trouble clearance code "CBM" - cleared by magic. Another was CCWT - came clear while testing.

[@kenunix1863]

I remember those and TWA Trouble Went Away.

[@ElmerCat]

Sarah, it is such a joy to see your devotion to this precious Panel switch. You give it life, you resurrect its soul, you are its savior. You are truly an angel, Sarah!

[@stevegeist3798]

@ConnectionsMuseum Maybe some day you could do a video focused on the tea wagons? What's in them, their capabilities, how they actually connect to a switch, what's different from this one to the one with the meters that you recently "re-tired" etc...

[@mfbfreak]

Fault finding is one of the most fun and satisfying things in electronics. Sometimes frustrating, but in the end it feels so good to have found it!

[@Bata.andrei]

Those old machines have soul! Not like the digital crap from the present. These videos bring me back to my highschool( telecommunications specialty highschool) years when I was practicing on the school's 50 lines cross bar switching system. I loved sitting and watching it in action...Love your channel!

[@t13fox67]

Reminds me of my bell system SxS cama system trouble shooting days. Very happy memories. Thank you so much, this was very interesting and gratifying.

[@graemedavidson499]

I'm in awe at the ingenuity of the engineers that designed this electromechanical marvel. Fault finding was much more satisfying back then, tracing the issue logically down to an individual part. Well done on tracing the fault!

[@TheSonicfrog]

This is such amazingly primal electro/mechanical technology, not a single tube, transistor, diode, nor integrated circuit. Great work troubleshooting by the way. Geez, the though of designing, engineering, manufacturing, installing, and documenting these amazing machines makes my head hurt. The scan you linked to is awesome! And navigating those docs and schematics ... wow! Also, first step in the official troubleshooting decision tree: did you mess with it yes/no? If yes, are you an idiot? etc ... and oh yeah, when all else fails, look at the diagram (RTFM in my troubleshooting vocabulary).

[@therealxunil2]

The more I see of telephone switching gear, the more I realize just how intelligent the people were who designed this stuff.

[@billmoran3812]

I enjoy troubleshooting and especially when doing it with someone of similar mindset. You would be fun to work with. Your thought process is very much like my own. I found myself talking to the screen during this video! I hope you do more troubleshooting videos.

[@michaeltidbury4835]

When I was a business analyst we used to use a methodology called SSADM which used states as a key component for designing old fashioned computer batch programs in Pl/1 or COBOL. I was taken back to them by your sequence switches 👍.

[@kenunix1863]

Yes, and COBOL is still in use !

[@wackyvorlon]

This video was excellent! I suspect that wear from continued working probably rubbed away enough crud from the contact to get it behaving.

[@therealchayd]

Sarah: "*incomprehensible-telephony-speak*...just like a car's camshaft..." Me: "Ah...now you're speaking my language!"

[@TufolphWritten]

Ppp

[@jessicam.4148]

Really interesting video, nice to see the details of how the switch works, and is supposed to work, and I really enjoy the real world troubleshooting, and your explanations of things. Please keep the videos coming!

[@JeffFrmJoisey]

I've liked, done some basic playing with phones back in the 70s. I have only have a very basic of idea how these switch machines work, but found this fascinating to watch you troubleshoot the old way. I suppose I never considered the effort that went in to designing them on paper before assembling them, right on through to maintaining them.

[@kenunix1863]

At 38:05 love that tone bar. Haven't heard that in a looooong time.

[@michaeltidbury4835]

I have only just found your channel. This was my first video and I found it fascinating. I have my own PAXes and know how satisfying it is to sort out a problem. I find the concept of the panel system difficult to follow but with your help I am getting here. Thanks 😉

[@johnpeterson7264]

You folks are fantastic ! I love what you’re doing !

[@shaunbc9279]

Fascinating, loved watching this video and appreciate the time taken to record and publish all your videos. I will be in Seattle in early October so maybe I’ll get to see the panel switch in real life, Strowger dominated the UK.

[@ds99]

Very interesting. I think your sanding you did likely got rid of the ghost. One thing you need is lots of patience for this stuff. I used to take violin lessons in the early 1970s and still have the violin and rosin even though I don’t play anymore. Thanks for the video. Please make some more. 😀

[@GDJason]

This was a great video. Good troubleshooting, it can be pretty satisfying when working on this kind of equipment and really having to trace out and understand the system to find the problem, even when it involves working out the "ghosts"

[@josugambee3701]

Funny - starting at about 14:07 I can hear a shaft squeaking. Evan Doorbell actually mentions this phenomenon in this recording (www.evan-doorbell.com/production/GE9_1.mp3 - at 10:35) and you can hear the squeaking in some of the phone tapes.

[@ConnectionsMuseum]

Josu Gambee Yep! I’ve heard that tape, and he’s exactly right. The vertical shafts get cranky sometimes and you have to fiddle with the bearings to quiet them down.

[@marvinrowen5894]

When i worked on ibm computers they said you never say bend it. You just made an adjustment Good job

[@bradmad8346]

Very impressive, I love to see a tech st work, the problem the process, the final repair, just doesn't get any better. Thank You, I always like these videos..

[@PiotrEsdenTempski]

Great video! Thank you for putting it together. Really enjoyed learning more about the system and the "ghosts". Looking forward to more. :D

[@ÐÞæ]

Good troubleshooting! Thanks for sharing this with us Sarah :-)

[@gregaluise5727]

👍 the the “thumbs up Sarah” photo stuck on the equipment frame in the background.

[@CSXRailfan675]

Awesome job on figuring this out. I especially like the "tea cart" that looks cool. Something MA Bell made? Or someone else's idea. Keep up the cool vids, very informative. I like switch gear.

[@tonyd6853]

Do you work in the 1900's?

[@compu85]

Did you manage to get a wire pair strung from the museum to your home?

[@edgeeffect]

Is that diagram glued to apiece of wood???? That's kinda "olden times" ;)

[@DandyDon1]

Maybe it wasn't so much as a "ghost in the machine"> It may have required a few cycles for the sanded brush to seat/burnish against the cam, or for the residual filings to clear away.

[@DandyDon1]

...and yes whenever time permits, a trouble shooting video like this is great.

[@ConnectionsMuseum]

I agree, that's almost certainly what happened. I still call it "ghosts" though, just because it feels so appropriate to think of it in that kind of esoteric, magical framework!

[@MatthijsvanDuin]

6:15 Even with the diagram linked in the video description it took me a bit to understand how to read this.... it's really unintuitive that the A cam's outer contact is labeled with its _open_ states while the contacts of all other cams are labeled with their _closed_ states.

[@ConnectionsMuseum]

I agree. It's not something that a reader would assume without any prior knowledge of their schematic representation. Also, it always messes with me that a slash (as in 2/5) represents "closed in 2 through 5 inclusive" while a dash (as in 2-5) represents "closed in 2 and 5 only".

[@MatthijsvanDuin]

@@ConnectionsMuseum Yeah that's also an interesting choice of notation.

[@charlesdorval394]

Wonderful video Sarah! It was great seeing you go through the whole process I was wondering, on that schematic you looked at, some of the numbers had fractions... why is that? multiple contacts?

[@ConnectionsMuseum]

Charles Thatisall those numbers represent the position the sequence switch is in at a particular time. So it rotates a full 360 degrees and has 18 positions it can stop in. Each position occupies 20 degrees of arc where the springs can be in contact with a cam cutting. The fractions are...fractional positions! So 5 and 1/2 means “closed in position 5 and also the first half of position 6”. Functionally, it’s used to allow one part of the circuit to make or break early or late, if required. Hope that helped a little :)

[@charlesdorval394]

I have to admit I had to read it a couple times lol Makes perfect sense :) thanks!

[@ConnectionsMuseum]

In case anyone else reads this and has more questions, here's a visual diagram of what we're talking about: archive.org/stream/a-132_panel-dial-systems/panel-book#page/n26/mode/1up (Figure 3: Illustration of Cam Cuttings and Circuit Conventions)

[@charlesdorval394]

Wow, amazing, thanks! I have to admit this stuff absolutely blows my mind

[@SeekingTheLoveThatGodMeans7648]

What a racket in the room... servicing this equipment must have been an arduous task.

[@Mr89netrom]

And I thought Industrial control was complicated :O, I can read modern drawings, but these are a whole different thing. As the symbols are way diffrent, Not only being way over 70 years old, but also bein american, as a europeian I dont get this as easy. I am too young to experience the simplicity, but at the same time complexety of the sequencer. But In PLC programming, the simpler way to program them is just the same. You index one instruction after another, And you can skip and jump in between tasks in certain situations.

[@CentiZen]

Hey Sarah, what's up with the strange spacing of the button panel on the Tea cart? 5:00 for example. Some of them seem to be much closer to each other. Is one bank out of alignment or was it designed that way?

[@ConnectionsMuseum]

Hm, I think you may be referring to the block of push-keys in the middle of the cart? Those stay depressed until you hit another key, so they're not actually spaced different. They're just pushed in, and the angle makes it look weird.

[@roysmith5902]

A comment and a question. The comment is that as amazingly wonderful as this stuff is, what's really mind-blowing is how failure resistant it is. You've got a state machine that's doing the wrong thing, and while obviously calls that get routed through that don't get processed correctly, the rest of the switch just keeps on chunking along. If I had corrupted memory in a modern digital computer, eventually, I'd end up with the whole machine crashing as pointers got trashed and the damage just kept spreading. The question has to do with the nomenclature. It get that "1/8" means "positions 1 through 8". So, why at 7:13, just to the right of your finger, does it say "1-2-3-4-5-6-7-8-9" instead of just "1/9"? And even more confusing, a little above that, what's "11/11-1/4" Does that mean a quarter of the way past position 11?

[@ConnectionsMuseum]

Ah, good question! This is a weird thing about this nomenclature. For that one terminal only, the numbers are where it is NOT connected. That terminal is always a ground, and if there were a connection in any positions, the sequence switch would spin forever. Notice the number 10 is missing. That means the ground is connected in 10, and the sequence switch will always advance through that position. (Imagine how confused I was before I figured that out!) 11/11/1/4 is also insane but yes, you got the meaning right. It's connected for just that 1/4 of a position. I too am amazed by how resilient these machines are. Not only does it keep on chooching along if a single selector encounters trouble, but it can still be fixed, 100 years later with a schematic and some basic tools. Wow!

[@kenunix1863]

@@ConnectionsMuseum The only downside is when a call fails the sender would time out. Turn on it's trouble indicator. And the caller would get dead after dialing. Sometimes you could release the sender by operating the prime key.

[@thewhitefalcon8539]

That's a matter of inflexibility. The other switch "channels" continue processing calls because that's all they can do. A computer can do many things, and most of them aren't the thing you want it to do.

[@ladams00]

Does the sequencer for the selector frame reference translators? I don't have a block diagram of a panel switch. I should probably go try to answer my own question 😊 awesome video BTW. Please keep them coming!

[@ConnectionsMuseum]

The sequence switch is its own thing. I think the translator that you're thinking of is the frame that the sender uses to decode an office code into a location on the district frame. (Translators were later replaced by Decoders, which worked a different way, but accomplished the exact same function)

[@roylamkin7177]

Brings be back to my roots!

[@dansummers2965]

Hey, some good and (eventually) productive confusion there. I recognise some of the expressions, coming as I do from a day of debugging software... More videos like this are good (but hopefully with less ouch).

[@harryminerly4967]

9971 mean anything to you. Or maybe 958 ??

[@ceelonium]

@2000 a narrative of the switchings

[@Fir3Chi3f]

It's just your magic touch Sarah. Should setup a webcam in your house pointed at your house phone for testing purposes!

[@Tomek1958100]

super film

[@thewhitefalcon8539]

comment: super film translate to english: great movie thanks Google!

[@Fopenplop]

"Ouch, many volts"

[@ConnectionsMuseum]

So many!

[@kenunix1863]

Actually most circuits have 48 volts d.c.