Apollo Guidance Computer Part 16: Digging into potted module B11

[@larryscott3982]

Do you intentionally write cliff hangers into the AGC saga? It’s an emotional roller coaster. Forget Game of Thrones, this is Game of Ones (and zeros).

[@mrl22222]

It's like watching episodes of batman....will our caped crusaders escape the clutches of the penguin the time? and what about viki vale...same bat time, same bat channel...

[@larryscott3982]

Mike Leslie Will the short circuit short circuit the project? Can the Caped Technician out smart the fault? Can team DSKY decode the archives in time? Will money run out? Will the team have to go back to their day jobs? It’s a race against time.

[@ReneSchickbauer]

Oh yes, Marc does that a lot. The "Model 19 Teletype restoration" and the "Xerox Alto restoration" also had those kind of cliffhangers. The Model 19 probably had one of the most aggravating ones in the form of "we just disassembled that hugely complicated desk and you wont see it again for MONTHS until we put it together again and find out if we had the correct wiring diagram"

[@mc2594]

'BAM' 'WHWACK' 'CRUNCH' 'OUCH' 'ZZZWAP'

[@smgvbest]

Makes me feel like "Lost in Space" more than batman. last thing we hear is "danger will robinson, danger" then the episide ends in a cliff hanger. will B11 survive, find out next week (hint, hint)

[@timgood4003]

You need to find a dentist who's into Apollo and add him to your team. I bet he'd clear out those modules real easy! This is really a most significant project. You guys are one of the greatest teams since...Apollo! 👍👍👍👍👍👍👍👍👍👍👍

[@CheckEmGG]

the dentist will probably tell the modules that they need to floss more.

[@Lazarus7000]

I'm actually seeing my dentist soon and I'd ask him, as he does have the hand so steady he'd be the one you want, but he's on the exact other end of the country. My old dentist would have been keen for sure, he drove an original H1 Hummer (may have been a privately civilianized HMMWV, actually, I never got that close a look at it) and as was a big ol' nerd, he had lasers as soon as those were a thing in dentistry and loved technology in all its forms, there was a Laserdisc on repeat in the waiting room and the halls were hung with deeply 3-D hologram images.

[@heatshield]

A dentist? Do you want all the wires ripped out, flushed with water and re-potted with cement?! No thank you. I'll just eat around it.

[@kbabioch]

@@heatshield Haha, good one.

[@CuriousMarc]

Belated answer: well you were right on. Another viewer just (3 years later after the video posted!) pointed us to kzbin.info/www/bejne/goHTdJKhg514aJY . It turns out that's how Raytheon repaired potted modules! Mary's group, who was using dental tools. We used the exact same procedure as Raytheon historically did without realizing it!

[@BrianBoniMakes]

You did really well with that. I have experience successfully unpotting electronics. One of my first jobs was at an engineering firm where I did prototyping and reverse engineering. A project I remember well was when a large electronics manufacturer paid us to take apart every shaver battery charger on the market and provide a schematic and a short analysis on its operation. For the potted circuits I had a large liquid nitrogen vessel that I would drop the circuit into until frozen solid then I would wave a small torch over surface. The temp change would cause the top layer of potting compound to crack and lift. I would scrape away all the loose material with small chisels and dental picks until everything was gone then repeat enough times until I could read the components and draw a schematic. The reverse also worked where a lab oven was used to warn the circuit then it was dropped into the nitrogen. All the solvents I tried either did nothing or made such a mess that discovery of the circuits was impossible. I think MEK softened the potting compound but it also removed component markings and softened plastics.

[@Akitene]

Electronics archaeology at its best. You guys are truly impressive.

[@VioletRene]

Fascinating! I was on the edge of my seat when you tested that module! Don't give up. Please! This series has become my favorite on all of KZbin! Best wishes guys! 👏

[@paultraynorbsc627]

Excellent guys keep going Paul, Liverpool UK

[@martinusvanbrederode4080]

Almost as awesome as this restoration project is the fact that there are still physical stores where you can walk in and browse through and buy components. I'm sure they welcome the exposure through your channel!

[@3x3-x3x-oXo]

for *one* dollar! that's mind-blowing.

[@soniclab-cnc]

"stay tuned for the 88th video in our AGC series" keep em coming, these are gold.

[@georgemurdocca4871]

Marc is the microsurgeon of electronic repair, just awesome to watch.

[@hamishgrove7722]

This is my favourite tech series on youtube. I am learning so much. Thank you.

[@timthompson468]

I’m too late offering this information, but it might be useful in the future. I’ve depotted dozens of potted modules, and found the best solution is to add heat. I use a cheap surface mount hot air soldering tool. I can set the temperature to 100 degrees C. That’s enough to soften the epoxy so it crumbles away, but it’s not so hot that it damages components. I use a set of hobby knife blades to dig out the hot epoxy around the components. It looks like you got the job done, but, for future reference, I think heat makes it a lot easier. Cool to see Anchor is still around. I worked down the street from them in the ‘80s through the ‘90s.

[@heinrichhein2605]

and any more temperature cycles means stress for the components and would be not recommend

[@maicod]

I can't say enough how much I consume this series like delicious food. I know it is easy standing by the sideline seeing you guys struggle through so from here I give you a lot of mental support.

[@williamogilvie6909]

Very interesting. You guys leave no stone unturned, figuratively.

[@fredflintstone8048]

Fantastic work by the Olympic team of vintage computer restoration nerds. Fixing one problem can of course flush out another problem which in some cases reveals the cause of the first problem. High current flow due to some short or ?? through a diode can be the cause of it's demise. You guys are the best and I'm sure you'll get it working eventually.

[@BigMouth380cal]

I am not nearly as witty as many of the respondents below, but I am equally thrilled with this project and the presentation. You guys are amazing.

[@lcalvom]

That AGC is giving a hell lotta work, but it's so amazing to have you on board on its restoration. Great work to all of you and greetings from Spain!

[@flymario8046]

geez... that many problems on one board. It was incredible to watch you guys repairing it!

[@nicksantos43]

Man, you guys are doing an AMAZING job. The amount of knowledge you've collected on these things, and the work you've put in to literally rebuild all the tools you need to debug them is astounding. Adam Savage is working on rebuilding the hatch from the Apollo CSM, but I'm thinking he should have gotten with you guys to build an AGC from scratch! Great Job!

[@yorgle]

Dangit. That's the quickest 14 minutes ever. Love you guys!

[@USWaterRockets]

I was really looking forward to this video from the hints you were dropping. Not disappointed!

[@UOttawaScotty]

This is an insane series, thank you so much Curious Marc, you guys are amazing !

[@johanrg70]

Wow that was some excellent work opening up that potted material! I remember seeing the preview of you starting to route it out on the machine wondering how it worked out, great result. Looking forward to the next episode.

[@numlockkilla]

Thankyou to everything you guys are doing to solve the history.

[@pvc988]

Just a few weeks ago I have tried to repair potted module. It was hell. Respect guys.

[@donmoore7785]

It is only a matter of time since you can get to the components - excellent work!

[@jeffreyplum5259]

My father worked at Raytheon Missile division, in the Sixties and Seventies. It did Sparrow and Sidewinder missile guidance elements. Potting was the bane of repairs to faulty units there. They had to do much the same thing as you are doing to fix faults. The very material which protects stuff from damage also protects it from repairs. LOL I love the " Continuing Saga of the Apollo AGC. " Move Over, Buck Rogers! Good Luck!

[@guynardin]

Amaizing... I have not words to discribe how admire your work with this treasure.... please, keep sharing!!!

[@wtmayhew]

Wow! That’s excellent detective work and a fantastic job of de-potting the problem area. Only one minor point - and perhaps why the original diodes were open - clipping the excess lead with diagonal cutters imparts quite a bit of shock to the component (as evidenced by the sharp click! heard in the video). It is better to use a scissors type cutting tool if one small enough to fit into the confined area. Great work and thanks so much for sharing. You guys are fantastic. I should add the shock imparted to the component by the chop type tool may not cause an immediate failure, but can damage the bond wire connection inside the component or crack the die - resulting in a latent failure the shows up later down the road.

[@Zone1242]

My favorite KZbin project keeps delivering - wonderful stuff!

[@arricammarques1955]

Video diaries are amazing too witness! Amazing work, guys!

[@AlanMimms]

Imagine the bugged-out eyes an engineer from 1960s would have seeing this team and their tools - if they hadn't helped to boil the frog to the point we're at now, I mean. Living through the course of the last 50 years makes it seem natural in a way that such fantastic science fictional progress completely IS NOT. Bravo to everyone in the field who worked to make all of this success possible. We all stand on the shoulders of giants.

[@DylanCorrales]

I'm always so excited to see a new video!

[@DavidThomasScorbal]

The breadth of knowledge displayed by this very small team is rather stunning.

[@oetken007]

It is great to see new content so frequently, this topic is extreme interesting and extraordinary! Please post the link to the auction in one of your videos.

[@arf20]

Wow, that was the most intense diode replacement ever! You guys are the best, luck with the short circuit! ;-)

[@randalltufts3321]

I have no doubt they will fix it. 💯% Guarantee I've done digging thru potted vco circuits many times very sucsesfully. They will get it

[@frustro4323]

Bad memories of numb fingers

[@400_billion_suns]

I just love these videos and seeing your progress. What an awesome team! Thanks for taking the time to document it and share it with us!

[@seanalanmorris199]

Outstanding work, thrilling to watch!

[@ambirbtruckdriver]

You guys are much braver than I am. Good luck with everything.

[@martinross6416]

This is so incredible. The agony and the ecstasy.

[@sadiqmohamed681]

Amazing job you guys are doing! I hope you have it all working in time for the anniversary.

[@tachelesreden]

You guys are unbelievable! Just great, gentlemen!

[@givemeakawasaki]

Such EPIC repairs guys! Cannot wait for the next video!!

[@williambello4089]

Wow! I can't believe you mentioned Haltek (used to be on Terra Bella avenue) Halted Specialties Corp. and Weird Stuff. Those were my old Hangouts, back in 1978. Haltek was, in fact, the first place I ever saw a real actual core memory. I used to buy many things from Haltek, but the best thing I ever got was a DEC LA120 KSR for $20. I've been an uber-geek for a long long time! Damn, I miss those places. They always had something special... I am just floored that you removed the potting compound and replaced those diodes!

[@growingknowledge]

The stunning work continues.

[@XPCTECH]

Absolutely amazing, I really applaud your efforts, seems like a passion, keep it up!

[@wm6h]

Thanks for the peak into Anchor Electronics

[@krz8888888]

This is better than any fiction

[@mymessylab]

Fantastic repairs so far. The best project ever seen. 👏👏

[@markbell9742]

Wow, some heavy lifting; Press-On. Cheers, Mark

[@phonotical]

A Swiss grandfather that was a watchmaker? That's great! 😁

[@JensAndree]

This is so awesome - and important for future generations, because there aren't many parts left from this era and hardly any of them still working, hence why this team assembled in the first place. I would've loved to have been part of this team because it's just the stuff that really turns me on but I live on a different continent... I was taught the NASA school of soldering back in the 80's when I started working with Ericsson, building and repairing telephone switch electronics and I still use the same soldering technique today (to a certain extent). Those solder joints I made in the 80's are probably still as good today as they were new and we who understood back then why we did the tedious work loved it whereas others left to do other tasks. How do you plan to close the opened up windows in the modules you open up? It's got to be a balance between conservation (stability), security, looks and personal preference of the owner. The potting compound doesn't seem to have affected the components at all so NASA knew what they were doing 60 years ago for sure! I assume you're writing down your findings and store this with the modules when done so future generations can read how you went at it repairing these modules? (youtube videos might not survive a couple generations...) Anyhow, huge thanks for doing this and for documenting all your hard work!!!

[@johannesdesloper8434]

Hmm suddenly realized.. Your AGC cane from the scrap yard instead of a museaum huh? haha.. great work guys..Absolutely something special.

[@CuriousMarc]

Yes. It’s a rescued AGC from a junkyard. Was going to be melted down :-(

[@johannesdesloper8434]

@@CuriousMarc could well have been that they did some destructive tests on it and that's the exact reason it ended up at the junkyard instead of a museum. I think a couple of years ago I saw the first parts of the serie and it looked bad so I gave up.. picked up recently and luckely you guys didn't give up. Enjoyed it so much. Something special to this. I myself worked first decade of this milenium with Siemens Teleperm-B from the 70's a system that was a bit of an evolution of this computer but much simpeler. The program was stored in mostly AND and OR gates.

[@CuriousMarc]

@@johannesdesloper8434We know from the recovered content of the core memory (in one of the later episode) that it had been functional until its last power down.

[@johannesdesloper8434]

@@CuriousMarc Aha, so components died of "natural"causes. Hard to find the right words , but this whole series of video's did a lot with me. Thanks for the effort. HA! I have a couple of large ferrite rings laying around, this inspired me and maybe one day I reproduce the old 4bit patent with wood and copper .

[@CuriousMarc]

@@johannesdesloper8434 We are not sure. The only modules that were faulty were the potted ones, which did not belong to the original configuration of the computer. One thought is that they failed or got damaged in another computer, then someone stole the good modules from this one and replaced them with the damaged ones. But we don’t know.

[@keithglaysher737]

Considering the great skill set you have between you guys, perhaps you would consider removing all the epoxy from the board & keep repairing & testing till you get the desired result? only a thought. Stellar skills magnificent work guys.

[@markfrombriz]

This is an amazing and inspirational series

[@praedormitium]

With Dichloromethane you can dissolve the potting compound. Leave it there overnight. When working please wear nitrile gloves and no latex gloves.

[@SeanBZA]

MEK will also work, though with some of those potting compounds I had to let it sit in there for nearly 3 months before the potting had absorbed enough to soften enough to pick it off. PCB survived, though I am taking no bets as to the attachment of the tracks to the board, I think most of them are held on by the vias and component leads.

[@edgeeffect]

I love my bottle of DCM

[@mogwopjr]

oh my goodness such great content! Thank you!!! I can only dream of working on projects like this. My digital archeology consists of 1970's/80's CNC equipment.

[@CoreyStup]

Love that you're using your grandfathers watch making tools as part of this! I was thinking like others here, would an ultrasonic based tool cut thru the epoxy better?

[@phonotical]

Been waiting to see this one, very interesting stuff!

[@tickertape1]

Oh my goodness those sprague caps my hp terminal has been screaming at me for replacements shame we don’t have such a thing in the uk

[@Akitene]

Exact same idea: my HP-9821A needs some new caps too. Anchor Electronics has an online store (anchor-electronics.com/), I'll give it a try.

[@littlejason99]

So many new videos recently, we are getting spoiled!

[@melancholicwaffle]

CuriousMarc: come for the computing content, stay for the accent!

[@dolphinclassical]

Lots of work, beautiful video! Thank you for sharing. Oh and isn't it annoying that day jobs get in the way of projects like these?

[@anotherriddle]

This is so amazing to watch! Awesome work! :D

[@kirktierney]

I watch every second of this, smiling like some voyeur.

[@Ethator11]

I've been to all 3 of the electronics stores you mentioned =) too bad there's only one left.

[@BlackEpyon]

Yeah, REAL electronics stores are becoming scarce these days (none of these "Source" gizmo shops that have sprung up the place of Radio Shack) I get most of my stuff online from Digi-Key these days.

[@sybergoosejr]

Keep up the great work

[@tomdewey9907]

Old early epoxy/plastic encapsulated transistors are failing now; just the way it is with aging components. Yet again, look at how reliable the Voyager-1 and Voyager-2 spacecraft continue to communicate since launched in the late 1970's; that to me is amazing.

[@-vermin-]

Impressive decapsulation!

[@dougmcclelland6139]

Miller Stephenson in Danbury CT makes a dissolver for the potting compound ??

[@RinoaL]

i wonder if in 50 years will there be some computer nerds taking this much detail to mess with a falcon 9 computer or shuttle computer.

[@user2C47]

Unlikely. They are like most other computers. All they would need is the software and a computer to run it on. A 60s shuttle computer might be interesting though.

[@RinoaL]

@@user2C47 IBM system 360s were like most other computers, and yet Marc is spending time trying to put one together. SpaceX's computers are just as boring and non-unique for the time as the AGC, just the AGC has all the patriotism behind it to bring interest.

[@jerryfraley5904]

Wait, first mention of 1N914B at 7:16, then you purchased 1N916B at 8:06. Just curious which one is accurate (but I haven't finished the video yet).

[@kevincozens6837]

Nice electronics parts store. Several decades ago I worked in a store that also had lots of shelves with those long narrow boxes of components. Did someone say the wrong part number? @8:10 the procured parts were referenced as 1N916B rather than 1N914B. The 1N914B diode is still very popular today. I still use them in projects. I have one envelope in my parts drawers with around 40 of them that are 30 to 40 years old.

[@florencegomer7937]

Kevin Cozens .... 1N916B is a higher-spec version of the 1N914B is it not?

[@kevincozens6837]

@@florencegomer7937 I looked at the basic information about the two diodes and I couldn't tell the difference between them.

[@macartm]

Great to see more progress, and curious as to how you're going to replace the broken core memory...? OK we have seen you boot it with emulated core but I am guessing you have other plans...? :)

[@Capt.Marco-Hawk-L.L.A.P]

when are you putting the diode on ebay please?

[@zfrenchy1716]

love Anchor Electronics

[@JPBennett]

As many issues as it has, I wonder if that was a test part that was intentionally driven to the point of failure.

[@SeanBZA]

Not likely, just early electronic parts that still had bugs in the process of making them. Make the first few hundred of any brand new product or part that is pushing the limits of technology and you will be doing a lot of failure analysis to see what causes failures, and how to get the process steps exactly right and (most importantly) reproducable each and every batch. Remember these diodes were probably made on first or second generation silicon processing machinery, and with small (around 3inch IIRC) slices to work with you had a lot of places for errors or mistakes to creep in. They probably have had some severe testing, and were just not strong enough to last, failing shortly after assembly, probably as the epoxy was stressing the leads. Bathtub curve of early infant mortality in electronics showing up there. There probably will be another 2 diodes that are marginal, but still work in that module at the moment, but when or if they fail is up to chance.

[@SweetBearCub]

How is it that components in a sealed module that were fed tightly regulated power by the AGC could have become damaged, especially after the exhaustive testing that the components went through before they were used in the AGC?

[@mdbssn]

I know you said it was archaeology, but does this also count as rocket surgery?

[@BillyBobCornontheCob]

Rocket brain surgery, even.

[@jaybrooks1098]

Anchor still exists?!? I remember ordering stuff from them with orvac didn’t have it

[@mikeloeven]

I am curious is there any method to chemically dissolve the potting using chemicals that would not damage the actual circuitry ?

[@HansCombee]

Impressive job!

[@neilolif]

Bringing back such horrible memories.., (pun intended..,) Having been a development technician at IBM, these types of activities were all in a day's work. The difference is, you folks hardly have the option to bin an assembly and buy a new one!! (As it seems some tech at "NASA" may have done with your bad module!!)

[@15743_Hertz]

Built on a Monday. (For those who don't know, the expression is "Never buy a car built on a Monday. That's when the workers get back from their weekend parties with hangovers".)

[@pdrg]

Or a Friday when everyone is rushing to get home for the weekend...

[@VegasCyclingFreak]

6:25 How do silicone diodes go bad? Too much current?

[@CuriousMarc]

That's one way, and was my first thought. Then Mike pointed out that this diode did not pass reliability tests and that the vendor was banned from selling to NASA. Later on another one died on us in normal usage. So it looks like Mike's theory (diodes with per-existing defects in them) has got some traction.

[@mikestewart8928]

According to the official reports, these diodes had "channeling and surface instabilities" which led to unacceptably high leakage currents and eventually failure.

[@VegasCyclingFreak]

CuriousMarc - Makes sense, still was the early days of semiconductors and manufacturing wasn’t terribly consistent.

[@525Lines]

I know they're not something you could easily buy, i don't think but could a laser programmed like the drill do a better job? A similar one was used to clear the dirt from a ceiling fresco in a roman tomb a few years ago.

[@TarnAlcock]

I am trying to find the diode on ebay. Is it sold already?

[@b43xoit]

$1 for exact replacement parts; love it.

[@simontay4851]

But _why_ did these diodes fail (were they short or open? Diodes usually fail short) and how did that shorted wire happen? Did they not test it before potting the module? You would've thought they would.

[@florencegomer7937]

Simon Tay ... These are very early Silicon semiconductors. Many possible reasons for failure.

[@CuriousMarc]

Mike told us that these particular diodes did fail long term qualification, and that the supplier was banned from ever selling to NASA. As you'll see later, another later failed on us during normal operation.They were replaced for the final solution (remember our AGC is from a prototype series that was supposed to flush such issues out). They must have had some manufacturing defect. It could be that some test or mishap was also done for that module, since it had so many faults. I was not able to extract the diodes fully intact, so we can't do a post mortem.

[@mikestewart8928]

Indeed -- the verbiage in the official reports was "channeling and surface instabilities".

[@TwoMagadan]

I would LOVE to buy that failed diode on eBay. Share the link and let the bidding begin!

[@ReneSchickbauer]

Marc has aparently watched other electronic repair channels on KZbin. He's got the EEVBlog "don't turn it on, take it apart" attitude, maybe with a bit of bigclives "let's use the vise of knowledge" way of doing it thrown in ;-)

[@mikestewart8928]

The short circuit in this module in particular is why we took it apart and exhaustively tested every single circuit instead of just powering it up. The short could have caused a lot more damage to other modules. It feels tedious when you're doing it, but it's worth it when you actually find a problem.

[@ReneSchickbauer]

@@mikestewart8928 This might be a stupid question, but how would the astronauts have dealt with a AGC hardware failure during a mission?

[@NetzKanal]

This is awesome!

[@edgeeffect]

If you need a Bridgeport to do electronics.... well, it's gonna get a bit scary!

[@markg735]

AvE says hold my beer!

[@MartynCole]

You have the patience of Job.

[@525Lines]

Properly mounted in a shadow box with some indication where it came from, I can see buying burnt components from your apollo computer.

[@gu4xinim]

Did you tell the store guy where those diodes would end up on?

[@electo99]

do you have the ebay link for the diode

[@DAVIDGREGORYKERR]

Can you guys use Acetone or Xylene solvent

[@brokenworld1]

If you need to do this again try Dichloromethane (also called methylene chloride). It is the active ingredient in paint stripper and is fantastic at removing all sorts of potting materials.

[@Awesomepotamus]

acts like it got a good blast of current at some point in its life. You may be chasing faults for a while on ol'B11

[@SeanBZA]

Reminds me of equipment with now unobtanium MC44 diodes in it, which had a very nasty thing of failing as either open circuit (not too much of a problem till 2 failed in the same section) or they went short circuit. Failing short meant you now were shorting part of a transformer secondary, but the unit would keep on running for a while, just with much higher ripple on the unregulated supply. Not a problem for the linear regulator following it, seeing as the headroom was in most cases 1.5 to 1.8 times the regulated voltage, just slightly higher noise on the rails. As the transformer had high resistance secondaries, and quite high impedance as well, this limited current somewhat till a second diode decided the abuse was enough, and it too went short circuit. Then you had a 3 phase transformer with 2 secondaries passing a high DC current, or a shorted secondary, and with a few hours running the transformer would overheat and start to carbonise, but no flames, as this is in a near hermetic enclosure. Fumes, char and dripping burnt varnish then would drip down and collect, till eventually the transformer finally finished shorting out properly, blowing 2 of the 3 1A primary side fuses. The internal fan would also merrily coat every board with a thin film of this gas, making cleaning the entire interior an imperative. you could smell those that arrived well cooked just opening the transit case, and getting that "well done, past edible range" smell out of it. We could not get the *exact* diode, but instead used the nearest thing we had to hand as replacement, one of a large supply of 1N4004 diodes, and would use 400 grit sandpaper on the leads to thin them down to fit the holes on the board. Worked for those with a single failed diode, or if you got it before the cooking started in earnest, replacing all 6 diodes in each rail 3 phase supply. Eventually some new ones showed up ( do not ask the price), and first thing, after testing them, was to see what the OEM did for those diodes. 1N4002 diodes in all of them, with a nice thin lead, made by Thompson CSF. Did I mention as well early plated through holes, and early PCB material, that was not too amenable to staying put with heating to remove parts. Superglue was our go to fix, along with thin copper wire reworks.