writing a hardware testing suite from scratch

I wrote a Python app that automatically tests a ton of different things about the LumenPnP to make sure it's working perfectly before we ship it! All the tests run by puppeteering the machine using Gcode, and they check things like max speed, pump and valve actuation, vacuum sensor values, and a whole bunch of other stuff!
You can get a LumenPnP here!: opulo.io/
-------------------------------------------------------------------
Check out my Patreon:
/ stephenhawes
Join the STR Discord Server:
discordapp.com/invite/TCwy6De
LumenPnP Github Repo:
github.com/opulo-inc/lumenpnp
-- FOLLOW --
Twitch bit.ly/stephen-twitch
Instagram bit.ly/stephen-instagram
Twitter bit.ly/stephen-twitter
-- GEAR --
These are affiliate links so if you buy anything through them I get a small percentage which helps out the channel. Thank you!
PCB Holder (Omnivice): amzn.to/344qhVD
Microscope: amzn.to/348aSnw
Soldering Iron: amzn.to/2KlHDGs
Wire Strippers: amzn.to/2IESgDV
Cutting Mat: amzn.to/3oNCwOz
Flush Cutters: amzn.to/3ncFwnc
Digital Calipers: amzn.to/3gIzZSL
Leatherman Sidekick: amzn.to/3a4sQuY
Part Organizer: amzn.to/3a9uDyV
Soldering Iron Tip Cleaner: amzn.to/3a3wZiD
Oscilloscope: amzn.to/2W4zrNt
Multimeter: amzn.to/2JVp6Ba
Crimping Tool: amzn.to/37WymwP
SpaceMouse: amzn.to/3gCWR64
Tweezers: amzn.to/3a1auL5
Solder Wick: amzn.to/347kUVI
Files: amzn.to/37bMXp0
Dremel: amzn.to/3gIADj9
Headlamp: amzn.to/3gE5VYw
Pelican Case: amzn.to/3ncFpbg
-------------------------------------------------------------------
Gundam Media Sources:
• GUNDAM EVOLUTION - Sor...
www.flickr.com/photos/calavet...

[@stephen_hawes]

You can check out the machine here!: opulo.io/products/lumenpnp

[@dinodubroja7433]

closed loop stepper drivers are solution to losing steps

[@mikeselectricstuff]

That test procedure is going to get real tedious once you start doing significant numbers. Much of this can be automated. What you ideally want is a single button for "go" (or not even that - just start when it detects the USB being plugged in) and red and green lights for pass/fail. As you have a vacuum sensor and a movable head, all this could be automated by moving the head onto something that blocks the nozzle. Remember that for production test all you are looking for are things like bad connections, incorrect assembly etc. so the testing only needs to be enough to detect each possible issue, e.g. you don't need to test all nozzles, just that the vacuum readings are in the right zone. If you can home by sensing motor current,you can also test for missed steps by counting how many steps ( or just the amount of time) it takes to home from a known position.

[@adamcarver9057]

I absolutely love how this machine is so simple for the end-user but so advanced at the same time. Good job, I want one but I don't have a need yet. Maybe I should make a need.

[@TheRainHarvester]

Haha that is my frame of mind too!!

[@UnexpectedMaker]

I know you're just starting out with Gundam - and so far it's pretty cool :) - apart from all of the human/manual intervention (I am sure you know this already) - Much of the movement testing can be done automatically with a camera that sits over the DUT and can detect the movement and homing tests etc. There are a lot of opportunities for removing human intervention over time as you continue to develop the system, but I'm excited to see your investing time into your testing tools - they are SOOO important! Great job!

[@stephen_hawes]

Ah yeah that's a good point! I should totally add fiducial scans (and just vision in general) as part of the test for skipped steps, and probably a ton of other things. Definitely very manual right now, but excited to automate more and more of it! :D

[@randycarter2001]

Fiducials. Start with the camera staring at a fiducial mark, run the motion test, return to where the fiducial is supposed to be. The camera image will show if any steps were lost. Then the imaging software will report x.x% error.

[@stephen_hawes]

Totally rad idea, thanks for the thought! This will totally be how I solve this in the next version!

[@Ziraya0]

Have you considered a test "harness"? I don't know how many things you can automate via 3D printed block full of widgets that you snap on for the test, but you could automate testing the vacuum tips by having the NO45 tip mounted in the block at a height it can shove the tip into, and with a solenoid to block the NO45. Put the head in position, do the open test, shove it into the nozzle, do the nozzle test, activate the solenoid, do the closed test. There's a bunch of options for a test block to facilitate checking for lost steps, you could have a mock homing post with microswitches; you could have a really big optical interrupter, almost certainly DIY, and probably a laser, that detects the gantry without being collideable. Doesn't need to be precision, just after homing have the gantry cross it and record where it is, then after the speed test cross it again and detect if it moved. Another option, 4 paddle microswitches like in an arcade stick (2 switches would probably be fine), at a height below the position where both heads are lifted, but high enough that you can reach a head down into it. This has less rigidity because the heads, but it's low force, especially with paddles giving you a lot of leverage. You could also use optical interrupters to check Z travel, the most extreme version is have 4 interrupters, two just inside the upper and lower limits, two just outside, as two stations. Put a head at an extreme, enter the station, one interrupter should read open, the other blocked. If both are blocked it's too low, if both are open it's too high. I think, not totally sure, but I think, if you check the lower limit of both heads and they're both nominal, then you can infer that the upper limit is also where you want it to be; which means when you put the head in neutral position, the heads are at the same height. So you don't need 4 interrupters, you don't need two stations, you just need a pair of interrupters at the lower position, which also means you don't need a thing sticking up that could get hit by a faulty machine. And now it can be a 3D printed hole housing the interrupter pairs in a cross so the two Z levels can be closer together without components hitting each other.

[@stephen_hawes]

Wow, all kinds of rad ideas, thank you so much for your thoughts! I really like the idea of a cal jig print that snaps onto the staging plate and helps facilitate automating these tests.

[@AMTunLimited]

This might be a bit *extra* but you could extend the definition of tests to "depend" on other tests and do a graph traversal to make a list. That way you can define certain sub assemblies and systems the test depends on and you don't have to think about test order, especially if you have assemblies of assemblies...

[@kentswan3230]

Speed checking for mis stepping can be delayed until you have the upward-looking optics operational. The basic theory is to auto-center the nozzle to the camera. Then run a speed test then move the nozzel back to the camera and it should be the same. A similar method can be used to calculate the dual nozzle offset.

[@davestech6357]

You should use a vacuum tip as a sensor for skip checking instead of eyeballing it. Also travel and all motion repeatability. Would be nice to have one of your machines. Great work.

[@niceride]

Quick tip: great UI/UX design avoids "Yes" "No" "Ok" "Retry" "Cancel" buttons in UI. For example "Run Test" button is great UI/UX design. "No motion" "Yes X motor moved" is better than "No" "Yes". Casually following your progress as you post these videos and am excited for you! :-)

[@Andreas-gh6is]

I think marlin has a TMC debug command with M122 that could be useful. For more automatic tests you could put a wireless arduino on the "head" and report the acceleration. This can easily be made fast enough to capture individual steps.

[@MalcolmCrabbe]

Watching the movement at around 6:20, the flexing of the LH cable near the plug might prove to be an issue in the long term. The outside of the multicore cable looks very stiff, so the flexing is where the inner cores enter the pin header plug, which may in time cause the conductors to crack and fail through the "wiggling" effects as the nozzles moves up and down. - Just an opinion, I think you have done a wonderful job in taking this idea to market, and appreciate you taking us along for the ride.

[@mikeselectricstuff]

Yes, that cable definitely needs proper support - having the cores flex at the connector will be a disaster.

[@TheRainHarvester]

You could populate a board without solder paste, automate a picture of various sections, then compare as the check. Then again it would use parts and if a failure disturbs the pcb, then all the SMDs could shift invalidating each placement (which signified different tests).

[@TheRainHarvester]

Maybe compare pics taken as the test progresses.

[@nicholaslabrecque]

Stephen just offset your axis by a known amount and after testing a speed "home" and count the steps back to home, if its not your known amount you skipped steps. might need to do it like ( home + offset => max - offset ) home to the max, max - offset != offset === skipped steps and vis a vis

[@Staphylokocke]

About measuring the force required to move the axes: You could iterate through various settings of HOMING_SENSITIVITY with the M914 gcode command. If you place the axis that you want to measure at a known distance to it's physical limits, you can measure the time that it takes the axis to home. If the axis reports that it is homed too early, the force required to move the axis must have exceeded the threshold for that HOMING_SENSITIVITY. If this works, you would just have to find the tipping point at which the HOMING_SENSITIVITY is just high enough to fail to move the axis. That specific sensitivity should correlate with the resistance of the axis.

[@Staphylokocke]

Okay Mr. Hawes, you really nerd-sniped me. You can actually just get the motor load from the M122 command, which gives you the sg_result. The sg_result is just a measure of the motor load at that time. So if you move the axis and average that value over a few seconds, you should know how much resistance your axis has.

[@markmcgookin]

Hey this looks amazing. I was watching this wondering if building a small jig/board would remove a lot of the manual intervention here? A board with sensors that could detect x/y gantries moving over head, some small sensors with different sized chips over sensors that could be lifted/placed to detect vacuum is working and maybe even rotation somehow. Just a thought, love your videos bud, really inspired me to learn a lot more about electronics.

[@electricCoookie]

Always making Serial uis with flutter now :D

[@mhdm]

On skip step detection, or a version of it: try out ENDSTOPS_ALWAYS_ON_DEFAULT in Marlin which is basically that when also using SENSORLESS_HOMING. Note you can tune the sensitivity with M914 command. Also if you enable TMC_DEBUG then M122 will give you detailed stepper driver information, including stallguard sg_result, which is a number representing the current motor load. Re 5:27 "back at perfectly zero". Well.. the above should catch missed full steps but the precision rabbit hole goes deeper. TMC motors also report the current micro step as "uStep count" in M122. Home, record the value per axis, do a lot of motion then re-home and check how different uStep count is. This should be precise enough to measure the repeatability of the microswitch I see you have on the Z axis.

[@haddow777]

I wonder. For positioning, you may be able to use magnetic sensors in combination to smartly placed magnets to give you more precise position feedback. The sensors nowadays can detect way more than just a magnet, but it's field strength and orientation. So, if positioned right, the sensor could not only detect a magnet, but how close it is and when it passes by to the other side. The could really help with positioning data. You could home the moving part, take the sensor input for that position, then track sensor output for each step as the part moves. Obviously, in many areas, you would just be able to have the sensor at the end points of the part path, but that would still give a tonne of feedback. For instance, you were mentioning about skipping during moving at certain speeds. Well, you could combine the expected position of the part to the sensor feedback long before having to home the part and recalibrate it. Also, it allows you to adjust end positions easily in software with the sensor verifying by part position rather than step count, so it's independent of alignment issues or step skipping.

[@sonosus]

You should have the script print out a small test report via a thermal printer on receipt paper and put one in the box for the customer to see their machine's test data.

[@ExplodingWaffle101]

m122 and m914 might be useable to check for missed steps, which is basically what sensorless homing does and i think prusa skipped steps is also done with stallguard

[@phillipsengineer]

Hi Stephen. My question isn't specifically about this video; it's about designing robotics. I find your LumenPnP absolutely fascinating. It would be awesome if you could point me in the right direction. I'm trying to add robotics to my custom, computer-automated printed circuit assembly tester. Wishing you an amazing day.

[@chaitanyasindagi1237]

You could home again at the end of your movements and see if your new home offset is zero or not. If you didn't skip any steps you should theoretically home at what you think is home from before, otherwise you'd hit the end stop sooner or later than you thought you would

[@decee1157]

The test at 5:30 You could make it do the different speed tests you already made and then by knowing the current position of the stepper motor you could add a feature that homes the axis and returns the length it moved from start homing sequence to home position. Then you know if the stepper motors skipped steps.

[@bracco23]

You surely already know but prusa includes a report of the tests run on their machines before shipping. Since you already have the results, it should be easy to attach a printer and print a summary for inclusion in the package (just generating a pdf and printing it should be enough)

[@alexandersidorenko5605]

Might make sense to also keep track of temperature you ran tests at. Realistically, might not make any difference in this particular application, but I have seen test/calibration data go useless because we didn't note what was the temperature of the board when we ran the test :) We realized this months later, when seasons changed and ambient temperatures in the test area dropped significantly.

[@OMGWTFBBQSHEEP]

Always exited when a new vid is up! Regarding checking if it skipped steps or not, what about sending a home command after the test and then timing how long the home sequence took? If it takes e.g. 0,05 seconds longer than normal, you know it skipped some steps. Just thinking out loud here, not sure if its actually possible with Marlin😊

[@ThePixelMint]

I came here to say the same thing, except that I thought maybe it could home and see if it’s home has moved relative to where it thinks it should be

[@Staphylokocke]

@@ThePixelMint Marlin doesn't report the steps it took to home, sadly @OMGWTFBBQSHEEP I looked up the steps per mm and it's at 5mm/fullstep so the timing could be viable if you go slow on your homing feedrate. Now the homing just has to be repeatable to at least 2.5mm, but that sounds reasonable.

[@stephen_hawes]

Ooooh this is a cheeky idea! I think this would be possible with the time measurement happening in Python, and waiting for a "finished homing" response from Marlin! I'll poke into this, thanks for the thought!

[@OMGWTFBBQSHEEP]

Great! Looking forward to see if it works out!

[@Aletaire]

One of the cool things prusa does with their testing is literally print out a receipt that shows all the test results, with every printer.

[@stevewalston7089]

@Robert McEwen - They are already doing this. It was in a video about a year ago.

[@benbionic]

Use a camera above to check location and see skipped steps. It could also see each axis movement so you don't need to hit enter over and over for each test.

[@bibinvarkey3041]

@Stephen Hawes: Can your Open PnP machine place BGA, LGA, PGA chips aswell? Also are you planning to make open source machine for Solder Paste Printing, Solder Stencil Printing, 3D Solder Paste Inspection (SPI), Automated Optical Inspection 3D (AOI) for top and bottom smd & pth component placement, Automated X-ray Inspection 3D (AXI), Robotic PTH Soldering machines aswell?

[@shelbyseelbach9568]

Now make a robot to test apps.

[@realmup3r]

Great progress! I bought one of the first "print your own parts kit". I'm still waiting for further information on the promised upgrade kit to convert the machine into a V3. Any news?

[@realmup3r]

*push

[@jbob6429]

Just a thought but I would probably do homing test first. If the homing works then there is no need for the movement test and the movement test requires user input. (Just from a more detailed max min prospective).

[@makermelissa]

You mentioned that a fork of marlin could check for skipped steps. Have you considered flashing a custom version of marlin for testing (maybe even just for that one test) and then flashing the final firmware on after that?

[@Personnenenparle]

To check the skipped steps, just rezero with the current spike and see if its at the same place

[@datawolk]

Isn't it possible to measure the belt tension in a certain position of the x axis? With a pressure sensor or something else.

[@michaelbishton9439]

I noted your comment about testing the stepper motors by running them into a hard stop and measuring the current. I read in Arduino forum that because the current constantly varies while running a stepper motor, it cannot be done. How do you do it?

[@mathiasaa1]

For checking if it skipped steps: Couldn't you make it stop for example 1cm or X amount of steps to the side after the speed test and then home? If it only moved 9mm or less before it's homed it was too close and if it moves 11 mm or more before it's homed it was too far away and obviously skipped steps in both cases?

[@M8KES]

Is the schematic and board files available from your PNP controller ? Btw , if o watch this videos , I don't feel lonely anymore if jumping around when things working

[@bjarkeistruppedersen8213]

Now for automating the entire test 🙂

[@M8KES]

Where I can find the part list that is used on the machine. Is there any or I'm just stupid? 😂

[@bestrider14]

just to mention "Qt" is pronounce "cute"

[@riccardosacchetti]

For steps.. you could try a closed loop system.

[@BigfootPrinting]

Klipper keeps track of the actual steps issued vs toolhead position for repeating movements and homing to check if steps were skipped. You're probably not going to get that functionality without implementing it yourself in Marlin.

[@genericcheesewedge4870]

There is a script for printers to detect step loss at increasing acceleration, but I’m pretty sure that only works on Klipper

Is this extensive testing needed because of problems with V1 and V2 models or is this just a precautionary QA?

[@vladbabii9025]

To test skipped steps on x and/or y you just need 4 contact points on the "bed" and one probe on the head. just move left/right or front-back a number of times then try to probe 1/2/3/4 points. based on results you can say if you skipped any steps.

[@danedwards_ee]

Hey cool!! I'm doing this for a board I designed.

[@gramps717]

P5 has most modern error messages for pixel point processing

[@cadeallen7210]

Which library do you use for the GUI?

[@InterFelix]

Regarding missed step detection: Since Prusa Printers seem to have it and their fork of Marlin is also open source, you could have a look into their source code and port that feature to your firmware. It'll probably not be quite easy, but I'm sure you can do it. Alternatively, reach out to Prusa3D and ask if they can help you with that, maybe by contributing the feature to mainline Marlin. I'm sure they'd be glad to help you out.

[@Staphylokocke]

Their Motherboard connects the uC to the diag pins of the TMC2130s. I assume they have an interrupt set on that pin. On the lumen motherboard on the other hand, that pin is not connected and instead they use SPI for communication with the stepper drivers. So I think you won't be able to just use Prusas implementation here.

[@InterFelix]

@@Staphylokocke oh, interesting! That makes a lot of sense, actually! The drivers will probably not deliver enough data over SPI to achieve similar functionality.

[@Staphylokocke]

@@InterFelix Hmm but at the same time, the lumen uses sensorless homing, which means there must be a way to get the load values in a high enough frequency for homing to work.

[@Staphylokocke]

I guess for a single axis while no print is running it is fine to just poll constantly over SPI

[@MrSmiley162]

New Relic could be a good alternative to your Google Sheets moving forward

[@InterFelix]

Sure you mean New Relic? All I can find from them are Observability / Monitoring tools for various cloud and On-Prem infrastructure. I thought about AirTable (and the alternatives, especially Baserow as it is open source).

[@MrSmiley162]

@@InterFelix New Relic definitely has logs, and I imagine it'll be free for the quantity of data they'd be ingesting. Just thought I'd throw the suggestion out there. Good to hear alternatives!

[@InterFelix]

@@MrSmiley162 oh, interesting! I didn't think about using a log aggregation service in this way, but it kind of makes sense, as I think about it!

[@iliya-malecki]

Surely skipping steps looks different to bumping axis limits on the current-vs-time graph. Maybe just train a little regression and make it hella conservative?

[@iliya-malecki]

Oh oh I know, pick n seconds after a "collision peak", fft it and voilà, even a hardcoded decision tree can categorize based on amplitudes of predefined frequencies.

[@iliya-malecki]

I would go as far as saying that picking a couple of points on the delta-current vs time graph can be good enough at indicating skipped steps. Now I just need to fit some models and show the accuracy, so if you find it worthy a try, just say (and give current data :))

[@OscarSommerbo]

Sure, sheets/excel, is a poor substitute for a "real" database, but it does work well for smaller datasets, and it is hard to beat the automation simplicity with csv-files and fixed columns. I did a fair bit of VBA programming for Excel in 98-2000 with Excel pretending to be a database, it was fun a frustrating in equal measures, but I finally convinced my betters to spring for actual databases. I think for automated testing, you probably have to use some sort of "active harness" ie something you hook up to the PnP, but it has the sensors it needs to check for skipped steps etc. So I would assume an Arduino or similar with a bunch of sensors that can talk back to Gundam to provide what actual values it saw, its name would clearly be "Haro", but that is just fast musings.

[@mikestaub]

Any plans on adding computer vision?

[@AJB2K3]

I don't know the legality of using the Gundam name for the software so please pst a followup on this please? However, Depending on which anime you followed (I think Seed/seed Destiny was the only one to Call the OS G.U.N.D.A.M) most other used G.O.S (Gundam Operating System.) 😛 Wish I could afford to invest in one of these as watching its design and evolution has been really interesting. Keep up the good work.

[@bracco23]

It's internal use so probably there isn't any issue legally as it will never release publicly.

[@antoniosanford4675]

Can all of your software run in linux?

[@petermuller608]

This procedure looks expensive as hell. How long does the whole process take, or will the operator be able to run multiple tests at once?

[@gramps717]

disable componet on purpose then build error messages around that

[@gramps717]

so the error messages point to a bad component such as a diode ete ete

[@timonsku]

For open source hardware you can get a free VID:PID pair from pid dot codes With a unique PID you can auto detect the USB connection of you machines in any software which can be quite useful (like your test software)

[@harryman01]

🤔why not use encoders , and optical endstops you will have a full close loop and even the homing and other calibration more precise

[@Jehty_]

Hm, are you planning on optimizing the procedure? Because there is a lot of time wasted. You could test multiple things at the same time. Like for example the test of the motors. You can let them move all at the same time. Or the test of the nozzles. Why test them one after another? You could do both at the same time. And then requiring to hit enter after every single test. That's not only wasted time but also really annoying for the operator.

[@cw5522]

Is gundam open-source?

[@Tedlasman]

Klipper version soon?

[@DissyFanart]

big ups

[@Alex_whatever]

Stephen, you are doing it again. You are asking for some copyright thing to happen again by using copyrighted names for things... Gundam is cool and all and I love it but, yeah.. I recommend you come up with another name :-)

[@saifsahory7105]

detail because U was so lost, I still am but not as much haha

[@user-fj4bn6ci2l]

太慢了，淘宝搜，openpnp已经有商家在卖了，而且很便宜。

[@jstro-hobbytech]

So c++ by proxy.....

[@lloydjones5871]

I love the energy, fanaticism, dedication and technical skills you are putting into this project, keep it up. But a small critism, I cant cope with your video splicing style of presentation in this video. The disjointed video distracts my mind from the subject being presented and gives me a headache. If you cant script and present in one shoot then i will have to stop watching your channel. sorry.

[@InterFelix]

That's just his style. I, for one, quite like it. Also, you can tell it's deliberate since the audio cut happens slightly before the video cut, which is more work than having both cuts be simultaneous.

[@blomproductions]

First