AI Learns Magic the Gathering Cards

[@VitaKet]

Super interesting to see how this is done.

[@jackbaumgartel]

thank you!

[@Sisyphus639]

I love this series! I was just starting to write a personal card database and attempt a card sorter myself. Keep up the great work! Sub well earned!

[@jackbaumgartel]

Thank you for the kind words! I appreciate the support

[@javids.9537]

Jack, this is so cool! Thank you for making this video!!!

[@jackbaumgartel]

Of course, glad you like it!

[@brunoduran2847]

Amazing work, hope you keep doing it!

[@jackbaumgartel]

Thank you, I appreciate it!

[@emilemil1]

I wonder if a scalable approach would be to have one network identify the set, and then have one network per set for identifying the card.

[@jackbaumgartel]

Heck yeah, you nailed it! That's the premise I'll be operating on.

[@BitShift11]

​@@jackbaumgartel, You haven't demoed the image quality of the pi camera yet but maybe a solution would be to train a network to identify text instead of the full card image. Using the name and the info at the bottom of each card, could be enough information and result in more accurate results if you had the image quality to do so. There are also edge cases for special format cards where using the full image would be a more generic/universal approach. You could also implement both solutions and if you are able to detect the card name and bottom text, you could use this for an additional layer of error handling / training data.

[@jackbaumgartel]

@@BitShift11 Thanks for the input! I did put out a video earlier demoing some OCR with magic cards with disappointing results, but it's not a bad idea!

[@DanOC1991]

Great work, this is exactly how this problem should be solved. Looks like the incorrect results are essentially just mistakes in identifying the set from the symbol. Lots of training would mitigate that, but you may also find it more helpful to zone in on the set icon itself. Perhaps you could train another net that looks specifically for and then at the set symbol and uses the other determined info (card name, cmc, colour, etc) from the first net, together with the pic of the set symbol to try to rectify mistakes. Would be interesting to see the confusion matrix to see in more detail what's confusing the algorithm

[@jackbaumgartel]

Thank you! And yes my next video will be implementing a similar strategy to the one you described to help resolve the issue!

[@astropgn]

I don't know where you are on the project right now, but one suggestion could be to treat specific parts of the card differently. It is easy, even without ML, to identify edges. The cards have some sort of pattern, with the collection symbol at a specific place. So, you could fairly easily correct the orientation and select these areas. Since most cards follow a similar pattern (the title is always in a rectangle at the top, the collection symbol is in a rectangle in the middle right, and there is information at the bottom), you can crop just these areas and process them to make them better for the ML. And since the areas of these crops will be smaller than the final scan, you can increase their resolution (using Lanczos or any other algorithm), enhance contrast, etc. With this, you can feed the training data using the original scan but also the specific areas that were post-processed as well.

[@jackbaumgartel]

I appreciate the input, I've implemented a very similar approach to help streamline the recognition beyond just a simple ML model.

[@astropgn]

@@jackbaumgartel Oh, great! This is a very interesting project, I wish you success =D

[@SirPoonga]

I have been doing the same thing. I am basing mine off the TCG sorter. Using an XY gantry and vacuum system. I picked up a cheap laser engraver system off Alliexpress that was big enough for what I needed. I will ditch the laser and put a vacuum in its place. I have been testing OpenCV's feature-matching capabilities. I also started out with imaging hashing and found it unreliable. The feature matching is very good at identifying the cards based on the artwork. What is nice about the feature matching is it doesn't matter the orientation of the card. Whereas with hashing and other techniques we needed to identify the edges of the card and transform it into a perfect vertical rectangle. However, that does not help with set identification. A friend pointed me to an MTG card identifying flow chart. I am working on figuring out how to programmatically do that. For now, though I just want to sort my collection. Since many cards are newer and have the exact card information in the lower left I am thinking about OCR that to identify. Then if that fails or is missing because of older cards then I will go off of feature matching. Right now I just want the card name, color, and rarity. Though cards have changed rarity based on set. Note, I have found not 100% of MTG cards have a multiverse ID. For example, the Secret Lair Decimate does not have a multiverse id. So I have decided to use the Scryfall id instead. Right now I am working on a program that downloads all of the images and train the feature-matching database. I assume you are also using the mtgjson files to get card info?

[@jackbaumgartel]

Whoa thanks for all the info, sounds like you've put some real effort into the project! I haven't tried OpenCV's feature matching yet, I may have to test it out. And yes set ID is certainly a bit trickier than simple card ID. I had also noticed that some cards lacked or had multiple multiverse IDs, I'll have to look at the scryfall dataset and see how it compares.

[@SirPoonga]

@@jackbaumgartel I get the scryfall id form mtgjson also.

[@jasferallenbalboa2343]

OMG, it really worked. Thank you so much!!

[@jackbaumgartel]

You're welcome!

Cool project. I am no expert either, but I think you might speed-up training if you train it for feature extractions. Meaning, instead of training of each card you would train to detect features like color, set symbol etc. The training set would be way smaller.

[@jevinhernandez8654]

Absolutely love your videos.

[@jackbaumgartel]

Thank you, I appreciate it.

[@brendand3765]

When confronted we that issue, I chose to use feature detection (you can find some in opencv such as SIFT and SURF). There is no learning, it's "just" about extracting features once and then matching them. Check it out ;)

[@jackbaumgartel]

Will do, thanks for the input!

[@ellevp7196]

working embodying me, thanks

[@jackbaumgartel]

You're welcome!

[@KianSheik]

The way I'm going it is using a mask rcnn and I labeled the card bounding box, name, and set, then I can use them to rotate and extract the text and set symbol for ocr and feature matching for the set

[@jackbaumgartel]

I like it, I’m excited to see how that works out!

[@DurgeshPatil-bv8ow]

yo bro, really thankya. Big respect

[@jackbaumgartel]

Of course, glad to help!

[@reillocb]

I wonder if there's a way to crowd source computational power like the old folding @ home project. Just imagine magic players volunteering to power neutral network training

[@reillocb]

Just did a quick search and found run-ai's page on distributed training. Tensorflow's library does have it built in

[@jackbaumgartel]

that'd be cool for sure, definitely above my paygrade though

[@spencercurtis86]

Id be down to volunteer to help for ai training

[@serratedsharp4854]

What are your thoughts on a scalable card recognition process? Do you think using something like Faiss to create a similarity embedding vector index would be viable?

[@sonnyv4858]

Any update on this great project? Not sure if I am doing things right , but would loved to get this working. Please help, TY

[@jackbaumgartel]

Yes! My last video was about a month ago, and I should be uploading again in the near future as well. Check it out here if you'd like kzbin.info/www/bejne/imTCpWOFaq-oo6M

[@InspireDosez]

Jack have ya finished this project yet?? And if so can it be incorporated with sports cards?

[@jackbaumgartel]

See my channel for the most recent progress updates, and sports card integration is on the feature list for later on down the road!

[@LunarRunner]

In the video, you mention that this specific process is not scalable for your sorter. Can you explain the factors that limit the scalability? I assume you were using a GPU for training and prediction, can you let use know what you used, and the size your state dictionary? Have you tried to train on grayscale transformed images? Also, in your production, have you considered preprocessing the image to adjust for rotation and hues?

[@KattDa]

Is this the currently favored method for your custom mtg sorter? Or is that another that you're looking for ATM?

[@jackbaumgartel]

Neural networks are super useful for this application, so I'll definitely be using them. There is a bit more logic that has to go into the software though

[@CygnusRising]

So, I don't know much about OCR and AI networks, but... Considering this and your OCR approach... Why should the code care about the title of the card? What I'm getting at is that the title text on a card seems to be 1) a fraction of the dimensions of the card 2) Text, which is more error-prone to examine due to complexity. Evaluating the entire card via AI is also perhaps just overkill, and there are too many small elements that I would expect throw off the evaluation The card art, however, is by my estimation both a much bigger chunk of the real estate on the card (and thus a far more unique and feature-rich element to evaluate than the title alone) but also has far fewer extraneous features (differing color frame style, card type, rules and flavor text, art and copyright lines, etc.) that I'm sure adds to errors in whole-card analysis. So, I'm just going to throw out the idea of "Only identify cards by two parameters: Art, and set symbol.' Perhaps consider building a mechanism that 1) Determines the bounds of a card in an image (ie, correct for the card being skewed or upside-down) so that the evaluation boundaries for the art and set symbol can be more accurately oriented and defined. 2) Captures the defined section that is determined to be 'the art' and uses that to find similar art based on cached 'perfect scans' of only card art. 3) If multiple printings of the same art exist, capture the defined section containing the set symbol of the card being evaluated and uses that to find similar set symbols based on cached 'perfect scans'. This might cause some issues with Alpha/Beta/Limited/Revised/4th Edition all of which shared art and had no set symbols, but I also doubt you'd be sending many Alpha/Beta/Limited cards through a card sorter so... maybe that's not too big of an issue if Revised/4th copies either end up together or in a 'failed to sort' bin -- which, you might actually want to add... Just my take, not sure if this is at all viable or if you've considered this.

[@jackbaumgartel]

Thanks for all the input, you do raise some good points! This demo was really just supposed to showcase how efficient neural networks can be for a challenge like this. You're certainly right that when it comes to implementation, there are much better approaches than simply scanning the whole card. It's definitely a viable solution to break the card down into areas of interest and go from there, and I'll hopefully be covering that a bit more in my next video!

[@CygnusRising]

@@jackbaumgartel Looking forward to seeing how you crack it. I don't need a card sorter, but this is a cool problem to solve.

[@jackbaumgartel]

@@CygnusRising Haha thank you, I appreciate the support and I agree the problem itself is a fun one to tackle!

[@btrbt8613]

This is super cool. I've actually been wondering if a CNN might be able to identify cross-card synergies and combos. MtG's mechanics syntax is very restrictive, so it seems _naively_ plausible using tf-idf or just a word-count mapping. Any thoughts?

[@jackbaumgartel]

Thank you! I hadn't really thought about that application much but I'm sure it could be done. Perhaps after the card sorter project I'll dive into that some more

[@sammelm5357]

I think a CNN is not the right tool for that. For finding synergies you would probably start with a structured dataset (maybe from mtgjson) and CNN don't deal optimally with text. Instead you would use a transformer to learn card text and use it to encode them into an embedding space. Another approach I am thinking about is actually writing a parser that can parse all mtg cards into some sort of ast on which you could apply symbolic methods or even mini rules engines.

[@btrbt8613]

​@@jackbaumgartel You're most welcome, friend. I hope your project works out well. Following with keen interest.

[@johnbertortega5086]

thanks helpful vid

[@jackbaumgartel]

Glad you liked it!

[@violetrosenzweig]

AI to recognize set symbol, and then have a different AI for each set?

[@jackbaumgartel]

Yep, that's the general plan for a scaled solution!

[@davidanderson8628]

Remember the goal of your recognizer. While it might be super cool to be able to differentiate the different printings of Llanowar Elves, but if your goal is to sort by color, then as long as it ends up in the Green bin, it doesn't matter if it's the M12 or M10 version. I wouldn't worry about the multiverse IDs or the different printings, just the card name.

[@jackbaumgartel]

Thanks for the input, I think there may be some confusion though. I personally sort by color, but a lot of people don’t! I’m building this project not just for myself, but for many others who want a more affordable sorting machine. It’s import to me that the software is able to truly know which printing it is scanning, that way people can sort however they prefer!

[@davidanderson8628]

@@jackbaumgartel gotcha. I was trying to find an easier path. I appreciate the ambition. Go forth and solve the hard problems!

[@jackbaumgartel]

@@davidanderson8628 Thank you and will do!

[@dakillaklown715]

Maybe using DelverLens as a baseline would be helpful for identifying cards.

[@jackbaumgartel]

Thanks for the input, but I don't want the project to rely on any external support. Ie if delver goes down then all of my machines are useless

[@dakillaklown715]

@@jackbaumgartel I understand that. I am simply pointing out that there is another program/app that does pretty good with the identification of cards. Thought it might be a place to look at the code for help.

[@jackbaumgartel]

@@dakillaklown715 Thanks, I appreciate it!

[@dakillaklown715]

@@jackbaumgartel is there a way to get in touch with you off TY?

[@jackbaumgartel]

@@dakillaklown715 Sure! My website here has a few options for doing that! jackbaumgartel.myportfolio.com/

[@BagelTiger]

Do you think the AI can branch out to Pokemon, Yu-Gi-Oh and the like?

[@jackbaumgartel]

Yes, those are certainly features on the list! The first priority will be MTG cards though

[@walney2008]

please speak identify pieces and moves chess

[@jackbaumgartel]

Sorry, I'm not sure what you are saying