3-Sweep: Extracting Editable Objects from a Single Photo, SIGGRAPH ASIA 2013

by Tao Chen · Zhe Zhu · Ariel Shamir · Shi-Min Hu · Daniel Cohen-Or

[@Joyfulmischief]

The only question remains. When will this wonderful piece of black software magic will hit us mere mortals ?

[@hanserdmann4058]

Alredy possible to do it on ISO Photo Perspective matched ortho RHINO CAD viewports..but without texturing which has to be performed manually afterwards!

[@bloomp7999]

2021 still not available

[@Joyfulmischief]

@@bloomp7999 Lol...because it was an elaborate hoax in the first place

[@bloomp7999]

@@Joyfulmischief waaaaaaat yes it Indeed looks like a hoax, but i was hoping , it could be Real! Do you have a proof of what you Say ?

[@Joyfulmischief]

@@bloomp7999 No. But obviously...It looked really incredible, but never anymore info than this video surfaced, so there is little doubt now 😉

[@joenap7158]

geosemantic snapping - this is a brilliant idea. The examples use objects pretty much built from primitives and symmetric textures so this video does show the best cases. A head would probably end up looking like a cylinder with another face on back. Photogrammetry would offer detail while this would be incredibly fast and require less. I'm probably going to dream about this tonight.

[@21EC]

the person behind this idea is a genius !

[@TheMimzez]

This is what people who don't know anything about how photoshop works thinks how photoshop works and now there is something that really works like that, amazing

[@MaximumADHD]

I don't really know. It seems too cool to believe.

[@EldritchEyeFilms]

My mouth was open throughout this entire video in wonder and amazement. OH MY GOD

[@xDP02]

1 year and 5 month later this and a few news articles of this video are all the evidence of this tool till today

[@KaltatheNobleMind]

neary every article i read on here just talk about how it's easy to model assest fr games and do 3d print. peopel are blind to the fact that this could speedboost machine vision 30 years ahead of what curently goes on since it captures volume,texture,spacial coordinates and context data all at once within a minute where traidtional photogrammetry would take sufficently longer with less accurate renderings. if google baught this and implimented it in their officla camera app as well as ties to google earth they would have had a virtual roadmap of nealry all major cities within a month which can help aid in the navigation of self driving cars:D if i make sense. tie it with regular learning systems too and you can generate i library of any sort of object known to man within a year,and it cna do it automatclaly once the system recognizes contours and what to trace leading to even more ditization:) this can very well be the key to creating an Artificla general intelligence

[@xDP02]

Meir Teichman No

[@KaltatheNobleMind]

xDP02 it cant build up AGI and machine vision by making fairly accurate input easy to create and therefor impliment?

[@xDP02]

"the key to creating an Artificla general intelligence" No ;) I don't even know how the ability to recognize contours relate to an AI in any way.

[@KaltatheNobleMind]

wantafanta01 also try syntheyes. supposedly 123d catch gives a better result but there is a maya plugin that works with syntheyes to get better texturing :) whatever works though.

[@Kantuva]

I'm in awe, really, this thing is beyond words guys, this software is one of these things that will change the world we live in, just like 3D printing will in a few years.

[@Donny67810]

Download Link?

[@victoralemandelgado4753]

no

[@danylipsker]

This is the best of the best I use 3d for more than 20 years and never watched such an achievement with apparent ease of use and simplistic solution Bravo

[@johnchapman5615]

WHERE DO I DOWNLOAD THE SOFTWARE. I DONT CARE IF I HAVE TO PAY FOR IT JUST GIVE ME A WEBSITE GOD DAMMIT!

[@JustinKoenigSilica]

+John Chapman i need this as well

[@aleksandarmilanovic7719]

+Justin Koenig Try here:www.siggraph.org/discover/news/siggraph-2014-app-available-download Click on link:,,Downolad the Siggraph 2015 for ....,, and in google play box enter Siggraph.The first one.

[@flashcat945]

+Aleksandar Milanovic There is no download for 3-Sweep? It's just a useless Siggraph event app?

[@jefftukangjeff]

+John Chapman publication. READ it. cg.cs.tsinghua.edu.cn/papers/SIGASIA-2013-3sweep.pdf author: www.faculty.idc.ac.il/arik/site/3Sweep.asp

[@flashcat945]

+Jefftukang Jeff No software demo, as he was requesting.

[@Zingsta]

I have never sympathised with how cartoon characters were unable to close their mouth from awe until now holy crap this is amazing

[@ravoniesravenshir3926]

Can you use it to tear a model of a anime char from a background and it will predict the missing parts?

[@diegoantoniorosariopalomin4977]

dont you dont this yet unreleased program for that , what you are looking for is the patmatch algorithim

[@ravoniesravenshir3926]

Well I happen to have some images I want made into 3d models.

[@diegoantoniorosariopalomin4977]

+Ravonies Ravenshir i dont 3 sweep would be able to do that , since it relies on shading information to calculate depth , which is flat in anime compared to real life

[@ravoniesravenshir3926]

Well for the program it shouldn't matter...

[@Leesment]

Been to school for 3 years studying 3d, seeing this will make life so much easier in the future, amazing!

[@batpower54]

Stickmasterluke brought me here

[@JosuahDemangeon]

The telescope case is very interesting: Since you have a 3D representation of this object, you can now exploit the reflection on the cylinders to reconstitute the scene on the back of the photographer ! and since you have multiple different reflection with different positions, you can even turn it into 3D. But this would require quite much hard work to be done! Congratulation for this amazing software !

[@balup4515]

plz give me download link.

[@juliobermudez7125]

yout.com/video/Oie1ZXWceqM/

[@jamesabell9494]

This is amazing. It will leave people to concentrate on creativity while being able to make quick textured models from their photos too. The future is now!

[@figs827]

Heterosexual and legitimate

[@ParasyteZero]

What slaughter? I've been using Adobe software for close to 20 years and my work with the software resulted in a great job. They pay for my software now, but I worked my ass off to buy my first copy of Photoshop 4 and subsequent upgrades. I have zero regrets.

[@derketzenable]

It's a fake.

[@hamhampangpang_customer]

cg.cs.tsinghua.edu.cn/3sweep/software/index.html

[@MarkBrockman]

What's really impressive is how the software rendered the background behind the obelisk around the 4:25 mark without ever seeing it. Technology you so crazy.

[@SaurabhSingh-nz6uu]

this total fake if any one know the software name tell me ?? it's total fake bull sit???

[@G7ennx]

This isn't an 'introduction to 3D modeling', it's the state of the art in technology for making 3D models from photos. Apples and Oranges and all that...

[@mohit96mb]

I really doubt its reality.

[@JorgeGamaliel]

Awesome. This is definitely the beginning of another great revolution.

[@lbpdluis]

fake.. you can do this with blender, those are pictures in 3D

[@dracconcomet]

this is seriously one of the coolest things ive ever seen.

[@buddieizreal]

FAKE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[@jordinwang9378]

Why?

[@buddieizreal]

Hell if I know fam...

[@TheDeathmask97]

At 1:10 they stated that they use another "patch match" program to fill in the holes created by removing the object. This is visible at 2:18 if you look to the far left at the candle's shadow. The area where the the candle used to be has a solid shadow with a noticeable "cut" in it, and in its place is a fuzzy area that is a color that results from mixing the shadow and the yellow wall together.

[@2wolfx]

For those who are confused with the scene behind the object, please search "image inpainting" in KZbin. There are many advanced techniques that recover much larger background than what is shown here.

[@TheRobustitron]

My first thought was pulling props directly from concept art and either polishing them or using them as a reference. Custom UV seams & packing could really make this a must-have for games, or you could just use it to bake a diffuse for an optimized mesh.

[@emart5]

The film industry is going to slowly transition into the Video Game/VR industry, at least to an extent (IMO). If we can capture actors completely in 3d along with models like these we can really attempt to recreate real life areas in a virtual space that will look almost 100% like real life. The future of the holodeck is actually starting to happen!

[@antoniy01]

This is just mindblowing. Period.

[@AldGregg]

That is brilliant! Well done. I would use this all the time for making quick objects. Great for game modders and pros alike.

[@jeanlukepicard4859]

I've seen iteration synthesis at the last SIGGRAPH. So basically algorithms that take some models (for example, five pairs of different sneakers) and from that set, these algorithms can create new sneakers (!!!). So you can imagine how that will work with this one here? You'll be able to "3D trace" an object from a photo here (for example a street lamp) and then once it's recognized as belonging to the category "Street Lamps", the algorithms will generate 50 different new/different design ones.

[@Usumgallu]

Couple of years ago some company claimed to develop a software that could automatically build scores from recorded music. Looked awesome in the introduction video, but in reality the software was never released. I hope this will be more successful.

[@pedites]

Since Photoshop it was difficult to trust images. With this piece of software it is not difficult... it is impossible. Amazing !!!!!

[@nlicky]

This is crazy good at how good this is as getting 3d models from photographs. It's not totally perfect but looks amazing at the convincingly good models from just a 2d image. While I don't think it will replace traditional modeling as some things are best left to work manually by hand, it can surely help with the process to generate realistic objects and textures easily.

[@sherri99516]

This is way too cool for school! Amazing 3D editing software. A must have. :)

[@leerdj]

Great demonstration, props for showing weaknesses too.

[@DavidLinkan]

It is real. The "failed cases" is a welcomed honestly from their part. Looks amazing, but I'd like to see the wires because it seems to put polygons everywhere without any optimization - and here's the retopo time. Amazing thought.

[@mladthelad]

I think its basically looping the texture around. Since the portion which we dont see is symmetrical to the visible geometry, the texture mapping probably follows the same thing.

[@hexygen1]

This will be published in SIGGRAPH Asia 2013, which hasn't happened yet. They still don't have the conference program on their website...

[@badmojo420]

Because it's not a cylinder getting wider. The program is looking for a specific shape that the user has selected and pointed out. The area the user painted went all the way to the top, so looking at the whole thing, it can see the arms are not part of a skinny cylinder and ignores them. It doesn't just look for edges, it looks at the shapes in the photo. I'm guessing this is why there was so much brushed metal, since it makes it easier to identify a shape visually from only 1 angle.

[@Q_20]

Why does like 99% of SIGGRAPH cool techs get forgotten like this?

[@aaronisgrate]

that was my first thought too! "how awesome would it be to scan whatever and plug that into a 3d printer??" (like minds think alike :)) you could make ANYTHING you wanted. i think the future is finally here ladies and gents.

[@HeliconPath]

It's a really clever photomanip algorithm that is actually fairly old now. You can even find it being used in mobile phones.

[@zero0101one]

That's simply brilliant, Thanks for your hardwork!

[@safranpollen]

I think 3D printing&software will be in close future times more important than any industrial evolution or software processing ever done. a great innovative milestone software demonstration . ThanXXX so fine:-)

[@DaveNicholas]

I need this!!! This is a dream for a programmer like me with no design skills.

[@delbarital]

Guys this video is not fake. Ariel Shamir, one of the researchers behind this technology is a professor at my university (IDC Herzelia). Search for the short TEDx lecture he gave few years ago about another research he was involved in "TEDxTelAviv - Ariel Shamir - The Vision of Image" however, as I understand, this is only a proof of concept and not an actual product yet.

[@JonKroeker]

That's true. You use the fill tool with the Content Aware setting. Doesn't always give intended results but impressive when it does work.

[@JustinSorensen17]

My mind has been completely blown.. future of game development

[@AliiqYT]

صراحه اكثر من رائع This is Awesome كيونتك شكرا

[@NektoDesigner]

This is just mind-blowing! Thanks for sharing :)

[@Silencer1337]

Depends on where you want to go with the model. Any serious modern 3D application wouldn't be able to use the image as is, because it needs it to have very specific lighting conditions, so that it can compute the actual lighting through virtual sources of light in whatever scene you put it. Look up things suchs as "normal map", "bump map", "gloss map" and "specular map". Not to mention it's all stretchy itself and depicts the object only from one side.

[@Krest0grad]

Dear MrFoxInc, sorry for my english%) We can build a reliable 3D model, being available to 2 and more photos. Build a model from one image we can to do, just knowing some of its proportions. That is, assuming that the cross section elements - a perfect circle, that the faces of the cube are perfectly aligned. A single image be produce optical illusions.

[@dowpcalleja9492]

1:12 "patch match algorithm" , this method is achieved by taking some parts around the object and patching it behind seamlessly. patch failure example: 3:33 as you can see the inconsistent shade around the object rendered the patch very visible.

[@johnne86sd]

To fill in the missing background, I believe the algorithm just recognizes the pixel information that's near the outside edge of the object in question and fills the background accordingly with that information once the object is extracted. The algorithm just assumes the background is naturally and logically the same. It's like using a stamp tool, but automatic.

[@SANTARII]

They explain that at 1:10. If you look at some of the other examples, you can see that it's not perfect, but good enough so that nothing stands out as wrong.

[@NoTraceFound]

There's powerful auto fill algorithms filling the blanks space that are behind the objects, this has been in Photoshop for quite some time if you want to test it

[@HikikomoriDev]

This is why we still model with image references. Besides, there is no better way to create a clean mesh that way.

[@forgottenalex]

O_O Im beginning to rethink the year of time I've spent crafting my art of 3D and then this pops out of nowhere

[@caulitomaz]

Actually, the algorithm they use is called PatchMatch, you can see the paper here: gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/ . It's from SIGGRAPH 2009.

[@futurfry]

dude thanks. I had totally forgotten.

[@CarlyndraTM]

WHAT A TIME TO BE ALIVE

[@shaderx666]

this will make matte painting extraction a blast!

[@vehbr]

Oh my god... HUMAN MIND FTW! People, you're AWESOME!

[@CrAZman231]

Absolutely amazing software, excited for when it comes out

[@MrFoxInc]

Probably because the branches are horizontal (more or less) and the stick is vertical, and as the lines were already being drawn vertically, the program simply assumed it should ignore the branches because they don't go vertical.

[@djivesp]

Holy cow. This is incredible

[@PedrosaJesus]

I guess it blends in some patterns found in the surrounding areas of the piece you remove. The candle around 2:08 shows an example of failure when the shadow on the wall of the arm that is being moved is filled with the color of the wall. It silhouettes the missing arm with plain colour. The amazing one with the telescope shows a flat horizon and a half town half mountain filling. It shows a fair ammount of "guessing" right there. It may even be configurable Not perfect, but pretty close :)

[@Lavr667]

God damn, i WANT this! Start kickstarter ASAP.

[@Tzctplus]

siggraph isn't a company, and as some other people have pointed out you can't make this in photoshop (or any other software for that matter) in the way it is shown in the software. The person that posted the clip could help to dispel these doubts, nor doing so raises even more suspicion.

[@Deetronic]

AMAZING!!!

[@BjornVerzijlbergh]

Indeed, That is called the "Content Aware" renderer. It also works without "deleting" but by using masks, layers or even the healing brush.

[@rashiga12345]

its explained in the video,they are sampling from around the object and patch it where the object is removed

[@rafagp33]

Para objetos Geometricos parece que va bien, pero, me quedé con las ganas de ver la imagen de una persona para observar la nueva perpestiva!! jejeje..

[@christopherlashwa9081]

For anyone that doesn't already know, this software is already released! Link: cg.cs.tsinghua.edu.cn/3sweep/software/index.html

[@LISTENSUBMITOBEY]

I misread the title and thought we were suddenly able to make food from pictures, ahaha...;; But still, this is pretty awesome.

[@barretthepburn165]

Actually some of us like this idea, It will eliminate people cashing in on simple 3d objects, and give alot more respect to those of us who can build a creative original 3d model from scratch.

[@CT99999]

This demonstrates the power of math, specifically computer graphics, a sub-field of Computer Science.

[@MorningLightMtn]

Good God, that Patch Match Algorithm is glorious Everything is glorious How did they program this?!

[@ChipMatthews]

It's mentioned briefly in the video. It's use a fill routine similar to Photoshop's content aware fill.

[@inuysha360]

3D modeling will never be the same.

[@FrankB1110]

Wow, it you made this open-source incredible things will come!

[@brightico]

Looks like it only builds the objects by mirroring to the other side. The difference is that it also calculates the angle of the object then builds the object at that precise angle. This will only work for objects that can be split in half. Cars, boxes, candles. For instance, I don't think this would work for something like a person unless you had a front and back image to mend together after building the object

[@BruceHeinlein]

This is truly remarkable!

[@oO_ox_O]

smart background texture generation, it's possible, it's basically what you done with a clone tool just automatically, old stuff, works sometimes quite well

[@Chainism]

Wow. This product has potential to change the game, I hope the Beta becomes accessible to the public for free. I'd invest endless hours in this program...

[@palasalchemist]

Ca-ra-ca-lhas. Estupefato estou. Wow. Quem encontra grandes dificuldades em explicar coisas para pessoas com a imaginação desorganizada/ansiosa/ausente sabe o quanto isso é precioso!! Wow. Simply... wow.

[@HeraBek]

It doesn't, they clearly stated that the recreated the background in photoshop first. Photoshop uses some really cool pattern recognition algorithms, but it's still quite a bit of manual work to get it looking nice.

[@jmalmsten]

I remember a test-video from a few years ago where you'd film your object, tracked the footage and then traced the contours of the 2D frames ending up with a texturemapped model. That program just kind of dissapeared since then. But I'd imagine that coupled with multiple images we could do some pretty detailed models. Even 123D would be great if you could actively trace contours instead of just pointing out points and if AE could do this all and then do projection-mapping. That would be great...

[@Usumgallu]

I cannot remember. The video went viral on some music production forums in 2009 or something. I'll post it if I happen to find it (if the website even exists anymore).

[@Hogscraper]

The information is there at the beginning. Its a submission to the SIGGRAPH conference for demonstration. The conference isn't until November so we will have to wait to see whether this is legit unless the submitter links to their work. I really, really want this to be real and an open source plugin:)

[@XandreasX]

Cool. Also so cool that 3 minutes felt like 10 minutes.

[@andiback]

This is that amazing! May be there's a way to compress also video data more efficiently in the future e.g. by identifying moving objects in this way instead of picking up two-dimensional plane clusters! Lot's of bytes in a given movie stream could be saved by only writing down the mathematical envelope curves of objects.

[@diegohiroshiysusvideos2909]

Ay no mames! ya lo habia leido pero no creia que fuera posible tal soft o-o eso es realmente Amazing.

[@korffyboy91]

In 1:13 they mention their algorithm used to kinda "fill in the blanks" behind the objects (the newer versions of Photoshop are capable of the same sort of sorcery btw). So I assume the shadow here was probably vanished within this very algorithm-process.