Selecting 3D Objects With The Mouse Using OpenGL // OpenGL Tutorial #31

In this video we're going to learn how to use render to texture in order to implement selection of a 3D objects using the mouse in OpenGL. This tutorial assumes knowledge about how I load 3D models using Assimp (check out my Assimp tutorial • Loading Models Using A... ).
Make sure to watch all the previous tutorials in the "OpenGL For Beginners" playlist at • OpenGL for Beginners
Please visit ogldev.org to see more of my tutorials on modern OpenGL.
Link to source: github.com/emeiri/ogldev/blob...
OpenGL 4.6 specification: www.khronos.org/registry/Open...
Timecodes
0:00 Intro
0:27 Render To Texture
2:02 Solution Architecture
3:23 Start of code review
3:28 Picking Texture declaration
4:42 Picking Texture implementation
6:50 Switching to GLFW
7:13 Main application code
7:38 Picking Phase
10:00 Render Phase
12:10 Demo and conclusion
Feel free to comment below.
Email: [e-mail]
Facebook: / ogldev-188319114585587
GIthub: github.com/emeiri/ogldev.git
Twitter: @ogldev
One time donations (Paypal): ogldev.org/donate.html
Patreon: / ogldev
Credits:
Sound effects - "Creator Assets" (youtube)
Image by Clker-Free-Vector-Images from Pixabay
Music - "Indian Walk" by Nico Staf (youtube audio library)
Enjoy,
Etay Meiri
#opengl #ogldev #opengtutorials

[@TiramisuCorleone]

It was not easy for me to follow you, but ultimately you have helped me. So thank you!

[@OGLDEV]

Glad I could help!

[@codingexpress4950]

Thank you so much! It helped me a lot. keep up the good work +1👍

[@OGLDEV]

You're welcome!

[@fudgeracoon2529]

Great tutorial !!!

[@OGLDEV]

Thanks!

[@arthurmoyer2657]

Hey, Thanks for this! This is one of the other problems I was having with my RTS code picking the ground. I am(was?) using a pure mathematical technique that failed far too often. I look forward to seeing how well this works. Thanks!

[@jw200]

glReadPixels is very slow

[@arthurmoyer2657]

@@jw200 Do you have a suggestion for an alternative then?

[@OGLDEV]

You're welcome!

[@OGLDEV]

As always, it should be weighed against the alternatives. This solution is very simple to implement which is an advantage. I haven't measured the performance of glReadPixels but I hope that the drivers have some way to optimize it since we are accessing a single pixel.

[@arthurmoyer2657]

@@OGLDEV I would agree I would hope that it wouldn't hurt the performance. I think the issue is it mainly grinds things to a halt while CPU and GPU sync up, so they aren't running async anymore, so glReadPixel will wait until the render is done before continuing, which means the CPU stalls out while waiting. So it would more be about when in the code the glReadPixel is called from what I've read (and even using a different thread to do the glReadPixel call so the main CPU thread doesn't stall). I've also read someone changing the viewport to something very small and then doing the render to get the requested ID position (I assume they offset the projection/view etc. so the mouse position is the center of the viewport and the scene is "shifted" like looking through a spy glass. I've seen others talking about other async buffers, but it was too late at night for me to understand all of it. One way or another, it's a useful direction that you have in your video. :)

[@MrFacciolone]

this was really great! Just one question though, in the glReadPixels() call I see you specified as format GL_RGB_INTEGER, but in the glReadPixels documentation it says only some format are accepted, and GL_RGB_INTEGER is NOT on the list. GL_RGB is though, and I would have thought that specifying the type as GL_UNSIGNED_INT would have been enough, but it turns out that it won't work that way. The correct format for ReadPixels is the one present in your code, GL_RGB_INTEGER. The question is: how did you figure out you had to pass GL_RGB_INTEGER as argument if even the Khronos documentation doesn't list it as accepted? Again thank you for the great series there is much stuff in here I would never have figured out by myself

[@OGLDEV]

Thanks! GL_RGB_INTEGER is indeed not part of the man page of glReadPixels but in the PDF of OpenGL it's not entirely clear. I guess I just played with it until I got it working...

[@francescobrischetto5023]

Hi, I start following your tutorials of websites and youtube because I'll need to use tessellation shader for my master thesis. Could you please update git solutions for tutorials with tessellation shader for Windows? (tutorials 30/31). I tried to setup a VM with Ubuntu but I had some problems in let it recognize the right videocard

[@OGLDEV]

I've just added projects for the two tutorials. I've named them 'Tessellation1' and 'Tessellation2' to avoid confusion with the current series. The first one works ok on my Windows machine with GT710. The second one fails initialization because for some reason it cannot find the uniform gTessellationLevel in the CS even though it is used. On the Linux machine with GTX1650 the same code works so it could be something specific on the Windows machine. Please let me know if this is working for you.

[@MaxMustermann-ey5sc]

This is great! I implemented a simple version of this (no triangle picking just entire objects). I am trying to get drag selection to work now. The big problem is that even if I decide to only read every 5th pixel and x and y, it's still taking a considerable amount of time (way too long even for in editor selection) despite the decreased accuracy. Assuming that the glReadPixel is actually the slow part, is this still a viable solution for drag selections. Is there potentially a better solution than checking the pixels in a loop? Or would you recommend something like a volume intersection?

[@OGLDEV]

Check out my Ray Casting tutorial for a math-only version: kzbin.info/www/bejne/opuYmatspcaSoKc

[@MaxMustermann-ey5sc]

​@@OGLDEV thank you!! I was actualling looking for 'box selection' when dragging the mouse. That explains that I did not find many resources online lol. I've since implemented this using stencils. But now I need to implement manipulation so I'll habe a closer look at your recommendation again ;D Thanks again!!

[@OGLDEV]

You're welcome :-)

[@nerijusvilcinskas7851]

Can I ask why are you not using renderbuffer to render pixel information about objects and chose to render to texture instead? I have a framebuffer class in my engine which supports renderbuffer attachment and gl_depth_component attachment rendering depth buffer to texture. So why would I want to use render to texture vs render to renderbuffer? I heard rendering to them is faster than render to texture

[@OGLDEV]

Laziness. I need to switch at some point.

[@nerijusvilcinskas7851]

@@OGLDEV Thanks for such a quick reply! I guess I will try to make this work using renderbuffer maybe, dealing with framebuffers in general is a bit frustrating so thank you for these amazing tutorials once again!

[@quentinquadrat9389]

How many objects can we pick ? 16 millions (color id) or 256 (limited by Z-buffer accuracy) ? Not sure to understand in what the Z-buffer helps, since with shaders you can pass directly the ID of your object. I have to experiment by myself.

[@OGLDEV]

You need the zbuffer in order to store the info of only the closest pixels. The actual info is stored in the GL_RGBUI32 texture so you have three int32 to specify the draw call, object index and primitive id.

[@quentinquadrat9389]

@@OGLDEV thanks

[@diamond32tutoriales49]

I don't understand what you mean by WorldTrans, is there no code where I can see it? or some repository

[@OGLDEV]

The code is located here: github.com/emeiri/ogldev/blob/master/tutorial31_youtube/ The WorldTrans object is used to maintain the local-to-world transformation of a single entity/instance. You can find the declaration here: github.com/emeiri/ogldev/blob/master/Include/ogldev_world_transform.h and the implementation: github.com/emeiri/ogldev/blob/master/Common/ogldev_world_transform.cpp

[@killereks]

5:44 after adding that filter parameter my texture just becomes transparent and nothing is ever rendered. When i remove it shows up as black (my glClearColor)

[@OGLDEV]

You mean the entire model disappears? This is the picking texture so it should not affect the model texture. Make sure you bind zero to GL_TEXTURE_2D when you finish initializing it. Is my code working correctly? It actually seem to work ok for me without the filter params but I think the texture might be incomplete in this case so it's best to set it.

[@Hiraghm]

I wish there was a version of this tutorial in C

[@OGLDEV]

I'm using a very basic version of C++ so changing it to C should not be a problem.

[@op-vw8sd]

👍

[@OGLDEV]

:-)

[@kishirisu1268]

Selection has zero relation to graphic backend. Make AABB, do raycat, if collide with AABB - you clicked object. What else do you need? Why invent strange texture renderimg methods? Just look how it work in Unity and Unreal.. No offence, but selection by color is the most “strange” approach I see in all GL tutorials. Do not use it in real life, its just demonstration of feature, not for real selection in games.

[@OGLDEV]

I explain ray casting in kzbin.info/www/bejne/opuYmatspcaSoKc. There's a big difference between the two methods. AABB intersection is very quick but also coarse. In many cases you need to use multiple boxes to make it more fine. The glReadPixels method provides you with pixel level granularity and also tells you the specific vertex that you hit (or you are close to). Different cases calls for different solutions.

[@seeenoooh]

Consider the case where you have many objects of various sizes that are in close proximity to one another in the scene. Their AABBs will overlap, making selection accuracy nearly impossible. Also, AABBs could occlude one another, even though their respective objects could be visible, which will make selection accuracy harder. OOBB-Ray intersections are a better solution. Not perfect, but alot better IMO. With that said, and in my humble experience, I'd vote for color picking if you are implementing it for "Editor" purposes. If you are implementing selections for in-game user logic, then OOBB-Ray intersections.

[@OGLDEV]

I agree. When I was working on the tutorial I had Blender in mind where you need to be very accurate and enable selection by vertices, edges and faces.

[@DreadDoom]

You're thinking too narrowly. I use this for my UI library in my game so developers can transform their UI elements however they want, as well as do any cutting (e.g. cutting a hole in the center of a circle using shader math to make a donut-shaped health bar with a texture). With a traditional approach developers would have to manually change the picker function to include their edge-case, "check if mouse is inside circle but not this smaller circle". There are also more complex structures such as Bezier curves that can be dragged around. Using this method I can be sure that whatever the developer decides to do they will always: 1. Press the topmost UI element (the topmost element will automatically replace the below ones) 2. Perfect picking for any shape generated in the fragment shader.

[@killereks]

Unity and Unreal literally use this to detect clicks.

[@ducroit]

Hi, I encountered some run errors with tutorials 31 to 32 (piking), 34 to 38 (shadow) and 40(Skeletal animation). The tutorial 33(sprite) is run fine. First I had the error of the texture2D function at line 319 and 513 in lighting_new.fs, I saw that the function is deprecated, so I put the texture function instead. "Error compiling '../Common/Shaders/lighting_new.fs': '0:319(21): error: no function with name 'texture2D' 0:319(21): error: type mismatch 0:319(21): error: operands to arithmetic operators must be numeric 0:513(19): error: no function with name 'texture2D' 0:513(19): error: operands to arithmetic operators must be numeric 0:518(40): warning: `TempColor' used uninitialized" As a result, I had the error with EdgeDistance0 in lighting_new.fs: "Error linking shader program: 'error: vertex shader output `EdgeDistance0' specifies no interpolation qualifier, but fragment shader input specifies noperspective interpolation qualifier" While trying to remove the noperspective qualification, I had the following error : "!!! Debug callback !!! Debug message: id 1, GL_INVALID_OPERATION in glDrawElements Message source: API Error type: Error Severity: High" I don’t know if the error is from my opengl (4.6) and glfw(3.3.6) versions. Could you please try to check out on your side?

[@OGLDEV]

I added the missing 'noperspective' to all the vertex shaders where it was missing and replaced texture2D with texture. Please let me know if you still have problems with any of these tutorials. Thanks for the feedback!

[@ducroit]

@@OGLDEV Thank you for responding so quickly! But I still have problems when I try to run these tutorials after pull your update : "Invalid shader program: 'active samplers with a different type refer to the same texture image unit' Warning! Unable to get the location of uniform 'gWireframeColor' !!! Debug callback !!! Debug message: id 1, GL_INVALID_OPERATION in glDrawElements Message source: API Error type: Error Severity: High"

[@OGLDEV]

I pushed a fix to the 'active samplers...' error. Let's see if it improves something.

[@ducroit]

@@OGLDEV Everything works now! Thank you very much, I learned a lot with your tutorials.

[@OGLDEV]

Glad to hear!