02: Simple Visual Servoing in V-Rep / CoppeliaSim | Tutorial

[@Intars5d]

I would go and say excellent explanation. Not rushed, well explained, nuances explored and mentioned. These camera and filter tools, as well as setup was missing puzzle for one robotics laboratory/homework assignment. You gave me light at the end of the tunnel :l

[@DemetrisStavrou]

Thanks for the V-Rep videos. They are really helpful!

[@RoboticsWithHrithik]

Image Processing and triggering (show filter dialog) is not showing in scene object properties. My vrep version is 4.0. Please help!!!

[@johnminyu9426]

great work! It helps me a lot! Thank you!

[@thoretrompf3552]

Hi I use version 3.2.2 I try to follow but my dummy doesn't move along the path. No little read point moves along the path. If I open the 'Scene Object Properties' I can't see the 'Motion properties'. If I drop the dummy to the path and check 'Follow parent path (only direct parent)' at the properties the dummy doesn't move to the path. What I doing wrong? Please help me.

[@ghettoreptile]

hi Nikolai, I did what you told Alastair Kuek, however my simulation is still not working. i keep getting these errors Lua runtime error: [string "SCRIPT cam"]:46: attempt to index global 'pack3' (a nil value) stack traceback: [string "SCRIPT cam"]:63: in function [string "SCRIPT cam"]:46: in function [C]: in function 'xpcall' [string "SCRIPT cam"]:63: in main chunk not sure what to do at all can you help me, thanks

[@riamycha]

Hi Nikolai, Thanks for the interesting video, it's very helpful. Still, I have a question: If we suppose that instead of having that vision sensor, we have a robot equipped with a camera (not a vision sensor), and once that robot detects a static object (the red sphere for example), it directly goes towards it. How can we do that?

[@NikolaiK]

Your question is pretty broad and will require more specifics (as to the type of robot). But in its simplest form: Once you detect the object, you can find a vector from your robot to the target. This vector can tell you whether your robot has to turn towards the target (if the target is to the side of the robot). You can also use the size of the target in the image to determine if the robot should drive towards it or if it currently at the correct distance.

[@riamycha]

Nikolai K. The robot is a differential drive robot, for example the dr12 robot already given as a model in V-REP. The dr12 is already equipped with a camera, is there a way to get information about what objects are present in the environment, through this camera? (A box for example) Then go towards it and push it to a target location. I want to resolve the box-pushing problem having the dr12 robot, a box, and a target locations changing for each simulation, any help please?

[@NikolaiK]

Myriam Riamy Hello, This is a very complex problem and it is pretty much impossible to explain every step in this comment window, because it combines object detection, path planning, and robot kinematics, but I will attempt to give you a few pointers that may help you get started. You should take a look at the filter dialog, where you have the option to get a "depth image" from an image sensor, which might be useful when you are trying to navigate relative to the cube. In the Scene Object Properties you also have the option under "Entity to render" to only detect the cube and disregard everything else. That way you won't clutter up your image with other objects.

[@FrankLopezx]

i was hoping to learn how to grab objects using servoing , is this possible thru this program? and in my case its a humanoid with five fingers so how do we go about bringing in my own model? none of the primitives robots have fingers let alone five like human articulation

[@anik52168]

M getting the following error while implementing this tutorial. Lua runtime error: [string "SCRIPT Cam"]:61: attempt to index global 'pack2' (a nil value) stack traceback: [string "SCRIPT Cam"]:61: in main chunk Error is in the following command xtarget=pack2[5] ytarget=pack2[6] Can anybody help me to resolve this error. Thanks in advance

[@sumairamanzoor2482]

Use if (pack2) in the script if (pack2) then xtarget=pack2[5] ytarget=pack2[6] simAuxiliaryConsolePrint(out,string.format(' x= %0.2f, y= %0.2f',xtarget, ytarget)) end

[@emjay9199]

I tried your method, Sumaira, but I am now getting another error: Lua runtime error: [string "SCRIPT Cam"]:26: bad argument #2 to 'format' (number expected, got nil) stack traceback: [C]: in function 'format' [string "SCRIPT Cam"]:26: in main chunk Any feedback would be much appreciated. Thank you.

[@emjay9199]

while (simGetSimulationState()~=sim_simulation_advancing_abouttostop) do result, pack1, pack2=simReadVisionSensor(Cam) if (pack1) then xTarget=pack1[5] end if (pack2) then yTarget=pack2[6] end simAuxiliaryConsolePrint(out,string.format(" x: %0.2f, y: %0.2f", xTarget, yTarget)) end

[@AliAnwar267]

Edit mode is not available in the latest version of v-rep

[@binli3539]

You can assess that by clicking the seventh toolbar button in the left panel of the V-Rep. My v-rep version is 3.6.2

[@flavioyamamoto9321]

Hi Nikolai. Very nice video! Is it possible to export the content of a vision sensor as an image (png or jpg format)? Suppose that a robot wants to store information (pictorial) about blue balls. Then, if the robot sees a "blue ball" the content of the vision sensor is stored as a jpg image (one snapshot).

[@sambitroy7280]

Hello. I am using vrep 3.2.2. I am trying to detect a disc using vision sensor. As suggested I am using the Binary work image and trigger filter. Still the result is coming as 0. I coded it as per the video

[@anukritisingh7204]

Hey any idea how to do it with ROS in Gazebo?

[@arkaye]

can i give two separate links of the same robot to follow two different paths at the same time?? I am not able to achieve it

[@user-hr9hg2eh5c]

life saverrrrrr

[@deepakajayakumar2806]

Hi Nikolai, My version is V-Rep Pro Edu 3.3.0. I've done the changes regarding "blob detection on work image" and managed to get the x,y values. But the code "simSetJointTargetVelocity(yaw, 1*(0.5-xtarget))" is not working. No errors are shown but the joints are not rotating. I've attached a child script to the 'Path' for moving the object along path as some options for changing the Target Velocity is disabled in new versions. Is that a problem??? The code is below moveToPosition=function(_pathHandle,startPos,endPos,maxVel,maxAccel,maxJerk) local posVelAccel={startPos,0,0} local targetPosVel={endPos,0} local maxVelAccelJerk={maxVel,maxAccel,maxJerk} local rmlHandle=simRMLPos(1,0.0001,-1,posVelAccel,maxVelAccelJerk,{1},targetPosVel) local newPosVelAccel=posVelAccel while true do local result,newPosVelAccel=simRMLStep(rmlHandle,simGetSimulationTimeStep()) simSetPathPosition(_pathHandle,newPosVelAccel[1]) -- update the path's intrinsic position if result~=0 then break end simSwitchThread() end simRMLRemove(rmlHandle) end pathHandle=simGetObjectHandle("Path") simSetPathTargetNominalVelocity(pathHandle,0) -- for backward compatibility moveToPosition(pathHandle,0,3.748,0.1,0.01,0.01)

[@saifuddinmohit8301]

I am having the same problem! Did you find a way to make it work?

[@manandreiioan]

Hello, Thanks for this video tutorial. Can anyone tell me where is saved the RGB information? In my simulation I have 3 different coloured pieces, and I want to grab a piece, let's say blue, with a robot and release in blue box. So I have 3 path, but I don't know what conditions to use in if statement. Any helps is welcome!

[@gabrielecianetti8557]

Hello, thanks for this video tutorial, it is very interesting and helpful. I have one question, could you please help? I am trying to do the same using the quadricopter instead of revolute joints. In V-Rep there is the corrispective of simSetJointTargetVelocity for the quadricopter. Since in V-rep it is possible to move the quadricopter moving its target (green sphere), I tried to set up the position of that target to allow the quadricopter to follow the red blobs, but till now I didn't get any good result. Do you know if there is an easier way to do it? Thanks

[@alak5665]

Hi Nicholai! i have followed your steps. however, my result for simReadVisionSensor (cam) is 0. does that mean that there is no detection of the blob?

[@NikolaiK]

Hello Alastair, Take a look at the Function List here: www.coppeliarobotics.com/helpFiles/en/apiFunctions.htm#simReadVisionSensor It is very useful for determining the output and input parameters for a function. When the function returns 0, then it means that nothing was detected. You should try verifying what the camera sees by viewing the output in a "Floating View"

[@NikolaiK]

kugelfang killaruna Alastair Kuek Hello Alastair, Kugelfang turns out there have been some recent updates with V-rep, which have changed how some vision sensor aspects behave. Unfortunately, the "blob detection on work image" does not work as previously with this new update (I have been having issues myself, but will keep trying to get it to work). However you can use the "Binary work image and trigger" filter instead. Please note that this filter is not good if you plan on using more than one target object, since it will give you the center of mass of all objects. You will have to change the settings of the Filter to the following: Proportion 0.5 Tolerance +/- 0.5 X-Position 0.5 Tolerance +/- 0.5 Y-Position 0.5 Tolerance +/- 0.5 Roundness Threshold 0.5 The dialog will also tell you that the center of mass x and y values are in the packet at index 2 and 3, respectively. Below the simReadVisionSensor function you should add a condition that checks if detection was successful. The rest of the code should be added within the if statement: if result>0 then posx=pack2[2] posy=pack2[3] --the rest of the code here end

[@kugelfangkillaruna]

Nikolai K. thanks sir.

[@FrankLopezx]

ok i try to follow along but i have the newest version and it seems like this video is outdated for new users as we cant just download that specific version you have

+Frank Lopez older versions are still available at coppeliarobotics.com

[@FrankLopezx]

oh nice then ill download the ver being used in this video, thanks for the file extension import, i have alot of good androids i design in autocad and i can do obj so that be fine for me to bring in here, but was i right in what i was doing ? when i bring in my obj file i can click and drag it from the hierarchy to and drop in the dummy>path ? so that it can follow dummy? rather than using the red ball that he made in this video from the base primitives?

[@buddhikawarnakulasooriya8692]

I'm a new to this field. Can someone please tell me, is it possible to implement these simmulations in a real robot via VREP?

[@bisma5586]

i try to follow but error ...Simulation started. Lua runtime error: [string "SCRIPT cam"]:9: attempt to index global 'pack2' (a nil value)

[@bisma5586]

threadFunction=function() out=simAuxiliaryConsoleOpen("debug",8,1) yaw=simGetObjectHandle("yaw_joint") pitch=simGetObjectHandle("pitch_joint") cam=simGetObjectHandle("cam") result,pack1,pack2=simReadVisionSensor(cam) xtarget=pack2[5] --ytarget=result[6] simAuxiliaryConsolePrint(out,string.format("x: %0.2f,y: %0.2f",xtarget,ytarget)) simSetJointTargetVelocity(yaw_joint,1*(0.5-xtarget)) simSetJointTargetVelocity(pitch_joint,1*(0.5-ytarget)) -- Put your thread code here (initialization and clean-up code should not be in here) -- Some EXTERNAL commands (e.g. socket commands provided by Lua libraries) -- might appear as blocking to the simulator. In that case, you can define -- a non-blocking section as following example shows: -- -- simSetThreadIsFree(true) -- Start of the non-blocking section -- -- Following 2 lines are meant as an example of EXTERNAL blocking commands: -- http = require("socket.http") -- print(http.request("www.google.com")) -- -- simSetThreadIsFree(false) -- End of the non-blocking section -- -- While in a non-blocking section, try to avoid calling sim-functions. Also -- never forget to close the blocking section, otherwise V-REP will hang. -- Make sure you read the information related to the simSetThreadIsFree -- API function in V-REP's documentation. -- If you wish to synchronize a threaded loop with each simulation pass, -- set the thread switch timing in the initialization phase of this script -- to the maximum (200), and manually switch thread here with the -- simSwitchThread() command. -- ( simSwitchThread() will suspend this script's execution until next -- simulation pass, i.e. until the simulation time has changed ) -- -- Following example illustrates this: -- -- while simGetSimulationState()~=sim_simulation_advancing_abouttostop do -- local p=simGetObjectPosition(objHandle,-1) -- p[1]=p[1]+0.001 -- simSetObjectPosition(objHandle,-1,p) -- simSwitchThread() -- end end -- Put some initialization code here: simSetThreadSwitchTiming(2) -- Default timing for automatic thread switching -- Here we execute the regular thread code: res,err=pcall(threadFunction) if not res then simAddStatusbarMessage('Lua runtime error: '..err) end -- Put some clean-up code here: threadFunction=function() out=simAuxiliaryConsoleOpen("debug",8,1) yaw=simGetObjectHandle("yaw_joint") pitch=simGetObjectHandle("pitch_joint") cam=simGetObjectHandle("cam") result,pack1,pack2=simReadVisionSensor(cam) xtarget=pack2[5] --ytarget=result[6] simAuxiliaryConsolePrint(out,string.format("x: %0.2f,y: %0.2f",xtarget,ytarget)) simSetJointTargetVelocity(yaw_joint,1*(0.5-xtarget)) simSetJointTargetVelocity(pitch_joint,1*(0.5-ytarget)) -- Put your thread code here (initialization and clean-up code should not be in here) -- Some EXTERNAL commands (e.g. socket commands provided by Lua libraries) -- might appear as blocking to the simulator. In that case, you can define -- a non-blocking section as following example shows: -- -- simSetThreadIsFree(true) -- Start of the non-blocking section -- -- Following 2 lines are meant as an example of EXTERNAL blocking commands: -- http = require("socket.http") -- print(http.request("www.google.com")) -- -- simSetThreadIsFree(false) -- End of the non-blocking section -- -- While in a non-blocking section, try to avoid calling sim-functions. Also -- never forget to close the blocking section, otherwise V-REP will hang. -- Make sure you read the information related to the simSetThreadIsFree -- API function in V-REP's documentation. -- If you wish to synchronize a threaded loop with each simulation pass, -- set the thread switch timing in the initialization phase of this script -- to the maximum (200), and manually switch thread here with the -- simSwitchThread() command. -- ( simSwitchThread() will suspend this script's execution until next -- simulation pass, i.e. until the simulation time has changed ) -- -- Following example illustrates this: -- -- while simGetSimulationState()~=sim_simulation_advancing_abouttostop do -- local p=simGetObjectPosition(objHandle,-1) -- p[1]=p[1]+0.001 -- simSetObjectPosition(objHandle,-1,p) -- simSwitchThread() -- end end -- Put some initialization code here: simSetThreadSwitchTiming(2) -- Default timing for automatic thread switching -- Here we execute the regular thread code: res,err=pcall(threadFunction) if not res then simAddStatusbarMessage('Lua runtime error: '..err) end -- Put some clean-up code here:

note that this one only works in 3.2.1 but not in 3.2.2

[@FrankLopezx]

+Ernő Horváth i just noticed this for myself, is their another tutorial i can follow more suited for our ver? i actually follow most of what was said but i couldn't get the dummy to move, and to top it off rather then making a ball primitive i used the asimo looking robot from the primitive section to place on top of the dummy as the dummy was on top of the path in the scene hierarchy and the bot just started walking not following the dummy "not that the dummy was moving " but not even the path i created but its own path not defined by me, hmmm is there a way to bring my own cad models? to use rather than this robots or the shape primitives ?

+Frank Lopez i think you can import dxf, obj, 3ds and obj

[@NikolaiK]

+Frank Lopez Hello Frank, take a look at the importing shapes tutorial on the V-rep website for help on importing shapes and how they fit relative to each other: www.coppeliarobotics.com/helpFiles/en/rigidBodyTutorial.htm

[@garry_ayra]

how about 3.6.2