Sarin Suriyakoon - 6 months ago 12
Java Question

# How to detect robot direction from Image?

I've developing the robot which can run in the corn plant and guided by compass sensors but I want to apply the camera as the eye of robot and use the image processing to detect error angle of movement.

This is the image examples.

processed image

raw image

segmented image

I use this following step

1. Step 1 : The current technique that I use is convert color value to HSV modified from this code

2. Step 2 : So It will detect selected color which is the brown or dirt color then I collect the most left and right of brown or selected color of each image row in two array(a red point).

3. Step 3 : I plot a 2 linear regression line as a blue point and calculate intersection point as a pink point

4. Step 4 : Draw the green line to compare the pink point with another images. I'm not sure what to do with this green line yet

5. The problem is the dirt or brown color is exist between the corn leaf too then I make my code to miss a calculation

The question is how to filter out the brown pixel that is between the corn leaf or another area that is not in the corn path? Which algorithm or methodology that I should study or apply in this problem?

EDIT1 : Using Spektre's answer and It looks better

Here is the result after I apply it with JAVA+Boofcv

• Step 1 : Thresholding or Color Segmentation

• Step 2 : Blured(use Gaussian and Median filter)

• Step 3 : Plot Linear Regression

Full Code Here

LinearRegression Class

10 example images with the same process

I would:

1. create mask of brown stuff

Just treshold `H,S` and ignore `V`, You already have this. I use integer `255` instead of color (Blue) for latter computation.

This will remove small clusters of wrongly selected parts. After this you should treshold the mask again because the mask value will be a bit smaller then `255` unless you got fully selected areas. The bigger the area the bigger the value (closer to `255`). I treshold with `>=150`

3. scan the mask by horizontal lines

4. for each line find the center of gravity of all selected pixels

After blurring and tresholding again the used mask is in Aqua. So compute the average point `x` coordinate of all masked pixels in each line. This point is marked with White.

5. regress line through all of the centers of gravity

I use mine approximation search for this but you can use any regression you want. The regressed line is marked with Red

I used line equation `x=x0+y*dx` where `y=<0,pic1.ys>`. And search the solution on intervals:

``````x0=<-pic1.xs,+2*pic1.xs>
dx=<-10,+10>
``````

Where `pic1.xs,pic1.ys` is the image resolution. So as you can see I do not cover full range of angles but I think this would not work on those edge cases anyway (near horizontal directions). For such cases you should do this on the Vertical lines instead and use `x=y0+x*dy` instead.

Here C++ source I did this with:

``````    picture pic0,pic1;
// pic0 - source img
// pic1 - output img
int x,y,h,s,v,px,pn,*p;
color c;
// copy source image to output
pic1=pic0;
pic1.save("cornbot0.png");
for (y=0;y<pic1.ys;y++)             // scan all H lines
for (x=0;x<pic1.xs;x++)            // scan actual H line
{
c=pic1.p[y][x];                 // get pixel color
rgb2hsv(c);                     // in HSV
h=WORD(c.db[picture::_h]);
s=WORD(c.db[picture::_s]);
v=WORD(c.db[picture::_v]);
// Treshold brownish stuff
if ((abs(h- 20)<10)&&(abs(s-200)<50)) c.dd=255; else c.dd=0;
pic1.p[y][x]=c;
}
pic1.save("cornbot1.png");
pic1.smooth(10);                    // blur a bit to remove small clusters as marked
pic1.save("cornbot2.png");

// compute centers of gravity
p=new int[pic1.ys];                 // make space for points
for (y=0;y<pic1.ys;y++)             // scan all H lines
{
px=0; pn=0;                     // init center of gravity (avg point) variables
for (x=0;x<pic1.xs;x++)         // scan actual H line
if (pic1.p[y][x].dd>=150)      // use marked points only
{
px+=x; pn++;                // add it to avg point
pic1.p[y][x].dd=0x00004080; // mark used points (after smooth) with Aqua
}
if (pn)                         // finish avg point computation
{
px/=pn;
pic1.p[y][px].dd=0x00FFFFFF;// mark it by White
p[y]=px;                    // store result for line regression
} else p[y]=-1;             // uncomputed value
}

// regress line
approx x0,dx;
double ee;
for (x0.init(-pic1.xs,pic1.xs<<1,100,3,&ee); !x0.done; x0.step())   // search x0
for (dx.init(-10.0   ,+10.0     ,1.0,3,&ee); !dx.done; dx.step())  // search dx
for (ee=0.0,y=0;y<pic1.ys;y++)                                    // compute actua solution distance to dataset
if (p[y]!=-1)                                                    // ignore uncomputed values (no brown stuff)
ee+=fabs(double(p[y])-x0.a-(double(y)*dx.a));
// render regressed line with Red
for (y=0;y<pic1.ys;y++)
{
x=double(x0.aa+(double(y)*dx.aa));
if ((x>=0)&&(x<pic1.xs))
pic1.p[y][x].dd=0x00FF0000;
}
pic1.save("cornbot2.png");
delete[] p;
``````

I use my own `picture` class for images so some members are:

• `xs,ys` size of image in pixels
• `p[y][x].dd` is pixel at `(x,y)` position as 32 bit integer type
• `p[y][x].dw[2]` is pixel at `(x,y)` position as 2x16 bit integer type for 2D fields
• `p[y][x].db[4]` is pixel at `(x,y)` position as 4x8 bit integer type for easy channel access
• `clear(color)` - clears entire image
• `resize(xs,ys)` - resizes image to new resolution
• `bmp` - VCL encapsulated GDI Bitmap with Canvas access
• `smooth(n)` - fast blur the image `n` times

You can further improve this with segmentation (removing small clusters) based on area and position. Also you can ignore too big peaks in avg points between neighbors. Also you can detect sky and ignore whole area where sky is present.

[edit1] smooth

This is how my smooth looks like:

``````void picture::smooth(int n)
{
color   *q0,*q1;
int     x,y,i,c0[4],c1[4],c2[4];
bool _signed;
if ((xs<2)||(ys<2)) return;
for (;n>0;n--)
{
#define loop_beg for (y=0;y<ys-1;y++){ q0=p[y]; q1=p[y+1]; for (x=0;x<xs-1;x++) { dec_color(c0,q0[x],pf); dec_color(c1,q0[x+1],pf); dec_color(c2,q1[x],pf);
#define loop_end enc_color(c0,q0[x  ],pf); }}
if (pf==_pf_rgba) loop_beg for (i=0;i<4;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u8(c0[i]);  } loop_end
if (pf==_pf_s   ) loop_beg                   { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])/ 4; clamp_s32(c0[0]); } loop_end
if (pf==_pf_u   ) loop_beg                   { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])>>2; clamp_u32(c0[0]); } loop_end
if (pf==_pf_ss  ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])/ 4; clamp_s16(c0[i]); } loop_end
if (pf==_pf_uu  ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u16(c0[i]); } loop_end
#undef loop_beg
#define loop_beg for (y=ys-1;y>0;y--){ q0=p[y]; q1=p[y-1]; for (x=xs-1;x>0;x--) { dec_color(c0,q0[x],pf); dec_color(c1,q0[x-1],pf); dec_color(c2,q1[x],pf);
if (pf==_pf_rgba) loop_beg for (i=0;i<4;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u8(c0[i]);  } loop_end
if (pf==_pf_s   ) loop_beg                   { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])/ 4; clamp_s32(c0[0]); } loop_end
if (pf==_pf_u   ) loop_beg                   { c0[0]=(c0[0]+c0[0]+c1[0]+c2[0])>>2; clamp_u32(c0[0]); } loop_end
if (pf==_pf_ss  ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])/ 4; clamp_s16(c0[i]); } loop_end
if (pf==_pf_uu  ) loop_beg for (i=0;i<2;i++) { c0[i]=(c0[i]+c0[i]+c1[i]+c2[i])>>2; clamp_u16(c0[i]); } loop_end
#undef loop_beg
#undef loop_end
}
}
``````

It simply weight average 3 pixels

``````(x,y)=(2*(x,y)+(x-1,y)+(x,y-1))/4
``````

and after that do the same with

``````(x,y)=(2*(x,y)+(x+1,y)+(x,y+1))/4
``````

to avoid shifting of image. Then this whole thing is looped `n` times and that is all. You can ignore the clamp and pixel-format options in this case it is `pf==_pf_rgba` but it uses only single channel anyway ... `dec_color,enc_color` just unpack,pack color channels into/from array of variables to avoid truncating and overflow issues on 8 bit channels and also to format/simplify the code a bit better (for different pixel format support)

btw smooth base is the same as convolution with

``````0.00 0.25 0.00
0.25 0.50 0.00
0.00 0.00 0.00
``````

and

``````0.00 0.00 0.00
0.00 0.50 0.25
0.00 0.25 0.00
``````
Source (Stackoverflow)