Real time paper sheet detection using OpenCV in Android

Question

First I really appreciate the the blog http://opencv-code.com/tutorials/automatic-perspective-correction-for-quadrilateral-objects/

I'm following this, but I'm not processing the image (Used in the blog above) but, rather I'm trying to detect the paper sheet of any size (A4, legal or any normal rectangular paper size) in real time camera preview.

Problem I'm getting is after "Expanding the hough line segments to fit the image" I'm getting large number of hough lines hence getting more than 4 intersecting points (not == 4).

like this http://s17.postimg.org/i0a57fb8v/device_2015_04_07_171351.png

How can I remove the rest invalid points, I just need the 4 corners points? I'm using OpenCV Library for Android

Please mainly focus on detectPaperSheet() method. Here is my code:

 package org.opencv.samples.tutorial1;

import java.util.ArrayList;

import org.opencv.android.BaseLoaderCallback;
import org.opencv.android.CameraBridgeViewBase;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2;
import org.opencv.android.LoaderCallbackInterface;
import org.opencv.android.OpenCVLoader;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.Point;
import org.opencv.core.Scalar;
import org.opencv.core.Size;
import org.opencv.imgproc.Imgproc;

import android.app.Activity;
import android.os.Bundle;
import android.util.Log;
import android.view.Menu;
import android.view.MenuItem;
import android.view.SurfaceView;
import android.view.WindowManager;
import android.widget.Toast;

public class Tutorial1Activity extends Activity implements
        CvCameraViewListener2 {
    private static final String TAG = "OCVSample::Activity";

private CameraBridgeViewBase mOpenCvCameraView;
private boolean mIsJavaCamera = true;
private MenuItem mItemSwitchCamera = null;

private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
    @Override
    public void onManagerConnected(int status) {
        switch (status) {
        case LoaderCallbackInterface.SUCCESS: {
            Log.i(TAG, "OpenCV loaded successfully");
            mOpenCvCameraView.enableView();
        }
            break;
        default: {
            super.onManagerConnected(status);
        }
            break;
        }
    }
};

public Tutorial1Activity() {
    Log.i(TAG, "Instantiated new " + this.getClass());
}

/** Called when the activity is first created. */
@Override
public void onCreate(Bundle savedInstanceState) {
    Log.i(TAG, "called onCreate");
    super.onCreate(savedInstanceState);
    getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);

    setContentView(R.layout.tutorial1_surface_view);

    if (mIsJavaCamera)
        mOpenCvCameraView = (CameraBridgeViewBase) findViewById(R.id.tutorial1_activity_java_surface_view);
    else
        mOpenCvCameraView = (CameraBridgeViewBase) findViewById(R.id.tutorial1_activity_native_surface_view);

    mOpenCvCameraView.setVisibility(SurfaceView.VISIBLE);

    mOpenCvCameraView.setCvCameraViewListener(this);
}

@Override
public void onPause() {
    super.onPause();
    if (mOpenCvCameraView != null)
        mOpenCvCameraView.disableView();
}

@Override
public void onResume() {
    super.onResume();
    OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_9, this,
            mLoaderCallback);
}

public void onDestroy() {
    super.onDestroy();
    if (mOpenCvCameraView != null)
        mOpenCvCameraView.disableView();
}

@Override
public boolean onCreateOptionsMenu(Menu menu) {
    Log.i(TAG, "called onCreateOptionsMenu");
    mItemSwitchCamera = menu.add("Toggle Native/Java camera");
    return true;
}

@Override
public boolean onOptionsItemSelected(MenuItem item) {
    String toastMesage = new String();
    Log.i(TAG, "called onOptionsItemSelected; selected item: " + item);

    if (item == mItemSwitchCamera) {
        mOpenCvCameraView.setVisibility(SurfaceView.GONE);
        mIsJavaCamera = !mIsJavaCamera;

        if (mIsJavaCamera) {
            mOpenCvCameraView = (CameraBridgeViewBase) findViewById(R.id.tutorial1_activity_java_surface_view);
            toastMesage = "Java Camera";
        } else {
            mOpenCvCameraView = (CameraBridgeViewBase) findViewById(R.id.tutorial1_activity_native_surface_view);
            toastMesage = "Native Camera";
        }

        mOpenCvCameraView.setVisibility(SurfaceView.VISIBLE);
        mOpenCvCameraView.setCvCameraViewListener(this);
        mOpenCvCameraView.enableView();
        Toast toast = Toast.makeText(this, toastMesage, Toast.LENGTH_LONG);
        toast.show();
    }

    return true;
}

public void onCameraViewStarted(int width, int height) {
}

public void onCameraViewStopped() {
}

public Mat onCameraFrame(CvCameraViewFrame inputFrame) {

    return detectPaperSheet(inputFrame.rgba());

}

private Mat detectPaperSheet(Mat original_image) {
    Mat imgSource = original_image;
    Mat untouched = original_image.clone();

    // Converting to grayscale
    Mat mHsvMat = new Mat(imgSource.rows(), imgSource.cols(),
            CvType.CV_8UC1, new Scalar(0));

    Imgproc.cvtColor(imgSource, mHsvMat, Imgproc.COLOR_BGRA2GRAY, 4);

    // apply gaussian blur to smoothen lines of dots
    Imgproc.GaussianBlur(imgSource, imgSource, new Size(11, 11), 0);

    // Applying Canny
    Imgproc.Canny(mHsvMat, mHsvMat, 80, 100);

    Mat lines = new Mat();
    int threshold = 100;
    int minLineSize = 150;
    int lineGap = 40;

    Imgproc.HoughLinesP(mHsvMat, lines, 1, Math.PI / 180, threshold,
            minLineSize, lineGap);

    // Expanding the Lines To Image Width and Height

    ArrayList<Point> corners = new ArrayList<Point>();

    for (int x = 0; x < lines.cols(); x++) {

        double[] vec = lines.get(0, x);
        double[] val = new double[4];

        val[0] = 0;
        val[1] = ((float) vec[1] - vec[3]) / (vec[0] - vec[2]) * -vec[0]
                + vec[1];
        val[2] = imgSource.cols();
        val[3] = ((float) vec[1] - vec[3]) / (vec[0] - vec[2])
                * (imgSource.cols() - vec[2]) + vec[3];

        lines.put(0, x, val);
    }

    for (int x = 0; x < lines.cols(); x++) {
        double[] vec = lines.get(0, x);
        double x1 = vec[0], y1 = vec[1], x2 = vec[2], y2 = vec[3];
        Point start = new Point(x1, y1);
        Point end = new Point(x2, y2);

        Core.line(imgSource, start, end, new Scalar(255, 0, 0), 1);

    }

    for (int i = 0; i < lines.cols(); i++) {
        for (int j = i + 1; j < lines.cols(); j++) {
            Point pt = computeIntersect(lines.get(0, i), lines.get(0, j));
            if (pt.x >= 0 && pt.y >= 0)
                corners.add(pt);
        }
    }

    if (corners.size() < 4) {
        Log.e("Corner < 4", corners.size() + " |");
        return untouched;
    } else {
        Log.e("Corner > 4", corners.size() + " |");
    }

    //Mat cornerPoints = new Mat();

    for (int j = 0; j < corners.size(); j++) {

        Core.circle(imgSource,
                new Point(corners.get(j).x, corners.get(j).y), 20,
                new Scalar(0, 0, 255), 2);
    }

    return imgSource;
}

private static Point computeIntersect(double[] a, double[] b) {
    double x1 = a[0], y1 = a[1], x2 = a[2], y2 = a[3], x3 = b[0], y3 = b[1], x4 = b[2], y4 = b[3];
    double denom = ((x1 - x2) * (y3 - y4)) - ((y1 - y2) * (x3 - x4));
    Point pt = new Point();
    if (denom != 0) {

        pt.x = ((x1 * y2 - y1 * x2) * (x3 - x4) - (x1 - x2)
                * (x3 * y4 - y3 * x4))
                / denom;
        pt.y = ((x1 * y2 - y1 * x2) * (y3 - y4) - (y1 - y2)
                * (x3 * y4 - y3 * x4))
                / denom;
        return pt;
    } else
        return new Point(-1, -1);
}

}

Were you able to do this? I am working on something similar and I am stuck — Gurankas
– Gurankas, Commented Sep 21, 2017 at 5:59

dianull · Accepted Answer · 2015-04-08 15:14:50Z

2

I remember poly approximation was of some help for me in such case (detecting rectangular shape). However I used it with findContours method (with CV_CHAIN_APPROX_SIMPLE mode), as it seemed to produce better results than Hough lines.

[edit] I did it in C++ with JNI, but in Java I think it should look something like this:

    Mat srcImg; //you may want to apply Canny or some threshold before searching for contours
    List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
    Mat hierarchy;
    Imgproc.findContours(srcImg, contours, hierarchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
    MatOfPoint2f mat2fsrc, mat2fdst;
    Scalar color =  new Scalar(250, 250, 255);

    for (int i = 0; i < contours.size(); i++) {
        contours.get(i).convertTo(mat2fsrc, CvType.CV_32FC2);
        Imgproc.approxPolyDP(mat2fsrc, mat2fdst, 0.01 * Imgproc.arcLength(mat2fsrc, true), true);
        mat2fdst.convertTo(contours.get(i), CvType.CV_32S);
        Imgproc.drawContours(srcImg, contours, i, color, 2, 8, hierarchy, 0, new Point());
    }

edited Apr 8, 2015 at 15:14

answered Apr 7, 2015 at 13:22

dianull

531 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Qadir Hussain Over a year ago

thanks for response. can you please provide sample code in java ?

Collectives™ on Stack Overflow

Real time paper sheet detection using OpenCV in Android

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related