expeV2.tex

\section{Experimental validation}

\label{sec:expe}

%The main goal of this work is to detect straight segments enriched with a
%quality measure through the associated width parameter.
%In lack of available reference tool (line detector and ground truth data)
%dealing with the thickness parameter, the evaluation stage first aims
%at quantifying the benefits of the new detector compared to the previous
%one in unsupervised context on synthetic data considered as a ground truth.
%Then comparisons are made with a well established recent detector
%\cite{LuAl15} in order to check that global performance (processing time,
%ground truth covering, detected lines count and mean length) are not
%degraded.

%The process flow of the former method (initial detection followed by two
%refinement steps) is integrated as an option into the code of the new
%detector, so that both methods rely on the same optimized basic routines.
%During all these experiments, only the blurred segment size and its
%orientation compared to the initial stroke are tested at the end of
%the initial detection, and only the segment size is tested at the end
%of the fine tracking stage.
%All other tests, sparsity or fragmentation, are disabled.
%The segment minimal size is set to 5 pixels, except where precised.

At first, the benefits of introduced concepts are evaluated through a
comparison of the performance of both versions of the detector
on a set of 1000 synthesized images containing 10 randomly
placed input segments with random width between 2 and 5 pixels.
As these values are controlled, these images can be considered as a
ground truth.
The absolute value of the difference of each found segment to its
matched input segment is measured.
%On these synthetic images, the numerical error on the gradient extraction
%biases the line width measures. This bias was first estimated using 1000
%images containing only one input segment (no possible interaction)
%and the found value (1.4 pixel) was taken into account in the test.
\RefTab{tab:synth} shows
slightly better width and angle measurements for the new detector.
The new detector shows more precise, with a smaller amount of false
detections and succeeds in finding most of the input segments.
Other experiments, also available at the {\it GitHub} repository, show
that the new detector is faster and finds more edges than the previous one.


%\begin{figure}[h]
%\center
%  \begin{tabular}{
%      c@{\hspace{0.1cm}}c@{\hspace{0.1cm}}c@{\hspace{0.1cm}}c@{\hspace{0.1cm}}c}
%    \includegraphics[width=0.19\textwidth]{Fig_synth/statsExample.png} &
%    \includegraphics[width=0.19\textwidth]{Fig_synth/statsoldPoints.png} &
%    \includegraphics[width=0.19\textwidth]{Fig_synth/statsoldBounds.png} &
%    \includegraphics[width=0.19\textwidth]{Fig_synth/statsnewPoints.png} &
%    \includegraphics[width=0.19\textwidth]{Fig_synth/statsnewBounds.png}
%    \begin{picture}(1,1)
%      \put(-310,0){a)}
%      \put(-240,0){b)}
%      \put(-170,0){c)}
%      \put(-100,0){d)}
%      \put(-30,0){e)}
%    \end{picture}
%  \end{tabular}
%  \caption{Evaluation on synthesized images:
%           a) one of the test images,
%           b) output blurred segments from the old detector and
%           c) their enclosing digital segments,
%           d) output blurred segments from the new detector and
%           e) their enclosing digital segments.}
%  \label{fig:synth}
%\end{figure}
\begin{table}
\centering
\input{Fig_synth/statsTable}
\caption{Measured performance of both versions of the detector on a set of
synthesized images.
Old refers to the previous version \cite{KerautretEven09}, whereas new is
the present detector augmented with ADS and CAW concepts.
$S$ is the set of all the input segments,
$D$ the set of all the detected blurred segments.}
\label{tab:synth}
\end{table}

Next experiments aim at comparing the achieved performance of the new
detector with those of other line detectors : LSD \cite{GioiAl10},
ED-Lines \cite{AkinlarTopal12} and CannyLines \cite{LuAl15}.
The tests are run on the York Urban database \cite{DenisAl08} composed
of 102 images with their ground truth lines.
As this database was set in the scope of Manhattan-world environments,
only lines in the three main directions are identified.
Results on one image are displayed in \RefFig{fig:york}.
Compared measures $M$ are execution time $T$, covering ratio $C$,
detected lines amount $N$, cumulated length of detected lines $L$ and
mean length ratio $L/N$.
On each image of the database we measure the execution time of 100 repetitions
of a complete detection, gradient extraction included, for each line detector;
$T$ is the mean value computed on the whole image set.
Tests are run on Intel Core i5 processor.
If we assume that a pixel of a ground truth line is identified
if there is a detected line in its 8-neighborhood, then the measure $C$ is
the mean ratio of the length of ground truth line pixels identified on the
total amount of ground truth line pixels.
For all these experiments, detected lines smaller than 10 pixels are
discarded for all the detectors.
Found measures are given in \RefTab{tab:comp}.

\begin{figure}[h]
\center
  \begin{tabular}{c@{\hspace{0.2cm}}c@{\hspace{0.2cm}}c}
    \includegraphics[width=0.32\textwidth]{Fig_york/P1020928.png} &
    \includegraphics[width=0.32\textwidth]{Fig_york/P1020928_york.png} &
    \includegraphics[width=0.32\textwidth]{Fig_york/P1020928_lsd.png} \\
    \includegraphics[width=0.32\textwidth]{Fig_york/P1020928_ed.png} &
    \includegraphics[width=0.32\textwidth]{Fig_york/P1020928_canny.png} &
    \includegraphics[width=0.32\textwidth]{Fig_york/P1020928_fbsd.png}
    \begin{picture}(1,1)(0,0)
      \put(-320,91.5){\circle{8}}
      \put(-202,91.5){\circle{8}}
      \put(-85,91.5){\circle{8}}
      \put(-322.5,89){a}
      \put(-204.5,88.5){b}
      \put(-87,89){c}
      \put(-320,4.5){\circle{8}}
      \put(-202,4.5){\circle{8}}
      \put(-85,4.5){\circle{8}}
      \put(-322.5,2){d}
      \put(-204,2){e}
      \put(-87,2){f}
    \end{picture}
  \end{tabular}
  \caption{Comparison of line detectors on one of the 102 ground truth
           images of the York Urban database : a) input image, % P1020928
           b) ground truth lines, c) LSD output, d) ED-Lines output,
           e) CannyLines output, f) thick lines of the new detector.}
  \label{fig:york}
\end{figure}

\begin{table}
\centering
\input{Expe_auto/compTable}

\caption{Measured performance of recent line detectors and of our detector
on the York Urban Database \cite{DenisAl08}.}
\label{tab:comp}
\end{table}

\begin{table}
\centering
\input{Expe_auto/compTable}

\caption{Measured performance of recent line detectors and of our detector
on the York Urban Database \cite{DenisAl08}.}
\label{tab:comp}
\end{table}

On these images, CannyLines provides longer lines and ED-Lines is much faster.
Globally, the performance of the new detector is pretty similar and
competitive to the other ones, and
additionnaly, our detector provides an indication
on the detected lines quality through the additional width parameter.