%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% This is a LaTeX abstract template for the NETTAB 2016 conference
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentclass[12pt,a4paper]{article}

\usepackage[english]{babel}
\usepackage[T1]{fontenc}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathrsfs}
\usepackage[colorlinks]{hyperref}
\usepackage{helvet}
\usepackage{graphicx}
\renewcommand{\familydefault}{\sfdefault}
\hypersetup{hidelinks}
\addtolength{\hoffset}{-2cm}
\addtolength{\textwidth}{4cm}
\addtolength{\voffset}{-2cm}
\addtolength{\textheight}{5cm}

\newcommand{\Hy}{{\em hyperSMURF}}


\begin{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Insert the abstract title in the curly brackets after the 
%% \section* tag
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Boosting deleterious genomic preidctions thorugh \Hy parameters tuning

\section*{Parameters tuning boosts \Hy~ predictions of rare deleterious non-coding genetic variants}

\vspace{0.5cm}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Author(s)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\noindent{{Alessandro Petrini$^1$}, {Max Schubach$^2$}, {Matteo Re$^{1}$},
{Marco Frasca$^{1}$}, {Marco Mesiti$^{1}$}, {Giuliano Grossi$^{1}$}, 
{Tiziana Castrignan\`o$^{3}$},
{Peter N. Robinson$^{4}$} and {Giorgio Valentini$^{1}$}
 }

\vspace{0.5cm}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Author's affiliation 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\noindent{$^1${\fontsize{11}{13.2}\selectfont AnacletoLab, Dipartimento di Informatica, Universit\`a degli Studi di Milano, Italy}}\\
\noindent{$^2${\fontsize{11}{13.2}\selectfont Institute for Medical and Human Genetics, Charité -- Universitätsmedizin Berlin, Germany}}\\
\noindent{$^3${\fontsize{11}{13.2}\selectfont CINECA, SCAI SuperComputing Applications and Innovation Department, Rome, Italy}\\
\noindent{$^4${\fontsize{11}{13.2}\selectfont The Jackson Laboratory for Genomic Medicine, Farmington CT, USA}}\\

\vspace{0.3cm}

\noindent
\textbf{Introduction}\\

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Introduction
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\noindent{
\fontsize{11}{13.2}\selectfont 
The regulatory code that determines whether and how a given genetic variant affects the function of a regulatory element remains poorly understood for most classes of regulatory variation.
Indeed the large majority of bioinformatics tools have been developed to predict the pathogenicity of genetic
variants in coding sequences or conserved splice sites~\cite{Ritchie14b}.

Computational algorithms for the prediction of non-coding deleterious variants associated with rare genetic diseases are faced with special challenges owing to the rarity of confirmed pathogenic mutations. Indeed in this context classical machine learning methods are biased toward neutral variants that constitute the large majority of genetic variation, and are not able to detect the potential deleterious variants that constitute only a tiny minority of all known genetic variation~\cite{Smedley16}.

We recently proposed \Hy, hyper-ensemble of SMOTE Undersampled Random Forests, an ensemble approach explicitly designed to deal with the huge imbalance between deleterious and neutral variants~\cite{Schubach17}, and able to significantly outperform state-of-the-art methods for the prediction of non-coding variants associated with Mendelian diseases.

Despite its successful application to the detection of deleterious single nucleotide variants (SNV) as well as to small insertions or deletions (indels),
{\em HyperSMURF} is a method that depends on several learning parameters, that strongly influence its overall performances.
In this work we experimentally show that by tuning \Hy~ parameters we can significantly boost the performance of the method, thus predicting with significantly better precision and recall rare SNVs associated with Mendelian diseases.




} 

\vspace{0.3cm}
\noindent
\textbf{Methods}\\

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Methods
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\noindent{\fontsize{11}{13.2}\selectfont 
{\em HyperSMURF} is a method specifically designed to provide genome-wide predictions of deleterious (e.g. disease-associated) variants explicitly taking into account the imbalance that characterize the number of deleterious variants (positive examples) vs neutral variants (negative examples) in the non-coding human genome.

To this end two main learning strategies are adopted: 1) Sampling techniques and 2) Ensembling and hyper-ensembling approaches. 
By oversampling the small set of available positive examples using SMOTE~\cite{Chawla2002} and at the same subsampling the set of negative examples, we can balance the data, thus avoiding the bias toward the majority class. At the same time by training a set of random forests, each one on a different sampled and balanced set of the data, we can obtain both accurate and diverse base learners and a large coverage of the available training data: these features represent key factors for the success of ensemble methods~\cite{Kuncheva14}. 
Note that with \Hy~we have an ensemble of ensembles, since each base learner is in turn a random forest (i.e. an ensemble of decision trees), thus obtaining an hyper-ensemble (ensemble of ensembles) approach. 
For a detailed description of the \Hy~algorithm, please see~\cite{Schubach17}.

{\em HyperSMURF} is characterized by a set of learning parameters (i.e. the number of random forests $n$, the oversampling $f$ and subsampling $m$ factors) that can significantly affect the overall performance of the method. Indeed these parameters largely affect the coverage of the training data, the accuracy and the diversity of the base learners, the balancing between positive and negative examples, the ratio of novel synthetic positive examples through the SMOTE algorithm~\cite{Chawla2002}, that on the one hand can add more valuable information to train the random forests, but on the other hand can add noisy examples to the training data. In other words it is not always straightforward the proper choice of the learning parameters of \Hy, since it also strongly depends on the underlying distribution of the data.
In principle, by properly tuning the learning parameters, we can better fit the data and significantly boost the overall performances.
Considering the complexity of a ``a priori'' evaluating of the ``optimal'' \Hy~parameters, we adopted an empirical approach by selecting through internal cross-validation on the training data the parameters that lead to the beast estimated performance using the Area Under the Precision Recall Curve (AUPRC) as the the metric to be maximized.

} 

\vspace{0.3cm}
\noindent
\textbf{Results}\\

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Results
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\noindent{\fontsize{11}{13.2}\selectfont 
We subdivided the Mendelian data that include $406$ manually annotated ``positive'' deleterious SNV and more than $15$ millions of neutral ``negative'' SNVs in a training set including about $9/10$ of the available data and a separated test set including the remaining $1/10$ of data, using the same set of genomic features described in~\cite{Smedley16}.
We then compared the \Hy~results obtained by using the default parameters (i.e. $n=100$, $f=2$, $m=3$) with the \Hy~results obtained by selecting the ``best'' learning parameters through cross-validation on the training data.
More precisely we considered all the possible combinations of the parameters $n \in \{10, 50, 100, 300\}$, $f \in \{1, 2, 3, 5, 10\}$ and $m \in \{1, 2, 3, 5, 10\}$. The resulting $100$ \Hy~models have been cross-validated using the Marconi cluster available at CINECA Supercomputing Applications and Innovation Department.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[ht]
%\vskip -0.1in   keepaspectratio
\centering
\begin{tabular}{cc}	
\includegraphics[width=8.8cm]{boxplot.eps} &  \hspace{-1.0cm}  
\includegraphics[width=9.8cm]{PRC.eps} \\
(a) & (b)\\ 
\end{tabular}
\small
\caption{\fontsize{11}{13.2}\selectfont 
(a): Distribution across $100$ different choices of \Hy~parameters $(n,f,m)$ of the AUPRC values obtained by cross-validation on the training data; (b) Comparison of precision-recall curves obtained by \Hy~with default parameters (red curve) and with the selected ``best'' parameters (cyan curve).}
\label{fig}
\normalsize
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


Fig~\ref{fig} a) shows the distribution of the cross-validated results across $100$ combinations of $(n,f,m)$ triplets of the learning parameters on the training set. The results widely vary, depending on the choice of the parameters, from a minimum of the Area Under the Precision Recall Curve (AUPRC) equal to $0.1741$ to a maximum AUPRC equal to $0.4529$, achieved with parameters $n=300, f=1, m=10$, i.e. with $300$ random forests trained on samples having a doubled number of positive examples ($f=1$), and a number of negatives $10$ times larger than that of positives ($m=10$), thus reducing in the training set the original imbalance between positive and negatives from about $1:36000$ to $1:10$. 

Fig~\ref{fig} b) shows the precision/recall curves obtained on the test set by using the default parameters and the best  parameters selected by cross-validation on the training data. Here we obtain a significant increment of the AUPRC from 
$0.3568$  to $0.4156$ (the difference is statistically significant according to the Wilcoxon rank sum test, $p-value = 10^{-16}$). These results are also confirmed by the $AUROC_{50}, AUROC_{100},$ and  $AUROC_{1000}$ results, where \Hy~ with tuned parameters largely and significantly outperforms \Hy~ with default parameters (Table~\ref{table}).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{table}[!h]
%\addcontentsline{toc}{table}{{\bf Supplementary Table XX}. Performance of \Hy~with different selection strategies for negatives}
\caption{\fontsize{11}{13.2}\selectfont Comparison of \Hy~results obtained respectively with default parameters ($n=100, f=2, m=3$) and with the best parameters obtained by internal cross-validation on the training data ($n=300, f=1, m=10$).}
\label{table}
\centering
\begin{tabular}{|lrrrrr|}
\hline
               & $AUPRC$	& $AUROC_{50}$ &  $AUROC_{100}$ & $AUROC_{500}$  & $AUROC_{1000}$ \\ \hline
\Hy~default par.  &  0.3568  &   0.8600  &  0.9300  &  0.9091   &   0.8868   \\
\Hy~best par.     &  0.4156  &   0.9220  &  0.9610  &  0.9407   &   0.9460   \\  \hline  \hline
\end{tabular}
\end{table}    

Note that we do not report $AUROC$ results, since in this highly imbalanced context pure $AUROC$ results are not as significant as $AUPRC$~\cite{Saito15}.

We outline that we previously showed that \Hy~with default parameters just significantly outperforms existing state-of-the-art methods~\cite{Kircher2014,Zhou2015,Ionita-Laza:2016} on the prediction of deleterious variants in Mendelian diseases~\cite{Schubach17}. 
Our results summarized in Table~\ref{table} show that the proper selection of \Hy~learning parameters can further significantly improve the overall performance of the method.
 
}


\vspace{0.3cm}
\textbf{Conclusions}\\

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Conclusions
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\noindent{\fontsize{11}{13.2}\selectfont 
This work shows the potentialities of \Hy~parameters tuning in the context of the detection and prioritization of deleterious genetic variants associated with Mendelian diseases, but we guess that also in other genomic contexts characterized by high imbalance between  deleterious and neutral variants fine tuning of \Hy~parameters may lead to improved results. 

Nevertheless  the training and testing of hyper-ensembles trained and tested on millions of genetic variants is highly time-consuming: in the context of Mendelian disease, where we used low-dimensional features (more precisely $26$ genomic features) training and testing a \Hy~model required from about $2$ to $20$ hours of computation with an Intel Xeon Processor $E5-2697 v4$, with a clock of $2.30$ GHz and $128$ GB of memory. By enlarging the number of features to thousands (e.g. by using features extracted from DNA with deep convolutional networks~\cite{Zhou2015}), the computational time can further dramatically increase. 
To overcome these drawbacks, a full parallel implementation of \Hy~by exploiting High Performance Computing architectures is a research line to be pursued to deeply explore the learning parameters of \Hy, as well as the learning parameters of the random forests that constitute the base learners of the hyper-ensemble.
On the other hand a parallel implementation could make feasible the automatic adaptation of \Hy~ to different learning tasks and different analyses of genomic big data, ranging from the detection of deleterious variants in genetic diseases to the detection of somatic driver mutations in cancer.

}

\noindent

{\fontsize{11}{13.2}\selectfont 


%\textbf{References}\\

\begin{thebibliography}{1}

\bibitem{Chawla2002}
N.~V. Chawla, K.~W. Bowyer, L.~O. Hall, and W.~P. Kegelmeyer.
\newblock {SMOTE: Synthetic Minority Over-sampling Technique}.
\newblock {\em Journal of Artificial Intelligence Research}, pages 321--357,
  2002.

\bibitem{Ionita-Laza:2016}
Iuliana Ionita-Laza, Kenneth McCallum, Bin Xu, and Joseph~D Buxbaum.
\newblock A spectral approach integrating functional genomic annotations for
  coding and noncoding variants.
\newblock {\em Nat Genet}, 48(2):214--20, Feb 2016.

\bibitem{Kircher2014}
Martin Kircher, Daniela~M. Witten, Preti Jain, Brian~J. O'Roak, Gregory~M.
  Cooper, and Jay Shendure.
\newblock A general framework for estimating the relative pathogenicity of
  human genetic variants.
\newblock {\em Nat Genet}, 46(3):310--315, Mar 2014.

\bibitem{Kuncheva14}
Ludmila~I. Kuncheva.
\newblock {\em Diversity in Classifier Ensembles}, pages 247--289.
\newblock John Wiley \& Sons, Inc., 2014.

\bibitem{Ritchie14b}
G.R. Ritchie and P.~Flicek.
\newblock Computational approaches to interpreting genomic sequence variation.
\newblock {\em Genome Med.}, 6(87), 2014.

\bibitem{Saito15}
T.~Saito and M.~Rehmsmeier.
\newblock The precision-recall plot is more informative than the roc plot when
  evaluating binary classifiers on imbalanced datasets.
\newblock {\em PLoS ONE}, 10:e0118432, 2015.

\bibitem{Schubach17}
M.~Schubach, M.~Re, P.N. Robinson, and G.~Valentini.
\newblock Imbalance-aware machine learning for predicting rare and common
  disease-associated non-coding variants.
\newblock {\em Scientific Reports}, 
\newblock doi: 10.1038/s41598-017-03011-5, 2017.

\bibitem{Smedley16}
Damian Smedley, Max Schubach, Julius~O.B. Jacobsen, Sebastian K{\"{o}}hler,
  Tomasz Zemojtel, Malte Spielmann, Marten J{\"{a}}ger, Harry Hochheiser,
  Nicole~L. Washington, Julie~A. McMurry, Melissa~A. Haendel, Christopher~J.
  Mungall, Suzanna~E. Lewis, Tudor Groza, Giorgio Valentini, and Peter~N.
  Robinson.
\newblock {A Whole-Genome Analysis Framework for Effective Identification of
  Pathogenic Regulatory Variants in Mendelian Disease}.
\newblock {\em The American Journal of Human Genetics}, 99(3):595--606, sep
  2016.

\bibitem{Zhou2015}
Jian Zhou and Olga~G Troyanskaya.
\newblock {Predicting effects of noncoding variants with deep learning-based
  sequence model.}
\newblock {\em Nature methods}, 12(10):931--934, August 2015.

\end{thebibliography}

}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% References
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\indent{\fontsize{10}{12}\selectfont 1. Authors. Title. Journal. Year. Volume(Issue):Pages. DOI}\\
%\indent{\fontsize{10}{12}\selectfont 2. Authors. Title. Journal. Year. Volume(Issue):Pages. DOI}\\
%\indent{\fontsize{10}{12}\selectfont 3. Authors. Title. Journal. Year. Volume(Issue):Pages. DOI}\\
%\indent{\fontsize{10}{12}\selectfont 4. Authors. Title. Journal. Year. Volume(Issue):Pages. DOI}\\

\end{document}
