publications
2017

arXiv
Active Learning amidst Logical Knowledge. Platanios, Emmanouil A, Kapoor, Ashish, and Horvitz, Eric In arXiv 2017 Abstract PDF
Structured prediction is ubiquitous in applications of machine learning such as knowledge extraction and natural language processing. Structure often can be formulated in terms of logical constraints. We consider the question of how to perform efficient active learning in the presence of logical constraints among variables inferred by different classifiers. We propose several methods and provide theoretical results that demonstrate the inappropriateness of employing uncertainty guided sampling, a commonly used active learning method. Furthermore, experiments on ten different datasets demonstrate that the methods significantly outperform alternatives in practice. The results are of practical significance in situations where labeled data is scarce.

NIPS
Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach. Platanios, Emmanouil A, Poon, Hoifung, Horvitz, Eric, and Mitchell, Tom M In Neural Information Processing Systems 2017 Abstract
We propose an efﬁcient method to estimate the accuracy of classiﬁers using only unlabeled data. We consider a setting with multiple classiﬁcation problems where the target classes may be tied together through logical constraints. For example, a set of classes may be mutually exclusive, meaning that a data instance can belong to at most one of them. The proposed method is based on the intuition that: (i) when classiﬁers agree, they are more likely to be correct, and (ii) when the classiﬁers make a prediction that violates the constraints, at least one classiﬁer must be making an error. Experiments on four realworld data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. Our models also outperform existing stateoftheart solutions in both estimating accuracies, and combining multiple classiﬁer outputs. The results emphasize the utility of logical constraints in estimating accuracy, thus validating our intuition.
2016

ICML
Estimating Accuracy from Unlabeled Data: A Bayesian Approach. Platanios, Emmanouil A, Dubey, Avinava, and Mitchell, Tom M In International Conference in Machine Learning 2016 Abstract PDF Supplementary Poster Slides
We consider the question of how unlabeled data can be used to estimate the true accuracy of learned classifiers, and the related question of how outputs from several classifiers performing the same task can be combined based on their estimated accuracies. To answer these questions, we first present a simple graphical model that performs well in practice. We then provide two nonparametric extensions to it that improve its performance. Experiments on two realworld data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. Our models also outperform existing stateoftheart solutions in both estimating accuracies, and combining multiple classifier outputs.
2015

AAAI
NeverEnding Learning. Mitchell, Tom M, Cohen, William W, Hruschka Jr, Estevam R, Pratim Talukdar, Partha, Betteridge, Justin, Carlson, Andrew, Dalvi, Bhanava, Gardner, Matt, Kisiel, Bryan, Krishnamurthy, Jayant, Lao, Ni, Mazaitis, Kathryn, Mohamed, Thahir P, Nakashole, Ndapakula, Platanios, Emmanouil A, Ritter, Alan, Samadi, Mehdi, Settles, Burr, Wang, Richard C, Wijaya, Derry, Gupta, Abhinav, Chen, Xinlei, Saparov, Abulhair, Greaves, Malcolm, and Welling, Joel In Association for the Advancement of Artificial Intelligence 2015 Abstract PDF
Whereas people learn many different types of knowledge from diverse experiences over many years, most current machine learning systems acquire just a single function or data model from just a single data set. We propose a neverending learning paradigm for machine learning, to better reflect the more ambitious and encompassing type of learning performed by humans. As a case study, we describe the NeverEnding Language Learner (NELL), which achieves some of the desired properties of a neverending learner, and we discuss lessons learned. NELL has been learning to read the web 24 hours/day since January 2010, and so far has acquired a knowledge base with over 80 million confidenceweighted beliefs (e.g., servedWith(tea, biscuits)). NELL has also learned millions of features and parameters that enable it to read these beliefs from the web. Additionally, it has learned to reason over these beliefs to infer new beliefs, and is able to extend its ontology by synthesizing new relational predicates. NELL can be tracked online at http://rtw.ml.cmu.edu, and followed on Twitter at @CMUNELL.

CMU
Platanios, E. A. (2015). Estimating Accuracy from Unlabeled Data. Master’s Thesis at Carnegie Mellon University. Abstract PDF Poster
We consider the question of how unlabeled data can be used to estimate the true accuracy of learned classiﬁers. This is an important question for any autonomous learning system that must estimate its accuracy without supervision, and also when classiﬁers trained from one data distribution must be applied to a new distribution (e.g., document classiﬁers trained on one text corpus are to be applied to a second corpus). We ﬁrst show how to estimate error rates exactly from unlabeled data when given a collection of competing classiﬁers that make independent errors, based on the agreement rates between subsets of these classiﬁers. We further show that even when the competing classiﬁers do not make independent errors, both their accuracies and error dependencies can be estimated by making certain relaxed assumptions. We then present an alternative approach based on graphical models that also allows us to combine the outputs of the classiﬁers into a single output label. A simple graphical model is introduced that performs well in practice. Then, two nonparametric extensions to it are presented, that signiﬁcantly improve its performance. Experiments on two realworld data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. We also obtain results demonstrating our graphical model approaches beating alternative methods for combining the classiﬁers’ outputs. These results are of practical signiﬁcance in situations where labeled data is scarce and shed light on the more general question of how the consistency among multiple functions is related to their true accuracies.
2014

UAI
Estimating Accuracy from Unlabeled Data. Platanios, Emmanouil A, Blum, Avrim, and Mitchell, Tom M In Conference on Uncertainty in Artificial Intelligence 2014 Abstract PDF Addendum Poster
We consider the question of how unlabeled data can be used to estimate the true accuracy of learned classifiers. This is an important question for any autonomous learning system that must estimate its accuracy without supervision, and also when classifiers trained from one data distribution must be applied to a new distribution (e.g., document classifiers trained on one text corpus are to be applied to a second corpus). We first show how to estimate error rates exactly from unlabeled data when given a collection of competing classifiers that make independent errors, based on the agreement rates between subsets of these classifiers. We further show that even when the competing classifiers do not make independent errors, both their accuracies and error dependencies can be estimated by making certain relaxed assumptions. Experiments on two data realworld data sets produce estimates within a few percent of the true accuracy, using solely unlabeled data. These results are of practical significance in situations where labeled data is scarce and shed light on the more general question of how the consistency among multiple functions is related to their true accuracies.

PAMI
Gaussian ProcessMixture Conditional Heteroscedasticity. Platanios, Emmanouil A, and Chatzis, Sotirios P IEEE Transactions on Pattern Analysis and Machine Intelligence 2014 Abstract PDF
Generalized autoregressive conditional heteroscedasticity (GARCH) models have long been considered as one of the most successful families of approaches for volatility modeling in ﬁnancial return series. In this paper, we propose an alternative approach based on methodologies widely used in the ﬁeld of statistical machine learning. Speciﬁcally, we propose a novel nonparametric Bayesian mixture of Gaussian process regression models, each component of which models the noise variance process that contaminates the observed data as a separate latent Gaussian process driven by the observed data. This way, we essentially obtain a Gaussian processmixture conditional heteroscedasticity (GPMCH) model for volatility modeling in ﬁnancial return series. We impose a nonparametric prior with powerlaw nature over the distribution of the model mixture components, namely the PitmanYor process prior, to allow for better capturing modeled data distributions with heavy tails and skewness. Finally, we provide a copulabased approach for obtaining a predictive posterior for the covariances over the asset returns modeled by means of a postulated GPMCH model. We evaluate the efﬁcacy of our approach in a number of benchmark scenarios, and compare its performance to stateoftheart methodologies.
2012

NIPS
Nonparametric Mixtures of MultiOutput Heteroscedastic Gaussian Processes for Volatility Modeling. Platanios, Emmanouil A, and Chatzis, Sotirios P In Neural Information Processing Systems Workshop on Modern Nonparametric Methods in Machine Learning 2012 Abstract PDF Poster
In this work, we present a nonparametric Bayesian method for multivariate volatility modeling. Our approach is based on postulation of a novel mixture of multioutput heteroscedastic Gaussian processes to model the covariance matrices of multiple assets. Speciﬁcally, we use the PitmanYor process prior as the non parametric prior imposed over the components of our model, which are taken as multioutput heteroscedastic Gaussian processes obtained by introducing appropriate convolution kernels that combine simple heteroscedastic Gaussian processes under a multioutput scheme. We exhibit the efﬁcacy of our approach in a volatility prediction task.