Dealing with Under-reported Variables: An Information Theoretic Solution

Research output: Contribution to journalArticle

Abstract

Under-reporting occurs in survey data when there is a reason for participants to give a false negative response to a question, e.g. maternal smoking in epidemiological studies. Failing to correct this misreporting introduces biases and it may lead to misinformed decision making. Our work provides methods of correcting for this bias, by reinterpreting it as a missing data problem, and particularly learning from positive and unlabelled data. Focusing on information theoretic approaches we have three key contributions: (1) we provide a method to perform valid independence tests with known power by incorporating prior knowledge over misreporting; (2) we derive corrections for point/interval estimates of the mutual information that capture both relevance and redundancy; and finally, (3) we derive different ways for ranking under-reported risk factors. Furthermore, we show how to use our results in real-world problems and machine learning tasks.

Bibliographical metadata

Original languageEnglish
Pages (from-to)159-177
Number of pages19
JournalInternational Journal of Approximate Reasoning
Volume85
Early online date7 Apr 2017
DOIs
StatePublished - Jun 2017