Question at position 3 Here’s a snippet from a hypothetical study: “Our dataset comprised 100,000 healthy individuals with blood markers, cognitive performance measures and clinical diagnosis (“Alzheimer’s disease” or “cognitively normal”). The number of individuals was well-balanced between the two diagnostic categories. Missing data was filled in using an iterative procedure in which missing values were predicted using existing values. The dataset was then divided into training and test sets in and 80%:20% ratio. We train a machine learning model in the training set to predict whether an individual has Alzheimer’s disease or is cognitively normal based on the individual’s blood markers and cognitive performance measures. Cross-validation was performed in the training set to obtain the best hyperparameters. The model was then retrained using the best hyperparameters in the training set, and then evaluated in the test set, yielding 90% sensitivity and 90% specificity.” Which of the following is correct? Select all correct answers below. There can be more than one correct answer. There is a penalty for selecting the wrong answer.The reported prediction performance is likely to be an under-estimate of true accuracyThe hyperparameter selection procedure is valid The hyperparameter selection procedure is invalidThe reported prediction performance is likely to be an over-estimate of true accuracyBlackTom Analysis

Question

Question at position 3 Here’s a snippet from a hypothetical study: “Our dataset comprised 100,000 healthy individuals with blood markers, cognitive performance measures and clinical diagnosis (“Alzheimer’s disease” or “cognitively normal”). The number of individuals was well-balanced between the two diagnostic categories. Missing data was filled in using an iterative procedure in which missing values were predicted using existing values. The dataset was then divided into training and test sets in and 80%:20% ratio. We train a machine learning model in the training set to predict whether an individual has Alzheimer’s disease or is cognitively normal based on the individual’s blood markers and cognitive performance measures. Cross-validation was performed in the training set to obtain the best hyperparameters. The model was then retrained using the best hyperparameters in the training set, and then evaluated in the test set, yielding 90% sensitivity and 90% specificity.” Which of the following is correct? Select all correct answers below. There can be more than one correct answer. There is a penalty for selecting the wrong answer.The reported prediction performance is likely to be an under-estimate of true accuracyThe hyperparameter selection procedure is valid The hyperparameter selection procedure is invalidThe reported prediction performance is likely to be an over-estimate of true accuracyBlackTom Analysis

BlackTom AI · Accepted Answer

Re-state the scenario in your own words: a large dataset of 100,000 healthy individuals with biomarkers and cognitive measures was split 80/20 into training and test sets. Missing data were filled via an iterative imputation method. Within the training set, cross-validation was used to select hyperparameters, then the model was retrained on the training data with those best hyperparameters and finally evaluated on the held-out test set, reporting 90% sensitivity and 90% specificity.

Option 1: The reported prediction performance is likely to be an under-estimate of true accuracy

This option would imply that the model’s performance on the test set is lower than its true generalization performance. Given the described procedure—training and hyperparameter tuning entirely within the training set, followed by evaluation on a separate test set—there is no inherent reason to expect the test performance to be systematically too low. In fact, if anything, withholding the test data from the tuning process protects against optimistic bias, not underestimation. Therefore, there is no justification here to label the reported performance as likely under-estimated.

Option 2: The hyperparameter selection procedure is valid

This statement treats the process as appropriate: hyperparameters were chosen using cross-validation within the training set, and then the model was retrained using those hyperparameters on the training data and evaluated on the independent test set. Since the test data were not used to select hyperparameters, the procedure avoids peeking at test labels when tuning, which is the core requirement for a valid evaluation setup. Although some practitioners advocate nested cross-validation for an unbiased performance estimate, the described approach is a standard and valid method that correctly separates training/hyperparameter tuning from test evaluation. Thus, this option is correct.

Option 3: The hyperparameter selection procedure is invalid

This would be true if hyperparameters were chosen using information from the test set or if the test data influenced the tuning process. The description explicitly states hyperparameters were selected via cross-validation in the training set, with evaluation performed on a separate test set. There is no indication of leakage from the test set into hyperparameter tuning in the described workflow. Therefore, labeling the procedure as invalid contradicts the given method and is not supported by the details provided.

Option 4: The reported prediction performance is likely to be an over-estimate of true accuracy

Over-estimation would occur if the tuning process used test data or if the imputation or preprocessing leaked information from the test set into training. The text mentions that hyperparameters were selected in the training set and then the model was evaluated on the test set, which is the standard way to obtain an unbiased test performance in many practical setups. Without explicit indication of leakage from test data into the training/imputation steps, this option is not warranted by the described method. Therefore, it is not correct in this context.

View Explanation

Log in for full answers

Similar Questions

Why was model-building strategy #1 used to estimate generalization performance on the income data for the decision tree, logistic regression, and random forest models?

In k-fold cross-validation, what happens in each iteration?

What is one advantage of k-fold cross-validation over a single holdout set?

Which of the following is TRUE about cross validation?

Question at position 1 True/False Question: k-fold cross-validation is used to find which fold of the data set gives the best model. TrueFalse

Which is not a benefit of using cross-validation for model evaluation?

Why does the code 4 notebook use cross-validation when selecting the alpha value for regularization?

More Practical Tools for Students Powered by AI Study Helper

Homework AI Solver

Stylized AI Paper Writer

Plagiarism Checker Assistant

Citation AI Academic Writing Tool

In-Class Translation Assistant

AI Note Generator

AI Quiz Answers

Past Exam Questions from University Test Bank

Smart Practice Assistant

Adaptive Practice

Making Your Study Simpler