Abstract:Predictive models are often used for real-time decision making. However, typical machine learning techniques ignore feature evaluation cost, and focus solely on the accuracy of the machine learning models obtained utilizing all the features available. We develop algorithms and indexes to support cost-sensitive prediction, i.e., making decisions using machine learning models taking feature evaluation cost into account. Given an item and a online computation cost (i.e., time) budget, we present two approaches to return an appropriately chosen machine learning model that will run within the specified time on the given item. The first approach returns the optimal machine learning model, i.e., one with the highest accuracy, that runs within the specified time, but requires significant up-front precomputation time. The second approach returns a possibly sub- optimal machine learning model, but requires little up-front precomputation time. We study these two algorithms in detail and characterize the scenarios (using real and synthetic data) in which each performs well. Unlike prior work that focuses on a narrow domain or a specific algorithm, our techniques are very general: they apply to any cost-sensitive prediction scenario on any machine learning algorithm.

What problem does this paper attempt to address?

This paper attempts to solve the problem of how prediction models can be optimally selected in real - time decision - making while considering the cost of feature evaluation. Specifically, traditional machine - learning techniques often focus only on the accuracy of the model and ignore the cost of feature evaluation (such as computing time and resource consumption). Therefore, when predictions need to be completed within a limited time budget, these traditional methods may not meet practical requirements. ### Main Problems of the Paper 1. **How to select the optimal machine - learning model within a limited time budget?** - Given a project and an online computing cost (i.e., time) budget, how to select the machine - learning model that can run within the specified time and has the highest accuracy. 2. **How to balance the prediction accuracy and computing cost?** - In some application scenarios, such as credit card fraud detection, system performance monitoring, and advertising delivery systems, accuracy is crucial, but at the same time, the limitations of computing resources also need to be considered. 3. **How to deal with cost - sensitive predictions under different input scales?** - The size of the input data (such as the number of image pixels) will affect the cost of feature calculation. Therefore, a method that can adapt to different input scales is needed to select an appropriate model. ### Overview of Solutions The paper proposes two main methods to solve the above problems: 1. **POLY - DOM Method**: - **Advantages**: It provides an optimal solution and ensures that the most accurate model is selected within a given time budget. - **Disadvantages**: It requires a large amount of offline pre - computation and has high computational complexity. - **Implementation**: By constructing a "Polydom Index", different optimal models under different feature sets can be effectively stored and retrieved. 2. **GREEDY Method**: - **Advantages**: It is simple and fast to calculate and does not require a large amount of offline pre - computation. - **Disadvantages**: It may not be the optimal solution, but it performs well in many cases. - **Implementation**: Based on the greedy algorithm, feature sets are gradually selected until the time budget is reached or a certain accuracy requirement is met. ### Key Technologies and Challenges - **Feature Set Pruning**: By removing those feature sets that cannot be optimal solutions, the amount of calculation is reduced. - **Polydom Index**: A new data structure for compactly representing candidate feature sets and supporting efficient online queries. - **Modeling of Cost Function and Accuracy Function**: By defining the cost function \( c(F_i, |x|) \) of feature calculation and the model accuracy function \( a(F_i) \), the pros and cons of different feature sets are quantified. ### Application Scenarios - **Advertising System**: It is necessary to control the delay while ensuring personalized recommendations for users. - **Financial Software**: Deal with portfolio predictions of different scales and ensure the best results within a fixed time. - **Autopilot System**: Respond to errors within a limited time to ensure flight safety. - **Mobile Sensor**: Decide whether to report error information under limited resources. ### Summary This paper proposes a general framework for considering the cost of feature evaluation in real - time prediction, aiming to improve the efficiency and practicality of the prediction system. By combining the POLY - DOM and GREEDY methods, the optimal strategy can be flexibly selected in different application scenarios, so as to better meet practical requirements.

Indexing Cost Sensitive Prediction

Predict-then-optimize or predict-and-optimize? An empirical evaluation of cost-sensitive learning strategies

Predicting ENR Construction Cost Index Using Machine-Learning Algorithms

Predictive Indexing

Cost-Sensitive Learning for Predictive Maintenance

Cost-sensitive Regression Learning on Small Dataset Through Intra-Cluster Product Favoured Feature Selection

Towards cost-sensitive adaptation: When is it worth updating your predictive model?

Cost Sensitive Support Vector Machines

SkipPredict: When to Invest in Predictions for Scheduling

Efficient Lifelong Model Evaluation in an Era of Rapid Progress

Example-dependent cost-sensitive adaptive boosting

Nonparametric active learning for cost-sensitive classification

Estimating Environmental Cost Throughout Model's Adaptive Life Cycle

Low-Cost Algorithmic Recourse for Users With Uncertain Cost Functions

Strategies for Imputing Missing Values and Removing Outliers in the Dataset for Machine Learning-Based Construction Cost Prediction

Optimal Credit Scorecard Model Selection Using Costs Arising from Both False Positives and False Negatives

Optimization of Selective Ensemble for Cost-Sensitive Classification: an Empirical Study

Online performance and proactive maintenance assessment of data driven prediction models

Trading-Off Cost of Deployment Versus Accuracy in Learning Predictive Models

Learning Decisions Offline from Censored Observations with ε-insensitive Operational Costs

On Statistical Efficiency in Learning