Conformal Transformation of Kernel Functions

WuSi,AmariShun-Ichi

IF: 2.565

2002-01-01

Neural Processing Letters

Abstract:In this paper we extend the conformal method of modifying a kernel function to improve the performance of Support Vector Machine classifiers 14, 15. The kernel function is conformally transformed i...

What problem does this paper attempt to address?

Conformal Transformation of Kernel Functions: A Data-Dependent Way to Improve Support Vector Machine Classifiers

Si Wu,Shun-Ichi Amari

DOI: https://doi.org/10.1023/A:1013848912046

IF: 2.565

2002-01-01

Neural Processing Letters

Abstract:In this paper we extend the conformal method of modifying a kernel function to improve the performance of Support Vector Machine classifiers [14, 15]. The kernel function is conformally transformed in a data-dependent way by using the information of Support Vectors obtained in primary training. We further investigate the performances of modified Gaussian Radial Basis Function and Polynomial kernels. Simulation results for two artificial data sets show that the method is very effective, especially for correcting bad kernels.
Conformal Transformation of Kernels: A Geometric Perspective on Text Classification

Ioana Rădulescu,Alexandra Băicoianu,Adela Mihai

2024-06-02

Abstract:In this article we investigate the effects of conformal transformations on kernel functions used in Support Vector Machines. Our focus lies in the task of text document categorization, which involves assigning each document to a particular category. We introduce a new Gaussian Cosine kernel alongside two conformal transformations. Building upon previous studies that demonstrated the efficacy of conformal transformations in increasing class separability on synthetic and low-dimensional datasets, we extend this analysis to the high-dimensional domain of text data. Our experiments, conducted on the Reuters dataset on two types of binary classification tasks, compare the performance of Linear, Gaussian, and Gaussian Cosine kernels against their conformally transformed counterparts. The findings indicate that conformal transformations can significantly improve kernel performance, particularly for sub-optimal kernels. Specifically, improvements were observed in 60% of the tested scenarios for the Linear kernel, 84% for the Gaussian kernel, and 80% for the Gaussian Cosine kernel. In light of these findings, it becomes clear that conformal transformations play a pivotal role in enhancing kernel performance, offering substantial benefits.

Machine Learning,Information Theory,Differential Geometry
Improving Support Vector Machine Classifiers By Modifying Kernel Functions

S Amari,S Wu

DOI: https://doi.org/10.1016/S0893-6080(99)00032-5

IF: 7.8

1999-01-01

Neural Networks

Abstract:We propose a method of modifying a kernel function to improve the performance of a support vector machine classifier. This is based on the structure of the Riemannian geometry induced by the kernel function. The idea is to enlarge the spatial resolution around the separating boundary surface, by a conformal mapping, such that the separability between classes is increased. Examples are given specifically for modifying Gaussian Radial Basis Function kernels. Simulation results for both artificial and real data show remarkable improvement of generalization errors, supporting our idea. (C) 1999 Elsevier Science Ltd. All rights reserved.
An information-geometrical method for improving the performance of support vector machine classifiers

shunichi amari,si wu

DOI: https://doi.org/10.1049/cp:19991089

1999-01-01

Abstract:The performance of support vector machine (SVM) largely depends on the kernel. There have been no theories concerning how to choose a good kernel in a data-dependent way. As a first step to this important problem, we propose an information-geometrical method of modifying a kernel function to improve the performance of a SVM classifier. The idea is to enlarge the spatial resolution around the separating boundary surface by a conformal mapping. We gave examples of modifying Gaussian Radial Basis Function kernels. Stability of such processes is also known. Simulation results for both artificial and real data turns out to support our idea.
Distance-Based Classifier Via the Kernel Trick.

Hongxin Zhang

2010-01-01

Abstract:For classifying large data sets, we propose a discriminant kernel that introduces a nonlinear mapping from the joint space of input data and output label to a discriminant space. Our method difiers from traditional ones, which correspond to map nonlinearly from the input space to a feature space. The induced distance of our discriminant kernel is Eu- clidean and Fisher separable, as it is deflned based on distance vectors of the feature space to distance vectors on the discriminant space. Unlike the support vector machines or the kernel Fisher discriminant analysis, the classifler does not need to solve a quadric program- ming problem or eigen-decomposition problems. Therefore, it is especially appropriate to the problems of processing large data sets. The classifler can be applied to face recognition, shape comparison and image classiflcation benchmark data sets. The method is signiflcantly faster than other methods and yet it can deliver comparable classiflcation accuracy.
Scaling the kernel function to improve performance of the support vector machine

Peter Williams,Sheng Li,Jianfeng Feng,Si Wu

DOI: https://doi.org/10.1007/11427391_133

2005-01-01

Abstract:The present study investigates a geometrical method for optimizing the kernel function of a support vector machine. The method is an improvement of the one proposed in [4,5]. It consists of using prior knowledge obtained from conventional SVM training to conformally rescale the initial kernel function, so that the separation between two classes of data is effectively enlarged. It turns out that the new algorithm works efficiently, has few free parameters, consumes very low computational cost, and overcomes the susceptibility of the original method.
Optimized Kernel-Based Conformal Predictor for Online Fault Detection

Fan YANG,Jian LUO,Hua-zhen WANG,Yan-qing PENG,Hong MI

DOI: https://doi.org/10.3969/j.issn.0493-2137.2009.07.009

2009-01-01

Abstract:In order to improve the computational efficiency of conformal predictor, a procedure of adaptive kernel-based distance metric learning was incorporated in the algorithm. The learning process was divided into two stages. Firstly, an optimized kernel was obtained by increasing the class separability of 75% of the training samples. Secondly, the k nearest neighbor classifier was used to design a nonconformity measure function in the optimized kernel space. And then the standard conformal predictor algorithm was conducted on the remaining 25% of the training samples. The new method was applied to the multiple fault diagnosis of Tennessee Eastman process. The results show that the new algorithm provides substantial reductions in computational time, and ensures high predictive efficiency as well.
A geometrical method to improve performance of the support vector machine.

Peter Williams,Sheng Li,Jianfeng Feng,Si Wu

DOI: https://doi.org/10.1109/TNN.2007.891625

2007-01-01

Abstract:The performance of a support vector machine (SVM) largely depends on the kernel function used. This letter investigates a geometrical method to optimize the kernel function. The method is a modification of the one proposed by S. Amari and S. Wu. Its concern is the use of the prior knowledge obtained in a primary step training to conformally rescale the kernel function, so that the separation between the two classes of data is enlarged. The result is that the new algorithm works efficiently and overcomes the susceptibility of the original method.
A support vector data description algorithm based on an adaptive kernel function

ZiQiang WANG,AiLing DUAN,DeXian ZHANG

DOI: https://doi.org/10.3969/j.issn.1671-4628.2008.02.020

2008-01-01

Abstract:A support vector data description classification algorithm based on an adaptive kernel function is proposed in order to further improve the classification accuracy and running speed of a support vector classifier. The essential features of the algorithm are as follows: A kernel modification algorithm is constructed according to the conformal mapping of information geometry and the modified kernel function is subsequently used to train the classification algorithm in support vector data description. The experimental results show that the proposed algorithm possesses higher classification accuracy and faster running speed.
Conformalized Kernel Ridge Regression

Evgeny Burnaev,Ivan Nazarov

DOI: https://doi.org/10.48550/arXiv.1609.05959

2016-09-20

Abstract:General predictive models do not provide a measure of confidence in predictions without Bayesian assumptions. A way to circumvent potential restrictions is to use conformal methods for constructing non-parametric confidence regions, that offer guarantees regarding validity. In this paper we provide a detailed description of a computationally efficient conformal procedure for Kernel Ridge Regression (KRR), and conduct a comparative numerical study to see how well conformal regions perform against the Bayesian confidence sets. The results suggest that conformalized KRR can yield predictive confidence regions with specified coverage rate, which is essential in constructing anomaly detection systems based on predictive models.

Machine Learning,Applications
Conformalization of Sparse Generalized Linear Models

Etash Kumar Guha,Eugene Ndiaye,Xiaoming Huo

2023-07-11

Abstract:Given a sequence of observable variables $\{(x_1, y_1), \ldots, (x_n, y_n)\}$, the conformal prediction method estimates a confidence set for $y_{n+1}$ given $x_{n+1}$ that is valid for any finite sample size by merely assuming that the joint distribution of the data is permutation invariant. Although attractive, computing such a set is computationally infeasible in most regression problems. Indeed, in these cases, the unknown variable $y_{n+1}$ can take an infinite number of possible candidate values, and generating conformal sets requires retraining a predictive model for each candidate. In this paper, we focus on a sparse linear model with only a subset of variables for prediction and use numerical continuation techniques to approximate the solution path efficiently. The critical property we exploit is that the set of selected variables is invariant under a small perturbation of the input data. Therefore, it is sufficient to enumerate and refit the model only at the change points of the set of active features and smoothly interpolate the rest of the solution via a Predictor-Corrector mechanism. We show how our path-following algorithm accurately approximates conformal prediction sets and illustrate its performance using synthetic and real data examples.

Machine Learning
Conformal Embeddings via Heat Kernel

Zhitong Su

2023-09-11

Abstract:For any n-dimensional compact Riemannian Manifold $M$ with smooth metric $g$, by employing the heat kernel embedding introduced by Bérard-Besson-Gallot'94, we intrinsically construct a canonical family of conformal embeddings $C_{t,k}$: $M\rightarrow\mathbb{R}^{q(t)}$, with $t>0$ sufficiently small, $q(t)\gg t^{-\frac{n}{2}}$, and $k$ as a function of $O(t^l)$ in proper sense. Our approach involves finding all these canonical conformal embeddings, which shows the distinctions from the isometric embeddings introduced by Wang-Zhu'15.

Differential Geometry
The Fast Kernel Transform

John Paul Ryan,Sebastian Ament,Carla P. Gomes,Anil Damle

DOI: https://doi.org/10.48550/arXiv.2106.04487

2021-06-09

Abstract:Kernel methods are a highly effective and widely used collection of modern machine learning algorithms. A fundamental limitation of virtually all such methods are computations involving the kernel matrix that naively scale quadratically (e.g., constructing the kernel matrix and matrix-vector multiplication) or cubically (solving linear systems) with the size of the data set $N.$ We propose the Fast Kernel Transform (FKT), a general algorithm to compute matrix-vector multiplications (MVMs) for datasets in moderate dimensions with quasilinear complexity. Typically, analytically grounded fast multiplication methods require specialized development for specific kernels. In contrast, our scheme is based on auto-differentiation and automated symbolic computations that leverage the analytical structure of the underlying kernel. This allows the FKT to be easily applied to a broad class of kernels, including Gaussian, Matern, and Rational Quadratic covariance functions and physically motivated Green's functions, including those of the Laplace and Helmholtz equations. Furthermore, the FKT maintains a high, quantifiable, and controllable level of accuracy -- properties that many acceleration methods lack. We illustrate the efficacy and versatility of the FKT by providing timing and accuracy benchmarks and by applying it to scale the stochastic neighborhood embedding (t-SNE) and Gaussian processes to large real-world data sets.

Machine Learning,Numerical Analysis
Computational Conformal Geometric Methods for Vision

Na Lei,Feng Luo,Shing-Tung Yau,Xianfeng Gu

DOI: https://doi.org/10.1007/978-3-030-98661-2_107

2022-01-01

Abstract:Conformal geometry studies the geometric properties of objects invariant under conformal transformation group. It is a powerful theoretic tool to study shape classification, surface deformation, and registration. Computational conformal geometry is an emerging field combining modern geometry and computer science and develops both theories in the discrete setting and computational algorithms. This work first briefly introduces the fundamental concepts, theorems in conformal geometry, such as the Riemann mapping theorem, the uniformization theorem, the Beltrami equation, Teichmuller space theory, and so on; then explains three categories of computational algorithms: discrete surface curvature flow, harmonic maps, and holomorphic differentials based on Hodge theory; and finally demonstrates practical applications in engineering and medical imaging fields. In computer vision, the work explains Teichmuller shape space for surface classification, landmark constrained surface registration based on Teichmuller map, and optimal transport map. In medical imaging, the work introduces brain mapping, brain morphology study, virtual colonoscopy, and so on.
Image augmentation with conformal mappings for a convolutional neural network

Oona Rainio,Mohamed M.S. Nasser,Matti Vuorinen,Riku Klén

2023-11-21

Abstract:For augmentation of the square-shaped image data of a convolutional neural network (CNN), we introduce a new method, in which the original images are mapped onto a disk with a conformal mapping, rotated around the center of this disk and mapped under such a Möbius transformation that preserves the disk, and then mapped back onto their original square shape. This process does not result the loss of information caused by removing areas from near the edges of the original images unlike the typical transformations used in the data augmentation for a CNN. We offer here the formulas of all the mappings needed together with detailed instructions how to write a code for transforming the images. The new method is also tested with simulated data and, according the results, using this method to augment the training data of 10 images into 40 images decreases the amount of the error in the predictions by a CNN for a test set of 160 images in a statistically significant way (p-value=0.0360).

Computer Vision and Pattern Recognition
Conformal calibrators

Vladimir Vovk,Ivan Petej,Paolo Toccaceli,Alex Gammerman

DOI: https://doi.org/10.48550/arXiv.1902.06579

IF: 5.414

2019-02-18

Machine Learning

Abstract:Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems that are fully adaptive. Our method consists in calibrating existing predictive systems; the input predictive system is not supposed to satisfy any properties of validity, whereas the output predictive system is guaranteed to be calibrated in probability. It is interesting that the method may also work without the IID assumption, standard in conformal prediction.
Conformal Prediction using Conditional Histograms

Matteo Sesia,Yaniv Romano

DOI: https://doi.org/10.48550/arXiv.2105.08747

2021-10-24

Abstract:This paper develops a conformal method to compute prediction intervals for non-parametric regression that can automatically adapt to skewed data. Leveraging black-box machine learning algorithms to estimate the conditional distribution of the outcome using histograms, it translates their output into the shortest prediction intervals with approximate conditional coverage. The resulting prediction intervals provably have marginal coverage in finite samples, while asymptotically achieving conditional coverage and optimal length if the black-box model is consistent. Numerical experiments with simulated and real data demonstrate improved performance compared to state-of-the-art alternatives, including conformalized quantile regression and other distributional conformal prediction approaches.

Methodology,Machine Learning
Making learning more transparent using conformalized performance prediction

Matthew J. Holland

DOI: https://doi.org/10.48550/arXiv.2007.04486

2020-07-09

Abstract:In this work, we study some novel applications of conformal inference techniques to the problem of providing machine learning procedures with more transparent, accurate, and practical performance guarantees. We provide a natural extension of the traditional conformal prediction framework, done in such a way that we can make valid and well-calibrated predictive statements about the future performance of arbitrary learning algorithms, when passed an as-yet unseen training set. In addition, we include some nascent empirical examples to illustrate potential applications.

Machine Learning
A new trigonometric kernel function for support vector machine

Sajad Fathi Hafshejani,Zahra Moberfard

DOI: https://doi.org/10.1007/s42044-022-00130-9

2022-12-15

Abstract:In the last few years, various types of machine learning algorithms, such as Support Vector Machine (SVM), Support Vector Regression (SVR), and Non-negative Matrix Factorization (NMF) have been introduced. The kernel approach is an effective method for increasing the classification accuracy of machine learning algorithms. This paper introduces a family of one-parameter kernel functions for improving the accuracy of SVM classification. The proposed kernel function consists of a trigonometric term and differs from all existing kernel functions. We show this function is a positive definite kernel function. Finally, we evaluate the SVM method based on the new trigonometric kernel, the Gaussian kernel, the polynomial kernel, and a convex combination of the new kernel function and the Gaussian kernel function on various types of datasets. Empirical results show that the SVM based on the new trigonometric kernel function and the mixed kernel function achieve the best classification accuracy. Moreover, some numerical results of performing the SVR based on the new trigonometric kernel function and the mixed kernel function are presented.

Machine Learning,Computer Vision and Pattern Recognition,Neural and Evolutionary Computing
The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines

Di Zhang,Suvrajeet Sen

2024-07-31

Abstract:Stochastic First-Order (SFO) methods have been a cornerstone in addressing a broad spectrum of modern machine learning (ML) challenges. However, their efficacy is increasingly questioned, especially in large-scale applications where empirical evidence indicates potential performance limitations. In response, this paper proposes an innovative method specifically designed for kernel support vector machines (SVMs). This method not only achieves faster convergence per iteration but also exhibits enhanced scalability when compared to conventional SFO techniques. Diverging from traditional sample average approximation strategies that typically frame kernel SVM as an 'all-in-one' Quadratic Program (QP), our approach adopts adaptive sampling. This strategy incrementally refines approximation accuracy on an 'as-needed' basis. Crucially, this approach also inspires a decomposition-based algorithm, effectively decomposing parameter selection from error estimation, with the latter being independently determined for each data point. To exploit the quadratic nature of the kernel matrix, we introduce a stochastic conjugate subgradient method. This method preserves many benefits of first-order approaches while adeptly handling both nonlinearity and non-smooth aspects of the SVM problem. Thus, it extends beyond the capabilities of standard SFO algorithms for non-smooth convex optimization. The convergence rate of this novel method is thoroughly analyzed within this paper. Our experimental results demonstrate that the proposed algorithm not only maintains but potentially exceeds the scalability of SFO methods. Moreover, it significantly enhances both speed and accuracy of the optimization process.

Machine Learning,Optimization and Control

Conformal Transformation of Kernel Functions

Conformal Transformation of Kernel Functions: A Data-Dependent Way to Improve Support Vector Machine Classifiers

Conformal Transformation of Kernels: A Geometric Perspective on Text Classification

Improving Support Vector Machine Classifiers By Modifying Kernel Functions

An information-geometrical method for improving the performance of support vector machine classifiers

Distance-Based Classifier Via the Kernel Trick.

Scaling the kernel function to improve performance of the support vector machine

Optimized Kernel-Based Conformal Predictor for Online Fault Detection

A geometrical method to improve performance of the support vector machine.

A support vector data description algorithm based on an adaptive kernel function

Conformalized Kernel Ridge Regression

Conformalization of Sparse Generalized Linear Models

Conformal Embeddings via Heat Kernel

The Fast Kernel Transform

Computational Conformal Geometric Methods for Vision

Image augmentation with conformal mappings for a convolutional neural network

Conformal calibrators

Conformal Prediction using Conditional Histograms

Making learning more transparent using conformalized performance prediction

A new trigonometric kernel function for support vector machine

The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines