新建
第四讲 图SVM (Graph SVM)
xuxh@dp.tech
doggyceneks
推荐镜像 :Basic Image:bohrium-notebook:2023-04-07
推荐机型 :c2_m4_cpu
赞
目录
数据集
辅助文件和数据(v1)
内容提要
本讲按照SVM技术的发展顺序依次介绍 linear SVM, soft-margin SVM, nonlinear SVM, 最后讲 graph SVM. 下面我们简单梳理本讲每部分的具体内容
Intro：SVM 在机器学习中的位置
- 五类主要学习算法：监督、非监督、半监督、强化、自监督
- SVM and graph SVM 属于：监督和半监督，连接了机器学习的重要话题
Linear SVM
- 介绍 SVM 算法的定义
- Linear SVM：在数据线性可分的假设下处理上述问题
- 介绍 Linear SVM 的构造思路和求解方法
- Lab 1 ：实现线性SVM，观察它在线性可分数据和非线性数据上的分类效果
- 介绍 SVM 算法的定义
Soft-margin SVM
- motivation：当数据中存在噪声，不能完全线性可分时，可以考虑采用 soft-margin SVM 解决这个问题
- 正则参数 和 惩罚项解读
- Soft-margin SVM 问题求解
- Lab 2: 实现线性 soft-margin SVM，观察它在加入噪声的线性可分数据和非线性数据上的分类效果
- motivation：当数据中存在噪声，不能完全线性可分时，可以考虑采用 soft-margin SVM 解决这个问题
Kernel techniques and Nonlinear/kernel SVM
- motivation：解决数据不线性可分的问题，采用核技巧将非线性数据映射至高维空间
- 介绍 Nonlinear/kernel SVM 优化问题的构造和求解
- Lab 3: 实现 Nonlinear/kernel SVM，观察它在加入噪声的非线性数据和真实世界的文本文档上的分类效果
Graph SVM
- motivation：利用未标记数据的信息，假设标记数据和未标记数据都来自同一个特征空间中的流行
- 介绍 Graph SVM 优化问题的构造和求解
- Lab 4: 实现 Nonlinear/kernel SVM，观察它在加入噪声的非线性数据和真实世界的文本文档上的分类效果
代码
文本
Lab 01 : Standard/Linear SVM
代码
文本
[3]
# Load libraries
import numpy as np
import scipy.io
%matplotlib inline
#%matplotlib notebook
from matplotlib import pyplot
import matplotlib.pyplot as plt
plt.rcParams.update({'figure.max_open_warning': 0})
from IPython.display import display, clear_output
import time
import sys; sys.path.insert(0, '/bohr/dataset-5kud/v1/')
from lib.utils import compute_purity
import warnings; warnings.filterwarnings("ignore")
代码
文本
Linearly separable data points
代码
文本
[4]
# Dataset
mat = scipy.io.loadmat('/bohr/dataset-5kud/v1/datasets/data_linearSVM.mat')
Xtrain = mat['Xtrain']
Cgt_train = mat['Cgt_train'] - 1; Cgt_train = Cgt_train.squeeze()
l_train = mat['l'].squeeze()
n = Xtrain.shape[0]
d = Xtrain.shape[1]
nc = len(np.unique(Cgt_train))
print(n,d,nc)
Xtest = mat['Xtest']
Cgt_test = mat['Cgt_test'] - 1; Cgt_test = Cgt_test.squeeze()
40 2 2
代码
文本
[5]
# Plot
plt.figure(figsize=(8,4))
p1 = plt.subplot(121)
size_vertex_plot = 100
plt.scatter(Xtrain[:,0], Xtrain[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_train, color=pyplot.jet())
plt.title('Training Data')
p2 = plt.subplot(122)
size_vertex_plot = 100
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_test, color=pyplot.jet())
plt.title('Test Data')
plt.tight_layout()
plt.show()
代码
文本
[6]
# Run Linear SVM
# Compute linear kernel, L, Q
Ker = Xtrain.dot(Xtrain.T)
l = l_train
L = np.diag(l)
Q = L.dot(Ker.dot(L))
# Time steps
tau_alpha = 10/ np.linalg.norm(Q,2)
tau_beta = 0.1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + np.eye(n)
# Pre-compute J.K(Xtest) for test data
LKXtest = L.dot(Xtrain.dot(Xtest.T))
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 201
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%10) or (diff_alpha<1e-3):
# Indicator function of support vectors
idx = np.where( np.abs(alpha)>0.25* np.max(np.abs(alpha)) )
Isv = np.zeros([n]); Isv[idx] = 1
nb_sv = len(Isv.nonzero()[0])
# Offset
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(L.dot(alpha)) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = alpha.T.dot(LKXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Plot
plt.figure(figsize=(8,4))
p1 = plt.subplot(121)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=f_test, color=pyplot.jet())
plt.title('Score function $s(x)=w^Tx+b$ \n iter=' + str(k)+ ', diff_alpha=' + str(diff_alpha)[:7])
plt.colorbar()
p2 = plt.subplot(122)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=C_test, color=pyplot.jet())
plt.title('Classification function $f(x)=sign(w^Tx+b)$\n iter=' + str(k) + ', acc=' + str(accuracy_test)[:5])
plt.tight_layout()
plt.colorbar()
plt.show()
if k<num_iter-1:
clear_output(wait=True)
代码
文本
Non-linearly separable data points
代码
文本
[7]
# Dataset
mat = scipy.io.loadmat('/bohr/dataset-5kud/v1/datasets/data_twomoons_softSVM.mat')
Xtrain = mat['Xtrain']
Cgt_train = mat['C_train_errors'] - 1; Cgt_train = Cgt_train.squeeze()
Cgt_train[:250] = 0; Cgt_train[250:] = 1
l_train = mat['l'].squeeze()
n = Xtrain.shape[0]
d = Xtrain.shape[1]
nc = len(np.unique(Cgt_train))
print(n,d,nc)
Xtest = mat['Xtest']
Cgt_test = mat['Cgt_test'] - 1; Cgt_test = Cgt_test.squeeze()
500 100 2
代码
文本
[8]
# Plot
plt.figure(figsize=(10,4))
p1 = plt.subplot(121)
size_vertex_plot = 33
plt.scatter(Xtrain[:,0], Xtrain[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_train, color=pyplot.jet())
plt.title('Training Data')
p2 = plt.subplot(122)
size_vertex_plot = 33
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_test, color=pyplot.jet())
plt.title('Test Data')
#plt.tight_layout()
plt.show()
代码
文本
[9]
# Run Linear SVM
# Compute linear kernel, L, Q
Ker = Xtrain.dot(Xtrain.T)
l = l_train
L = np.diag(l)
Q = L.dot(Ker.dot(L))
# Time steps
tau_alpha = 10/ np.linalg.norm(Q,2)
tau_beta = 0.1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + np.eye(n)
# Pre-compute J.K(Xtest) for test data
LKXtest = L.dot(Xtrain.dot(Xtest.T))
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 201
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%10) or (diff_alpha<1e-3):
# Indicator function of support vectors
idx = np.where( np.abs(alpha)>0.25* np.max(np.abs(alpha)) )
Isv = np.zeros([n]); Isv[idx] = 1
nb_sv = len(Isv.nonzero()[0])
# Offset
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(L.dot(alpha)) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = alpha.T.dot(LKXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Plot
size_vertex_plot = 33
plt.figure(figsize=(12,4))
p1 = plt.subplot(121)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=f_test, color=pyplot.jet())
plt.title('Score function $s(x)=w^Tx+b$ \n iter=' + str(k)+ ', diff_alpha=' + str(diff_alpha)[:7])
plt.colorbar()
p2 = plt.subplot(122)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=C_test, color=pyplot.jet())
plt.title('Classification function $f(x)=sign(w^Tx+b)$\n iter=' + str(k) + ', acc=' + str(accuracy_test)[:5])
#plt.tight_layout()
plt.colorbar()
plt.show()
if k<num_iter-1:
clear_output(wait=True)
代码
文本
Lab 02 : Soft-Margin SVM
代码
文本
[10]
# Load libraries
import numpy as np
import scipy.io
%matplotlib inline
#%matplotlib notebook
from matplotlib import pyplot
import matplotlib.pyplot as plt
plt.rcParams.update({'figure.max_open_warning': 0})
from IPython.display import display, clear_output
import time
import sys; sys.path.insert(0, '/bohr/dataset-5kud/v1/')
from lib.utils import compute_purity
from lib.utils import compute_SVM
import warnings; warnings.filterwarnings("ignore")
代码
文本
Linearly separable data points
代码
文本
[12]
# Data matrix X = linearly separable data points
mat = scipy.io.loadmat('/bohr/dataset-5kud/v1/datasets/data_softSVM.mat')
Xtrain = mat['Xtrain']
Cgt_train = mat['C_train_errors'] - 1; Cgt_train = Cgt_train.squeeze()
l_train = mat['l'].squeeze()
n = Xtrain.shape[0]
d = Xtrain.shape[1]
nc = len(np.unique(Cgt_train))
print(n,d,nc)
Xtest = mat['Xtest']
Cgt_test = mat['Cgt_test'] - 1; Cgt_test = Cgt_test.squeeze()
40 2 2
代码
文本
[13]
# Plot
plt.figure(figsize=(8,4))
p1 = plt.subplot(121)
size_vertex_plot = 100
plt.scatter(Xtrain[:,0], Xtrain[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_train, color=pyplot.jet())
plt.title('Training Data with 25% ERRORS')
p2 = plt.subplot(122)
size_vertex_plot = 100
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_test, color=pyplot.jet())
plt.title('Test Data')
plt.tight_layout()
plt.show()
代码
文本
[14]
# Run soft-margin SVM
# Compute linear kernel, L, Q
Ker = Xtrain.dot(Xtrain.T)
l = l_train
L = np.diag(l)
Q = L.dot(Ker.dot(L))
# Time steps
tau_alpha = 10/ np.linalg.norm(Q,2)
tau_beta = 0.1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + np.eye(n)
# Pre-compute J.K(Xtest) for test data
LKXtest = L.dot(Xtrain.dot(Xtest.T))
# Error parameter
lamb = 0.001 # acc: 90%
#lamb = 0.01 # acc: 97.5%
lamb = 0.1 # acc: 100% <==
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 201
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
alpha[alpha>lamb] = lamb # Projection on [-infty,lamb]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%10) or (diff_alpha<1e-3):
# Indicator function of support vectors
idx = np.where( np.abs(alpha)>0.25* np.max(np.abs(alpha)) )
Isv = np.zeros([n]); Isv[idx] = 1
nb_sv = len(Isv.nonzero()[0])
# Offset
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(L.dot(alpha)) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = alpha.T.dot(LKXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Plot
plt.figure(figsize=(8,4))
p1 = plt.subplot(121)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=f_test, color=pyplot.jet())
plt.title('Score function $s(x)=w^Tx+b$ \n iter=' + str(k)+ ', diff_alpha=' + str(diff_alpha)[:7])
plt.colorbar()
p2 = plt.subplot(122)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=C_test, color=pyplot.jet())
plt.title('Classification function $f(x)=sign(w^Tx+b)$\n iter=' + str(k) + ', acc=' + str(accuracy_test)[:5])
plt.tight_layout()
plt.colorbar()
plt.show()
if k<num_iter-1:
clear_output(wait=True)
代码
文本
Non-linearly separable data points
代码
文本
[15]
# Dataset
mat = scipy.io.loadmat('/bohr/dataset-5kud/v1/datasets/data_twomoons_softSVM.mat')
Xtrain = mat['Xtrain']
Cgt_train = mat['C_train_errors'] - 1; Cgt_train = Cgt_train.squeeze()
l_train = mat['l'].squeeze()
n = Xtrain.shape[0]
d = Xtrain.shape[1]
nc = len(np.unique(Cgt_train))
print(n,d,nc)
Xtest = mat['Xtest']
Cgt_test = mat['Cgt_test'] - 1; Cgt_test = Cgt_test.squeeze()
500 100 2
代码
文本
[16]
# Plot
plt.figure(figsize=(10,4))
p1 = plt.subplot(121)
size_vertex_plot = 33
plt.scatter(Xtrain[:,0], Xtrain[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_train, color=pyplot.jet())
plt.title('Training Data with 25% ERRORS')
p2 = plt.subplot(122)
size_vertex_plot = 33
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_test, color=pyplot.jet())
plt.title('Test Data')
#plt.tight_layout()
plt.show()
代码
文本
[17]
# Run soft-margin SVM
# Compute linear kernel, L, Q
Ker = Xtrain.dot(Xtrain.T)
l = l_train
L = np.diag(l)
Q = L.dot(Ker.dot(L))
# Time steps
tau_alpha = 10/ np.linalg.norm(Q,2)
tau_beta = 0.1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + np.eye(n)
# Pre-compute J.K(Xtest) for test data
LKXtest = L.dot(Xtrain.dot(Xtest.T))
# Error parameter
lamb = 0.001 # acc: 80.4%
#lamb = 0.01 # acc: 81%
lamb = 0.1 # acc: 81.6%
#lamb = 1 # acc: 82.6% <==
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 201
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
alpha[alpha>lamb] = lamb # Projection on [-infty,lamb]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%10) or (diff_alpha<1e-3):
# Indicator function of support vectors
idx = np.where( np.abs(alpha)>0.25* np.max(np.abs(alpha)) )
Isv = np.zeros([n]); Isv[idx] = 1
nb_sv = len(Isv.nonzero()[0])
# Offset
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(L.dot(alpha)) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = alpha.T.dot(LKXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Plot
size_vertex_plot = 33
plt.figure(figsize=(12,4))
p1 = plt.subplot(121)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=f_test, color=pyplot.jet())
plt.title('Score function $s(x)=w^Tx+b$ \n iter=' + str(k)+ ', diff_alpha=' + str(diff_alpha)[:7])
plt.colorbar()
p2 = plt.subplot(122)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=C_test, color=pyplot.jet())
plt.title('Classification function $f(x)=sign(w^Tx+b)$\n iter=' + str(k) + ', acc=' + str(accuracy_test)[:5])
#plt.tight_layout()
plt.colorbar()
plt.show()
if k<num_iter-1:
clear_output(wait=True)
代码
文本
Lab 03 : Kernel/Non-Linear SVM
代码
文本
[18]
# Load libraries
import numpy as np
import scipy.io
%matplotlib inline
#%matplotlib notebook
from matplotlib import pyplot
import matplotlib.pyplot as plt
from IPython.display import display, clear_output
plt.rcParams.update({'figure.max_open_warning': 0})
import time
import sys; sys.path.insert(0, 'lib/')
from lib.utils import compute_purity
from lib.utils import compute_SVM
import warnings; warnings.filterwarnings("ignore")
import sklearn.metrics.pairwise
代码
文本
Non-linearly separable data points
代码
文本
[19]
# Dataset
mat = scipy.io.loadmat('/bohr/dataset-5kud/v1/datasets/data_twomoons_kernelSVM.mat')
Xtrain = mat['Xtrain']
Cgt_train = mat['C_train_errors'] - 1; Cgt_train = Cgt_train.squeeze()
l_train = mat['l'].squeeze()
n = Xtrain.shape[0]
d = Xtrain.shape[1]
nc = len(np.unique(Cgt_train))
print(n,d,nc)
Xtest = mat['Xtest']
Cgt_test = mat['Cgt_test'] - 1; Cgt_test = Cgt_test.squeeze()
500 100 2
代码
文本
[20]
# Plot
plt.figure(figsize=(10,4))
p1 = plt.subplot(121)
size_vertex_plot = 33
plt.scatter(Xtrain[:,0], Xtrain[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_train, color=pyplot.jet())
plt.title('Training Data with 25% ERRORS')
p2 = plt.subplot(122)
size_vertex_plot = 33
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_test, color=pyplot.jet())
plt.title('Test Data')
#plt.tight_layout()
plt.show()
代码
文本
Run soft-margin SVM
代码
文本
[21]
# Run soft-margin SVM
# Compute linear kernel, L, Q
Ker = Xtrain.dot(Xtrain.T)
l = l_train
L = np.diag(l)
Q = L.dot(Ker.dot(L))
# Time steps
tau_alpha = 10/ np.linalg.norm(Q,2)
tau_beta = 0.1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + np.eye(n)
# Pre-compute J.K(Xtest) for test data
LKXtest = L.dot(Xtrain.dot(Xtest.T))
# Error parameter
lamb = 0.1 # acc: 83%
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 201
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
alpha[alpha>lamb] = lamb # Projection on [-infty,lamb]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%10) or (diff_alpha<1e-3):
# Indicator function of support vectors
idx = np.where( np.abs(alpha)>0.25* np.max(np.abs(alpha)) )
Isv = np.zeros([n]); Isv[idx] = 1
nb_sv = len(Isv.nonzero()[0])
# Offset
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(L.dot(alpha)) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = alpha.T.dot(LKXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Plot
size_vertex_plot = 33
plt.figure(figsize=(12,4))
p1 = plt.subplot(121)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=f_test, color=pyplot.jet())
plt.title('Score function $s(x)=w^Tx+b$ \n iter=' + str(k)+ ', diff_alpha=' + str(diff_alpha)[:7])
plt.colorbar()
p2 = plt.subplot(122)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=C_test, color=pyplot.jet())
plt.title('Classification function $f(x)=sign(w^Tx+b)$\n iter=' + str(k) + ', acc=' + str(accuracy_test)[:5])
#plt.tight_layout()
plt.colorbar()
plt.show()
if k<num_iter-1:
clear_output(wait=True)
代码
文本
Run kernel SVM
代码
文本
[22]
# Run kernel SVM
# Compute Gaussian kernel, L, Q
sigma = 0.5; sigma2 = sigma**2
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtrain, metric='euclidean', n_jobs=1)
Ker = np.exp(- Ddist**2 / sigma2)
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtest, metric='euclidean', n_jobs=1)
KXtest = np.exp(- Ddist**2 / sigma2)
l = l_train
L = np.diag(l)
Q = L.dot(Ker.dot(L))
# Time steps
tau_alpha = 10/ np.linalg.norm(Q,2)
tau_beta = 0.1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + np.eye(n)
# Pre-compute J.K(Xtest) for test data
LKXtest = L.dot(KXtest)
# Error parameter
lamb = 3 # acc: 95.4
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 201
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
alpha[alpha>lamb] = lamb # Projection on [-infty,lamb]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%10) or (diff_alpha<1e-3):
# Indicator function of support vectors
idx = np.where( np.abs(alpha)>0.25* np.max(np.abs(alpha)) )
Isv = np.zeros([n]); Isv[idx] = 1
nb_sv = len(Isv.nonzero()[0])
# Offset
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(L.dot(alpha)) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = alpha.T.dot(LKXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Plot
size_vertex_plot = 33
plt.figure(figsize=(12,4))
p1 = plt.subplot(121)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=f_test, color=pyplot.jet())
plt.title('Score function $s(x)=w^T\phi(x)+b$ \n iter=' + str(k)+ ', diff_alpha=' + str(diff_alpha)[:7])
plt.colorbar()
p2 = plt.subplot(122)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=C_test, color=pyplot.jet())
plt.title('Classification function $f(x)=sign(w^T\phi(x)+b)$\n iter=' + str(k) + ', acc=' + str(accuracy_test)[:5])
#plt.tight_layout()
plt.colorbar()
plt.show()
if k<num_iter-1:
clear_output(wait=True)
代码
文本
Real-world graph of articles
代码
文本
[23]
# Dataset
mat = scipy.io.loadmat('/bohr/dataset-5kud/v1/datasets/data_20news_50labels.mat')
Xtrain = mat['Xtrain']
l_train = mat['l'].squeeze()
n = Xtrain.shape[0]
d = Xtrain.shape[1]
nc = len(np.unique(Cgt_train))
print(n,d,nc)
Xtest = mat['Xtest']
Cgt_test = mat['Cgt_test'] - 1; Cgt_test = Cgt_test.squeeze()
50 3684 2
代码
文本
Run linear SVM
代码
文本
[24]
# Run linear SVM
# Compute Gaussian kernel, L, Q
Ker = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtrain, metric='euclidean', n_jobs=1)
KXtest = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtest, metric='euclidean', n_jobs=1)
l = l_train
L = np.diag(l)
Q = L.dot(Ker.dot(L))
# Time steps
tau_alpha = 10/ np.linalg.norm(Q,2)
tau_beta = 0.1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + np.eye(n)
# Pre-compute J.K(Xtest) for test data
LKXtest = L.dot(KXtest)
# Error parameter
lamb = 3 # acc: 83.5
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 201
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
alpha[alpha>lamb] = lamb # Projection on [-infty,lamb]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%10) or (diff_alpha<1e-3):
# Indicator function of support vectors
idx = np.where( np.abs(alpha)>0.25* np.max(np.abs(alpha)) )
Isv = np.zeros([n]); Isv[idx] = 1
nb_sv = len(Isv.nonzero()[0])
# Offset
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(L.dot(alpha)) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = alpha.T.dot(LKXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Print
print('Linear SVM, iter, diff_alpha, acc :',str(k),str(diff_alpha)[:7],str(accuracy_test)[:5])
Linear SVM, iter, diff_alpha, acc : 4 0.0 83.5
代码
文本
Run kernel SVM
代码
文本
[25]
# Run kernel SVM
# Compute Gaussian kernel, L, Q
sigma = 0.5; sigma2 = sigma**2
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtrain, metric='euclidean', n_jobs=1)
Ker = np.exp(- Ddist**2 / sigma2)
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtest, metric='euclidean', n_jobs=1)
KXtest = np.exp(- Ddist**2 / sigma2)
l = l_train
L = np.diag(l)
Q = L.dot(Ker.dot(L))
# Time steps
tau_alpha = 10/ np.linalg.norm(Q,2)
tau_beta = 0.1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + np.eye(n)
# Pre-compute J.K(Xtest) for test data
LKXtest = L.dot(KXtest)
# Error parameter
lamb = 3 # acc: 87.5
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 201
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
alpha[alpha>lamb] = lamb # Projection on [-infty,lamb]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%10) or (diff_alpha<1e-3):
# Indicator function of support vectors
idx = np.where( np.abs(alpha)>0.25* np.max(np.abs(alpha)) )
Isv = np.zeros([n]); Isv[idx] = 1
nb_sv = len(Isv.nonzero()[0])
# Offset
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(L.dot(alpha)) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = alpha.T.dot(LKXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Print
# print('iter, diff_alpha',str(k),str(diff_alpha)[:7])
# print('acc',str(accuracy_test)[:5])
print('Kernel SVM iter, diff_alpha :',str(k),str(diff_alpha)[:7])
print(' acc :',str(accuracy_test)[:5])
Kernel SVM iter, diff_alpha : 100 0.00099
acc : 87.5
代码
文本
Lab 04 : Graph SVM
代码
文本
[26]
# Load libraries
import numpy as np
import scipy.io
%matplotlib inline
#%matplotlib notebook
from matplotlib import pyplot
import matplotlib.pyplot as plt
from IPython.display import display, clear_output
plt.rcParams.update({'figure.max_open_warning': 0})
import time
import sys; sys.path.insert(0, '/bohr/dataset-5kud/v1/')
from lib.utils import compute_purity
from lib.utils import compute_SVM
from lib.utils import construct_knn_graph
from lib.utils import graph_laplacian
import warnings; warnings.filterwarnings("ignore")
import sklearn.metrics.pairwise
代码
文本
Non-linear separable data points
代码
文本
[27]
# Dataset
mat = scipy.io.loadmat('/bohr/dataset-5kud/v1/datasets/data_twomoons_graphSVM.mat')
Xtrain = mat['Xtrain']
Cgt_train = mat['Cgt_train'] - 1; Cgt_train = Cgt_train.squeeze()
l_train = mat['l'].squeeze()
nb_labeled_data_per_class = mat['nb_labeled_data_per_class'].squeeze()
n = Xtrain.shape[0]
d = Xtrain.shape[1]
nc = len(np.unique(Cgt_train))
print(n,d,nc)
Xtest = mat['Xtest']
Cgt_test = mat['Cgt_test'] - 1; Cgt_test = Cgt_test.squeeze()
print('l_train:',l_train)
print('number of labeled data per class:',nb_labeled_data_per_class)
print('number of unlabeled data:',n-2*nb_labeled_data_per_class)
500 2 2 l_train: [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] number of labeled data per class: 1 number of unlabeled data: 498
代码
文本
[28]
# Plot
plt.figure(figsize=(12,4))
p1 = plt.subplot(121)
size_vertex_plot = 33
plt.scatter(Xtrain[:,0], Xtrain[:,1], s=size_vertex_plot*np.ones(n), c=l_train, color=pyplot.jet())
plt.title('Training Data: Labeled Data in red (first class)\n and blue (second class), \n and unlabeled Data in green (data geometry)')
plt.colorbar()
p2 = plt.subplot(122)
size_vertex_plot = 33
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=Cgt_test, color=pyplot.jet())
plt.title('Test Data')
plt.colorbar()
#plt.tight_layout()
plt.show()
代码
文本
Run kernel SVM
代码
文本
[29]
# Run kernel SVM
# Compute Gaussian kernel, L, Q
sigma = 0.5; sigma2 = sigma**2
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtrain, metric='euclidean', n_jobs=1)
Ker = np.exp(- Ddist**2 / sigma2)
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtest, metric='euclidean', n_jobs=1)
KXtest = np.exp(- Ddist**2 / sigma2)
l = l_train
L = np.diag(l)
Q = L.dot(Ker.dot(L))
# Time steps
tau_alpha = 10/ np.linalg.norm(Q,2)
tau_beta = 0.1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + np.eye(n)
# Pre-compute J.K(Xtest) for test data
LKXtest = L.dot(KXtest)
# Error parameter
lamb = 3 # acc: 95.4
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 201
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
alpha[alpha>lamb] = lamb # Projection on [-infty,lamb]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%100) or (diff_alpha<1e-3):
# Indicator function of support vectors
idx = np.where( np.abs(alpha)>0.25* np.max(np.abs(alpha)) )
Isv = np.zeros([n]); Isv[idx] = 1
nb_sv = len(Isv.nonzero()[0])
# Offset
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(L.dot(alpha)) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = alpha.T.dot(LKXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Plot
size_vertex_plot = 33
plt.figure(figsize=(12,4))
p1 = plt.subplot(121)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=f_test, color=pyplot.jet())
plt.title('Score function $s(x)=w^T\phi(x)+b$ \n iter=' + str(k)+ ', diff_alpha=' + str(diff_alpha)[:7])
plt.colorbar()
p2 = plt.subplot(122)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=C_test, color=pyplot.jet())
plt.title('Classification function $f(x)=sign(w^T\phi(x)+b)$\n iter=' + str(k) + ', acc=' + str(accuracy_test)[:5])
#plt.tight_layout()
plt.colorbar()
plt.show()
if k<num_iter-1:
clear_output(wait=True)
代码
文本
Run Graph SVM
代码
文本
[30]
# Run Graph SVM
# Compute Gaussian kernel
sigma = 0.15; sigma2 = sigma**2
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtrain, metric='euclidean', n_jobs=1)
Ker = np.exp(- Ddist**2 / sigma2)
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtest, metric='euclidean', n_jobs=1)
KXtest = np.exp(- Ddist**2 / sigma2)
# Compute kNN graph
kNN = 10
gamma = 25
A = construct_knn_graph(Xtrain, kNN, 'euclidean')
Lap = graph_laplacian(A).todense()
# Compute Indicator function of labels
H = np.zeros([n])
H[np.abs(l_train)>0.0] = 1
H = np.diag(H)
# Compute L, Q
L = np.diag(l_train)
l = l_train
T = np.eye(n)
T += gamma* Lap.dot(Ker)
Tinv = np.linalg.inv(T)
Q = L.dot(H.dot(Ker.dot(Tinv.dot(H.dot(L)))))
# Time steps
tau_alpha = 1/ np.linalg.norm(Q,2)
tau_beta = 1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + np.eye(n)
# Error parameter
lamb = 1 # acc: 98.6
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 201
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
alpha[alpha>lamb] = lamb # Projection on [-infty,lamb]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%100) or (diff_alpha<1e-3):
# xi vector
xi = Tinv.dot(H.dot(L.dot(alpha)))
# Offset
idx_unlabeled_data = np.where( np.abs(l)<1./2 )
alpha_labels = alpha; alpha_labels[idx_unlabeled_data] = 0
idx = np.where( np.abs(alpha_labels)>0.25* np.max(np.abs(alpha_labels)) )
Isv = np.zeros([n]); Isv[idx] = 1 # Indicator function of Support Vectors
nb_sv = len(Isv.nonzero()[0])
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(np.squeeze(np.array(xi))) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = xi.dot(KXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Plot
size_vertex_plot = 33
plt.figure(figsize=(12,4))
p1 = plt.subplot(121)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=f_test, color=pyplot.jet())
plt.title('Score function $s(x)=w^T\phi(x)+b$ \n iter=' + str(k)+ ', diff_alpha=' + str(diff_alpha)[:7])
plt.colorbar()
p2 = plt.subplot(122)
plt.scatter(Xtest[:,0], Xtest[:,1], s=size_vertex_plot*np.ones(n), c=C_test, color=pyplot.jet())
plt.title('Classification function $f(x)=sign(w^T\phi(x)+b)$\n iter=' + str(k) + ', acc=' + str(accuracy_test)[:5])
#plt.tight_layout()
plt.colorbar()
plt.show()
if k<num_iter-1:
clear_output(wait=True)
k-NN graph with euclidean distance
代码
文本
Real-world graph of articles
代码
文本
Datasets has 10 labeled data and 40 unlabeled data
代码
文本
[32]
# Dataset
mat = scipy.io.loadmat('/bohr/dataset-5kud/v1/datasets/data_20news_10labels_40unlabels.mat')
Xtrain = mat['Xtrain']
n = Xtrain.shape[0]
l_train = mat['l'].squeeze()
d = Xtrain.shape[1]
Xtest = mat['Xtest']
Cgt_test = mat['Cgt_test'] - 1; Cgt_test = Cgt_test.squeeze()
nc = len(np.unique(Cgt_test))
print(n,d,nc)
num_labels = np.sum(np.abs(l_train)>0.0)
print('l_train:',l_train)
print('number of labeled data per class:',num_labels//2)
print('number of unlabeled data:',n-num_labels)
50 3684 2 l_train: [-1 0 -1 -1 -1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 -1 0 0 0 0 0 0 1 1 0] number of labeled data per class: 5 number of unlabeled data: 40
代码
文本
Run kernel SVM
代码
文本
[33]
# Run Kernel SVM (no graph information)
# Compute Gaussian kernel
sigma = 0.5; sigma2 = sigma**2
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtrain, metric='euclidean', n_jobs=1)
Ker = np.exp(- Ddist**2 / sigma2)
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtest, metric='euclidean', n_jobs=1)
KXtest = np.exp(- Ddist**2 / sigma2)
# Compute kNN graph
kNN = 5
gamma = 0 # <= no graph information
A = construct_knn_graph(Xtrain, kNN, 'cosine')
Lap = graph_laplacian(A).todense()
# Compute Indicator function of labels
H = np.zeros([n])
H[np.abs(l_train)>0.0] = 1
H = np.diag(H)
# Compute L, Q
L = np.diag(l_train)
l = l_train
T = np.eye(n)
T += gamma* Lap.dot(Ker)
Tinv = np.linalg.inv(T)
Q = L.dot(H.dot(Ker.dot(Tinv.dot(H.dot(L)))))
# Time steps
tau_alpha = 1/ np.linalg.norm(Q,2)
tau_beta = 1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + 1* np.eye(n)
# Error parameter
lamb = 100
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 1001
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
alpha[alpha>lamb] = lamb # Projection on [-infty,lamb]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%100) or (diff_alpha<1e-3):
# xi vector
xi = Tinv.dot(H.dot(L.dot(alpha)))
# Offset
idx_unlabeled_data = np.where( np.abs(l)<1./2 )
alpha_labels = alpha; alpha_labels[idx_unlabeled_data] = 0
idx = np.where( np.abs(alpha_labels)>0.25* np.max(np.abs(alpha_labels)) )
Isv = np.zeros([n]); Isv[idx] = 1 # Indicator function of Support Vectors
nb_sv = len(Isv.nonzero()[0])
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(np.squeeze(np.array(xi))) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = xi.dot(KXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Print
# print('iter, diff_alpha',str(k),str(diff_alpha)[:7])
# print('acc',str(accuracy_test)[:5])
print('Kernel SVM iter, diff_alpha :',str(k),str(diff_alpha)[:7])
print(' acc :',str(accuracy_test)[:5])
k-NN graph with cosine distance
Kernel SVM iter, diff_alpha : 1001 0.68597
acc : 65.5
代码
文本
Run Graph SVM
代码
文本
[34]
# Run Graph SVM
# Compute Gaussian kernel
sigma = 0.5; sigma2 = sigma**2
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtrain, metric='euclidean', n_jobs=1)
Ker = np.exp(- Ddist**2 / sigma2)
Ddist = sklearn.metrics.pairwise.pairwise_distances(Xtrain, Xtest, metric='euclidean', n_jobs=1)
KXtest = np.exp(- Ddist**2 / sigma2)
# Compute kNN graph
kNN = 8
gamma = 100
A = construct_knn_graph(Xtrain, kNN, 'cosine')
Lap = graph_laplacian(A).todense()
# Compute Indicator function of labels
H = np.zeros([n])
H[np.abs(l_train)>0.0] = 1
H = np.diag(H)
# Compute L, Q
L = np.diag(l_train)
l = l_train
T = np.eye(n)
T += gamma* Lap.dot(Ker)
Tinv = np.linalg.inv(T)
Q = L.dot(H.dot(Ker.dot(Tinv.dot(H.dot(L)))))
# Time steps
tau_alpha = 1/ np.linalg.norm(Q,2)
tau_beta = 1/ np.linalg.norm(L,2)
# For conjuguate gradient
Acg = tau_alpha* Q + 1* np.eye(n)
# Error parameter
lamb = 1
# Initialization
alpha = np.zeros([n])
beta = 0.0
alpha_old = alpha
# Loop
k = 0
diff_alpha = 1e6
num_iter = 1001
while (diff_alpha>1e-3) & (k<num_iter):
# Update iteration
k += 1
# Update alpha
# Approximate solution with conjuguate gradient
b0 = alpha + tau_alpha - tau_alpha* l* beta
alpha, _ = scipy.sparse.linalg.cg(Acg, b0, x0=alpha, tol=1e-3, maxiter=50)
alpha[alpha<0.0] = 0 # Projection on [0,+infty]
alpha[alpha>lamb] = lamb # Projection on [-infty,lamb]
# Update beta
beta = beta + tau_beta* l.T.dot(alpha)
# Stopping condition
diff_alpha = np.linalg.norm(alpha-alpha_old)
alpha_old = alpha
# Plot
if not(k%100) or (diff_alpha<1e-3):
# xi vector
xi = Tinv.dot(H.dot(L.dot(alpha)))
# Offset
idx_unlabeled_data = np.where( np.abs(l)<1./2 )
alpha_labels = alpha; alpha_labels[idx_unlabeled_data] = 0
idx = np.where( np.abs(alpha_labels)>0.25* np.max(np.abs(alpha_labels)) )
Isv = np.zeros([n]); Isv[idx] = 1 # Indicator function of Support Vectors
nb_sv = len(Isv.nonzero()[0])
if nb_sv > 1:
b = (Isv.T).dot( l - Ker.dot(np.squeeze(np.array(xi))) )/ nb_sv
else:
b = 0
# Continuous score function
f_test = xi.dot(KXtest) + b
# Binary classification function
C_test = np.sign(f_test) # decision function in {-1,1}
accuracy_test = compute_purity(0.5*(1+C_test),Cgt_test,nc) # 0.5*(1+C_test) in {0,1}
# Print
# print('iter, diff_alpha',str(k),str(diff_alpha)[:7])
# print('acc',str(accuracy_test)[:5])
print('Graph SVM iter, diff_alpha :',str(k),str(diff_alpha)[:7])
print(' acc :',str(accuracy_test)[:5])
k-NN graph with cosine distance
Graph SVM iter, diff_alpha : 2 0.0
acc : 78.5
代码
文本
点个赞吧
本文被以下合集收录
Graph Machine learning
xuxh@dp.tech
更新于 2024-10-08
44 篇0 人关注
推荐阅读
公开
scATAC数据原始数据预处理流程zhouziyi
发布于 2023-11-12
公开
如何用Coze手搓一个学术版Perplexity（极简）
wangmin@dp.tech
发布于 2024-05-13
6 赞