Random Feature Expansions for Deep Gaussian Processes

Kích thước: px

Bắt đầu hiển thị từ trang:

Download "Random Feature Expansions for Deep Gaussian Processes"

Lâm Khang
4 năm trước
Lượt xem:

2 ) = ) θ ) )... ) θ ))) ) ) ) ) ) ) ) ) )

3 , θ) = ) ), θ ) ) ) ), θ )... ), θ )) )... )

12 Φ =Φ Ω )=N, )

14 σ Φ = [ Ω), Ω)] ) Ω θ = N, Λ ) σ Λ =,..., )

15 σ Φ =, Ω) ) Ω θ = N, Λ )

16 σ Φ = [ Ω), Ω)] Ω θ ) = N, Λ ) σ Λ =,..., ) σ Φ =, Ω) Ω θ ) = N, Λ )

17 Φ ) ) Φ ) ) Ω ) ) Ω ) ) θ ) θ )

19 Ψ = Ω ),...,Ω ), ),..., )) [ θ)] E Ψ) [ Ψ)]) [Ψ) Ψ θ)] Ψ) Ψ, θ)

20 , Ψ, θ) =, Ψ, θ) E Ψ) [, Ψ, θ)]) E Ψ) [, Ψ, θ)]) I E Ψ) ) [, Ψ, θ)] [, Ψ, θ)] = Ψ Ψ)

21 Ψ) = Ω ) ) ) ) Ω ) ) = N ) µ ), σ ) ) ) ) = N ), ) ) )

22 Ψ) = Ω ) ) ) ) Ω ) ) = N ) µ ), σ ) ) ) ) = N ), ) ) ) ) ) = ) ) ϵ ) + ) ϵ ) N, )

23 Ω RMSE MNLL log 10 RFs) log 10 RFs) prior-fixed var-fixed var-resampled

25 dgp-rbf dgp-arc dgp-ep dnn var-gp

26 dgp-rbf dgp-arc dgp-ep dnn var-gp

27 Dataset Accuracy RBF ARC MNLL RBF ARC MNIST8M 99.14% 99.04% AIRLINE 78.55% 72.76%

28 layers 10 layers 20 layers 30 layers SV-DKL

29 layers 10 layers 20 layers 30 layers SV-DKL

..dF 1) ; GPs are single layered Neural Nets with an infinite number of hidden units; Taking a weight-space view of a GP: F =ΦW ; The priors over the weights are: p W i) =N 0,I) ; The RBF kernel: [

34 Deep probabilistic models; Composition of functions: fx) = h Nh 1) θ Nh 1))... h 0) θ 0))) x); Fig. 1: Illustration of how stochastic processes may be composed. Inference requires calculating the marginal likelihood: py X, θ) = Extremely challenging! p Y F Nh), θ Nh)) p... p F 1) X, θ 0)) df Nh)...dF 1) ; GPs are single layered Neural Nets with an infinite number of hidden units; Taking a weight-space view of a GP: F =ΦW ; The priors over the weights are: p W i) =N 0,I) ; The RBF kernel: [ krbfx, x )=exp 1 ] 2 x x ) x x ) = X Ω Φ W Fig. 2: Single layered GP. pω)exp ιx x ) ω ) dω can be approximated using trigonometric functions: σ Φrbf = 2 [cos F Ω), sin F Ω)] with p Ω j θ) =N 0, Λ 1), NRF Meanwhile, the first order Arc-Cosine kernel: karcx, x )= 1 π x x Jα) =2 max0, ω x) max0, ω x )N ω 0,I) dω x x can be approximated using Rectified Linear Units ReLU): 2σ Φarc = 2 max 0,FΩ) with p Ω j θ) =N 0, Λ 1). NRF ) F where DGPs with RFs become DNNs with low-rank weight matrices! X Φ 0) 1) 2) F Φ1) F Y 0) 0) 1) Ω W Ω1) W θ 0) Define Ψ =Ω 0),...,W 0),...); Lower bound on Marginal Likelihood: θ1) Fig. 3: Diagram of the proposed DGP with random features. log [py θ)] E qψ) log [p Y Ψ)]) DKL [qψ) p Ψ θ)], where qψ) approximates pψ Y, θ); Factorized approximate posterior: with q ) = N ) q and ) q ijl q ), ) = N Assuming factorized likelihood, we can use mini-batch stochastic gradient optimization: n E qψ)log[pyk Ψ)]) DKL[qΨ) pψ θ)]; m k Im The expectation can be estimated using Monte Carlo: with Ψr qψ); NMC log[pyk Ψr)], NMC r=1 Computational cost dominated by low-rank matrix multiplication - no inverses required; Various optimization strategies for Ω available RMSE log10rfs) MNLL log10rfs) prior-fixed var-fixed var-resampled Fig. 4: Performance of different strategies for dealing with Ω as a function of the number of random features.these can be fixed PRIOR-FIXED), or treated variationally with fixed randomness VAR-FIXED and resampled at each iteration VAR-RESAMPLED). ) ; Competing methods: DGP with RBF Kernel DGP-RBF); DGP with first order Arc-Cosine Kernel DGP-ARC); DGP with Expectation Propagation DGP-EP); Sparse GP with Variational Inference VAR-GP) Regression Power plant Dataset n = 9568, d=4) Spam Dataset n = 4061, d=57) log10sec) log10sec) log10sec) log10sec) Classification EEG Dataset n = 14979, d=14) log10sec) log10sec) 1 Protein Dataset n = 45730, d=9) log10sec) log10sec) MNIST Dataset n = 60000, d=784) log10sec) DGP-RBF DGP-ARC DGP-EP VAR-GP log10sec) Fig. 5: Progression of RMSE and MNLL on test data over time for competing DGP models. Easily extendable architecture with option to feed-forward inputs Error rate log10sec) MNLL log10sec) NELBO Layers 2 layers 10 layers 20 layers 30 layers SV-DKL Fig. 6: Left and center - Performance of our model on the AIRLINE dataset as function of time for different depths. Right - The box plot of the negative lower bound confirms this is a suitable objective for model selection. Our contributions: Complete specification and evaluation of DGPs based on random features; Scalable and practical DGP inference - no inverses; Synchronous/asynchronous distributed implementation available; Ongoing work: Fastfood and other kernels; Convolutional DGPs for image processing. [1] A. Damianou and N. Lawrence. Deep Gaussian Processes, AISTATS [2] Y. Gal and Z. Gharamani. Dropout as a Bayesian Approximation, ICML Random Feature Expansions for Deep Gaussian Processes Kurt Cutajar 1, Edwin V. Bonilla 2, Pietro Michiardi 1, Maurizio Filippone EURECOM, Sophia Antipolis, France 2 - University of New South Wales, Australia Deep Gaussian Processes Model Architecture Experimental Setup and Results h 0) x) h 1) x) h 0) h 1) x) ) DGPs with Random Features F Nh) F Nh 1), θ Nh 1)) allowing for scaling factors σ 2 and Λ = diagl 2 1,...,l 2 d ) for the kernel and the features; Jα) =sinα +π α) cos α and α = cos 1 x x.. Stochastic Variational Inference W l) ij qψ) = ijl µ l) ij, σ2 ) l) ij Ω l) ij E qψ) log [pyk Ψ)]) 1 W l) ij Ω l) ij m l) ij, s2 ) l) ij Accuracy MNLL Conclusions References RMSE MNLL Accuracy MNLL RMSE MNLL Accuracy MNLL

Tài liệu tương tự

PHÂN LỚP DỮ LIỆU MẤT CÂN BẰNG VỚI THUẬT TOÁN HBU 1. GIỚI THIỆU NGUYỄN THỊ LAN ANH Khoa Tin học, Trường Đại học Sư phạm, Đại học Huế Tóm tắt: Dữ liệu m

PHÂN LỚP DỮ LIỆU MẤT CÂN BẰNG VỚI THUẬT TOÁN HBU 1. GIỚI THIỆU NGUYỄN THỊ LAN ANH Khoa Tin học, Trường Đại học Sư phạm, Đại học Huế Tóm tắt: Dữ liệu mất cân bằng là một trong những nguyên nhân làm giảm