KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agents #developer-tools #codex #claude-code #mcp #rust #deepseek-harness #typescript #ai-agent #cli

deep_gp_random_features

★ 40 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# README #

This repository contains code to reproduce the results in the paper:

K. Cutajar, E. V. Bonilla, P. Michiardi, and M. Filippone. Random feature expansions for deep Gaussian processes. In Proceedings of the 33rd International Conference on Machine Learning, ICML 2017, Sydney, Australia, August 6-11, 2017, 2017.
(Link: http://proceedings.mlr.press/v70/cutajar17a/cutajar17a.pdf)

The code is written in Python and uses the TensorFlow module; follow https://www.tensorflow.org to install TensorFlow.

Currently the code is structured so that the learning of DGPs is done using stochastic gradient optimization and a loss function of interest is displayed on the test error every fixed number of iterations.

29/04/2019 - The original implementation has been updated to work with Python 3.6 and the latest version of TensorFlow (1.13.1). This latest version of the code also introduces an additional option for utilising the local reparameterization trick in order to accelerate convergence during training.

## Flags ##

The code implements variational inference for a deep Gaussian process approximated using random Fourier features. The code accepts the following options:

*   -h, --help            Show help message and exit
*   --batch_size          Batch size
*   --learning_rate       Initial learning rate
*   --n_iterations        Number of iterations (batches) to train the DGP model
*   --display_step        Display progress every FLAGS.display_step iterations
*   --mc_train            Number of Monte Carlo samples used to compute stochastic gradients
*   --mc_test             Number of Monte Carlo samples for predictions
*   --n_rff               Number of random features for each layer
*   --df                  Number of GPs per hidden layer
*   --nl                  Number of layers
*   --optimizer           Select the optimizer: it can be adam, adagrad, adadelta, sgd
*   --kernel_type         Kernel: it can be RBF or arccosine
*   --kernel_arccosine_degree  Degree parameter of arc-cosine kernel
*   --is_ard              Using ARD kernel or isotropic
*   --feed_forward        Feed original inputs to each layer
*   --local_reparam       Use the local reparameterization trick
*   --q_Omega_fixed       Number of iterations to keep posterior over Omega fixed
*   --theta_fixed         Number of iterations to keep theta fixed
*   --learn_Omega         How to treat Omega - it can be 'prior_fixed' for Omega obtained from the prior with fixed randomness, 'var_fixed' for variational with fixed randomness, or 'var_resampled' for variational with resampling
*   --duration            Duration of job in minutes
*   --dataset             Dataset name
*   --fold                Dataset fold
*   --seed                Seed for random tf and np operations
*   --less_prints         Disables evaluations involving on the complete training set

Flags for the distributed version

*   --ps_hosts            Comma-separated list of hostname:port pairs
*   --worker_hosts        Comma-separated list of hostname:port pairs
*   --job_name            One of 'ps', 'worker'
*   --task_index          Index of task within the job

## Examples ##

Here are a few examples to run the Deep GP model on various datasets (we assume that the code directory is in PYTHONPATH - otherwise, please append PYTHONPATH=. at the beginning of the commands below):

### Regression ###

```
#!bash
# Learn a DGP model with two layers (nl=2) and three GPs in the hidden layer (df=3) on a regression problem (dataset=concrete). The kernel is RBF by default and use the ARD formulation. Approximate the GPs with 100 random Fourier features.
# Set the optimizer to adam with step-size of 0.01 with a batch size of 200. Use 100 Monte Carlo samples to estimate stochastic gradients (mc_train=100) and use 100 Monte Carlo samples to carry out predictions (mc_test=100).
# Cap the running of the code to 60min and to 100K iterations. Learn Omega variationally, and fix the approximate posterior over Omega and the GP covariance parameters for the first 1000 and 4000 iterations, respectively (q_Omega_fixed=1000 and theta_fixed=4000).

python experiments/dgp_rff_regression.py --seed=12345 --dataset=concrete --fold=1 --q_Omega_fixed=1000 --theta_fixed=4000 --is_ard=True --optimizer=adam --nl=2 --learning_rate=0.01 --n_rff=100 --df=3 --batch_size=200 --mc_train=100 --mc_test=100 --n_iterations=100000 --display_step=250 --duration=60 --learn_Omega=var_resampled

```

```
#!bash
# Here is an example where we fix the random Fourier features from the prior induced by the GP approximation (learn_Omega=prior_fixed).

python experiments/dgp_rff_regression.py --seed=12345 --dataset=concrete --fold=1 --theta_fixed=4000 --is_ard=True --optimizer=adam --nl=2 --learning_rate=0.01 --n_rff=100 --df=3 --batch_size=200 --mc_train=100 --mc_test=100 --n_iterations=100000 --display_step=250 --duration=60 --learn_Omega=prior_fixed --local_reparam=True

```

### Binary Classification ###
```
#!bash
# Same as the first example but for a classification problem (dataset=credit) and optimizing the spectral frequencies Omega (learn_Omega=var_fixed).

python experiments/dgp_rff_classification.py --seed=12345 --dataset=credit --fold=1 --q_Omega_fixed=1000 --theta_fixed=4000 --is_ard=True --optimizer=adam --nl=2 --learning_rate=0.01 --n_rff=100 --df=3 --batch_size=200 --mc_train=100 --mc_test=100 --n_iterations=100000 --display_step=250 --duration=60 --learn_Omega=var_fixed --local_reparam=True

```

```
#!bash
# Here is an example with the arc-cosine kernel of degree 1.

python experiments/dgp_rff_classification.py --seed=12345 --dataset=credit --fold=1 --q_Omega_fixed=0 --theta_fixed=4000 --is_ard=True --optimizer=adam --nl=2 --learning_rate=0.01 --n_rff=100 --df=3 --batch_size=200 --mc_train=100 --mc_test=100 --n_iterations=100000 --display_step=250 --duration=60 --learn_Omega=var_fixed --kernel_type=arccosine --kernel_arccosine_degree=1 --local_reparam=True

```

### MNIST (Multiclass classification) ###

```
#!bash
# Here is the MNIST example, where we use a two-layer DGP with 50 GPs in the hidden layer. We use 500 random Fourier features to approximate the GPs.
# In this example we use the option less_prints to avoid computing the loss on the full training data every 250 iterations.

python experiments/dgp_rff_mnist.py --seed=12345 --theta_fixed=4000 --is_ard=True --optimizer=adam --nl=2 --learning_rate=0.001 --n_rff=500 --df=50 --batch_size=1000 --mc_train=100 --mc_test=50 --n_iterations=100000 --display_step=250 --less_prints=True --duration=1200 --learn_Omega=var_fixed --local_reparam=True

```

### MNIST8M (Multiclass classification) ###

```
#!bash
# Here is the MNIST8M example - same settings as the MNIST example
# NOTE: Before running the code, please download the infinite MNIST dataset from here: http://leon.bottou.org/_media/projects/infimnist.tar.gz

python experiments/dgp_rff_infmnist.py --seed=12345 --theta_fixed=4000 --is_ard=True --optimizer=adam --nl=2 --learning_rate=0.001 --n_rff=500 --df=50 --batch_size=1000 --mc_train=40 --mc_test=100 --n_iterations=100000 --display_step=1000 --less_prints=True --duration=1200 --learn_Omega=var_fixed

```

deep_gp_random_features

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README