Intrinsic-Dimension

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

Intrinsic-Dimension

★ 10 Open GitHub ↗

Code implementing the paper "Measuring intrinsic dimension of objective landscapes" to perform some research

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Measuring the Intrinsic Dimension of Objective Landscapes
Code implementing the paper ["Measuring the Intrinsic Dimension of Objective Landscapes"](https://arxiv.org/pdf/1804.08838.pdf).

I can reproduce, with minor variations, the overall results from the paper in Pytorch, and extend the research to a different projection matrix. As part of this work, I have provided the results obtained from this implementation and almost all the references (and resources that I find useful) utilized to complete it (see also the report in pdf for other references).

All the results follow the most similar implementation of the specific architecture shown in the paper. The projection used to obtain those results are the **Dense** and **Fastfood** ones (sparse projections give me some problems in Pytorch, see an issue in the reference) following the same rules of the paper.

As you can see from the plots there is no mean or std for the measured data. This is first because the released code is provided with seed (I really hope reproducible) and second, because providing those data means running all the computations at least three times and the computations of some of these architectures are very time-consuming and with limited computing resources at my disposal this is the best I can do.

All plots below can be investigated further by the code in id_plots.ipynb.
All the training can be started in windows systems utilizing the *.ps1 or if you have a *nix system all the .ps1 can be easily translated to .sh as the one provided in the example, I have moved all those scripts in the folder "automation_scripts" so to start one of this one can do copy the desired script in the main directory and start it from this point or modify the paths inside the script. Here there is some code to perform this operation:
```sh
# execute these line from the main directory
cp automation_scripts/<name_of_script>.sh .

# modify the scripts to run the specific version of id90 desired or change hyperparameters  
<name_of_script>.sh

```

# MNIST Results

![MNIST Results](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_general.png "MNIST Results")

## Fully Connected
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global| FC  | 199210  | 750  |
|Author Local | FC  | 199210  | ?  |
|Global| FC  | 199210  | 600  |
|Local | FC  | 199210  | 525  |

### Fully Connected - Global Baseline
![MNIST Fully Connected - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_FC_global.png "MNIST Fully Connected - Global Baseline")

### Fully Connected - Local Baseline
![MNIST Fully Connected - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_FC_local.png "MNIST Fully Connected - Local Baseline")

## LeNet 
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global|LeNet|  44426  | 290 |
|Author Local |LeNet|  44426  | 275 |
|Global|LeNet|  44426  | 170 |
|Local |LeNet|  44426  | 160 |

### Lenet - Global Baseline
![MNIST LeNet - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_LeNet_global.png "MNIST LeNet - Global Baseline")

### LeNet - Local Baseline
![MNIST LeNet - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_LeNet_global.png "MNIST LeNet - Local Baseline")

## Untied LeNet
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global| Untied LeNet  | 286334  | 600  |
|Author Local | Untied LeNet  | 286334  | 450  |
|Global| Untied LeNet  | 286334  | 350  |
|Local | Untied LeNet  | 286334  | 350  |

### Untied LeNet - Global Baseline
![MNIST Untied LeNet - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_Untied_LeNet_global.png "MNIST Untied LeNet - Global Baseline")

### Untied LeNet - Local Baseline
![MNIST Untied LeNet - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_Untied_LeNet_local.png "MNIST Untied LeNet - Local Baseline")

## FC LeNet
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global| FC LeNet  |  3640574 | 2000  |
|Author Local | FC LeNet  | 3640574  | 1400  |
|Global| FC LeNet  | 3640574  | 900  |
|Local | FC LeNet  | 3640574  | 800  |

### FC LeNet - Global Baseline
![MNIST FC LeNet - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_FC_LeNet_global.png "MNIST FC LeNet - Global Baseline")

### FC LeNet - Local Baseline
![MNIST FC LeNet - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_FC_LeNet_local.png "MNIST FC LeNet - Local Baseline")

## FC Tied LeNet
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global| FCTied LeNet  |  ? | 425  |
|Author Local | FCTied LeNet  |  ? | 400  |
|Global| FCTied LeNet  | 193370  | 400  |
|Local | FCTied LeNet  | 193370  | 400  |

### FC Tied LeNet - Global Baseline
![MNIST FC Tied LeNet - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_FC_Tied_LeNet_global.png "MNIST FC Tied LeNet - Global Baseline")

### FC Tied LeNet - Local Baseline
![MNIST FC Tied LeNet - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/MNIST_FC_Tied_LeNet_local.png "MNIST FC Tied LeNet - Local Baseline")

# CIFAR10 Results

![CIFAR10 Results](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_general.png "CIFAR10 Results")

## Fully Connected
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global| FC  | 1055610  | 9000  |
|Author Local | FC  | 1055610  | 8000  |
|Global| FC  | 1051930  | 5000  |
|Local | FC  | 1051930  | 10000  |
|FastJL | FC  | 1051930  | 4000  |

### Fully Connected - Global Baseline
The results for this case are different due to problems of unnmatching dimension for the architecture (I tried various dimension and used the one that is more similar).

![CIFAR10 Fully Connected - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_FC_global.png "CIFAR10 Fully Connected - Global Baseline")

### Fully Connected - Local Baseline
![CIFAR10 Fully Connected - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_FC_local.png "CIFAR10 Fully Connected - Local Baseline")

### Fully Connected - FastJL
![CIFAR10 Fully Connected - FastJL](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_FC_FastJL.png "CIFAR10 Fully Connected - FastJL")

## LeNet
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global|LeNet|  62006 | 1000 |
|Author Local |LeNet|  62006 | 2900 |
|Global|LeNet|  62006  | 600 |
|Local|LeNet|  62006  | 1700 |
|FastJL |LeNet|  62006  | 1250 |

### LeNet - Global Baseline
![CIFAR10 LeNet - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_LeNet_global.png "CIFAR10 LeNet - Global Baseline")

### LeNet - Local Baseline
![CIFAR10 LeNet - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_LeNet_local.png "CIFAR10 LeNet - Local Baseline")

### LeNet - FastJL
![CIFAR10 LeNet - FastJL](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_LeNet_FastJL.png "CIFAR10 LeNet - FastJL")

## Untied LeNet
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global| Untied LeNet  |  658238 | 9000? 2750*  |
|Author Local | Untied LeNet  |  658238 | 15000 |
|Global| Untied LeNet  | 658238  | 10000  |
|Local | Untied LeNet  | 658238  | >40000  |
|FastJL | Untied LeNet  | 658238  | 2000  |

\* in the paper they claim a $d_{int90}$ of 2750 but in the plots from the implementation it seems to be 9000, very close to my implementation

### Untied LeNet - Global Baseline
![CIFAR10 Untied LeNet - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_Untied_LeNet_global.png "CIFAR10 Untied LeNet - Global Baseline")

### Untied LeNet - Local Baseline
Unable to reproduce results, because my local implementation performs better than the global baseline of the paper so the baseline is very skewed and the 90% baseline is very difficult to reach with few parameters!

![CIFAR10 Untied LeNet - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_Untied_LeNet_local.png "CIFAR10 Untied LeNet - Local Baseline")

### Untied LeNet - FastJL

Little sparsification and variance problems after the drop in performance

![CIFAR10 Untied LeNet - FastJL](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_Untied_LeNet_FastJL.png "CIFAR10 Untied LeNet - FastJL")

## FC LeNet
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global| FC LeNet  | 16397726  | 35000  |
|Author Local | FC LeNet  | 16397726  |  >100000 |
|Global| FC LeNet  | 16397726  | 5000  |
|Local | FC LeNet  | 16397726  | 27000  |
|FastJL | FC LeNet  | 16397726  | 5000  |

Here the results are very different!

### FC LeNet - Gobal Baseline

![CIFAR10 FC LeNet - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_FC_LeNet_global.png "CIFAR10 FC LeNet - Global Baseline")

### FC LeNet - Local Baseline
![CIFAR10 FC LeNet - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_FC_LeNet_local.png "CIFAR10 FC LeNet - Local Baseline")

### FC LeNet - FastJL
![CIFAR10 FC LeNet - FastJL](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_FC_LeNet_FastJL.png "CIFAR10 FC LeNet - FastJL")

## FC Tied LeNet
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global| FCTied LeNet  |  ? | 2500  |
|Author Local | FCTied LeNet  |  ? | 4500  |
|Global| FCTied LeNet  |  297734 | 3000  |
|Local | FCTied LeNet  |  297734 | 8000  |
|FastJL | FCTied LeNet  |  297734 | 2500  |

### FC Tied LeNet - Global Baseline
![CIFAR10 FC Tied LeNet - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_FC_Tied_LeNet_global.png "CIFAR10 FC Tied LeNet - Global Baseline")

### FC Tied LeNet - Local Baseline
![CIFAR10 FC Tied LeNet - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_FC_Tied_LeNet_local.png "CIFAR10 FC Tied LeNet - Local Baseline")

### FC Tied LeNet - FastJL
Here the results of FastJL are very unstable the architecture reach the intrinsic dimension and then drop, I think because the sparsification and the difference in parameter's variance here play an important role.
![CIFAR10 FC Tied LeNet - FastJL](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_FC_Tied_LeNet_FastJL.png "CIFAR10 FC Tied LeNet - FastJL")

## ResNet
$Baseline$ | $Network$  | $D$  | $d_{int90}$  |
|:---|:---:|:---:|---:|
|Author Global| ResNet  | 280000?  | 1000-2000  |
|Author Local | ResNet  | 280000?  | 20000-50000  |
|Global| ResNet  | 292954 | 1000  |
|Local | ResNet  | 292954 | 12000  |
|FastJL | ResNet  | 292954 | 1000  |

The implementation of this architecture is very hard to match so the results are a bit different from the paper!

### ResNet - Global Baseline

![CIFAR10 Resnet - Global Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_Resnet_global.png "CIFAR10 Resnet - Global Baseline")

### ResNet - Local Baseline
![CIFAR10 Resnet - Local Baseline](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_Resnet_local.png "CIFAR10 Resnet - Local Baseline")

### ResNet - FastJL
![CIFAR10 Resnet - FastJL](https://raw.githubusercontent.com/AntoAndGar/Intrinsic-Dimension/main/figures/CIFAR10_Resnet_FastJL.png "CIFAR10 Resnet - FastJL")

# Reference
1. https://www.uber.com/en-IT/blog/intrinsic-dimension/
2. https://tomroth.com.au/notes/intdim/intdim/
3. https://github.com/uber-research/intrinsic-dimension
4. https://twitter.com/JevGamper/status/1240335205807816705?s=20
5. https://github.com/jgamper/intrinsic-dimensionality
6. https://github.com/tnwei/intrinsic-dimension
7. https://greydanus.github.io/2017/10/30/subspace-nn/
8. https://discuss.pytorch.org/t/locally-connected-layers/26979/2
9. https://www.cs.princeton.edu/~runzhey/demo/Geo-Intrinsic-Dimension.pdf
10. https://github.com/LangLeon/thesis-intrinsic-dimension
11. https://towardsdatascience.com/interesting-projections-where-pca-fails-fe64ddca73e6
12. https://www.oreilly.com/library/view/hands-on-convolutional-neural/9781789130331/a33f17be-9d32-4499-aa1c-c1a81e023eb7.xhtml
13. https://cs231n.github.io/convolutional-networks/#convert
14. https://math.stackexchange.com/questions/995623/why-are-randomly-drawn-vectors-nearly-perpendicular-in-high-dimensions/995678
15. https://www.youtube.com/watch?v=Y_Ac6KiQ1t0&t=1s
16. https://github.com/josuni/Intrinsic-Dimensionality-for-various-datasets
17. https://arxiv.org/pdf/1408.3060.pdf
18. https://johnthickstun.com/docs/fast_jlt.pdf
19. https://www.cs.technion.ac.il/~nailon/fjlt.pdf
20. https://arxiv.org/pdf/2204.01800.pdf
21. https://github.com/pytorch/pytorch/issues/88053

# Future works (time permitting)

- [ ] Using a beautiful projection like: https://arxiv.org/abs/1202.3033
- Try other datasets:
    - [x] FMNIST
    - [ ] EMNIST
    - [ ] FLOWERS102
    - [ ] ImageNet
- Try other architectures:
    - [ ] RNN
    - [ ] LSTM
    - [ ] Transformer
    - [ ] Inception
- [ ] Implementing RL tasks
- [ ] Evaluate performace with Regularization and Dropout

Intrinsic-Dimension

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README