ntds_2016

KO | EN

gitlite — search

#javascript #php #ruby #python #java #erlang #c-sharp #jquery #logic-programming #android #game-engine #module

★ 99 Open GitHub ↗

Material for the EPFL master course "A Network Tour of Data Science", edition 2016.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# A Network Tour of Data Science, edition 2016

[![Binder](https://mybinder.org/badge_logo.svg)][binder]

[binder]: https://mybinder.org/v2/gh/mdeff/ntds_2016/with_outputs?urlpath=tree

This repository contains the material for the practical work associated with the EPFL master course [EE-558 A Network Tour of Data Science][epfl] ([moodle]), taught in fall 2016.
The course material revolves around the following topics:
[Data Science](https://en.wikipedia.org/wiki/Data_science),
[Machine Learning](https://en.wikipedia.org/wiki/Machine_learning),
[Deep Learning](https://en.wikipedia.org/wiki/Deep_learning),
[Graph Signal Processing](https://arxiv.org/abs/1211.0053).

Theoretical knowledge is taught during lectures.
Practical knowledge is taught through [exercises](#content).
Both are practiced and evaluated through [four assignments](#content) and a [semester project](#projects).

Look at the [2017], [2018], and [2019] editions for a course more focused on graphs and networks (instead of deep learning).

[epfl]: http://edu.epfl.ch/coursebook/en/a-network-tour-of-data-science-EE-558
[moodle]: http://moodle.epfl.ch/course/view.php?id=15299
[2017]: https://github.com/mdeff/ntds_2017
[2018]: https://github.com/mdeff/ntds_2018
[2019]: https://github.com/mdeff/ntds_2019

## Content

There is two types of exercises.

The **Data Scientist toolkit**, a set of [tools][toolkit], mostly in Python, to
help during the Data Science process.

1. [Introduction][t00_intro].
2. Data acquisition & exploration: [demo][t01_demo], [exercise][t01_ex], [solution][t01_sol].
3. Data exploitation: [demo][t02_demo], [exercise][t02_ex], [solution][t02_sol].
4. High Performance Computing: [exercise][t03_ex], [solution][t03_sol].
5. Data visualization: [exercise][t04_ex], [solution][t04_sol].

[toolkit]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/tree/with_outputs/toolkit
[t00_intro]:  https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/00_introduction.ipynb
[t01_demo]:   https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/01_demo_acquisition_exploration.ipynb
[t01_ex]:     https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/01_ex_acquisition_exploration.ipynb
[t01_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/01_sol_acquisition_exploration.ipynb
[t02_demo]:   https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/02_demo_exploitation.ipynb
[t02_ex]:     https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/02_ex_exploitation.ipynb
[t02_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/02_sol_exploitation.ipynb
[t03_ex]:     https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/03_ex_hpc.ipynb
[t03_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/03_sol_hpc.ipynb
[t04_ex]:     https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/04_ex_visualization.ipynb
[t04_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/toolkit/04_sol_visualization.ipynb

**Machine Learning** (ML) & **Graph Signal Processing** (GSP) [algorithms].
These exercises are designed so as to familiarize yourself with the algorithms
presented in class.

1. Graph Science: [exercise][a01_ex], [solution][a01_sol].
2. Clustering: [exercise][a02_ex], [solution][a02_sol], [assignment][a02_ass], [solution][a02_sass].
3. Classification: [exercise][a03_ex], [solution][a03_sol].
4. TensorFlow: [exercise][a04_ex], [solution][a04_sol].
5. Neural Networks: [assignment][a05_ass], [solution][a05_sol].
6. Recurrent Neural Networks: [assignment][a06_ass], [solution][a06_sol].
7. Graph Fourier Transform: [exercise][a07_ex], [solution][a07_sol].
8. Transductive Learning using Graphs: [assignment][a08_ass], [solution][a08_sol].

[algorithms]: https://nbviewer.jupyter.org/github/mdeff/ntds_2016/tree/with_outputs/algorithms
[a01_ex]:     https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/01_ex_graph_science.ipynb
[a01_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/01_sol_graph_science.ipynb
[a02_ex]:     https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/02_ex_clustering.ipynb
[a02_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/02_sol_clustering.ipynb
[a02_ass]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/02_ass_clustering.ipynb
[a02_sass]:   https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/02_sol_assignment.ipynb
[a03_ex]:     https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/03_ex_classification.ipynb
[a03_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/03_sol_classification.ipynb
[a04_ex]:     https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/04_ex_tensorflow.ipynb
[a04_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/04_sol_tensorflow.ipynb
[a05_ass]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/05_ass_convnet.ipynb
[a05_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/05_sol_convnet.ipynb
[a06_ass]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/06_ass_recurrent_nn.ipynb
[a06_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/06_sol_recurrent_nn.ipynb
[a07_ex]:     https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/07_ex_graph_fourier.ipynb
[a07_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/07_sol_graph_fourier.ipynb
[a08_ass]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/08_ass_graph_inpainting.ipynb
[a08_sol]:    https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/algorithms/08_sol_graph_inpainting.ipynb

## Projects

Part of the course is evaluated by a [project][desc], proposed and carried out
by groups of one to three students.
Below is the work of the 38 students enrolled that year.

* [[proposal][01p], [analysis][01r], [slides][01s]] Breast Cancer Classifcation, *Robin Demesmaeker*
* [[proposal][02p], [analysis][02r], [slides][02s]] How Fake News Go Viral?, *Victor Kristof, William Trouleau*
* [[proposal][03p], [analysis][03r], [slides][03s]] Twitter User Gender Classification, *Gaétan Ramet, Benjamin Schloesing, Yuan Yao*
* [[proposal][04p], [analysis][04r], [slides][04s]] Youtube Fame Predictor, *Benoît Steinmann, Cyrille Rolland*
* [[proposal][05p], [analysis][05r], [slides][05s]] Emotion Recognition from Faces, *Patryk Oleniuk, Carmen Galotta*
* [[proposal][06p], [analysis][06r], [slides][06s]] Airbnb New User Bookings, *Pecoraro Cyril, Jaume Guillaume, Grisard Malo*
* [[proposal][07p], [analysis][07r], [slides][07s]] Predicting an Election from Tweets, *Ercolani Chiara, Vorobiev Mikhail, Juillard Michaël*
* [[proposal][08p], [analysis][08r], [slides][08s]] Fisheries Monitoring, *Damian Pascual Ortiz, Pablo Mainar Jovaní*
* [[proposal][09p], [analysis][09r], [slides][09s]] El Nino, *Cotting Matthieu, Bonelli Alberto, Allani Mohamed*
* [[proposal][10p], [analysis][10r], [slides][10s]] Alzheimer Disease Detection, *Christian Abbet, Maxime Bonhenblust, Nicolas Masserey*
* [[proposal][11p], [analysis][11r], [slides][11s]] Estimating Hyper-Parameters for Compressed Sensing, *Dimitris Perdios*
* [[proposal][12p], [analysis][12r], [slides][12s]] Global Warming, *Effrosyni Simou*
* [[proposal][13p], [analysis][13r], [slides][13s]] Daily News for Stock Market Prediction, *Jeroen Le Maire*
* [[proposal][14p], [analysis][14r], [slides][14s]] Open Source Software Support, *Pavlos Nikolopoulos, Matthaios Olma, Stefanos Skalistis*
* [[proposal][15p], [analysis][15r], [slides][15s]] Epileptic Seizures Prediction, *Sophie du Bois*
* [[proposal][16p], [analysis][16r], [slides][16s]] Product Recommendation, *Berke Aral Sönmez, Alper Köse*
* [[proposal][17p], [analysis][17r], [slides][17s]] Sentiment Analysis, *Meryem Wehbe, Samuel Beuret, Valentine Santarelli*
* [[proposal][18p], [analysis][18r], [slides][18s]] Bike Sharing Demand, *Vincent Hardy*

![projects](https://raw.githubusercontent.com/mdeff/ntds_2016/master/project/projects.jpg)

[desc]: https://nbviewer.jupyter.org/github/mdeff/ntds_2016/blob/with_outputs/project/description.pdf

[01p]: project/proposals/breast_cancer.pdf
[01r]: project/reports/breast_cancer/
[01s]: project/slides/breast_cancer.pdf
[02p]: project/proposals/fake_news.pdf
[02r]: project/reports/fake_news/
[02s]: project/slides/fake_news.pdf
[03p]: project/proposals/twitter_gender.pdf
[03r]: project/reports/twitter_gender/
[03s]: project/slides/twitter_gender.pdf
[04p]: project/proposals/youtube_fame.pdf
[04r]: project/reports/youtube_fame/
[04s]: project/slides/youtube_fame.pdf
[05p]: project/proposals/emotion_recognition.pdf
[05r]: project/reports/emotion_recognition/
[05s]: project/slides/emotion_recognition.pdf
[06p]: project/proposals/airbnb_booking.pdf
[06r]: project/reports/airbnb_booking/
[06s]: project/slides/airbnb_booking.pdf
[07p]: project/proposals/election_prediction.pdf
[07r]: project/reports/election_prediction/
[07s]: project/slides/election_prediction.pdf
[08p]: project/proposals/fisheries_monitoring.pdf
[08r]: project/reports/fisheries_monitoring/
[08s]: project/slides/fisheries_monitoring.pdf
[09p]: project/proposals/el_nino.pdf
[09r]: project/reports/el_nino/
[09s]: project/slides/el_nino.pdf
[10p]: project/proposals/alzheimer_detection.pdf
[10r]: project/reports/alzheimer_detection/
[10s]: project/slides/alzheimer_detection.pdf
[11p]: project/proposals/compressed_sensing.pdf
[11r]: project/reports/compressed_sensing/
[11s]: project/slides/compressed_sensing.pdf
[12p]: project/proposals/global_warming.pdf
[12r]: project/reports/global_warming/
[12s]: project/slides/global_warming.pdf
[13p]: project/proposals/stock_market.pdf
[13r]: project/reports/stock_market/
[13s]: project/slides/stock_market.pdf
[14p]: project/proposals/software_support.pdf
[14r]: project/reports/software_support/
[14s]: project/slides/software_support.pdf
[15p]: project/proposals/epileptic_seizures.pdf
[15r]: project/reports/epileptic_seizures/
[15s]: project/slides/epileptic_seizures.pdf
[16p]: project/proposals/product_recommendation.pdf
[16r]: project/reports/product_recommendation/
[16s]: project/slides/product_recommendation.pdf
[17p]: project/proposals/sentiment_analysis.pdf
[17r]: project/reports/sentiment_analysis/
[17s]: project/slides/sentiment_analysis.pdf
[18p]: project/proposals/bike_sharing.pdf
[18r]: project/reports/bike_sharing/
[18s]: project/slides/bike_sharing.pdf

## Installation with Docker [![](https://images.microbadger.com/badges/image/mdeff/ntds_2016.svg)][image]

The easiest way to play with the code is to run it inside a [docker] container,
a [lightweight virtualization method][virt].

[docker]: https://www.docker.com
[virt]: https://en.wikipedia.org/wiki/Operating-system-level_virtualization

1. [Install Docker][install] on your Windows, Mac or Linux machine.

2. Run the [image], which is automatically updated from this git repository.
   ```sh
   docker pull mdeff/ntds_2016  # to update it
   docker run --rm -i -p 8871:8888 -v ~/:/data/mount mdeff/ntds_2016
   ```

3. Access the container's Jupyter notebook at <http://localhost:8871>. There
   you'll find two folders:

* `repo` contains a copy of this git repository. Nothing you modify in this
	 folder is persistent. If you want to keep your modifications, use `File`,
	 `Download as`, `Notebook` in the Jupyter interface.
   * `mount` contains a view of your home directory, from which you can
     persistently modify any of your files.

Windows and Mac users may need to [redirect the port in
   VirtualBox][redirect].

[install]: https://docs.docker.com/engine/installation/
[image]: https://hub.docker.com/r/mdeff/ntds_2016/
[redirect]: https://stackoverflow.com/a/33642903/3734066

### Container modification

If you want to use it for your projects and need additional software or Python
packages, you'll need to install them into the container.

1. Create your named container.
   ```sh
   docker run -i -p 8871:8888 -v ~/:/data/mount --name myproject mdeff/ntds_2016
   ```

2. Once you stop it, you'll be able to start it again with `docker start
   myproject`.

3. In another terminal, install packages while the container is running.
   ```sh
   docker exec -i myproject /bin/bash
   pip install mypackage
   apt-get install myotherpackage
   ```

## Manual installation

**Warning**: this may be problematic for Windows users, as TensorFlow does not
support Windows yet.

1. Install Python.
	* Windows: we recommend to install [Anaconda]. Please install version 3.5.
	  Most of the packages we'll use during the exercises are included in the
	  distribution. An other option is the [Windows Subsystem for Linux][wsl],
	  available on Windows 10, which allows you to install packages as if you
	  were on Ubuntu.
	* Mac: we recommend that you use the [Homebrew] package manager and install
	  Python with `brew install python3`. You can also use [Anaconda].
	* Linux: please use your package manager to install the latest Python 3.x.

2. Clone the course repository. You may need to first install [git].
   ```sh
   git clone https://github.com/mdeff/ntds_2016.git
   cd ntds_2016
   ```

3. Optionally, create a [virtual environment][venv].
   ```sh
   pyvenv /path/to/new/virtual/env
   . /path/to/new/virtual/env/bin/activate
   ```
   > A virtual environment allows you to install a different set of packages for
   > each of your Python project. Each project thus stays cleanly separated from
   > each other. It is a good practice but by no means necessary. You can read
   > more about virtual environments on this [blog post][venv_blog]. Anaconda
   > users, see [here][conda_venv].

4. Install the packages we'll use from [PyPI], the Python Package Index.
   ```sh
   pip install -r requirements.txt  # or make install
   ```

* If it fails, it is probably because you need to install some native
	 packages with your package manager. Please read the error messages and
	 remember, Google is your friend ! You may look at the
	 [dockerfile](dockerfile) to get an idea of which setup is necessary on
	 a Debian / Ubuntu system.

* Depending on your installation, `pip` may refer to Python 2 (you can
	 verify with `pip -V`). In that case, use `pip3` instead of `pip`.

* Anaconda users can also install packages with `conda install packname`.
	 See [here][conda_install] for your options.

5. Verify that you have a working installation by running a simple test.
   Again, you may need to call `python3`.
   ```sh
   python check_install.py  # or make test
   ```

* If you are on Windows with Anaconda and get
	 `WARNING (theano.configdefaults): g++ not detected!`, you may want to
	 install [mingw-w64](http://mingw-w64.org) with `conda install mingw
	 libpython`. Otherwise your Deep Learning models will run extremly slowly.
	 This may however not work for Python 3.5, see this [GitHub
	 issue][theano_windows_py35] for a workaround.

6. Open the jupyter web interface and play with the notebooks !
   ```sh
   jupyter notebook
   ```

[Homebrew]: http://brew.sh
[wsl]: https://msdn.microsoft.com/en-us/commandline/wsl/about
[Anaconda]: https://www.continuum.io/downloads#windows
[conda_install]: http://stackoverflow.com/a/18640601/3734066
[conda_venv]: http://conda.pydata.org/docs/using/envs.html
[venv]: https://docs.python.org/3/library/venv.html
[venv_blog]: https://realpython.com/blog/python/python-virtual-environments-a-primer/
[PyPI]: https://pypi.python.org
[git]: https://git-scm.com/downloads
[theano_windows_py35]: https://github.com/Theano/Theano/issues/3376#issuecomment-235034897

## Team

* Lecturers:
[Xavier Bresson](http://www.ntu.edu.sg/home/xbresson),
[Pierre Vandergheynst](https://people.epfl.ch/pierre.vandergheynst).
* Assistant:
[Michaël Defferrard](http://deff.ch).

## License

All codes and examples are released under the terms of the [MIT
License](LICENSE.txt).

ntds_2016

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README