driving-with-data-detroit

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

driving-with-data-detroit

★ 8 Open GitHub ↗

Code for paper "Driving With Data: Modeling and Forecasting Vehicle Fleet Maintenance in Detroit".

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Driving with Data: Modeling and Forecasting Vehicle Fleet Maintenance in Detroit

Note: the data used for this analysis is not publicly available and is governed by a data non-disclosure agreement with the City of Detroit Operations and Infrastructure Group. This repository contains the complete source files used for the paper:

```
Driving with Data: Modeling and Forecasting Vehicle Fleet Maintenance in Detroit
J. Gardner, D. Koutra, J. Mroueh, V. Pang, A. Farahi, S. Krassenstein, and J. Webb.
https://arxiv.org/abs/1710.06839
```

![Detroit](https://raw.githubusercontent.com/jpgard/driving-with-data-detroit/master/img/readme/thumb_DSC_0437_1024.jpg)

This README provides guidance on how to replicate the PARAFAC/PRISM analysis presented in our paper, along with the LSTM maintenance prediction model. The ARIMA models are currently provided in a separate repository.

For the full results of our PARAFAC analysis on the Detroit vehicle maintenance dataset, see the [vehicle-year](img/3_way_plots/vehicle_year_log/README.md) and [month-year](img/3_way_plots/month_year_log/README.md) results pages.

# Installation

To replicate the "Driving with Data in Detroit" (DDD) analysis, first install the `ddd` module. The easiest way to do this is to clone the repo, create a virtual environment, and then simply run

``` 
$ pip3 install -e .
```

# PARAFAC/PRISM

This section shows the steps to preprocess a dataset, conduct PARAFAC and PRISM (PaRafac-Informed Sequence Mining), and visualize the results using three-way plots.

1. **Data preprocessing**

a. Run the preprocessing scripts to generate the 3-way data tensors needed in PARAFAC:

``` 
    $ python2 scripts/python2/generate_data_tensor.py -td month_year
    $ python2 scripts/python2/generate_data_tensor.py -td vehicle_year
    ```
    This will deposit CSV files of the data in `tensor-data/`.
    
    b. Generate the sequence data for each vehicle make/model. This will write sequence data in individual text files in `./freq-pattern-data/seqs/`.
    
    ``` 
    $ python2 scripts/python2/generate_vehicle_maintenance_sequences.py
    ```

2. **Run PARAFAC (this step requires a licensed MATLAB installation)**

a. In MATLAB, first run the `scripts/matlab/install_tensor_toolbox.m` script to load the PARAFAC libraries (included in this repo).
    
    b. Edit the file `scripts/matlab/tensor_decomp.m` to point to the relevant input files, and to have the correct output files. 
    
    c. Run `scripts/matlab/tensor_decomp.m` in MATLAB. As this script runs, you should see output of the following form:
    
    ``` 
    Nonnegative PARAFAC:
     Iter  1: fit = 1.495350e-01 fitdelta = 1.5e-01
     Iter  2: fit = 2.156228e-01 fitdelta = 6.6e-02
     Iter  3: fit = 2.765258e-01 fitdelta = 6.1e-02
     Iter  4: fit = 3.227131e-01 fitdelta = 4.6e-02
     Iter  5: fit = 3.548078e-01 fitdelta = 3.2e-02
     ... 
   ```

3. **Using the factor loading matrices, compute the in-group and out-groups via PRISM.** Note that the PRISM scripts are written in *Python 3* due to the Bayesian modeling libraries being only available in Python 3.

``` 
    $ python3 scripts/python3/run_prism.py -td month_year
   $ python3 scripts/python3/run_prism.py -td vehicle_year
    ```

4. **Generate the 3-way plots in R using `scripts/R/tensor_plots.Rmd`.**

a. Edit the file `scripts/R/tensor_plots.Rmd` to point to the correct factor loading matrices (by default these should be correct) and set the `analysis_type` variable to either `vehicle_year_log` or `month_year_log` as desired.
    
    b. Run `scripts/R/tensor_plots.Rmd`. This script can be run in an IDE (e.g. RStudio), or from the command line via
    
    ```
    $ Rscript scripts/R/tensor_plots.R
    ```
    This will produce a series of plots in `img` similar to those shown below.
    
    ![3way-example](https://raw.githubusercontent.com/jpgard/driving-with-data-detroit/master/img/readme/3way_monthyear_example.png)

# Maintenance Prediction LSTM

This section describes the steps to train and evaluate the LSTM described in the paper for maintenance prediction. Note that the LSTM model is modified from the original code in the Tensorflow 1.x repository [here](https://github.com/tensorflow/docs/blob/master/site/en/r1/tutorials/sequences/recurrent.md). Note that TensorFlow 1.x is in *maintenance mode only* as of 2020.

1. **Create the files with maintenance sequences by make/model.** Each line in the file represents a unique vehicle; each file represents a specific make/model.
```$ python3 generate_vehicle_maintenance_sequences.py```

2. **Preprocess the sequences specifically to match the input format for the TensorFlow script.**
```$ python3 scripts/python3/lstm_preproc.py ```

3. **Train the model.** Note that this script is in PYTHON 2; the other scripts are in PYTHON 3!
```$ python2 ptb/ptb_word_lm.py  --data_path=/path/to/output/dir/from/lstm_preproc/ ```

```
    ...
    Epoch: 1 Learning rate: 1.000
    Epoch: 1 Train Perplexity: 593.959
    Epoch: 1 Valid Perplexity: 114.834
    Epoch: 2 Learning rate: 1.000
    Epoch: 2 Train Perplexity: 77.896
    Epoch: 2 Valid Perplexity: 41.869
    Epoch: 3 Learning rate: 1.000
    Epoch: 3 Train Perplexity: 48.058
    Epoch: 3 Valid Perplexity: 47.353
    Epoch: 4 Learning rate: 1.000
    Epoch: 4 Train Perplexity: 38.648
    Epoch: 4 Valid Perplexity: 29.654
    Epoch: 5 Learning rate: 0.500
    Epoch: 5 Train Perplexity: 23.264
    Epoch: 5 Valid Perplexity: 22.252
    Epoch: 6 Learning rate: 0.250
    Epoch: 6 Train Perplexity: 17.788
    Epoch: 6 Valid Perplexity: 15.532
    Epoch: 7 Learning rate: 0.125
    Epoch: 7 Train Perplexity: 16.802
    Epoch: 7 Valid Perplexity: 15.344
    Epoch: 8 Learning rate: 0.062
    Epoch: 8 Train Perplexity: 16.635
    Epoch: 8 Valid Perplexity: 15.284
    ...
    ```

driving-with-data-detroit

Today's picks

Was this content helpful?

Recent Feedback

driving-with-data-detroit

Today's picks

Was this content helpful?

Recent Feedback

Download README