pycodesuggest - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #react #go #golang #rust #deep-learning #nodejs #java #windows #vue #pytorch #mcp #open-source

pycodesuggest

(★ 154)

Learning to Auto-Complete using RNN Language Models

파일 탐색기

최종 버전 다운로드 (.zip)

- cloned_repos.dat
- test_files.txt
- train_files.txt
- valid_files.txt
- ast_transforms.py
- astwalker.py
- norm.out0
- normalisation.py
- normalise.sh
- processFiles.py
- PyRepo.py
- scraper.py
- attention_rnn.py
- batcher.py
- beamSearch.py
- BeamSearchTree.py
- datawriter.py
- evaluation.py
- hooks.py
- model.py
- prebatcher.py
- pyreader.py
- pythonLanguageModel.py
- rnn.py
- termcolor.py
- tfutils.py
- Trainer.py
- utils.py
- ngram.arpa
- ngram.py
- reader.py
- convolution_attention_sourcecode.pdf
- Dependency_RNN_For_Sentence_Completion.pdf
- latent_predictor_networks_for_code.pdf
- Mining_idioms_from_source_code.pdf
- on_naturalness_of_software.pdf
- rnn_grammars.pdf
- source_code_visualisation.pdf
- toward_deep_learning_of_software_repos.pdf
.gitignore
.gitmodules
LICENSE.txt
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Learning Python Code Suggestion with a Sparse Pointer Network

This repository contains the code used in the paper "Learning Python Code Suggestion with a Sparse Pointer Network"

## Prerequisites
* [Python 3.5+](https://www.python.org/)
* [Git 1.7+](https://git-scm.com/)
* [Tensorflow 0.9+](www.tensorflow.org) (Tested on 0.9 - 0.11)
* [Github3.py](https://github3py.readthedocs.io/en/master/)
* [GitPython](http://gitpython.readthedocs.io/en/stable/intro.html)
* An account at [Github](https://github.com)

## Generating the Corpus
### Step 1: Cloning the Repos
To recreate the corpus used in the paper, run:

`python3 github-scraper/scraper.py --mode=recreate --outdir=<PATH-TO-OUTPUT-DIR> --dbfile=/FULL/PATH/TO/pycodesuggest/data/cloned_repos.dat --githubuser=<GITHUB USERNAME>`

Where outdir is the path on your local machine where the repos will be cloned. Note that the dbfile path should be the *full* path on your machine. You may be prompted for your Github password.

---

To obtain a fresh corpus based on a new search of Github, using the same criteria as the paper, run:

`python3 github-scraper/scraper.py --mode=new --outdir=<PATH-TO-OUTPUT-DIR> --dbfile=cloned_repos.dat --githubuser=<GITHUB USERNAME>`

Note that you may interrupt the process and continue where it left off later by providing the same dbfile.

---

There are a number of other parameters that allow you to create your own custom corpus, specifying the *programming language* or search term used to query Github amongst others. Run  `python3 github-scraper/scraper.py -h` for more information

### Step 2: (OPTIONAL): Remove unnecessary files
Linux/Mac OS: Run the following command in your output directory to remove non Python files

`find . -type f ! -name "*.py" -delete`

### Step 3: Normalisation
Run the following command to normalise all files with a .py extension by providing the output directory of step 1 as the path. The normalised files will be written to a new directory with "normalised" appended to the path.

`python3 github-scraper/normalisation.py --path=<PATH TO DOWNLOADED CORPUS>`

Files which can't be parsed as valid Python3 will be ignored. The list of successfully processed files is written to PATH/processed.txt which also allows for the normalisation to continue if interrupted.

### Step 4: Split into train/dev/test
To use the same train/dev/test split as used in the paper, copy the files train_files.txt, valid_files.txt and test_files.txt from the data directory into the downloaded corpus and normalised corpus directories.

---

To generate a new split, run the following command which generates the list of train files (train_files.txt), validation files (valid_files.txt) and test files (test_files.txt) in the ratio 0.5/0.2/0.3. Use the **normalised** path from the previous step. This will ensure that the list of files is available in both the normalised and unnormalised data sets.

`python3 github-scraper/processFiles.py --path=<PATH TO NORMALISED CORPUS>`

Then copy the 3 generated lists to the original un-normalised path.

## Citing
If you make use of this code or the Python corpus, please cite:

```
@article{pycodesuggest,
  author    = {Avishkar Bhoopchand and
               Tim Rockt{\"{a}}schel and
               Earl Barr and
               Sebastian Riedel},
  title     = {Learning Python Code Suggestion with a Sparse Pointer Network},
  year      = {2016},
  url       = {http://arxiv.org/abs/1611.08307}
}
```