spinarago

KO | EN

gitlite — search

#javascript #hacktoberfest #react #llm #golang #android #rust #deep-learning #docker #java #windows #pytorch #openai #ios

(★ 25)

A basic web crawler written in Go

spinarago Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# spinarago

A basic web crawler written in Go.

## Why "Spinarago"?

Spinarak = A spider Pokémon  
Go = Go

You do the math. :)

## Description

This is a basic web crawler that prints the site map of given URL. It does print external URLs, but doesn't follow them. This project is still a work in progress so
it's not feature complete. Feel free to make suggestions for improvements, either by creating issues or submiting pull requests.

## Install

```sh
$ go get github.com/danicat/spinarago
```

## Usage

```sh
$ spinarago --hostname <host> --delay <milliseconds> --level <max-depth>
```

I highly recommend for you to install [jq](https://stedolan.github.io/jq/) to pretty print the json output. Example:

```sh
$ spinarago --hostname http://example.com | jq
```

You can also redirect the stdout to a json file to make a site map dump:

```sh
$ spinarago --hostname http://example.com -level 1 -delay 10 > example_level1.json
```

`jq` is really handy to filter the output:

```sh
$ cat example_level1.json | jq '.[] | { url: .url }'
```

## TODO

- Handle relative paths

## Contributing

I'm open to contributions. Just create an issue and/or submit a pull request.

## Contact

Any comments please feel free to reach out to me at @danicat83 on Twitter.

spinarago

spinarago Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

spinarago

spinarago Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README