LogAnalyser

KO | EN

gitlite — search

#javascript #hacktoberfest #ai #typescript #llm #golang #linux #docker #nodejs #awesome #vue #security #ios #openai

(★ 9)

Winning submission of Cloud.init.io (A nationwide cloud-computing hackathon by Nutanix)

LogAnalyser 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LogAnalyser
A repo to host all of the code for Cloud.init.io

Our App, <Placeholder> provides an easy-to-use interface to handle distributed logging in the microservice world. As we all know,
microservices run on multiple hosts. To fulfill a single business requirement, we might need to talk to multiple services running
on different machines. So, the log messages generated by microservices are distributed across multiple hosts.

Therefore as a developer or administrator, if you want to troubleshoot any issue, that leaves you clueless. 
  You don’t know which microservice running on which host served your request. Even if you know which host(s) served your
  request, going to different hosts and grepping the logs and correlating them across all the microservices requests is a 
  cumbersome process. If your environment is auto-scaled, then troubleshooting an issue is unimaginable.
  
  To tackle all of the above mentioned issues, we have developed a web app that will make life easier when troubleshooting 
  issues in the microservices world.
  
  ## Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

### Prerequisites

* django
* python3-tk
* matplotlib
* re
* pandas

Use the following command to install any unmet dependencies
```
sudo apt-get install <package>
```
or
```
pip3 install <package>
```

### Installing

A step by step series of examples that tell you how to get a development env running

Clone the repo

```
git clone https://github.com/SnehalRaj/LogAnalyser.git
```
create and start a virtual environment

```
virtualenv -p python3 env --no-site-packages

source env/bin/activate
```
Install the project dependencies:
```
sudo pip3 install -r requirements.txt

```
Install django (Run the command again even if you have it)
```
pip3 install django
```

Run the script addkey to add a secret key in your Django project
```
./addkey
```

Then run
```
python3 manage.py migrate
```
NOTE: The script might give errors as some packages(pandas,matplotlib etc) would be missing on the local device. Please install any missing packages using the following command
```
pip3 install <package-name>
```

And Finally run

```
python3 manage.py runserver
```
Now, you can go to [localhost](http://127.0.0.1:8000/) and access the site

### Navigating the site

The Site has an interactive dashboard having different options to operate the site. To start logging you have enter
the number of nodes and processes and then click on start

This will start logging data into a folder called "log_simulator" present inside your repository.

Now you can click on to any of the buttons given in the dashboard to navigate the site.

#### Live
  This tab would show the live logs corresponding to the nodes and processes entered. The logs are separated into separate tables for separate nodes and updates in real time to show the latest data to the user.
#### Statistics
This tab features an interactive viewing experience of the data collected while logging. The user can input multiple keywords in the search box (separated by comma) and get the data regarding the occurences of the those keywords in a visual format
#### Timestamp View
The Timestamp view tab allows the user to filter out logs between two given timestamps. The user can input two timestamps in the format specified and our app automatically filters out logs between the given timestamps and shows the logs in a tabular format
#### Timeline
This tab shows all of the logs which are marked by timestamps in a tabular format.

In all of the tabs, we have used dynamic datatables to shows our logged data. The table allows the user to search and sort based on specific keywords in real time and provides a lot of information in an interactive manner.
### Disclaimer
 The logs are continously stored in a file named "log_simulator" inside the cloned repo. Remember to stop the logging process by clicking the stop button in the home tab of the app so that you don't run out of memory!

### Features

* Live Debugging : Owing to the nature of general microservices loggers and the vast amount of logs generated, we have included
live debugging in our web app which shows the real-time log data to a user for a better understanding of the requests and errors.
Our App has an inbuilt delay (20 secs) in the processing of requests and thus provides the latest logs to the user in a dynamic table
where S/he can order the data based on any of fields in the table.

* Log Structures Data : We have structured the logs in order to maximize the ease of use of an Administrator to debug
 the issues present in the microservice. We have used dynamic tables to display the logs with each column storing a relevant data field
 so that the logs can easily be grouped and searched for relevant information easily
 
* Dynamic logging levels and Async Logging : We have provided the functionality to use different log levels in our code and 
deal with a logger which has a lot of logging statements in the code as well. We have tried to provided the liberty to 
change the log level dynamically. A user may choose if he wants to include tracebacks dynamically within the table itself which
will allow the user to avoid the overhead of excessive logging that way.

* Timeline view : Our App shows the logs in an interactive timeline view which helps the user to view only those logs which are marked with a timestamp. This view ignores all other logs and error and thus has sorted timeline data corresponding to the logs of a specific node and process entered by the user.

* Searchable Logs : We have made sure that all the fields available in the logs are searchable. For example, If you get a hold
of the Exception type, you can search all the logs based on that type to find out the request flow.This helps the user pin point errors relating to a particular file, exception or time.

* Timestamp Extraction: Our web app includes the feature of searching the logs via timestamps using which a user can easily extract
the logs present between two given timestamps. The logs are specific to the node and process which the user has entered.

* Correlation of logs by identifiers: We have included real time dynamic data of the frequency comparisons of certain keyword
identifiers in the logs of various processes across all nodes in a cluster. Our web app includes visual representation of 
different statistical data collected across the logs and compares them using pie charts, bar plots and histogram.

## Authors

* **Harsh Agarwal** - (https://github.com/Harshagarwal19)
* **Keshav Bansal** - (https://github.com/keshav22bansal)
* **Snehal Raj** - (https://github.com/SnehalRaj)

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

## Acknowledgments

* Sample code was provided by Nutanix during the hackathon Cloud.init.io

LogAnalyser

LogAnalyser 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

LogAnalyser

LogAnalyser 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README