RestaurantInspectionsSparkMLNET

KO | EN

gitlite — search

#javascript #ai #llm #golang #go #deep-learning #linux #macos #windows #awesome #cli #kubernetes #pytorch #open-source

RestaurantInspectionsSparkMLNET

(★ 15)

ETL & Data Enrichment with Spark.NET and ML.NET Automated (Auto) ML

RestaurantInspectionsSparkMLNET Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Restaurant Inspections ETL & Data Enrichment in Spark.NET and ML.NET Automated (Auto) ML

This sample takes a restaurant violation dataset from the NYC Open Data portal and process it using Spark.NET. Then, the processed data will be used to train a machine learning model that attempts to predict the grade an establishment will receive after an inspection. The model will be trained using ML.NET, an open-source, cross-platform machine learning framework. Finally, data for which no grade currently exists will be enriched using the trained model to assign an expected grade.

For a detailed write-up, check out the [ Restaurant Inspections ETL & Data Enrichment with Spark.NET and ML.NET Automated (Auto) ML blog post](http://luisquintanilla.me/2019/09/15/restaurant-inspections-etl-data-enrichment-spark-auto-ml-net/).

## Pre-requisites

This project was built using Ubuntu 18.04 but should work on Windows and Mac devices.

- .NET Core 2.1
- Java 8
- Apache Spark 2.4.1 with Hadoop 2.7
- .NET Spark Worker 0.4.0

## Solution description

### Understand the data

The dataset used in this solution is the [*DOHMH New York City Restaurant Inspection Results*](https://data.cityofnewyork.us/Health/DOHMH-New-York-City-Restaurant-Inspection-Results/43nn-pn8j) and comes from the NYC Open Data portal. It is updated daily and contains assigned and pending inspection results and violation citations for restaurants and college cafeterias. The dataset excludes establishments that have gone out of business. Although the dataset contains several columns, only a subset of them are used in this solution. For a detailed description of the dataset, visit the site.

### Understand the solution

This solution is made up of different .NET Core applications:

- RestaurantInspectionsETL: .NET Core Console application that takes raw data and uses Spark.NET to clean and transform the data into a format that is easier to use as input for training and making predictions with a machine learning model built with ML.NET.
- RestaurantInspectionsML: .NET Core Class Library that defines the input and output schema of the ML.NET machine learning model. Additionally, this is where the trained model is saved to.
- RestaurantInspectionsTraining: .NET Core Console application that uses the graded data generated by the RestaurantInspectionsETL application to train a multiclass classification machine learning model using ML.NET's AutoML.
- RestaurantInspectionsEnrichment: .NET Core Console application that uses the ungraded data generated by the RestaurantInspectionsETL application as input for the trained ML.NET machine learning model to predict what grade an establishment is most likely to receive based on the violations found during inspection.

## Get the code

```bash
git clone https://github.com/lqdev/RestaurantInspectionsSparkMLNET.git
```

## Update solution locations

Before building the code, update the location of the solution in the *RestaurantInspectionsTraining* and *RestaurantInspectionsEnrichment*.

Replace the value of `solutionDirectory` with the path of where your solution is saved.

Original:

```csharp
string solutionDirectory = "/home/lqdev/Development/RestaurantInspectionsSparkMLNET";
```

New:

```csharp
string solutionDirectory = "<YOUR-SOLUTION-PATH>/RestaurantInspectionsSparkMLNET";
```

## Build

### RestaurantInspectionsETL

```bash
dotnet publish -f netcoreapp2.1 -r ubuntu.18.04-x64
```

### RestaurantInspectionsTraining

```bash
dotnet build
```

### RestaurantInspectionsEnrichment

```bash
dotnet publish -f netcoreapp2.1 -r ubuntu.18.04-x64
```

## Run

### RestaurantInspectionsETL

From the project directory run the application with spark-submit.

```bash
spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin/Debug/netcoreapp2.1/ubuntu.18.04-x64/publish/microsoft-spark-2.4.x-0.4.0.jar dotnet bin/Debug/netcoreapp2.1/ubuntu.18.04-x64/publish/RestaurantInspectionsETL.dll
```

### RestaurantInspectionsTraining

```bash
dotnet run
```

### RestaurantInspectionsEnrichment

Navigate to the *publish* directory. In this case, it's *bin/Debug/netcoreapp2.1/ubuntu.18.04-x64/publish*.

From the *publish* directory, run the application with `spark-submit`.

```bash
spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local microsoft-spark-2.4.x-0.4.0.jar dotnet RestaurantInspectionsEnrichment.dll
```

RestaurantInspectionsSparkMLNET

RestaurantInspectionsSparkMLNET Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

RestaurantInspectionsSparkMLNET

RestaurantInspectionsSparkMLNET Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README