PipeFlow - GitLite

KO | EN

gitlite — search

#javascript #python #hacktoberfest #android #machine-learning #rust #deep-learning #linux #docker #nodejs #java #windows #awesome-list #security

PipeFlow

(★ 136)

High-performance ETL pipeline library for .NET. Process CSV, JSON, Excel, and SQL data with minimal memory usage through streaming operations.

File Explorer

Download Latest Version (.zip)

- dotnet-tools.json
- - build.yml
  - dependency-check.yml
  - release.yml
- - PipeFlow.Sandbox.csproj
  - Program.cs
- - DataRow.cs
  - IOrderedPipeline.cs
  - IOrderedQueryablePipeline.cs
  - IPipeline.cs
  - IQueryablePipeline.cs
  - PipeFlow.csproj
  - PipeFlowOptions.cs
  - PipelineContext.cs
- - GlobalUsings.cs
  - PipeFlow.Tests.csproj
  - xunit.runner.json
.editorconfig
.gitattributes
.gitignore
CHANGELOG.md
CONTRIBUTING.md
Directory.Build.props
Directory.Packages.props
global.json
icon.png
LICENSE
PipeFlow.slnx
README.md
Version.props

# Installation Guide

1. Get the code

git clone https://github.com/Nonanti/PipeFlow

Downloads the entire project code from GitHub to your computer.

cd PipeFlow

Moves into the project folder you just downloaded.

2. .NET

Medium Recommended

Prerequisites

Git Needed to download the project code from GitHub.
.NET SDK The SDK required for the dotnet command.

dotnet add package PipeFlowCore

Type this command into your terminal and run it.

dotnet build

Type this command into your terminal and run it.

dotnet test

Type this command into your terminal and run it.

dotnet run --project samples/PipeFlow.Sandbox -c Release

Builds the project and runs it immediately.

✅ After dotnet run, check the message or address shown in the terminal.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# PipeFlow

[![NuGet](https://img.shields.io/nuget/v/PipeFlowCore.svg)](https://www.nuget.org/packages/PipeFlowCore)
[![Build](https://github.com/Nonanti/PipeFlow/actions/workflows/build.yml/badge.svg)](https://github.com/Nonanti/PipeFlow/actions)
[![License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://raw.githubusercontent.com/Nonanti/PipeFlow/main/LICENSE)
[![.NET](https://img.shields.io/badge/.NET-8%20%7C%2010-512BD4)](https://dotnet.microsoft.com/download)

A .NET ETL pipeline library. Read, transform, and write data with a fluent, async-first API.

> **v3 is under active development on this branch.** The stable v2.1.0 package is
> still the one on NuGet as `PipeFlowCore`. v3 will ship split across
> `PipeFlow` + `PipeFlow.SqlServer` + `PipeFlow.PostgreSql` + `PipeFlow.MongoDb` +
> `PipeFlow.Excel` + `PipeFlow.EntityFrameworkCore` + `PipeFlow.Aws` +
> `PipeFlow.Azure` + `PipeFlow.GoogleCloud`. Until then use v2 or build from source.

## Install (v2, current stable)

```bash
dotnet add package PipeFlowCore
```

The v2 source lives at tag [`v2.1.0-archived`](https://github.com/Nonanti/PipeFlow/tree/v2.1.0-archived).

## Quick start (v3, from source)

```csharp
using PipeFlow;

// DataRow is a case-insensitive, structurally-equal row type
var row = new DataRow
{
    ["Id"] = 1,
    ["Email"] = "alice@example.com",
    ["Age"] = "30",
};

// InvariantCulture type conversion with Nullable<T> support
int id = row.GetValue<int>("Id");
int? age = row.GetValue<int?>("Age");
string email = row.GetValue<string>("EMAIL")!; // key lookup is case-insensitive
```

The pipeline, source, and sink abstractions are defined; concrete CSV / JSON / HTTP /
database sources land in Wave 0.B and beyond. See `samples/PipeFlow.Sandbox/Program.cs`
for every public API exercised end-to-end.

## Build and test locally

Requires the .NET 10 SDK (also builds for net8.0).

```bash
dotnet build
dotnet test
dotnet run --project samples/PipeFlow.Sandbox -c Release
```

## Status

- Core abstractions, `DataRow`, exception hierarchy: done (Wave 0.A).
- `Pipeline<T>`, `ParallelPipeline<T>`, builder, DI, options: next (Wave 0.B).
- Source/sink concrete implementations (CSV, JSON, HTTP, DB, cloud): Wave 0.C onward.
- NuGet release as split packages: v3.0.0 GA.

## Contributing

PRs welcome. See [CONTRIBUTING.md](CONTRIBUTING.md). Run `dotnet format` before committing.

## License

MIT. See [LICENSE](LICENSE).

## Author

Berkant - [Nonanti](https://github.com/Nonanti)