bigdata-playground
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
File Explorer
Download Latest Version (.zip)- airports.json
- airports.parquet
- build.properties
- plugins.sbt
- app.conf
- log4j.properties
- Airport.scala
- BatchConfig.scala
- BatchDbConfig.scala
- Config.scala
- ConfigurationFactory.scala
- Main.scala
- build.sbt
- Dockerfile-dev
- airports.json
- airports.parquet
- Application.java
- MongoReducer.java
- ParquetMapper.java
- airport.schema.avsc
- app.conf
- log4j.properties
- Dockerfile-dev
- pom.xml
- airports.json
- airports.parquet
- build.properties
- plugins.sbt
- Application.java
- app.conf
- log4j.properties
- BatchConfig.scala
- BatchDbConfig.scala
- Config.scala
- ConfigurationFactory.scala
- Main.scala
- build.sbt
- Dockerfile
- Dockerfile-dev
- pom.xml
- batch-flink.yml
- batch-hadoop.yml
- batch-spark.yml
- ml-spark.yml
- streaming-flink.yml
- streaming-spark.yml
- streaming-storm.yml
- webapp.yml
- docker-compose.yml
- flink.yml
- hadoop-hbase.yml
- kafka.yml
- kylo.yml
- mongo.yml
- webapp.yml
- zookeeper.yml
- Dockerfile-model-dev
- Dockerfile-stream-dev
- flight-info.schema.avsc
- model.py
- stream.py
- build.properties
- plugins.sbt
- app.conf
- flight-info.schema.avsc
- hbase-site.xml
- log4j.properties
- CabinClass.scala
- FlightInfoBean.scala
- TripType.scala
- Config.scala
- ProducerConfig.scala
- StreamingConfig.scala
- StreamingDbConfig.scala
- ConfigurationFactory.scala
- FlightInfoAvroSchema.scala
- FlightInfoHBaseOutputFormat.scala
- JsonUtils.scala
- Main.scala
- build.sbt
- Dockerfile-dev
- build.properties
- plugins.sbt
- Application.java
- FlightInfoBean.java
- app.conf
- flight-info.schema.avsc
- log4j.properties
- CabinClass.scala
- FlightInfoBean.scala
- TripType.scala
- Config.scala
- ProducerConfig.scala
- StreamingConfig.scala
- StreamingDbConfig.scala
- ConfigurationFactory.scala
- JsonUtils.scala
- Main.scala
- build.sbt
- Dockerfile
- Dockerfile-dev
- hbase-site.xml
- pom.xml
- KafkaFlightInfoBolt.java
- KafkaSpoutTopology.java
- Application.java
- FlightInfoBean.java
- app.conf
- flight-info.schema.avsc
- hbase-site.xml
- log4j.properties
- Dockerfile-dev
- pom.xml
- airport.dto.ts
- index.ts
- flight.mutations.ts
- airports.queries.ts
- airports.service.ts
- flight.service.ts
- index.ts
- tweet.service.ts
- tweet.subscription.ts
- airport-dto-mapper.ts
- airport-mapper.ts
- index.ts
- search-flight.component.html
- search-flight.component.scss
- search-flight.component.ts
- dashboard.component.html
- dashboard.component.scss
- dashboard.component.ts
- dashboard.module.ts
- dashboard.routes.ts
- airport.model.ts
- flight-info.model.ts
- index.ts
- material.module.ts
- app.component.html
- app.component.scss
- app.component.ts
- app.module.ts
- app.routes.ts
- .gitkeep
- environment.prod.ts
- environment.ts
- browserslist
- favicon.ico
- index.html
- karma.conf.js
- main.ts
- polyfills.ts
- styles.scss
- test.ts
- tsconfig.app.json
- tsconfig.spec.json
- tslint.json
- .editorconfig
- angular.json
- package.json
- protractor.conf.js
- README.md
- tsconfig.json
- tslint.json
- yarn.lock
- database.ts
- airport.ts
- flight-info.ts
- tweet.ts
- index.ts
- flight-info-avro.mapper.ts
- flight-info-avro.ts
- flight-info.producer.ts
- index.ts
- tweet.consumer.ts
- resolvers.ts
- types.ts
- index.ts
- index.ts
- package.json
- server.js
- tsconfig.json
- tslint.json
- yarn.lock
- Dockerfile
- Dockerfile-dev
- package.json
- yarn.lock
- .gitignore
- LICENSE
- README.md
# Installation Guide
1. Get the code
git clone https://github.com/Chabane/bigdata-playground
Downloads the entire project code from GitHub to your computer.
cd bigdata-playground
Moves into the project folder you just downloaded.
2. Docker
Easy RecommendedPrerequisites
- Git Needed to download the project code from GitHub.
- Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
docker network create vnet
Type this command into your terminal and run it.
cd docker
Moves into the project folder you just downloaded.
docker-compose -f mongo.yml -f zookeeper.yml -f kafka.yml -f hadoop-hbase.yml -f flink.yml up -d
Runs the command against the services defined in the compose file.
docker-compose -f dev/webapp.yml up -d
Runs the command against the services defined in the compose file.
docker-compose -f dev/batch-spark.yml up -d
Runs the command against the services defined in the compose file.
Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.
Pulled directly from this repo's README.
3. Maven (Java)
MediumPrerequisites
- Git Needed to download the project code from GitHub.
- JDK (Java) Required to build and run Java projects.
- Maven The build tool used for the mvn command.
cd batch/hadoop && mvn clean package && cd ../..
Moves into the project folder you just downloaded.
cd streaming/storm && mvn clean package && cd ../..
Moves into the project folder you just downloaded.
A BUILD SUCCESS message means it worked. The output is created under target/.
Pulled directly from this repo's README.
4. Node.js
EasyPrerequisites
npm install yarn -g
Downloads and installs the libraries listed in package.json.
cd webapp && yarn && cd client && yarn && cd ../server && yarn && cd ../ && npm run build:dev && cd ../
Moves into the project folder you just downloaded.
After running the command, open the address shown in the terminal (usually something like http://localhost:3000) in your browser.
Pulled directly from this repo's README.
// repository documentation
Was this content helpful?
(0 ratings)