public-datasets-pipelines
Cloud-native, data onboarding architecture for Google Cloud Datasets
파일 탐색기
최종 버전 다운로드 (.zip)- python-lint.yaml
- terraform-lint.yaml
- unit-tests.yaml
- yaml-lint.yaml
- CONTRIBUTING.md
- ISSUE_TEMPLATE.md
- PULL_REQUEST_TEMPLATE.md
- release-please.yml
- ahr_pipeline.tf
- america_health_rankings_dataset.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- ahr_dag.py
- pipeline.yaml
- dataset.yaml
- austin_dataset.tf
- austin_pipeline.tf
- provider.tf
- variables.tf
- austin_311_service_requests_schema.json
- austin_bikeshare_stations_schema.json
- austin_bikeshare_trips_schema.json
- austin_crime_schema.json
- austin_waste_and_diversion_schema.json
- csv_transform.py
- Dockerfile
- requirements.txt
- austin_dag.py
- pipeline.yaml
- dataset.yaml
- austin_bikeshare_dataset.tf
- bikeshare_stations_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- bikeshare_stations_dag.py
- pipeline.yaml
- dataset.yaml
- austin_crime_dataset.tf
- crime_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- crime_dag.py
- pipeline.yaml
- dataset.yaml
- austin_waste_dataset.tf
- provider.tf
- variables.tf
- waste_and_diversion_pipeline.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- waste_and_diversion_dag.py
- dataset.yaml
- bls_dataset.tf
- c_cpi_u_pipeline.tf
- cpi_u_pipeline.tf
- cpsaat18_pipeline.tf
- employment_hours_earnings_pipeline.tf
- employment_hours_earnings_series_pipeline.tf
- provider.tf
- unemployment_cps_pipeline.tf
- unemployment_cps_series_pipeline.tf
- variables.tf
- wm_pipeline.tf
- wm_series_pipeline.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- c_cpi_u_dag.py
- pipeline.yaml
- cpi_u_dag.py
- pipeline.yaml
- cpsaat18_dag.py
- pipeline.yaml
- employment_hours_earnings_dag.py
- pipeline.yaml
- employment_hours_earnings_series_dag.py
- pipeline.yaml
- pipeline.yaml
- unemployment_cps_dag.py
- pipeline.yaml
- unemployment_cps_series_dag.py
- pipeline.yaml
- wm_dag.py
- pipeline.yaml
- wm_series_dag.py
- dataset.yaml
- provider.tf
- variables.tf
- copy_gcs_bucket_dag.py
- pipeline.yaml
- dataset.yaml
- cdc_chronic_disease_indicators_dataset.tf
- chronic_disease_indicators_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- chronic_disease_indicators_dag.py
- pipeline.yaml
- dataset.yaml
- cdc_places_dataset.tf
- local_data_for_better_health_county_data_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- local_data_for_better_health_county_data_dag.py
- pipeline.yaml
- dataset.yaml
- provider.tf
- variables.tf
- celeba_dag.py
- pipeline.yaml
- dataset.yaml
- cbsa_2019_1yr_pipeline.tf
- cbsa_2019_5yr_pipeline.tf
- census_bureau_acs_dataset.tf
- census_bureau_acs_pipeline.tf
- censustract_2019_5yr_pipeline.tf
- congressionaldistrict_2019_1yr_pipeline.tf
- congressionaldistrict_2019_5yr_pipeline.tf
- county_2019_1yr_pipeline.tf
- county_2019_5yr_pipeline.tf
- place_2019_1yr_pipeline.tf
- place_2019_5yr_pipeline.tf
- provider.tf
- puma_2019_1yr_pipeline.tf
- puma_2019_5yr_pipeline.tf
- schooldistrictelementary_2019_1yr_pipeline.tf
- schooldistrictelementary_2019_5yr_pipeline.tf
- schooldistrictsecondary_2019_1yr_pipeline.tf
- schooldistrictsecondary_2019_5yr_pipeline.tf
- schooldistrictunified_2019_1yr_pipeline.tf
- schooldistrictunified_2019_5yr_pipeline.tf
- state_2019_1yr_pipeline.tf
- state_2019_5yr_pipeline.tf
- variables.tf
- zcta_2019_5yr_pipeline.tf
- csv_transform.py
- Dockerfile
- group_ids.json
- requirements.txt
- state_codes.json
- schema.json
- census_bureau_acs_dag.py
- pipeline.yaml
- dataset.yaml
- age_specific_fertility_rates_pipeline.tf
- census_bureau_international_dataset.tf
- country_names_area_pipeline.tf
- midyear_population_age_sex_pipeline.tf
- midyear_population_agespecific_pipeline.tf
- midyear_population_pipeline.tf
- mortality_life_expectancy_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- csv_transform.py
- Dockerfile
- requirements.txt
- csv_transform.py
- Dockerfile
- requirements.txt
- csv_transform.py
- Dockerfile
- requirements.txt
- csv_transform.py
- Dockerfile
- requirements.txt
- csv_transform.py
- Dockerfile
- requirements.txt
- csv_transform.py
- Dockerfile
- requirements.txt
- age_specific_fertility_rates_dag.py
- pipeline.yaml
- country_names_area_dag.py
- pipeline.yaml
- midyear_population_dag.py
- pipeline.yaml
- midyear_population_age_sex_dag.py
- pipeline.yaml
- midyear_population_agespecific_dag.py
- pipeline.yaml
- mortality_life_expectancy_dag.py
- pipeline.yaml
- dataset.yaml
- census_opportunity_atlas_dataset.tf
- census_opportunity_atlas_pipeline.tf
- provider.tf
- tract_covariates_pipeline.tf
- tract_outcomes_pipeline.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- tract_covariates_schema.json
- tract_outcomes_schema.json
- census_opportunity_atlas_dag.py
- pipeline.yaml
- dataset.yaml
- cfe_calculator_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- copy_cfe_data_dag.py
- pipeline.yaml
- dataset.yaml
- cfpb_complaints_dataset.tf
- complaint_database_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- complaint_database_dag.py
- pipeline.yaml
- dataset.yaml
- chicago_crime_dataset.tf
- crime_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- crime_dag.py
- pipeline.yaml
- dataset.yaml
- chicago_taxi_trips_dataset.tf
- provider.tf
- taxi_trips_pipeline.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- taxi_trips_dag.py
- dataset.yaml
- chdb_data_city_all_pipeline.tf
- chdb_data_tract_all_pipeline.tf
- city_health_dashboard_dataset.tf
- city_health_dashboard_full_load_pipeline.tf
- city_health_dashboard_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- chdb_data_city_schema.json
- chdb_data_tract_schema.json
- city_health_dashboard_dag.py
- pipeline.yaml
- chdb_data_city_schema.json
- chdb_data_tract_schema.json
- city_health_dashboard_full_load_dag.py
- pipeline.yaml
- dataset.yaml
- csv_transform.py
- Dockerfile
- requirements.txt
- clemson_dice_traffic_vision_dag.py
- pipeline.yaml
- cloud_datasets_dataset.tf
- pdp_extract_tabular_metadata_pipeline.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- pdp_extract_tabular_metadata_dag.py
- pipeline.yaml
- dataset.yaml
- cloud_storage_geo_index_dataset.tf
- landsat_index_pipeline.tf
- provider.tf
- sentinel_2_index_pipeline.tf
- variables.tf
- cloud_storage_geo_index_landsat_index_schema.json
- cloud_storage_geo_index_sentinel_2_schema.json
- csv_transform.py
- Dockerfile
- requirements.txt
- cloud_storage_geo_index_dag.py
- pipeline.yaml
- dataset.yaml
- cms_medicare_dataset.tf
- hospital_general_info_pipeline.tf
- inpatient_charges_pipeline.tf
- outpatient_charges_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- hospital_general_info_dag.py
- pipeline.yaml
- inpatient_charges_dag.py
- pipeline.yaml
- outpatient_charges_dag.py
- pipeline.yaml
- dataset.yaml
- covid19_cds_eu_dataset.tf
- global_cases_by_province_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- global_cases_by_province_dag.py
- pipeline.yaml
- dataset.yaml
- covid19_genome_sequence_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- covid19_genome_sequence_dag.py
- pipeline.yaml
- dataset.yaml
- covid19_google_mobility_dataset.tf
- mobility_report_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- covid19_google_mobility_mobility_report_schema.json
- mobility_report_dag.py
- pipeline.yaml
- dataset.yaml
- covid19_govt_response_dataset.tf
- oxford_policy_tracker_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- oxford_policy_tracker_dag.py
- pipeline.yaml
- dataset.yaml
- covid19_italy_dataset.tf
- data_by_province_pipeline.tf
- data_by_region_pipeline.tf
- national_trends_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- data_by_province_dag.py
- pipeline.yaml
- data_by_region_dag.py
- pipeline.yaml
- national_trends_dag.py
- pipeline.yaml
- dataset.yaml
- covid19_jhu_csse_dataset.tf
- provider.tf
- variables.tf
- covid19_jhu_csse_schema.json
- csv_transform.py
- Dockerfile
- requirements.txt
- Dockerfile
- load_bq_eu.py
- requirements.txt
- covid19_jhu_csse_dag.py
- pipeline.yaml
- dataset.yaml
- covid19_nyt_dataset.tf
- load_from_github_pipeline.tf
- provider.tf
- variables.tf
- load_from_github_dag.py
- pipeline.yaml
- dataset.yaml
- covid19_symptom_search_dataset.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- Dockerfile
- requirements.txt
- script.py
- data_covid19_symptom_search_symptom_search_country_daily_schema.json
- data_covid19_symptom_search_symptom_search_country_weekly_schema.json
- data_covid19_symptom_search_symptom_search_sub_region_1_daily_schema.json
- data_covid19_symptom_search_symptom_search_sub_region_1_weekly_schema.json
- data_covid19_symptom_search_symptom_search_sub_region_2_daily_schema.json
- data_covid19_symptom_search_symptom_search_sub_region_2_weekly_schema.json
- covid19_symptom_search_dag.py
- pipeline.yaml
- dataset.yaml
- covid19_vaccination_access_dataset.tf
- provider.tf
- vaccination_access_dataset.tf
- vaccination_access_to_bq_pipeline.tf
- variables.tf
- pipeline.yaml
- vaccination_access_to_bq_dag.py
- dataset.yaml
- covid19_vaccination_search_insights_dataset.tf
- covid19_vaccination_search_insights_pipeline.tf
- provider.tf
- variables.tf
- covid19_vaccination_search_insights_dag.py
- pipeline.yaml
- dataset.yaml
- decennial_census_quick_facts_dataset.tf
- provider.tf
- united_states_census_bureau_pipeline.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- united_states_census_bureau_dag.py
- dataset.yaml
- deepmind_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- alphafold_dag.py
- pipeline.yaml
- alphafold_v4_dag.py
- pipeline.yaml
- dataset.yaml
- chembl_30_pipeline.tf
- ebi_chembl_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- csv_transform.py
- Dockerfile
- requirements.txt
- chembl_30_dag.py
- pipeline.yaml
- ebi_chembl_old_data_dag.py
- pipeline.yaml
- dataset.yaml
- artifact.yaml
- co_levels.ipynb
- co_levels_test.py
- epa_historical_air_quality_dataset.tf
- epa_historical_air_quality_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- epa_annual_summaries_schema.json
- epa_co_daily_summary_schema.json
- epa_co_hourly_summary_schema.json
- epa_hap_daily_summary_schema.json
- epa_hap_hourly_summary_schema.json
- epa_lead_daily_summary_schema.json
- epa_no2_daily_summary_schema.json
- epa_no2_hourly_summary_schema.json
- epa_nonoxnoy_daily_summary_schema.json
- epa_nonoxnoy_hourly_summary_schema.json
- epa_ozone_daily_summary_schema.json
- epa_ozone_hourly_summary_schema.json
- epa_pm10_daily_summary_schema.json
- epa_pm10_hourly_summary_schema.json
- epa_pm25_frm_daily_summary_schema.json
- epa_pm25_frm_hourly_summary_schema.json
- epa_pm25_nonfrm_daily_summary_schema.json
- epa_pm25_nonfrm_hourly_summary_schema.json
- epa_pm25_speciation_daily_summary_schema.json
- epa_pm25_speciation_hourly_summary_schema.json
- epa_pressure_daily_summary_schema.json
- epa_pressure_hourly_summary_schema.json
- epa_rh_and_dp_daily_summary_schema.json
- epa_rh_and_dp_hourly_summary_schema.json
- epa_so2_daily_summary_schema.json
- epa_so2_hourly_summary_schema.json
- epa_temperature_daily_summary_schema.json
- epa_temperature_hourly_summary_schema.json
- epa_voc_daily_summary_schema.json
- epa_voc_hourly_summary_schema.json
- epa_wind_daily_summary_schema.json
- epa_wind_hourly_summary_schema.json
- requirements.txt
- epa_historical_air_quality_dag.py
- pipeline.yaml
- epa_historical_air_quality_full_load_dag.py
- pipeline.yaml
- dataset.yaml
- eumetsat_dataset.tf
- provider.tf
- variables.tf
- pipeline.yaml
- solar_forecasting_dag.py
- dataset.yaml
- fashion_mnist_dataset.tf
- provider.tf
- variables.tf
- fashion_mnist_dag.py
- pipeline.yaml
- dataset.yaml
- drug_enforcement_pipeline.tf
- fda_drug_dataset.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- drug_enforcement_dag.py
- pipeline.yaml
- dataset.yaml
- fda_food_dataset.tf
- food_enforcement_pipeline.tf
- food_events_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- food_enforcement_dag.py
- pipeline.yaml
- food_events_dag.py
- pipeline.yaml
- dataset.yaml
- fdic_dataset.tf
- institutions_pipeline.tf
- locations_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- institutions_dag.py
- pipeline.yaml
- locations_dag.py
- pipeline.yaml
- dataset.yaml
- candidate_2016_pipeline.tf
- candidate_2018_pipeline.tf
- candidate_2020_pipeline.tf
- candidate_committee_2016_pipeline.tf
- candidate_committee_2018_pipeline.tf
- candidate_committee_2020_pipeline.tf
- committee_2016_pipeline.tf
- committee_2018_pipeline.tf
- committee_2020_pipeline.tf
- committee_contributions_2016_pipeline.tf
- committee_contributions_2018_pipeline.tf
- committee_contributions_2020_pipeline.tf
- fec_dataset.tf
- individuals_2016_pipeline.tf
- individuals_ingest_2020_pipeline.tf
- opex_2016_pipeline.tf
- opex_2018_pipeline.tf
- opex_2020_pipeline.tf
- other_committee_tx_2016_pipeline.tf
- other_committee_tx_2018_pipeline.tf
- other_committee_tx_2020_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- candidate_2016_dag.py
- pipeline.yaml
- candidate_2018_dag.py
- pipeline.yaml
- candidate_2020_dag.py
- pipeline.yaml
- candidate_committee_2016_dag.py
- pipeline.yaml
- candidate_committee_2018_dag.py
- pipeline.yaml
- candidate_committee_2020_dag.py
- pipeline.yaml
- committee_2016_dag.py
- pipeline.yaml
- committee_2018_dag.py
- pipeline.yaml
- committee_2020_dag.py
- pipeline.yaml
- committee_contributions_2016_dag.py
- pipeline.yaml
- committee_contributions_2018_dag.py
- pipeline.yaml
- committee_contributions_2020_dag.py
- pipeline.yaml
- individuals_2016_dag.py
- pipeline.yaml
- individuals_ingest_2020_dag.py
- pipeline.yaml
- opex_2016_dag.py
- pipeline.yaml
- opex_2018_dag.py
- pipeline.yaml
- opex_2020_dag.py
- pipeline.yaml
- other_committee_tx_2016_dag.py
- pipeline.yaml
- other_committee_tx_2018_dag.py
- pipeline.yaml
- other_committee_tx_2020_dag.py
- pipeline.yaml
- dataset.yaml
- artifact.yaml
- find_single_region_species.sql
- artifact.yaml
- australia.ipynb
- australia_test.py
- gbif_dataset.tf
- gcs_to_bq_pipeline.tf
- provider.tf
- variables.tf
- gcs_to_bq_dag.py
- pipeline.yaml
- dataset.yaml
- geos_fp_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- copy_files_rolling_basis_dag.py
- pipeline.yaml
- dataset.yaml
- provider.tf
- variables.tf
- copy_gcs_bucket_dag.py
- pipeline.yaml
- dataset.yaml
- datacenter_cfe_pipeline.tf
- google_cfe_dataset.tf
- provider.tf
- variables.tf
- datacenter_cfe_dag.py
- pipeline.yaml
- dataset.yaml
- google_cloud_release_notes_dataset.tf
- provider.tf
- release_notes_pipeline.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- pipeline.yaml
- release_notes_dag.py
- dataset.yaml
- diversity_annual_report_pipeline.tf
- google_dei_dataset.tf
- google_diversity_dataset.tf
- provider.tf
- variables.tf
- diversity_annual_report_dag.py
- pipeline.yaml
- dataset.yaml
- google_political_ads_dataset.tf
- process_csvs_and_load_to_bq_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- process_csvs_and_load_to_bq_dag.py
- dataset.yaml
- google_trends_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- copy_trends_data_dag.py
- pipeline.yaml
- dataset.yaml
- full_pipeline.tf
- hacker_news_dataset.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- schema.json
- full_dag.py
- pipeline.yaml
- dataset.yaml
- human_variant_annotation_dataset.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- clinvar_dag.py
- pipeline.yaml
- db_snp_dag.py
- pipeline.yaml
- dataset.yaml
- idc_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- Dockerfile
- requirements.txt
- script.py
- copy_tcia_data_dag.py
- pipeline.yaml
- dataset.yaml
- imdb_dataset.tf
- interfaces_pipeline.tf
- provider.tf
- reviews_pipeline.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- interfaces_dag.py
- pipeline.yaml
- pipeline.yaml
- reviews_dag.py
- dataset.yaml
- artifact.yaml
- liquor_sales_predictions.ipynb
- sales_prediction_test.py
- iowa_liquor_sales_dataset.tf
- provider.tf
- sales_pipeline.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- sales_dag.py
- dataset.yaml
- 2020_sales_train_pipeline.tf
- 2021_sales_predict_pipeline.tf
- iowa_liquor_sales_dataset.tf
- iowa_liquor_sales_forecasting_dataset.tf
- provider.tf
- variables.tf
- 2020_sales_train_dag.py
- pipeline.yaml
- 2021_sales_predict_dag.py
- pipeline.yaml
- dataset.yaml
- irs_990_2014_pipeline.tf
- irs_990_2015_pipeline.tf
- irs_990_2016_pipeline.tf
- irs_990_2017_pipeline.tf
- irs_990_dataset.tf
- irs_990_ein_pipeline.tf
- irs_990_ez_2014_pipeline.tf
- irs_990_ez_2015_pipeline.tf
- irs_990_ez_2016_pipeline.tf
- irs_990_ez_2017_pipeline.tf
- irs_990_pf_2014_pipeline.tf
- irs_990_pf_2015_pipeline.tf
- irs_990_pf_2016_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- irs_990_2014_dag.py
- pipeline.yaml
- irs_990_2015_dag.py
- pipeline.yaml
- irs_990_2016_dag.py
- pipeline.yaml
- irs_990_2017_dag.py
- pipeline.yaml
- irs_990_ein_dag.py
- pipeline.yaml
- irs_990_ez_2014_dag.py
- pipeline.yaml
- irs_990_ez_2015_dag.py
- pipeline.yaml
- irs_990_ez_2016_dag.py
- pipeline.yaml
- irs_990_ez_2017_dag.py
- pipeline.yaml
- irs_990_pf_2014_dag.py
- pipeline.yaml
- irs_990_pf_2015_dag.py
- pipeline.yaml
- irs_990_pf_2016_dag.py
- pipeline.yaml
- dataset.yaml
- dependencies_pipeline.tf
- libraries_io_dataset.tf
- projects_pipeline.tf
- provider.tf
- repositories_pipeline.tf
- repository_dependencies_pipeline.tf
- tags_pipeline.tf
- variables.tf
- versions_pipeline.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- dependencies_dag.py
- pipeline.yaml
- pipeline.yaml
- projects_dag.py
- pipeline.yaml
- repositories_dag.py
- pipeline.yaml
- repository_dependencies_dag.py
- pipeline.yaml
- tags_dag.py
- pipeline.yaml
- versions_dag.py
- dataset.yaml
- cycle_stations_pipeline.tf
- london_bicycles_dataset.tf
- provider.tf
- variables.tf
- csv_transform.py
- cycle_hire_schema.json
- Dockerfile
- requirements.txt
- london_bicycles_dag.py
- pipeline.yaml
- dataset.yaml
- cell_by_gene_pipeline.tf
- cell_metadata_pipeline.tf
- detected_transcripts_pipeline.tf
- merfish_dataset.tf
- provider.tf
- variables.tf
- cell_by_gene_dag.py
- pipeline.yaml
- cell_metadata_dag.py
- pipeline.yaml
- detected_transcripts_dag.py
- pipeline.yaml
- dataset.yaml
- mimic_iii_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- mimic_iii_dag.py
- pipeline.yaml
- dataset.yaml
- iris_pipeline.tf
- ml_datasets_dataset.tf
- penguins_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- iris_dag.py
- pipeline.yaml
- penguins_dag.py
- pipeline.yaml
- dataset.yaml
- mlcommons_dataset.tf
- provider.tf
- variables.tf
- mswc_dag.py
- pipeline.yaml
- dataset.yaml
- mnist_dataset.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- mnist_dag.py
- pipeline.yaml
- dataset.yaml
- multilingual_spoken_words_corpus_dataset.tf
- multilingual_spoken_words_corpus_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- multilingual_spoken_words_corpus_dag.py
- pipeline.yaml
- dataset.yaml
- nasa_wildfire_dataset.tf
- past_week_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- past_week_dag.py
- pipeline.yaml
- dataset.yaml
- ipeds_pipeline.tf
- nces_dataset.tf
- provider.tf
- variables.tf
- ipeds_dag.py
- pipeline.yaml
- dataset.yaml
- new_york_dataset.tf
- new_york_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- citibike_stations_schema.json
- ny_311_service_requests_schema.json
- nypd_mv_collisions_schema.json
- tree_census_1995_schema.json
- new_york_dag.py
- pipeline.yaml
- dataset.yaml
- csv_transform.py
- Dockerfile
- green_trips_schema.json
- requirements.txt
- yellow_trips_schema.json
- new_york_taxi_trips_dataset.tf
- new_york_taxi_trips_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- green_trips_schema.json
- requirements.txt
- yellow_trips_schema.json
- new_york_taxi_trips_dag.py
- pipeline.yaml
- dataset.yaml
- provider.tf
- new_york_trees_dataset.tf
- provider.tf
- tree_census_2005_pipeline.tf
- tree_census_2015_pipeline.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- tree_census_2005_dag.py
- pipeline.yaml
- tree_census_2015_dag.py
- dataset.yaml
- hatecrimes_pipeline.tf
- news_hatecrimes_dataset.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- hatecrimes_dag.py
- pipeline.yaml
- dataset.yaml
- nhtsa_traffic_fatalities_dataset.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- nhtsa_traffic_fatalities_accident_2015_schema.json
- nhtsa_traffic_fatalities_accident_2016_2019_schema.json
- nhtsa_traffic_fatalities_accident_2020_schema.json
- nhtsa_traffic_fatalities_cevent_2015_2020_schema.json
- nhtsa_traffic_fatalities_damage_2015_2020_schema.json
- nhtsa_traffic_fatalities_distract_2015_2020_schema.json
- nhtsa_traffic_fatalities_drimpair_2015_2020_schema.json
- nhtsa_traffic_fatalities_factor_2015_2020_schema.json
- nhtsa_traffic_fatalities_maneuver_2015_2020_schema.json
- nhtsa_traffic_fatalities_nmcrash_2015_2020_schema.json
- nhtsa_traffic_fatalities_nmimpair_2015_2020_schema.json
- nhtsa_traffic_fatalities_nmprior_2015_2020_schema.json
- nhtsa_traffic_fatalities_parkwork_2015_schema.json
- nhtsa_traffic_fatalities_parkwork_2016_2017_schema.json
- nhtsa_traffic_fatalities_parkwork_2018_schema.json
- nhtsa_traffic_fatalities_parkwork_2019_schema.json
- nhtsa_traffic_fatalities_parkwork_2020_schema.json
- nhtsa_traffic_fatalities_pbtype_2015_2020_schema.json
- nhtsa_traffic_fatalities_person_2015_2017_schema.json
- nhtsa_traffic_fatalities_person_2018_schema.json
- nhtsa_traffic_fatalities_person_2019_schema.json
- nhtsa_traffic_fatalities_person_2020_schema.json
- nhtsa_traffic_fatalities_safetyeq_2015_2016_schema.json
- nhtsa_traffic_fatalities_safetyeq_2017_2020_schema.json
- nhtsa_traffic_fatalities_vehicle_2015_schema.json
- nhtsa_traffic_fatalities_vehicle_2016_2017_schema.json
- nhtsa_traffic_fatalities_vehicle_2018_2019_schema.json
- nhtsa_traffic_fatalities_vehicle_2020_schema.json
- nhtsa_traffic_fatalities_vevent_2015_2020_schema.json
- nhtsa_traffic_fatalities_vindecode_2015_schema.json
- nhtsa_traffic_fatalities_violatn_2015_2020_schema.json
- nhtsa_traffic_fatalities_vision_2015_2020_schema.json
- nhtsa_traffic_fatalities_vsoe_2015_2020_schema.json
- nhtsa_traffic_fatalities_dag.py
- pipeline.yaml
- dataset.yaml
- nih_gudid_dataset.tf
- nih_gudid_pipeline.tf
- provider.tf
- variables.tf
- nih_gudid_dag.py
- pipeline.yaml
- dataset.yaml
- csv_transform.py
- Dockerfile
- requirements.txt
- rxnatomarchive_schema.json
- rxnconso_schema.json
- rxncui_schema.json
- rxncuichange_schema.json
- rxndoc_schema.json
- rxnrel_schema.json
- rxnsab_schema.json
- rxnsat_schema.json
- rxnsty_schema.json
- nlm_rxnorm_dag.py
- pipeline.yaml
- gsod_stations_pipeline.tf
- noaa_dataset.tf
- noaa_pipeline.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- ghcnd_by_year_schema.json
- ghcnd_countries_schema.json
- ghcnd_hurricanes_schema.json
- ghcnd_inventory_schema.json
- ghcnd_states_schema.json
- ghcnd_stations_schema.json
- gsod_stations_schema.json
- noaa_ghcn_m_data_schema.json
- noaa_ghcn_m_stations_schema.json
- noaa_goes_16_cmip_schema.json
- noaa_goes_16_glm_schema.json
- noaa_goes_16_mcmip_schema.json
- noaa_goes_16_radiance_schema.json
- noaa_goes_17_cmip_schema.json
- noaa_goes_17_glm_schema.json
- noaa_goes_17_mcmip_schema.json
- noaa_goes_17_radiance_schema.json
- noaa_gsod_2020_schema.json
- noaa_gsod_2022_schema.json
- noaa_gsod_by_year_schema.json
- noaa_historic_severe_storms_schema.json
- noaa_lightning_strikes_schema.json
- noaa_spc_hail_schema.json
- noaa_spc_tornado_schema.json
- noaa_spc_wind_schema.json
- nws_forecast_regions_schema.json
- noaa_dag.py
- pipeline.yaml
- dataset.yaml
- noaa_passive_bioacoustic_dataset.tf
- provider.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- schema.json
- noaa_passive_bioacoustic_dag.py
- pipeline.yaml
- dataset.yaml
- csv_transform.py
- Dockerfile
- requirements.txt
- npi_raw_schema.json
- nppes_dag.py
- pipeline.yaml
- open_buildings_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- schema.json
- script.py
- open_buildings_dag.py
- pipeline.yaml
- dataset.yaml
- open_buildings_v2_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- schema.json
- script.py
- open_buildings_v2_dag.py
- pipeline.yaml
- dataset.yaml
- open_targets_dataset.tf
- provider.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- copy_genetics_data_dag.py
- pipeline.yaml
- copy_platform_data_dag.py
- pipeline.yaml
- dataset.yaml
- commuting_zone_income_rank_statistics_by_race_and_parent_income_percentile_pipeline.tf
- crosswalk_between_parent_and_child_income_percentiles_and_dollar_values_pipeline.tf
- intergenerational_transition_matrices_of_educational_attainment_by_race_and_gender_pipeline.tf
- national_child_and_parent_income_transition_matrices_by_race_and_gender_for_children_with_mothers_pipeline.tf
- national_child_and_parent_income_transition_matrices_by_race_and_gender_pipeline.tf
- national_statistics_by_parent_income_percentile_gender_race_pipeline.tf
- non_parametric_estimates_of_income_ranks_for_second_generation_immigrant_pipeline.tf
- parametric_estimates_of_income_ranks_for_second_generation_immigrant_children_pipeline.tf
- provider.tf
- race_and_economic_opportunity_dataset.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- commuting_zone_income_rank_statistics_by_race_and_parent_income_percentile_dag.py
- pipeline.yaml
- crosswalk_between_parent_and_child_income_percentiles_and_dollar_values_dag.py
- pipeline.yaml
- intergenerational_transition_matrices_of_educational_attainment_by_race_and_gender_dag.py
- pipeline.yaml
- national_child_and_parent_income_transition_matrices_by_race_and_gender_dag.py
- pipeline.yaml
- national_child_and_parent_income_transition_matrices_by_race_and_gender_for_children_with_mothers_dag.py
- pipeline.yaml
- national_statistics_by_parent_income_percentile_gender_race_dag.py
- pipeline.yaml
- non_parametric_estimates_of_income_ranks_for_second_generation_immigrant_dag.py
- pipeline.yaml
- parametric_estimates_of_income_ranks_for_second_generation_immigrant_children_dag.py
- pipeline.yaml
- dataset.yaml
- provider.tf
- san_francisco_dataset.tf
- san_francisco_pipeline.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- sf_311_service_requests_schema.json
- sf_bikeshare_station_info_schema.json
- sf_bikeshare_station_status_schema.json
- sf_bikeshare_trips_schema.json
- sf_calendar_schema.json
- sf_film_locations_schema.json
- sf_muni_fares_schema.json
- sf_muni_routes_schema.json
- sf_muni_shapes_schema.json
- sf_muni_stop_times_schema.json
- sf_muni_stops_schema.json
- sf_muni_trips_schema.json
- sf_sffd_service_calls_schema.json
- sf_sfpd_incidents_schema.json
- sf_street_trees_schema.json
- pipeline.yaml
- san_francisco_dag.py
- dataset.yaml
- artifact.yaml
- category_by_complaint_source.sql
- artifact.yaml
- sf_311_weekly_calls_forecast.ipynb
- sf_311_weekly_calls_forecast_test.py
- 311_service_requests_pipeline.tf
- provider.tf
- san_francisco_311_dataset.tf
- san_francisco_311_service_requests_dataset.tf
- variables.tf
- 311_service_requests_dag.py
- pipeline.yaml
- csv_transform.py
- Dockerfile
- requirements.txt
- dataset.yaml
- artifact.yaml
- average_trip_per_month.sql
- bikeshare_station_info_pipeline.tf
- bikeshare_station_status_pipeline.tf
- bikeshare_stations_pipeline.tf
- provider.tf
- san_francisco_bikeshare_dataset.tf
- san_francisco_bikeshare_stations_dataset.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- csv_transform.py
- Dockerfile
- requirements.txt
- bikeshare_station_info_dag.py
- bikeshare_stations_dag.py
- pipeline.yaml
- bikeshare_station_status_dag.py
- bikeshare_status_dag.py
- pipeline.yaml
- dataset.yaml
- bikeshare_status_pipeline.tf
- provider.tf
- san_francisco_bikeshare_status_dataset.tf
- variables.tf
- dataset.yaml
- film_locations_pipeline.tf
- provider.tf
- san_francisco_film_locations_dataset.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- film_locations_dag.py
- pipeline.yaml
- dataset.yaml
- provider.tf
- san_francisco_trees_dataset.tf
- street_trees_pipeline.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- street_trees_dag.py
- dataset.yaml
- provider.tf
- scalable_open_source_dataset.tf
- variables.tf
- Dockerfile
- requirements.txt
- script.py
- Dockerfile
- requirements.txt
- script.py
- deps_dev_dag.py
- pipeline.yaml
- dataset.yaml
- fails_to_deliver_pipeline.tf
- provider.tf
- sec_failure_to_deliver_dataset.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- fails_to_deliver_dag.py
- pipeline.yaml
- dataset.yaml
- provider.tf
- solar_potential_by_censustract_pipeline.tf
- solar_potential_by_postal_code_pipeline.tf
- sunroof_solar_dataset.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- solar_potential_by_censustract_dag.py
- pipeline.yaml
- solar_potential_by_postal_code_dag.py
- dataset.yaml
- provider.tf
- the_general_index_dataset.tf
- the_general_index_pipeline.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- the_general_index_schema.json
- pipeline.yaml
- the_general_index_dag.py
- dataset.yaml
- provider.tf
- thelook_ecommerce_dataset.tf
- thelook_ecommerce_pipeline.tf
- variables.tf
- distribution_centers.csv
- products.csv
- world_pop.csv
- Dockerfile
- fake.py
- requirements.txt
- pipeline.yaml
- thelook_ecommerce_dag.py
- dataset.yaml
- flights_impact_data_pipeline.tf
- metadata_pipeline.tf
- provider.tf
- travel_impact_model_dataset.tf
- variables.tf
- flights_impact_data_dag.py
- pipeline.yaml
- metadata_dag.py
- pipeline.yaml
- dataset.yaml
- provider.tf
- uniref50_dataset.tf
- uniref50_pipeline.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- uniref50_schema.json
- pipeline.yaml
- uniref50_dag.py
- dataset.yaml
- normals_hourly_pipeline.tf
- provider.tf
- us_climate_normals_dataset.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- us_climate_normals_dag.py
- dataset.yaml
- provider.tf
- san_jose_311_translation_pipeline.tf
- usa_cities_dataset.tf
- variables.tf
- pipeline.yaml
- san_jose_311_translation_dag.py
- dataset.yaml
- provider.tf
- usa_1910_current_pipeline.tf
- usa_names_dataset.tf
- variables.tf
- pipeline.yaml
- usa_1910_current_dag.py
- dataset.yaml
- provider.tf
- usda_nass_agriculture_dataset.tf
- variables.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- schema.json
- Dockerfile
- requirements.txt
- script.py
- pipeline.yaml
- usda_nass_agriculture_dag.py
- dataset.yaml
- mouse_brain_map_pipeline.tf
- provider.tf
- variables.tf
- vizgen_merfish_dataset.tf
- mouse_brain_map_dag.py
- pipeline.yaml
- dataset.yaml
- provider.tf
- variables.tf
- vqa_dataset.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- vqa_annotations_detail_schema.json
- vqa_annotations_schema.json
- vqa_complementary_pairs_schema.json
- vqa_questions_detail_schema.json
- vqa_questions_schema.json
- pipeline.yaml
- vqa_dag.py
- pipeline.yaml
- vqa_image_load_dag.py
- dataset.yaml
- country_series_definitions_pipeline.tf
- country_summary_pipeline.tf
- provider.tf
- series_summary_pipeline.tf
- series_times_pipeline.tf
- variables.tf
- world_bank_health_population_dataset.tf
- world_bank_health_population_pipeline.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- world_bank_health_population_dag.py
- dataset.yaml
- country_series_definitions_pipeline.tf
- country_summary_pipeline.tf
- provider.tf
- series_summary_pipeline.tf
- series_times_pipeline.tf
- variables.tf
- world_bank_intl_debt_dataset.tf
- world_bank_intl_debt_pipeline.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- world_bank_intl_debt_dag.py
- dataset.yaml
- country_series_definitions_pipeline.tf
- country_summary_pipeline.tf
- provider.tf
- series_summary_pipeline.tf
- variables.tf
- world_bank_intl_education_dataset.tf
- world_bank_intl_education_pipeline.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- pipeline.yaml
- world_bank_intl_education_dag.py
- dataset.yaml
- country_series_definitions_pipeline.tf
- country_summary_pipeline.tf
- footnotes_pipeline.tf
- provider.tf
- series_summary_pipeline.tf
- series_time_pipeline.tf
- variables.tf
- world_bank_wdi_dataset.tf
- csv_transform.py
- Dockerfile
- requirements.txt
- schema.json
- pipeline.yaml
- world_bank_wdi_dag.py
- dataset.yaml
- __init__.py
- added_project.png
- adding_project.png
- architecture.png
- fetch_upstream_contribute.png
- Dockerfile
- requirements.txt
- script.py
- dataset.yaml
- dataset_variables.json
- pipeline.yaml
- tutorial.ipynb
- dag_imports.json
- deploy_dag.py
- generate_dag.py
- generate_terraform.py
- scaffold.py
- dag.py.jinja2
- dag_context.py.jinja2
- default_args.py.jinja2
- license_header.py.jinja2
- task.py.jinja2
- backend.tf.jinja2
- google_bigquery_dataset.tf.jinja2
- google_bigquery_table.tf.jinja2
- google_storage_bucket.tf.jinja2
- license_header.tf.jinja2
- provider.tf.jinja2
- terraform.tfvars.jinja2
- variables.tf.jinja2
- test_deploy_dag.py
- test_generate_dag.py
- test_generate_terraform.py
- test_scaffold.py
- conftest.py
- test_checks_for_all_dags.py
- test_dag_integrity.py
- .flake8
- .gitignore
- .pre-commit-config.yaml
- .yamllint.yaml
- CHANGELOG.md
- CODE_OF_CONDUCT.md
- header-checker-lint.yml
- LICENSE
- OWNERS
- poetry.lock
- pyproject.toml
- README.md
- renovate.json
- version.txt
# 설치 가이드
1. 코드 내려받기
git clone https://github.com/GoogleCloudPlatform/public-datasets-pipelines
깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.
cd public-datasets-pipelines
방금 내려받은 프로젝트 폴더 안으로 이동합니다.
2. Docker
쉬움 추천사전 준비물
- Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
- Docker Desktop 컨테이너를 빌드하고 실행하려면 필요합니다. 설치 후 실행해서 백그라운드에 켜두세요.
⚠️ 이 프로젝트는 규모가 큰 저장소라, 이 방법이 실제 핵심 제품이 아니라 내부 하위 패키지를 가리키는 것일 수 있습니다. README 전체를 함께 확인해보세요.
docker build -f datasets/america_health_rankings/pipelines/_images/run_csv_transform_kub/Dockerfile -t public-datasets-pipelines .
Dockerfile을 기반으로 실행 가능한 이미지를 빌드합니다.
docker run -p 8080:80 public-datasets-pipelines
빌드된 이미지를 실제 컨테이너로 실행합니다.
터미널에 docker compose ps 를 입력해 컨테이너들이 Up 상태인지 확인하세요. README에 포트 번호가 적혀있다면 브라우저에서 http://localhost:포트번호 로 접속해보세요.
3. Python
쉬움사전 준비물
⚠️ 이 프로젝트는 규모가 큰 저장소라, 이 방법이 실제 핵심 제품이 아니라 내부 하위 패키지를 가리키는 것일 수 있습니다. README 전체를 함께 확인해보세요.
pip install -r datasets/america_health_rankings/pipelines/_images/run_csv_transform_kub/requirements.txt
requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.
jupyter notebook
브라우저에서 노트북(.ipynb) 파일들을 열람하고 실행할 수 있는 Jupyter 화면을 켭니다.
에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.
// repository documentation
Was this content helpful?
(0 ratings)
