DevOps-Python-tools
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
File Explorer
Download Latest Version (.zip)- pipeline.yml
- config.yml
- *_centos.yaml.disabled
- alpine.yaml
- alpine_3.yaml
- centos.yaml.disabled
- centos7.yaml.disabled
- centos8.yaml.disabled
- checkov.yaml
- codeowners.yaml
- codeql-analysis.yml
- debian.yaml
- debian_10.yaml
- debian_11.yaml
- debian_12.yaml
- debian_6.yaml.disabled
- debian_7.yaml.disabled
- debian_8.yaml.disabled
- debian_9.yaml.disabled
- docker_pytools_alpine.yaml
- docker_pytools_debian.yaml
- docker_pytools_fedora.yaml
- docker_pytools_ubuntu.yaml
- fedora.yaml
- fork-sync.yaml
- fork-update-pr.yaml
- ghcr_python_ubuntu.yaml.disabled
- grype.yaml
- json.yaml
- kics.yaml
- mac.yaml
- mac_11.yaml
- mac_12.yaml
- markdown.yaml
- pypy2.yaml.disabled
- pypy3.yaml.disabled
- python2.7.yaml.disabled
- python3.10.yaml
- python3.11.yaml
- python3.6.yaml.disabled
- python3.7.yaml
- python3.8.yaml
- python3.9.yaml
- semgrep-cloud.yaml
- semgrep.yaml
- shellcheck.yaml
- trivy.yaml
- ubuntu.yaml
- ubuntu_14.04.yaml.disabled
- ubuntu_16.04.yaml.disabled
- ubuntu_18.04.yaml.disabled
- ubuntu_20.04.yaml
- ubuntu_22.04.yaml
- ubuntu_github.yaml
- validate.yaml
- xml.yaml
- yaml.yaml
- CODEOWNERS
- ISSUE_TEMPLATE.md
- semaphore.yml
- hdp-2.3-cluster-basic-named.json
- hdp-2.3-cluster-basic.json
- hdp-2.3-cluster-hdfs-ha.json
- hdp-2.3-cluster-hdfs-yarn-ha.json
- hdp-2.3-cluster-hdfs-yarn-hbase-ha.json
- hostmappings-hdfs-ha-random-hosts.json
- hostmappings-hdfs-ha.json
- hostmappings-hdfs-yarn-ha-random-hosts.json
- hostmappings-hdfs-yarn-ha.json
- hostmappings-hdfs-yarn-hbase-ha-random-hosts.json
- hostmappings-hdfs-yarn-hbase-ha.json
- hostmappings-named-random-hosts.json
- hostmappings-named.json
- hostmappings-random-hosts.json
- hostmappings.json
- README.md
- .concourse.yml
- .gocd.yml
- buildspec.yml
- cloudbuild.yaml
- centos7-12nodes-encrypted.json
- centos7-12nodes.json
- centos7-1node-encrypted.json
- centos7-1node.json
- centos7-3nodes-encrypted.json
- centos7-3nodes.json
- README.md
- barclaycard_crunch_accounting_csv_statement_converter.sh
- barclays_crunch_accounting_csv_statement_converter.sh
- lib.sh
- .gcloudignore
- deploy.sh
- main.py
- Makefile
- README.md
- requirements.txt
- .gcloudignore
- deploy.sh
- main.py
- Makefile
- README.md
- requirements.txt
- .gcloudignore
- deploy.sh
- main.py
- Makefile
- README.md
- requirements.txt
- serverless.yml
- __init__.py
- hive_impala_cli.py
- postgres_cli.py
- apk-packages-dev.txt
- apk-packages-pip.txt
- apk-packages.txt
- bootstrap.sh
- brew-packages.txt
- ci_bootstrap.sh
- ci_git_set_dir_safe.sh
- deb-packages-dev.txt
- deb-packages-optional.txt
- deb-packages-pip.txt
- deb-packages.txt
- gocd_config_repo.json
- jenkins-job.xml
- rpm-packages-dev.txt
- rpm-packages-optional.txt
- rpm-packages-pip.txt
- rpm-packages.txt
- .teamcity.vcs.json
- .teamcity.vcs.oauth.json
- .teamcity.vcs.ssh.json
- add_ou.ldif
- cd_catalog.xml
- embedded_double_quotes.json
- header.csv
- header_quoted.csv
- ldap.ldif
- ldap_download.sh
- ldap_upload.sh
- multirecord.json
- multirecord_single_double_mixed_quotes.notjson
- multirecord_single_quotes.notjson
- multirecord_single_quotes_embedded_double_quotes.notjson
- multirecord_single_quotes_embedded_double_quotes_unescaped.notjson
- multirecord_with_blank_lines.notjson
- plant_catalog.xml
- quoted.csv
- simple.xml
- single_quotes.notjson
- single_quotes_embedded_double_quotes.notjson
- single_quotes_embedded_double_quotes_unescaped.notjson
- test.csv
- test.ini
- test.ini-colons
- test.ini-hashes
- test.json
- test.toml
- test.yaml
- uk_marriage_rates_2022.xslx
- apache-drill-docker-compose.yml
- common.yml
- elasticsearch-docker-compose.yml
- elasticsearch-elastic.co-docker-compose.yml
- hadoop-docker-compose.yml
- hbase-docker-compose.yml
- presto-dev-docker-compose.yml
- presto-docker-compose.yml
- registry-docker-compose.yml
- solrcloud-docker-compose.yml
- all.sh
- check.sh
- compile.sh
- excluded.sh
- help.sh
- python3.sh
- syntax.sh
- test_anonymize.py
- test_anonymize.sh
- test_apache-drill.sh
- test_center.sh
- test_docker.sh
- test_docker_registry_show_tags.sh
- test_dockerfiles_check_git_branches.sh
- test_dockerfiles_check_git_tags.sh
- test_dockerhub_search.sh
- test_dockerhub_show_tags.sh
- test_elasticsearch.sh
- test_find_active_server.sh
- test_find_duplicate_files.sh
- test_find_python_library_path.sh
- test_getent.sh
- test_git_check_branches_upstream.sh
- test_hadoop.sh
- test_hbase.sh
- test_headtail.sh
- test_hexanonymize.sh
- test_json.sh
- test_json_docs_to_bulk_multiline.sh
- test_json_to_xml.sh
- test_json_to_yaml.sh
- test_opentsdb.sh
- test_presto.sh
- test_quay_show_tags.sh
- test_serf_event_handler.sh
- test_solrcloud.sh
- test_spark_csv_to_avro.sh
- test_spark_csv_to_parquet.sh
- test_spark_json_to_avro.sh
- test_spark_json_to_parquet.sh
- test_spark_z_avro_to_parquet.sh
- test_spark_z_parquet_to_avro.sh
- test_strip_ansi_escape_codes.sh
- test_timeout.sh
- test_validate_avro.sh
- test_validate_cson.sh
- test_validate_csv.sh
- test_validate_ini.sh
- test_validate_json.sh
- test_validate_ldap_ldif.sh
- test_validate_multimedia.sh
- test_validate_parquet.sh
- test_validate_toml.sh
- test_validate_xml.sh
- test_validate_yaml.sh
- test_welcome.sh
- test_xml_to_json.sh
- test_xml_to_yaml.sh
- test_yamllint.sh
- utils.sh
- .appveyor.yml
- .checkov.yaml
- .cirrus.yml
- .coafile
- .drone.yml
- .editorconfig
- .envrc
- .envrc-python
- .flake8
- .gitallowed
- .gitignore
- .gitlab-ci.yml
- .gitmodules
- .ipython-notebook-pyspark.00-pyspark-setup.py
- .ipython-notebook-pyspark.ipython_notebook_config.py.j2
- .mdl.rb
- .mdlrc
- .pre-commit-config.yaml
- .pylintrc
- .pyup.yml
- .sonarcloud.properties
- .travis.yml
- .yamllint
- ambari_ams_metrics.sh
- ambari_blueprints.py
- ambari_cancel_all_requests.sh
- ambari_trigger_service_checks.py
- anonymize.py
- anonymize_custom.conf
- anonymize_ignore.conf
- anonymize_parallel.sh
- aws_s3_presign.py
- aws_users_access_key_age.py
- aws_users_last_used.py
- aws_users_pw_last_used.py
- aws_users_unused_access_keys.py
- azure-pipelines.yml
- bash-tools
- bitbucket-pipelines.yml
- boot
- buddy.yml
- center.py
- cloudera_navigator_tables_used.py
- cloudera_navigator_tables_used_postgres.py
- codefresh.yml
- crunch_accounting_csv_statement_converter.py
- docker_pull_all_images.sh
- docker_pull_all_images_all_tags.sh
- docker_pull_all_tags.sh
- docker_registry_show_tags.py
- dockerfiles_check_git_branches.py
- dockerfiles_check_git_tags.py
- dockerhub_search.py
- dockerhub_show_tags.py
- find_active_apache_drill.py
- find_active_apache_drill2.py
- find_active_cassandra.py
- find_active_elasticsearch.py
- find_active_hadoop2_namenode.py
- find_active_hadoop_namenode.py
- find_active_hadoop_yarn_resource_manager.py
- find_active_hbase_master.py
- find_active_hbase_stargate.py
- find_active_hbase_thrift.py
- find_active_impala.py
- find_active_impala_catalog.py
- find_active_impala_statestore.py
- find_active_kubernetes_api.py
- find_active_oozie.py
- find_active_presto_coordinator.py
- find_active_server.py
- find_active_solrcloud.py
- find_duplicate_files.py
- find_missing_files_in_sequence.py
- gcp_service_account_credential_keys.py
- getent.py
- git_check_branches_upstream.py
- hbase_compact_tables.py
- hbase_flush_tables.py
- hbase_generate_data.py
- hbase_region_requests.py
- hbase_regions_by_memstore_size.py
- hbase_regions_by_size.py
- hbase_regions_least_used.py
- hbase_regionserver_requests.py
- hbase_scan_table_column_names.sh
- hbase_show_table_region_ranges.py
- hbase_table_region_row_distribution.py
- hbase_table_regions_by_regionserver.sh
- hbase_table_regionserver_distribution.sh
- hbase_table_row_key_distribution.py
- hbase_uncompacted_regions.sh
- hdfs_files_native_checksums.jy
- hdfs_files_stats.jy
- hdfs_find_replication_factor_1.py
- hdfs_time_block_reads.jy
- headtail.py
- hexanonymize.py
- hive_compute_table_stats.py
- hive_foreach_table.py
- hive_schemas_csv.py
- hive_tables_column_counts.py
- hive_tables_list.py
- hive_tables_locations.py
- hive_tables_metadata.py
- hive_tables_null_columns.py
- hive_tables_null_rows.py
- hive_tables_row_column_counts.py
- hive_tables_row_counts.py
- hive_tables_row_counts_any_nulls.py
- impala_compute_table_stats.py
- impala_foreach_table.py
- impala_schemas_csv.py
- impala_tables_column_counts.py
- impala_tables_list.py
- impala_tables_locations.py
- impala_tables_metadata.py
- impala_tables_null_columns.py
- impala_tables_null_rows.py
- impala_tables_row_column_counts.py
- impala_tables_row_counts.py
- impala_tables_row_counts_any_nulls.py
- ipython_notebook_pyspark.py
- Jenkinsfile
- json_docs_to_bulk_multiline.py
- json_to_xml.py
- json_to_yaml.py
- jython_autoinstall.exp
- jython_install.sh
- kics.config
- LICENSE
- Makefile
- opentsdb_import_metric_distribution.py
- opentsdb_list_metrics.sh
- opentsdb_list_metrics_hbase.sh
- pig-text-to-elasticsearch.pig
- pig-text-to-solr.pig
- pig_udfs.jy
- plot_uk_marriage_rates.py
- pylib
- python_find_library_path.py
- pythonpath.py
- quay_show_tags.py
- README.md
- requirements.txt
- selenium_hub_browser_test.py
- serf_event_handler.py
- sonar-project.properties
- spark_avro_to_parquet.py
- spark_csv_to_avro.py
- spark_csv_to_parquet.py
- spark_json_to_avro.py
- spark_json_to_parquet.py
- spark_parquet_to_avro.py
- sql
- strip_ansi_escape_codes.py
- templates
- timeout.py
- travis_debug_session.py
- travis_last_log.py
- urldecode.py
- urlencode.py
- validate_avro.py
- validate_cson.py
- validate_csv.py
- validate_ini.py
- validate_ini2.py
- validate_json.py
- validate_ldap_ldif.py
- validate_multimedia.py
- validate_parquet.py
- validate_properties.py
- validate_toml.py
- validate_xml.py
- validate_yaml.py
- welcome.py
- xml_to_json.py
- xml_to_yaml.py
- yaml_to_json.py
# Installation Guide
1. Get the code
git clone https://github.com/HariSekhon/DevOps-Python-tools
Downloads the entire project code from GitHub to your computer.
cd DevOps-Python-tools
Moves into the project folder you just downloaded.
2. Official Install Script
Easy RecommendedPrerequisites
- Homebrew A package manager for macOS/Linux.
curl -L https://git.io/python-bootstrap | sh
Downloads and runs the official install script in one line β this handles the full setup automatically.
brew install openssl wget
Installs the pre-built package via Homebrew β no source build required.
After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.
Pulled directly from this repo's README.
3. Python
EasyPrerequisites
sudo pip install -r requirements.txt
Installs the Python libraries listed in requirements.txt (or similar).
sudo pip install --index https://host.domain.com/api/pypi/repo/simple --trusted host.domain.com -r requirements.txt
Installs the Python libraries listed in requirements.txt (or similar).
sudo pip install --proxy hari:mypassword@proxy-host:8080 -r requirements.txt
Installs the Python libraries listed in requirements.txt (or similar).
sudo pip install --upgrade pip==9.0.1
Installs the Python libraries listed in requirements.txt (or similar).
sudo pip install --upgrade pip
Installs the Python libraries listed in requirements.txt (or similar).
If it runs without errors and prints output in the terminal, it worked.
Pulled directly from this repo's README.
4. Make
MediumPrerequisites
- Git Needed to download the project code from GitHub.
- Make Usually pre-installed on Linux/macOS. On Windows, install separately (e.g. via MSYS2 or WSL).
make
Compiles the code based on the generated build configuration to produce an executable.
make anonymize.pyc
Compiles the code based on the generated build configuration to produce an executable.
make jython
Compiles the code based on the generated build configuration to produce an executable.
make update
Compiles the code based on the generated build configuration to produce an executable.
make test
Compiles the code based on the generated build configuration to produce an executable.
If it finishes without errors, it worked. Try running the generated executable directly.
Pulled directly from this repo's README.
// repository documentation
Was this content helpful?
(0 ratings)