init_ec2

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #go #macos #windows #awesome #awesome-list #security #pytorch #mcp #openai

(★ 11)

init EC2 cluster, for free-password-login(ubuntu and root). for hostname, for hosts file.

init_ec2 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# init_ec2
This project helps users to initialize a new cluster.

For example, you can watch a use case vedio on Youtube: https://youtu.be/OvVaib5xllo

It is suitable for:
 
 * a real cluster, which consists of PCs, small servers or other  physical machines you own.
 * cloud environment, such as Amazon EC2, aliyun.

The only conditions you should have are:

- you have installed ubuntu (tested on 14.04) OS on all the nodes. (it is always eligibly in cloud environments, e.g., EC2, aliyun)
- you  know or have set the ip address on each node. 
- **python 2.7** has been installed on your laptop/PC. (unless you use a Windows OS, otherwise you are eligibly. on Windows, you need to install Python yourself)
- **fabric** (version >=1.10, but not compatible with 2.0 version) has been installed on your laptop/PC
	- it seems a little harder than above steps. But believe me, you deserve it! 
	- on Mac, install easy_install and then fabric:
		- download file https://pypi.python.org/pypi/ez_setup 
		- sudo python ez_setup.py
		- sudo easy_install "fabric == 1.10.5"
		
	- on Linux (Ubuntu), the same way (notice, do not use pip to install fabric, you will be disappointed; besides, apt-get install fabric is ok, but the version is 1.8 now (2016.4))
		- sudo apt-get install python-setuptools python-dev build-essential
		- sudo easy_install "six==1.6.0" # I am not sure that whether it is necessary,
		- sudo easy_install "fabric == 1.10.5"
		- notice that, if there is something wrong, google it: e.g., lack of some .h file, just use apt-get install
	- on Windows,... Good luck, guy. I have not tried how to install fabric on Windows. But it should not be a probelm.  
init EC2 cluster, for free-password-login(ubuntu and root). for hostname, for hosts file.

##how to use
1. modify passwd.ini
 
    this is an example
		
		[default]
		#which section is actived
		activeSession=
		#the sudo user, only used for sudo tasks
		root =
		#passwd of sudoer
		passwd =
		
		#example of section
		[server]
		#uesr name, if use createUser task, this user name will be the new user
		newuser =
		#password of newuser
		passwd =
		#cluster ips, use comma to split
		hosts=
		#hostnames of cluster , the number of hostnames must equal with hosts. the hostnames is used for changeHostname tass
		hostnames=
		#locally jdk file. the related path is ./
		jdk_source_file=
		# when the jdk.tar.gz is unziped, the folder in the tar file.
		jdk_folder=
		# the ntp server address you  want to sync
		ntp_server=
		# the allowed net that can sync with your ntp server
		ntp_net=
		# the allowed net mask that can sync with your ntp server
		ntp_net_mask=
		
	this is an instance:
	
		[default]
		activeSession=server
		root = fit
		passwd =111
		
		[server]
		newuser = hxd
		passwd = 1112
		hosts=192.168.130.3, 192.168.130.5		hostnames=s3, s5, 
		jdk_source_file=files/jdk1.8.77.tar.gz
		jdk_folder=jdk1.8.0_77
		ntp_server=192.168.130.2
		[client]
		newuser = hxd
		passwd = 1112
		hosts = 192.168.130.2
		hostnames = s2
		jdk_source_file=files/jdk1.8.77.tar.gz
		jdk_folder=jdk1.8.0_77
		ntp_net=192.168.130.0
		ntp_net_mask=255.255.255.0
		ntp_server=192.168.130.2
		[all]
		newuser = hxd
		passwd = 1112
		hosts=192.168.130.2,192.168.130.3, 192.168.130.5
		hostnames=s2,s3, s5
		jdk_source_file=files/jdk1.8.77.tar.gz
		jdk_folder=jdk1.8.0_77
2. you can use this script like this 
   
   `fab task_name` 
   
   you can use `fab help` to show how many tasks supportted.
   
   current supportted tasks:
   
    * createUser
        - create a new user (username is 'newuser' in the configurations, password is 'passwd' in the active configurations).
    * removeUser
    	- delete the user we created
    * addIntoHostFile
    	- add all the <ip,hostname> into /etc/hosts
    * changeHostname
    	- change all the hostnames as the values in the actived conf.
    * installnptserver
    	- install ntp server on nodes. We suggest you only install ntp server on one node in the cluster, instead of all the nodes.(how to do: enable a configruation which has only one ip in 'hosts')
    * distributeJDK
    	- after you manual download a jdk file, you can use this task to push it to all the nodes and modify the $JAVA_HOME and $PATH 
    * setnpttaks
    	- add a scheduled task every day for ntpdate time with the ntp server
    * password-free ssh
    	- setting password-free ssh. how to use: 
    		- fab ssh1  (this can generate pem files)
    		- fab ssh2  (this finish the password-free)
    		- fab ssh3  (cleanup)
    * restart
    	- reboot the cluster
    * correct_bashrc
    	- normally, we use `ssh hxd@node1 "jps"`, the terminal says: jps command not found, though we have set $JAVA_HOME and $PATH in .bashrc file. The reason is that .bashrc is ignored when we run a non-interactive login command. To solve that, use this task to repair the bashrc file.
    * modifyAtpSource
    	- modify the apt sources.list as files/sources.list.
    * addDisk
    	- when you add a new disk, you should format it and then mount it. this task help you. use `fab addDisk:device,location`. e.g., `fab addDisk:/dev/sdb,/datab`
    * TODO: we will add new tasks to help user add new nodes in an existed cluster

3. some bigdata system and monitor system. you can use it like `fab -f fab_bigdata taskname`

current supported task:

* installCollectd
        - you need add a section in passwd.ini

[collectd]
            \#server ip tell the node that where to send the collected data
            \#the receiver should have a receive process. e.g. influxdb and others
            \#the most easy receiver is collected itself. In this way, you should modify the receiver's collectd configuration file yourself: /etc/collectd/collectd.conf
            \#in network plugin: replace "Server" by "Listen"
            server_ip=
            server_port=
            \#collect interval unit: second
            interval=5
    * runCollectd
        - `fab -f fab_bigdata runCollectd:stop` or `fab -f fab_bigdata runCollectd:start`
    * installGangliaClient
        - ganglia is a master-slave architecture. now you can only install the client, ganglia-monitor. The master processes, gmetad  and ganglia-webfrontend, are not included.
        - if you want to install the master processes, see https://www.digitalocean.com/community/tutorials/introduction-to-ganglia-on-ubuntu-14-04.
        - to install the client process, you need a gmond.conf in files folder. Notice, modify the ip address in the file as your master ip, and modify the cluster name as the same as your master configuration.

* installCouch
        - you can install CouchDB.
        - normally, couchdb is a single-node. therefore, your active section.hosts should have only one ip. But if you exactly want to install couchdb on all the nodes, use `fab -f fab_bigdata installCouch:y`
        - in current version, we do not modify the bind-address of CouchDB, the default is  localhost.
    * runCouch
        - you can start or stop CouchDB
        - how to use: `fab -f fab_bigdata runCouch:'start'` or `fab -f fab_bigdata runCouch:'stop'` or `fab -f fab_bigdata runCouch:'start','y'` or  `fab -f fab_bigdata runCouch:'stop','y'`. 'y' means you have  couchdb on each node

##Amazon Ec2 (or other pem based ssh)
in Amazon Ec2, we use a pem file to login instead of password.

To use the script on Amazon EC2. you need modify the script now. But do not worry, just a little thing you need to do:

* comment "env.password=..." in rootUser() 
* add env.key_filename=['filepath'] in rootUser()
* Cheers!

init_ec2

init_ec2 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

init_ec2

init_ec2 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README