openstack_kubernetes

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

openstack_kubernetes

★ 8 Open GitHub ↗

Deploy a Fleet Rktnetes cluster over OpenStack

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Openstack Juno

Note: Some improvement to do with the Liberty for etcd add / remove members field.

The image used is a Ubuntu 16.04 server.
The fleet stack also works with Debian Jessie.

# Fleet Only

## Stages

### Image

First, *easily* generate an image with `make instance`, this will produced this output:

...
    2016-09-13 16:10:21Z [random]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:10:21Z [generate_userdata]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:10:22Z [random]: CREATE_COMPLETE  state changed
    2016-09-13 16:10:22Z [generate_userdata]: CREATE_COMPLETE  state changed
    2016-09-13 16:10:22Z [generate_securitygroup]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:10:23Z [network]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:10:24Z [generate_securitygroup]: CREATE_COMPLETE  state changed
    2016-09-13 16:10:24Z [network]: CREATE_COMPLETE  state changed
    2016-09-13 16:10:24Z [subnet]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:10:26Z [subnet]: CREATE_COMPLETE  state changed
    2016-09-13 16:10:26Z [port]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:10:26Z [router]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:10:28Z [port]: CREATE_COMPLETE  state changed
    2016-09-13 16:10:28Z [router]: CREATE_COMPLETE  state changed
    2016-09-13 16:10:28Z [interface]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:10:30Z [interface]: CREATE_COMPLETE  state changed
    2016-09-13 16:10:30Z [instance]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:10:32Z [floating]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:10:34Z [floating]: CREATE_COMPLETE  state changed
    2016-09-13 16:11:54Z [instance]: CREATE_COMPLETE  state changed
    +---------------------+------------------------------------------------------------------+
    | Field               | Value                                                            |
    +---------------------+------------------------------------------------------------------+
    | id                  | 00000000-aaaa-bbbb-cccc-999999999999                             |
    | stack_name          | instance                                                         |
    | description         | All in one to prepare image for Fleet and Kubernetes main stacks |
    |                     |                                                                  |
    | creation_time       | 2016-09-13T16:10:21Z                                             |
    | updated_time        | None                                                             |
    | stack_status        | CREATE_COMPLETE                                                  |
    | stack_status_reason | Stack CREATE completed successfully                              |
    +---------------------+------------------------------------------------------------------+

The instance will shutting down and write a healthy result over the console.
Ensure with `make is_instance_finished_well` and `make is_instance_off`

Then, `make image`

Server snapshotting... 25% complete
    ...
    Server snapshotting... 100% complete
    Finished

Remove the current image with

make iclean
    2016-09-13 16:32:39Z [instance]: DELETE_IN_PROGRESS  state changed
    2016-09-13 16:32:40Z [floating]: DELETE_IN_PROGRESS  state changed
    2016-09-13 16:32:43Z [floating]: DELETE_COMPLETE  state changed
    2016-09-13 16:32:49Z [instance]: DELETE_COMPLETE  state changed
    2016-09-13 16:32:49Z [port]: DELETE_IN_PROGRESS  state changed
    2016-09-13 16:32:49Z [generate_userdata]: DELETE_IN_PROGRESS  state changed

### Fleet    
    
Once the fleet image is done, you're ready to launch a fleet cluster.

make fleet
    
    2016-09-13 16:22:06Z [router]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:22:07Z [router]: CREATE_COMPLETE  state changed
    2016-09-13 16:22:07Z [etcd_static]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:25:03Z [etcd_static]: CREATE_COMPLETE  state changed
    2016-09-13 16:25:03Z [worker_statefull]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:28:17Z [worker_statefull]: CREATE_COMPLETE  state changed
    2016-09-13 16:28:17Z [worker_stateless]: CREATE_IN_PROGRESS  state changed
    2016-09-13 16:30:09Z [worker_stateless]: CREATE_COMPLETE  state changed
    +---------------------+------------------------------------------------------------------------------------------+
    | Field               | Value                                                                                    |
    +---------------------+------------------------------------------------------------------------------------------+
    | id                  | 00000000-aaaa-bbbb-cccc-999999999999                                                     |
    | stack_name          | fleet                                                                                    |
    | description         | Fleet cluster over etcd static cluster, with Flannel, Kafka + ELK access over Traefik RP |
    |                     |                                                                                          |
    | creation_time       | 2016-09-13T16:22:06Z                                                                     |
    | updated_time        | None                                                                                     |
    | stack_status        | CREATE_COMPLETE                                                                          |
    | stack_status_reason | Stack CREATE completed successfully                                                      |
    +---------------------+------------------------------------------------------------------------------------------+

Now, the following infrastructure is ready:

![fleet](https://raw.githubusercontent.com/JulienBalestra/openstack_kubernetes/master/docs/fleet.png)

The stateless instance expose two services, Kibana and Inventory

You can import the kibana/sample.json dashboard to visualize this:

![fleet](https://raw.githubusercontent.com/JulienBalestra/openstack_kubernetes/master/docs/kibana.png)

### Technologies

* ResourceGroup of 3 instances of etcd members
* AutoScalingGroup of 3 instances marked as stateful
* AutoScalingGroup of 2 instances marked as stateless

#### Etcd static

* Etcd member (as static bootstrap)
* Flannel
* Fleet
* SkyDNS
* Journald Stream to Kafka

#### Stateful

* Etcd proxy
* Flannel
* Fleet
* SkyDNS
* Journald Stream to Kafka
* Torus (inactive)
* Elasticsearch
* Zookeeper
* Kafka

#### Stateless

* Etcd proxy
* Flannel
* Fleet
* SkyDNS
* Journald Stream to Kafka
* Traefik
* Logstash
* Kibana

# Fleet for Kubernetes - Rktnetes

Based on the Fleet stack, this stack append:

* AutoScalingGroup of 3 Kubemaster (api-server) 
* AutoScalingGroup of 2 Kubenode (kubelet)

Using the Makefile:

make kubernetes
    
    2016-09-14 14:52:43Z [router]: CREATE_IN_PROGRESS  state changed
    2016-09-14 14:52:45Z [router]: CREATE_COMPLETE  state changed
    2016-09-14 14:52:45Z [etcd_static]: CREATE_IN_PROGRESS  state changed
    2016-09-14 14:54:47Z [etcd_static]: CREATE_COMPLETE  state changed
    2016-09-14 14:54:47Z [worker_statefull]: CREATE_IN_PROGRESS  state changed
    2016-09-14 14:56:44Z [worker_statefull]: CREATE_COMPLETE  state changed
    2016-09-14 14:56:44Z [worker_stateless]: CREATE_IN_PROGRESS  state changed
    2016-09-14 14:58:40Z [worker_stateless]: CREATE_COMPLETE  state changed
    2016-09-14 14:58:40Z [kubemaster]: CREATE_IN_PROGRESS  state changed
    2016-09-14 15:00:45Z [kubemaster]: CREATE_COMPLETE  state changed
    2016-09-14 15:00:45Z [kubenode]: CREATE_IN_PROGRESS  state changed
    2016-09-14 15:02:47Z [kubenode]: CREATE_COMPLETE  state changed
    +---------------------+-------------------------------------------------------------------------------------------------------------------------+
    | Field               | Value                                                                                                                   |
    +---------------------+-------------------------------------------------------------------------------------------------------------------------+
    | id                  | 00000000-aaaa-bbbb-cccc-999999999999                                                                                    |
    | stack_name          | kubernetes                                                                                                              |
    | description         | Fleet cluster over etcd static cluster, with Flannel, Kafka + ELK access over Traefik RP to deploy a Kubernetes cluster |
    |                     |                                                                                                                         |
    | creation_time       | 2016-09-14T14:52:43Z                                                                                                    |
    | updated_time        | None                                                                                                                    |
    | stack_status        | CREATE_COMPLETE                                                                                                         |
    | stack_status_reason | Stack CREATE completed successfully                                                                                     |
    +---------------------+-------------------------------------------------------------------------------------------------------------------------+

### Deployed

Now, the following infrastructure is ready:

![kubernetes](https://raw.githubusercontent.com/JulienBalestra/openstack_kubernetes/master/docs/kubernetes.png)

With a quick overview of the machine deployment:

![kubernetes](https://raw.githubusercontent.com/JulienBalestra/openstack_kubernetes/master/docs/machines.png)

With, as example, the following network distribution:

![kubernetes](https://raw.githubusercontent.com/JulienBalestra/openstack_kubernetes/master/docs/interfaces.png)

Checking the network connectivity and latency (express in milliseconds) between each host accross differents networks (Regular, Flannel)

![kubernetes](https://raw.githubusercontent.com/JulienBalestra/openstack_kubernetes/master/docs/connections.png)

Of course, the Kibana is available over the FloatingIP of the stateless group.

### Using Kubernetes

Finally, you can run the kubectl against the Kubemaster's FloatingIP as follow:

kubectl -s kubeMaster0,kubeMaster1,kubeMaster2 get cs
    NAME                 STATUS    MESSAGE              ERROR
    controller-manager   Healthy   ok                   
    scheduler            Healthy   ok                   
    etcd-0               Healthy   {"health": "true"}  
    
    kubectl -s kubeMaster0,kubeMaster1,kubeMaster2 get no
    NAME                   STATUS    AGE
    kube-node-achruumzsn   Ready     1m
    kube-node-yvkndqjpbm   Ready     1m

The Kubernetes cluster point to http://127.0.0.1:2379 for service discovery, who's a proxy to the static etcd cluster, see Under the Hood for more details about etcd.

## Under the hood

### Etcd management

etcdctl cluster-health
    member 1eeb69136ece2e20 is healthy: got healthy result from http://192.168.1.10:2379
    member 761d10027054fc0a is healthy: got healthy result from http://192.168.1.12:2379
    member 9e2fef28ec9ef1b0 is healthy: got healthy result from http://192.168.1.11:2379
    cluster is healthy

To add an extra etcd member in the cluster (fleet, kubernetes):

make fleet_add_extra_member
    # Will produce the following stdout:
    
    subnet="00000000-aaaa-bbbb-cccc-999999999999"
    network="00000000-aaaa-bbbb-cccc-999999999999"
    key_name="ubuntu"
    image="fleet"
    etcd_initial_cluster_ip="http://192.168.1.10:2379,http://192.168.1.11:2379,http://192.168.1.12:2379"
    etcd_initial_cluster="static0=http://192.168.1.10:2380,static1=http://192.168.1.11:2380,static2=http://192.168.1.12:2380"
    prefix="192.168.1.1"
    anti_affinity="00000000-aaaa-bbbb-cccc-999999999999"
    etcd_initial_cluster_token="iTRDy5eNvi"
    security_group="00000000-aaaa-bbbb-cccc-999999999999"
    flavor="m1.medium"
    name="static"
    ...
    2016-09-13 17:02:19Z [port]: CREATE_IN_PROGRESS  state changed
    2016-09-13 17:02:20Z [userdata]: CREATE_IN_PROGRESS  state changed
    2016-09-13 17:02:21Z [userdata]: CREATE_COMPLETE  state changed
    2016-09-13 17:02:21Z [port]: CREATE_COMPLETE  state changed
    2016-09-13 17:02:21Z [instance]: CREATE_IN_PROGRESS  state changed
    2016-09-13 17:05:09Z [instance]: CREATE_COMPLETE  state changed
    +---------------------+---------------------------------------+
    | Field               | Value                                 |
    +---------------------+---------------------------------------+
    | id                  | 00000000-aaaa-bbbb-cccc-999999999999  |
    | stack_name          | etcd_static-3-iTRDy5eNvi              |
    | description         | Etcd instance for static Etcd cluster |
    |                     |                                       |
    | creation_time       | 2016-09-13T17:02:19Z                  |
    | updated_time        | None                                  |
    | stack_status        | CREATE_COMPLETE                       |
    | stack_status_reason | Stack CREATE completed successfully   |
    +---------------------+---------------------------------------+
    
    etcdctl cluster-health
    member 1eeb69136ece2e20 is healthy: got healthy result from http://192.168.1.10:2379    
    member 761d10027054fc0a is healthy: got healthy result from http://192.168.1.12:2379
    member 9e2fef28ec9ef1b0 is healthy: got healthy result from http://192.168.1.11:2379
    member 2b271486cef2a7b4 is healthy: got healthy result from http://192.168.1.13:2379
    cluster is healthy
    
And delete the latest extra member
    
    make fleet_delete_extra_member
    2016-09-13 17:06:52Z [instance]: DELETE_IN_PROGRESS  state changed
    2016-09-13 17:07:18Z [instance]: DELETE_COMPLETE  state changed
    2016-09-13 17:07:18Z [port]: DELETE_IN_PROGRESS  state changed
    
    etcdctl cluster-health
    member 1eeb69136ece2e20 is healthy: got healthy result from http://192.168.1.10:2379
    failed to check the health of member 2b271486cef2a7b4 on http://192.168.1.13:2379: Get http://192.168.1.13:2379/health: dial tcp 192.168.1.13:2379: getsockopt: no route to host
    member 2b271486cef2a7b4 is unreachable: [http://192.168.1.13:2379] are all unreachable
    member 761d10027054fc0a is healthy: got healthy result from http://192.168.1.12:2379
    member 9e2fef28ec9ef1b0 is healthy: got healthy result from http://192.168.1.11:2379
    cluster is healthy

Then, re-add one

make fleet_add_extra_member
    ...
    
    etcdctl cluster-health
    member 1eeb69136ece2e20 is healthy: got healthy result from http://192.168.1.10:2379
    member 73d3ac441595fb2c is healthy: got healthy result from http://192.168.1.13:2379
    member 761d10027054fc0a is healthy: got healthy result from http://192.168.1.12:2379
    member 9e2fef28ec9ef1b0 is healthy: got healthy result from http://192.168.1.11:2379
    cluster is healthy

Again, delete all extra **members**

make fleet_delete_extra_members # memberS

Periodic *etcd-gc.timer* will store any missing member in /var/lib/etcd/losts

Sep 14 15:05:56 static-2-j9cjctlpec etcd-gc[3097]: /bin/cat: /var/lib/etcd/losts: No such file or directory
    Sep 14 15:06:02 static-2-j9cjctlpec etcd-gc[3120]: bc887d297aeffc9d
    Sep 14 15:06:02 static-2-j9cjctlpec systemd[1]: Started Purge orphan Etcd Members.
    
If a missing member stored in /var/lib/etcd/losts still missing, *etcd-gc.timer* remove it
    
    Sep 14 16:05:57 static-2-j9cjctlpec systemd[1]: Starting Purge orphan Etcd Members...
    Sep 14 16:05:57 static-2-j9cjctlpec etcd-gc[3272]: bc887d297aeffc9d
    Sep 14 16:06:03 static-2-j9cjctlpec etcd-gc[3275]: Actual losts: bc887d297aeffc9d
    Sep 14 16:06:03 static-2-j9cjctlpec etcd-gc[3275]: Checking bc887d297aeffc9d
    Sep 14 16:06:21 static-2-j9cjctlpec etcd-gc[3275]: PING 192.168.1.13 (192.168.1.13) 56(84) bytes of data.
    Sep 14 16:06:21 static-2-j9cjctlpec etcd-gc[3275]: --- 192.168.1.13 ping statistics ---
    Sep 14 16:06:21 static-2-j9cjctlpec etcd-gc[3275]: 3 packets transmitted, 0 received, 100% packet loss, time 2015ms
    Sep 14 16:06:21 static-2-j9cjctlpec etcd-gc[3275]: Member is lost: bc887d297aeffc9d
    Sep 14 16:06:22 static-2-j9cjctlpec etcd-gc[3275]: Removed member bc887d297aeffc9d from cluster
    Sep 14 16:06:26 static-2-j9cjctlpec systemd[1]: Started Purge orphan Etcd Members.

## Add a Kibana
    
    dig @localhost kibana.skydns.local +short
    10.1.56.2
    
    systemctl start fleet-kibana@1
    dig @localhost kibana.skydns.local +short
    10.1.56.2
    10.1.47.2
  
    
## Add a Logstash
    
    dig @localhost logstash.skydns.local +short
    10.1.14.2
    
    systemctl start fleet-logstash@1
    dig @localhost logstash.skydns.local +short
    10.1.14.2
    10.1.105.3

# Setup steps

This stack is design to works behind corporate proxies and use local object store as cache

#### Python eggs
    pip install python-openstackclient
    pip install python-heatclient
    
#### Common packages
    curl jq
    
#### ACI builders
    docker2aci
    
    
#### Environment
    cp -v environment.template environment
    # FILL-IT
    # ...
    
    source environment
    make check
    
#### Versions
    cat image/versions
    
All versions listed above have to be stored in public dedicated Swift Buckets (aka containers)

swift list 
    # Will produce this stdout:
    
    calico
    cni
    confd
    docker
    etcd
    flannel
    fleet
    inventory
    kubernetes
    netenv
    rkt
    torus

### ACI

All ACIs have to be stored like this:
    
    swift list aci 
    # Will produce this stdout:
    
    calico-node.aci
    elasticsearch.aci
    etcd.aci
    hyperkube.aci
    jds_kafka.aci
    kafka.aci
    kibana.aci
    logstash.aci
    skydns.aci
    stage1-coreos.aci
    traefik.aci
    zookeeper.aci

### Checks

Be sure by running:

make check_versions

openstack_kubernetes

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README