Kolla

简介

Kolla是OpenStack Big Tent Governace下的一个项目，项目的目标是

To provide production-ready containers and deployment tools for operating
OpenStack clouds.

Kolla使用Docker容器和Anisble playbooks来实现这个目标。Kolla是开箱即用的，即使你是个新手也可以很快的使用kolla快速部署你的openstack集群。Kolla也允许你根据实际的需求来定制化的部署。

kolla目前已经可以部署以下openstack项目

可以部署的基础组件包括

Kolla体验

可以参照kolla官方文档https://github.com/openstack/kolla/blob/master/doc/quickstart.rst 进行部署。

Kolla解决的问题

可配置的灵活架构

可以看下默认的多节点架构

  1. # These initial groups are the only groups required to be modified. The
  2. # additional groups are for more control of the environment.
  3. [control]
  4. # These hostname must be resolvable from your deployment host
  5. control01
  6. control02
  7. control03
  9. # The above can also be specified as follows:
  10. #control[01:03]     ansible_ssh_user=kolla
  12. # The network nodes are where your l3-agent and loadbalancers will run
  13. # This can be the same as a host in the control group
  14. [network]
  15. network01
  17. [compute]
  18. compute01
  20. # When compute nodes and control nodes use different interfaces,
  21. # you can specify "api_interface" and another interfaces like below:
  22. #compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1
  24. [storage]
  25. storage01
  27. [baremetal:children]
  28. control
  29. network
  30. compute
  31. storage
  33. # You can explicitly specify which hosts run each project by updating the
  34. # groups in the sections below. Common services are grouped together.
  35. [kibana:children]
  36. control
  38. [elasticsearch:children]
  39. control
  41. [haproxy:children]
  42. network
  44. [mariadb:children]
  45. control
  47. [rabbitmq:children]
  48. control
  50. [mongodb:children]
  51. control
  53. [keystone:children]
  54. control
  56. [glance:children]
  57. control
  59. [nova:children]
  60. control
  62. [neutron:children]
  63. network
  65. [cinder:children]
  66. control
  68. [memcached:children]
  69. control
  71. [horizon:children]
  72. control
  74. [swift:children]
  75. control
  77. [heat:children]
  78. control
  80. [murano:children]
  81. control
  83. [ironic:children]
  84. control
  86. [ceph-mon:children]
  87. control
  89. [ceph-rgw:children]
  90. control
  92. [ceph-osd:children]
  93. storage
  97. # Additional control implemented here. These groups allow you to control which
  98. # services run on which hosts at a per-service level.
  99. #
  100. # Word of caution: Some services are required to run on the same host to
  101. # function appropriately. For example, neutron-metadata-agent must run on the
  102. # same host as the l3-agent and (depending on configuration) the dhcp-agent.
  104. # Glance
  105. [glance-api:children]
  106. glance
  108. [glance-registry:children]
  109. glance
  111. # Nova
  112. [nova-api:children]
  113. nova
  115. [nova-conductor:children]
  116. nova
  118. [nova-consoleauth:children]
  119. nova
  121. [nova-novncproxy:children]
  122. nova
  124. [nova-scheduler:children]
  125. nova
  127. [nova-spicehtml5proxy:children]
  128. nova
  130. [nova-compute-ironic:children]
  131. nova
  133. # Neutron
  134. [neutron-server:children]
  135. control
  137. [neutron-dhcp-agent:children]
  138. neutron
  140. [neutron-l3-agent:children]
  141. neutron
  143. [neutron-lbaas-agent:children]
  144. neutron
  146. [neutron-metadata-agent:children]
  147. neutron

默认我们会把haproxy放到network节点，如果我想把haproxy放到一个单独的节点，那么我只需要到这样修改

  1. -[haproxy:children]
  2. -network
  3. +[haproxy]
  4. +haproxy01
  5. +haproxy02

配置文件管理

每个openstack服务都运行在一个容器中，那kolla是怎么管理openstack的配置的呢? 我们拿nova-compute的配置管理来举例

首先kolla会使用ansible为nova-compute生成一份配置文件放在/etc/kolla/nova-compute/目录下。

  1. #nova_custom_config默认是/etc/kolla/configs/nova
  2. #node_config_directory默认是 /etc/kolla
  3. - name: Copying over nova.conf
  4.   merge_configs:
  5.     vars:
  6.       service_name: "{{ item }}"
  7.     sources:
  8.       - "{{ role_path }}/templates/nova.conf.j2"
  9.       - "{{ node_custom_config }}/global.conf"
  10.       - "{{ node_custom_config }}/database.conf"
  11.       - "{{ node_custom_config }}/messaging.conf"
  12.       - "{{ node_custom_config }}/nova.conf"
  13.       - "{{ node_custom_config }}/nova/{{ item }}.conf"
  14.       - "{{ node_custom_config }}/nova/{{ inventory_hostname }}/nova.conf"
  15.     dest: "{{ node_config_directory }}/{{ item }}/nova.conf"
  16.   with_items:
  17.     - "nova-api"
  18.     - "nova-compute"
  19.     - "nova-compute-ironic"
  20.     - "nova-conductor"
  21.     - "nova-consoleauth"
  22.     - "nova-novncproxy"
  23.     - "nova-scheduler"
  24.     - "nova-spicehtml5proxy"

大家可能会注意到kolla使用merge_configs来完成配置文件的合并，那么merge_configs是干什么的呢?顾名思义，merge_configs就是把多个配置文件合成一个,kolla为什么要这样做呢？
openstack配置选项非常多但是真正需要管理的则很少，对这部分选项kolla使用模版的方式管理，同时由于merge_configs的使用，使得用户可以非常方便的添加自己的定制化选项。比如你部署kolla在一台虚拟机上，你必须使用QEMU hypervisor来替代KVM hypervisor。那么你可以在/etc/kolla/config/nova/nova-compute.conf中添加以下配置

  1. [libvirt]
  2. virt_type=qemu

merge_configs的代码在 ansible/action_plugins/merge_configs.py

启动容器时/etc/kolla以docker卷的形式挂载到/var/lib/kolla/config_files目录下

  1. - name: Starting nova-libvirt container
  2.   kolla_docker:
  3.     action: "start_container"
  4.     common_options: "{{ docker_common_options }}"
  5.     image: "{{ nova_libvirt_image_full }}"
  6.     name: "nova_libvirt"
  7.     pid_mode: "host"
  8.     privileged: True
  9.     volumes:
  10.       - "{{ node_config_directory }}/nova-libvirt/:{{ container_config_directory }}/:ro"
  11.       - "/etc/localtime:/etc/localtime:ro"
  12.       - "/lib/modules:/lib/modules:ro"
  13.       - "/run/:/run/"
  14.       - "/dev:/dev"
  15.       - "/sys/fs/cgroup:/sys/fs/cgroup"
  16.       - "kolla_logs:/var/log/kolla/"
  17.       - "libvirtd:/var/lib/libvirt"
  18.       - "nova_compute:/var/lib/nova/"
  19.       - "nova_libvirt_qemu:/etc/libvirt/qemu"
  20.   when: inventory_hostname in groups['compute']

容器启动脚本会根据nova-compute.json来将配置文件拷贝到/etc并设置合适的权限

  1. {
  2.     "command": "nova-compute",
  3.     "config_files": [
  4.         {
  5.             "source": "{{ container_config_directory }}/nova.conf",
  6.             "dest": "/etc/nova/nova.conf",
  7.             "owner": "nova",
  8.             "perm": "0600"
  9.         }{% if nova_backend == "rbd" %},
  10.         {
  11.             "source": "{{ container_config_directory }}/ceph.*",
  12.             "dest": "/etc/ceph/",
  13.             "owner": "nova",
  14.             "perm": "0700"
  15.         }{% endif %}
  16.     ]
  17. }

关于kolla配置文件的管理还可以参考这里

nova-fake测试控制平台性能

这里

compute节点升级问题

由于所有服务都运行在容器中，那么是不是我升级compute节点时，该节点的虚机都会进入关机状态呢，kolla使用super-privilege的容器来解决了这个问题具体可以参考kolla PTL的文章https://sdake.io/2015/01/28/an-atomic-upgrade-process-for-openstack-compute-nodes/

平滑升级

kolla为升级也编写了upgrade.yaml这个playbook,我们还是拿nova-compute的升级为例

  1. # kolla/ansible/roles/nova/tasks/upgrade.yml
  2. ---
  3. # Create new set of configs on nodes
  4. - include: config.yml
  6. # TODO(inc0): since nova is creating new database in L->M, we need to call it.
  7. # It should be removed later
  8. - include: bootstrap.yml
  10. - include: bootstrap_service.yml
  12. - name: Checking if conductor container needs upgrading
  13.   kolla_docker:
  14.     action: "compare_image"
  15.     common_options: "{{ docker_common_options }}"
  16.     name: "nova_conductor"
  17.     image: "{{ nova_conductor_image_full }}"
  18.   when: inventory_hostname in groups['nova-conductor']
  19.   register: conductor_differs
  21. # Short downtime here, but from user perspective his call will just timeout or execute later
  22. - name: Stopping all nova_conductor containers
  23.   kolla_docker:
  24.     action: "stop_container"
  25.     common_options: "{{ docker_common_options }}"
  26.     name: "nova_conductor"
  27.   when:
  28.     - inventory_hostname in groups['nova-conductor']
  29.     - conductor_differs['result']
  31. - include: start_conductors.yml
  33. - include: start_controllers.yml
  34.   serial: "30%"
  36. - include: start_compute.yml
  37.   serial: "10%"
  39. - include: reload.yml
  40.   serial: "30%"

使用

查看log

  1. cd /var/lib/docker/volumes/kolla_logs/

进入容器调试

  1. docker exec -it service_name  bash

root权限问题

出于安全考虑很多kolla服务都是运行在非root下，进入容器后拿不到root权限，我们还以nova_compute为例,可以修改/etc/kolla/nova_compute/config.json改为以下

  1. {
  2.     "command": "nova-compute",
  3.     "config_files": [
  4.         {
  5.             "source": "/var/lib/kolla/config_files/nova.conf",
  6.             "dest": "/etc/nova/nova.conf",
  7.             "owner": "nova",
  8.             "perm": "0600"
  9.         },
  10.         {
  11.             "source": "/var/lib/kolla/config_files/nova.sudo",
  12.             "dest": "/etc/sudoers.d/nova.sudo",
  13.             "owner": "root",
  14.         }    ]
  15. }

然后在/etc/kolla/nova-compute添加nova.sudo

  1. nova       ALL=(ALL)       NOPASSWD: ALL

重启容器后即可sudo到root用户下调试

定制化build镜像

参考 https://github.com/openstack/kolla/blob/master/doc/image-building.rst

总结

优点

  • 配置管理灵活方便
  • 可以平滑升级
  • 部署简单
  • 环境隔离
  • 多种安装源
  • 支持的部署的服务多

缺点

  • 对新手的友好程度
  • debug不方便