*** jascott1 has quit IRC | 00:05 | |
*** yee37922 has joined #openstack-kolla | 00:09 | |
*** openstack has joined #openstack-kolla | 00:18 | |
*** ducttape_ has quit IRC | 00:19 | |
*** sambetts_ has joined #openstack-kolla | 00:19 | |
*** cmceniry has joined #openstack-kolla | 00:20 | |
*** igordcard has joined #openstack-kolla | 00:20 | |
*** PramodJ has quit IRC | 00:22 | |
*** strigazi has joined #openstack-kolla | 00:24 | |
*** yingjun has joined #openstack-kolla | 00:32 | |
*** manheim has joined #openstack-kolla | 00:34 | |
*** yuanying has quit IRC | 00:36 | |
*** manheim has quit IRC | 00:39 | |
*** jaosorior has joined #openstack-kolla | 00:40 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CEILOMETER CHANGES https://review.openstack.org/475452 | 00:42 |
---|---|---|
*** rwsu has quit IRC | 00:48 | |
*** srwilkers_ has joined #openstack-kolla | 00:48 | |
*** srwilkers_ has quit IRC | 00:49 | |
*** TxGirlGe_ has quit IRC | 00:51 | |
*** cuongnv has joined #openstack-kolla | 00:56 | |
*** tonyb_ is now known as tonyb | 00:57 | |
*** dciabrin has joined #openstack-kolla | 01:01 | |
*** caowei has joined #openstack-kolla | 01:02 | |
*** harlowja has joined #openstack-kolla | 01:04 | |
*** jascott1 has joined #openstack-kolla | 01:05 | |
*** jtriley has joined #openstack-kolla | 01:09 | |
*** tovin07_ has joined #openstack-kolla | 01:12 | |
*** rwsu has joined #openstack-kolla | 01:12 | |
*** caowei_ has joined #openstack-kolla | 01:13 | |
*** caowei has quit IRC | 01:16 | |
*** caowei_ is now known as caowei | 01:16 | |
*** magicboiz has quit IRC | 01:16 | |
*** ducttape_ has joined #openstack-kolla | 01:16 | |
*** yangyapeng has joined #openstack-kolla | 01:20 | |
openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Fix the database user name for all roles https://review.openstack.org/475258 | 01:26 |
*** ducttape_ has quit IRC | 01:26 | |
*** ducttap__ has joined #openstack-kolla | 01:31 | |
*** xinliang has quit IRC | 01:31 | |
*** xinliang has joined #openstack-kolla | 01:34 | |
*** xinliang has quit IRC | 01:34 | |
*** xinliang has joined #openstack-kolla | 01:34 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CEILOMETER CHANGES https://review.openstack.org/475452 | 01:34 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Revert "Fix a typo and complement doc in ceph-guide.rst" https://review.openstack.org/475611 | 01:36 |
*** ducttap__ has quit IRC | 01:36 | |
*** xinliang_ has joined #openstack-kolla | 01:44 | |
*** yangyape_ has joined #openstack-kolla | 01:48 | |
*** yangyapeng has quit IRC | 01:52 | |
*** yangyape_ has quit IRC | 01:56 | |
*** yangyapeng has joined #openstack-kolla | 01:56 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CEILOMETER CHANGES https://review.openstack.org/475452 | 02:01 |
*** xinliang has quit IRC | 02:02 | |
openstackgerrit | Li Yingjun proposed openstack/kolla-ansible master: Add option ha_vrrp_health_check_interval for l3 https://review.openstack.org/475614 | 02:07 |
*** eanylin_ has quit IRC | 02:11 | |
openstackgerrit | Li Yingjun proposed openstack/kolla-ansible master: Add option ha_vrrp_health_check_interval for l3 https://review.openstack.org/475614 | 02:12 |
*** unicell has quit IRC | 02:18 | |
openstackgerrit | Li Yingjun proposed openstack/kolla master: ping is needed for ha router https://review.openstack.org/475617 | 02:21 |
*** cuongnv_ has joined #openstack-kolla | 02:21 | |
*** cuongnv has quit IRC | 02:24 | |
*** Guest8722 is now known as sdake | 02:29 | |
*** sdake has joined #openstack-kolla | 02:29 | |
*** ducttape_ has joined #openstack-kolla | 02:32 | |
*** masber has joined #openstack-kolla | 02:34 | |
*** yuanying has joined #openstack-kolla | 02:35 | |
*** masuberu has quit IRC | 02:37 | |
*** masuberu has joined #openstack-kolla | 02:38 | |
*** masber has quit IRC | 02:42 | |
*** ducttape_ has quit IRC | 02:46 | |
*** zhugaoxiao has joined #openstack-kolla | 02:51 | |
*** cuongnv_ has quit IRC | 02:57 | |
*** yuanying has quit IRC | 03:04 | |
*** yuanying has joined #openstack-kolla | 03:04 | |
openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add webconsole support in ironic https://review.openstack.org/475628 | 03:05 |
*** spsurya has joined #openstack-kolla | 03:05 | |
spsurya | morning | 03:06 |
*** cuongnv has joined #openstack-kolla | 03:08 | |
*** yuanying has quit IRC | 03:09 | |
*** kbaegis has quit IRC | 03:17 | |
openstackgerrit | caoyuan proposed openstack/kolla master: Update min Ansible version to 2.2.0 https://review.openstack.org/452529 | 03:20 |
*** eanylin_ has joined #openstack-kolla | 03:39 | |
*** ducttape_ has joined #openstack-kolla | 03:43 | |
*** jamesbenson has joined #openstack-kolla | 03:45 | |
*** jamesbenson has quit IRC | 03:49 | |
*** ducttap__ has joined #openstack-kolla | 04:00 | |
*** ducttape_ has quit IRC | 04:00 | |
*** ducttap__ has quit IRC | 04:05 | |
*** kbaegis has joined #openstack-kolla | 04:07 | |
*** ducttape_ has joined #openstack-kolla | 04:08 | |
*** ducttape_ has quit IRC | 04:13 | |
*** ducttape_ has joined #openstack-kolla | 04:16 | |
*** kbaegis1 has joined #openstack-kolla | 04:19 | |
*** kbaegis has quit IRC | 04:19 | |
*** caowei has quit IRC | 04:21 | |
*** ducttape_ has quit IRC | 04:21 | |
*** ducttape_ has joined #openstack-kolla | 04:21 | |
*** ducttape_ has quit IRC | 04:26 | |
*** jamesbenson has joined #openstack-kolla | 04:26 | |
*** jamesbenson has quit IRC | 04:31 | |
*** kbaegis1 has quit IRC | 04:35 | |
*** ducttape_ has joined #openstack-kolla | 04:39 | |
*** zhubingbing_ has joined #openstack-kolla | 04:42 | |
zhubingbing_ | Hi guys | 04:43 |
*** ducttape_ has quit IRC | 04:43 | |
*** salv-orlando has joined #openstack-kolla | 04:48 | |
*** salv-orlando has quit IRC | 04:49 | |
*** salv-orlando has joined #openstack-kolla | 04:49 | |
spsurya | zhubingbing_: sup | 04:50 |
*** salv-orlando has quit IRC | 04:51 | |
*** salv-orlando has joined #openstack-kolla | 04:51 | |
*** janki has joined #openstack-kolla | 04:53 | |
*** zhubingbing_ has quit IRC | 04:57 | |
*** rstarmer has quit IRC | 04:58 | |
*** unicell has joined #openstack-kolla | 05:06 | |
*** jamesbenson has joined #openstack-kolla | 05:08 | |
*** jtriley has quit IRC | 05:10 | |
*** yuanying has joined #openstack-kolla | 05:11 | |
*** jamesbenson has quit IRC | 05:12 | |
*** pcaruana has joined #openstack-kolla | 05:14 | |
*** salv-orl_ has joined #openstack-kolla | 05:15 | |
*** salv-orlando has quit IRC | 05:18 | |
*** skramaja has joined #openstack-kolla | 05:21 | |
*** pcaruana has quit IRC | 05:30 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CEILOMETER CHANGES https://review.openstack.org/475452 | 05:34 |
*** zhubingbing_ has joined #openstack-kolla | 05:35 | |
zhubingbing_ | Hi spsurya | 05:35 |
spsurya | zhubingbing_: trying to help caoyuan into https://review.openstack.org/#/c/464673/ | 05:36 |
spsurya | wanted to find the changes related to celiometer | 05:37 |
spsurya | w.r.t event_connection | 05:37 |
spsurya | did you see anywhere ? | 05:37 |
zhubingbing_ | I. Don 't agree | 05:37 |
zhubingbing_ | I have see event_connection in ceilomete.conf | 05:38 |
spsurya | me too | 05:38 |
spsurya | I was looking Into master just in case | 05:39 |
zhubingbing_ | If remove ,kilometer maybe not working | 05:44 |
zhubingbing_ | ceilometer | 05:44 |
zhubingbing_ | https://github.com/openstack/ceilometer/blob/master/ceilometer/dispatcher/__init__.py#L38 | 05:44 |
spsurya | agree | 05:47 |
*** jamesbenson has joined #openstack-kolla | 05:49 | |
*** ducttape_ has joined #openstack-kolla | 05:53 | |
*** jamesbenson has quit IRC | 05:53 | |
masuberu | good afternoon all, | 06:11 |
*** iniazi_ has quit IRC | 06:12 | |
*** iniazi has joined #openstack-kolla | 06:12 | |
*** manheim has joined #openstack-kolla | 06:15 | |
*** magicboiz has joined #openstack-kolla | 06:22 | |
*** unicell1 has joined #openstack-kolla | 06:23 | |
*** manheim has quit IRC | 06:23 | |
*** unicell has quit IRC | 06:24 | |
masuberu | I am running an openstack kolla-ansible environment for a few weeks now. Just realized that the compute node is running our of memory, the reason is elastic search is taking all the memory, is this a knows issue? | 06:26 |
*** caowei has joined #openstack-kolla | 06:26 | |
*** magicboiz has quit IRC | 06:27 | |
*** jamesbenson has joined #openstack-kolla | 06:30 | |
*** magicboiz has joined #openstack-kolla | 06:33 | |
*** jamesbenson has quit IRC | 06:35 | |
*** pcaruana has joined #openstack-kolla | 06:49 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Remove ceilometer-api and ceilometer-collector services https://review.openstack.org/470752 | 06:53 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CLEAN CEILOMETER https://review.openstack.org/475670 | 06:56 |
*** jamesbenson has joined #openstack-kolla | 06:59 | |
*** salv-orl_ has quit IRC | 07:01 | |
*** salv-orlando has joined #openstack-kolla | 07:02 | |
*** jamesbenson has quit IRC | 07:04 | |
*** salv-orlando has quit IRC | 07:07 | |
*** salv-orlando has joined #openstack-kolla | 07:09 | |
*** manheim has joined #openstack-kolla | 07:11 | |
*** rmart04 has joined #openstack-kolla | 07:15 | |
*** manheim has quit IRC | 07:17 | |
*** manheim has joined #openstack-kolla | 07:17 | |
*** huzhengchuan has joined #openstack-kolla | 07:29 | |
*** dave-mccowan has quit IRC | 07:30 | |
*** sbezverk has quit IRC | 07:31 | |
*** manheim has quit IRC | 07:32 | |
*** pcaruana|afk| has joined #openstack-kolla | 07:35 | |
*** pcaruana|afk| has quit IRC | 07:35 | |
*** pcaruana has quit IRC | 07:36 | |
*** dave-mccowan has joined #openstack-kolla | 07:36 | |
spsurya | masuberu: yes there a known problem with elasticseach https://github.com/elastic/elasticsearch/issues/12208 | 07:39 |
spsurya | issue is still in new state | 07:40 |
spsurya | not so sure whether your problem related to this or not | 07:40 |
spsurya | logs may help into that | 07:40 |
*** pcaruana has joined #openstack-kolla | 07:40 | |
*** reidrac has joined #openstack-kolla | 07:42 | |
*** egonzalez has joined #openstack-kolla | 07:43 | |
openstackgerrit | Li Yingjun proposed openstack/kolla master: ping is needed for ha router https://review.openstack.org/475617 | 07:47 |
*** tovin07_ has quit IRC | 07:54 | |
spsurya | masuberu: running openstack with kolla-asible from 4 weeks with no issue on memory | 07:58 |
spsurya | https://www.irccloud.com/pastebin/1ZJEODga/ | 07:59 |
spsurya | for your reference | 08:00 |
openstackgerrit | zhubingbing proposed openstack/kolla master: Alphabetical order install item https://review.openstack.org/475205 | 08:00 |
openstackgerrit | Mathias Ewald proposed openstack/kolla master: Fix apache configuration for ceilometer_api https://review.openstack.org/475694 | 08:03 |
*** mewald has joined #openstack-kolla | 08:03 | |
*** manheim has joined #openstack-kolla | 08:06 | |
*** manheim has quit IRC | 08:08 | |
*** manheim has joined #openstack-kolla | 08:09 | |
*** Administrator_ has joined #openstack-kolla | 08:17 | |
*** zhubingbing_ has quit IRC | 08:18 | |
*** serlex has joined #openstack-kolla | 08:20 | |
*** zhugaoxiao has quit IRC | 08:20 | |
*** mgoddard has joined #openstack-kolla | 08:20 | |
*** cuongnv has quit IRC | 08:21 | |
openstackgerrit | Li Yingjun proposed openstack/kolla-ansible master: Add option ha_vrrp_health_check_interval for l3 https://review.openstack.org/475614 | 08:24 |
openstackgerrit | Li Yingjun proposed openstack/kolla master: ping is needed for ha router https://review.openstack.org/475617 | 08:27 |
*** openstackgerrit has quit IRC | 08:48 | |
reidrac | anyone familiar with on_keystone_name kolla ansible module? | 08:51 |
reidrac | oh, sorry; I meant "on_keystone_role" | 08:52 |
*** openstackgerrit has joined #openstack-kolla | 08:53 | |
openstackgerrit | Li Yingjun proposed openstack/kolla-ansible master: Add option ha_vrrp_health_check_interval for l3 https://review.openstack.org/475614 | 08:53 |
*** tvignaud has quit IRC | 08:55 | |
egonzalez | reidrac, with this https://docs.ansible.com/ansible/os_keystone_role_module.html ? | 08:55 |
reidrac | right; thanks egonzalez! | 08:55 |
reidrac | it is failing here randomly; I found that adding a short sleep before it fixes the issue | 08:56 |
reidrac | when deploying keysytone | 08:56 |
*** zhubingbing_ has joined #openstack-kolla | 08:57 | |
*** karimb has joined #openstack-kolla | 08:58 | |
reidrac | OK, so that ansible module is used to wait until the user role is created | 08:59 |
reidrac | I'll make a bug report, it may be nothing, but can't find what's going on | 09:00 |
egonzalez | probably some kind of network issue | 09:00 |
reidrac | egonzalez: in an all-in-one deployment on vagrant? | 09:00 |
egonzalez | we had retries before but we removed it, maybe we should recover them https://github.com/openstack/kolla-ansible/commit/38ad05a8729c00ccc012f234135885907065035d | 09:00 |
reidrac | let me reproduce it so I can tell you the exact error | 09:02 |
*** gfidente has joined #openstack-kolla | 09:04 | |
*** gfidente has quit IRC | 09:04 | |
*** gfidente has joined #openstack-kolla | 09:04 | |
*** n0isyn0i1e is now known as n0isyn0ise | 09:05 | |
*** tvignaud has joined #openstack-kolla | 09:08 | |
*** salv-orlando has quit IRC | 09:09 | |
*** salv-orlando has joined #openstack-kolla | 09:09 | |
*** magicboiz has quit IRC | 09:09 | |
reidrac | egonzalez: OK, it fails onn os_keystone_role with "Could not determine a suitable URL for the plugin"; and if I re-run it, it will work fine (so I tried the sleep after the bootstrap) | 09:11 |
reidrac | this is all-in-one, with Vagrant and libvirt | 09:11 |
*** salv-orlando has quit IRC | 09:13 | |
openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add webconsole support in ironic https://review.openstack.org/475628 | 09:18 |
*** sambetts_ is now known as sambetts | 09:19 | |
reidrac | egonzalez: any suggestion would be appreciated; I can open a bug report, but I'd like to investigate a bit more first | 09:27 |
openstackgerrit | Mathias Ewald proposed openstack/kolla master: Fix apache configuration for ceilometer_api https://review.openstack.org/475694 | 09:28 |
egonzalez | reidrac, I'd try reverting the commit I posted before, I think that have sth to do. Anyway, verify haproxy is working fine and is not any issue with balancing to non working nodes | 09:32 |
egonzalez | pbourke, around? | 09:32 |
reidrac | egonzalez: good stuff; I'll check both things, thanks! | 09:33 |
*** openstackgerrit has quit IRC | 09:33 | |
*** yingjun has quit IRC | 09:38 | |
*** shardy has joined #openstack-kolla | 09:39 | |
*** hrw has quit IRC | 09:42 | |
reidrac | egonzalez: haproxy reports on active server and 0 backups servers; so looks like that's OK; I'll try reverting the patch | 09:44 |
*** jascott1 has quit IRC | 09:45 | |
*** jascott1 has joined #openstack-kolla | 09:46 | |
reidrac | oh, wait... keystone logs creating def role @ 11:41:55.833 and haproxy repotrs keystone servers up @ 11:41:57 | 09:47 |
reidrac | on_keystone_role runs just after the default role has been created, so that's the problem | 09:48 |
*** jascott1 has quit IRC | 09:51 | |
reidrac | yes, the retries fix the problem; also name of the task "Creating default user role" is somewhat misleading as is not actually doing that but checking that the role was created | 10:13 |
*** dims has quit IRC | 10:16 | |
*** yangyapeng has quit IRC | 10:16 | |
*** yangyapeng has joined #openstack-kolla | 10:17 | |
*** pbourke has quit IRC | 10:17 | |
egonzalez | reidrac, the bootstrap creates admin role, Creating default user role: creates _member_ role | 10:18 |
reidrac | OK; thanks | 10:18 |
reidrac | but is uses the API instead of SQL directly like keystone-manage, isn't it? | 10:18 |
reidrac | that's why that one fails | 10:18 |
egonzalez | reidrac, yep | 10:19 |
*** pbourke has joined #openstack-kolla | 10:19 | |
egonzalez | is failing because ansible goes faster that keystone and haproxy to be marked as UP | 10:19 |
reidrac | so may be the retries aren't really required all over the place but in this one (at least0 | 10:19 |
reidrac | ) | 10:19 |
egonzalez | maybe a wait for the port to be ready | 10:19 |
reidrac | OK | 10:20 |
*** dims has joined #openstack-kolla | 10:21 | |
*** yangyapeng has quit IRC | 10:21 | |
*** myoung has quit IRC | 10:21 | |
*** yangyapeng has joined #openstack-kolla | 10:22 | |
*** myoung has joined #openstack-kolla | 10:23 | |
*** yangyapeng has quit IRC | 10:27 | |
manheim | hi guys | 10:32 |
manheim | the presentation is up | 10:32 |
manheim | https://www.youtube.com/watch?v=dujGk5FML14 | 10:32 |
reidrac | filed a bug: https://bugs.launchpad.net/kolla-ansible/+bug/1699096 | 10:32 |
openstack | Launchpad bug 1699096 in kolla-ansible "Race condition between haproxy and keystone register task" [Undecided,New] | 10:32 |
manheim | enjoy and let me know what can I improve | 10:32 |
*** rhallisey has quit IRC | 10:33 | |
reidrac | I'll work on a fix now | 10:33 |
*** caowei_ has joined #openstack-kolla | 10:36 | |
*** caowei has quit IRC | 10:36 | |
*** caowei_ is now known as caowei | 10:36 | |
*** ansmith has quit IRC | 10:38 | |
*** rstarmer has joined #openstack-kolla | 10:45 | |
*** jascott1 has joined #openstack-kolla | 10:46 | |
*** karimb has quit IRC | 10:50 | |
*** jascott1 has quit IRC | 10:51 | |
*** karimb has joined #openstack-kolla | 10:51 | |
*** ansmith has joined #openstack-kolla | 10:51 | |
*** mewald has left #openstack-kolla | 11:01 | |
*** janki has quit IRC | 11:06 | |
*** janki has joined #openstack-kolla | 11:07 | |
*** ansmith has quit IRC | 11:11 | |
*** jchhatbar has joined #openstack-kolla | 11:16 | |
*** janki has quit IRC | 11:19 | |
*** jchhatbar is now known as janki | 11:26 | |
*** yangyapeng has joined #openstack-kolla | 11:28 | |
*** schwicht has joined #openstack-kolla | 11:28 | |
*** karimb has quit IRC | 11:31 | |
*** rhallisey has joined #openstack-kolla | 11:40 | |
*** schwicht has quit IRC | 11:42 | |
serlex | Hi, I see 4.0.2 is out. Any special steps to upgrade from 4.0.1? | 11:49 |
*** ducttape_ has quit IRC | 11:56 | |
*** ducttape_ has joined #openstack-kolla | 11:59 | |
*** openstackgerrit has joined #openstack-kolla | 12:00 | |
openstackgerrit | jianyi proposed openstack/kolla-ansible master: all.yml should be in alphabetical order https://review.openstack.org/459956 | 12:00 |
*** caowei_ has joined #openstack-kolla | 12:05 | |
*** caowei has quit IRC | 12:09 | |
*** caowei_ is now known as caowei | 12:09 | |
zhubingbing_ | Hi guys | 12:10 |
*** tumble has joined #openstack-kolla | 12:12 | |
egonzalez | sup zhubingbing_ | 12:14 |
*** caowei has quit IRC | 12:14 | |
*** caowei has joined #openstack-kolla | 12:14 | |
zhubingbing_ | hi | 12:14 |
*** ducttape_ has quit IRC | 12:15 | |
*** salv-orlando has joined #openstack-kolla | 12:19 | |
*** trozet has joined #openstack-kolla | 12:21 | |
*** yingjun has joined #openstack-kolla | 12:30 | |
openstackgerrit | Li Yingjun proposed openstack/kolla master: ping is needed for ha router https://review.openstack.org/475617 | 12:35 |
*** kbaegis has joined #openstack-kolla | 12:37 | |
*** bmace has quit IRC | 12:40 | |
*** salv-orlando has quit IRC | 12:52 | |
*** salv-orlando has joined #openstack-kolla | 12:52 | |
*** janki has quit IRC | 12:52 | |
*** bmace has joined #openstack-kolla | 12:55 | |
-openstackstatus- NOTICE: restarting gerrit to address slowdown issues | 12:56 | |
*** rwallner has joined #openstack-kolla | 12:56 | |
*** salv-orlando has quit IRC | 12:56 | |
*** rwallner has quit IRC | 12:58 | |
*** rwallner has joined #openstack-kolla | 12:58 | |
*** lucasxu has joined #openstack-kolla | 13:00 | |
*** zhubingbing_ has quit IRC | 13:02 | |
*** awebber has joined #openstack-kolla | 13:02 | |
sdake | manheim fwiw fuel is dead - ref mailing list | 13:07 |
*** jtriley has joined #openstack-kolla | 13:08 | |
*** lpetrut has joined #openstack-kolla | 13:09 | |
*** karimb has joined #openstack-kolla | 13:10 | |
manheim | yep | 13:10 |
manheim | it was just an example | 13:10 |
manheim | please let me know what can be impproved | 13:11 |
*** schwicht has joined #openstack-kolla | 13:12 | |
*** goldyfruit has joined #openstack-kolla | 13:12 | |
*** salv-orlando has joined #openstack-kolla | 13:12 | |
*** caowei_ has joined #openstack-kolla | 13:16 | |
*** eanylin_ has quit IRC | 13:18 | |
*** jtriley has quit IRC | 13:19 | |
*** caowei has quit IRC | 13:19 | |
*** caowei_ is now known as caowei | 13:19 | |
openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Fix the database user name for all roles https://review.openstack.org/475258 | 13:19 |
sdake | manheim mechanism drive exqmaple, /etc/kolla/nova/config/nova.conf | 13:21 |
sdake | is how you override rather then modifyign /etc/kolla/nofa.conf | 13:21 |
sdake | looks good tho ;) | 13:21 |
manheim | ty | 13:22 |
sdake | liberty was 1.1.0 as well - we killed 1.0.0 because it had a fatal issue with storage | 13:27 |
*** jrist_ is now known as jrist | 13:31 | |
*** jrist has quit IRC | 13:31 | |
*** jrist has joined #openstack-kolla | 13:31 | |
*** caowei has quit IRC | 13:33 | |
*** ducttape_ has joined #openstack-kolla | 13:33 | |
*** yingjun has quit IRC | 13:34 | |
*** eanylin_ has joined #openstack-kolla | 13:35 | |
*** yingjun has joined #openstack-kolla | 13:38 | |
*** ducttape_ has quit IRC | 13:38 | |
*** kbaegis has quit IRC | 13:39 | |
*** kbaegis has joined #openstack-kolla | 13:39 | |
*** wahnsinn has joined #openstack-kolla | 13:41 | |
*** kbaegis1 has joined #openstack-kolla | 13:41 | |
openstackgerrit | Dave McCowan proposed openstack/kolla-kubernetes master: [WIP] Add support for HTTPS on external API network https://review.openstack.org/471992 | 13:42 |
*** kbaegis1 has quit IRC | 13:44 | |
*** kbaegis has quit IRC | 13:45 | |
*** kbaegis has joined #openstack-kolla | 13:46 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add warning while upgrading with dockerhub images N->O https://review.openstack.org/475802 | 13:49 |
dmsimard | sdake: do you guys have a registry inside infra yet ? | 13:50 |
sdake | dmsimard no | 13:50 |
dmsimard | :( | 13:50 |
dmsimard | I stood up an openshift registry for RDO recently, it was a wild ride | 13:51 |
*** spsurya has quit IRC | 13:55 | |
rwellum | hey sdake - I'm attending kubernetes training hosted by Cisco today. :) | 13:55 |
sdake | rwellum fascinating | 13:56 |
sdake | who is providing it | 13:56 |
rwellum | Apprenda + cisco | 13:56 |
*** karimb has quit IRC | 13:58 | |
*** skramaja has quit IRC | 13:59 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add warning while upgrading with dockerhub images N->O https://review.openstack.org/475802 | 14:00 |
reidrac | egonzalez: wiating for the port doesn't work; haproxy is always bound, but the backend isn't available, I'll go with retry unless you have a better idea | 14:00 |
egonzalez | reidrac, retry is fine | 14:00 |
reidrac | egonzalez: cheers! | 14:01 |
*** Administrator_ has quit IRC | 14:01 | |
*** Administrator_ has joined #openstack-kolla | 14:02 | |
*** srwilkers_ has joined #openstack-kolla | 14:02 | |
*** mickt has joined #openstack-kolla | 14:09 | |
*** mickt has left #openstack-kolla | 14:09 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add warning while upgrading with dockerhub images N->O https://review.openstack.org/475802 | 14:10 |
*** karimb has joined #openstack-kolla | 14:10 | |
*** magicboiz has joined #openstack-kolla | 14:12 | |
*** tumble has quit IRC | 14:12 | |
wahnsinn | hey, I have problems pushing images to a private registry: "ERROR:kolla.image.build.base:open /dev/mapper/docker-254:3-3605282-0d1f95af4ae76c1c535e296b64838d7a58a7700f980f7978542085cf6cb0a52a: no such file or directory" | 14:12 |
wahnsinn | neither gitlab registry nor a registry deployed with the start-registry script is working | 14:13 |
wahnsinn | does somebody have an idea where I should look for the problem? | 14:14 |
sdake | rwellum not familiar with apprenda | 14:14 |
wahnsinn | the weird thing is that some images were pushed sometimes | 14:14 |
sdake | wahnsinn my first recommendation would be to not use devicemapper | 14:15 |
sdake | docker themselves says "not supported, deprecated" | 14:15 |
sdake | wahnsinn my second recommdnation would be to hceck disk space | 14:16 |
*** magicboiz has quit IRC | 14:16 | |
sdake | even though your system may say 30-40% free, devicemapper actually uses more of the storage than df reports, resulting in all sorts of errors | 14:16 |
*** huzhengchuan has quit IRC | 14:17 | |
wahnsinn | thanks, I'll have a look | 14:17 |
rwellum | sdake: they and cisco have a long running partnership (apparently) | 14:18 |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add warning while upgrading with dockerhub images N->O https://review.openstack.org/475802 | 14:20 |
*** magicboiz has joined #openstack-kolla | 14:21 | |
*** salv-orlando has quit IRC | 14:24 | |
*** tumble has joined #openstack-kolla | 14:24 | |
openstackgerrit | Juan J. MartÃnez proposed openstack/kolla-ansible master: Retry Keystone's default user role creation https://review.openstack.org/475820 | 14:25 |
egonzalez | sdake, mind taking a look at https://review.openstack.org/#/c/475802/ | 14:26 |
egonzalez | quite important warning imo, here is the rendered release note http://docs-draft.openstack.org/02/475802/4/check/gate-kolla-ansible-releasenotes/49c598c//releasenotes/build/html/unreleased.html#id1 | 14:26 |
*** ducttape_ has joined #openstack-kolla | 14:27 | |
*** salv-orlando has joined #openstack-kolla | 14:28 | |
egonzalez | sdake, note is a change directly to stable/ocata release notes | 14:28 |
*** salv-orlando has quit IRC | 14:30 | |
*** yangyape_ has joined #openstack-kolla | 14:30 | |
*** salv-orlando has joined #openstack-kolla | 14:30 | |
*** yangyapeng has quit IRC | 14:33 | |
*** jmccarthy has joined #openstack-kolla | 14:37 | |
jmccarthy | Just checking, any cores around who can bump these along ? | 14:38 |
jmccarthy | https://review.openstack.org/#/c/469511Add possibility to configure tenant network types and type drivers | 14:38 |
jmccarthy | https://review.openstack.org/#/c/464675Add possibility to configure tenant and provider VLANs | 14:38 |
openstackgerrit | caoyuan proposed openstack/kolla master: Update the "prechecks" to "precheck" https://review.openstack.org/475827 | 14:38 |
*** ducttape_ has quit IRC | 14:39 | |
*** ducttape_ has joined #openstack-kolla | 14:39 | |
*** dciabrin has quit IRC | 14:40 | |
*** yingjun has quit IRC | 14:40 | |
openstackgerrit | caoyuan proposed openstack/kolla master: Update the "prechecks1" to "prechecks" https://review.openstack.org/475827 | 14:40 |
*** jamesbenson has joined #openstack-kolla | 14:46 | |
*** tovin07 has joined #openstack-kolla | 14:52 | |
*** dciabrin has joined #openstack-kolla | 14:54 | |
*** rhallisey is now known as rhallisey-brb | 14:57 | |
*** rhallisey-brb is now known as rhallisey-away | 14:57 | |
*** spsurya has joined #openstack-kolla | 14:57 | |
jamesbenson | morning all | 14:58 |
manheim | hi | 14:58 |
spsurya | morning jamesbenson manheim | 15:04 |
*** hrw has joined #openstack-kolla | 15:04 | |
spsurya | hrw sup | 15:05 |
jamesbenson | morning spsurya | 15:05 |
*** sbezverk has joined #openstack-kolla | 15:08 | |
*** rwallner has quit IRC | 15:10 | |
*** rmart04 has quit IRC | 15:14 | |
*** ducttape_ has quit IRC | 15:20 | |
*** schwicht has quit IRC | 15:21 | |
*** ducttape_ has joined #openstack-kolla | 15:21 | |
*** rstarmer has quit IRC | 15:22 | |
*** rstarmer has joined #openstack-kolla | 15:23 | |
*** rwallner has joined #openstack-kolla | 15:25 | |
inc0 | good morning everyone | 15:30 |
inc0 | egonzalez: hey, https://review.openstack.org/#/c/466815/17 I think it's ready | 15:31 |
inc0 | answering your question:) | 15:31 |
egonzalez | inc0, cool | 15:31 |
egonzalez | mind checking https://review.openstack.org/#/c/475802/ ? all gates are red because we have not merge any image with kolla 4.0.3 yet | 15:32 |
inc0 | hmm, ok yeah | 15:32 |
inc0 | we need to do it, there is no easy way to manually trigger | 15:32 |
inc0 | I'll propose fake change, merge it and rollback afterwards | 15:33 |
*** karimb has quit IRC | 15:33 | |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla stable/ocata: Fake change to kickoff image publishing https://review.openstack.org/475849 | 15:34 |
inc0 | done | 15:35 |
inc0 | I'll remove it when images are published | 15:35 |
openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Add possibility to configure tenant network types and type drivers https://review.openstack.org/469511 | 15:39 |
openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Add possibility to configure tenant network types and type drivers https://review.openstack.org/469511 | 15:39 |
*** rstarmer_ has joined #openstack-kolla | 15:40 | |
*** rstarmer has quit IRC | 15:42 | |
*** awebber has quit IRC | 15:43 | |
kfox1111 | weird... | 15:46 |
kfox1111 | k8s is moving so fast now they are releasing alpha's for 1.8 while 1.7 isn't even released. | 15:46 |
jamesbenson | inc0, are the gates why I can't build the docker images? | 15:46 |
inc0 | jamesbenson: no, building locally should just work | 15:47 |
kfox1111 | I wonder if that was a mistake... | 15:47 |
srwilkers_ | kfox1111, really? damn, thats crazy | 15:48 |
inc0 | kfox1111: yeah, I'd bet... | 15:48 |
inc0 | probably typo | 15:48 |
inc0 | or something as trivial | 15:48 |
inc0 | srwilkers_: what's up man? | 15:48 |
srwilkers_ | inc0, hey dude | 15:49 |
kfox1111 | nice. finally. https://github.com/kubernetes/kubernetes/pull/46076 node secret authorization. | 15:49 |
kfox1111 | that will help security significantly. | 15:49 |
kfox1111 | yeah. that feature is targeted for 1.7 but was in the 1.8alpha1 notes. | 15:51 |
kfox1111 | I bet the release should have been another 1.7 alpha but something slipped. | 15:51 |
*** manheim has quit IRC | 15:57 | |
*** manheim has joined #openstack-kolla | 15:57 | |
*** dciabrin has quit IRC | 16:00 | |
*** egonzalez has quit IRC | 16:01 | |
*** manheim has quit IRC | 16:02 | |
sdake | kfox1111 question - i have a cratered multlinode deployment of kubernetes - mariadb mounted via nfs | 16:03 |
kfox1111 | ok. | 16:03 |
sdake | i suspect two copies of mariadb are running even though only one copy is reported running by kubernetes | 16:04 |
jamesbenson | inc0: thanks, I figured out the issue.... | 16:04 |
sdake | is there any way to validate or invalidate this theory that you are aware of? | 16:04 |
kfox1111 | sdake: possible. | 16:04 |
sdake | i am getting an errror no 11 on read of ibdata | 16:04 |
jamesbenson | inc0: I'm building the 4.0.2 images (pip install kolla==4.0.2 kolla-ansible==4.0.2) and going to try to deploy on freshly reimaged systems. I'll let you know if I have ceph issues still... I'm hoping not.. | 16:04 |
kfox1111 | docker ps on all hosts or ps ax | grep mariadb on all hosts. | 16:04 |
sdake | ps -ax does not seem to return any results | 16:04 |
sdake | let me check out docker ps | 16:05 |
jamesbenson | Do have one question though, how come a lot of the monitoring tools are default to controller and not monitoring node? | 16:05 |
kfox1111 | ther is only one running? | 16:05 |
kfox1111 | (Not sure if mariadb on nfs is really supported or not..) | 16:05 |
inc0 | jamesbenson: our monitoring was never finished | 16:05 |
inc0 | you can try it out, but can't vouch for it | 16:05 |
*** jascott1 has joined #openstack-kolla | 16:07 | |
sdake | kfox1111 i hear ya re support - apparently docs say it works | 16:07 |
*** JamesBenson-mobi has joined #openstack-kolla | 16:07 | |
sdake | kfox1111 it is definately having a problem with FCNTL on ibdata1 | 16:08 |
kfox1111 | yeah, thats a locking primitive I think. | 16:08 |
kfox1111 | nfs can support locking but tricky to get right. | 16:08 |
kfox1111 | fyi, I think your playing with fire. | 16:09 |
jamesbenson | ah okay, what things aren't complete? | 16:11 |
*** dciabrin has joined #openstack-kolla | 16:13 | |
sdake | kfox1111 i think i am plying with TNT actually | 16:13 |
sdake | knowing that - i am proceeding ahead :) | 16:13 |
inc0 | sdake: I have something cool to show ;) later this day | 16:14 |
kfox1111 | ok. fairly warned ye be says I. :) | 16:14 |
sdake | kfox1111 ya - i'm gonna say I told ya so too... :) | 16:14 |
*** tovin07 has left #openstack-kolla | 16:15 | |
sdake | kfox1111 ya I know all about fcntl | 16:15 |
sdake | kfox1111 i'm curious if you have any other suggestions for what may be the root cause (of a system you know nohting about in detail:) | 16:15 |
kfox1111 | if I had to guess, I'd say you need to setup nfs locking on the client/server. | 16:16 |
kfox1111 | lockd I think it is called. | 16:16 |
kfox1111 | (been a long time since I've played with it) | 16:16 |
kfox1111 | or, I think it may be built into v4. | 16:17 |
*** sudipto has joined #openstack-kolla | 16:17 | |
*** sudipto_ has joined #openstack-kolla | 16:17 | |
*** pcaruana has quit IRC | 16:19 | |
*** unicell has joined #openstack-kolla | 16:21 | |
*** unicell1 has quit IRC | 16:23 | |
*** JamesBenson|phon has joined #openstack-kolla | 16:26 | |
*** JamesBenson-mobi has quit IRC | 16:26 | |
wahnsinn | sdake : thanks for the tip - it's working fine now | 16:31 |
*** unicell has quit IRC | 16:36 | |
*** schwicht has joined #openstack-kolla | 16:36 | |
*** JamesBenson-mobi has joined #openstack-kolla | 16:37 | |
*** JamesBenson|phon has quit IRC | 16:37 | |
*** JamesBenson|phon has joined #openstack-kolla | 16:38 | |
*** lpetrut has quit IRC | 16:40 | |
*** jmccarthy has left #openstack-kolla | 16:41 | |
*** JamesBenson-mobi has quit IRC | 16:42 | |
*** SamYaple has quit IRC | 16:44 | |
*** reidrac has quit IRC | 16:44 | |
*** SamYaple has joined #openstack-kolla | 16:44 | |
*** SamYaple has quit IRC | 16:44 | |
*** SamYaple has joined #openstack-kolla | 16:44 | |
*** sambetts is now known as sambetts|afk | 16:45 | |
*** gfidente is now known as gfidente|afk | 16:47 | |
*** sbezverk has quit IRC | 16:51 | |
*** itlinux has joined #openstack-kolla | 16:52 | |
*** itlinux_ has joined #openstack-kolla | 16:52 | |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: WIP Dockerfile to run ansible orchiestration https://review.openstack.org/474770 | 16:57 |
inc0 | kfox1111: sdake ^ if we do this we don't need to create configmaps manually any more | 16:59 |
*** ducttape_ has quit IRC | 17:02 | |
*** kbaegis has quit IRC | 17:05 | |
*** manheim has joined #openstack-kolla | 17:08 | |
kfox1111 | looking... | 17:09 |
*** mgoddard has quit IRC | 17:09 | |
*** manheim has quit IRC | 17:12 | |
*** manheim has joined #openstack-kolla | 17:12 | |
kfox1111 | inc0: generally +1. a bunch of comments inline. | 17:18 |
inc0 | yeah it's wip | 17:19 |
inc0 | I'm not sure about having more than one distro on container side tbh | 17:20 |
*** kbaegis has joined #openstack-kolla | 17:21 | |
*** unicell has joined #openstack-kolla | 17:25 | |
kfox1111 | your favorite's ubuntu. others are centos, etc. | 17:25 |
kfox1111 | easier adoption if you support what peples favorites are. | 17:25 |
kfox1111 | kolla already has pretty much all the plumbing. | 17:26 |
inc0 | yes, but maknig kolla built container with that is not going to be easy | 17:26 |
inc0 | and not that much value | 17:26 |
inc0 | imho | 17:26 |
inc0 | I mean it's short lived | 17:26 |
inc0 | I'd go for alpine if I would have time for figuring it out | 17:27 |
kfox1111 | why is it hard? its just a couple of statically linked go binaries and ansible? | 17:28 |
kfox1111 | plus, you could use distro packages for ansible, making it even easier. | 17:28 |
inc0 | kfox1111: putting kolla-k8s inside is harder | 17:29 |
kfox1111 | we dont' use any fancy newfangled ansible stuff. | 17:29 |
inc0 | ADD ansible | 17:29 |
inc0 | that's what's hard | 17:29 |
kfox1111 | why is that hard with centos? | 17:29 |
inc0 | it's not harder with centos | 17:29 |
inc0 | it's hard with kolla - kolla assumes Dockerfile.j2 being in Kolla | 17:29 |
inc0 | and Kolla doesn't have access to Kolla-k8s code | 17:30 |
kfox1111 | ah. | 17:30 |
kfox1111 | is that exteran repo ps far out then? | 17:30 |
inc0 | this is just plain Dockerfile without templating | 17:30 |
kfox1111 | external | 17:30 |
inc0 | dockerhub you mean? | 17:30 |
kfox1111 | no. the exteral to kolla, kolla build support. | 17:30 |
inc0 | well it's coming, but this will be lots of added complication imho | 17:31 |
inc0 | for little benefit | 17:31 |
kfox1111 | its complicated when there are more then one way to do things. | 17:31 |
kfox1111 | "build all the things" and then build this other thing. | 17:31 |
inc0 | we can do busybox/alpine and just keep it native Dockerifle | 17:31 |
inc0 | don | 17:31 |
kfox1111 | I don't think this will be the only container specific to kolla-kubernetes either. | 17:32 |
inc0 | don't forget that we'll have kolla docker container in dockerhub | 17:32 |
kfox1111 | so cant be just /Dockerfile | 17:32 |
openstackgerrit | Merged openstack/kolla stable/ocata: Fake change to kickoff image publishing https://review.openstack.org/475849 | 17:32 |
kfox1111 | I'm not forgetting. | 17:32 |
inc0 | so far it's only one having kolla-k8s code in it | 17:32 |
kfox1111 | only becase we've successfully landed some other kolla-k8s specific contaienrs in kolla. | 17:32 |
kfox1111 | this is the first one we havent tried. | 17:32 |
kfox1111 | if kolla-kubernetes is to get containers, the others probably belong here too. | 17:33 |
inc0 | yeah I agree | 17:33 |
inc0 | but then we will have harder time publishing them | 17:33 |
inc0 | since I build publisher pipeline for kolla | 17:33 |
inc0 | adding kolla-k8s is going to be...trickier | 17:33 |
kfox1111 | yeah. but I think its going to be important. | 17:33 |
inc0 | agree | 17:34 |
kfox1111 | how far out do you think that other ps is? | 17:34 |
inc0 | it's orthogonal | 17:34 |
kfox1111 | I think its maybe a prereq. depends just how far out it is though. | 17:35 |
inc0 | but well, still pretty far, not sure what bradjones has locally | 17:35 |
kfox1111 | if its a week or two, probably best to finish it first. | 17:35 |
serlex | kolla-ansible upgrade seems to be stuck on "Sighup nova-compute" anyone seen that before? | 17:35 |
inc0 | they can go together, after it merges we'll just add new job | 17:36 |
inc0 | serlex: no... | 17:36 |
kfox1111 | I guess I'd be ok with the dockerfile as is provided a techdebt bug is filed and the other ps is being worked on. | 17:36 |
inc0 | you can try to restart container | 17:36 |
serlex | ok.An hour in. Ctrl+C rerun upgrade? | 17:36 |
serlex | ok | 17:37 |
serlex | restarting container didnt do much, I will try rerunning upgrade | 17:38 |
inc0 | kk | 17:39 |
*** sudipto has quit IRC | 17:40 | |
*** sudipto_ has quit IRC | 17:40 | |
inc0 | serlex: all nodes hung? | 17:41 |
inc0 | or some succeed? | 17:41 |
*** srwilkers_ has quit IRC | 17:43 | |
serlex | It hang on "Sighup nova-compute" after completing on 4 out of the 5 nodes | 17:51 |
inc0 | anything special on 5th node? | 17:51 |
serlex | I cancelled it ran it again, finished pretty quick. should all out containers be restarted or relevant ones? | 17:52 |
serlex | 5th is just another compute node, 4th was compute node, seem to have ran with "changed" output | 17:52 |
inc0 | if it's finished it's done, restarting is built into play | 17:52 |
inc0 | we added sighub instead of restart to avoid well..restarts | 17:52 |
serlex | i see | 17:53 |
serlex | any way I can confirm successful upgrade? | 17:53 |
serlex | look through the commits i guess? | 17:53 |
inc0 | check images | 17:53 |
serlex | sure | 17:53 |
inc0 | if it's tagged with 4.0.2 then it's upgraded | 17:53 |
inc0 | docker ps should tell you that | 17:53 |
serlex | checking... | 17:57 |
*** JamesBenson|phon has quit IRC | 17:59 | |
*** JamesBenson-mobi has joined #openstack-kolla | 17:59 | |
*** athomas has quit IRC | 17:59 | |
*** ducttape_ has joined #openstack-kolla | 18:01 | |
*** wahnsinn has quit IRC | 18:02 | |
*** rwsu has quit IRC | 18:03 | |
*** JamesBenson-mobi has quit IRC | 18:05 | |
*** MasterOfBugs has joined #openstack-kolla | 18:06 | |
openstackgerrit | Andy Smith proposed openstack/kolla-ansible master: Add qdrouterd role for messaging infrastructure component https://review.openstack.org/469462 | 18:13 |
*** serlex has quit IRC | 18:13 | |
*** TxGirlGeek has joined #openstack-kolla | 18:15 | |
*** dciabrin has quit IRC | 18:16 | |
*** tonanhngo has joined #openstack-kolla | 18:24 | |
*** tonanhngo has quit IRC | 18:25 | |
*** tonanhngo has joined #openstack-kolla | 18:26 | |
*** iniazi_ has joined #openstack-kolla | 18:28 | |
*** ducttape_ has quit IRC | 18:29 | |
*** iniazi has quit IRC | 18:29 | |
*** dciabrin has joined #openstack-kolla | 18:31 | |
*** klindgren has quit IRC | 18:33 | |
*** sbezverk has joined #openstack-kolla | 18:33 | |
inc0 | kfox1111: did you try to run built helm repo on k8s? | 18:35 |
*** ducttape_ has joined #openstack-kolla | 18:36 | |
kfox1111 | a little ambigious | 18:36 |
kfox1111 | you talking about the helm repo microservice chart? | 18:36 |
inc0 | yes | 18:36 |
kfox1111 | I have not tested it. I think only sbezverk has. | 18:37 |
inc0 | use my (or similar) container to build and host repo as job | 18:37 |
inc0 | or actally even better I guess - pv | 18:37 |
kfox1111 | its a service. | 18:37 |
inc0 | kk | 18:37 |
kfox1111 | so deployment. | 18:37 |
inc0 | then a service | 18:37 |
inc0 | well yeah | 18:37 |
sbezverk | inc0: yes I have and we have chart for it and it starts in the gate workflow | 18:37 |
*** salv-orlando has quit IRC | 18:38 | |
inc0 | btw sbezverk https://review.openstack.org/#/c/474770/ besides distro, how do you like idea? | 18:38 |
*** salv-orlando has joined #openstack-kolla | 18:39 | |
sbezverk | inc0: I have huge outage to deal, I will try to find some time later today.. but currently I spend 24 hours per day on con calls | 18:40 |
inc0 | sure | 18:40 |
inc0 | good luck | 18:40 |
*** rstarmer has joined #openstack-kolla | 18:41 | |
*** pradk has joined #openstack-kolla | 18:41 | |
rstarmer | still trying with ceph, looks like there's something not right when I deploy with erasure coding, but replication works. | 18:41 |
jamesbenson | inc0: what version of python do we need for kolla? genpwd is yelling at me... | 18:42 |
inc0 | 2.7 | 18:42 |
jamesbenson | odd... | 18:42 |
inc0 | what's the yelling about? | 18:42 |
rstarmer | bigger question: my mon/osd/pools all look happy, cinder is happy, volumes "act" right, but systems keep booting against a blank volume. Anyone gotten ceph working via kolla-ansible? | 18:42 |
inc0 | rstarmer: hmm I don't know anything about erasure conding unfortunately:/ | 18:42 |
inc0 | I have | 18:43 |
inc0 | rstarmer: but boots work in general? | 18:43 |
jamesbenson | https://pastebin.com/DURaZvQd | 18:43 |
inc0 | because glance and nova also uses ceph | 18:43 |
inc0 | so if no-volume ephemerals work, that means ceph is ok | 18:43 |
rstarmer | I'm less worried about erasure coding, woudl like but not critical. But no, boot doesnt/ The logs all look "right", but in the end the vms boot with "no bootable device" | 18:44 |
jamesbenson | rstarmer, I've got ceph working a few times, but always seems to be a rough spot for me. | 18:44 |
inc0 | jamesbenson: did you copy empty passwords.yml to /etc/kolla? | 18:44 |
jamesbenson | yep | 18:44 |
rstarmer | inc0: yeah, I think Ceph is happy. Cinder seems happy talking to ceph, cinder and nova talk, but ther's agap somewhere. | 18:44 |
inc0 | rstarmer: glance add image works? | 18:45 |
jamesbenson | inc0: there and empty... | 18:45 |
inc0 | jamesbenson: readable too? | 18:45 |
rstarmer | yes, image in glance. cinder volume log shows cinder grabing the image from a rbd target, and asssociating it with the volume id that is then mapped to the VM | 18:45 |
jamesbenson | 0644 | 18:46 |
inc0 | so, if you try to boot with ephemeral it's still bad right | 18:46 |
inc0 | ? | 18:46 |
jamesbenson | readable by the world | 18:46 |
inc0 | (no cinder involved) | 18:46 |
rstarmer | hmm, you know, I didn't try that, let me :D | 18:46 |
inc0 | jamesbenson: duh...wierd indeed...try to debug python code | 18:46 |
inc0 | rstarmer: fyi that will also use ceph | 18:47 |
inc0 | ephemerals lands on ceph as well | 18:47 |
rstarmer | will try now, and let you know | 18:47 |
inc0 | after that try to manually mount rbd with glance image to see if it's good | 18:47 |
inc0 | maybe something fails at initalizing volume from image | 18:47 |
jamesbenson | I am using pip 4.0.2 for both kolla and kolla-ansible. | 18:48 |
inc0 | duh...we test that in gates and is consistently good... | 18:48 |
rstarmer | ok, so ephemeral also ends up with no bootable device | 18:48 |
jamesbenson | I figured, sort of a core/basic thing. | 18:49 |
rstarmer | is there a pointer to the manual glance/rbd mount? I've never done that | 18:49 |
inc0 | http://docs.ceph.com/docs/master/start/quick-rbd/ | 18:49 |
*** ducttape_ has quit IRC | 18:59 | |
*** dciabrin has quit IRC | 19:00 | |
rstarmer | Found it. wrong image file type. Selected 'raw' instead of qcow2... | 19:00 |
inc0 | happened to me more than once | 19:00 |
rstarmer | all is right with the world. Now, to figure out erasure coding, and how to separate our "big" storage from our "little" storage. One for glance/backups/rgw and one for volumes... | 19:01 |
*** rwallner has quit IRC | 19:03 | |
*** itoperatorguy has joined #openstack-kolla | 19:04 | |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP: Dockerfile publishing pipeline https://review.openstack.org/475469 | 19:11 |
itoperatorguy | hi all. I am trying to install kolla. And during deploy I get error messeges about missing docker images. for example the kolla-ansible try to use fluentd:4.0.2 and toolbox:4.0.2 but they does not exists on hub.docker.com. What am I missing? | 19:13 |
*** itlinux_ has quit IRC | 19:13 | |
*** itlinux has quit IRC | 19:13 | |
inc0 | itoperatorguy: we haven't pushed images yet, sorry | 19:13 |
inc0 | it's on us | 19:13 |
inc0 | I encourage you to build them | 19:13 |
jamesbenson | you'll have to build them yourself itoperatorguy | 19:13 |
inc0 | we are working on automation for timely pushes but in general I discourage using dockerhub images before that | 19:14 |
inc0 | because these images are stale and updates haven't been included | 19:14 |
itoperatorguy | this is the right way to do it? "./build.py -b centos --registry kolla:5000 --push" | 19:15 |
inc0 | well if you made pip install kolla | 19:15 |
inc0 | then you have kolla-build | 19:15 |
inc0 | but yeah that's that it means | 19:15 |
itoperatorguy | "kolla" is the hostname of my deployment mashine where I previously run ./start-registry | 19:15 |
inc0 | yeah that works | 19:15 |
inc0 | then in globals.yml in "docker_registry" specify kolla:5000 | 19:16 |
inc0 | and it'll work | 19:16 |
*** rhallisey-away is now known as rhallisey | 19:16 | |
itoperatorguy | well then I am still in trouble because I did. I started the registry, I run the push command above. I run the kolla-ansible prechecks then I wanted to do a deploy and I got the error messeges that I have missing docker images. | 19:17 |
inc0 | what is exact error you have? | 19:18 |
itoperatorguy | first for fluentd. I checked on docker hub and yes there is no 4.0.2 tag only 4.0.0 so I pulled it retagged it and pushed it to my local repo as a workaround. I run the deploy agan and now I have same issue with toolbox. | 19:19 |
itoperatorguy | fatal: [network1]: FAILED! => {"changed": false, "failed": true, "msg": "The requested image does not exist: kolla:5000/kolla/centos-binary-kolla-toolbox:4.0.2"} | 19:19 |
inc0 | strange, did you build and push all of them? | 19:19 |
inc0 | ahh | 19:19 |
inc0 | you pulled and retagged | 19:19 |
inc0 | yeah, well, that won't work | 19:20 |
inc0 | I mena it will | 19:20 |
inc0 | but will require you to do all manual | 19:20 |
inc0 | try just doing pip install -U kolla | 19:20 |
inc0 | and kolla-build --registry kolla:5000 --push -b centos -t binary | 19:20 |
inc0 | it will build and tag images to 4.0.2 automatically | 19:21 |
inc0 | if you're hell bend on not building | 19:21 |
inc0 | just add openstack_release: "4.0.0" to globals.yml | 19:21 |
inc0 | and then you cna pull from dockerhub | 19:21 |
inc0 | but again, old, not updated images so good for exploration but not prod | 19:21 |
itoperatorguy | if I run "docker images | wc -l" my results are 204 and if I run just run docker images then I see that most of them are tagged with "5.0.0" | 19:23 |
inc0 | if you checkout master it will be 5.x | 19:23 |
inc0 | as in Pike | 19:23 |
inc0 | 5.x is development now | 19:23 |
jamesbenson | inc0: wierd... my passwords file was really empty.... not sure how that happened... | 19:23 |
inc0 | a yeah | 19:23 |
jamesbenson | fixed now | 19:23 |
inc0 | well | 19:23 |
itoperatorguy | what should be the latest stable tag? | 19:24 |
inc0 | 4.0.2 | 19:24 |
inc0 | if you do git checkout stable/ocata | 19:24 |
inc0 | that will be one you'll get | 19:25 |
inc0 | or rather | 19:25 |
inc0 | 4.0.3 is what you get because it's always stable tag+next minor | 19:25 |
itoperatorguy | so I need something like this? "./build.py -b centos --registry kolla:5000 --tag 4.0.3 --push" + openstack_release: "4.0.3" to globals.yml ? | 19:27 |
inc0 | itoperatorguy: you have code from github? | 19:27 |
inc0 | then if you want ocata (stable | 19:27 |
inc0 | you need to run git checkout stable/ocata | 19:27 |
inc0 | to swithc to stable branch | 19:28 |
inc0 | after that run pip install -e -U . | 19:28 |
inc0 | (from kolla dir) | 19:28 |
inc0 | and kolla-build -b centos --registry kolla:5000 --push | 19:28 |
inc0 | no need to specify tag | 19:28 |
itoperatorguy | ok which repo? Because I see to on the official documentation (BTW what is the difference)? kolla or kolla-ansible? | 19:30 |
itoperatorguy | probably my English is the "barrier factor" but I can not really get the difference | 19:33 |
*** ducttape_ has joined #openstack-kolla | 19:33 | |
*** ducttape_ has quit IRC | 19:34 | |
*** ducttape_ has joined #openstack-kolla | 19:35 | |
*** spsurya has quit IRC | 19:35 | |
inc0 | itoperatorguy: kolla - building images | 19:36 |
inc0 | kolla-ansible - deploying images | 19:36 |
inc0 | when you're use this method, your deployment is split into two steps | 19:36 |
inc0 | 1. building set of images and populating your registry | 19:37 |
inc0 | that's Kolla's job | 19:37 |
inc0 | after that you run ansible-playbooks which downloads these images, prepares config and runs them in correct way | 19:37 |
inc0 | that's kolla-ansible | 19:37 |
*** manheim has quit IRC | 19:37 | |
itoperatorguy | inc0: ok it makes much more sense. Thanks. I see the light now :D | 19:37 |
inc0 | haha | 19:37 |
inc0 | yeah, let me know if you have any further questions | 19:38 |
*** mgkwill_ is now known as mgkwill | 19:46 | |
Reepicheep | what is the proper way when adding new nodes to an existing cluster to bootstrap-server on the new nodes? | 19:57 |
Reepicheep | if I add it to my inventory and run bootstrap-server it is restarting docker on all my nodes which breaks mariadb | 19:58 |
*** rwallner has joined #openstack-kolla | 19:59 | |
*** rstarmer has quit IRC | 20:03 | |
*** rstarmer has joined #openstack-kolla | 20:11 | |
*** sgrasley has joined #openstack-kolla | 20:12 | |
inc0 | Reepicheep: you can just make temp inventory for that | 20:13 |
inc0 | (but it's bug) | 20:13 |
inc0 | temporarily specify [baremetal] group | 20:13 |
inc0 | or you can use ansible --limit | 20:13 |
*** kbaegis has quit IRC | 20:15 | |
*** jascott1 has quit IRC | 20:15 | |
*** jascott1 has joined #openstack-kolla | 20:16 | |
*** kbaegis has joined #openstack-kolla | 20:16 | |
*** jascott1 has quit IRC | 20:17 | |
Reepicheep | inc0: thanks that is what I was gonig to do then handle the host files elseware | 20:19 |
inc0 | deploy is idempotent so it won't break | 20:19 |
inc0 | however bootstrap servers seems to have issue there | 20:19 |
inc0 | but yeah, hostfiles | 20:20 |
inc0 | damn | 20:20 |
inc0 | well, I'll fix it quickly | 20:20 |
*** kbaegis has quit IRC | 20:20 | |
Reepicheep | inc0: lol, I can confirm there are issues .. it stops all the mariadb containers at once .. which leaves the galera in a unkown state with no host "safe_to_bootstrap" and all "seqno" as -1 | 20:22 |
*** kbaegis has joined #openstack-kolla | 20:23 | |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Fix baremetal role idempotency https://review.openstack.org/475916 | 20:24 |
inc0 | Reepicheep: ^ | 20:24 |
inc0 | that patch should fix it | 20:24 |
Reepicheep | inc0: thanks.. I will give it a try | 20:26 |
jamesbenson | inc0: clean system and having issues with ceph : Bootstrapping Ceph OSDs. Fails here. was this related to the ceph issues yesterday we were stuck on? | 20:35 |
inc0 | show me log please | 20:36 |
Reepicheep | inc0: that worked on stable/ocata thanks again | 20:39 |
Reepicheep | I'm guessing that patch will not be backported so I can just track it localy for now, correct? | 20:39 |
*** shardy has quit IRC | 20:40 | |
inc0 | it actually might | 20:40 |
inc0 | mind submitting bug? | 20:40 |
inc0 | (or I can do it) | 20:41 |
Reepicheep | I can do that | 20:41 |
inc0 | if we tag this to be backported, it will | 20:41 |
inc0 | and should be released in 4.0.3 | 20:41 |
inc0 | thanks | 20:41 |
*** goldyfruit has quit IRC | 20:43 | |
jamesbenson | inc0: here's the ansible output https://pastebin.com/TSQy9RUc | 20:49 |
jamesbenson | inc0: docker logs bootstrap_osd_0 and ceph_mon | 20:50 |
jamesbenson | https://pastebin.com/ntD7gAwe | 20:50 |
inc0 | jamesbenson: whenever you see "container returned non-zero code" you should check docker logs <<container>> | 20:50 |
inc0 | yeah | 20:50 |
jamesbenson | and I'm getting the others | 20:50 |
inc0 | check /var/lib/docker/volumes/kolla_logs/ | 20:50 |
inc0 | for ceph mon and ceph osd | 20:51 |
inc0 | logs | 20:51 |
jamesbenson | inc0: https://pastebin.com/mL9Sgk1r for ceph-mon | 20:51 |
inc0 | yeah I saw similar thing locally:/ no idea what's happening | 20:52 |
jamesbenson | inc0: ceph_client: https://pastebin.com/Qz1Tk5Rr | 20:52 |
inc0 | I saw something like this when ntp wasn't synchronized | 20:52 |
*** manheim has joined #openstack-kolla | 20:52 | |
jamesbenson | they seem to be all synced...at least ntpd is running and pointing to a local ntp server. | 20:52 |
inc0 | yeah | 20:52 |
inc0 | I couldn't figure it out few days ago too:/ | 20:53 |
jamesbenson | yeah, no idea here either.... | 20:53 |
jamesbenson | I know it was working a few days ago (before I did some kolla/kolla-ansible updates | 20:53 |
Reepicheep | inc0: https://bugs.launchpad.net/kolla/+bug/1699335 | 20:54 |
openstack | Launchpad bug 1699335 in kolla "running bootstrap-servers on an existing install breaks mariadb" [Undecided,New] | 20:54 |
jamesbenson | but looking at the changelogs, I couldn't find anything that should have impacted it. | 20:54 |
inc0 | jamesbenson: I would say this can be ceph issue | 20:54 |
inc0 | that wouldn't be reflected in changelogs | 20:54 |
jamesbenson | is stuff pulled directly from ceph's repo? | 20:55 |
inc0 | no | 20:55 |
inc0 | apt/yum installed | 20:55 |
jamesbenson | oh... | 20:55 |
jamesbenson | and it's not marked to a specific version? | 20:55 |
inc0 | it's marked to major release (as in jewel) | 20:56 |
inc0 | but I don't know if it's issue | 20:56 |
inc0 | how many control nodes do you have? | 20:56 |
inc0 | 3? | 20:56 |
*** manheim has quit IRC | 20:56 | |
jamesbenson | yeah | 20:56 |
jamesbenson | 3 | 20:56 |
inc0 | no idea:/ | 21:00 |
jamesbenson | now I understand why ceph is such a pain though, if we are just apt it from somewhere else ... | 21:01 |
jamesbenson | inc0: where is that in kolla's repo? | 21:02 |
*** manheim has joined #openstack-kolla | 21:02 | |
inc0 | ceph installation? | 21:02 |
jamesbenson | yeah, the apt | 21:02 |
inc0 | https://github.com/openstack/kolla/blob/master/docker/ceph/ceph-base/Dockerfile.j2 here | 21:03 |
inc0 | and repos are added here https://github.com/openstack/kolla/blob/master/docker/base/sources.list.ubuntu | 21:04 |
inc0 | hmm interesting, we don't pin to jewel in ubuntu it seems | 21:05 |
jamesbenson | yeah, wanted to check these things.. | 21:05 |
jamesbenson | that might be the issue then, no? | 21:05 |
*** lucasxu has quit IRC | 21:05 | |
inc0 | seems like it, I was pinning this but that was long ago | 21:06 |
inc0 | however still unlikely because we're using native ubuntu | 21:06 |
inc0 | so maintainers are supposed to provice these things | 21:06 |
inc0 | and also gates uses latest images and theyre just fine | 21:06 |
*** rwallner has quit IRC | 21:06 | |
*** rwallner has joined #openstack-kolla | 21:07 | |
jamesbenson | still worth a try... | 21:08 |
inc0 | yeah we removed pin as ubuntu cloud archive provices jewel | 21:08 |
*** rhallisey has quit IRC | 21:10 | |
*** rwallner has quit IRC | 21:11 | |
jamesbenson | so do we just need to add in the ceph repo here:https://github.com/openstack/kolla/blob/stable/ocata/docker/base/sources.list.ubuntu | 21:13 |
jamesbenson | I'm looking at your commit back in Sept 6 2016, removed the version lock because it was only used in cent, not ubuntu and risky if we miss a security patch on ceph..... | 21:14 |
jamesbenson | https://github.com/openstack/kolla/commit/0a50cb8889f245bc89b49fbb2dacb9c02ded5f88 | 21:14 |
*** salv-orl_ has joined #openstack-kolla | 21:17 | |
*** jbadiapa_ has quit IRC | 21:19 | |
*** salv-orlando has quit IRC | 21:20 | |
*** gfidente|afk has quit IRC | 21:21 | |
jamesbenson | has there been much luck with ceph-ansible, I remember seeing people talk about it in the past.... | 21:25 |
*** dciabrin has joined #openstack-kolla | 21:27 | |
*** rstarmer has quit IRC | 21:30 | |
inc0 | well they're going to be better at ceph deploy than we will | 21:33 |
openstackgerrit | Eric Peterson proposed openstack/kolla-kubernetes master: Add missing setting for novncproxy_base_url https://review.openstack.org/475933 | 21:33 |
*** dave-mccowan has quit IRC | 21:35 | |
*** schwicht has quit IRC | 21:35 | |
*** Mannheim has joined #openstack-kolla | 21:35 | |
*** pramodrj07 has joined #openstack-kolla | 21:37 | |
*** MasterOfBugs has quit IRC | 21:40 | |
*** kong has left #openstack-kolla | 21:40 | |
*** rwallner has joined #openstack-kolla | 21:41 | |
*** rwallner_ has joined #openstack-kolla | 21:43 | |
*** manheim has quit IRC | 21:43 | |
*** Mannheim has quit IRC | 21:44 | |
*** rwallner has quit IRC | 21:45 | |
*** dciabrin has quit IRC | 21:45 | |
*** Manheim has joined #openstack-kolla | 21:46 | |
*** rwallner_ has quit IRC | 21:47 | |
jamesbenson | got to run inc0, chat more tomorrow :-) | 21:50 |
inc0 | have a good one | 21:50 |
jamesbenson | you too | 21:50 |
*** jamesbenson has quit IRC | 21:50 | |
*** Pavo has quit IRC | 21:50 | |
*** jamesbenson has joined #openstack-kolla | 21:52 | |
*** calbers has quit IRC | 21:54 | |
*** jamesbenson has quit IRC | 21:57 | |
*** jamesbenson has joined #openstack-kolla | 21:59 | |
*** itoperatorguy has quit IRC | 21:59 | |
*** omenv has quit IRC | 22:03 | |
*** TxGirlGeek has quit IRC | 22:03 | |
*** schwicht has joined #openstack-kolla | 22:03 | |
*** jamesbenson has quit IRC | 22:04 | |
*** kolla-slack has quit IRC | 22:10 | |
*** yee37923 has joined #openstack-kolla | 22:10 | |
*** yee37923 has quit IRC | 22:10 | |
*** ducttape_ has quit IRC | 22:11 | |
*** kolla-slack has joined #openstack-kolla | 22:11 | |
*** kolla-slack has quit IRC | 22:12 | |
*** openstack has joined #openstack-kolla | 22:18 | |
*** jrist has quit IRC | 22:18 | |
*** kolla-slack1 has quit IRC | 22:19 | |
*** yee379 has quit IRC | 22:20 | |
*** jrist has joined #openstack-kolla | 22:20 | |
*** jrist has quit IRC | 22:20 | |
*** kolla-slack has joined #openstack-kolla | 22:20 | |
*** dosaboy has joined #openstack-kolla | 22:20 | |
*** sklgromek has joined #openstack-kolla | 22:20 | |
*** yee379 has joined #openstack-kolla | 22:20 | |
*** jascott1 has joined #openstack-kolla | 22:20 | |
*** jrist has joined #openstack-kolla | 22:21 | |
openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible master: Upgrade from docker-py to docker https://review.openstack.org/475941 | 22:21 |
*** goldyfruit has joined #openstack-kolla | 22:23 | |
*** Pavo has joined #openstack-kolla | 22:27 | |
*** goldyfruit has quit IRC | 22:27 | |
*** manheim has joined #openstack-kolla | 22:30 | |
*** jascott1 has quit IRC | 22:31 | |
sdake | nottzomg could it be hotter | 22:33 |
sdake | 120DEGREES F | 22:33 |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla master: WIP: Dockerfile publishing pipeline https://review.openstack.org/475469 | 22:37 |
inc0 | sdake: global warming is real | 22:39 |
inc0 | it'll be close to 100 in Oregon too | 22:39 |
inc0 | and it's Oregon | 22:39 |
inc0 | I moved here because it's colder ;( | 22:39 |
*** schwicht has quit IRC | 22:39 | |
inc0 | sdake: check this out https://review.openstack.org/#/c/474770/ | 22:40 |
*** TxGirlGeek has joined #openstack-kolla | 22:40 | |
inc0 | kubectl create -f /this_file | 22:40 |
inc0 | and it will create all configs and secrets | 22:40 |
inc0 | (when I'm dnoe with it) | 22:40 |
*** sbezverk has quit IRC | 22:42 | |
*** manheim has quit IRC | 22:43 | |
*** manheim has joined #openstack-kolla | 22:45 | |
kolla-slack | <jascott1> sdake its nor really 120 is it? | 22:45 |
*** jascott1 has joined #openstack-kolla | 22:46 | |
inc0 | jascott1: yes, yes it is | 22:46 |
jascott1 | HFS! | 22:46 |
jascott1 | i would think I would be digging underground in that climate | 22:47 |
inc0 | almost 50 C | 22:47 |
inc0 | jascott1: if you touch ground, it'll burn you | 22:48 |
inc0 | also happy digging:P | 22:48 |
jascott1 | so basically living in hell | 22:48 |
jascott1 | ya if warming keeps up we will be in the primo spot ;) | 22:49 |
jascott1 | well we already are imo | 22:49 |
jascott1 | oregon tropical beachfront property | 22:49 |
*** manheim has quit IRC | 22:50 | |
inc0 | just a tad north and I'll be good | 22:50 |
jascott1 | heheh | 22:50 |
jascott1 | hey I got your changes working for me wrt to playbooks. I had to correct something or other and also added var for helm version as they had a problem with google storage yesterday | 22:51 |
jascott1 | 2.5 release was broken most day and all night | 22:52 |
*** pramodrj07 has quit IRC | 22:52 | |
*** pramodrj07 has joined #openstack-kolla | 22:53 | |
*** goldyfruit has joined #openstack-kolla | 22:55 | |
*** manheim has joined #openstack-kolla | 22:57 | |
*** goldyfruit has quit IRC | 22:58 | |
*** salv-orl_ has quit IRC | 22:59 | |
*** salv-orlando has joined #openstack-kolla | 22:59 | |
*** manheim has quit IRC | 23:01 | |
*** salv-orlando has quit IRC | 23:03 | |
*** schwicht has joined #openstack-kolla | 23:10 | |
*** harlowja has quit IRC | 23:19 | |
*** spiette has joined #openstack-kolla | 23:20 | |
*** jamesben_ has joined #openstack-kolla | 23:28 | |
openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible master: Upgrade from docker-py to docker https://review.openstack.org/475941 | 23:33 |
*** jamesben_ has quit IRC | 23:33 | |
openstackgerrit | Hongbin Lu proposed openstack/kolla master: Upgrade from docker-py to docker https://review.openstack.org/475939 | 23:43 |
*** sbezverk has joined #openstack-kolla | 23:44 | |
*** tonanhngo has quit IRC | 23:47 | |
*** eaguilar has joined #openstack-kolla | 23:50 | |
*** tonanhngo has joined #openstack-kolla | 23:52 | |
*** tonanhngo has quit IRC | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!