| *** awidders_ has joined #openstack-kolla | 00:01 | |
| *** awiddersheim has quit IRC | 00:01 | |
| *** lakerzhou has quit IRC | 00:02 | |
| *** dardelean_ has quit IRC | 00:09 | |
| *** dave-mccowan has joined #openstack-kolla | 00:10 | |
| kfox1111 | arg... wonder if the gate's broken now... .http://logs.openstack.org/26/449826/15/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/aa4c03c/console.html#_2017-09-19_23_57_55_706154 | 00:16 |
|---|---|---|
| kfox1111 | looks like deps changed out from under us. | 00:16 |
| *** hongbin has quit IRC | 00:18 | |
| kfox1111 | looks lilke we need pycrypto in requirements.txt | 00:21 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Add pycrypto https://review.openstack.org/505445 | 00:23 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Native Values Step 1. https://review.openstack.org/449826 | 00:24 |
| *** huzhengchuan has quit IRC | 00:26 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Native Values Step 1. https://review.openstack.org/449826 | 00:37 |
| *** duonghq has joined #openstack-kolla | 00:38 | |
| duonghq | morning guys | 00:38 |
| kfox1111 | morning. | 00:39 |
| kfox1111 | just in time for me to head out. :/ | 00:40 |
| kfox1111 | have a good one. :) | 00:40 |
| duonghq | nice to meet you | 00:40 |
| *** dave-mcc_ has joined #openstack-kolla | 00:45 | |
| *** yangyapeng has quit IRC | 00:48 | |
| *** dave-mccowan has quit IRC | 00:48 | |
| *** yangyapeng has joined #openstack-kolla | 00:48 | |
| *** hrw has quit IRC | 00:50 | |
| *** hrw has joined #openstack-kolla | 00:51 | |
| *** yangyapeng has quit IRC | 00:52 | |
| *** dave-mcc_ has quit IRC | 00:54 | |
| *** zhurong has joined #openstack-kolla | 00:54 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Remove unused configuration items from tacker role https://review.openstack.org/505075 | 00:56 |
| *** caowei has joined #openstack-kolla | 01:02 | |
| *** lakerzhou1 has quit IRC | 01:04 | |
| *** huzhengchuan has joined #openstack-kolla | 01:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/485905 | 01:08 |
| *** kevzha01 has joined #openstack-kolla | 01:08 | |
| *** kevzha01 is now known as kevinz | 01:16 | |
| *** rwsu has quit IRC | 01:25 | |
| *** klindgren_ has joined #openstack-kolla | 01:28 | |
| *** klindgren has quit IRC | 01:31 | |
| *** tovin07_ has joined #openstack-kolla | 01:40 | |
| *** itlinux has joined #openstack-kolla | 01:54 | |
| *** yangyapeng has joined #openstack-kolla | 01:58 | |
| *** zhangfei has joined #openstack-kolla | 01:58 | |
| *** itlinux has quit IRC | 02:08 | |
| *** dardelean_ has joined #openstack-kolla | 02:09 | |
| *** dardelean_ has quit IRC | 02:14 | |
| *** dave-mccowan has joined #openstack-kolla | 02:19 | |
| *** dave-mcc_ has joined #openstack-kolla | 02:20 | |
| *** dave-mccowan has quit IRC | 02:23 | |
| *** unicell has quit IRC | 02:24 | |
| *** yangyapeng has quit IRC | 02:33 | |
| *** dave-mcc_ has quit IRC | 02:38 | |
| *** jascott1 has quit IRC | 02:48 | |
| *** jascott1 has joined #openstack-kolla | 02:49 | |
| *** jascott1 has quit IRC | 02:53 | |
| *** rwsu has joined #openstack-kolla | 03:16 | |
| *** kollian has joined #openstack-kolla | 03:25 | |
| spsurya | morning all | 03:29 |
| spsurya | duonghq: morning | 03:29 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/485905 | 03:35 |
| duonghq | morning spsurya | 03:36 |
| *** coolsvap has joined #openstack-kolla | 03:53 | |
| *** janki has joined #openstack-kolla | 04:00 | |
| *** huanxie has joined #openstack-kolla | 04:02 | |
| *** caowei has quit IRC | 04:08 | |
| *** huanxie has quit IRC | 04:19 | |
| *** caowei has joined #openstack-kolla | 04:40 | |
| *** skramaja has joined #openstack-kolla | 04:44 | |
| *** sbezverk has joined #openstack-kolla | 04:45 | |
| *** duonghq has quit IRC | 04:51 | |
| *** zhurong has quit IRC | 04:53 | |
| *** mdnadeem has joined #openstack-kolla | 04:53 | |
| *** jaosorior has joined #openstack-kolla | 05:01 | |
| *** caowei has quit IRC | 05:01 | |
| *** unicell has joined #openstack-kolla | 05:09 | |
| *** magicboiz has quit IRC | 05:33 | |
| *** magicboiz has joined #openstack-kolla | 05:34 | |
| *** caowei has joined #openstack-kolla | 05:43 | |
| *** jascott1 has joined #openstack-kolla | 05:54 | |
| *** daidv has quit IRC | 05:56 | |
| *** daidv has joined #openstack-kolla | 05:57 | |
| *** hieulq has quit IRC | 05:57 | |
| *** hieulq has joined #openstack-kolla | 05:57 | |
| *** robbbe has quit IRC | 06:03 | |
| *** jiriprox has quit IRC | 06:04 | |
| *** lrensing has joined #openstack-kolla | 06:06 | |
| *** kollian has quit IRC | 06:08 | |
| *** magicboiz has quit IRC | 06:39 | |
| *** magicboiz has joined #openstack-kolla | 06:40 | |
| *** yingjun has joined #openstack-kolla | 06:43 | |
| *** yangyapeng has joined #openstack-kolla | 06:44 | |
| *** zhurong has joined #openstack-kolla | 06:45 | |
| *** lrensing has quit IRC | 07:01 | |
| *** pcaruana has joined #openstack-kolla | 07:02 | |
| *** shardy has joined #openstack-kolla | 07:02 | |
| *** serlex has joined #openstack-kolla | 07:02 | |
| *** openstackgerrit has quit IRC | 07:02 | |
| *** jiriprox has joined #openstack-kolla | 07:08 | |
| *** openstackgerrit has joined #openstack-kolla | 07:09 | |
| openstackgerrit | yanpeifei proposed openstack/kolla-ansible master: common: fix ansible location and use docker in interactive model https://review.openstack.org/492018 | 07:09 |
| openstackgerrit | yanpeifei proposed openstack/kolla-ansible master: add ceph guide https://review.openstack.org/497705 | 07:09 |
| openstackgerrit | yanpeifei proposed openstack/kolla-ansible master: update python version https://review.openstack.org/505511 | 07:09 |
| *** dardelean_ has joined #openstack-kolla | 07:10 | |
| *** mnasiadka has joined #openstack-kolla | 07:12 | |
| *** yangyapeng has quit IRC | 07:12 | |
| *** yangyapeng has joined #openstack-kolla | 07:13 | |
| *** yangyapeng has quit IRC | 07:13 | |
| *** yangyapeng has joined #openstack-kolla | 07:14 | |
| *** yangyapeng has quit IRC | 07:15 | |
| *** yangyapeng has joined #openstack-kolla | 07:15 | |
| *** yangyapeng has quit IRC | 07:15 | |
| *** sgordon has quit IRC | 07:16 | |
| *** yangyapeng has joined #openstack-kolla | 07:16 | |
| *** yangyapeng has quit IRC | 07:17 | |
| *** yangyapeng has joined #openstack-kolla | 07:17 | |
| *** yangyapeng has quit IRC | 07:17 | |
| *** yangyapeng has joined #openstack-kolla | 07:18 | |
| *** yangyapeng has quit IRC | 07:18 | |
| *** dardelean_ has quit IRC | 07:18 | |
| *** yangyapeng has joined #openstack-kolla | 07:19 | |
| *** bjolo has joined #openstack-kolla | 07:19 | |
| bjolo | morning | 07:19 |
| *** yangyapeng has quit IRC | 07:19 | |
| *** yangyapeng has joined #openstack-kolla | 07:20 | |
| *** dardelean has joined #openstack-kolla | 07:20 | |
| *** yangyapeng has quit IRC | 07:21 | |
| *** sgordon has joined #openstack-kolla | 07:21 | |
| *** yangyapeng has joined #openstack-kolla | 07:21 | |
| *** dardelean_ has joined #openstack-kolla | 07:26 | |
| *** caowei has quit IRC | 07:27 | |
| *** caowei has joined #openstack-kolla | 07:29 | |
| *** caowei has quit IRC | 07:30 | |
| *** huanxie has joined #openstack-kolla | 07:34 | |
| *** caowei has joined #openstack-kolla | 07:35 | |
| *** jascott1 has quit IRC | 07:42 | |
| *** zhurong has quit IRC | 07:48 | |
| *** yangyapeng has quit IRC | 07:50 | |
| *** yangyapeng has joined #openstack-kolla | 07:51 | |
| *** jaosorior has quit IRC | 07:55 | |
| *** rmart04 has joined #openstack-kolla | 08:02 | |
| *** bmace has quit IRC | 08:04 | |
| *** dardelean_ has quit IRC | 08:05 | |
| *** yangyapeng has quit IRC | 08:05 | |
| *** yangyapeng has joined #openstack-kolla | 08:06 | |
| *** huzhengchuan has quit IRC | 08:08 | |
| *** yangyapeng has quit IRC | 08:10 | |
| *** gfidente has joined #openstack-kolla | 08:10 | |
| *** gfidente has quit IRC | 08:10 | |
| *** gfidente has joined #openstack-kolla | 08:10 | |
| *** zhubingbing__ has joined #openstack-kolla | 08:12 | |
| *** zhubingbing_ has quit IRC | 08:15 | |
| *** lxdong has joined #openstack-kolla | 08:15 | |
| *** mgoddard has joined #openstack-kolla | 08:18 | |
| *** jaosorior has joined #openstack-kolla | 08:19 | |
| lxdong | change /etc/kolla/config/neutron.conf to [DEFAULT] service_plugins=trunk and reconfigured everything works well except occasional dashboard error :Unable to connect to Neutron. Is this an issue? | 08:19 |
| *** dardelean_ has joined #openstack-kolla | 08:19 | |
| *** dardelean has quit IRC | 08:20 | |
| *** dardelean_ has quit IRC | 08:23 | |
| *** yangyapeng has joined #openstack-kolla | 08:25 | |
| *** jiriprox has quit IRC | 08:28 | |
| *** yangyapeng has quit IRC | 08:30 | |
| *** bmace has joined #openstack-kolla | 08:31 | |
| *** huzhengchuan has joined #openstack-kolla | 08:35 | |
| *** egonzalez has joined #openstack-kolla | 08:35 | |
| *** zhurong has joined #openstack-kolla | 08:44 | |
| *** pbourke has quit IRC | 08:54 | |
| *** pbourke has joined #openstack-kolla | 08:54 | |
| magicboiz | morning | 09:10 |
| magicboiz | I'm testing kolla-ansible upgrade (ocata -> pike), but I'm facing some issues with mariadb cluster. Any quick idea? | 09:11 |
| egonzalez | magicboiz, what kind of issues? | 09:16 |
| magicboiz | egonzalez: bootstrap.yml mariadb role fails, "RUNNING HANDLER [mariadb : wait for slave mariadb]" fails 10 attempts | 09:20 |
| egonzalez | magicboiz, how many mariadb nodes? using ansible serial or something? | 09:21 |
| magicboiz | egonzalez: 3 mariadb nodes. What's ansible serial? I tried with kolla-ansible upgrade | 09:22 |
| magicboiz | egonzalez: I did a fresh ocata deployment, it went ok, and then pike kolla-ansible upgrade | 09:23 |
| egonzalez | magicboiz, nvm then, can you share upgrade logs for the part in mariadb upgrade? also mariadb logs too | 09:24 |
| openstackgerrit | Pierre Hanselmann proposed openstack/kolla-ansible master: WSGI configuration is missing the directive "WSGIApplicationGroup %{GLOBAL}" after "WSGIProcessGroup" in the horizon template. Of all WSGI configuration templates it is the only one that does not have the "WSGIApplicationGroup" line. https://review.openstack.org/505569 | 09:26 |
| magicboiz | egonzalez: sure: https://pastebin.com/vH7nmXnL | 09:27 |
| *** sambetts|afk is now known as sambetts | 09:28 | |
| *** huanxie has quit IRC | 09:31 | |
| openstackgerrit | Pierre Hanselmann proposed openstack/kolla-ansible master: WSGI configuration is missing the directive "WSGIApplicationGroup %{GLOBAL}" after "WSGIProcessGroup" in the horizon template. Of all WSGI configuration templates it is the only one that does not have the "WSGIApplicationGroup" line. https://review.openstack.org/505569 | 09:33 |
| egonzalez | magicboiz, could you check logs in /var/lib/docker/volumes/kolla_logs/_data/mariadb , control01 was master node in your cluster, others slaves | 09:33 |
| magicboiz | egonzalez: mariadb.log https://pastebin.com/6sZn40kU | 09:34 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible stable/ocata: Add ovs section in neutron lbaas configuration https://review.openstack.org/505571 | 09:37 |
| magicboiz | egonzalez: https://pastebin.com/yzNwEPb7 this is from /var/lib/docker/volumes/kolla_logs/_data/mariadb | 09:38 |
| openstackgerrit | Pierre Hanselmann proposed openstack/kolla-ansible master: WSGI configuration missing directive https://review.openstack.org/505569 | 09:42 |
| *** jascott1 has joined #openstack-kolla | 09:43 | |
| openstackgerrit | Merged openstack/kolla master: Enable xtrace on start scripts https://review.openstack.org/494258 | 09:47 |
| *** jascott1 has quit IRC | 09:47 | |
| *** dardelean_ has joined #openstack-kolla | 09:49 | |
| openstackgerrit | Merged openstack/kolla master: Use immutable journal path rather than device name https://review.openstack.org/492796 | 09:50 |
| *** yingjun has quit IRC | 09:52 | |
| *** dardelean_ has quit IRC | 09:55 | |
| openstackgerrit | Merged openstack/kolla master: Add more collectd plugins to the container https://review.openstack.org/502956 | 10:04 |
| *** tovin07_ has quit IRC | 10:10 | |
| *** jchhatbar has joined #openstack-kolla | 10:15 | |
| *** jchhatbar has quit IRC | 10:15 | |
| *** janki has quit IRC | 10:17 | |
| *** jchhatbar has joined #openstack-kolla | 10:17 | |
| *** jiriprox has joined #openstack-kolla | 10:20 | |
| *** jchhatbar_ has joined #openstack-kolla | 10:22 | |
| *** jchhatbar_ has quit IRC | 10:22 | |
| *** zhurong has quit IRC | 10:22 | |
| *** janki has joined #openstack-kolla | 10:23 | |
| *** jchhatbar has quit IRC | 10:25 | |
| *** caowei has quit IRC | 10:34 | |
| *** huanxie has joined #openstack-kolla | 10:36 | |
| *** dardelean_ has joined #openstack-kolla | 10:37 | |
| *** rstarmer has joined #openstack-kolla | 10:39 | |
| *** huanxie has quit IRC | 10:40 | |
| *** magicboiz has quit IRC | 10:56 | |
| *** magicboiz has joined #openstack-kolla | 10:56 | |
| *** skramaja_ has joined #openstack-kolla | 11:10 | |
| *** skramaja has quit IRC | 11:10 | |
| *** inc0 has quit IRC | 11:11 | |
| *** zhangfei has quit IRC | 11:15 | |
| *** dave-mccowan has joined #openstack-kolla | 11:15 | |
| *** ArminderSingh- has joined #openstack-kolla | 11:21 | |
| *** ArminderSingh has quit IRC | 11:23 | |
| *** ansmith has quit IRC | 11:30 | |
| *** awiddersheim has joined #openstack-kolla | 11:32 | |
| *** serlex has quit IRC | 11:36 | |
| dardelean_ | lhi | 11:36 |
| *** awidders_ has quit IRC | 11:36 | |
| *** skramaja has joined #openstack-kolla | 11:43 | |
| *** skramaja_ has quit IRC | 11:43 | |
| *** rstarmer has quit IRC | 11:59 | |
| *** lakerzhou has joined #openstack-kolla | 12:00 | |
| *** Croata2 has joined #openstack-kolla | 12:02 | |
| egonzalez | daidv, sup | 12:06 |
| mnasiadka | hi | 12:06 |
| mnasiadka | anybody has any idea how get keepalived logs from neutron-l3-agent container? | 12:07 |
| *** inc0 has joined #openstack-kolla | 12:07 | |
| *** janki has quit IRC | 12:08 | |
| *** rhallisey has joined #openstack-kolla | 12:09 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fluentd: fix Mariadb mysqld_safe log not match https://review.openstack.org/502169 | 12:13 |
| *** janki has joined #openstack-kolla | 12:20 | |
| *** kevinz has quit IRC | 12:22 | |
| *** zhubingbing has joined #openstack-kolla | 12:37 | |
| *** kbaegis has joined #openstack-kolla | 12:52 | |
| *** skramaja has quit IRC | 12:52 | |
| *** Zophar has joined #openstack-kolla | 12:54 | |
| *** ansmith has joined #openstack-kolla | 13:06 | |
| *** yingjun has joined #openstack-kolla | 13:07 | |
| *** rstarmer has joined #openstack-kolla | 13:08 | |
| rstarmer | Anyone seen this issue with 5.0 nova-compute: 'Could not access KVM kernel module: Permission denied' | 13:09 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: base: add Linaro repository for Debian with some backports https://review.openstack.org/505620 | 13:11 |
| hrw | this ^^ patch adds repo with libvirt 3.6.0 and qemu 2.10-rc3 backports. both aarch64 and x86_64 are provided | 13:12 |
| *** sean-k-mooney has quit IRC | 13:18 | |
| egonzalez | i'm not sure about adding non distro repos or bug fixes | 13:21 |
| hrw | egonzalez: neither do I so wanted to get opinions | 13:22 |
| hrw | egonzalez: especially when kolla allows for custom overrides | 13:23 |
| kolla-slack | <britthouser> @rstarmer might check owner of /dev/kvm ? | 13:23 |
| kolla-slack | <britthouser> https://review.openstack.org/505286 | 13:24 |
| rstarmer | looked, it's still root:root, but the /etc/libvirt/qemu.conf has services run as nova:nova. Hmm. I see the change is supposed to have /dev/kvm in the kvm group, and that's not what I'm seeing. will try a change to the device and see. | 13:25 |
| rstarmer | although i don't see the nova user being part of the kvm group either :/ | 13:25 |
| kolla-slack | <britthouser> that review points to https://review.openstack.org/#/c/501653/ | 13:27 |
| *** yingjun has quit IRC | 13:28 | |
| rstarmer | britthouser: thanks for the pointers, will check as soon as I get back online (hopefully shortly). | 13:28 |
| *** jheroux has joined #openstack-kolla | 13:32 | |
| *** rstarmer has quit IRC | 13:41 | |
| *** huzhengchuan has quit IRC | 13:42 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/pike: Pin ansible to <2.4 to make ara work in gates https://review.openstack.org/505634 | 13:43 |
| *** jascott1 has joined #openstack-kolla | 13:45 | |
| *** rmart04 has quit IRC | 13:46 | |
| *** rmart04_ has joined #openstack-kolla | 13:46 | |
| *** jascott1 has quit IRC | 13:49 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/504645 | 13:50 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/485905 | 13:50 |
| *** kelv_ has joined #openstack-kolla | 13:53 | |
| *** lxdong has quit IRC | 13:54 | |
| *** zhurong has joined #openstack-kolla | 13:55 | |
| *** gkadam has joined #openstack-kolla | 14:04 | |
| egonzalez | do we have meeting today? | 14:04 |
| *** janki has quit IRC | 14:07 | |
| hrw | good point | 14:12 |
| *** rmart04_ has quit IRC | 14:14 | |
| *** rmart04 has joined #openstack-kolla | 14:16 | |
| egonzalez | Jeffrey4l, we need to merge https://review.openstack.org/#/c/503712/ with https://review.openstack.org/#/c/505634/ to make gates work, both are failures that cannot be merged | 14:16 |
| egonzalez | *regarding stable/pike | 14:16 |
| *** huzhengchuan has joined #openstack-kolla | 14:18 | |
| *** athomas has quit IRC | 14:18 | |
| Jeffrey4l | egonzalez, then we have to merge these two patches into one. | 14:18 |
| *** athomas has joined #openstack-kolla | 14:21 | |
| *** bmace has quit IRC | 14:26 | |
| *** rstarmer has joined #openstack-kolla | 14:28 | |
| *** aagate has joined #openstack-kolla | 14:29 | |
| *** aagate has quit IRC | 14:29 | |
| *** mnasiadka has quit IRC | 14:32 | |
| *** rmart04 has quit IRC | 14:32 | |
| *** ntpttr_laptop has joined #openstack-kolla | 14:34 | |
| *** rstarmer has quit IRC | 14:34 | |
| *** gkadam is now known as gkadam-afk | 14:37 | |
| *** lrensing has joined #openstack-kolla | 14:38 | |
| *** bmace has joined #openstack-kolla | 14:41 | |
| *** zhubingbing has quit IRC | 14:44 | |
| *** rmart04 has joined #openstack-kolla | 14:50 | |
| *** duonghq has joined #openstack-kolla | 14:50 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/504645 | 14:51 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/485905 | 14:51 |
| *** aagate has joined #openstack-kolla | 14:52 | |
| *** jamesbenson has joined #openstack-kolla | 14:52 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove deprecated oslo_messaging_rabbit options https://review.openstack.org/498692 | 14:55 |
| hrw | Jeffrey4l: two branches into one patch? | 14:56 |
| Jeffrey4l | hrm, two patches into one. and yes. | 14:56 |
| *** hongbin has joined #openstack-kolla | 14:56 | |
| *** sbezverk has quit IRC | 14:57 | |
| gema | is there a meeting today? | 14:57 |
| *** mdnadeem has quit IRC | 14:58 | |
| gema | oh, I see we all have the same doubts, will show up then just in case | 14:58 |
| duonghq | I guess that we'll have | 15:00 |
| Jeffrey4l | this is why voting jobs annoying at this point. hrw | 15:02 |
| gema | Jeffrey4l: we are going to be adding the Linaro dev cloud to the gates | 15:03 |
| gema | to be able to test kolla on arm64 | 15:03 |
| Jeffrey4l | that's cool lol | 15:03 |
| Jeffrey4l | when it will be happen? | 15:03 |
| gema | Jeffrey4l: but it is going to take me a couple of weeks to get started because we are at Connect next week, in SFO | 15:03 |
| gema | and then I need to scale out one of our clouds | 15:04 |
| gema | so we have more capacity | 15:04 |
| gema | Jeffrey4l: I will keep you posted | 15:04 |
| gema | Jeffrey4l: because I am sure I will need your guidance | 15:05 |
| Jeffrey4l | roger. np. | 15:05 |
| *** zhurong has quit IRC | 15:05 | |
| rwellum | Hey gema egonzalez - you had good trips back? | 15:08 |
| egonzalez | rwellum, yeah and you? | 15:09 |
| rwellum | Other than flight was delayed and my car broke down - yes. :) | 15:09 |
| *** jbrooks has quit IRC | 15:09 | |
| *** rmart04 has quit IRC | 15:09 | |
| inc0 | good morning | 15:10 |
| rwellum | Morning | 15:11 |
| kfox1111 | morning | 15:11 |
| rwellum | Hey kfox1111 - I pinged you back yesterday - but think we missed each other | 15:11 |
| kfox1111 | yeah. no worries. | 15:12 |
| kfox1111 | can't remember the question. | 15:12 |
| rwellum | I took a look at the crypto patch - lgtm. | 15:13 |
| *** kollian has joined #openstack-kolla | 15:13 | |
| kfox1111 | cool. inc0 can you second it plz? | 15:13 |
| kfox1111 | should unbreak the gate I think. | 15:13 |
| inc0 | kfox1111: link plz | 15:13 |
| *** LuigiT33 has joined #openstack-kolla | 15:14 | |
| *** ntpttr_laptop has quit IRC | 15:15 | |
| kfox1111 | hmm.... looks like there is an alternative up for review too: https://review.openstack.org/#/c/484946 | 15:17 |
| kfox1111 | is pycrypto going away? | 15:17 |
| *** jamesbenson has quit IRC | 15:17 | |
| *** dave-mcc_ has joined #openstack-kolla | 15:18 | |
| *** robbbe has joined #openstack-kolla | 15:19 | |
| rwellum | Not sure but I see sdake has -1'd this | 15:19 |
| *** dave-mccowan has quit IRC | 15:19 | |
| *** jamesbenson has joined #openstack-kolla | 15:21 | |
| kfox1111 | just talked with infra. looks like pycrpto is being deprecated. | 15:22 |
| kfox1111 | so this other review is preffered. | 15:22 |
| kfox1111 | lets finish it up. | 15:22 |
| *** jamesbenson has quit IRC | 15:22 | |
| hrw | kfox1111: pycrypto is dead. cryptography is new crypto in python | 15:22 |
| *** jbrooks has joined #openstack-kolla | 15:27 | |
| *** jamesbenson has joined #openstack-kolla | 15:30 | |
| *** robbbe has quit IRC | 15:30 | |
| *** zhubingbing has joined #openstack-kolla | 15:34 | |
| *** zhubingbing has joined #openstack-kolla | 15:35 | |
| *** zhubingbing has quit IRC | 15:35 | |
| *** zhubingbing has joined #openstack-kolla | 15:36 | |
| *** vhosakot has joined #openstack-kolla | 15:39 | |
| *** LuigiT33 has quit IRC | 15:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/504645 | 15:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/485905 | 15:42 |
| *** leifmadsen has quit IRC | 15:46 | |
| rwellum | Agreed kfox1111 - let poke sdake totake another look? | 15:46 |
| *** leifmadsen has joined #openstack-kolla | 15:47 | |
| inc0 | we'd probably want to do the same in ansible | 15:47 |
| egonzalez | ansible already changed to cryptography https://review.openstack.org/#/c/459195/ | 15:48 |
| inc0 | ah I missed this oen | 15:49 |
| kfox1111 | ok. looks like the ansible one fixed sdake's issues. lets just clone it and merge. | 15:49 |
| inc0 | right | 15:50 |
| kfox1111 | I'll subit a fix in a min. | 15:50 |
| *** zhubingbing has quit IRC | 15:51 | |
| rwellum | Cool | 15:51 |
| kfox1111 | incoming. | 15:51 |
| *** jamesbenson has quit IRC | 15:52 | |
| kfox1111 | https://review.openstack.org/484946 | 15:52 |
| inc0 | we have no agenda for today | 15:54 |
| dardelean_ | has anyone tried magnum with k8s lately? | 15:55 |
| *** jascott1 has joined #openstack-kolla | 15:55 | |
| rwellum | inc0: I'd like to point out the kolla-k8s schedule for 1.x - see if anyone has any issues/thoughts. | 15:55 |
| inc0 | we can make summary of ptg | 15:56 |
| sdake | hey folks | 15:57 |
| rwellum | +1 inc0 | 15:57 |
| sdake | rwellum link pl? | 15:57 |
| rwellum | https://etherpad.openstack.org/p/kolla-queens-ptg-k8s-release-roadmap sdake | 15:58 |
| sdake | ok - ya I think my -1 on that crypto review was basically "copy from kollla-ansible' | 15:58 |
| *** mnasiadka has joined #openstack-kolla | 15:58 | |
| sdake | rwellum i was speaking of the crypto review ;) | 15:58 |
| rwellum | Ah :) | 15:58 |
| duonghq | +1 inc0 | 15:58 |
| rwellum | Is it ok to do +w - even before ci has run? Newbie core question... | 15:59 |
| *** jascott1 has quit IRC | 15:59 | |
| *** britthouser has joined #openstack-kolla | 16:00 | |
| *** dardelean_ has quit IRC | 16:00 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 16:01 | |
| *** dardelean_ has joined #openstack-kolla | 16:02 | |
| *** Jeffrey4l has quit IRC | 16:06 | |
| *** Jeffrey4l has joined #openstack-kolla | 16:07 | |
| *** jascott1 has joined #openstack-kolla | 16:08 | |
| *** Jeffrey4l_ has quit IRC | 16:08 | |
| *** jascott1 has quit IRC | 16:09 | |
| *** jascott1 has joined #openstack-kolla | 16:09 | |
| *** jascott1 has quit IRC | 16:10 | |
| *** jascott1 has joined #openstack-kolla | 16:10 | |
| *** huzhengchuan has quit IRC | 16:11 | |
| sdake | rwellum looks like alot of work :) | 16:11 |
| sdake | rwellum we always wait for +w on ci - as we want to verify the patch si good | 16:12 |
| *** jiriprox has quit IRC | 16:12 | |
| rwellum | Yeah agreed. | 16:12 |
| rwellum | ok cool. | 16:12 |
| sdake | rwellum your welcome to add a 2nd =2 tho | 16:12 |
| rwellum | And why aren't you in the kolla meeting? :) | 16:12 |
| sdake | U2 | 16:12 |
| rwellum | I saw my name ;) | 16:13 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/504645 | 16:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/485905 | 16:15 |
| *** glycerine has joined #openstack-kolla | 16:16 | |
| *** glycerine has quit IRC | 16:16 | |
| vhosakot | rwellum: I always do +workflow for a green gate. If gate is not green, I usually don't +w. | 16:17 |
| vhosakot | I'll start reviewing more... | 16:17 |
| rwellum | Thanks guys | 16:19 |
| rwellum | Makes sense but I've seen others +w without green gate - so wasn't sure. | 16:19 |
| *** jamesbenson has joined #openstack-kolla | 16:21 | |
| *** ntpttr_laptop has joined #openstack-kolla | 16:21 | |
| kfox1111 | rwellum: as you get more familior with the gate, you can recognize when all the gates have passed and when there is a test that failed due to some other issue. | 16:22 |
| rwellum | +1 | 16:23 |
| kfox1111 | at minimum, sometimes just running the gate tests twice and seeing that the same tests passed all the gates but not all at the same time. | 16:23 |
| *** mnasiadka has quit IRC | 16:24 | |
| kfox1111 | ideally all the gates would pass all the time, but the kolla gates do an aweful lot of things so its been hard to make them pass 100% of the time on all the clouds infra has. | 16:24 |
| *** janki has joined #openstack-kolla | 16:26 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix to use . to source script files https://review.openstack.org/498408 | 16:33 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix prechecks fail with docker not in deployment host https://review.openstack.org/478796 | 16:33 |
| openstackgerrit | Merged openstack/kolla-ansible stable/pike: Modify the conf in skydive https://review.openstack.org/501153 | 16:34 |
| *** mnasiadka has joined #openstack-kolla | 16:36 | |
| *** daidv_ has joined #openstack-kolla | 16:36 | |
| egonzalez | hrw, the patch for ceph https://review.openstack.org/#/c/502014/ feel free to abandon and create a new one :) | 16:40 |
| *** sbezverk has joined #openstack-kolla | 16:40 | |
| *** mnasiadka has quit IRC | 16:41 | |
| hrw | thx | 16:41 |
| *** unicell has quit IRC | 16:41 | |
| hrw | egonzalez: I think that xenial-updates ceph will be good enough (and all archs) - will look | 16:41 |
| *** sambetts is now known as sambetts|afk | 16:44 | |
| *** janki has quit IRC | 16:46 | |
| *** shardy has quit IRC | 16:48 | |
| *** dardelean_ has quit IRC | 16:54 | |
| *** mnasiadka has joined #openstack-kolla | 16:56 | |
| *** gfidente is now known as gfidente|afk | 16:59 | |
| jamesbenson | gema: link for that? | 17:00 |
| *** britthouser has quit IRC | 17:00 | |
| gema | jamesbenson: you should start with refstack-client | 17:00 |
| gema | it will land there | 17:00 |
| *** mnasiadka has quit IRC | 17:01 | |
| gema | https://github.com/openstack/refstack-client | 17:01 |
| inc0 | so idea behind tempest is really elegant one | 17:01 |
| *** egonzalez has quit IRC | 17:01 | |
| gema | let me find the etherpad | 17:01 |
| inc0 | since we're building reference implementations for various things | 17:01 |
| inc0 | it will be good for gates, testing | 17:01 |
| jamesbenson | yeah, I think I've tried this before, but had issues :-( | 17:01 |
| inc0 | and docs | 17:01 |
| inc0 | we can write docs pointing to it "this is how you deploy scenario x" | 17:01 |
| gema | inc0: I can help drive that | 17:01 |
| gema | I am doing it internally also | 17:01 |
| gema | for our own tesitng | 17:01 |
| inc0 | cool, that'd be awesome gema | 17:01 |
| gema | testing | 17:01 |
| Jeffrey4l | inc0, then how can we push 5.0.0 image to docker now? | 17:01 |
| inc0 | you're face of this effort then:) | 17:01 |
| inc0 | Jeffrey4l: manually I guess | 17:02 |
| gema | inc0: sounds good | 17:02 |
| inc0 | although I'm reluctant to do it | 17:02 |
| gema | jamesbenson: https://etherpad.openstack.org/p/InteropDenver2017PTG_TempestAutoconfig | 17:02 |
| Jeffrey4l | yes. and any volunteer for this now? | 17:02 |
| inc0 | too many people uses these images | 17:02 |
| jamesbenson | inc0: following Jeffrey4l, things still aren't pushed to dockerhub then? | 17:02 |
| inc0 | and they're bad images to use. | 17:02 |
| Walex | guys, what are typical times from download Kolla/Kolla-ansible to getting a small cluster (even all-in-one) running? | 17:02 |
| jamesbenson | thanks gema | 17:02 |
| gema | np | 17:02 |
| Jeffrey4l | jamesbenson, yes. it is not pushed. | 17:02 |
| jamesbenson | k | 17:03 |
| inc0 | Walex: depends on your disk/net | 17:03 |
| *** duonghq has quit IRC | 17:03 | |
| inc0 | ~20 min for buliding images | 17:03 |
| inc0 | + few more min to deploy | 17:03 |
| inc0 | + time it takes to write configuration and stuff | 17:03 |
| jamesbenson | Walex, I can get it to about 30 minutes for a full prod minus building | 17:03 |
| Walex | it took me a day and a half to work around all the bugsd | 17:03 |
| inc0 | what bugs?:) | 17:03 |
| gema | anyway, night folks, I am in the uk and it is supper time | 17:03 |
| inc0 | good night gema we can discuss later how to organize scenario thingy | 17:04 |
| jamesbenson | Walex, I think the first time around takes a little bit to make sure networking and setting are correct. But then it's pretty easy. | 17:04 |
| inc0 | thanks for taking it up | 17:04 |
| gema | inc0: sure thing | 17:04 |
| gema | no probs | 17:04 |
| Walex | plus several hours to build the images and an hour or two to do the deploy | 17:04 |
| jamesbenson | night gema, thanks for the info | 17:04 |
| gema | inc0: I will also sync up with you after next week on the upgrading our clouds to kolla thing | 17:04 |
| gema | inc0: we can get more people involved if they want to be | 17:04 |
| inc0 | wow Walex that's pretty strange | 17:04 |
| *** mgoddard has quit IRC | 17:04 | |
| jamesbenson | gema, count me in | 17:04 |
| Walex | jamesbenson: I was asking "from scratch", not doing it having alreadyu done it several times | 17:04 |
| inc0 | it took 20min for me to deploy 120 hardware ndoes... | 17:04 |
| Jeffrey4l | Walex, are u in China lol.. | 17:05 |
| jamesbenson | ^_^ | 17:05 |
| *** gkadam-afk is now known as gkadam-away | 17:05 | |
| inc0 | Walex: check which docker storage driver you use | 17:05 |
| inc0 | if it's devicemapper, that will be very slow | 17:05 |
| Walex | inc0: thats indeed devicemapper. | 17:05 |
| Walex | inc0: but also I have had to do it several times to fix stuff not in the docs. | 17:06 |
| inc0 | if you switch to btrfs or overlay2 it'll speed up build significantly | 17:06 |
| inc0 | right, docs needs fixing for us, in fact this is my personal theme of Queens release | 17:06 |
| Jeffrey4l | Walex, please feel free to file bug for the issue you meet. | 17:07 |
| inc0 | if you could sumbit bugs or provide feedback in any form, that'd be really helpful | 17:07 |
| inc0 | ofc you can also submit doc fix itself:) that'd would be ever better! | 17:07 |
| Walex | Jeffrey4l: I files one with Horizon Dockerfile.j2, the others have been around a while. | 17:07 |
| Walex | Jeffrey4l: some of them are Docker workarounds | 17:08 |
| Jeffrey4l | Walex, could u provide the bug link? | 17:08 |
| inc0 | anyway, got to go for a few, need to feed my cat and such | 17:08 |
| inc0 | tt y'all lateer | 17:08 |
| sdake | inc0 = crazy cat guy | 17:08 |
| inc0 | sdake: you don't know half of it | 17:09 |
| sdake | oh Ithink i do :) | 17:09 |
| jamesbenson | lol | 17:09 |
| rwellum | Why are the images 'bad'? Jeffrey4l ? | 17:09 |
| rwellum | Trying to follwo | 17:09 |
| rwellum | follow | 17:09 |
| inc0 | dockerhub ones? | 17:09 |
| Jeffrey4l | rwellum, sorry? | 17:10 |
| inc0 | becayse they're not updated | 17:10 |
| inc0 | as in, all deps are frozen in time from moment we buld them | 17:10 |
| inc0 | so CVEs and stuff aren't getting fixed | 17:10 |
| *** dave-mcc_ has quit IRC | 17:10 | |
| Jeffrey4l | btw, centos 7.4 is released | 17:10 |
| inc0 | yeah, I heard people had issues | 17:10 |
| rwellum | Oh right - from that point yes. But no worse than any others currently on duckerhub | 17:10 |
| *** dardelean_ has joined #openstack-kolla | 17:11 | |
| inc0 | rwellum: no worse doesn't mean good;) | 17:11 |
| rwellum | ack | 17:11 |
| *** dave-mccowan has joined #openstack-kolla | 17:11 | |
| Jeffrey4l | someone complain that why 5.0.0 image is not pushed to docker ;( | 17:11 |
| inc0 | I think we shouldn't do anything with dockerhub until it's automated and mechanism for this is in place | 17:11 |
| kfox1111 | rwellum: got a bit of time to work on a ps? | 17:11 |
| inc0 | right, let's just ask them to wait till we make it proper please | 17:12 |
| inc0 | it's for their own good;) | 17:12 |
| Walex | this is the most obvious bugs I encountered in doing a 5.0.0 setup: http://paste.ubuntu.com/25580933/ | 17:12 |
| rwellum | kfox1111: definitely - what's up? | 17:12 |
| inc0 | thanks Walex I'll read through that in a moment | 17:12 |
| kfox1111 | rwellum: if you have interest, it would be great to finally get rid of the resolv.conf workaround for the net=host containers. | 17:12 |
| kfox1111 | now that k8s has a feature that lets us work around it properly. | 17:13 |
| rwellum | Using the new network namespaces? | 17:14 |
| *** daidv_ has quit IRC | 17:15 | |
| kfox1111 | no. there's a pod attribute now: | 17:15 |
| kfox1111 | dnsPolicy: ClusterFirst | 17:15 |
| *** dardelean_ has quit IRC | 17:15 | |
| kfox1111 | that we can set on the net=host pods that will cause it to do dns the same way as non net=host pods. | 17:15 |
| *** mnasiadka has joined #openstack-kolla | 17:16 | |
| kfox1111 | which was what the resolv.conf workaround was doing. | 17:16 |
| kfox1111 | but with a captured resolv.conf to a configmap and volume mounting it. | 17:16 |
| kfox1111 | the new way is way cleaner. :) | 17:16 |
| rwellum | Oh - sounds good. I'll take a look. | 17:17 |
| *** dardelean has joined #openstack-kolla | 17:17 | |
| kfox1111 | cool. thanks. :) | 17:18 |
| kfox1111 | if you have any questions, please let me know. | 17:18 |
| rwellum | I am sure I will :) | 17:18 |
| *** mnasiadka has quit IRC | 17:21 | |
| *** awiddersheim has quit IRC | 17:24 | |
| *** awiddersheim has joined #openstack-kolla | 17:24 | |
| openstackgerrit | Christopher Nipper proposed openstack/kolla-ansible master: Add support for Neutron SRIOV agent container https://review.openstack.org/487610 | 17:25 |
| *** awidders_ has joined #openstack-kolla | 17:25 | |
| openstackgerrit | Christopher Nipper proposed openstack/kolla-ansible master: Add details about SRIOV enablement to the networking guide https://review.openstack.org/498112 | 17:25 |
| *** awiddersheim has quit IRC | 17:29 | |
| Walex | ah my | 17:32 |
| *** Veggis has joined #openstack-kolla | 17:34 | |
| *** mnasiadka has joined #openstack-kolla | 17:34 | |
| kolla-slack | <egonzalez> Walex, regarding "Need to run cleanup containers", was a new deployment or a shutdown are start again? | 17:35 |
| Walex | ah my notes on the install so far: http://www.sabi.co.uk/blog/17-two.html?170918#170918 | 17:35 |
| jascott1 | kfox1111 looks like that helm binary test error'd out... was that something with helm or the test? | 17:36 |
| Walex | egonzalez: every time I do 'kolla-ansible stop' when I do a 'deploy' afterwards 'kolla-toolbox' does not start or something else | 17:36 |
| kfox1111 | jascott1: haven't looked... sec... | 17:36 |
| *** schwicht has joined #openstack-kolla | 17:36 | |
| Walex | egonzalez: looks like either the containers are left in a half-dead state or tghere is a race condition in the order in which they start. | 17:37 |
| Walex | egonzalez: it is usually the 'mariab' container that cannot start because 'kolla-toolbox' has not started yet. | 17:38 |
| inc0 | kolla-toolbox doesn't run any service really so thats really strange behaviour | 17:38 |
| hrw | looks like I sorted out ceph/jewel for ubuntu ;D | 17:38 |
| inc0 | I know galera hates (hates!) full cluster stop | 17:38 |
| hrw | will create patch tomorrow | 17:38 |
| * hrw off | 17:38 | |
| inc0 | Walex: right, that's not kolla-toolbos issue | 17:38 |
| inc0 | that's galera | 17:38 |
| Walex | inc0: the error I get is that 'mariadb' cannot create the haproxy user unless 'kolla-toolbo# is running | 17:38 |
| kfox1111 | jascott1: looks like I missed a chmod: | 17:39 |
| kfox1111 | http://logs.openstack.org/26/449826/17/check/gate-kolla-kubernetes-python35/6a20135/console.html#_2017-09-20_00_41_43_959643 | 17:39 |
| Walex | inc0: I have disabled haproxy | 17:39 |
| inc0 | well one thing is, kolla-ansible stop is not cleanup | 17:39 |
| inc0 | if you run kolla-ansible stop it will halt your env but you need to restart it manually really | 17:39 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Native Values Step 1. https://review.openstack.org/449826 | 17:39 |
| hrw | Jeffrey4l: https://review.openstack.org/#/c/502014 got comment in case my remote machine dies | 17:39 |
| inc0 | because of galera | 17:39 |
| * hrw off | 17:39 | |
| Walex | inc0: and it should not be... I don't want to reconfigure OpenStack every time I shut it down | 17:40 |
| inc0 | cleanup is done with kolla-ansible destroy | 17:40 |
| jascott1 | kfox1111 oh ok, let me know if I need to re-roll | 17:40 |
| inc0 | Walex: why would you shut down openstack tho | 17:40 |
| inc0 | not a use case we handle very well because...why? | 17:40 |
| kfox1111 | jascott1: will do. | 17:40 |
| Walex | inc0: to shut down the server? | 17:40 |
| kfox1111 | thx. | 17:40 |
| inc0 | well you need to handle mariadb recovery then | 17:40 |
| inc0 | but that's good feedback for our "devmode" | 17:41 |
| inc0 | maybe we should handle aio better - as in don't deploy clustered mariadb for aio | 17:41 |
| *** mnasiadka has quit IRC | 17:41 | |
| Walex | inc0: sometimes I do need to so the mariadb recovery, but the usual error is that mariadb cannotr creeate the hapreoxy user. | 17:41 |
| inc0 | I think real issue is that kolla sees mariadb as stopped so it tries to bootstrap it, or rather doesn't see it started | 17:42 |
| inc0 | let me test this a bit, in fact aio devenv is what I'm focused on now so we'll sort these things out | 17:42 |
| Walex | BTW part of my punishment is to reinstall the lot on stop of an existing OpenStack instance, which is overloaded, so 'deploy' takes 16 minutes... | 17:43 |
| kolla-slack | <egonzalez> Walex, problem is kolla_docker module does not start containers if are stopped. Which to be honest I'm not sure if is the best behavior, as if containers are started in wrong order(galera) may broke your whole environment | 17:43 |
| inc0 | harsh :( | 17:43 |
| inc0 | yeah that too | 17:43 |
| inc0 | well not for aio | 17:43 |
| inc0 | but our first priority is prod, so that's why aio is a bit untested in these scenarios | 17:44 |
| Walex | egonzalez: that should be documented... | 17:44 |
| kolla-slack | <egonzalez> Build times may me caused by your network, docker config or access to repos in case there is no repo near you | 17:44 |
| kolla-slack | <egonzalez> Actually i would drop stop command | 17:44 |
| inc0 | +1 to that | 17:45 |
| inc0 | well | 17:45 |
| inc0 | it might be useful for devenv | 17:45 |
| Walex | egonzalez: but theere is also the issue of: how do you stop a cluster and then restart it, if stopping it might cause containers to restart in the wrong order... | 17:45 |
| inc0 | Walex: then usually they'll keep restarting until it sorts itself out | 17:45 |
| Walex | egonzalez: inc0: because one *must* occasionally stop even prod. | 17:45 |
| inc0 | issue is galera really | 17:46 |
| inc0 | well yes, then you perform manual galera recovery | 17:46 |
| kolla-slack | <egonzalez> https://review.openstack.org/#/c/499205 | 17:46 |
| kolla-slack | <egonzalez> This fixes the non container start in kolla_docker | 17:47 |
| *** lrensing has quit IRC | 17:47 | |
| Walex | inc0: the error I get is actually from Kolla I think not MariaDB/Galera. | 17:47 |
| inc0 | but it's caused by galera ultimatelyu | 17:47 |
| inc0 | because we purposefully not start galera | 17:47 |
| kolla-slack | <egonzalez> But don't think will help with Maria recover | 17:47 |
| inc0 | because if we mess it up | 17:48 |
| inc0 | you lose data | 17:48 |
| inc0 | and we'd rather keep it manual than cause dataloss | 17:48 |
| iniazi | so I'm trying to deploy using 5.0.0 instead of 4.0.2... image building works fine (or atleast I can get all but the dpdk & kafka built...which I guess is expected behaviour), but deploying 5.0.0 getting issues with grafana + panko... or atleast where it stops | 17:48 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP init-container from json to yaml. https://review.openstack.org/505761 | 17:49 |
| inc0 | Walex: fwiw all use cases you talk about are very relevant now for us | 17:50 |
| inc0 | because in Queens we want Kolla to be alternative to devstack | 17:50 |
| inc0 | so AIO restart and stuff like that are part of that | 17:50 |
| inc0 | which means, we'll fix everything you talk about over next weeks | 17:50 |
| iniazi | panko says that service_name is undefined (although it keeps going) and then stops at grafana, "TASK [grafana : Wait for grafana application ready]" ... "503 Service Unavailable nNo server is available"..."url": "http://10.0.0.20:3000/login" | 17:50 |
| iniazi | that IP is the internal vip | 17:51 |
| inc0 | iniazi: so grafana, being part of monitoring, is still tech preview | 17:51 |
| inc0 | read unmaintained reallyh | 17:51 |
| inc0 | we should get rid of it or fix it | 17:51 |
| iniazi | ok. so disable it now, and deploy manually? | 17:51 |
| iniazi | is there a list of projects that aren't maintained/unsupported? so I can just skip them if they have an issue and I can try to deploy them manually. | 17:52 |
| inc0 | yeah I wouldn't recommend monitoring really here | 17:52 |
| iniazi | is it the same with telegraf, ceilometer? | 17:52 |
| iniazi | ok | 17:52 |
| inc0 | ceilometer should be fine | 17:52 |
| inc0 | we need to do this kind of house cleaning, we have a lot of stuff where people merged something year ago and left afterwards | 17:53 |
| kolla-slack | <egonzalez> iniazi, which task fail in panko and release? | 17:53 |
| Walex | uhm, are the Nova container sharing '/var/log/'? | 17:54 |
| Walex | uhm, are the Nova container sharing '/var/log/kolla? | 17:54 |
| iniazi | egonzalez: looks like it went past my buffer. let me redeploy with panko on | 17:54 |
| kolla-slack | <egonzalez> Walex yep, all containers are mounting logs in a docker volume | 17:55 |
| inc0 | Walex: /var/log/kolla is mounted to docker volume kolla_logs | 17:55 |
| inc0 | you can find contents in /var/lib/docker/volumes/kolla_logs/_data | 17:55 |
| iniazi | inc0: is telegraf stable or is that in that twilight zone as well? :) | 17:56 |
| *** mnasiadka has joined #openstack-kolla | 17:56 | |
| Walex | ah thanks! | 17:56 |
| inc0 | twilight or even dark night that's full of terrors | 17:56 |
| Walex | inc0: that's very useful and should be documented... | 17:57 |
| inc0 | logs or terrors? | 17:58 |
| inc0 | because terrors I'd rather deprecate;) | 17:58 |
| iniazi | <booming voice>Log of Terrors</> | 17:58 |
| inc0 | The Log is dark and full of terrors ... :D | 17:59 |
| iniazi | oh general question. if deploy fails, is it best to cleanup and redeploy? thats what I do everytime. | 17:59 |
| iniazi | hehe | 17:59 |
| inc0 | cleanup | 17:59 |
| Walex | uhmmm I am looking at logs because I may have done a minor misconfig and wondering how to fix it quickly | 18:00 |
| inc0 | and redeploy | 18:00 |
| inc0 | deploy should be safe but if you can do cleanup, do cleanup | 18:00 |
| inc0 | Walex: kolla-ansible reconfigure | 18:00 |
| Walex | the issue is that i have all-in-one and I put in '/etc/hosts' two names for it | 18:00 |
| Walex | inc0: ahhh excellent, but I have to find what I did wrong | 18:00 |
| *** mnasiadka has quit IRC | 18:00 | |
| Walex | one name test-m the other fully qualified. | 18:00 |
| inc0 | right, that will make rabbitmq cryu | 18:00 |
| *** kollian has quit IRC | 18:01 | |
| Walex | inc0: as I suspected... | 18:01 |
| Walex | I get: Host [u'test-m', u'test-m'] fails. Previously tried hosts: [[u'test-m', u'test-m']] | 18:01 |
| inc0 | bbiaf, need to go to hunt for some food | 18:02 |
| iniazi | ok, i just run cleanup-host & cleanup-containers on all the hosts. | 18:02 |
| Walex | when I try to create a compute instance, and the message is that 'No valid host was found. There are not enough hosts available.' | 18:02 |
| inc0 | I heard our caffeteria is full of game, so I'll be back once I acquire some sustinance | 18:02 |
| iniazi | speaking of food, i need to have breakfast & lunch... and have a CAB/CRB meeting to attend | 18:02 |
| kolla-slack | <egonzalez> iniazi kolla-ansible destroy will do in all hosts for you | 18:03 |
| inc0 | right kolla-ansible bootstrap-servers is great tool | 18:03 |
| iniazi | yeah, i haven't tried bootstrap-servers. it would have made it easier when I was setting it up. i'll run it after I try the destroy (another first time) | 18:03 |
| iniazi | oh, was thinking of bootstrap-servers... what does the containers do? | 18:05 |
| iniazi | oh, i guess you meant servers | 18:06 |
| inc0 | which containers? | 18:06 |
| inc0 | ah | 18:06 |
| inc0 | it installs docker and stuff | 18:06 |
| inc0 | make your clean centos/ubuntu ready for kolla | 18:06 |
| iniazi | yeah. I did that manually from trial and error. running it now. its changing lots of stuff... | 18:07 |
| iniazi | anyway, i'll try panko agaain after all is finished and let you know where deploy failed | 18:07 |
| * inc0 off | 18:08 | |
| *** mnasiadka has joined #openstack-kolla | 18:15 | |
| *** mnasiadka has quit IRC | 18:19 | |
| *** lucasxu has joined #openstack-kolla | 18:21 | |
| vhosakot | yeah, I need to play more with "kolla-ansible bootstrap-servers" | 18:21 |
| jascott1 | kfox1111 have you seen this plush templating lib in go? https://github.com/gobuffalo/plush | 18:22 |
| jascott1 | if its worthy, might be able to use in helm | 18:22 |
| *** mnasiadka has joined #openstack-kolla | 18:26 | |
| *** jiriprox has joined #openstack-kolla | 18:28 | |
| *** kbaegis has quit IRC | 18:29 | |
| Walex | ahhhh I think I found my problem and someone was mentioning a 'chmod' mistake erarlier: | 18:31 |
| Walex | I get in 'nova-compute.log': | 18:31 |
| Walex | : libvirtError: internal error: process exited while connecting to monitor: 2017-09-20T18:21:11.936613Z qemu-kvm: -chardev pty,id=charserial0,logfile=/var/lib/nova/instances/cd5dbccb-9ec8-4f35-a915-7a0df3bfb692/console.log,logappend=off: Unable to open logfile /var/lib/nova/instances/cd5dbccb-9ec8-4f35-a915-7a0df3bfb692/console.log: Permission denied | 18:31 |
| *** mnasiadka has quit IRC | 18:31 | |
| Walex | and that prevents instances from starting. | 18:31 |
| Walex | I have 'pip'-installed 'kolla' 'kolla-ansible' a few hours ago. I can start images on a system installed a few days ago.,.. | 18:33 |
| *** mnasiadka has joined #openstack-kolla | 18:35 | |
| *** kbaegis has joined #openstack-kolla | 18:37 | |
| *** mnasiadka has quit IRC | 18:39 | |
| vhosakot | Walex: the default permissions that nova uses to create the console.log works for qemu. I've seen in the past that changing permissions of that file does not work when we reboot the nova VM. | 18:40 |
| Walex | vhosakot: that's for a newly created VM on first startup though | 18:41 |
| vhosakot | Walex: ah, for a brand new VM's perm.. I see | 18:42 |
| *** ntpttr_laptop has quit IRC | 18:42 | |
| Walex | vhosakot: it is the VM created just after 'init-runonce' even :-) | 18:43 |
| vhosakot | then, it's very brand new totally from scratch | 18:44 |
| Walex | on the other newly created all-in-one I get it right: http://paste.ubuntu.com/25581461/ | 18:45 |
| *** mnasiadka has joined #openstack-kolla | 18:55 | |
| *** gfidente|afk has quit IRC | 18:56 | |
| *** egonzalez has joined #openstack-kolla | 18:56 | |
| Walex | so.... | 18:58 |
| *** mnasiadka has quit IRC | 18:59 | |
| Walex | on the cluster where I can start instance the 'libvirtd' dir is all owned by the same user: http://paste.ubuntu.com/25581563/ | 18:59 |
| Walex | error... same for 'libvirtd' i was comparing by mistake the libvirtd and nova_compute | 19:01 |
| *** lrensing has joined #openstack-kolla | 19:03 | |
| *** coolsvap has quit IRC | 19:14 | |
| Walex | the error I am getting repeatedly on 'deploy' is: "TASK [mariadb : Creating haproxy mysql user]" "fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "kolla_toolbox container is not running."}" | 19:15 |
| *** mnasiadka has joined #openstack-kolla | 19:15 | |
| Walex | and the issue abover with nested OpenStack and logfile creation in Nova apparently is a known issue: https://bugs.launchpad.net/kolla/+bug/1699712 | 19:16 |
| openstack | Launchpad bug 1699712 in kolla "Nova boot doesn't work on Ubuntu in case of nested KVM" [High,In progress] - Assigned to Bertrand Lallau (bertrand-lallau) | 19:16 |
| Walex | but I have virt_type set to "qemu" | 19:18 |
| *** mnasiadka has quit IRC | 19:19 | |
| egonzalez | Walex set also cpu_mode = none | 19:22 |
| *** lrensing has quit IRC | 19:25 | |
| egonzalez | Walex deploy wont start stopped containers | 19:26 |
| *** schwicht has quit IRC | 19:26 | |
| Walex | egonzalez: I have done that too. | 19:30 |
| Walex | egonzalez: I have done cpu_mode to none too | 19:30 |
| Walex | IIRC, checking | 19:30 |
| Walex | egonzalez: I found this https://bugs.launchpad.net/kolla-ansible/+bug/1668654 | 19:31 |
| openstack | Launchpad bug 1668654 in kolla-ansible ocata "ubuntu libvirtError Unable to open logfile" [Critical,Fix released] | 19:31 |
| Walex | that suggests that ut had something to do with "dynamic_ownership" and libvirt version, so explorijng that | 19:31 |
| Walex | both hosts have libvirt 3.2.0 but one is SL7.4 and the other Centos7.3 | 19:32 |
| Walex | egonzalez: 'docket ps -a' lists all containers as 'exited' | 19:33 |
| Walex | test-m# docker ps -a | grep -v Exited | wc -l | 19:33 |
| Walex | 1 | 19:33 |
| Walex | that 1 line is the header line | 19:33 |
| Walex | egonzalez: but as you suggested the sledgehammer of the cleanup tool unsticks that | 19:34 |
| egonzalez | Walex that means are not running, if you stopped have to manually start them | 19:34 |
| *** mnasiadka has joined #openstack-kolla | 19:35 | |
| *** lrensing has joined #openstack-kolla | 19:35 | |
| Walex | egonzalez: then I am missing how to start them with 'kolla-ansible' if that's whats needed | 19:36 |
| Walex | I am trying 'docker start' :-) | 19:37 |
| egonzalez | deploy wont start stopped containers, will create them if not existing or restart if something in config,images or volumes changed | 19:37 |
| Walex | egonzalez: I suppose I am missing the states the Kolla containers can be in and how they overlap with the states of Docker | 19:39 |
| *** Zophar has quit IRC | 19:39 | |
| egonzalez | Walex nothing changes in the states, kolla does not have any docker state at all | 19:39 |
| egonzalez | once the container is created, everything is from docker | 19:40 |
| *** mnasiadka has quit IRC | 19:40 | |
| Walex | egonzalez: that's why you wanted then to remove 'stop' I guess | 19:41 |
| Walex | egonzalez: but I guess 'deploy' starts the "right" containers | 19:41 |
| Walex | s/starts/instantiates/ | 19:42 |
| Walex | the docs say that doing 'deploy' N times is fine, but maybe I have misunderstood that. | 19:42 |
| egonzalez | Walex you can deploy as many times , and nothing will change/restart if the state is the same. but start stopped containers is dangerous | 19:43 |
| inc0 | aaaand I'm back | 19:44 |
| egonzalez | Walex to apply a change in nova.conf, only containers using nova.conf will be restarted | 19:44 |
| inc0 | egonzalez: I'm thinknig about doing mariadb in non-clustered mode for devmode | 19:45 |
| inc0 | and aio | 19:45 |
| inc0 | as in if len(group mariadb) == 1 -> wsrep_driver: none | 19:45 |
| egonzalez | inc0 i thing is already that | 19:45 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/pike: base: use ceph/jewel on ubuntu https://review.openstack.org/505786 | 19:45 |
| egonzalez | s/thing/think | 19:46 |
| hrw | egonzalez: something for you: ^^ | 19:47 |
| inc0 | egonzalez: let me check if deploy aio then restart aio works | 19:47 |
| egonzalez | inc0 https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/mariadb/templates/galera.cnf.j2#L26 not set cluster address | 19:47 |
| inc0 | right, will it turn off galera tho | 19:47 |
| inc0 | kfox1111: ^ do you know if aio deploy will make galera not care about full shutdown? | 19:48 |
| inc0 | SamYaple: ^ | 19:48 |
| kfox1111 | in kolla-kubernetes? | 19:48 |
| egonzalez | inc0 we should move to newer galera version in this release | 19:48 |
| inc0 | in galera | 19:48 |
| egonzalez | hrw many thanks | 19:48 |
| kfox1111 | I don't know if you can aio galera. | 19:49 |
| inc0 | I'm trying to do single node galera in ansible too | 19:49 |
| hrw | egonzalez: yw | 19:49 |
| inc0 | so we should set wsrep_driver: none right? | 19:49 |
| inc0 | for aio | 19:49 |
| kfox1111 | if you do that, its not galera. just normal mysql. | 19:49 |
| inc0 | right, which is what we're after | 19:49 |
| * hrw off | 19:49 | |
| kfox1111 | I think that should work. | 19:49 |
| *** goldyfruit has quit IRC | 19:50 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP init-container from json to yaml. https://review.openstack.org/505761 | 19:52 |
| Walex | inc0: if I may remind you of a workaround that _might_ be affecting me... You have a note that it needs to be revisited for Pike. | 19:52 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Turn off galera in aio https://review.openstack.org/505787 | 19:52 |
| inc0 | no, this won't get to Pike for sure | 19:52 |
| Walex | inc0: do you inc0 not that one... | 19:53 |
| Walex | inc0: this one: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=af9927661dc0a1e514c661da6cf6a7e2b9f90724 | 19:53 |
| Walex | it works on Scientific Linux 7.4 but not CentOS 7.3 as far as I can see | 19:53 |
| inc0 | it is in pike | 19:53 |
| egonzalez | Walex selinux? | 19:54 |
| inc0 | hmm CentOS is affected by it as well from 7.4? | 19:54 |
| Walex | egonzalez: set permissive. | 19:54 |
| inc0 | that might be the case | 19:54 |
| Walex | inc0: no idea, I don't have CentOS 7.4 | 19:55 |
| Walex | I have 'libvirt' 3.2.0 on both | 19:55 |
| inc0 | well correct me if I'm wrong but scientific linux is centos on speed | 19:55 |
| Walex | inc0: it is the same identical thing actually | 19:55 |
| *** mnasiadka has joined #openstack-kolla | 19:55 | |
| Walex | inc0: maybe slightly difdferent speeds of release security updates | 19:55 |
| egonzalez | Walex centos 7.3 but images are in 7.4, gates are broken atm but i have it working locally | 19:56 |
| inc0 | mhm | 19:56 |
| inc0 | should we pin to 7.3 for now? | 19:56 |
| egonzalez | inc0 in kolla gates works too, but in ansible not , weird, no idea what could be different | 19:56 |
| Walex | BTW on SL 7.4 I am actually running CentOS 7.3 containers | 19:56 |
| inc0 | container distro is what matters | 19:57 |
| inc0 | egonzalez: maybe tarballs are stale | 19:57 |
| inc0 | pike tarballs are from 7.9 | 19:58 |
| Walex | inc0: if I have still your attention, the issue is that: qemu-kvm: -chardev pty,id=charserial0,logfile=/var/lib/nova/instances/cd5dbccb-9ec8-4f35-a91 5-7a0df3bfb692/console.log,logappend=off: Unable to open logfile | 19:58 |
| inc0 | master tarballs are fresh | 19:58 |
| Walex | with permission denied | 19:58 |
| egonzalez | inc0 i waited for new tarballs to be published but failed again(master) | 19:58 |
| egonzalez | send a recheck to 505082 and see if now works | 19:58 |
| egonzalez | sent | 19:58 |
| inc0 | Walex: interesting, haven't seen that before | 19:58 |
| inc0 | does this file exists? | 19:59 |
| inc0 | btw that's about nova console log which lives in /var/lib/docker/volumes/nova_compute on host | 19:59 |
| *** mnasiadka has quit IRC | 20:00 | |
| Walex | and what inc0 you actually fixed that issue in this bug: | 20:00 |
| Walex | https://bugs.launchpad.net/kolla-ansible/+bug/1668654 | 20:00 |
| openstack | Launchpad bug 1668654 in kolla-ansible ocata "ubuntu libvirtError Unable to open logfile" [Critical,Fix released] | 20:00 |
| Walex | BTW the CentOS 7.3 system is OpenStack on OpenStack, and I have 'qemu' and 'none' for virt type | 20:01 |
| inc0 | ya looks familiar | 20:01 |
| inc0 | soi Walex if you built contianers now I think you'll get centos 7.4 in container | 20:01 |
| inc0 | which would mean it's the case | 20:01 |
| inc0 | I guess we need to remove this conditional and make it a thing across all distros | 20:02 |
| Walex | inc0: on the system that works I see this: http://paste.ubuntu.com/25581928/ | 20:03 |
| Walex | inc0: ahhhhh interesting about 7.4 | 20:03 |
| inc0 | right we base of Centos:7 image | 20:03 |
| inc0 | which probably got bumped | 20:03 |
| inc0 | that might mean libvirt-bin also got same bump as ealier in ubuntu | 20:04 |
| Walex | inc0: where it works the instance directory is owned by 42436 but the console file is owned by 42427 | 20:04 |
| Walex | inc0: I can very quickly comment out the conditional in the template and re-deploy and see if that works. | 20:04 |
| egonzalez | it is working me in with 7.4 locally | 20:04 |
| Walex | inc0: good idea? | 20:04 |
| inc0 | that'd be awesome | 20:04 |
| inc0 | egonzalez: did you try to spanwn vm and access console-log? | 20:05 |
| Walex | redeploy or reconfigure when I update a template? | 20:05 |
| egonzalez | not the log but yes access to the vm | 20:05 |
| inc0 | Walex: do redeploy just to be sure | 20:05 |
| inc0 | egonzalez: I can't remmber if that doesnt show up when you want to access console log | 20:06 |
| egonzalez | inc0 was during instance creation | 20:06 |
| Walex | its running a new 'deploy', will take a bit on virtual host | 20:07 |
| Walex | wiht DM/LVM2 storage probably 2-3 ;layers of it :-) | 20:07 |
| inc0 | man srsly switch to overlay | 20:07 |
| * Walex admins the SL7.4 host, but not the CentOS 7.3 one | 20:08 | |
| Walex | inc0: should it be defauilt? | 20:08 |
| inc0 | well I assume reason it's not is taht you'll lose access to your containers/images when you switch | 20:08 |
| inc0 | so if they'd change default people would be really unhappy during upgrade | 20:09 |
| *** sunnyve has joined #openstack-kolla | 20:09 | |
| sunnyve | Hi expert, can you please let me know where I can find etherpad or any document for Containerization of Openstack neutron with kola ? | 20:09 |
| *** mgoddard has joined #openstack-kolla | 20:09 | |
| Walex | inc0: that's a v ery good point indeed | 20:10 |
| sunnyve | I want to understand containerization for neutron how It can affect existing mech driver ? | 20:10 |
| inc0 | it's contenerized already...what's exactly your question sunnyve | 20:10 |
| inc0 | ? | 20:10 |
| Walex | sunnyve: my own understanding of 'kolla' and 'kolla-ansible': http://www.sabi.co.uk/blog/17-two.html | 20:11 |
| inc0 | sunnyve: are you trying to move your existing cloud to kolla? | 20:11 |
| inc0 | or deploy brand new? | 20:11 |
| Walex | sunnyve: my own understanding of 'kolla' and 'kolla-ansible': http://www.sabi.co.uk/blog/17-two.html?170918#170918 more precisely | 20:11 |
| Walex | this deploy was quite quick, only "changed=2" and "ok=163" | 20:11 |
| inc0 | Walex: also if you're aiming for scientific reasarch openstack deployment, look at that https://github.com/stackhpc/kayobe | 20:12 |
| sunnyve | in0: I have my neutron plugin mech driver and since we are moving to containerized I want to make understand how does that changes for us. ? Also how to deploy kolla openstack services on devstack | 20:12 |
| inc0 | Walex: do docker restart nova-libvirt tho | 20:12 |
| inc0 | sunnyve: kolla is deployument tool, it's alternative to devstack not built on top of it | 20:13 |
| inc0 | you deploy openstack with kolla or devstack | 20:13 |
| inc0 | moving existing openstack to kolla, or brownfield, is going to be tricky so make sure you know what you're doing | 20:14 |
| inc0 | doing it while keeping vms alive is going to be really tricky and I, for one, never done it | 20:14 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Native Values Step 1. https://review.openstack.org/449826 | 20:14 |
| inc0 | I have theory how to do it but really I'd suggest using live migration to move vms out of host, clean host, deploy containers, move vms back | 20:15 |
| *** mnasiadka has joined #openstack-kolla | 20:15 | |
| inc0 | it's ovs contenerisation rather than neutron which you should be worrried about | 20:16 |
| sunnyve | I have a new VM , Will I be able to install openstack with kolla on my machine(it's ubuntu 16.04 VM) ?. I assume answer is yes, so where I can find the instructions to play with kolla ? | 20:16 |
| inc0 | https://docs.openstack.org/kolla-ansible/latest/ | 20:17 |
| sunnyve | Thanks | 20:17 |
| SamYaple | inc0: kfox1111 AIO galera is just the same as multinode but with and empty 'gcomm://' | 20:18 |
| SamYaple | inc0: if gcomm is empty, it will always start up | 20:19 |
| inc0 | mhm, ok, didn't know that | 20:19 |
| kfox1111 | SamYaple: ah. gotcha. | 20:19 |
| SamYaple | inc0: thats why i have the length check on the gcomm line | 20:19 |
| inc0 | allright | 20:19 |
| Walex | inc0: I changed that and same result. I'lld dopuble check | 20:19 |
| SamYaple | basically empty gcomm is --wsrep-new-cluster (more or less) | 20:19 |
| *** mnasiadka has quit IRC | 20:20 | |
| Walex | inc0: I don't get though why it works where it works: http://paste.ubuntu.com/25582020/ | 20:20 |
| Walex | inc0: the instance directory is owned by 42436:42436 buit the files within by 42427:42427 except for 'disk.info' | 20:21 |
| SamYaple | egonzalez: you should move to mariadb 10.2 (or at least 10.1). since 10.1 galera is compiled into mariadb-server and is not a seperate install | 20:21 |
| SamYaple | mariadb 10.0 is i EOL i think | 20:21 |
| inc0 | Walex: since you redeployed, but I bed you didn't cleanup, I'm not sure if libvirt took new change;) | 20:21 |
| Walex | inc0: I didn't cleanup! Double checking inside the container | 20:22 |
| inc0 | kolla-ansible destroy is best cleanup | 20:22 |
| inc0 | kolla-ansible reconfigure should be fine | 20:22 |
| *** sunnyve has quit IRC | 20:26 | |
| Walex | same sen from inside: http://paste.ubuntu.com/25582049/ http://paste.ubuntu.com/25582045/ | 20:27 |
| Walex | I'll try reconfigure, then destroy | 20:28 |
| *** kbaegis has quit IRC | 20:29 | |
| Walex | part of the story is that what the various 'kolla-ansible' commands actually do is not very clear from the docs... | 20:31 |
| *** ansmith has quit IRC | 20:35 | |
| *** mnasiadka has joined #openstack-kolla | 20:36 | |
| kfox1111 | uh.... where did docker go? | 20:38 |
| kfox1111 | 2017-09-20 20:23:45.557883 | No package docker-1.12.6-16.el7.centos available. | 20:38 |
| kfox1111 | centos extras has docker-1.12.6-48.git0fdc778.el7.centos.x86_64.rpm ... | 20:40 |
| *** mnasiadka has quit IRC | 20:40 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Drop pycrypto from utility genpwd.py https://review.openstack.org/484946 | 20:44 |
| *** csuttles has quit IRC | 20:47 | |
| Walex | kfox1111: Scientific Linux 7.4 had this available for Docker: http://paste.ubuntu.com/25582175/ | 20:49 |
| Walex | yesterday I mean | 20:50 |
| *** rwsu has quit IRC | 20:50 | |
| *** jistr has quit IRC | 20:52 | |
| *** lakerzhou has quit IRC | 20:56 | |
| *** mnasiadka has joined #openstack-kolla | 20:56 | |
| *** jascott1 has quit IRC | 20:59 | |
| *** pcaruana has quit IRC | 20:59 | |
| *** jascott1 has joined #openstack-kolla | 20:59 | |
| *** jascott1 has quit IRC | 21:00 | |
| *** mgoddard has quit IRC | 21:00 | |
| *** mnasiadka has quit IRC | 21:00 | |
| *** jheroux has quit IRC | 21:00 | |
| *** rwsu has joined #openstack-kolla | 21:03 | |
| *** kbaegis has joined #openstack-kolla | 21:05 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Drop pycrypto from utility genpwd.py https://review.openstack.org/484946 | 21:06 |
| iniazi | egonzalez: ok, panko is failing at "TASK [ceilometer : Copying over panko.conf]" with following fatal error on the controllers: "FAILED! => {"failed": true, "msg": "'service_name' is undefined"}" | 21:08 |
| *** jascott1 has joined #openstack-kolla | 21:08 | |
| iniazi | took a little while (took a little break) and also the bootstrap-servers broke my docker setup-- docker wouldn't start up on all the nodes. reverted my old docker config and documented, and restarted deploy | 21:09 |
| *** lrensing has quit IRC | 21:12 | |
| *** jascott1 has quit IRC | 21:12 | |
| egonzalez | iniazi i see, is missing a var used in panko.conf https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/ceilometer/tasks/config.yml#L87 | 21:13 |
| egonzalez | iniazi https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/zun/tasks/config.yml#L24 | 21:14 |
| *** jascott1 has joined #openstack-kolla | 21:14 | |
| egonzalez | inc0 just deployed with images in current tarball registry over centos 7.4 with kvm and all works with no issue locally | 21:16 |
| iniazi | ok, but so how do I fix it? i see the same config.yml on my kolla-ansible install on the server. does this get put into panko.conf? | 21:16 |
| egonzalez | iniazi add the service_name var in zun role to ceilometer | 21:17 |
| *** mnasiadka has joined #openstack-kolla | 21:17 | |
| egonzalez | just after service_require_panko_conf var | 21:17 |
| egonzalez | iniazi deploy again, not need to destroy in this case | 21:18 |
| *** rhallisey has quit IRC | 21:18 | |
| *** jascott1 has quit IRC | 21:19 | |
| iniazi | uhh, I'm not exactly sure where I'd find the zun roles | 21:20 |
| *** lrensing has joined #openstack-kolla | 21:21 | |
| *** mnasiadka has quit IRC | 21:21 | |
| iniazi | i'm an ansible n00b | 21:21 |
| egonzalez | iniazi add service_name: "{{ item.key }}" in https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/ceilometer/tasks/config.yml#L85 | 21:22 |
| egonzalez | iniazi http://paste.openstack.org/show/621570/ | 21:23 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Native Values Step 1. https://review.openstack.org/449826 | 21:26 |
| *** jascott1 has joined #openstack-kolla | 21:27 | |
| kfox1111 | jascott1: looks like its working. there is one template that is having an issue but all the rest are working. so just gotta figure out why. | 21:28 |
| kfox1111 | maybe a simple typo or something. | 21:28 |
| *** lucasxu has quit IRC | 21:29 | |
| *** jascott1 has quit IRC | 21:29 | |
| *** dave-mccowan has quit IRC | 21:32 | |
| *** jascott1 has joined #openstack-kolla | 21:33 | |
| iniazi | also, in 4.0.2 and 5.0.0, i see this warning for "Copying over telegraf plugin files" : [WARNING]: Unable to find '/usr/share/kolla-ansible/ansible/roles/telegraf/templates/config' in expected paths. (7 times, maybe per host) | 21:34 |
| iniazi | egonzalez, many thanks again... I'm redeploying now | 21:34 |
| *** jascott1 has quit IRC | 21:35 | |
| egonzalez | iniazi telegraf fails or skip the task? | 21:36 |
| *** mnasiadka has joined #openstack-kolla | 21:36 | |
| egonzalez | maybe a skip missing is needed here https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/telegraf/tasks/config.yml#L47 | 21:37 |
| *** csuttles has joined #openstack-kolla | 21:37 | |
| *** jascott1 has joined #openstack-kolla | 21:39 | |
| *** csuttles has quit IRC | 21:39 | |
| *** mnasiadka has quit IRC | 21:41 | |
| rwellum | kfox1111: on a running OS, kolla-k8s, if I cannot access the console because "Firefox can’t find the server at nova-novncproxy." - And I manually replace "nova-novncproxy" with the ip address of my host and it works - is this a config missing somewhere in my setup? | 21:42 |
| *** lakerzhou has joined #openstack-kolla | 21:43 | |
| *** jascott1 has quit IRC | 21:43 | |
| kfox1111 | hmm.... | 21:47 |
| kfox1111 | we don't currently have a gate check on nova-novncproxy. so its entirely possible it got broken at some point. | 21:48 |
| kfox1111 | that feels like it is advertising the internal svc name rather then the external name. | 21:48 |
| kfox1111 | I think that is a nova.conf option maybe? | 21:49 |
| rwellum | Yeah could be - was looking through gate code for clues | 21:50 |
| *** dardelean has quit IRC | 21:50 | |
| *** jamesbenson has quit IRC | 21:50 | |
| kfox1111 | rwellum: that the url that ends in /vnc_auto.html, or something different? | 21:50 |
| *** jascott1 has joined #openstack-kolla | 21:50 | |
| kfox1111 | we need a solenium test for the novnc stuff. | 21:50 |
| kfox1111 | to really work right though, I think we'd need a way to do a simple OCR to see if we show a login prompt. which I think is non trivial. | 21:51 |
| *** dardelean_ has joined #openstack-kolla | 21:51 | |
| kfox1111 | but maybe we could get away with just taking a screenshot and seeing if it is non black or something. | 21:51 |
| rwellum | kfox111 https://www.irccloud.com/pastebin/aWFNEGoC/ | 21:52 |
| rwellum | The addr is correct | 21:52 |
| kfox1111 | and novncproxy_base_url = http://10.240.43.201:6080/vnc_auto.html | 21:53 |
| kfox1111 | is the external address or internal? | 21:53 |
| rwellum | It's external | 21:53 |
| kfox1111 | hmm... ok. | 21:53 |
| kfox1111 | what did the url lool like that you tweaked? | 21:53 |
| rwellum | And like I said if I manually replace with the ip, I am straight into the console. | 21:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/504645 | 21:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/485905 | 21:54 |
| rwellum | http://10.240.43.201:6080/vnc_auto.html?token=7421905b-ab8c-4896-a44b-b34b0d9817d3&title=demo1(f5fa9ecf-ca74-4ccc-bd03-7c3c320ab357) | 21:54 |
| rwellum | kfox1111: ^^^ | 21:55 |
| *** jascott1 has quit IRC | 21:55 | |
| kfox1111 | so it was: | 21:55 |
| kfox1111 | http://nova-novnc:6080/vnc_auto.html?token=7421905b-ab8c-4896-a44b-b34b0d9817d3&title=demo1(f5fa9ecf-ca74-4ccc-bd03-7c3c320ab357) | 21:55 |
| kfox1111 | before? | 21:55 |
| rwellum | http://nova-novncproxy:6080/vnc_auto.html?token=7421905b-ab8c-4896-a44b-b34b0d9817d3&title=demo1(f5fa9ecf-ca74-4ccc-bd03-7c3c320ab357) | 21:55 |
| rwellum | One more nova | 21:55 |
| *** lrensing has quit IRC | 21:55 | |
| kfox1111 | k hmm.... | 21:56 |
| iniazi | egonzalez: it goes on.. so I guess its skipping? it doesn't say 'skipping' but deploy finishes fine | 21:56 |
| egonzalez | iniazi that option does not have option to skip hhehe, if no file is found nothing happens | 21:56 |
| *** mnasiadka has joined #openstack-kolla | 21:57 | |
| kfox1111 | rwellum: looks like its wrongly being overridden in: | 21:58 |
| kfox1111 | kolla-kubernetes/helm/microservice/nova-compute-daemonset/templates/nova-compute.yaml | 21:58 |
| iniazi | thats cool. | 21:58 |
| rwellum | Bug you think kfox1111 ? | 22:00 |
| kfox1111 | yes. two seperate ones. | 22:00 |
| kfox1111 | whoever developed that assumed it required the compute nodes ip address. which isn't correct. | 22:00 |
| kfox1111 | the other is there is no gate test. | 22:01 |
| kfox1111 | whatever's not tested is broken. | 22:01 |
| egonzalez | inc0 maybe i found why in kolla centos7.4 works and why in ansible not, kolla does not pin docker version to 1.12 (install 17.06) while ansible does. this explains why i cannot reproduce locally as also have 17 | 22:01 |
| iniazi | btw, that service_name worked... it went past it fine. still deploying... | 22:01 |
| *** mnasiadka has quit IRC | 22:01 | |
| egonzalez | *re gates | 22:01 |
| kolla-slack | <jascott1> @kfox1111 glad to hear | 22:02 |
| egonzalez | iniazi cool, you have a fix to push ;) | 22:02 |
| kfox1111 | @jascott1 yeah. thanks for fixing the bug. any idea when the next version will land with it? | 22:02 |
| *** lakerzhou has quit IRC | 22:02 | |
| kolla-slack | <jascott1> its 2.7 and maybe 2 weeksish? | 22:03 |
| kfox1111 | rwellum: try pulling crudini --set /nova/nova.conf vnc novncproxy_base_url {{ $novncProxyBaseUrl }}; from the chart and relaunching. | 22:03 |
| iniazi | egonzalez, haven't used github much. should fork to my account, ci the fix, and then create a PR? | 22:03 |
| kfox1111 | @jascott1: ok. thanks. | 22:03 |
| egonzalez | iniazi nope, openstack does not use gihub for that | 22:03 |
| kfox1111 | so, probably shortly after 1.8. | 22:03 |
| kfox1111 | that would make sence. | 22:03 |
| egonzalez | iniazi https://docs.openstack.org/infra/manual/developers.html | 22:04 |
| egonzalez | iniazi will need to create account in openstack, launchpad | 22:04 |
| egonzalez | iniazi take your time and ask if need some help | 22:05 |
| iniazi | I'll read through that doc | 22:05 |
| iniazi | thanks | 22:05 |
| kfox1111 | sdake: https://review.openstack.org/#/c/484946/ looks like it unwedges the gate. | 22:06 |
| kfox1111 | can I get a second review please? | 22:07 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: DONOMERGE: test if docker 1.12 breaks gate with centos 7.4 https://review.openstack.org/505832 | 22:12 |
| rwellum | kfox1111: I've made the change to the yaml. Now cleanest way to restart nova-compute and pick up this change? | 22:13 |
| kfox1111 | currently, to helm upgrade the package, | 22:14 |
| kfox1111 | and then delete one of the nova-compute pods to test on. | 22:14 |
| kfox1111 | we probably should add rolling upgrade attributes to the daemonsets, now that they are things.... | 22:14 |
| kfox1111 | thats another thing you could work in if your interested. | 22:15 |
| *** mnasiadka has joined #openstack-kolla | 22:17 | |
| *** lvdombrk has quit IRC | 22:19 | |
| *** lvdombrk has joined #openstack-kolla | 22:19 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/504645 | 22:21 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/485905 | 22:21 |
| *** mnasiadka has quit IRC | 22:21 | |
| kfox1111 | sdake: thanks. | 22:22 |
| sdake | kfox1111 no problem - looks liek the patch was fixed to be compiatlbe | 22:22 |
| sdake | and that removed the need for the strip | 22:22 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Drop pycrypto from utility genpwd.py https://review.openstack.org/484946 | 22:24 |
| kfox1111 | yup. | 22:25 |
| rwellum | kfox1111: helm init --upgrade ? | 22:25 |
| kfox1111 | no. | 22:28 |
| kfox1111 | helm upgrade nova-compute kolla/nova-compute | 22:28 |
| kfox1111 | or sometomething like that. | 22:28 |
| *** Croata2 has quit IRC | 22:30 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Native Values Step 1. https://review.openstack.org/449826 | 22:32 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP init-container from json to yaml. https://review.openstack.org/505761 | 22:33 |
| *** mnasiadka has joined #openstack-kolla | 22:37 | |
| rwellum | thanks kfox1111 - testing it now. | 22:38 |
| rwellum | No obvious improvement - wondering if I need to restart horizon pod too? | 22:40 |
| kfox1111 | hmm.... | 22:40 |
| kfox1111 | and you replaced the nova compute where the vm is running? | 22:40 |
| rwellum | yes | 22:40 |
| kfox1111 | hmm... | 22:41 |
| kfox1111 | I could have swarn thats where its used... lets see... | 22:41 |
| rwellum | What I am not sure is, I modified: microservice/nova-compute-daemonset/templates/nova-compute.yaml | 22:41 |
| rwellum | But restarted the service | 22:41 |
| *** mnasiadka has quit IRC | 22:41 | |
| rwellum | helm upgrade nova-compute kolla-kubernetes/helm/service/nova-compute | 22:41 |
| kfox1111 | maybe exec into the nova-compute pod and douuble check /etc/nova/nova.conf has the right value? | 22:42 |
| kfox1111 | and then deleted the nova-compute pod? | 22:42 |
| kfox1111 | that upgrade line looks ok I think. | 22:42 |
| rwellum | I did reset | 22:47 |
| rwellum | but... | 22:47 |
| rwellum | https://www.irccloud.com/pastebin/cLnjUB6m/ | 22:47 |
| kfox1111 | resetting the container won't apply the config change. | 22:48 |
| kfox1111 | its gotta be deleted. | 22:48 |
| rwellum | I mean I did a kubectl delete pod | 22:48 |
| kfox1111 | ah. ok. | 22:48 |
| kfox1111 | hmm... | 22:48 |
| kfox1111 | do a kubectl get pod <nova-compute-name> -o yaml on it plz. | 22:48 |
| rwellum | that's a goodly sized file - you want it all? | 22:53 |
| *** jistr has joined #openstack-kolla | 22:53 | |
| kfox1111 | maybe just the annotation section? | 22:53 |
| *** mnasiadka has joined #openstack-kolla | 22:57 | |
| rwellum | https://www.dropbox.com/s/6d24zxvqw9plxpu/compute.yaml?dl=0 | 22:58 |
| rwellum | kfox1111: ^^^ | 22:58 |
| kfox1111 | the baseurl override's still showing up in it... | 22:59 |
| kfox1111 | crudini --set /nova/nova.conf vnc novncproxy_base_url http://nova-novncproxy:6080/vnc_auto.html; | 22:59 |
| kfox1111 | can you kubectl get daemonset nova-compute -o yaml | 23:00 |
| kfox1111 | and look in its attributes for the same thing. | 23:00 |
| *** dardelean_ has quit IRC | 23:01 | |
| *** mnasiadka has quit IRC | 23:01 | |
| vhosakot | kubectl fix_everything | 23:02 |
| vhosakot | ;) | 23:02 |
| *** jistr has quit IRC | 23:02 | |
| kfox1111 | ;) | 23:03 |
| rwellum | kfox1111: that one didn't work | 23:03 |
| kfox1111 | kubectl get daemonset | 23:03 |
| kfox1111 | and see what its named. not remembering off hand. | 23:03 |
| *** jistr has joined #openstack-kolla | 23:04 | |
| rwellum | https://www.irccloud.com/pastebin/LGIPLkW6/ | 23:04 |
| rwellum | I checked and I def. removed from the microservice yaml | 23:05 |
| rwellum | "novncproxy_base_url http://nova-novncproxy:6080/vnc_auto.html; '" | 23:06 |
| kfox1111 | hmm... | 23:06 |
| rwellum | Wondering what I did wrong... | 23:07 |
| kfox1111 | seems like the helm upgrade didn't take. | 23:07 |
| rwellum | I did: helm upgrade nova-compute kolla-kubernetes/helm/service/nova-compute | 23:07 |
| kfox1111 | and it returned 0 with no output? | 23:08 |
| rwellum | https://www.irccloud.com/pastebin/GWMMnPgS/ | 23:08 |
| kfox1111 | and the base url stuff is still in the daemonset after that? | 23:09 |
| rwellum | Yeah | 23:09 |
| kfox1111 | I wonder if helm's not actually updating it as its just an attribute change... | 23:10 |
| kfox1111 | that would be.... unfortunate... | 23:10 |
| *** egonzalez has quit IRC | 23:10 | |
| kfox1111 | not sure if it would work or not, but could delete daemonset nova-compute and upgrade again. | 23:11 |
| rwellum | I guess I have to redeploy - or deploy another - making sure I change the chart before initially deploying. | 23:11 |
| kfox1111 | maybe that would work. if not, you'd have to delete the instance of the chart and install it again? | 23:11 |
| rwellum | How do I delete the daemonset? | 23:12 |
| kfox1111 | kubectl delete daemonset <daemonset-name> -n kolla | 23:12 |
| rwellum | ok done - and testing | 23:15 |
| rwellum | https://www.irccloud.com/pastebin/fvIG65wh/ | 23:16 |
| rwellum | Still seems to be in the daemonset | 23:16 |
| *** mnasiadka has joined #openstack-kolla | 23:16 | |
| kfox1111 | so you deleted the daemonset, then upgraded and then the crudini showed b ack up in the daemonset code? | 23:19 |
| rwellum | Yes. Actually I also deleted the pod | 23:19 |
| kfox1111 | weirded out by that... | 23:20 |
| rwellum | :( | 23:20 |
| rwellum | Guess I | 23:20 |
| rwellum | ll redeploy | 23:21 |
| *** mnasiadka has quit IRC | 23:21 | |
| rwellum | (tomorrow) | 23:21 |
| kfox1111 | maybe delete the nova-compute chart and redeploy jus that one thing. | 23:23 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: DONOMERGE: test if docker 1.12 breaks gate with centos 7.4 https://review.openstack.org/505832 | 23:27 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP init-container from json to yaml. https://review.openstack.org/505761 | 23:28 |
| *** mnasiadka has joined #openstack-kolla | 23:37 | |
| *** mnasiadka has quit IRC | 23:41 | |
| *** vhosakot has quit IRC | 23:42 | |
| *** hongbin has quit IRC | 23:44 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Native Values Step 1. https://review.openstack.org/449826 | 23:49 |
| *** itlinux has joined #openstack-kolla | 23:53 | |
| *** mnasiadka has joined #openstack-kolla | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!