Wednesday, 2020-04-22

*** threestrands has joined #openstack-kolla00:04
*** cdearborn has quit IRC00:06
*** verdurin has quit IRC00:07
*** verdurin has joined #openstack-kolla00:14
*** born2bake has quit IRC00:18
openstackgerritMerged openstack/kayobe master: Add support for stopping overcloud services  https://review.opendev.org/71519400:28
openstackgerritMerged openstack/kayobe master: CI: Test SSH connectivity to deployed instances  https://review.opendev.org/70760400:28
*** happyhemant has quit IRC00:35
openstackgerritMerged openstack/kayobe master: Don't use become for Kolla Ansible  https://review.opendev.org/71631400:50
openstackgerritMerged openstack/kayobe master: Avoid writing out requirements.txt in kolla-ansible role  https://review.opendev.org/70785300:50
openstackgerritJames Kirsch proposed openstack/kolla master: Add stunnel container for Glance service  https://review.opendev.org/72085000:54
*** zhanglong has joined #openstack-kolla01:25
srinI installed kolla-ansible on centos 8 a week ago and I got errors about yum install python2, but vanilla minimal centos8 hasn't python2 package so I cannot install it01:28
*** bersace has joined #openstack-kolla01:43
openstackgerritJeffrey Zhang proposed openstack/kolla stable/queens: nova-libvirt: add UEFI packages to support UEFI instances  https://review.opendev.org/72176401:46
*** seco has joined #openstack-kolla02:16
srinoh I misunderstanded02:26
*** jcmdln has joined #openstack-kolla02:28
*** oncall-pokemon has quit IRC02:28
*** seco has quit IRC02:31
*** ricolin has joined #openstack-kolla02:33
openstackgerritya.wang proposed openstack/kolla-ansible stable/train: Fix that cyborg conductor failed to communicate with placement  https://review.opendev.org/72177002:47
openstackgerritya.wang proposed openstack/kolla-ansible master: Fix that cyborg agent failed to start privsep daemon.  https://review.opendev.org/72113902:59
srinis master removed supports about installing ceph?03:01
*** bersace has quit IRC03:05
*** negronjl has quit IRC04:22
*** kevinz has quit IRC04:24
*** ykarel|away is now known as ykarel04:25
*** negronjl has joined #openstack-kolla04:25
*** evrardjp has quit IRC04:35
*** evrardjp has joined #openstack-kolla04:35
*** cah_link has joined #openstack-kolla05:09
*** skramaja has joined #openstack-kolla05:39
mnasiadkasri_: yes05:47
*** ricolin has quit IRC05:51
*** ricolin has joined #openstack-kolla05:53
sri_morning, mnasiadka06:03
sri_srin: ^^06:03
*** seco has joined #openstack-kolla06:31
*** jbadiapa has joined #openstack-kolla06:42
yoctozeptodcapone2004: yes, must be from git atm06:43
*** threestrands has quit IRC06:53
*** threestrands has joined #openstack-kolla06:54
*** threestrands has quit IRC06:55
*** threestrands has joined #openstack-kolla06:55
*** threestrands has quit IRC06:56
*** threestrands has joined #openstack-kolla06:56
*** threestrands has quit IRC06:58
*** threestrands has joined #openstack-kolla06:58
*** threestrands has quit IRC06:59
*** threestrands has joined #openstack-kolla06:59
*** dswebb has quit IRC07:00
*** threestrands has quit IRC07:01
*** born2bake has joined #openstack-kolla07:01
*** threestrands has joined #openstack-kolla07:01
hrwmorning07:01
*** threestrands has quit IRC07:02
*** seco has quit IRC07:02
*** threestrands has joined #openstack-kolla07:02
*** threestrands has quit IRC07:04
*** threestrands has joined #openstack-kolla07:04
*** seco has joined #openstack-kolla07:05
*** threestrands has quit IRC07:05
*** threestrands has joined #openstack-kolla07:06
*** bengates has joined #openstack-kolla07:06
*** ricolin has quit IRC07:06
*** bengates has quit IRC07:06
*** threestrands has quit IRC07:07
*** bengates has joined #openstack-kolla07:07
*** threestrands has joined #openstack-kolla07:07
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: use Ansible 2.9+ binary packages  https://review.opendev.org/72125107:08
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: kolla-toolbox: use only binary packages  https://review.opendev.org/72125207:08
*** threestrands has quit IRC07:08
hrwrebased07:08
*** threestrands has joined #openstack-kolla07:08
*** threestrands has quit IRC07:10
*** threestrands has joined #openstack-kolla07:10
*** threestrands has quit IRC07:11
*** threestrands has joined #openstack-kolla07:12
*** tonythomas has joined #openstack-kolla07:12
*** threestrands has quit IRC07:13
*** threestrands has joined #openstack-kolla07:13
*** ricolin has joined #openstack-kolla07:13
*** threestrands has quit IRC07:14
*** threestrands has joined #openstack-kolla07:14
*** threestrands has quit IRC07:16
*** threestrands has joined #openstack-kolla07:16
born2bakejust letting you know guys that this issue related to kvm permissions happens when you build images, in my case amphora,trove images, on that host07:16
born2bakeI assume that kvm, qemu and etc are installed on the host and they somehow cant properly work together with libvirt docker container07:17
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: kolla-toolbox: use only binary packages  https://review.opendev.org/72125207:17
*** threestrands has quit IRC07:17
*** threestrands has joined #openstack-kolla07:18
hrwmgoddard: your comments were valuable. applied changes, added reno07:18
*** threestrands has quit IRC07:19
*** threestrands has joined #openstack-kolla07:19
*** threestrands has quit IRC07:20
*** threestrands has joined #openstack-kolla07:21
*** dswebb has joined #openstack-kolla07:21
*** threestrands has quit IRC07:22
*** threestrands has joined #openstack-kolla07:22
*** threestrands has quit IRC07:23
*** threestrands has joined #openstack-kolla07:24
*** rpittau|afk is now known as rpittau07:24
*** threestrands has quit IRC07:25
*** threestrands has joined #openstack-kolla07:25
*** threestrands has quit IRC07:26
*** threestrands has joined #openstack-kolla07:27
*** threestrands has quit IRC07:28
*** threestrands has joined #openstack-kolla07:28
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Drop handling of 'easy_install'  https://review.opendev.org/72129007:28
hrwrebased07:28
*** threestrands has quit IRC07:29
*** threestrands has joined #openstack-kolla07:30
*** threestrands has quit IRC07:31
*** threestrands has joined #openstack-kolla07:31
*** threestrands has quit IRC07:32
*** threestrands has joined #openstack-kolla07:33
*** threestrands has quit IRC07:34
*** threestrands has joined #openstack-kolla07:34
*** threestrands has quit IRC07:35
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix that cyborg agent failed to start privsep daemon.  https://review.opendev.org/72113907:35
*** threestrands has joined #openstack-kolla07:36
*** threestrands has quit IRC07:37
*** threestrands has joined #openstack-kolla07:37
*** threestrands has quit IRC07:38
*** threestrands has joined #openstack-kolla07:39
*** threestrands has quit IRC07:40
*** threestrands has joined #openstack-kolla07:40
*** threestrands has quit IRC07:41
*** threestrands has joined #openstack-kolla07:41
*** threestrands has quit IRC07:43
*** threestrands has joined #openstack-kolla07:43
*** threestrands has quit IRC07:44
*** threestrands has joined #openstack-kolla07:45
*** threestrands has quit IRC07:46
*** threestrands has joined #openstack-kolla07:46
*** threestrands has quit IRC07:47
*** threestrands has joined #openstack-kolla07:47
*** threestrands has quit IRC07:49
*** threestrands has joined #openstack-kolla07:49
mgoddardthanks hrw :)07:50
*** threestrands has quit IRC07:50
mgoddardmorning07:50
mgoddardI'll be out until the meeting today07:50
*** threestrands has joined #openstack-kolla07:51
hrwmgoddard: c8/train will see update later today07:51
*** threestrands has quit IRC07:52
*** threestrands has joined #openstack-kolla07:52
*** threestrands has quit IRC07:53
*** threestrands has joined #openstack-kolla07:54
*** vishalmanchanda has joined #openstack-kolla07:54
*** threestrands has quit IRC07:55
*** threestrands has joined #openstack-kolla07:55
*** threestrands has quit IRC07:56
*** threestrands has joined #openstack-kolla07:57
*** threestrands has quit IRC07:58
*** threestrands has joined #openstack-kolla07:58
*** jopdorp has joined #openstack-kolla07:59
*** threestrands has quit IRC07:59
*** threestrands has joined #openstack-kolla08:00
*** threestrands has quit IRC08:01
*** threestrands has joined #openstack-kolla08:01
*** threestrands has quit IRC08:02
*** threestrands has joined #openstack-kolla08:02
yoctozeptomnasiadka: btw, did you notice wsrep problems are extremely rare nowadays?08:03
*** threestrands has quit IRC08:04
*** threestrands has joined #openstack-kolla08:04
*** threestrands has quit IRC08:05
*** threestrands has joined #openstack-kolla08:06
*** threestrands has quit IRC08:07
*** threestrands has joined #openstack-kolla08:07
*** threestrands has quit IRC08:08
dswebbmorning all, just a quick q / potential bug report.  The decision to run ovs 2.12 in the train deployment, is that something that comes from upstream (eg neutron project) or internally within kolla?  Our experience with kollas train deployment and ovs 2.12 has been riddled with issues (show stopper type issues).  Once we downgraded to 2.11 for ovs_vswitchd all of our problems went away.08:10
yoctozeptodswebb: upstream (distro)08:16
yoctozeptowhat are the issues? distros should know them08:17
dswebbhttps://bugzilla.redhat.com/show_bug.cgi?id=1782834, https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/185222108:17
openstackbugzilla.redhat.com bug 1782834 in openvswitch "Changing protocols in Bridge table doesn't take effect" [High,New] - Assigned to aconole08:17
yoctozepto(and obviously it works for us (TM) )08:17
openstackLaunchpad bug 1852221 in openvswitch (Ubuntu) "ovsdb-server needs to be restarted after adding protocols to bridges" [Undecided,Confirmed]08:17
yoctozeptoso distros know it08:19
yoctozeptohmm, why not fix ovsdb-server, I see canonical charm team is doing some huge workaround for it, lol08:19
yoctozeptodswebb: thanks for the heads up08:20
yoctozeptodswebb: did you manage to get this issue with kolla-ansible?08:20
dswebbthose two showed up in our deployment, but the main issue we had (we patched those with with the upstream patch from master for neutron).  but the main issue we kept hitting was weird OVS issues where security group rules on an instance with a floating ips just not working really.08:20
yoctozeptowas it new deployment or upgrade?08:21
dswebbyes, new deployment08:21
yoctozeptohmm08:21
mnasiadkayoctozepto: didn't have time in couple of last days to check up anything in kolla08:21
dswebbwe completely re-deployed last week08:21
dswebbthe ovs SG issue was almost like ovs was failing closed, but without any sort of RMQ communication issues or anything like that08:23
yoctozeptodswebb: well, ovs doesn't speak rmq08:25
yoctozeptoit's neutron agent that would08:25
dswebbyeah, that is what I meant, there were no communication issues from the agent that I could see08:25
yoctozeptodswebb: I marked the launchpad's bug for kolla too08:25
dswebbcool08:25
yoctozeptodswebb: so these two are the same08:25
dswebbyes08:25
yoctozeptodswebb: and is the other one you are speaking of reported somewhere?08:26
yoctozeptowould be good to include it in that bug report, since it's still 2.12 regression08:26
dswebbno, we can raise it, would it go to neutron bug report08:27
dswebb?08:27
dswebbalso, the fix for the version mismatch issue is here: https://review.opendev.org/#/c/698715/908:27
patchbotpatch 698715 - neutron - Set OpenFlow 1.0, 1.3 and 1.4 by default on bridges (MERGED) - 9 patch sets08:27
cosmicsoundgood day08:28
cosmicsoundhttps://bugs.launchpad.net/kolla/+bug/1874129 any idea what could cause this?08:31
openstackLaunchpad bug 1874129 in kolla "cloudkitty zombie processes" [Undecided,New]08:31
*** abdysn has joined #openstack-kolla08:40
openstackgerritMichal Nasiadka proposed openstack/kolla master: Add mariadb_clustercheck image  https://review.opendev.org/71021708:46
*** e0ne has joined #openstack-kolla08:56
*** zhanglong has quit IRC09:02
*** k_mouza has joined #openstack-kolla09:03
*** zhanglong has joined #openstack-kolla09:09
yoctozeptomnasiadka: we are not using upstream mariadb in c8 now?09:11
yoctozeptofeeling confused09:11
mnasiadkayoctozepto: no, I think only in Ubuntu?09:11
yoctozeptodswebb: yes, as long as it is 2.12-related09:11
yoctozeptodswebb: it's more like workaround, neutron wishes to revert it09:11
yoctozeptomnasiadka: crap09:11
yoctozepto(sorry for language)09:12
mnasiadkahttps://github.com/openstack/kolla/blob/master/docker/base/sources.list.ubuntu#L5109:12
mnasiadkayoctozepto: ^^09:12
yoctozeptohttps://ethercalc.openstack.org/kolla-infra-service-matrix looks gone argh09:14
yoctozeptoubuntu is 10.3 mariadb upstream09:14
yoctozeptodebian is 10.3 mariadb distro (buster)09:14
yoctozeptoc8 is 10.3 mariadb distro (c8 stream current)09:16
yoctozeptoI wonder if they move09:16
*** ykarel is now known as ykarel|lunch09:18
yoctozeptoso focal in victoria will move us up to 10.4, hmm09:19
yoctozeptoanyways, these look pretty stable, as they are not revamping galera09:20
*** ricolin has quit IRC09:22
*** ricolin has joined #openstack-kolla09:23
*** ricolin has quit IRC09:24
*** ricolin has joined #openstack-kolla09:25
*** ricolin has quit IRC09:25
*** ricolin_ has joined #openstack-kolla09:26
hrwspeaking of stuff...09:27
hrwas we are now at rmq 3.8 in ussuri I would like to move centos8 to use in-distro version of erlang and rabbitmq09:27
hrwfor x86-64 and ppc64le it will give us rmq 3.8 version and erlang 2209:28
hrwaarch64 has to wait for centos 8.209:28
born2bakeyoctozepto just letting you know that this issue related to kvm permissions happens when you build images, in my case amphora,trove images, on the same host where you run compute.  I assume that kvm, qemu and etc are installed on the host and they somehow cant properly work together with libvirt docker container09:29
yoctozeptoborn2bake: qemu is together with libvirt in the container09:30
yoctozeptoborn2bake: kvm is part of kernel09:30
yoctozeptobut this is a good hint09:30
*** k_mouza has quit IRC09:31
yoctozeptomaybe something reconfigures something09:31
born2bakenot sure but I checked twice. Create instances - all good. Build amphora/trove images on the host. Create instances again - error.09:32
*** e0ne has quit IRC09:34
*** e0ne has joined #openstack-kolla09:35
*** ricolin_ is now known as ricolin09:36
yoctozeptoborn2bake: btw, what does the building? external programs?09:36
born2bakehttps://docs.openstack.org/octavia/latest/admin/amphora-image-build.html09:38
*** k_mouza has joined #openstack-kolla09:38
yoctozeptoborn2bake: uhm, so you did those instructions on host level?09:39
born2bakeyup09:39
yoctozeptohmm, there is a chance it somehow hijacked the kvm device09:41
yoctozeptomakes sense to add it to the report09:41
yoctozeptohrw: I think it makes sense; we wanted 3.8 for prometheus and to get better ipv609:43
yoctozeptoso it's what it is09:43
hrwyoctozepto: c8/aarch64 will stay broken for a while09:43
yoctozeptomy dream is to be able to drop rabbitmq but it is not happenning any time soon ;p09:43
*** dcapone2004 has quit IRC09:43
hrwyoctozepto: as we need centos 8.2 to get erlang on aarch6409:44
yoctozeptohrw: define 'a while'09:44
yoctozeptois it going to take months or days09:44
hrwyoctozepto: it is broken already09:44
yoctozeptohrw: then why ask09:44
*** ykarel|lunch is now known as ykarel09:45
hrwanyway it was useless question ;D09:46
hrwon c8 we use delorean repos. they depend on rmq38 repo from centos already09:46
hrwso my change just enables that repo officially.09:46
openstackgerritNick Jones proposed openstack/kolla-ansible master: [skydive] fix: Use Keystone backend to authenticate API users  https://review.opendev.org/71759609:46
hrwrabbitmq/erlang ones we have defined are not used at all09:47
yoctozeptook09:48
osmanlicilegimorning09:49
hrwhi osmanlicilegi09:49
osmanlicilegihow's going folks09:49
hrwosmanlicilegi: life sucks09:50
osmanlicilegiagree09:50
hrwhopefully then you die ;)09:51
*** evrardjp has quit IRC09:51
*** evrardjp has joined #openstack-kolla09:51
yoctozeptohttps://bugs.launchpad.net/kolla-ansible/+bug/187421809:54
openstackLaunchpad bug 1874218 in kolla-ansible "swift sanity checks broken" [Low,Triaged]09:54
*** zhanglong has quit IRC09:54
yoctozeptoso users tried enabling those lovely sanity checks09:54
yoctozeptomgoddard, mnasiadka: deprecate (remove) sanity checks?09:55
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: use Ansible 2.9+ binary packages  https://review.opendev.org/72125109:55
hrwyoctozepto: disabled now09:55
osmanlicilegihrw: today I have another topic which sucks too. it's surveys. I learned that we can get rich by stopping working for OpenStack and starting to do something related with surveys. people spend tens of dollars to ask more than 10 questions to people.09:57
osmanlicilegihere is the topic for todays meeting: https://ethercalc.openstack.org/46c7q4lnww6j09:57
yoctozeptohrw: ++09:58
hrwosmanlicilegi: we can also go and burn some 5G towers!09:58
osmanlicilegiaha good idea. let's occupy 5g!09:59
hrwI wonder how many neighbours noticed 'Testing 5G station' wifi network09:59
yoctozeptohrw: they died already09:59
yoctozeptocancer, autism and covid-1909:59
yoctozeptoAT ONCE09:59
* osmanlicilegi brb10:00
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CentOS: use RabbitMQ and Erlang from official repos  https://review.opendev.org/72180010:02
hrwread commit message please10:02
*** JangwonLee__ has joined #openstack-kolla10:04
*** gfidente|afk is now known as gfidente10:05
*** JangwonLee has joined #openstack-kolla10:06
*** JangwonLee_ has quit IRC10:07
*** JangwonLee_ has joined #openstack-kolla10:08
*** JangwonLee__ has quit IRC10:09
*** JangwonLee has quit IRC10:12
hrwthere are some other c8/aarch64 failures10:15
hrwwill work on them but first need some centos-release-* files (in progress)10:15
mnasiadkayoctozepto: deprecate swift :)10:22
*** k_mouza has quit IRC10:22
*** k_mouza has joined #openstack-kolla10:31
*** bengates has quit IRC10:33
*** bengates has joined #openstack-kolla10:34
*** abdysn has quit IRC10:39
*** priteau has joined #openstack-kolla10:39
*** cah_link has quit IRC10:41
born2bakeWhen creating multiple instances with volume, volume takes ages to create/stuck at downloading state. Thus, instances fail with error timeout msg. If I create 1 volume manually, there is no issues.10:41
born2bakecan someone suggest where to dig10:41
*** rpittau is now known as rpittau|bbl10:42
openstackgerritMerged openstack/kolla-ansible master: Drop handling of 'easy_install'  https://review.opendev.org/72129010:48
*** JangwonLee__ has joined #openstack-kolla10:59
*** dswebb has quit IRC11:01
*** JangwonLee_ has quit IRC11:02
*** k_mouza has quit IRC11:07
*** cah_link has joined #openstack-kolla11:23
*** k_mouza has joined #openstack-kolla11:31
*** k_mouza has quit IRC11:41
*** k_mouza has joined #openstack-kolla11:42
*** rpittau|bbl is now known as rpittau11:50
hrwc8/train gets closer11:52
*** prernadembla05 has joined #openstack-kolla11:56
*** wuchunyang has joined #openstack-kolla12:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/train: [WIP] switch to official CentOS 8 repositories  https://review.opendev.org/72132912:10
hrwall images built.12:11
hrwstill WIP as we wait for centos-release-* packages12:11
*** wuchunyang has quit IRC12:13
*** hjensas is now known as hjensas|afk12:15
*** wuchunyang has joined #openstack-kolla12:16
*** sorin-mihai has joined #openstack-kolla12:17
*** JangwonLee has joined #openstack-kolla12:19
sorin-mihaifor a deployment on a network with jumbo frames, is there anything that needs to be done? documentation is not clear for me regarding this. i can set the MTU on the VIP interface, but not on the other one, is that bein taken care by neutron somehow?12:20
*** wuchunyang has quit IRC12:20
*** JangwonLee__ has quit IRC12:22
*** JangwonLee_ has joined #openstack-kolla12:23
*** JangwonLee__ has joined #openstack-kolla12:24
*** JangwonLee has quit IRC12:27
*** wuchunyang has joined #openstack-kolla12:27
yoctozeptosorin-mihai: you might want to tinker with [DEFAULT]\global_physnet_mtu in neutron12:27
*** JangwonLee_ has quit IRC12:28
*** dougsz has joined #openstack-kolla12:30
*** wuchunyang has quit IRC12:31
sorin-mihaibased on openstack documentation, i guess it should go in /etc/kolla/config/neutron/{config.json,ml2_conf.ini} ?12:32
sorin-mihaiwhat about the docker daemon, does it need to be adjusted as well?12:33
kplant/etc/kolla/config/neutron.conf for [DEFAULT]/global_physnet_mtu12:33
kplantand /etc/kolla/config/neutron/ml2_conf.ini for [ml2]/path_mtu12:34
kplantthat, and the physical interfaces ofc, is all you have to change12:34
yoctozepto^ and path_mtu defaults to global_physnet_mtu12:34
yoctozeptoso no need to change both12:34
yoctozeptoin fact, you can limit this to neutron-server most likely, as it dictates network mtus12:35
yoctozeptoso /etc/kolla/config/neutron/neutron-server.conf would be minimal impact12:35
yoctozeptowe use host networking so docker is not to be touched12:35
hrwhow many of you do builds with separate logs?12:38
*** sorin-mihai has quit IRC12:38
openstackgerritMerged openstack/kayobe master: Fix seed VM provisioning on a remote seed hypervisor  https://review.opendev.org/71851812:38
*** hjensas|afk is now known as hjensas12:41
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Do not try to build skipped/unbuildable images  https://review.opendev.org/72184712:41
hrwsay goodbye to 'skipped' logs12:41
*** sorin-mihai has joined #openstack-kolla12:42
sorin-mihaiyoctozepto, thanks12:48
sorin-mihaiand kplant ^_^12:48
kplantyw12:51
*** ykarel is now known as ykarel|afk13:00
hrwmgoddard: as you use bifrost... can you patch it to not try to install debootstrap on c8?13:03
hrwmgoddard: last time it was present in centos was queens era - in cloud/openstack-queens/ repo13:03
*** abdysn has joined #openstack-kolla13:09
*** cah_link1 has joined #openstack-kolla13:18
*** cah_link has quit IRC13:20
*** cah_link1 is now known as cah_link13:20
* osmanlicilegi back13:22
*** jimcrowleyibm has joined #openstack-kolla13:31
*** abdysn has quit IRC13:32
*** Abdallahyas has joined #openstack-kolla13:32
mnasiadkamgoddard: can't join todays meeting13:39
* mgoddard strips mnasiadka of core status13:39
mnasiadkafinally, can I go home? :D13:39
kplanteye for an eye13:39
mgoddardmnasiadka: if you aren't home, where have you been? :)13:42
mgoddardhrw: debootstrap exists in epel on c8. We just need to enable it13:43
*** e0ne has quit IRC13:44
yoctozeptomnasiadka: policy violation detected, initiating destruction processes13:44
*** e0ne has joined #openstack-kolla13:44
yoctozeptohrw: if you find some leisure time, it would be good to fill in https://ethercalc.openstack.org/kolla-infra-service-matrix :-)13:45
*** TrevorV has joined #openstack-kolla13:50
*** ykarel|afk is now known as ykarel13:51
mgoddardyoctozepto: that page would be more useful in docs13:52
*** e0ne has quit IRC14:00
*** e0ne has joined #openstack-kolla14:01
osmanlicilegimgoddard: I have to skip today's meeting cause there will a lockdown here for the next 4 days. I need to visit pharmacy and market. You can find my survey tools comparison at https://ethercalc.openstack.org/46c7q4lnww6j. If there's anything missing for this task, just ping me.14:02
*** dking has quit IRC14:07
*** zhanglong has joined #openstack-kolla14:08
*** Fl1nt has joined #openstack-kolla14:10
Fl1ntgood afternoon dear people of the chat !14:10
Fl1ntquestion, does this warning: "If you are using ceph-ansible - please copy ceph.client.cinder.keyring as /etc/kolla/config/nova/ceph.client.nova.keyring" from https://docs.openstack.org/kolla-ansible/train/reference/storage/external-ceph-guide.html apply to any release of k-a with c-a ? If so, why do we still need a nova keyring ?14:13
*** zhanglong has quit IRC14:13
*** zhanglong has joined #openstack-kolla14:15
Fl1ntbecause for now, even if I copy the content of my ceph.client.cinder.keyring within ceph.client.nova.keyring I've got a ErrNO 13 Permission denied which is quite obvious as we try to connect() with the nova user. If we need to also adapt the rbd_user to cinder, then what's the point of having nova one still ?14:15
Fl1ntbtw14:15
Fl1ntas ceph is smart enough to compare its username and id to what should be there14:15
Fl1ntif you don't fill the ceph.conf with [client.nova] keyring = <PATH> that won't work neither14:16
mgoddardthanks for doing that osmanlicilegi14:17
Fl1ntso, I'm a bit confuse at that moment because of that warning, any thoughts on this ?14:18
Fl1nthum... and that commit completely blow the doc ^^ https://opendev.org/openstack/kolla-ansible/commit/a9fecc41a31749c97e89ba6b66c9c46714fa6da814:20
Fl1ntI'll try with that train patch14:20
*** Abdallahyas has quit IRC14:21
*** abdysn has joined #openstack-kolla14:21
openstackgerritMark Goddard proposed openstack/kayobe stable/train: Add support for stopping overcloud services  https://review.opendev.org/72198214:22
mnasiadkamgoddard: in the garage, my family can’t stand me working :)14:22
openstackgerritMark Goddard proposed openstack/kayobe stable/train: Fix seed VM provisioning on a remote seed hypervisor  https://review.opendev.org/72198314:22
openstackgerritMark Goddard proposed openstack/kayobe stable/stein: Fix seed VM provisioning on a remote seed hypervisor  https://review.opendev.org/72198414:23
openstackgerritMark Goddard proposed openstack/kayobe stable/rocky: Fix seed VM provisioning on a remote seed hypervisor  https://review.opendev.org/72198514:23
hrwmgoddard: ok14:31
hrwyoctozepto: debian and centos8 updated for train/ussuri14:38
*** zhanglong has quit IRC14:40
*** factor has quit IRC14:45
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: CI: Test CentOS 7 to 8 migration  https://review.opendev.org/71160114:47
*** suff has joined #openstack-kolla14:49
mgoddardmeeting in 10 minutes14:50
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi ^14:50
hrwok14:51
Fl1ntI'm in ^^14:52
*** ykarel is now known as ykarel|away14:57
yoctozeptohrw: thx15:00
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Apr 22 15:00:18 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:01
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:01
mgoddard\o15:01
yoctozeptoo/15:01
Fl1ntnoob question, what's rollcall ?15:01
hrw~o~15:01
hrwFl1nt: 'who is here'15:01
priteauhttps://en.wikipedia.org/wiki/Roll_call15:02
Fl1ntoh ok15:02
Fl1nto/15:02
priteau"A taking of attendance as part of a meeting agenda"15:02
Fl1ntso ^^15:02
priteauo/15:02
mgoddard#topic announcements15:03
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:03
mgoddard#undo15:03
headphoneJameso/15:03
openstackRemoving item from minutes: #topic announcements15:03
mgoddard#topic agenda15:03
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:03
mgoddard* Roll-call15:03
mgoddard* Announcements15:03
mgoddard** Proposed cross-project goal for Victoria: https://review.opendev.org/72010715:03
patchbotpatch 720107 - governance - goals: add container-images - 3 patch sets15:03
mgoddard** Kolla builder next generation proposal: http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014255.html15:03
mgoddard* Review action items from last meeting15:03
mgoddard* CI status15:03
mgoddard* Kolla Klub https://etherpad.openstack.org/p/kolla-sig15:03
mgoddard** Survey tools comparison https://ethercalc.openstack.org/46c7q4lnww6j15:03
mgoddard* Ussuri release planning (kayobe)15:03
mgoddard* Ussuri release planning (kolla & kolla ansible)15:03
*** skramaja has quit IRC15:03
mgoddard* Virtual PTG15:03
mgoddard* Train release planning (centos 8)15:03
mgoddard* Announcements15:03
mgoddard#topic announcements15:03
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:04
mgoddard#info Proposed cross-project goal for Victoria on container images15:04
mgoddard#link https://review.opendev.org/72010715:04
patchbotpatch 720107 - governance - goals: add container-images - 3 patch sets15:04
mgoddardSeems there could be some interesting outcomes from this15:05
mgoddardAnd some opening up to the potential of using kolla15:05
yoctozeptoso this is mnaser's proposal that we could participate in as we have some relevant experience15:05
yoctozeptobindep is interesting to finally get nice15:05
hrwI decided to limit myself to read comments15:06
mgoddardright, although it's a long way from magically installing all your deps IMO15:06
yoctozeptoactually sean-k-mooney has just written up something relevant to bindep on the ml, regarding the next announcement15:06
yoctozepto(as well)15:06
mgoddard#info Kolla builder next generation proposal15:06
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014255.html15:06
mgoddardThanks yoctozepto for starting this conversation. I encourage people to get involved15:07
yoctozeptomgoddard: sure, I think we should do these kinds of things on ml from time to time, if not just to raise community awareness15:08
mgoddard#info OpenStack projects proposing RC1 releases this week15:08
yoctozepto(not kolla!)15:08
cosmicsoundo/15:09
mgoddard#info Kolla feature freeze next week15:09
*** sorin-mihai_ has joined #openstack-kolla15:09
yoctozepto:O15:09
mgoddardyup15:09
yoctozepto:D15:09
hrwlet me check what I have open...15:09
mgoddardok, that's enough #info15:09
mgoddardReview action items from last meeting15:09
* Fl1nt reading15:09
mgoddard#topic Review action items from last meeting15:09
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)"15:09
*** wuchunya_ has joined #openstack-kolla15:09
mgoddardmgoddard to prepare project update for kolla klub15:10
mgoddardosmanlicilegi to adapt popularity-contest for kolla needs15:10
mgoddardosmanlicilegi to look into kolla klub survey/poll15:10
mgoddardmgoddard to read PTG emails and request slots15:10
mgoddardhrw to switch kolla centos 8 train from delorean to centos-release-openstack-train15:10
mgoddardwe will find out if I have done mine tomorrow15:10
mgoddardI haven't requested PTG slots yet, but did put a proposal on the mailing list15:10
hrwhttps://review.opendev.org/721329 does my part. We are waiting for centos-release-* packages (on a way).15:10
patchbotpatch 721329 - kolla (stable/train) - [WIP] switch to official CentOS 8 repositories - 3 patch sets15:10
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014174.html15:11
hrwall images build fine.15:11
mgoddardthanks15:11
mgoddardosmanlicilegi did his, but can't attend the meeting15:11
*** sorin-mihai has quit IRC15:11
yoctozeptoyeah, thanks osmanlicilegi!15:11
mgoddard#action mgoddard to sign up for PTG slots15:11
yoctozeptonice summary there15:11
hrw16:02 < osmanlicilegi> mgoddard: I have to skip today's meeting cause there will a lockdown here for the next 4 days. I need to visit pharmacy and market. You can find my survey tools comparison at https://ethercalc.openstack.org/46c7q4lnww6j. If there's anything missing for this task, just ping me.15:12
yoctozepto#link https://ethercalc.openstack.org/46c7q4lnww6j15:12
mgoddard#topic CI status15:12
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:12
hrwc8/aarch64 is RED due to lack of erlang15:13
openstackgerritJames Kirsch proposed openstack/kolla master: Add stunnel container for Glance service  https://review.opendev.org/72085015:13
hrwboth train and ussuri. we have to wait for CentOS 8.2 ;(15:13
yoctozeptootherwise pretty green15:14
mgoddardhrw: can you mark the rabbitmq image as unbuildable until then so we're green?15:14
mgoddardthen at least we can catch regressions15:14
hrwmgoddard: kolla-toolbox is red too :(15:14
hrwmgoddard: ok15:14
mgoddard#action hrw mark rabbitmq and kolla-toolbox unbuildable for c8/aarch6415:15
mgoddard#topic Kolla Klub https://etherpad.openstack.org/p/kolla-sig15:15
*** openstack changes topic to "Kolla Klub https://etherpad.openstack.org/p/kolla-sig (Meeting topic: kolla)"15:15
mgoddardMeeting tomorrow, 1500 UTC15:15
mgoddardI'll give a project update, and we can feed back about surveys15:16
mgoddard#topic Survey tools comparison https://ethercalc.openstack.org/46c7q4lnww6j15:16
*** openstack changes topic to "Survey tools comparison https://ethercalc.openstack.org/46c7q4lnww6j (Meeting topic: kolla)"15:16
Fl1ntWe should also15:16
yoctozeptoit's interesting to learn details on kolla usage too15:16
mgoddardindeed. I think some case studies would be nice15:17
Fl1nthave a little agenda for what we as a community expect from this klub15:17
mgoddardideally I won't be presenting every session :)15:17
yoctozeptomgoddard: that would be regression!15:17
yoctozeptoFl1nt: yup, that discussion should be part of agenda15:18
yoctozeptothe homework was to think about it15:18
mgoddardFl1nt: I did try to ask last time, but didn't get too much response15:18
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: No kolla-toolbox & rabbitmq for aarch64/centos8  https://review.opendev.org/72199915:18
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Revert "No kolla-toolbox & rabbitmq for aarch64/centos8"  https://review.opendev.org/72200015:18
mgoddardfeedback seemed to be 'not too much naval gazing - just get on with it'15:18
Fl1ntyep, unfortunately, I think, on my own part, that many operators wait for a more vivid community in terms of support channels.15:19
*** dcapone2004 has joined #openstack-kolla15:19
mgoddardAgenda is here, feel free to add something: https://docs.google.com/document/d/1EwQs2GXF-EvJZamEx9vQAOSDB5tCjsDCJyHQN5_4_Sw/edit#heading=h.u784zrqsv2k115:19
Fl1ntok, I'll think about it.15:20
*** abdysn has quit IRC15:20
mgoddardif it's something you want to drive, that might encourage others to get involved15:20
mgoddardok, let's look at osmanlicilegi's results, since he put time into it15:20
mgoddardhttps://ethercalc.openstack.org/46c7q4lnww6j15:20
yoctozeptoat the end of the day, google forms look the best15:21
mgoddardkolla klub agena item 1:15:21
mgoddardSurvey tools survey results15:21
yoctozeptobut I guess China cannot access them15:21
mgoddardyeah that's a shame15:23
Fl1ntyeah, such constraints definitely slowdown collaboration.15:23
mgoddardtypeform seems nice, apart from the 10 question limit15:23
Fl1ntCan we achieve the survey in 10 questions?15:24
mgoddardIt seems a bit low15:24
yoctozeptobut maybe it fits15:25
Fl1ntMaybe the trick is to create multiple survey and explain that multiple strategy has we want to include and ease the discussion with everyone? So no one will complain about having multiple poll.15:25
yoctozeptowe also need to consider multianswer and freeform parts15:25
mgoddard+115:25
mgoddardosmanlicilegi has some multi answer15:25
yoctozeptolet's extend the calc for that15:26
mgoddardI don't see freeform though15:26
mgoddardIMO that would be one of the most valuable parts15:26
hrwFl1nt: the problem is analizing results15:26
Fl1nthum... 45€ for the professional plan, can't we pay for it ?15:27
mgoddardOne thing I worry about with these surveys is how discoverable the data will be15:27
Fl1ntthat's the killer feature of Google15:27
mgoddardI think at a minimum we need data export15:27
mgoddardbut also we need to be able to host that somewhere15:28
*** e0ne has quit IRC15:28
yoctozeptoit would be nice to learn about osf's solution15:28
Fl1nt+115:28
*** e0ne has joined #openstack-kolla15:28
mgoddardI see a few uses for the results15:29
mgoddard1. give the team a better picture of what people do and don't use15:29
Fl1ntyou can connect typeform anserws to google sheets15:29
mgoddard2. allow to connect up questions to sources of knowledge15:29
mgoddarda survey is probably quite good at 1., I don't know how well it serves 215:30
yoctozeptomgoddard: could you elaborate on 2.?15:30
mgoddardi.e. user comes in asking about storlets + kolla, who do we connect them to?15:31
mgoddardmaybe I'm just thinking too siloed, and we should send them to #openstack-storlets15:31
mgoddardalmost what I want is a social network :)15:32
mgoddardlinkedin for openstack15:32
yoctozeptomgoddard: I was about to say "linkedin for kolla"15:32
yoctozeptobut yeah15:32
mgoddardmaybe it's a pipe dream15:32
yoctozeptoyou already know I'm a dreamer too, so you have my full support15:33
mgoddardI'd just like to avoid the problem that questions often go unanswered because not all experts read IRC, and when we send people to $project-channel it's often to /dev/null15:33
yoctozeptoyeah, especially those least used ones15:33
Fl1ntyeah, exactly my point of why I suggested having discord, let's face it, nobody except old school devs/operators like us will use IRC, new comers to Openstack tends to be rather younger and have different tools/usage to communicate.15:33
* osmanlicilegi is back after a quick shopping15:34
yoctozeptoosmanlicilegi: quick, huh? ;D15:34
yoctozeptoprobably after cilegi ;-)15:34
mgoddardFl1nt: it's not just about the medium though, most people just don't want to consume the firehose15:34
Fl1nthonestly, kolla channel is for sure the most active one.15:34
osmanlicilegiyoctozepto: ;]15:35
Fl1ntsure, it was an exemple to support the fact that this community can't be based on either IRC or List.15:35
mgoddardmaybe the answer is we should direct people to the mailing list more often15:35
mgoddardat least there people filter out threads15:35
yoctozeptojust let's not talk ask.oo, it s... is not decent enough :-)15:35
mgoddardindeed15:35
mgoddardanyways, in terms of surveys15:36
mgoddardgoogle forms is best, but limited15:36
Fl1ntif you ask you'll get 10000 answers of the form to use, if you build it and propose it, you'll get adoption if it fit.15:36
mgoddardis there a runner up to forms?15:36
mgoddardtypeform? apart from 10 question limit15:37
Fl1ntyep typeform15:37
mgoddardosmanlicilegi: did any surveys offer freeform text?15:37
yoctozeptoit feels tedious to fill in15:37
yoctozeptoand has 100 answers limit monthly15:37
*** scottsol has joined #openstack-kolla15:37
Fl1nttypeform and surveymonkey.com are the both big names in that industry15:37
Fl1ntoutside of google form I mean15:37
osmanlicilegimgoddard: iirc all has freeform text support15:38
mgoddardI think 100 answers per month would be plenty yoctozepto!15:38
yoctozeptomgoddard: be a dreamer! ;p15:38
osmanlicilegibtw, typeform seems best after google forms if 10 questions is ok15:38
Fl1ntif 10 questions is your blocker, I'll subvention the typeform if necessary.15:39
yoctozeptoseems so15:39
yoctozeptoI don't like one q per page though15:39
mgoddardI think we should move on :)15:39
yoctozepto++15:39
mgoddard#topic Ussuri release planning (kayobe)15:39
*** openstack changes topic to "Ussuri release planning (kayobe) (Meeting topic: kolla)"15:39
Fl1ntyep15:39
mgoddardpriteau, dougsz, jovial[m]15:39
osmanlicilegiyoctozepto: typeform is customizable. one question per page can be my fault15:39
yoctozeptoosmanlicilegi: would you drop a question to ml and osf about their framework for surveys?15:40
mgoddardIf you didn't catch it earlier, feature freeze next week15:40
yoctozeptomaybe someone responds15:40
yoctozeptoosmanlicilegi: ack15:40
yoctozepto(now onto kayobe)15:40
osmanlicilegiyoctozepto: I will, today.15:40
mgoddardhttps://review.opendev.org/#/q/project:openstack/kayobe+branch:master+status:open15:40
priteaumgoddard: thanks for the reminder15:40
*** dking has joined #openstack-kolla15:40
priteauLots of open patches :o15:40
mgoddardyep15:41
mgoddardquite a few +2'd or approved though15:41
priteaumnasiadka: are you planning to update https://review.opendev.org/#/c/709689/ to answer Mark's comments?15:41
patchbotpatch 709689 - kayobe - Add framework to deploy user-defined containers on... - 11 patch sets15:41
mgoddardif anyone has time then adding a second +2 would help clear some15:42
mgoddardwould be nice to get https://review.opendev.org/#/c/658815/ in but I don't know if I have time to test the migration15:42
patchbotpatch 658815 - kayobe - WIP: Switch default docker storage driver to overlay2 - 2 patch sets15:42
dkingFor Kayboe, should cloud_vip_address be set for localhost? I'm receiving the error that kolla_internal_vip_address is not set for localhost. And if so, where's the right place to set that?15:43
mgoddardif someone else has time feel free to pick it up15:43
mgoddarddking: we're just in a meeting at the moment15:43
mgoddarddking: but you can set it in networks.yml15:43
dkingOkay. Thank you. I'll probably try back in an hour or so, then.15:43
mgoddarddking: which is global for all hosts15:43
mgoddardany more for kayobe15:44
mgoddard?15:44
openstackgerritMerged openstack/kolla-ansible master: Fix that cyborg agent failed to start privsep daemon.  https://review.opendev.org/72113915:44
mgoddard#topic Ussuri release planning (kolla & kolla ansible)15:44
*** openstack changes topic to "Ussuri release planning (kolla & kolla ansible) (Meeting topic: kolla)"15:44
yoctozeptoall good15:45
hrwDebian packages partially available15:45
mgoddardadded RP+1 to https://review.opendev.org/#/c/721632/115:46
patchbotpatch 721632 - kolla-ansible - Add docs and release note for CentOS 8 - 1 patch set15:46
hrwfor CentOS I wait for centos-release-* packages and will add some of them to cover RDO needs15:46
hrwrabbitmq/erlang are now in queue: https://review.opendev.org/721800 then advanced-virtualization and ceph-nautilus will follow15:46
patchbotpatch 721800 - kolla - CentOS: use RabbitMQ and Erlang from official repos - 1 patch set15:46
mgoddardhrw: is that for train?15:46
hrwmgoddard: both train and ussuri15:47
mgoddardok15:47
hrwmgoddard: only centos-release-openstack-{train,ussuri} will differ15:47
mgoddardwe have a few tls backend patches in flight. Hopefully we can get them in15:48
mgoddardwould be nice to do it for all services already using httpd, but getting late15:49
hrwI will add relnote to kolla/rmq patch for aarch64 to make it visible that c8/aarch64 is not part of initial release15:49
mgoddardok, thanks15:49
yoctozeptohttps://review.opendev.org/70780015:49
patchbotpatch 707800 - kolla - [community goal]: Add contributor and PTL guide - 2 patch sets15:49
yoctozeptothis needs some love15:49
mgoddard+115:50
mgoddardalthough not affected by freeze15:50
mnasiadkapriteau: yes, tomorrow15:50
mgoddard#topic Virtual PTG15:51
*** openstack changes topic to "Virtual PTG (Meeting topic: kolla)"15:51
mgoddardDo we have much to say on it?15:52
mgoddardWe can gather topics in a few weeks I think15:52
mgoddard#topic Train release planning (centos 8)15:52
*** openstack changes topic to "Train release planning (centos 8) (Meeting topic: kolla)"15:52
hrwas usual: this cycle summary, c8 summary, next cycle goals15:52
hrwtrain for C8 is splendid15:53
hrwwe have all images buildable15:53
mgoddardI think this was just a request to make a release when train/c8 is ready?15:53
hrwyes15:53
mgoddardok. If I backport the k-a docs/reno patch to train then I think we're ready for a release15:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: No kolla-toolbox & rabbitmq for aarch64/centos8  https://review.opendev.org/72199915:54
openstackgerritMerged openstack/kolla-ansible stable/train: Fix that cyborg conductor failed to communicate with placement  https://review.opendev.org/72177015:54
hrwadded relnote15:55
mgoddardit should at least work for greenfield deploys. I'll do 7-8 migration docs separately to not block a releaase15:55
hrwok15:55
hrwthanks15:55
mgoddard#topic Open discussion15:55
*** openstack changes topic to "Open discussion (Meeting topic: kolla)"15:55
Fl1ntDocumentation question15:55
Fl1ntis rocky release considered as EOL ?15:55
yoctozeptoFl1nt: extended maintenance15:56
yoctozeptoFl1nt: we accept patches as lons as CI works (it works afaik)15:56
yoctozeptobut do not really propose ourselves etc.15:56
*** suff has quit IRC15:56
Fl1ntbecause the k-a with external ceph needs some fixes, I'll be happy to write them, but it also need some fixes on the nova role as it doesn't work at this moment.15:56
yoctozeptoand I'd like to discuss https://bugs.launchpad.net/kolla-ansible/+bug/1874218 a bit15:56
openstackLaunchpad bug 1874218 in kolla-ansible "swift sanity checks (smoke tests) broken" [Low,Triaged]15:56
yoctozeptoare we into sanity checks / smoke tests?15:56
yoctozeptothey look pooey now15:57
openstackgerritJames Kirsch proposed openstack/kolla master: Modify Heat and Nova containers to execute httpd setup  https://review.opendev.org/72202815:57
yoctozeptoFl1nt: only rocky?15:57
priteauFl1nt: I am curious, what doesn't work with nova and external ceph?15:57
Fl1ntmaybe stein, but I didn't have HW/CI to test that release for now15:57
Fl1nt@priteau, the keyring15:57
Fl1ntbecause15:57
Fl1ntwith external ceph15:58
Fl1ntyou don't handle which ceph client/release you're using15:58
Fl1ntand so15:58
Fl1ntwhen using ceph.client.nova.keyring filename with rbd_user = cinder and inner key = ceph.client.cinder.keyring content15:58
Fl1ntceph client saw the trick15:58
Fl1ntand complain about it and raise an error15:59
yoctozeptomgoddard: no thoughts on smoke tests / sanity checks?16:00
Fl1ntthe only way to overcome that, is to upload the ceph.client.cinder.keyring on nova_compute and nova_libvirt container instead of ceph.client.nova.keyring.16:00
Fl1ntor16:00
Fl1ntto add a client directive within your ceph.conf like16:01
Fl1nt[client.nova]16:01
Fl1ntkeyring = <path_to_cinder_keyring>16:01
Fl1ntwhich is... ugly for the least ^^16:01
Fl1ntexemple: https://opendev.org/openstack/kolla-ansible/src/tag/7.2.1/ansible/roles/nova/tasks/external_ceph.yml16:01
Fl1ntbut on train: https://opendev.org/openstack/kolla-ansible/commit/a9fecc41a31749c97e89ba6b66c9c46714fa6da816:02
mgoddardFl1nt: I would suggest talking to mnasiadka. He made quite a few changes to external ceph config in master16:02
Fl1ntcool16:02
mgoddardwe're out of time I'm afraid16:02
mgoddardThanks all16:02
mgoddard#endmeeting16:03
*** openstack changes topic to "Please feel welcome to join Kolla SIG https://etherpad.openstack.org/p/kolla-sig | Kolla IRC meetings on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL | Whiteboard: https://etherpad.openstack.org/p/KollaWhiteBoard | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:03
openstackMeeting ended Wed Apr 22 16:03:02 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:03
Fl1ntif we can cherry pick the fixes may be good16:03
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-04-22-15.00.html16:03
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-04-22-15.00.txt16:03
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-04-22-15.00.log.html16:03
yoctozeptomgoddard: I don't mind your answer after the meeting16:03
osmanlicilegithanks folks16:03
hrwhttps://review.opendev.org/#/c/721847/ anyone?16:03
patchbotpatch 721847 - kolla - Do not try to build skipped/unbuildable images - 1 patch set16:03
Fl1nt@mgoddard, any idea of @mnasiadka timezone ?16:04
priteauFl1nt: mnasiadka's commit suggests that this is needed only from Train16:04
hrwFl1nt: 18:04 at his place16:04
Fl1nt@priteau, it's not unfortunately16:04
priteauAnd stupid question, but why not just configure a nova user? :-)16:04
Fl1ntget the issue confirmed on rocky 7.1.1 kolla container using centos-7 with ceph-common packages from nautilus and ceph-ansible cluster using nautilus too.16:05
hrwFl1nt: you did a fresh modified build of rocky/c7 images?16:06
Fl1ntbecause booting from volume is handled by cinder.16:06
Fl1nt@hrw, fresh? is a week ago enough?16:06
mnasiadkaFl1nt: rocky? Oh boy16:07
mnasiadkaFl1nt: I did make some changes to external-ceph, but mainly in master - not backported since it was rather an extension, not bug fixing16:07
hrwFl1nt: fresh as other than official ones16:07
*** jimcrowleyibm has quit IRC16:08
*** jimcrowleyibm has joined #openstack-kolla16:08
Fl1nt@mnasiadka, all in all, let's just nova use cinder key and username, put in the doc that due to some shinaningan within libraries within rocky you need to use cinder and add the vms pools to your cinder user, that's it16:08
Fl1nt@hrw, yep, they've been rebuild last week.16:09
*** wuchunya_ has quit IRC16:09
*** suff has joined #openstack-kolla16:10
hrwFl1nt: have you considered r->s->t?16:10
mnasiadkaFl1nt: rocky is in extended maintenance, if you feel like caring for it - raise a bug and fix it :)16:10
Fl1nt@hrw, that's in the process but I can't for now as the PoC requester explicitely required the acceptance test to be PoC on rocky, then upgrade validation to train and prod on train.16:11
Fl1nt@mnasiadka, I'll do.16:11
*** igordc has joined #openstack-kolla16:12
Fl1ntFWIW @mnasiadka this, isn't accurate neither: https://docs.openstack.org/kolla-ansible/train/reference/storage/external-ceph-guide.html16:14
Fl1ntbecause kolla doesn't control the ceph-common client version.16:14
Fl1ntused within image builds16:14
* hrw off16:14
sorin-mihai_i'm getting this error when trying to deploy from master https://pastebin.com/raw/9RrYBaL1 where should that cert file be on the node to get sourced?16:15
Fl1nt@sorin-mihai_, within <node_config>/config/cinder/ directory16:17
sorin-mihai_strange, because i tried that, i was hopping it should be somewhere else16:17
openstackgerritMerged openstack/kolla stable/train: CentOS 8: Add rabbitmq-3.7.24 image (CentOS 7 only)  https://review.opendev.org/71522416:19
sorin-mihai_that would be /etc/kolla/config/cinder/ with the defaults, right?16:20
Fl1ntyep, except if you deploy using kolla-ansible from pip16:21
sorin-mihai_yeah, double checked. i have /etc/kolla/config/cinder/cinder-cert.pem16:22
*** k_mouza has quit IRC16:24
*** k_mouza has joined #openstack-kolla16:25
mnasiadkaFl1nt: it does - in a way - we install what distro gives us - you’re free to change it using template override16:25
Fl1nt@sorin-mihai_, do you have your ansible run history? does it copied the file or skipped it?16:25
sorin-mihai_can always run again16:26
Fl1nt@mnasiadka, yes, but I mean, for instance, we don't really get a compatibility matrix such as rocky = ceph >13.x <15.y etc16:26
Fl1ntand16:26
*** k_mouza has quit IRC16:26
*** k_mouza has joined #openstack-kolla16:27
Fl1ntHere is an example, your company have two departments, one which handle the storage part, one which provide platforms that are supposed to use/run over it.16:27
Fl1ntthen you've got maybe storage team using ceph octopus, but the platforms requiring rocky/stein/whatever16:28
Fl1ntand maybe that required platform version isn't compatible with a specific client.16:28
Fl1ntbut you can't know it16:28
mnasiadkaFl1nt: https://ethercalc.openstack.org/kolla-infra-service-matrix16:28
*** rpittau is now known as rpittau|afk16:29
mnasiadkaFl1nt: can’t think of ceph client incompatibility for RBD... unless your storage team limits the version of clients allowed to connect16:29
Fl1nt@mnasiadka, where did you get that? :D that's exactly that kind of matrix that should be frontpage of the project doc.16:29
Fl1nt@mnasiadka, exactly that.16:30
mnasiadkaFl1nt: I think yoctozepto/hrw did that16:30
Fl1ntooooh sweet of them!! kudos to anyone did it!16:30
mnasiadkaFl1nt: so kick the storage team in the head for not thinking straight :)16:30
*** bengates has quit IRC16:31
mnasiadkaWhen it comes to RBD - even Jewel clients are fully compatible with Nautilus cluster16:31
mnasiadka(Doesn’t mean bugs are fixed in Jewel clients though)16:31
Fl1ntWell, as operators, we're dealing with company, I'm not David going to beat the ass of goliath ;-) I think that enterprise oriented software should at least know about such use-case and provide explicit material on it, vmware do it for instance.16:32
Fl1ntyeah, it's not an incompatibility issue in here, it's just the way corporate work.16:32
*** k_mouza has quit IRC16:33
Fl1ntoh and also the way the library used as switched within nova during a release I guess :D16:33
Fl1nt-as +have16:33
*** evrardjp has quit IRC16:35
sorin-mihai_https://pastebin.com/raw/vC5WY1Te16:35
*** evrardjp has joined #openstack-kolla16:35
Fl1nt@sorin-mihai_, did you set your node_config var? because you run your kolla-ansible from venv, it won't be able to find it otherwise.16:37
sorin-mihai_no, the line is actually commented, i thought it would keep it as 'default' #node_custom_config: "/etc/kolla/config"16:39
*** prernadembla05 has quit IRC16:42
sorin-mihai_but is not commented in group_vars/all.yml and i thought it would take it from there16:43
Fl1nt@mnasiadka, ok, I confirm the issue, I'll do a bugfix this weekend and propose it, that's nothing too complex so it should be easy to approve (or not) it.16:44
*** riuzen has joined #openstack-kolla16:45
*** e0ne has quit IRC16:47
*** e0ne has joined #openstack-kolla16:48
riuzenAnyone have a luck with magnum on k-a? I try to deploy it on Train version then got this error: http://paste.openstack.org/show/792556/. Magnum service always using public keystone endpoint instead internal and keystone_auth_default_policy.json missing.16:51
*** riuzen has quit IRC16:51
*** e0ne has quit IRC16:56
*** e0ne has joined #openstack-kolla16:56
*** scottsol has quit IRC17:00
yoctozeptowell, that was a quick leave17:03
hrwmnasiadka: yoctozepto did, I updated17:04
dcapone2004saw some response on my questions about openstack train and CentOS817:08
dcapone2004there was a ntoe that the git clone version was needed17:08
dcapone2004however, I was monitoring your meeting and noted some conversation about a CentOS8/Train "release", did this imply that this support is being backported into the pip channel and will work from pip install -U kolla-ansible in the near future?17:10
yoctozeptomgoddard, hrw: I updated https://ethercalc.openstack.org/kolla-infra-service-matrix17:12
yoctozepto(just now)17:12
yoctozeptoit represents current state17:12
yoctozeptothere are two apparent issues, marked red and added a comment17:12
yoctozeptothis is before proposed hrw's changes *17:13
*** dougsz has quit IRC17:13
*** seco has quit IRC17:13
hrwyoctozepto: I would love to have facts checked before leaving comments17:16
hrwyoctozepto: Debian in train uses luminous despite backports being enabled17:16
hrwyoctozepto: Debian in ussuri has nautilus pinned17:16
hrwyoctozepto: if someone wants then Debian/stein/nautilus is also possible. We have such images built in Linaro.17:18
yoctozeptohrw: we don't have backports repo in train, mr fact checker ;-)17:18
*** priteau has quit IRC17:18
hrwyoctozepto: then I do not understand your comment on debian/ussuri/ceph version17:18
yoctozeptothe issue is that we are 1) using general backports when ceph has its own home, 2) we are using train for ussuri :-)17:18
yoctozeptoit's not a breaker, just something to keep in mind17:19
hrwyoctozepto: https://review.opendev.org/71936417:19
patchbotpatch 719364 - kolla - Debian: use Ussuri binary packages - 1 patch set17:19
yoctozeptodespite the red colour17:19
hrwyoctozepto: when it will be ready then it will be using ussuri binaries17:19
hrwyoctozepto: if you want to help Debian OpenStack team then go for it.17:19
hrwyoctozepto: they work on packages After RC1 of openstack17:20
hrwit is not UCA or RDO with people paid to do that stuff17:20
yoctozeptohrw: I don't know why you are taking this personally :-) it's just a thing to remember ;-)17:20
hrwyoctozepto: such a day.week.month17:21
yoctozeptoyeah, I'm depressed too :/17:21
hrwyoctozepto: I am tired of keeping all those distros in sync ;d17:21
yoctozeptointeresting uca brought octopus to the table though17:22
hrwnot suprising. they did that in past too17:22
hrwwe released centos/debian with luminous and ubuntu with mimic17:22
hrwetc17:22
yoctozeptohrw: yeah, it's quite demanding to keep them all in a worky state17:22
yoctozeptohmm, mimic looks gone from tables though17:23
yoctozeptonot seein it17:23
hrwyoctozepto: I like the fact of knowing openstack people in rdo and debian as this makes our life easier17:23
hrwyoctozepto: RDO had some changes done especially for us in last few days17:23
hrwas effect of 'guys, we lack those packages: x, y, z, za. they were built here: urlx, urly, urlz, urlza'17:24
hrwcentos-release-ansible-29 got fixed as it broke our CI17:24
hrwcentos-release-* (ceph-nautilus, advanced-virt, openstack-train) will be released this week17:25
hrwc8/train will use mirrors.centos.org on all three architectures17:25
hrwc8/ussuri will use same (except openstack-train) + RDO delorean17:26
hrwso some repos will duplicate in master but not in stable/ussuri17:26
hrwetc17:26
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting glance api  https://review.opendev.org/70713117:26
yoctozeptohrw: that17:26
yoctozepto's cool17:26
hrwfor vicky we can choose to use either centos or rdo repos for components17:26
hrwetc17:26
hrwI really like how c8 looks for us.17:27
yoctozeptoc8 feels better than c7 for sure17:27
hrwwe probably can ignore external repos for most of stuff.17:27
hrwELK will be external, grafana too but rmq? c817:27
hrwonce c8/train gets released I will take ppc64le machine and check how it looks there17:28
yoctozeptohrw: and then other distros move to rmq 4.0 which requires magic and we have to cater for that in k-a :D17:28
hrwyoctozepto: when they do then we will worry17:30
yoctozeptohrw: indeed17:30
hrwI am tired of that IT cargoculture of using random binaries17:30
hrwwhich in projects like that keeps us nicely out of time for working on new stuff17:31
*** gfidente is now known as gfidente|afk17:33
yoctozeptohrw: agreed17:34
yoctozeptohrw: so you are saying you've got good contacts in rdo and debian17:34
yoctozeptomaybe what we need is a way to coordinate such things17:35
yoctozeptoand also include uca in that of course17:35
yoctozeptoideally we would not need any upstreams and coordinate all 3 to roughly match their versions17:36
yoctozeptoor else it would help greatly if we could just consume "official" containers17:36
yoctozeptofor non-openstack things and then focus on openstack and better features for it17:36
hrwyoctozepto: those contacts are 'go to proper irc channels, ask nicely, be gentle and help'17:37
hrwyoctozepto: we can drop binary images and pray for infra ones to work ;D17:38
openstackgerritRadosław Piliszek proposed openstack/kolla master: [DNM] test something  https://review.opendev.org/72210217:49
yoctozeptohrw: deprecated everything!17:49
yoctozeptodeprecate*17:49
hrwyoctozepto: deprecate base. rest will not matter17:51
*** cdearborn has joined #openstack-kolla17:55
openstackgerritMerged openstack/kolla master: Do not try to build skipped/unbuildable images  https://review.opendev.org/72184717:58
*** jopdorp has quit IRC17:58
openstackgerritMerged openstack/kayobe master: Fix passwords.yml generation with vault encryption on Python 3  https://review.opendev.org/72128117:59
openstackgerritMerged openstack/kayobe master: Cleanup py27 support  https://review.opendev.org/72076417:59
dcapone2004any response on my Train/CentOs8 support?18:03
*** suff has quit IRC18:32
*** k_mouza has joined #openstack-kolla18:33
*** e0ne has quit IRC18:34
*** e0ne has joined #openstack-kolla18:34
*** k_mouza has quit IRC18:38
*** vishalmanchanda has quit IRC18:43
*** e0ne_ has joined #openstack-kolla18:43
*** e0ne has quit IRC18:43
*** suff has joined #openstack-kolla18:55
*** e0ne_ has quit IRC19:04
*** dougsz has joined #openstack-kolla19:06
*** suff has quit IRC19:06
*** tonythomas has left #openstack-kolla19:07
*** suff has joined #openstack-kolla19:09
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: Add swift jobs  https://review.opendev.org/72212719:44
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Custom haproxy script for monitoring galera  https://review.opendev.org/71021319:52
openstackgerritMerged openstack/kayobe master: Use upper constraints when installing Tenks  https://review.opendev.org/72073620:11
*** suff has quit IRC20:13
*** seco has joined #openstack-kolla20:23
*** seco has quit IRC20:32
openstackgerritRadosław Piliszek proposed openstack/kolla master: [DNM] test something  https://review.opendev.org/72210220:41
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting glance api  https://review.opendev.org/70713120:42
yoctozeptomgoddard: http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014293.html, hmm?20:44
openstackgerritJames Kirsch proposed openstack/kolla master: Modify Heat and Nova containers to execute httpd setup  https://review.opendev.org/72202820:44
*** dougsz has quit IRC20:47
*** devfaz has quit IRC20:54
*** devfaz has joined #openstack-kolla20:55
*** Fl1nt has quit IRC21:02
*** yevi has left #openstack-kolla21:09
*** egonzalez has quit IRC21:45
*** oklhost has quit IRC21:45
*** egonzalez has joined #openstack-kolla21:45
*** pescobar has quit IRC21:45
*** seco has joined #openstack-kolla21:55
*** jimcrowleyibm has quit IRC22:09
*** dciabrin has quit IRC22:33
*** dciabrin has joined #openstack-kolla22:34
*** seco has quit IRC22:54
*** TrevorV has quit IRC22:59
*** born2bake has quit IRC23:00
*** dciabrin has quit IRC23:14
*** dciabrin has joined #openstack-kolla23:25

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!