15:00:48 <mgoddard> #startmeeting kolla
15:00:49 <openstack> Meeting started Wed Oct 16 15:00:48 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:52 <openstack> The meeting name has been set to 'kolla'
15:00:53 <mgoddard> #topic rollcall
15:00:55 <mgoddard> \o
15:01:05 <jovial[m]> 0/
15:01:19 <yoctozepto> o/
15:01:51 <generalfuzz> o/
15:01:55 <priteau> o/
15:02:15 <osmanlicilegi> o/
15:02:42 <mgoddard> #topic agenda
15:02:46 <chason> o/
15:02:50 <mgoddard> * Roll-call
15:02:52 <mgoddard> * Announcements
15:02:54 <mgoddard> ** OpenStack Train final release this week
15:02:56 <mgoddard> ** OpenStack Queens becomes EM next week on Friday
15:02:58 <mgoddard> * Review action items from last meeting
15:03:00 <mgoddard> * CI status
15:03:02 <mgoddard> * Train release planning
15:03:04 <mgoddard> * Review priorities
15:03:06 <mgoddard> * Which images should we mark as maintained in the support matrix?
15:03:08 <mgoddard> * Reducing load of build & publishing jobs on CI & Dockerhub (continued) https://etherpad.openstack.org/p/kolla-train-image-evaluation
15:03:10 <mgoddard> #topic announcements
15:03:12 <mgoddard> #info OpenStack Train final release this week
15:03:14 <mgoddard> I think it should happen today
15:03:28 <mgoddard> #info OpenStack Train final release this week
15:03:34 <mgoddard> #info OpenStack Queens becomes EM next week on Friday
15:03:51 <yoctozepto> mgoddard: it happened
15:04:00 <mgoddard> final release?
15:04:09 <yoctozepto> #link http://lists.openstack.org/pipermail/openstack-announce/2019-October/002024.html
15:04:14 <mgoddard> oh yeah
15:04:19 <mgoddard> thanks
15:04:31 <yoctozepto> np
15:05:06 <mgoddard> I think we agreed to stop publishing images in extended maintenance (EM), so we should disable queens soon
15:05:15 <yoctozepto> next Friday
15:05:24 <yoctozepto> or later if they postpone
15:05:35 <yoctozepto> but yeah, we should do this as soon
15:05:51 <mgoddard> #topic Review action items from last meeting
15:05:54 <mgoddard> yoctozepto to investigate tacker + glance store
15:05:56 <mgoddard> mgoddard to EOL pike
15:05:58 <mgoddard> mgoddard to backport publishing job timeout increase
15:06:00 <mgoddard> mnasiadka to look at not building or publishing deprecated images
15:06:02 <mgoddard> we had a few
15:06:05 <mgoddard> yoctozepto: I saw your email
15:06:18 <mgoddard> so we need shared storage now?
15:07:30 <yoctozepto> mgoddard: indeed :<
15:07:54 <yoctozepto> it does not look like tacker is too organized now
15:08:03 <openstackgerrit> Mark Goddard proposed openstack/kolla stable/stein: CI: increase both limits to 3h  https://review.opendev.org/688954
15:08:04 <yoctozepto> they are not too responsive either
15:08:35 <mgoddard> should we do the same as glance and enable only on one host?
15:08:53 <mnasiadka> and mark that limitation in docs?
15:08:59 <mnasiadka> (or a red note)
15:09:03 <mnasiadka> rel note :)
15:09:15 <mgoddard> if only we had some location independent way of storing images :/
15:09:28 <mgoddard> sounds fair
15:09:57 <mgoddard> yoctozepto: are you going to pick it up?
15:11:25 <yoctozepto> mgoddard: one instance of tacker?
15:11:31 <mgoddard> yes
15:11:41 <openstackgerrit> Mark Goddard proposed openstack/kolla stable/rocky: CI: increase both limits to 3h  https://review.opendev.org/688956
15:11:53 <yoctozepto> ok, let me see to that
15:11:56 <mgoddard> thanks
15:12:02 <yoctozepto> I don't know what tacker does really
15:12:07 <yoctozepto> will need to read up ;-)
15:12:27 <mgoddard> #action yoctozepto to fix tacker & NFV scenario job
15:12:36 <mgoddard> something NFV something
15:13:31 <yoctozepto> that's what I know :-)
15:13:31 <mgoddard> I proposed a patch for EOL pike, still waiting for +A
15:13:46 <mgoddard> just now I did the other action item :)
15:14:05 <mgoddard> mnasiadka: I don't think you did yours did you?
15:14:35 <mgoddard> #action mnasiadka to look at not building or publishing deprecated images
15:14:49 <mnasiadka> mgoddard: nope, I started locally, it wasn't a two-liner - and got a lot of work on the OVN PoC this week, but hopefully I can look into that tomorrow
15:15:00 <mgoddard> k
15:15:06 <mgoddard> #topic CI status
15:15:20 <mgoddard> NFV still borked
15:15:38 <mgoddard> we have some issues on ubuntu train in kolla?
15:15:47 <yoctozepto> yes
15:15:53 <mgoddard> #link https://bugs.launchpad.net/kolla/+bug/1848295
15:15:53 <openstack> Launchpad bug 1848295 in kolla train "trove and rally unbuildable for ubuntu binary" [Critical,Triaged]
15:15:54 <yoctozepto> ubuntu ceph scenario is broken atm
15:15:59 <mgoddard> ah, that too
15:16:00 <yoctozepto> and this
15:16:04 <yoctozepto> yeah, both
15:16:12 <mgoddard> is there a bug for the ceph issue?
15:16:32 <mgoddard> looks like no
15:16:33 <yoctozepto> nope
15:16:35 <mnasiadka> I don't think so
15:16:37 <mnasiadka> let me raise it
15:16:44 <yoctozepto> but proposal is there
15:16:51 <yoctozepto> :D
15:17:17 <mgoddard> thanks
15:17:52 <mgoddard> otherwise I think we're ok on the kolla side?
15:18:06 <mgoddard> jovial[m] is looking at baremetal boot issues in kayobe CI
15:18:55 <yoctozepto> mnasiadka: fails bslurp now https://1ef7fd5154b0aa7f60f5-cfb50cc5198678afd23d9756bd7ac094.ssl.cf5.rackcdn.com/688392/6/check/kolla-ansible-ubuntu-source-ceph/f4a235d/primary/logs/ansible/deploy
15:19:08 <yoctozepto> mgoddard: kolla yup
15:19:32 <jovial[m]> yeah, not much luck so far. I've got a wild hunch that it may be because we are using `cpu-model` and this is broken with haswell cpus.
15:19:37 <mnasiadka> yoctozepto: it was the same with previous version of the patch - I'll look into it tomorrow (or you can)
15:19:38 <jovial[m]> `host-model`
15:20:37 <yoctozepto> mnasiadka: let it be on you then
15:20:49 <mgoddard> jovial[m]: certain it's not a networking issue?
15:20:56 <mgoddard> reproduced it locally?
15:21:28 <jovial[m]> pretty sure - unless the kernel is getting corrupted on transit
15:21:58 <mgoddard> would be good if you could pass a checksum argument
15:22:24 <mgoddard> we could try switching from pxe to ipxe?
15:22:31 <mgoddard> udp -> tcp
15:23:00 <mgoddard> anyways, better move on
15:23:11 <mgoddard> #topic Train release planning
15:23:16 <mgoddard> The train approaches
15:23:39 <mgoddard> and the opportunity for puns is running out
15:23:48 <jovial[m]> yeah, using IPXE seems like a good idea.
15:24:36 <mgoddard> we have IPv6 and cells with FFEs, both have a +2 but last to merge will need a rebase
15:25:05 <mgoddard> generalfuzz asked about FFE for TLS CA patches
15:25:27 <mgoddard> #link https://review.opendev.org/#/c/686024/
15:26:16 <mgoddard> I've had my head down on cells, and am away tomorrow and friday
15:26:20 <generalfuzz> I sure did
15:26:55 <yoctozepto> train releases just today
15:27:00 <yoctozepto> in two weeks we could get it in
15:27:06 <mnasiadka> or three months
15:27:07 <yoctozepto> ffe +1 from me ;-)
15:27:12 <yoctozepto> mnasiadka: then not
15:27:14 <yoctozepto> but yeah
15:27:20 <yoctozepto> we have 3 months formally ;p
15:27:26 <mnasiadka> yes
15:27:28 <mgoddard> well we did agree on this friday as a deadline for features
15:27:55 <yoctozepto> hmm, indeed
15:28:05 <yoctozepto> regarding order
15:28:14 <yoctozepto> I am in favor of ipv6 first
15:28:22 <yoctozepto> then I can rebase cells on that
15:28:27 <yoctozepto> since mgoddard is gone
15:28:37 <yoctozepto> CI and dougsz verify that
15:28:49 <yoctozepto> and mnasiadka + hrw make it w+1
15:28:55 <mgoddard> if we have at least two cores sign up to review TLS and its dependency I'd accept an FFE
15:29:33 <mnasiadka> I can sign up to review it - it would be a waste to merge the installation method in base container, but not the deployment part.
15:30:42 <yoctozepto> me too
15:30:51 <mgoddard> ok, thanks
15:31:21 <generalfuzz> +1 thanks
15:31:23 <yoctozepto> mnasiadka: opv6 -> https://review.opendev.org/681573
15:31:28 <yoctozepto> ipv6*
15:31:41 <yoctozepto> or we can accept hrw's +2
15:32:25 <mgoddard> #topic Review priorities
15:32:54 <mgoddard> We have a few feature patches with RP+1
15:33:15 <mgoddard> I just added it to the TLS CA patch too
15:33:35 <mnasiadka> yoctozepto: just to be sure - are we merging ipv6 first and rebasing cells afterwards, or the other way around? :)
15:34:25 <mgoddard> I'd say it would be easier to merge cells first, as ipv6 is more search + replace friendly
15:34:31 <mgoddard> but whatever works
15:35:34 <yoctozepto> mnasiadka: ipv6 first for less CI work
15:35:51 <yoctozepto> mgoddard: you do realize it's more or less the same stuff either way? ;p
15:35:59 <yoctozepto> you either get it incoming or existing
15:36:01 <neolee> hello guys
15:36:02 <neolee> {"msg": "The conditional check 'easy_install_available' failed. The error was: template error while templating string: no test named 'version'. String: {{ not (ansible_distribution == 'Ubuntu' and\n        ansible_distribution_major_version is version(18, 'ge'))\n   and\n   not (ansible_distribution == 'Debian' and\n
15:36:02 <neolee> ansible_distribution_major_version is version(10, 'ge')) }}\n\nThe error appears to have been in '/usr/share/kolla-ansible/ansible/roles/baremetal/tasks/install.yml': line 104, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Install pip\n  ^ here\n"}
15:36:07 <yoctozepto> oh my
15:36:27 <mgoddard> yoctozepto: sort of
15:36:53 <mgoddard> ipv6 can lean on grep to verify correctness, cells cannot
15:37:26 <mgoddard> anyways
15:37:36 <mgoddard> neolee: we are currently in a meeting
15:37:56 <yoctozepto> mgoddard: what diffs are for
15:38:11 <yoctozepto> I guess you just have a different approach at things ;p
15:38:11 <neolee> ok
15:38:12 <openstackgerrit> Will Szumski proposed openstack/kayobe-config-dev master: Switch to using IPXE  https://review.opendev.org/688962
15:38:15 <yoctozepto> anyways, let's continue
15:38:20 <neolee> sorry
15:38:27 <mgoddard> for priorities, there will be some release patches coming. I'll RP+1 those
15:38:42 <mgoddard> Then let's focus on bugs
15:39:22 <mgoddard> Please check launchpad bug lists, so we can merge any fixes we have proposed
15:39:31 <mgoddard> Also gerrit review dashboards
15:40:16 <mgoddard> #link https://bugs.launchpad.net/kolla/train/+bugs
15:40:19 <mgoddard> #link https://bugs.launchpad.net/kolla-ansible/train/+bugs
15:40:31 <mgoddard> and beware any bugs not yet tagged as train
15:41:11 <mgoddard> Anything else we need to prioritise?
15:41:29 <yoctozepto> *chuckles* nfv *chuckles*
15:41:32 <yoctozepto> but it's on me
15:41:55 <mgoddard> #topic Which images should we mark as maintained in the support matrix?
15:42:18 <mgoddard> #link https://docs.openstack.org/kolla/latest/support_matrix.html
15:43:05 <yoctozepto> that is the question
15:43:19 <mgoddard> #link https://etherpad.openstack.org/p/kolla-train-image-evaluation
15:43:26 <mgoddard> I made a new section
15:43:31 <mgoddard> Images to mark as maintained
15:43:33 <mgoddard> L58
15:43:47 <mgoddard> throw up your suggestions, comment on others
15:44:17 <mgoddard> The definition of 'maintained' in this case is 'kolla core team is maintaining versions'
15:44:30 <mgoddard> but not tested in CI
15:45:03 <pbing19> I am building the container using local horizon source code. but somehow my customization changes are not coming into the horizon container
15:45:27 <yoctozepto> pbing19: meeting ongoing
15:45:46 <yoctozepto> mgoddard: images we care about I guess
15:45:50 <yoctozepto> we can repeat those T
15:46:09 <mgoddard> I guess anything with T & C, we bump C to M
15:46:36 <pbing19> yoctozepto:ok
15:48:06 <yoctozepto> mgoddard: yup
15:48:08 <mgoddard> anyone seeing the list in that etherpad?
15:49:29 <mnasiadka> yup
15:50:59 <mgoddard> are there others we should bump to M?
15:51:42 <mgoddard> does 'maintaining versions' have meaning for binary images?
15:52:09 <mgoddard> and for source images can we apply M to anything managed by version-check.py? i.e. most official openstack
15:52:36 <yoctozepto> mgoddard: not sure, I think M needs some knowledge from us
15:52:37 <yoctozepto> so yeah
15:52:40 <yoctozepto> at least source
15:52:44 <yoctozepto> binary excluded
15:52:54 <yoctozepto> and those we have experience with
15:53:03 <yoctozepto> (or at least pretend)
15:53:14 <mgoddard> what do you mean by knowledge?
15:54:11 <yoctozepto> mgoddard: have you ever deployed dragonflow
15:54:24 <mgoddard> yoctozepto: no, and it is deprecated :)
15:54:32 <yoctozepto> but surely you could find more :-)
15:54:52 <mgoddard> maybe we need a better definition of M
15:55:14 <mgoddard> we're not saying it's tested and working, that's T
15:55:25 <mgoddard> we're just saying we bump versions from time to time right?
15:56:10 <mnasiadka> yes
15:56:16 <yoctozepto> mgoddard: hmm, I think that's too lazy
15:56:35 <mgoddard> compare with elasticsearch, it will only be updated if someone proposes it
15:56:36 <yoctozepto> I thought it would be something like we don't test but we generally know how to troubleshoot
15:56:55 <yoctozepto> let's see what the list looks like when your idea is applied
15:56:59 <mgoddard> aodh though will be updated each release, and point releases within it
15:57:01 <yoctozepto> and then excluded exotic things
15:57:03 <yoctozepto> shall we
15:57:26 <mgoddard> ok
15:59:57 <hrw> morning
16:02:19 <mgoddard> new list up
16:03:15 <mgoddard> I think considering our job as maintainers here might help
16:03:25 <mgoddard> we don't promise to fix all bugs
16:03:48 <mgoddard> but part of our job for the images involves keeping the versions of deployed software up to date
16:04:17 <mgoddard> M could be interpreted as those things we regularly keep updated
16:04:32 <mgoddard> C is more community updated
16:04:53 <yoctozepto> hmm, ok
16:05:19 <mgoddard> if you have a better deinition I'm open to it
16:06:21 <mgoddard> anyway, we're out of time
16:06:28 <mgoddard> thanks all
16:06:34 <mgoddard> #endmeeting