15:58:16 <inc0> #startmeeting kolla
15:58:17 <openstack> Meeting started Wed Jan 10 15:58:16 2018 UTC and is due to finish in 60 minutes.  The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:58:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:58:21 <openstack> The meeting name has been set to 'kolla'
15:58:24 <inc0> #topic w00t!
15:58:26 <duonghq> o/
15:58:29 <duonghq> wo0t
15:58:39 <inc0> good morning/afternoon/evening everyone!
15:59:11 <chmarkus> hello
16:00:04 <chmarkus> I'm completely new to this but would like to attend the kolla meeting too, is this okay?
16:00:12 <coolsvap> Ao/
16:00:12 <hrw> o/
16:00:21 <rwellum> o/
16:00:26 <Jeffrey4l> o/
16:00:33 <inc0> chmarkus: not only okay but encouraged and great:) welcome!
16:00:54 <chmarkus> inc0, do I need to raise my arm too? :D
16:01:00 <egonzalez> o/ woot
16:01:13 <gema> o/
16:01:33 <inc0> yes, raising and waving arm is mandatory.
16:01:40 <chmarkus> o/
16:02:31 * gema gema giggles at the waving "arm"
16:02:41 * hrw waves arm64
16:02:41 <chason_> o/
16:03:02 * rwellum groans at hrw joke
16:03:02 <hrw> no arm32 in range ;d
16:03:03 <inc0> lol
16:03:14 <inc0> 64 arms of Kolla
16:03:30 <hrw> inc0: good idea for ptg photo
16:03:52 <inc0> my managers would disagree;)
16:03:52 <hrw> but first have to find 32 users/devs
16:04:07 <inc0> anyway, let's move on to actual meeting
16:04:11 <inc0> #topic announcements
16:04:45 <inc0> I'll be away for next 2 weeks or so people, Jeffrey4l will handle next 2 meetings
16:05:21 <Jeffrey4l> np inc0 :D
16:05:28 <inc0> thank you:)
16:05:48 <inc0> #link https://etherpad.openstack.org/p/kolla-rocky-ptg-planning
16:06:14 <inc0> rwellum was nice enough to start ptg etherpad, let's start planning sessions and what we want to achieve there
16:07:16 <inc0> any community announcements?
16:07:52 <inc0> guess not
16:07:57 <hrw> debian/aarch64 support fully merged into kolla - we may not have to push things into queens ;D
16:08:15 <hrw> one patch left in kolla-ansible and also ready for testing
16:08:16 <inc0> rocky*;) queens is current release
16:08:24 <inc0> also congrats! awesome job
16:08:46 <inc0> make sure to add juicy release note about that
16:08:48 <inc0> and some docs
16:08:48 <egonzalez> hrw, you done? awesome o/
16:08:50 <rwellum> +1
16:08:56 <egonzalez> +1
16:09:08 <Jeffrey4l> +1
16:09:10 <hrw> egonzalez: you as Linaro team I assume ;d
16:09:18 <egonzalez> yep
16:09:34 <gema> we are doing manual testing
16:09:43 <gema> and starting the ARM64 gates
16:09:48 <gema> which may or may not make it for queens
16:10:06 <inc0> let's talk about this on this meeting please
16:10:16 <inc0> for now, let's move on to agenda item
16:10:21 <inc0> #topic mariadb 10.1
16:10:29 <inc0> Jeffrey4l: that was yours?
16:10:34 <Jeffrey4l> np..
16:11:01 <hrw> we still not moved rpm distros and ubbuntu to 10.1
16:11:37 <inc0> ah right, and debian has just 10.1
16:11:43 <hrw> or in other words: debian got everything for 10.1 now
16:11:56 <inc0> including tested upgrade?
16:12:02 <hrw> https://review.openstack.org/531656 is the only debian/mariadb change
16:12:11 <hrw> inc0: we do not have 10.0 setups
16:12:42 <Jeffrey4l> there is no xtrabackup in debian?
16:12:57 <hrw> and no one tested centos/aarch64 nor ubuntu/aarch64 - both may be at 10.1 anyway too
16:12:58 <inc0> we will need to make 100% sure we can upgrade before putting this to buntu/centos
16:13:00 <Jeffrey4l> aha,ignore above.
16:13:11 <Jeffrey4l> mariabackup is a better choice.
16:13:35 <hrw> inc0: indeed.
16:14:19 <inc0> we're 3 weeks before rc1, do you guys think it's enough time to do it?
16:14:39 <duonghq> inc0, have we had upgrade logic?
16:14:45 <inc0> I mean, change itself shouldn't be that hard but it's data we're talking about so we should be very careful
16:15:08 <inc0> duonghq: we didn't really upgrade mariadb between middle versions
16:15:19 <inc0> which means we didn't really need it till now
16:17:12 <duonghq> as mariadb docs, it should be done quite easy, but anybody can confirm it?
16:17:34 <duonghq> #link https://mariadb.com/kb/en/library/upgrading-from-mariadb-100-to-mariadb-101/
16:18:20 <duonghq> https://mariadb.com/kb/en/library/upgrading-from-mariadb-galera-cluster-100-to-mariadb-101/
16:18:38 <duonghq> we have galera for test too
16:18:49 <duonghq> I think it's risky enough
16:20:14 <inc0> hmm...can we bump it to rocky?
16:20:42 <duonghq> IMHO, +1 for Rocky
16:20:45 <inc0> persistent data upgrades is not something I'd like to rush
16:21:01 <inc0> hrw: debian works now right?
16:21:06 <duonghq> or should we start vote on ml?
16:21:10 <gema> we have not tested upgrades either
16:21:24 <Jeffrey4l> f you are running an older 10.0 version with Galera 25.2.x, it is recommended to first upgrade to the latest 10.0 version, running Galera 25.3.x. See Galera versions for an indication.
16:21:30 <inc0> vote will take us even closer to release
16:21:31 <Jeffrey4l> we need confirm this at first.
16:21:47 <hrw> inc0: yes
16:22:05 <inc0> let's do this, if someone picks up this task with requirement of thorough upgrade testing, we might squeeze it into Q
16:22:19 <hrw> Jeffrey4l: so maybe let upgrade centos/ubuntu to latest 10.0 + 25.3.x and then in rocky do 10.1?
16:22:30 <inc0> but we need volunteer for that. unless we find this person, it's one of first things we'll work on in Rocky
16:23:25 <Jeffrey4l> hrw i am not sure. the official docs said that. no idea why will happen if we don't do this.
16:23:37 <hrw> that way we get latest stuff in Q, people will upgrade to Q and then with R they migrate
16:24:02 <hrw> and release note of R needs to have "you have to go through Q or upgrade eats your kitten"
16:24:47 <hrw> kind of 'safe way'
16:25:37 <duonghq> hrw, afaik, nobody in OPS world want to jump over one version :P
16:25:52 <inc0> duonghq: you'd be surprised...
16:25:59 <hrw> duonghq: never had to work in devops
16:26:11 <inc0> anyway, openstack itself doesn't support skip version, neither will we
16:26:12 <hrw> duonghq: we will do jump newton-queens at linaro clouds
16:26:27 <inc0> and non-kolla to kolla
16:26:30 <hrw> yep
16:26:31 <egonzalez> centos is using latest 10.0 version, we must be good with this requirement
16:26:32 <inc0> you guys will have interesting times;)
16:26:47 <gema> inc0: you washing your hands of it? :D
16:26:58 <Jeffrey4l> the good new is: centos is using mariadb 10.0 + galera 25.3
16:27:12 <duonghq> inc0, I agreed
16:27:13 <Jeffrey4l> so it is safe to upgrade to 10.1
16:27:27 <duonghq> Jeffrey4l, good news
16:27:27 <inc0> cool
16:27:41 <hrw> Jeffrey4l: Q or P you mean?
16:27:49 <Jeffrey4l> since ocata
16:27:55 <hrw> coolio
16:28:22 <hrw> what about ubuntu?
16:29:18 <Jeffrey4l> checking ~~
16:29:28 <hrw> ubuntu has 10.0.33 + 25.3.22 at http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.0/ubuntu/pool/main/
16:29:33 <egonzalez> repos point to 10.0 too, is not pinned afik
16:31:22 <Jeffrey4l> cool. at least all the distro are ready to bump to mariadb 10.1
16:32:15 <inc0> ok, soo...anyone up for a challenge?
16:33:25 <inc0> crickets
16:33:27 <gema> xD
16:33:28 <Jeffrey4l> based on the doc, what's the challenge is?
16:33:48 <inc0> well bump versions and test upgrade
16:34:13 <Jeffrey4l> i think just like upgrade from ocata to pike ( which mariadb version is not changed )
16:34:17 <Jeffrey4l> test is need ..
16:35:05 <inc0> right, that's the challenge:)
16:35:24 <inc0> anyway, if someone wants to take on it, cool, if not, Rocky
16:35:48 <inc0> and high priority task in R
16:35:54 <inc0> do we agree on this?
16:36:19 <Jeffrey4l> +1
16:36:20 <hrw> +2
16:36:21 <chason_> +1
16:36:24 <duonghq> +1 for make it high priority in R if we cannot make it roll in Q
16:36:25 <gema> +1
16:36:31 <inc0> ok, cool, let's move on:)
16:36:38 <inc0> #topic open discussion
16:36:44 <chmarkus> o/ may I add another topic?
16:36:48 <hrw> chmarkus: go
16:36:58 <inc0> chmarkus: yes, now is the time
16:36:59 <chmarkus> https://blueprints.launchpad.net/kolla/+spec/support-alpine-distro-base-image
16:37:04 <hrw> chmarkus: next time add topic to wiki page too
16:37:18 <chmarkus> yes, sorry need to make an OpenID account yet
16:37:20 <inc0> but before we move to it, we already started discussion about different architectures
16:37:22 <chmarkus> next time ;)
16:37:27 <inc0> let's finish this and we'll move to alpine
16:37:32 <chmarkus> ok
16:37:55 <inc0> so, biggest issue for us now will be zuul support for archs
16:38:15 <inc0> we'll need to work with infra to add somethign like this
16:38:40 <gema> inc0: happy to take on that task
16:38:45 <hrw> gema: how things are when it comes to infra talk?
16:39:02 <gema> hrw: email sent this morning, PTL aligned with us but wanted to open the discussion to wider audience
16:39:04 <inc0> they're really cool people and we'll figure it out
16:39:18 <inc0> yeah, let's join this openstack-dev discussion
16:39:20 <gema> people congratulating us for the work privately but no open responses yet on the ML
16:39:29 <gema> it is a bad time in the cycle also
16:39:33 <gema> with the release happening
16:40:12 <gema> plus this is not something we dump on infra, we'll make it work
16:40:15 <gema> and submit patches, as usual
16:40:15 <inc0> and I'm sure zuulv3 is still having hickups every now and then
16:40:29 <gema> inc0: but kolla is not fully on zuulv3 yet, right?
16:40:34 <inc0> it is
16:40:41 <gema> ok, so starting there will be good
16:40:51 <inc0> kolla-k8s is not yet, but that's not relevant here
16:40:55 <gema> ack
16:40:55 <hrw> there is only zuul. same on nova
16:41:04 <inc0> good news is the moment we'll figure out gates, publisher comes automatically
16:41:12 <gema> goodio
16:41:28 <gema> so let's see what infra say and then I can start adding stuff, adding yaml files in places, and installing stuff
16:41:29 <hrw> o yes. end of 'out of disk space' in a middle of a build
16:41:34 <gema> I don't want to dump this on them
16:41:45 <gema> but we'll need their expertise to make it work
16:41:47 <gema> most likely
16:42:00 <inc0> you'll need their help anyway, but again, I'm sure they'll be happy to assist
16:42:07 <gema> yup
16:42:49 <gema> inc0: and as soon as we have the thing going, our members will happily add more hardware to it
16:42:49 <inc0> PTG is close enough as well so if it turns out to be tricky, we can discuss it there
16:43:00 <gema> yep
16:43:04 <inc0> that's great to hear, thank you!
16:43:28 <inc0> ok, anything else on this topic?
16:43:40 <gema> nope, thanks for volunteering as guinea pigs :D
16:43:58 <egonzalez> #link https://blueprints.launchpad.net/kolla-ansible/+spec/mariadb-version-upgrade
16:44:15 <inc0> thanks egonzalez
16:44:23 <inc0> ok, let's move to alpine
16:44:28 <inc0> chmarkus: you have the floop
16:44:30 <inc0> floor
16:44:33 <chmarkus> thank you
16:44:53 <inc0> so on that note, we discussed alpine before
16:45:00 <inc0> like 2 years ago
16:45:15 <inc0> back then alpine didn't have many packages we require
16:45:17 <inc0> like ceph
16:45:20 <inc0> or galera
16:45:27 <inc0> so it was a no-go for us
16:45:33 <inc0> now things might've changed
16:45:57 <chmarkus> for my company I'm currently trying to add alpine support to kolla - I'd like to collab with you as much as possible to provide any enhancement upstream
16:46:15 <inc0> well if you guys work on it, let's do it upstream
16:46:21 <inc0> many people would be happy with this
16:46:28 <inc0> and I'm sure you'd have help
16:46:35 <gema> yep, we are interested
16:46:38 <chmarkus> that would be great
16:46:38 <inc0> biggest issue were this package availability
16:46:42 <chmarkus> I'm aware that not all services/containers will be compatible with alpine
16:47:17 <hrw> I think that first you need to get base image working. then mariadb, openstack-base and target all-in-one setup
16:47:18 <inc0> well if some less popular services would be ok I guess, we can document that some things won't work with alpine
16:47:38 <inc0> but ceph was one example of *really* important service that was missing
16:47:39 <chmarkus> the goal is to provide alpine support for every core service that proves to be compatible
16:48:05 <chmarkus> I would like to fallback to other base images (e.g. ubuntu) for incompatible ones
16:48:19 <hrw> chmarkus: you are adding new distro, new way of installing packages etc. so getting base may take a bit
16:48:34 <inc0> soooo ceph is very outdated
16:48:36 <chmarkus> I started looking into base today
16:48:37 <inc0> out there
16:49:08 <inc0> well due to layering if you need to fallback to different distros, you'll lose all advantages from alpine
16:49:36 <hrw> inc0: are there no plans to move to ceph-ansible in R anyway?
16:49:51 <chmarkus> but aren't the services separate containers?
16:49:58 <inc0> I guess compute node services are all possible to be on alpine so that's good
16:50:06 <inc0> hrw: we don't have it fully planned yet
16:50:12 <hrw> ko
16:50:13 <egonzalez> hrw, we need a upgrade path to them
16:50:22 <gema> hrw: the move will happen after luminous, (according to last discussion)
16:50:26 <inc0> chmarkus: they are, but alpine's big benefit is being small
16:50:55 <inc0> and if you have ubuntu already downloaded, adding next ubuntu container will only consume space of layers added
16:51:00 <hrw> linaro/debian-source-nova-compute                queens-20180109     dc3287faeab7        22 hours ago        1.29GB
16:51:07 <hrw> beat that ;d
16:51:18 <inc0> so having 10 ubuntu based containers will not consume 10*image size, just 1*base image + diff
16:51:22 <chmarkus> inc0, okay I get your point
16:51:40 <chmarkus> but isn't it also an improvement in regards to security/reliability?
16:51:43 <inc0> it will be beneficial to have compute node services in alpine tho
16:51:49 <inc0> as usually they're more numerous
16:51:58 <inc0> alpine over say centos?
16:52:07 <hrw> chmarkus: 60 debian images (3 different base images) take 8.5GB
16:52:20 <inc0> I wouldn't argue otherwise - centos/ubuntu repos are more curated than alpine
16:53:06 <inc0> but having control on ubuntu and compute on alipne should be possible
16:53:15 <inc0> maybe something to consider
16:53:27 <inc0> I'd start with use case like that
16:53:35 <hrw> :)
16:54:03 <inc0> chmarkus: in any case, I'd start with some PoC
16:54:45 <chmarkus> that's what I'm planning to do
16:54:57 <inc0> cool
16:55:03 <inc0> I'd be interested to see it:)
16:55:10 <chmarkus> going from base to openstack-base towards one service, let's say nova
16:55:32 <hrw> good idea
16:55:35 <inc0> so try doing all compute services on alpine
16:55:49 <inc0> that's nova, openvswitch, neutron, libvirt
16:56:21 <duonghq> I think nova + libvirt is enough for PoC
16:56:26 <inc0> yeah
16:56:59 <inc0> I'm pretty sure we won't be able to make storage/control nodes on alpine at least until ceph is better supported
16:56:59 <chmarkus> okay, so I guess it would be beneficial If I attend this meeting regularly for the organizational part?
16:57:11 <gema> +1
16:57:16 <inc0> make sure to join #openstack-kolla irc channel
16:57:22 <chmarkus> and discuss the technical stuff on the other channel
16:57:24 <inc0> this is our day-to-day communication
16:57:39 <hrw> and release early, release often
16:57:58 <inc0> for harder stuff [openstac-dev] mailing list will ensure longer discussin, but broader audience
16:58:10 <inc0> and welcome:)
16:58:22 <duonghq> btw, we will remove internal ceph soon, so do we still need ceph officially?
16:58:28 <inc0> anyway, we're running out of time
16:58:34 <chmarkus> thanks, looking forward to the collab
16:58:41 <inc0> duonghq: well *if* we remove it soon then no
16:58:49 <duonghq> ack
16:58:53 <inc0> but that's not decided 100% and won't be easy
16:59:09 <duonghq> yep
16:59:10 <inc0> because we need to provide migration path to whatever our new solution for ceph would be
16:59:11 <gema> we need ceph until ceph-ansible supports ARM64 :S
16:59:17 <gema> which we are working on
16:59:33 <inc0> ok, we ran out of time
16:59:37 <inc0> thank you all for comming!
16:59:44 <gema> thank you!
16:59:44 <inc0> #endmeeting