15:59:52 <inc0> #startmeeting kolla
15:59:52 <hrw> or today rather /o\
15:59:54 <openstack> Meeting started Wed Dec 20 15:59:52 2017 UTC and is due to finish in 60 minutes.  The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:59:56 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:59:58 <openstack> The meeting name has been set to 'kolla'
16:00:03 <inc0> #topic w00t!
16:00:10 <inc0> you know what to do
16:00:10 <duonghq> wo0t
16:00:14 <hrw> /o\
16:00:16 <egonzalez90> Woot
16:00:22 <chason> Woot
16:00:23 <coolsvap> o/
16:00:25 <niedbalski> o/
16:00:38 <Jeffrey4l> o/
16:00:50 <rwellum> o/
16:00:58 <gema> o/
16:01:36 <dardelean_> o/
16:01:50 <krtaylor> w00t
16:02:17 <dardelean_> w00t
16:02:26 <inc0> #topic announcements
16:02:35 <inc0> so, next meeting will be canceled
16:02:54 <inc0> so merry chrismas, happy hannukah, have a good holiday
16:03:26 <inc0> any community announcements?
16:04:09 <Jeffrey4l> ocata 4.0.3 is finally released at the end of last week
16:04:23 <inc0> ah so was pike
16:04:27 <Jeffrey4l> and pike 5.0.1 is on the way
16:04:37 <inc0> right
16:04:39 <Jeffrey4l> waiting for review :D
16:05:05 <Jeffrey4l> that is all from me.
16:05:06 <inc0> ok, let's get to agenda, it's fairly busy today
16:05:12 <inc0> or anyone else have announcement?
16:05:34 <inc0> guess not:)
16:05:44 <inc0> #topic Image containers lifecycle: how to differentiate on docker registry between tested/untested images? Staging/stable? (gema)
16:05:49 <inc0> gema shoot
16:06:05 <gema> inc0: we were discussing today how to handle images lifecycle
16:06:17 <gema> if having one copy of them is good, or if we should have them tagged by date
16:06:25 <inc0> #link https://review.openstack.org/#/c/526469/
16:06:30 <inc0> check that out
16:06:32 <gema> or if we could make them staging until we test them and then promote them to released
16:06:39 <hrw> now we have 'queens-YYYYMMDD' tag in use
16:07:05 <inc0> that's what I came up for now - build => :pike-staging => pull/test => retag to :pike => push
16:07:16 <inc0> one thing, one image can have multiple tags
16:07:48 <gema> inc0: ok, we'll look into it
16:07:59 <gema> inc0: for queens we are going to have to do something similar but with manual testing
16:08:25 <gema> inc0: thanks
16:08:30 <inc0> well...once you have nodepool for arm, it'd be super easy to use this mechanism;)
16:08:36 <gema> inc0: yep
16:08:47 <gema> inc0: on the topic, I'd like to introduce niedbalski , our new devops lead
16:08:57 <gema> inc0: he is going to be fixing one of our clouds to be able to add that nodepool :)
16:09:04 <inc0> o/
16:09:19 <gema> inc0: and he'll be in dublin with us also
16:09:19 <krtaylor> hi niedbalski, welcome!
16:09:30 <niedbalski> krtaylor, inc0 o/
16:09:35 <egonzalez90> o/
16:09:58 <niedbalski> egonzalez90, o/
16:10:12 <inc0> so, gema we also discussed having tag like
16:10:14 <inc0> pike-123
16:10:19 <inc0> where 123 is autoincrement
16:10:37 <gema> inc0: but then how do you know which images are tested and good, vs just built and bad?
16:10:41 <inc0> so images won't be repulled automatically
16:10:51 <inc0> well staging can be the same
16:11:00 <inc0> as I said, one image can have multiple tags
16:11:03 <gema> ok
16:11:07 <gema> you can tag them after testing
16:11:08 <rwellum> Why push images that are bad?
16:11:13 <gema> rwellum: to be able to test them?
16:11:14 <inc0> so you can have both pike and pike-123 pointing to same image
16:11:16 <hrw> also the issue can be when image is not built because of some reason
16:11:26 <rwellum> Why test a bad image?
16:11:35 <gema> rwellum: you don't know is bad until you test it
16:11:39 <inc0> rwellum: you son't know if it's bad until it's tested
16:11:47 <gema> rwellum: I don't want my team building all different images they built themselves
16:11:49 <rwellum> We should test before pushing
16:11:52 <gema> then testing becomes non reproducible
16:11:54 <inc0> I mean if it fails building, you don't push it obviously
16:12:02 <gema> rwellum: hrw cannot test everything
16:12:10 <gema> rwellum: and he is our builder :D
16:12:17 <rwellum> Nah - we add tempest and run basic tests?
16:12:28 <inc0> rwellum: pushing -staging is just requirement of zuul really - we don't have good way to pass built images between gates
16:12:31 <hrw> and our CI job is simple: ./tools/build.py --push --some-other-options
16:12:51 <inc0> right, change I linked also deploys kolla-ansible
16:13:19 <hrw> inc0: deploy is another on a list ;D
16:13:24 <inc0> and have place for different scenarios, so at the end we can have testing of both k-ans and k-k8s in multiple scenarios
16:13:38 <gema> rwellum: we need to get to a point where we can run tempest on a deployed cloud automatically, we are just not there yet
16:13:38 <inc0> before pushing
16:14:03 <inc0> rwellum: what we could use is tempest in our regular gates
16:14:20 <inc0> after we have job like that we can add it to push pipeline
16:14:34 <rwellum> Right because to hrw point, CI without testing is not really CI imo.
16:14:53 <inc0> build alone is smoketest
16:15:02 <inc0> after that you jsut add more testing:)
16:15:06 <gema> yep
16:15:16 <hrw> rwellum: now our CI job is my replacement for days when I am under bus ;d
16:15:19 <gema> rwellum: but I am 100% in agreement
16:15:25 <rwellum> Wow inc0 :)
16:15:40 <gema> rwellum: it just takes some time to get to a point where tests can be automated without some workaround or another
16:16:19 <rwellum> I feel it's my mission to prove you wrong there :)
16:16:24 <Jeffrey4l> how about just extend our ci, and test the image before pushing them?
16:16:43 <gema> you guys are preaching to the converted here :D
16:16:45 <inc0> Jeffrey4l: review this https://review.openstack.org/#/c/526469/
16:16:47 <hrw> we are stage where all possible images can be built (aka those without 3rdparty and x86-only stuff). took far too long to get there but at least something
16:16:53 <inc0> this is exactly what you asked for:)
16:17:52 <gema> inc0: I will review that patch also
16:18:00 <gema> seems to answer our question exactly
16:18:02 <inc0> thanks
16:18:13 <hrw> and we do two new things at once: 1. !x86  2. Debian
16:18:19 <Jeffrey4l> ah, right.
16:18:36 <inc0> hrw: as I said, when we get arm nodepool
16:18:53 <hrw> inc0: soon (tm)
16:18:53 <inc0> we'll just create slew of jobs for arm there and all that work will be applicable
16:19:24 <inc0> we'll need new tag for you, like pike-arm-staging => pike-arm
16:19:26 <inc0> but that's fine
16:19:31 <inc0> easy enough
16:19:45 <hrw> s/arm/aarch64 ;D
16:19:54 <rwellum> tm = tomorrow hrw ?
16:19:54 <gema> hrw: missing the point x)
16:20:05 <hrw> rwellum: haha. I wish
16:20:07 <inc0> yeah, well, I wouldn't know, not that I work for processor maker...
16:20:15 <gema> inc0: lol
16:20:21 <hrw> inc0: then arm64. not arm
16:20:32 <inc0> anyway, can we move on to AArch64 topic?;)
16:20:36 <gema> yep
16:20:39 <hrw> yes
16:20:41 <inc0> #topic State of deploying on AArch64 (hrw)
16:20:45 <hrw> ok
16:20:49 <inc0> hrw: so what's the state:)
16:20:50 <hrw> so 1. docker
16:20:51 <inc0> ?
16:21:05 <hrw> does not look good without hacks
16:21:28 <hrw> https://review.openstack.org/#/c/522712/ takes care of 'docker daemon or dockerd' stuff
16:21:53 <hrw> I really wonder how many people use docker 1.12 (or whichever used 'docker daemon')
16:22:08 <inc0> but gates are bloody red so we can't merge it...before...wait for it...nodepool!;)
16:22:18 <hrw> without that patch 'bootstrap-servers' crash perfectly working docker-ce installation
16:22:41 <hrw> inc0: none of your gates run debian. we change debian only
16:23:28 <hrw> without it merged in this form or other we do not have deploy at all
16:23:56 <hrw> questions to this part?
16:24:07 <rwellum> We use docker-ce?
16:24:11 <hrw> we do
16:24:16 <hrw> 17.06
16:24:19 <Jeffrey4l> could you fix the gate at the same time?
16:24:28 <Jeffrey4l> hrw, something is broken.
16:24:35 <inc0> well...1.12/1.13 is what distros carry usually
16:24:43 <Jeffrey4l> http://logs.openstack.org/12/522712/8/check/kolla-ansible-centos-source/8f4125e/primary/ansible/bootstrap-servers
16:24:50 <inc0> http://logs.openstack.org/12/522712/8/check/kolla-ansible-centos-source/8f4125e/primary/ansible/bootstrap-servers
16:24:54 <inc0> ;)
16:24:57 <hrw> Jeffrey4l: will look
16:25:21 <Jeffrey4l> why can't we have a widely support for docker version?
16:25:29 <Jeffrey4l> kolla works if docker > 1.10
16:25:45 <Jeffrey4l> hrw thanks.
16:25:47 <hrw> Jeffrey4l: ask docker guys to handle upgrades etc. they do not care much
16:25:56 <hrw> ;'d
16:26:23 <hrw> I would love to see docker package in debian/stable but docker way of development == no package for stable at all
16:26:27 <Jeffrey4l> hrw, fyi, we use docker 1.12.6 for a long time. and migrate to docker-ce 17.9 recently.
16:26:39 <inc0> that's pretty much why we pin to 1.12 / 1.13
16:26:53 <rwellum> I thought the move was to using CNFI docker - but maybe that's more specific to kubernetes community
16:27:02 <inc0> upgrades/breaking things is not docker inc first priority
16:27:10 <Jeffrey4l> docker changed too much after 1.13. including its name and deployment progress.
16:27:19 <hrw> #action hrw to check http://logs.openstack.org/12/522712/8/check/kolla-ansible-centos-source/8f4125e/primary/ansible/bootstrap-servers failures
16:27:45 <hrw> 2. cpu_mode for nova - just ignore now. have to debug and improve
16:28:12 <hrw> 3. mariadb boostrap. or rather galera
16:28:46 <hrw> looks like centos/ubuntu on x86 has galera enabled always (wsrep_on = ON)
16:29:00 <hrw> in Debian it has to be set in config or via cli
16:29:45 <hrw> with http://paste.debian.net/1001704/ I have mariadb container which says " wsrep_ready   | ON "
16:30:09 <hrw> but I assume that it will break other distros etc as this is a hack
16:30:39 <inc0> well gates pass so it's not horrendous
16:30:41 <Jeffrey4l> hrw, with that patch, debian will not support cluster.
16:30:44 <inc0> it might break on multinode
16:31:03 <hrw> so far I am unable to do all-in-one
16:31:16 <Jeffrey4l> still do not get the point what should be done in debian side.
16:31:27 <Jeffrey4l> hrw, it will break multi node in debian.
16:31:33 <egonzalez90> Try with mariadb 10.0 rather 10.1
16:31:44 <hrw> Jeffrey4l: neither do I. times when I used mysql as admin were gone 10y ago
16:31:49 <hrw> egonzalez90: 10.1 is in repo
16:31:57 <hrw> egonzalez90: we do not use 3rdparty repos with Debian
16:32:22 <Jeffrey4l> hrw what error did you get?
16:32:27 <egonzalez90> Is what we use in other distros
16:32:59 <Jeffrey4l> why not try to enable wsrep by configuration?
16:33:05 <hrw> Jeffrey4l: thats where fun starts - fatal: [10.101.3.104]: FAILED! => {"changed": false, "failed": true, "msg": "unable to connect to database, check login_user and login_password are correct or /var/lib/ansible/.my.cnf
16:33:09 <hrw> has the credentials. Exception message: (1045, \"Access denied for user 'root'@'c8n1' (using password: YES)\")"}
16:33:54 <hrw> egonzalez90: I got used to be that crazy guy who does some stuff first in kolla
16:34:38 <gema> hrw: he has a point though, if 10.0 works then we can be sure it is the new version
16:34:38 <hrw> I will probably start again with deploys in x86-64 vm to find out how the hell it works at each step
16:34:42 <gema> else it could be configuration
16:34:49 <gema> can we use ubuntu's package to try?
16:35:14 <gema> or that, sounds good also
16:35:18 <hrw> gema: I would rather dig out 10.0 from Debian. that would also require digging out old galera package
16:35:30 <gema> hrw: ok, sound good
16:35:48 <hrw> gema: using Ubuntu package would be like using CentOS rpms basically - no idea what was changed and why
16:35:58 <gema> hrw: understood
16:36:44 <hrw> I hope that Xinliang will get further than I did. He has more experience now with k-a
16:36:51 <gema> hrw: he will
16:36:55 <gema> give him some time
16:37:15 <hrw> gema: I do not touch kolla stuff this year probably.
16:37:45 <gema> anyway, I think we can move to your next topic
16:37:47 <gema> if you had more
16:37:51 <hrw> nope, finished
16:37:58 <inc0> that's it with agenda
16:38:02 <inc0> #topic open discussion
16:38:06 <inc0> anyone?
16:38:25 <hrw> I wish kolla*start etc would be in k-a instead of images ;d
16:38:36 <hrw> easier to hack and deploy ;D
16:39:01 <rwellum> I have a fairly large PS for kolla-k8s - adding V3 api's if anyone wants to look...
16:39:02 <Jeffrey4l> yep i thought of this too.
16:39:14 <inc0> https://review.openstack.org/#/c/526200/ speaking of which:)
16:39:31 <inc0> I'll address stuff from review today but I'll need more feedback
16:39:55 <inc0> and agree, after hub images kolla becomes much less useful for users
16:40:04 <inc0> kolla-ansible is standalone now
16:40:31 <inc0> building images moved from requirement to advanced usage
16:41:02 <hrw> s/eth0/eno1 s/eth1/eno1d1 ;d
16:41:15 <rwellum> inc0: most companies probably have their own source tho right? Not sure that statement is correct unless you are deploying pure upstream.
16:41:25 <gema> rwellum: for some images
16:41:40 <hrw> speaking of hub. is there a way to list someone's hub.docker images?
16:41:46 <rwellum> gema: ack
16:41:51 <inc0> rwellum: I don't think so tbh, I think most companies (at least smaller ones) uses just openstack
16:41:57 <hrw> kind of 'docker hubimages kolla' for example
16:42:26 <inc0> hrw: https://review.openstack.org/#/c/526469/11/tests/playbooks/publish_release.yml line 35
16:42:43 <hrw> inc0: thanks
16:43:32 <hrw> anything else?
16:44:06 <inc0> so projects are now separate, kolla and kolla-ansible
16:44:17 <inc0> there is no "default" doc now
16:45:47 <chason> inc0 Maybe we can link both kolla and kolla-ansible's docs together. :)
16:46:43 <inc0> yeah
16:46:52 <inc0> we do it now kinda
16:46:59 <inc0> but maybe it's not fully clear
16:47:23 <inc0> anything else on topic? anyone want to discuss anything?
16:47:48 * hrw nope
16:48:34 <inc0> allright
16:48:41 <inc0> let's get 10min back in our lives:)
16:48:49 <inc0> merry chrismas everyone!
16:49:04 <inc0> #endmeeting kolla