16:00:13 <Jeffrey4l> #startmeeting kolla
16:00:14 <openstack> Meeting started Wed Apr 18 16:00:13 2018 UTC and is due to finish in 60 minutes.  The chair is Jeffrey4l. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:18 <openstack> The meeting name has been set to 'kolla'
16:00:18 <Jeffrey4l> #topic rollcall
16:00:27 <spsurya> \0
16:00:40 <caoyuan> \o
16:00:41 <mgoddard> o/
16:00:45 <bmace> o/
16:00:52 <sgoel_> o/
16:00:59 <mgiles> o/
16:01:49 <chason> o/
16:01:52 <yankcrime> \o
16:02:21 <Jeffrey4l> #topic Announcements
16:02:32 <pbourke> o/
16:03:00 <Jeffrey4l> after some discuss, we still think kolla-k8s should be retired.
16:03:25 <Jeffrey4l> i started a vote on ML, feel free to reply it.
16:03:27 <Jeffrey4l> #link http://lists.openstack.org/pipermail/openstack-dev/2018-April/129452.html
16:03:35 <rwellum> o/
16:03:43 <Jeffrey4l> any others from community?
16:04:52 <Jeffrey4l> ok move on
16:04:58 <Jeffrey4l> #topic ceph pg limits in luminous (pbourke)
16:05:05 <Jeffrey4l> pbourke, yours floor
16:05:13 <pbourke> thanks, one moment just trying to find the bug
16:05:56 <duonghq> o/
16:05:58 <duonghq> sorry for late
16:06:01 <Jeffrey4l> #link https://bugs.launchpad.net/bugs/1763356
16:06:02 <openstack> Launchpad bug 1763356 in kolla-ansible "ceph deployment fails with not enough pages" [Critical,Confirmed]
16:06:04 <Jeffrey4l> pbourke, ^^
16:06:23 <pbourke> thanks Jeffrey4l
16:06:26 <pbourke> that's the one
16:06:29 <pbourke> in summary
16:06:46 <pbourke> our images are currently using ceph 12.2.1
16:07:02 <pbourke> in 12.2.2 they seem to have changed the limit on how many pgs you can have per osd
16:07:20 <pbourke> im not sure why our images haven't picked up this version, but as soon as we do, things will break
16:07:48 <pbourke> vbel has posted a solution on the bug, I just wanted to bring this up and see if we're happy with this solution
16:08:30 <pbourke> essentially, we should set the pg size for the pools we create to a very small value
16:08:48 <pbourke> and make it very clear via documentation or a warning message that operators need to revise these before deploying to production
16:09:12 <pbourke> kolla cannot accurately calculate these up front as it depends on the size and shape of each individual cluster
16:09:23 <Jeffrey4l> the issue is kolla don't know what's the ceph cluster size before deploying.
16:09:26 <pbourke> right
16:09:57 <pbourke> I think this makes sense, but Im not a ceph expert :)
16:10:15 <Jeffrey4l> use a smaller number for "ceph_pool_pg_num" works.
16:10:41 <pbourke> ok, we'll post a patch and people can review. Far as I can tell it wont be an issue for upgrades but will check this also
16:12:21 <Jeffrey4l> no matter how the number is. the operator still could hit this issue.
16:12:43 <Jeffrey4l> for example there is only one ceph-osd process
16:13:06 <Jeffrey4l> hrm... ignore ^^
16:13:10 <pbourke> perhaps we could perform some calcuations up front?
16:13:32 <pbourke> *calculations
16:13:55 <Jeffrey4l> pbourke, there are some recommendation from https://ceph.com/pgcalc/
16:14:26 <pbourke> yeah
16:14:52 <pbourke> I think in the future this problem may go away anyhow, I read something about ceph looking to make pg configuration more transparent
16:15:16 <Jeffrey4l> cool. interesting
16:15:56 <pbourke> #link https://ceph.com/community/new-luminous-pg-overdose-protection/
16:16:30 <Jeffrey4l> too long. will read it later.
16:16:32 <pbourke> so ok I think that's it for now
16:16:53 <caoyuan> pbourke thanks pbourke:)
16:17:35 <Jeffrey4l> for kolla i think we can reduce the ceph pool pg num and add some note in doc at the same time. because this issue can not be ignore so far in kolla.
16:18:56 <pbourke> can we move on?
16:19:07 <Jeffrey4l> yes.
16:19:19 <Jeffrey4l> #topic  Review clean up (pbourke)
16:19:33 <Jeffrey4l> still yours pbourke
16:19:33 <pbourke> #link https://review.openstack.org/#/q/project:openstack/kolla-ansible+status:open
16:19:49 <pbourke> so just wanted to highlight that we still have reviews open from mid 2017
16:20:01 <pbourke> I propose that these are tidied up
16:20:24 <pbourke> this looks better overall and makes it easier for reviewers to find whats important
16:20:39 <pbourke> unsure what criteria other projects use, I was going to suggest either 3 or 6 months
16:20:49 <pbourke> what do people think?
16:20:57 <Jeffrey4l> reasonable.
16:21:08 <harlowja> +1
16:21:10 <spsurya> pbourke: reasonable to me too
16:21:11 <mgoddard> +1, with a warning before abandoning
16:21:16 <caoyuan> +
16:21:17 <caoyuan> 1
16:21:26 <Jeffrey4l> i wonder is there any bot could do this in openstack now.
16:21:47 <pbourke> what Ive seen in the past is a core will simply abandon with something like "thanks for looking at this, if you're interested in continuing this work feel free to reopen"
16:22:06 <harlowja> ya i'd recommend ^
16:22:07 <pbourke> a bot would be useful but I dont think there's too many
16:22:13 <duonghq> +1, I prefer 6mo, we should left a bit more detail like: no activity in xxx mo
16:22:26 <pbourke> so I'm happy to go ahead and do this
16:22:31 <harlowja> i'd rather these have the human touch to it; people may have put in blood/sweat and tears into some of those changes
16:22:31 <pbourke> duonghq: 6 months is fine
16:22:54 <pbourke> is trivial for people to reopen, it just takes them off our radar
16:22:59 <pbourke> *it's
16:23:00 <Jeffrey4l> i saw some simliar bot in launchpad. btw we should cleanup bugs in the same way
16:23:11 <pbourke> oh, yes even more so for bugs
16:23:18 <spsurya> Also before that if we try to reach author at once then post the comment about abandon etc
16:24:01 <Jeffrey4l> for bugs, i found this script https://github.com/openstack-infra/release-tools/blob/master/expire_old_bug_reports.py
16:24:12 <mgoddard> if a bug is confirmed but not fixed, should it still be closed?
16:24:59 <pbourke> mgoddard: hard to say, if old enough if may no longer be valid
16:25:11 <mgoddard> I recently attended a bug triage session with the ironic team. That could be another way to tackle it
16:25:32 <pbourke> sounds good
16:25:54 <pbourke> I find it difficult to keep up with the bugs, if we had a formal session every now and then we might do better at keeping on top of them
16:26:25 <Jeffrey4l> from the script, it will mark the bug as close  if there is no update for 18 month
16:26:27 <mgoddard> they also have a bug liaison person assigned each week, who keeps track of new bugs
16:26:45 <mgoddard> 18 months seems a reasonable amount of time
16:27:23 <mgoddard> although I think harlowja's point about the human touch applies here too
16:27:46 <harlowja> ya, just keep in mind that there are people behind those bugs/reviews ...
16:27:51 <harlowja> with feelings and such :-P
16:28:02 <Jeffrey4l> mgoddard, +1 https://github.com/openstack-infra/release-tools/blob/master/expire_old_bug_reports.py#L162-L173
16:28:17 <chason> Maybe we can just fix  bugs of the maintained realease.
16:29:10 <Jeffrey4l> mgoddard, triage session sound a good solution too.
16:29:24 <Jeffrey4l> chason, what is that meaning?
16:29:52 <mgoddard> Jeffrey4l: It allows us to pool knowledge, rather than rely on one person with a limited domain
16:30:40 <Jeffrey4l> mgoddard, do you konw how the session is orgranized?
16:31:15 <mgoddard> Jeffrey4l: there was a ML post, I'll find it
16:31:23 <Jeffrey4l> chason, since we haven't stable-policy tag now. in fact we could backport some bp-like patch check http://lists.openstack.org/pipermail/openstack-dev/2018-April/129265.html
16:31:39 <Jeffrey4l> cool. i will check that later.
16:32:43 <Jeffrey4l> so for this topic, the conclusion for now is: we will close the bp(6 months)/bugs(18months) with a proper comments.
16:32:49 <Jeffrey4l> manully or automatically.
16:33:16 <mgoddard> https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg117769.html
16:33:25 <mgoddard> also see follow up emails for format
16:33:35 <mgoddard> it was a video call via bluejeans
16:34:01 <Jeffrey4l> yep
16:34:47 <Jeffrey4l> maybe we could scheduler one if there is enough guy who interested this
16:35:26 <Jeffrey4l> ok. seems we are done for this topic
16:35:29 <Jeffrey4l> let's move
16:35:38 <Jeffrey4l> #topic Open Discussion
16:35:46 <Jeffrey4l> any volunteer?
16:36:21 <mgoddard> I made a demo of kayobe
16:36:35 <mgoddard> Fast version: https://asciinema.org/a/176888?speed=2
16:36:45 <mgoddard> Real time: https://asciinema.org/a/176883
16:37:05 <mgoddard> There will be a blog post to follow shortly
16:37:11 <Jeffrey4l> cool
16:37:50 <chason> mgoddard nice work! :)
16:38:19 <mgoddard> Also, I recently created a proof of concept CI job for bifrost
16:38:26 <pbourke> nice one
16:38:29 <mgoddard> https://review.openstack.org/549775
16:38:38 <harlowja> so i have a question
16:38:42 <mgoddard> it worked at one point, but is now failing again :)
16:38:53 <harlowja> i've been trying to get the sensu images to build (we use source mode)
16:38:55 <mgoddard> is this something the kolla team would be interested in having?
16:39:03 <spsurya> mgoddard: will watch your nice job
16:39:03 <harlowja> but i've noticed http://paste.openstack.org/show/719424/ happens in source mode (on centos)
16:39:24 <harlowja> so i'm thinking that we may want to emit some kind of error for kolla building of sensu in source mode (until thats adddressed)
16:40:01 <Jeffrey4l> mgoddard, yep. more jobs is welcome.
16:40:26 <mgoddard> cool, I'll fix the issue and add some reviewers
16:40:41 <harlowja> i think it will work in binary mode, though on centos i'm having to force add `centos-release-openstack-pike`
16:40:55 <harlowja> so wondering if anyone else has tried those sensu containers?
16:41:02 <pbourke> harlowja: I'm not very familiar but  - we dont build source containers for non openstack components, so the binary/source workflow for sensu should be the same?
16:41:24 <harlowja> pbourke well it can be tried, it just craps out pretty late in the build process
16:41:29 <harlowja> so it almost feels better to die early
16:41:57 <Jeffrey4l> sensu is a binary install. and it depends on some oenstack client.
16:42:10 <pbourke> harlowja: ping me the review and I'll take a look
16:42:31 <Jeffrey4l> but why the novaclient is installed? the repo is well configured in source install type
16:42:37 <Jeffrey4l> is not installed*
16:42:44 <harlowja> which repo has it :)
16:42:46 <harlowja> on centos
16:44:00 <Jeffrey4l> python-novaclient? delorean repo? let me check that
16:45:12 <harlowja> ya, let me know
16:45:18 <harlowja> maybe it's something we messed up also
16:45:35 <harlowja> (on centos not rhel)
16:45:41 <Jeffrey4l> python2-novaclient.noarch                                                                    1:9.1.0-0.20170804194758.0a53d19.el7.centos                                                                     delorean
16:45:52 <harlowja> seems like a different name no?
16:45:56 <Jeffrey4l> yes.
16:46:25 <Jeffrey4l> novaclient may chagned name during this cycle. prepare for python3?
16:46:35 <Jeffrey4l> so i guess this maybe a bug in sensu side.
16:46:38 <harlowja> ya, i'll check to see if delorean is active
16:46:47 <harlowja> and do some double-checking
16:46:49 <Jeffrey4l> and sensu should fix the its dependency
16:47:11 <Jeffrey4l> btw, here is another silimar issue happend on kolla_toolbox https://review.openstack.org/561797
16:47:40 <Jeffrey4l> harlowja, delorean is rdo master repo. so it is active.
16:48:19 <Jeffrey4l> re kolla_toolbox, we are using source install ansible for kolla_toolbox no matter what the install_type is
16:48:20 <harlowja> ya, https://github.com/openstack-packages/osops-tools-monitoring/blob/rdo-liberty/osops-tools-monitoring-oschecks.spec may also just need to be updated
16:48:58 <Jeffrey4l> but recently, i found "pip install ansible" breaks the OS on ocata branch for oraclelinux
16:49:09 <Jeffrey4l> check https://review.openstack.org/561573
16:49:28 <Jeffrey4l> pbourke, ^^ you may be insteresting.
16:49:31 <Jeffrey4l> harlowja, cool
16:49:44 <harlowja> it'd be nice if it could just be in a virtualenv for https://github.com/openstack/osops-tools-monitoring
16:49:52 <harlowja> instead of a package (especially in source mode)
16:49:59 <harlowja> but just thought i'd ask if people have seen this :-P
16:50:42 <Jeffrey4l> harlowja, sensu is a ruby project.
16:50:52 <harlowja> ya, i know
16:50:56 <Jeffrey4l> https://github.com/sensu/sensu
16:51:05 <harlowja> more of the checks in https://github.com/openstack/osops-tools-monitoring/tree/master/monitoring-for-openstack are the things getting installed
16:51:22 <harlowja> afaik that's what is in the rpm that is broke(maybe broke?)
16:52:20 <Jeffrey4l> but centos add the client depency when install sensu.
16:53:25 <ktibi> Jeffrey4l, I see you answer about repo for stable branch. but for master do you know the repo used ?
16:53:40 <Jeffrey4l> i still think this should be fixed on centos side.
16:53:59 <Jeffrey4l> ktibi, yep for master we are using passed-ci repo
16:54:10 <ktibi> it's about https://review.openstack.org/#/c/562242/ I add package in passed-ci :/
16:54:31 <Jeffrey4l> ktibi, check https://github.com/openstack/kolla/blob/master/kolla/common/config.py#L41
16:54:57 <harlowja> Jeffrey4l i'll do some checking before saying more :)
16:55:40 <ktibi> Jeffrey4l, package is in https://trunk.rdoproject.org/centos7-pike/current-passed-ci :/
16:55:41 <ktibi> INFO:kolla.image.build.horizon:No package openstack-designate-ui available
16:56:06 <Jeffrey4l> ktibi, so it is not in the finally pike repo?
16:56:45 <Jeffrey4l> ktibi, if so, you may have to wait until the package is rolled into centos-release-openstack-pike repo.
16:56:49 <ktibi> ho ok. I understand now. For backport in stable branch, we use stable repo ><
16:57:01 <Jeffrey4l> yep
16:57:13 <ktibi> ok I'll ask to #rdo ^^
16:57:23 <Jeffrey4l> cool
16:58:01 <Jeffrey4l> ok, we are almost run out of time.
16:58:08 <Jeffrey4l> thanks guys for coming.
16:58:15 <Jeffrey4l> have a nice day&night
16:58:23 <Jeffrey4l> let us end the meeting.
16:58:26 <Jeffrey4l> #endmeeting