16:00:13 #startmeeting kolla 16:00:14 Meeting started Wed Apr 18 16:00:13 2018 UTC and is due to finish in 60 minutes. The chair is Jeffrey4l. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:18 The meeting name has been set to 'kolla' 16:00:18 #topic rollcall 16:00:27 \0 16:00:40 \o 16:00:41 o/ 16:00:45 o/ 16:00:52 o/ 16:00:59 o/ 16:01:49 o/ 16:01:52 \o 16:02:21 #topic Announcements 16:02:32 o/ 16:03:00 after some discuss, we still think kolla-k8s should be retired. 16:03:25 i started a vote on ML, feel free to reply it. 16:03:27 #link http://lists.openstack.org/pipermail/openstack-dev/2018-April/129452.html 16:03:35 o/ 16:03:43 any others from community? 16:04:52 ok move on 16:04:58 #topic ceph pg limits in luminous (pbourke) 16:05:05 pbourke, yours floor 16:05:13 thanks, one moment just trying to find the bug 16:05:56 o/ 16:05:58 sorry for late 16:06:01 #link https://bugs.launchpad.net/bugs/1763356 16:06:02 Launchpad bug 1763356 in kolla-ansible "ceph deployment fails with not enough pages" [Critical,Confirmed] 16:06:04 pbourke, ^^ 16:06:23 thanks Jeffrey4l 16:06:26 that's the one 16:06:29 in summary 16:06:46 our images are currently using ceph 12.2.1 16:07:02 in 12.2.2 they seem to have changed the limit on how many pgs you can have per osd 16:07:20 im not sure why our images haven't picked up this version, but as soon as we do, things will break 16:07:48 vbel has posted a solution on the bug, I just wanted to bring this up and see if we're happy with this solution 16:08:30 essentially, we should set the pg size for the pools we create to a very small value 16:08:48 and make it very clear via documentation or a warning message that operators need to revise these before deploying to production 16:09:12 kolla cannot accurately calculate these up front as it depends on the size and shape of each individual cluster 16:09:23 the issue is kolla don't know what's the ceph cluster size before deploying. 16:09:26 right 16:09:57 I think this makes sense, but Im not a ceph expert :) 16:10:15 use a smaller number for "ceph_pool_pg_num" works. 16:10:41 ok, we'll post a patch and people can review. Far as I can tell it wont be an issue for upgrades but will check this also 16:12:21 no matter how the number is. the operator still could hit this issue. 16:12:43 for example there is only one ceph-osd process 16:13:06 hrm... ignore ^^ 16:13:10 perhaps we could perform some calcuations up front? 16:13:32 *calculations 16:13:55 pbourke, there are some recommendation from https://ceph.com/pgcalc/ 16:14:26 yeah 16:14:52 I think in the future this problem may go away anyhow, I read something about ceph looking to make pg configuration more transparent 16:15:16 cool. interesting 16:15:56 #link https://ceph.com/community/new-luminous-pg-overdose-protection/ 16:16:30 too long. will read it later. 16:16:32 so ok I think that's it for now 16:16:53 pbourke thanks pbourke:) 16:17:35 for kolla i think we can reduce the ceph pool pg num and add some note in doc at the same time. because this issue can not be ignore so far in kolla. 16:18:56 can we move on? 16:19:07 yes. 16:19:19 #topic Review clean up (pbourke) 16:19:33 still yours pbourke 16:19:33 #link https://review.openstack.org/#/q/project:openstack/kolla-ansible+status:open 16:19:49 so just wanted to highlight that we still have reviews open from mid 2017 16:20:01 I propose that these are tidied up 16:20:24 this looks better overall and makes it easier for reviewers to find whats important 16:20:39 unsure what criteria other projects use, I was going to suggest either 3 or 6 months 16:20:49 what do people think? 16:20:57 reasonable. 16:21:08 +1 16:21:10 pbourke: reasonable to me too 16:21:11 +1, with a warning before abandoning 16:21:16 + 16:21:17 1 16:21:26 i wonder is there any bot could do this in openstack now. 16:21:47 what Ive seen in the past is a core will simply abandon with something like "thanks for looking at this, if you're interested in continuing this work feel free to reopen" 16:22:06 ya i'd recommend ^ 16:22:07 a bot would be useful but I dont think there's too many 16:22:13 +1, I prefer 6mo, we should left a bit more detail like: no activity in xxx mo 16:22:26 so I'm happy to go ahead and do this 16:22:31 i'd rather these have the human touch to it; people may have put in blood/sweat and tears into some of those changes 16:22:31 duonghq: 6 months is fine 16:22:54 is trivial for people to reopen, it just takes them off our radar 16:22:59 *it's 16:23:00 i saw some simliar bot in launchpad. btw we should cleanup bugs in the same way 16:23:11 oh, yes even more so for bugs 16:23:18 Also before that if we try to reach author at once then post the comment about abandon etc 16:24:01 for bugs, i found this script https://github.com/openstack-infra/release-tools/blob/master/expire_old_bug_reports.py 16:24:12 if a bug is confirmed but not fixed, should it still be closed? 16:24:59 mgoddard: hard to say, if old enough if may no longer be valid 16:25:11 I recently attended a bug triage session with the ironic team. That could be another way to tackle it 16:25:32 sounds good 16:25:54 I find it difficult to keep up with the bugs, if we had a formal session every now and then we might do better at keeping on top of them 16:26:25 from the script, it will mark the bug as close if there is no update for 18 month 16:26:27 they also have a bug liaison person assigned each week, who keeps track of new bugs 16:26:45 18 months seems a reasonable amount of time 16:27:23 although I think harlowja's point about the human touch applies here too 16:27:46 ya, just keep in mind that there are people behind those bugs/reviews ... 16:27:51 with feelings and such :-P 16:28:02 mgoddard, +1 https://github.com/openstack-infra/release-tools/blob/master/expire_old_bug_reports.py#L162-L173 16:28:17 Maybe we can just fix bugs of the maintained realease. 16:29:10 mgoddard, triage session sound a good solution too. 16:29:24 chason, what is that meaning? 16:29:52 Jeffrey4l: It allows us to pool knowledge, rather than rely on one person with a limited domain 16:30:40 mgoddard, do you konw how the session is orgranized? 16:31:15 Jeffrey4l: there was a ML post, I'll find it 16:31:23 chason, since we haven't stable-policy tag now. in fact we could backport some bp-like patch check http://lists.openstack.org/pipermail/openstack-dev/2018-April/129265.html 16:31:39 cool. i will check that later. 16:32:43 so for this topic, the conclusion for now is: we will close the bp(6 months)/bugs(18months) with a proper comments. 16:32:49 manully or automatically. 16:33:16 https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg117769.html 16:33:25 also see follow up emails for format 16:33:35 it was a video call via bluejeans 16:34:01 yep 16:34:47 maybe we could scheduler one if there is enough guy who interested this 16:35:26 ok. seems we are done for this topic 16:35:29 let's move 16:35:38 #topic Open Discussion 16:35:46 any volunteer? 16:36:21 I made a demo of kayobe 16:36:35 Fast version: https://asciinema.org/a/176888?speed=2 16:36:45 Real time: https://asciinema.org/a/176883 16:37:05 There will be a blog post to follow shortly 16:37:11 cool 16:37:50 mgoddard nice work! :) 16:38:19 Also, I recently created a proof of concept CI job for bifrost 16:38:26 nice one 16:38:29 https://review.openstack.org/549775 16:38:38 so i have a question 16:38:42 it worked at one point, but is now failing again :) 16:38:53 i've been trying to get the sensu images to build (we use source mode) 16:38:55 is this something the kolla team would be interested in having? 16:39:03 mgoddard: will watch your nice job 16:39:03 but i've noticed http://paste.openstack.org/show/719424/ happens in source mode (on centos) 16:39:24 so i'm thinking that we may want to emit some kind of error for kolla building of sensu in source mode (until thats adddressed) 16:40:01 mgoddard, yep. more jobs is welcome. 16:40:26 cool, I'll fix the issue and add some reviewers 16:40:41 i think it will work in binary mode, though on centos i'm having to force add `centos-release-openstack-pike` 16:40:55 so wondering if anyone else has tried those sensu containers? 16:41:02 harlowja: I'm not very familiar but - we dont build source containers for non openstack components, so the binary/source workflow for sensu should be the same? 16:41:24 pbourke well it can be tried, it just craps out pretty late in the build process 16:41:29 so it almost feels better to die early 16:41:57 sensu is a binary install. and it depends on some oenstack client. 16:42:10 harlowja: ping me the review and I'll take a look 16:42:31 but why the novaclient is installed? the repo is well configured in source install type 16:42:37 is not installed* 16:42:44 which repo has it :) 16:42:46 on centos 16:44:00 python-novaclient? delorean repo? let me check that 16:45:12 ya, let me know 16:45:18 maybe it's something we messed up also 16:45:35 (on centos not rhel) 16:45:41 python2-novaclient.noarch 1:9.1.0-0.20170804194758.0a53d19.el7.centos delorean 16:45:52 seems like a different name no? 16:45:56 yes. 16:46:25 novaclient may chagned name during this cycle. prepare for python3? 16:46:35 so i guess this maybe a bug in sensu side. 16:46:38 ya, i'll check to see if delorean is active 16:46:47 and do some double-checking 16:46:49 and sensu should fix the its dependency 16:47:11 btw, here is another silimar issue happend on kolla_toolbox https://review.openstack.org/561797 16:47:40 harlowja, delorean is rdo master repo. so it is active. 16:48:19 re kolla_toolbox, we are using source install ansible for kolla_toolbox no matter what the install_type is 16:48:20 ya, https://github.com/openstack-packages/osops-tools-monitoring/blob/rdo-liberty/osops-tools-monitoring-oschecks.spec may also just need to be updated 16:48:58 but recently, i found "pip install ansible" breaks the OS on ocata branch for oraclelinux 16:49:09 check https://review.openstack.org/561573 16:49:28 pbourke, ^^ you may be insteresting. 16:49:31 harlowja, cool 16:49:44 it'd be nice if it could just be in a virtualenv for https://github.com/openstack/osops-tools-monitoring 16:49:52 instead of a package (especially in source mode) 16:49:59 but just thought i'd ask if people have seen this :-P 16:50:42 harlowja, sensu is a ruby project. 16:50:52 ya, i know 16:50:56 https://github.com/sensu/sensu 16:51:05 more of the checks in https://github.com/openstack/osops-tools-monitoring/tree/master/monitoring-for-openstack are the things getting installed 16:51:22 afaik that's what is in the rpm that is broke(maybe broke?) 16:52:20 but centos add the client depency when install sensu. 16:53:25 Jeffrey4l, I see you answer about repo for stable branch. but for master do you know the repo used ? 16:53:40 i still think this should be fixed on centos side. 16:53:59 ktibi, yep for master we are using passed-ci repo 16:54:10 it's about https://review.openstack.org/#/c/562242/ I add package in passed-ci :/ 16:54:31 ktibi, check https://github.com/openstack/kolla/blob/master/kolla/common/config.py#L41 16:54:57 Jeffrey4l i'll do some checking before saying more :) 16:55:40 Jeffrey4l, package is in https://trunk.rdoproject.org/centos7-pike/current-passed-ci :/ 16:55:41 INFO:kolla.image.build.horizon:No package openstack-designate-ui available 16:56:06 ktibi, so it is not in the finally pike repo? 16:56:45 ktibi, if so, you may have to wait until the package is rolled into centos-release-openstack-pike repo. 16:56:49 ho ok. I understand now. For backport in stable branch, we use stable repo >< 16:57:01 yep 16:57:13 ok I'll ask to #rdo ^^ 16:57:23 cool 16:58:01 ok, we are almost run out of time. 16:58:08 thanks guys for coming. 16:58:15 have a nice day&night 16:58:23 let us end the meeting. 16:58:26 #endmeeting