16:00:59 <kozhukalov> #startmeeting Fuel
16:01:01 <openstack> Meeting started Thu Apr 23 16:00:59 2015 UTC and is due to finish in 60 minutes.  The chair is kozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:05 <openstack> The meeting name has been set to 'fuel'
16:01:06 <kozhukalov> #chair kozhukalov
16:01:07 <openstack> Current chairs: kozhukalov
16:01:13 <kozhukalov> hey guys
16:01:16 <docaedo> hello
16:01:20 <mattymo> hi!
16:01:21 <kozhukalov> agenda as usual
16:01:30 <kozhukalov> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:01:43 <kozhukalov> who else is here?
16:01:47 <mwhahaha> hi
16:02:01 <sambork> hi
16:02:01 <mihgen> hi
16:02:08 <xarses> o/
16:02:26 <kozhukalov> great, 7 people
16:02:33 <kozhukalov> let's start
16:02:53 <salmon_> hi
16:02:55 <kozhukalov> #topic Fuel CI migrated to new Jenkins instance ci.fuel-infra.org
16:03:17 <kozhukalov> who added this topic?
16:03:38 <mihgen> I think devops? bookwar is not here
16:03:39 <angdraug> I think it was bookwar
16:04:02 <mihgen> this is excellent actually, I'm very glad that this happened finally
16:04:14 <bookwar> hi all
16:04:21 <kozhukalov> bookwar: hi
16:04:22 <Tatyanka_Leontov> hi :)
16:04:23 <mkwiek> hello
16:04:23 <mihgen> I'd add just redirect from old fuel-jenkins.
16:04:30 <angdraug> bookwar: we're talking about ci.fuel-infra.org
16:04:36 <alex_bh> hello everyone
16:04:43 <alex_didenko> +1 for redirect from old CI
16:05:01 <mihgen> oh wait a sec, "This server is switched to read-only mode to provide logs for older builds. It will be switched off in July 2015"
16:05:03 <bookwar> we don't want to do redirect because we need to keep the history of builds and logs, so all links from gerrit reviews should work
16:05:16 <mihgen> do we need logs folks?
16:05:34 <alex_didenko> ok, what about link to new jobs in old jobs description?
16:05:51 <alex_didenko> like "New job is here"
16:06:03 <bookwar> openstack infra says that CI should store logs for about three months, we can come up with smaller limit, but still we need direct links to job results to work for some time
16:06:17 <mihgen> alex_didenko: why would we need it?
16:06:26 <bookwar> in gerrit reviews you have direct link to job with build number
16:06:44 <alex_didenko> last week I gave links to our CI jobs in slides, so now they are not accurate
16:06:44 <mihgen> ok let's keep it as bookwar suggests
16:07:12 <mihgen> alex_didenko: people should be able to find it still…
16:07:40 <mihgen> I just don't think we should over-engineer and spend 2h of bookwar going over old builds and providing links to new builds
16:07:48 <bookwar> if we have old review, for example https://review.openstack.org/#/c/175388/ links to test results should be available
16:07:50 <angdraug> can we do a banner at the top of every jenkins page with a link to ci.fuel-infra.org?
16:08:17 <angdraug> where can == much less than 2 hours of work :)
16:08:19 <bookwar> i can replace the topbar (the blue one) with a huge banner
16:08:36 <angdraug> alex_didenko: would that solve your problem?
16:08:49 <alex_didenko> yep
16:08:55 <bookwar> i almost did that, if you not scared by my crazy design skills i can do it this evening :)
16:09:42 <bookwar> ok, let's do it then
16:09:56 <docaedo> I like crazy design skills
16:10:00 <kozhukalov> #action bookwar adds banner at the top of every jenkins page with a link to ci.fuel-infra.org
16:10:20 <kozhukalov> ok, looks like we are done on this topic
16:10:23 <kozhukalov> moving on
16:10:37 <kozhukalov> #topic Failed community builds on ci.fuel-infra.org (mihgen)
16:10:52 <mihgen> I just noticied red builds over there.
16:11:11 <mihgen> may be the reason is in configuration after moving to new ci
16:11:37 <mihgen> but I saw many fails before, on old instance too. The question is why those are failing more often, than on jenkins-product ci
16:11:46 <bookwar> the reason is the recent merge
16:12:08 <bookwar> which switched iso builds to use internal mirrors by default
16:12:27 <mihgen> whoo what internal mirrors?
16:12:38 <mihgen> why do we have those?
16:12:39 <kozhukalov> bookwar: how are we going to solve this?
16:12:41 <bookwar> centos stuff
16:12:52 <mihgen> why those are not public?
16:12:56 <bookwar> i've just got aglarendil's confirmation here that fix for that is already merged
16:13:09 <mihgen> ok sounds better
16:13:10 <kozhukalov> bookwar: great
16:13:27 <bookwar> mihgen: these are centos upstream mirror which were used internally in our infra
16:13:38 <bookwar> just a snapshot of upstream
16:14:03 <mihgen> we need to keep looking at those community builds - ideally those should be just of same color as those on jenkins-product
16:14:03 <bookwar> so the fix has landed and we will check if it is ok
16:14:22 <kozhukalov> btw, i am working on introducing priorities for centos mirrors https://review.openstack.org/#/c/176438/
16:14:28 <kozhukalov> still in progress
16:14:37 <bookwar> mihgen: they were the same, internal staging builds were broken too :)
16:15:04 <mihgen> ohh
16:15:09 <mihgen> ok kozhukalov let's move on
16:15:26 <kozhukalov> #topic Voting fuel library unit tests https://review.fuel-infra.org/#/c/6150
16:15:47 <bookwar> that's just a small notification, that we are going to switch the job to voting
16:15:53 <bookwar> if there are no objections
16:16:17 <mihgen> rspec tests?
16:16:19 <alex_didenko> 3.. 2
16:16:22 <bookwar> #link https://review.fuel-infra.org/#/c/6150
16:16:34 <alex_didenko> yep, per-module puppet-rspec tests
16:16:53 <mihgen> let's do it finally )
16:16:56 <alex_didenko> we're running them in non-voting mode for a very long time
16:17:12 <bookwar> CR+1 are welcome
16:17:22 <kozhukalov> bookwar: thanx for this info
16:17:29 <kozhukalov> moving on ?
16:17:32 <bookwar> yep
16:17:44 <kozhukalov> #topic New proposals (alex_bh)
16:17:48 <alex_bh> thanks. I will be very fast
16:17:55 <alex_bh> In Create-Net (http://www.create-net.org) we are working on Fuel from the 3.2.1 version and we developed some new modules. Now, we would like to contribute to Fuel with new  features.
16:18:02 <alex_bh> I wrote 2 draft blueprints (we have also other ideas ;) ). The first one is on the installation of Calamari UI I order to provide a monitoring and management UI for Ceph.
16:18:11 <alex_bh> The second proposal is about the multiple regions deployment.
16:18:17 <alex_bh> I would like to submit to you these two ideas and start a discussion, in order to address our development.
16:18:54 <mihgen> alex_bh: that's interesting actually
16:18:56 <kozhukalov> alex_bh: could you please so kind to prepare specs in rst format? https://github.com/stackforge/fuel-specs
16:19:00 <angdraug> I like the calamari proposal
16:19:12 <angdraug> on both accounts, +1 to what kozhukalov just said :)
16:19:21 <mihgen> are you guys extensively using ceph deployment via Fuel?
16:19:42 <angdraug> #link https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Propose_enhancements
16:19:57 <alex_bh> well, now we are deploying ceph in a few production env
16:20:33 <alex_bh> and yes, I have deployed by  Fuel
16:21:02 <angdraug> alex_bh: please keep xarses in the loop on your progress, he will help you get fuel developers attention for reviews
16:21:03 <kozhukalov> alex_bh: great to hear that
16:21:36 <mihgen> let's take a look at proposals, and see if work can be done as plugins to Fuel
16:21:40 <xarses> alex_bh: sounds great. For both we would likely want to weigh the implementation directly in fuel, vs creating a fuel plugi-n. Both sound good in the core and I'd almost think that multi-region would have to have some parts in the core
16:22:12 <alex_bh> about Calamari the idea is to write a plugin
16:22:20 <kozhukalov> alex_bh: yes, and depending on your time zone please ping me or xarses if you have questions about spec preparations or any other questions (for me email is preferable)
16:22:37 <alex_bh> ok...thanks a lot for your cooperation
16:22:52 <xarses> alex_bh: please feel free to poke me if you need anything
16:23:19 <alex_bh> will update the doc according to your suggestions
16:23:25 <alex_bh> thanks a lot
16:23:28 <mihgen> good. should we move on?
16:23:36 <kozhukalov> alex_bh: great, thank you, moving on
16:23:42 <alex_bh> thanks
16:23:51 <kozhukalov> #topic recent provisioning improvements
16:24:06 <kozhukalov> there are some sub-topics here
16:24:22 <kozhukalov> they are written down in agenda
16:24:32 <mihgen> yeah thanks for summarizing those
16:24:54 <mihgen> question is how far we from being not that busy with bugs here )
16:25:19 <kozhukalov> mihgen: pretty close )
16:25:19 <mihgen> so you could start looking at volume-manager refactoring, at least from the design perspective
16:25:39 <mihgen> did you hear any issues from scale lab?
16:25:59 <mihgen> were folks able to do 200 nodes after rmoe's fix?
16:26:05 <kozhukalov> mihgen: actually i did look in december 2014 and i have some ideas about that
16:27:04 <kozhukalov> no, i didn't, last time when i heard about issues was that but which was fixed by rmoe
16:27:11 <mihgen> kozhukalov: we'd need to do cross validation of use cases with some of other folks who are close to users
16:27:21 <kozhukalov> https://review.openstack.org/#/c/174662/ this one
16:27:43 <kozhukalov> mihgen: yes, i see
16:27:57 <kozhukalov> will try to summarize feedback
16:28:12 <kozhukalov> from guys in the field
16:28:46 <mihgen> ok cool, thanks
16:28:58 <kozhukalov> another point here is that we can workaround some volume manager bugs in 6.1
16:29:19 <kozhukalov> in progress here
16:29:28 <kozhukalov> any other questions?
16:29:50 <kozhukalov> in not, let's move on
16:30:13 <kozhukalov> #topic Fuel design sessions during the Vancouver summit https://etherpad.openstack.org/p/fuel-7.0-roadmap (zigo)
16:30:56 <kozhukalov> #link https://etherpad.openstack.org/p/fuel-7.0-roadmap
16:31:02 <kozhukalov> zigo: around?
16:31:57 <kozhukalov> looks like he is not here
16:32:07 <angdraug> zigo has created the linked etherpad to kick off the discussion of what we can cover in an open design session (or two) at the Vancouver summit
16:32:38 <mihgen> we won
16:32:45 <angdraug> his priority is supporting Fuel on Debian, as you can see from the ideas he's thrown in
16:32:50 <xarses> ya, so we want to have public sessions in Vancouver
16:32:59 <mihgen> we won't have many ppl going there, but still
16:33:11 <angdraug> but it doesn't mean that we can't add more things to the list
16:33:12 <mihgen> aglarendil, mattymo, bookwar should be there
16:33:13 <xarses> and we'd like to see some of you propose topics to discuss, and we will also pass it around on the ML
16:33:26 <angdraug> for example bookwar will be there, so worth including ci related topics
16:34:11 <mihgen> from the CI stuff, I'd love to see more reusage of what upstream infra does, and contributing our improvements back there
16:34:14 <bookwar> i agree with mihgen, we can and will discuss things but for full-scale design session there are not enough people there
16:35:08 <angdraug> there only need to be enough fuel people there to host it, and enough interesting topics for community people to show up with their ideas
16:35:28 <angdraug> you won't walk out of there with semi-ready blueprint specs
16:35:43 <bookwar> so it would be more like brainstorming session
16:35:53 <angdraug> yes, brainstorming and feedback gathering
16:36:04 <mattymo> basically you can brainstorm and you can propose ideas and see if it is generally acceptable
16:36:31 <mattymo> same as what angdraug said, basically:)
16:36:34 <mihgen> ok cool, moving on?)
16:36:44 <mihgen> I'm inpatient today )
16:36:57 <angdraug> yup
16:37:02 <kozhukalov> yep, moving on
16:37:09 <bookwar> mattymo:  we don't have the 'generally' part, minoringly acceptable :)
16:37:16 <kozhukalov> #topic Help to fuel-library team in squashing high/criticals, even assigned bugs (mihgen)
16:37:52 <mihgen> so you folks might be aware, majority of most complicated / time consuming bugs we have in fuel-library
16:38:06 <mihgen> basically everything related to puppet modules and linux configuration
16:38:17 <kozhukalov> looks like python team gonna be in time with bugs by HCF, but library needs help
16:39:01 <mihgen> yes. So, if you guys / your team members who are not at the meeting now can go over those bugs and help in any form, that would be awesome
16:39:13 <kozhukalov> mihgen: should we contact aglarendil and ask where and how we can help?
16:39:16 <mihgen> of course if you are not over subscribed with your own area
16:39:22 <mattymo> I can comment on that. I'm just getting lots of -1 with "fix this", followed with another -1 "fix that".. slowing down bug squashing velocity
16:39:45 <mattymo> if you have multiple concerns with a patch, please just put it all out there, rather than clogging CI workflow with drip-coffee-action negative feedback
16:39:56 <angdraug> +1
16:39:57 <mihgen> yep, aglarendil should be able to help identify those bugs
16:40:14 <kozhukalov> mattymo: +1
16:40:26 <angdraug> I've also seen this pattern: please go through whole commit when reviewing, don't stop after you've found one problem
16:40:35 <mattymo> I'm not going to name names, but I had a wonderful case where 1 person managed to -1 my patch 9 times
16:40:44 <kozhukalov> please be as constructive as possible while reviewing
16:40:54 <mihgen> mattymo: +1. Folks just get into communication with those who are keep doing this, and try to fix it
16:41:14 <xarses> also, if it's a minor/cosmetic issue, lets consider creating a bug to follow it up and allow it to stand
16:41:15 <mihgen> if it doesn't fly - let me know and I'll do something )
16:41:31 <mihgen> xarses: +1
16:41:42 <mattymo> xarses, +1 especially for gartantuan multi-project commits that take 2-3 hrs to test each iteration
16:41:50 <mihgen> critical things should be fixed fast. remaining polishing can be considered as separate bugs
16:42:23 <kozhukalov> mihgen: sounds frightening
16:42:57 <angdraug> no, just common sense
16:43:05 <mihgen> no worries, you won't feel pain =)
16:43:49 <mihgen> closing meeting?
16:43:52 <kozhukalov> mihgen: yes, it is gonna be rather pleasant (couldn't resist to say this)
16:43:59 <alex_didenko> 1 more thing
16:44:08 <mihgen> I'm not that way )
16:44:15 <alex_didenko> just wanted to mention that we've started running noop tests (puppet-rspec) for puppet tasks in non-voting mode https://ci.fuel-infra.org/job/fuellib_noop_tests/
16:44:46 <mihgen> alex_didenko: great! how fast those pass?
16:44:58 <alex_didenko> Took 3 min 10 sec
16:45:02 <kozhukalov> alex_didenko: thanx for this info
16:45:11 <bookwar> #link https://ci.fuel-infra.org/job/fuellib_noop_tests/buildTimeTrend
16:45:33 <mihgen> that's excellent. Will we be running remaining long -running tests only after this "smoke" passes?
16:45:51 <mihgen> noop -> rspec -> heavy system
16:46:06 <mihgen> I think we run in parallel now, don't we?
16:46:07 <alex_didenko> eventually yes
16:46:12 <alex_didenko> I hope :)
16:46:16 <kozhukalov> mihgen: +1 sounds like "must do"
16:46:40 <alex_didenko> we had a meeting with devops on it some time ago
16:46:50 <mihgen> yeah let's try to make it sooner, that's how we can free up some hardware and people's time :)
16:46:54 <alex_didenko> so it should be doable
16:47:01 <bookwar> it is not simple from jenkins configuration side, but it is doable
16:47:26 <bookwar> so we are not getting it today or next week, but eventually, yes :)
16:47:48 <kozhukalov> bookwar: cool
16:48:04 <kozhukalov> folks, any other questions?
16:48:09 <kozhukalov> suggestions?
16:48:15 <kozhukalov> announcements?
16:48:19 <mihgen> bookwar: I thought it's just child job.. ok I hope you'll find easy and best solution
16:48:42 <kozhukalov> looks like we are done
16:48:58 <kozhukalov> great meeting, thanks everyone for attending
16:49:02 <kozhukalov> closing
16:49:04 <mihgen> yep, thanks, let's move on
16:49:07 <mihgen> =)
16:49:22 <kozhukalov> #endmeeting