16:00:39 <xarses> #startmeeting fuel
16:00:40 <openstack> Meeting started Thu Sep 17 16:00:39 2015 UTC and is due to finish in 60 minutes.  The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:44 <openstack> The meeting name has been set to 'fuel'
16:00:51 <xarses> #chair xarses
16:00:52 <openstack> Current chairs: xarses
16:01:06 <xarses> Todays Agenda:
16:01:06 <xarses> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:01:10 <kozhukalov_> hi
16:01:12 <mwhahaha> hi
16:01:12 <maximov> hi
16:01:14 <tatyana> hi
16:01:16 <ashtokolov> hi
16:01:20 <IvanKliuk> hi
16:01:52 <angdraug> \o
16:02:02 <akislitsky> рш
16:02:04 <aglarendil> hi
16:02:07 <akislitsky> hi
16:02:25 <mihgen> hi
16:02:26 <agordeev> hi
16:02:51 <xarses> #topic Fuel at OpenStack Summit (Xarses)
16:03:14 <vkramskikh> hi
16:03:19 <xarses> I wanted to get a sense of who's going, If we should have any sessions / work groups
16:03:39 <warpc> hi
16:03:54 <xarses> topics everyone is interested in and the like
16:04:48 <mihgen> I'd say proper, scalable multi-rack support would be topic #1
16:05:25 <mihgen> and HA track
16:05:25 <angdraug> my personal favorite is setting up gating CI on openstack-infra
16:05:52 <mihgen> if there are other HA groups, we would need to consider collaboration with those if possible
16:06:21 <mihgen> if there are no large efforts - we should lead one
16:06:49 <xarses> ok, So I started an etherpad just now
16:06:53 <xarses> #link https://etherpad.openstack.org/p/fuel-mitaka
16:07:31 <aglarendil> well, I guess, there is not so many people going to the summit. so should those who go be proxies of others? I think it would be better to raise this dicussion in ML and separate etherpad. It might take more than an hour
16:08:01 <kozhukalov_> what about plugin howto workshop?
16:08:18 <angdraug> kozhukalov_: +1
16:08:25 <mihgen> aglarendil: +1
16:08:32 <xarses> ya, I will post this to the ML too
16:08:40 <xarses> In a minute will move along
16:08:48 <mihgen> there are just a few folks, yes. We need to be proxies
16:09:19 <mihgen> openstack-puppet etherpad:
16:09:20 <mihgen> #link https://etherpad.openstack.org/p/HND-puppet
16:09:49 <mihgen> we should participate there as well. aglarendil, other - if you have anything for those guys, let's use xarses as proxy
16:10:35 <aglarendil> so are we discussing which sessions/groups we should participate in or about our own Fuel design session at the summit ?
16:11:27 <xarses> both?
16:11:31 <mihgen> sorry I hajacked, it was about Fuel
16:12:17 <mihgen> but yeah - we'd need to think of both, figure out what sessions it's imprortant to go to and ensure that we go
16:12:50 <xarses> Ok, lets move along then. I'll follow up on the ML
16:13:06 <xarses> #topic Elections (angdraug)
16:13:06 <aglarendil> let's start this in ML - I think we have  have plenty of ideas - but we have not yet sorted most of them
16:13:16 <angdraug> we need an independent election official,
16:13:21 <angdraug> I've asked SergeyLukjanov to run the elections for us
16:13:26 <angdraug> he recommended to separate PTL and component lead elections so that people can't end up with both roles
16:13:50 <angdraug> now we wait for SergeyLukjanov to post a call for self-nominations to openstack-dev
16:14:17 <kozhukalov_> who is going to self nominate?
16:14:35 <kozhukalov_> xarses, angdraug ?
16:14:41 <aglarendil> mihgen: angdraug: I have question regarding component lead election procedure. There is no such entity in OpenStack. Is there a clear description of the procedure of CL elections?
16:14:43 <mihgen> so people can self-nomintate like other in openstack-dev do for PTL?
16:14:53 <angdraug> I'll self-nominate for PTL
16:15:08 <angdraug> https://wiki.openstack.org/wiki/Election_Officiating_Guidelines
16:15:11 <angdraug> aglarendil: ^
16:15:23 <mihgen> aglarendil: there is no one for CL, but we will use same procedure
16:15:32 <mihgen> unless there are any objections
16:15:33 <angdraug> same process as PTL
16:15:43 <angdraug> the only difference is going to be the list of people who can vote
16:15:50 <aglarendil> well, there are lots of procedurial questions
16:16:04 <aglarendil> and obviously who can nominate
16:17:09 <angdraug> aglarendil: it's all very straight-forward, see that link above
16:18:08 <angdraug> there's no restriction on who can self-nominate
16:18:21 <angdraug> voters are committers from project's repositories
16:18:24 <mattymo_> other OpenStack projects allow self-nominations
16:18:55 <kozhukalov_> the only restriction is that fuel members have to know who that guy is
16:19:02 <angdraug> the recommendation is not to nominate others (only self-nominate) because otherwise it's a headache to confirm that nominee actually wants the job
16:19:59 <aglarendil> is not there a restriction that eligible candidate should also be an eligible voter?
16:20:30 <xarses> I think it would be ok to assume that, but who's going to vote for a random?
16:20:44 <angdraug> xarses: +1
16:21:17 <angdraug> I think having enough people willing to self-nominate is going to be a bigger problem
16:21:33 <angdraug> do we have more volunteers for PTL? for fuel-library CL? for fuel-python CL?
16:22:06 <kozhukalov_> yes, that could be a problem
16:22:19 <aglarendil> let's see election announcement first
16:22:38 <mattymo_> we could see surprises
16:22:43 <alex_didenko> I think we may have more volunteers after we send an email about it
16:23:12 <angdraug> well, track down SergeyLukjanov and get him to send it :)
16:23:20 <alex_didenko> we have some core reviewers on vacation right now or not attending the meeting
16:23:42 <angdraug> yeah, I proposed that we have a 2 week self-nomination period instead of just 1 week to account for that
16:24:53 <kozhukalov_> probably those who do not attend the meeting regularly would not like to be PTL or CL which assumes a lot of community related work
16:25:09 <aglarendil> angdraug: he is 1000kms away from me - won't it be easier for you to fly along the meridian through the North Pole?
16:26:02 <angdraug> at least he's in the same time zone )
16:26:13 <angdraug> moving on?
16:26:21 <xarses> yep
16:26:37 <xarses> #topic https://bugs.launchpad.net/fuel/7.0.x/+bug/1494684 is it critical or not. There is workaround (mkdir). Full solution will be available soon.  (kozhukalov)
16:26:40 <openstack> Launchpad bug 7 in Launchpad itself "Need help for novice translators" [High,Fix released] - Assigned to Данило Шеган (danilo)
16:26:51 <kozhukalov_> This particular bug does not seem to be critical
16:27:01 <xarses> I'm guessing this is a release blocking bug maybe?
16:27:02 <aglarendil> I think we merge the work around and deliver the full solution in updates
16:27:32 <kozhukalov_> there is workaround for it https://review.openstack.org/#/c/224581/
16:27:43 <maximov> aglarendil we don't merge workarounds for non critical bugs after HCF
16:27:44 <aglarendil> workaround is very trivial and we can get to the release then
16:27:45 * mihgen has to drop, sorry
16:27:49 <kozhukalov_> aglarendil, +1
16:27:58 <mwhahaha> would it ever accidentaly fill the disk?
16:28:09 <aglarendil> maximov: after we merge the workaround we can downgrade it to high
16:28:12 <aglarendil> not before
16:28:24 <aglarendil> mwhahaha: it may
16:28:26 <kozhukalov_> sorry, but the whole solution requires one more day to make it ready to merge
16:28:28 <maximov> kozhukalov_ said it is not critical
16:28:43 <kozhukalov_> and then we would need to test it thoroughly
16:28:48 <maximov> let's decided if it is critical or not
16:28:49 <maximov> first
16:28:51 <aglarendil> maximov: I insist it is critical without a workaround
16:28:57 <aglarendil> it can fill the disk
16:28:58 <maximov> kozhukalov_ ?
16:29:05 <aglarendil> these files will never be rotated or compressed
16:29:10 <aglarendil> and will never be visible to the user
16:29:15 <aglarendil> until it is too late
16:29:30 <aglarendil> so, one-line change is worth of saving us the trouble
16:29:34 <xarses> ok, so i sounds critical then
16:29:38 <kozhukalov_> does upstart write so many logs?
16:29:41 <mwhahaha> yes
16:29:49 <aglarendil> each time you restart a service with upstart
16:29:52 <kozhukalov_> that it could fill the whole disk?
16:29:54 <mwhahaha> openstack services write to them too
16:29:55 <mwhahaha> i think
16:30:16 <mwhahaha> that was the whole point of getting the ability to disable stderr configurations into the puppet scripts
16:30:21 <maximov> 100 bytes in log after every restart ?
16:30:26 <mwhahaha> i forget who was working on that
16:31:02 <aglarendil> maximov: there are not only openstack services
16:31:54 <kozhukalov_> is there anything that prevents us from merging this trivial patch except the priority status of the bug
16:32:00 <kozhukalov_> ?
16:32:25 <maximov> we need clear explanation re user impact
16:32:29 <maximov> in the bug
16:32:41 <kozhukalov_> if there is no such things, then let's raise the priority to critical
16:33:00 <aglarendil> user impact is obvious - on the long run you will get your root partition filled without ability to recover it easily which may lead to cloud misbehaviour
16:33:01 <maximov> we need to provide clear explanation why you raised the priority to critical
16:33:28 <maximov> what is the probability of this scenario ?
16:33:39 <angdraug> since mihgen had to drop off, I guess it falls to me to remind everyone that merging anything at the last moment, even most trivial fixes, is a risk
16:33:40 <kozhukalov_> maximov, aglarendil have just given the explanation
16:33:53 <angdraug> can it wait until 7.0-mu1?
16:33:57 <aglarendil> nope
16:34:05 <aglarendil> it cannot be easily applied to running nodes
16:34:30 <aglarendil> well, it can be - but it is embarassing
16:34:46 <kozhukalov_> aglarendil, -)
16:34:57 <maximov> do we have any ideas how fast upstart logs are growing ?
16:34:57 <xarses> should we vote then?
16:35:27 <xarses> missing logs is data loss if you ask me, we will end up with support calls over it
16:35:49 <kozhukalov_> on my desktop it is just 400K now
16:36:00 <kozhukalov_> i mean du -sh /var/log/upstart
16:36:27 <maximov> maybe you have logrotate running for these files
16:36:35 <maximov> 400K for what period
16:36:40 <kozhukalov_> maximov, sure
16:36:41 <maximov> year, week
16:37:11 <angdraug> xarses: missing log is not data loss, but it is a support nightmare, yes
16:37:12 <kozhukalov_> it does not matter, desktop env have nothing in common with servers
16:37:42 <mwhahaha> the question would be for a controller what is the size of the upstart logs
16:37:47 <angdraug> does upstart write to the log only when it starts/stops the server?
16:37:48 <mwhahaha> but i can't tell you because i can't see them
16:38:05 <angdraug> server->service
16:38:19 <mwhahaha> no openstack services may be writing too them
16:38:36 <maximov> so guys, let's maybe understand first, how serious the problem is
16:38:49 <mwhahaha> this was the puppet thing i was refering to, https://bugs.launchpad.net/puppet-openstack/+bug/1482564
16:38:50 <openstack> Launchpad bug 1482564 in puppet-openstack "Puppet modules should have an ability to set use_stderr parameter" [Undecided,Fix committed] - Assigned to Sergey Kolekonov (skolekonov)
16:38:58 <kozhukalov_> angdraug, likely not only when starts/stops services, but when any events come up
16:39:11 <angdraug> so it can be a lot of logs
16:39:29 <angdraug> and without logroate it can run the log partition out of space
16:39:37 <kozhukalov_> angdraug, potentially yes, if for example a serivce flaps
16:39:39 <mwhahaha> the root partition may run out of space
16:39:44 <mwhahaha> not the log partition
16:39:51 <angdraug> right. even worse
16:40:12 <skolekonov> Hi guys. Yes, I was working on this bug in fuel and in upstream. Services will write all logs with ERR level
16:40:19 <mattymo_> root full leads to inability to create devices needed for launching vms, doing network ops, etc.. that's the reason we have a logs partition now
16:40:20 <angdraug> possibility to run / out of space is a critical bug by itself
16:40:34 <kozhukalov_> probably we have there supervisor or something which tries to start service when it falls
16:40:41 <mattymo_> if we can't contain our logs, we're just exposing ourselves to unnecessary risk
16:41:17 <angdraug> ok, I see strong consensus in favor of merging kozhukalov_'s workaround
16:41:20 <mattymo_> we're breaking upstart because that dir /var/log/upstart in the logs partition is missing, causing upstart to not be able to flip its logs over correctly
16:41:22 <angdraug> what are the risks?
16:41:33 <maximov> ok, kozhukalov_ did you build custom iso w/ your fix and run some system tests ?
16:42:07 <xarses> Ok, do we have enough info here?
16:42:17 <kozhukalov_> maximov, not yet
16:42:48 <angdraug> kozhukalov_: how did you test it?
16:43:43 <kozhukalov_> angdraug, have not tested yet, because i was working on the whole correct solution last night
16:43:58 <xarses> Ok, lets move this to the ML and get to the last topic
16:44:09 <kozhukalov_> i did this small patch just couple hours ago
16:44:33 <maximov> ok, RC3 build already started
16:45:26 <angdraug> lets move on then
16:45:30 <xarses> #topic Package updates on fuel-upgade (mattymo/ikalnitsky)
16:45:49 <mattymo_> So, ikalnitsky is not here to discuss this
16:46:03 <mattymo_> fuel-upgrade (from 6.1 to 7.0) doesn't do a full yum update
16:46:06 <maximov> please provide your opinion mattymo_
16:46:23 <mattymo_> leaving old packages (kernel, python libs, other system packages) unupdated
16:46:35 <mattymo_> it doesn't break anything at the moment, but it really should be done. automatically if possible, I would argue
16:46:37 <maximov> how it worked previously from 6.0 to 6.1 ?
16:46:55 <mattymo_> we didn't update any either
16:47:13 <mattymo_> we were 6.5 -> 6.5 centos in that release. for 6.1 to 7.0 it's centos 6.5 to 6.6
16:47:14 <angdraug> slightly too late in the release cycle to bring this up :(
16:47:19 <mattymo_> http://paste.openstack.org/show/VDVyrUkVgCUEuSKPaG6X/ this is the ~200 packages of un-updated packages
16:47:21 <maximov> why we escalating this problem to critical now then ?
16:47:34 <mattymo_> because the # is quite large
16:47:48 <mattymo_> and now we have a justifiable reason for including this change
16:48:09 <angdraug> is there anything preventing operator from running yum update manually on the fuel node?
16:48:12 <mattymo_> no
16:48:15 <mattymo_> never ever was
16:48:27 <angdraug> so there's an obvious workaround, ergo not a critical
16:48:41 <maximov> +1 angdraug
16:48:51 <maximov> the bug is high if there is a workaround
16:48:57 <maximov> we just need to document it
16:49:04 <mattymo_> ok release notes item it is
16:49:11 <angdraug> yup
16:49:14 <maximov> mattymo_ +1 thanks
16:49:27 <mattymo_> in fuel 7.0 we don't update system packages because we forgot to do it before SCF. run yum update
16:49:55 <maximov> mattymo_ well, we need to be better next time
16:50:10 <mattymo_> maximov, there is an objection from Igor Kalnitsky saying that it _might_ fail
16:50:11 <mattymo_> that's it
16:50:25 <mattymo_> I pushed previously on this issue and was met with the same answer
16:50:30 <mattymo_> in 5.1 in 6.0 and in 6.1
16:51:00 <maximov> mattymo_ did you always push after HCF ? just in a couple of days before release?
16:51:03 <angdraug> mattymo_: propose a fix for 8.0 and lets merge it early so that we can see how likely it is to fail
16:51:15 <mattymo_> angdraug, we're upgrading to centos 7.0 for 8.0, so now it's moot
16:51:22 <mattymo_> python 2.7 and the rest
16:51:52 <angdraug> no its not moot
16:52:02 <angdraug> even centos7 gets package updates occasionally :)
16:52:29 <mattymo_> let's move along
16:52:43 <xarses> ok any thing else we need to cover?
16:52:50 <xarses> #topic open discuss
16:52:57 <xarses> otherwise we can close early
16:54:05 <xarses> seems like no on has anything else to raise, thanks guys
16:54:20 <xarses> #endmeeting