16:03:52 <kozhukalov> #startmeeting Fuel
16:03:53 <openstack> Meeting started Thu Apr 16 16:03:52 2015 UTC and is due to finish in 60 minutes.  The chair is kozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:03:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:03:57 <openstack> The meeting name has been set to 'fuel'
16:04:04 <kozhukalov> #chair kozhukalov
16:04:05 <openstack> Current chairs: kozhukalov
16:04:15 <kozhukalov> is anybody here?
16:04:19 <docaedo> hi
16:04:20 <agordeev> hi
16:04:21 <mwhahaha> hi
16:04:24 <daniel3_> hi
16:04:26 <siwos_> hello
16:04:32 <mkwiek> hello
16:04:44 <kozhukalov> great
16:04:53 <prmtl> o/
16:05:01 <mattymo> hey!
16:05:05 <kozhukalov> tiny announcement
16:05:18 <akislitsky> hi
16:05:52 <kozhukalov> yesterday dpyzhov wrote the letter where suggested to cancel our meeting
16:06:14 <kozhukalov> looks like current format is not the best what we can have
16:06:52 <kozhukalov> so, my suggestion is to continue running this meeting
16:07:08 <kozhukalov> but we definitely need to change its format
16:07:35 <mihgen> what format do you propose?
16:07:36 <kozhukalov> and in the end of this meeting i'd hear your ideas guys on this
16:08:00 <kozhukalov> let's start from first topic
16:08:07 <kozhukalov> it is more important
16:08:26 <kozhukalov> #topic Need help on this https://bugs.launchpad.net/fuel/+bug/1444459
16:08:29 <openstack> Launchpad bug 1444459 in Fuel for OpenStack "Boot sector signature not found in case with changed disks configuration for Ceph node" [High,Confirmed] - Assigned to Vladimir Kozhukalov (kozhukalov)
16:08:58 <kozhukalov> short: a node is successfully provisioned but then it is unable to boot
16:09:18 <mihgen> why?
16:09:42 <kozhukalov> what is bad is this bug appeard 3 times today on the scale lab
16:09:51 <kozhukalov> i don't know
16:10:10 <kozhukalov> a month ago we fixed similar bug
16:10:26 <mihgen> but we managed to reproduce it in VMs?
16:10:33 <kozhukalov> the reason was that a node was rebooted using sysrq
16:11:05 <kozhukalov> yes, Nastya managed to reprduce it on her lab
16:11:10 <kozhukalov> on vbox
16:11:34 <kozhukalov> actually any ideas what it could be are welcome
16:11:40 <mihgen> may be we should run sync before sysrq?
16:11:55 <mihgen> why would we run sysrq, and not reboot?
16:12:27 <kozhukalov> i mean, we install image on a node, we check its md5, everything is ok, then we mount it and run grub-install in chroot
16:12:55 <kozhukalov> grub is installed (according to log) but then a node can not boot
16:13:19 <kozhukalov> this time is not sysrq
16:13:30 <kozhukalov> likely not sysrq
16:13:33 <kozhukalov> not sure
16:13:40 <mihgen> do you have that node to take a look if actually there is a grub installed / other stuff saved?
16:13:46 <kozhukalov> it was previous similar bug
16:14:26 <mihgen> there is should not be any magic, if data is correct on disk then it would boot
16:14:34 <mihgen> sounds like data is not correct
16:14:36 <kozhukalov> yes, the problem is this bug is floating
16:15:00 <mihgen> then question is whether data was not completely saved before reboot, or corrupted during reset
16:15:10 <kozhukalov> yes
16:15:12 <mwhahaha> did it run grub on the wrong disk?
16:15:25 <kozhukalov> but what could damage the data?
16:15:45 <kozhukalov> running sync before is not going to help
16:15:58 <agordeev> may it be related to disk caches? we can add few sync() calls in fuel-agent to be sure that we've asked system to sync data.
16:16:00 <kozhukalov> because afaik sync flushes file systems
16:16:20 <mihgen> disk cache?
16:16:21 <kozhukalov> mwhahaha: we install grub on all disks
16:16:38 <docaedo> but an orderly shutdown/reboot does trigger a cache flush AFAIK
16:16:40 <kozhukalov> mihgen: what do you mean? hardware cache?
16:16:48 <mihgen> yes
16:16:55 <kozhukalov> docaedo: yes
16:17:20 <kozhukalov> i believe we need even more detailed loggin
16:17:40 <mihgen> not sure if any logging would help if it's hardware cache
16:17:59 <mihgen> we just need to ensure that we do a trigger to cache flush
16:18:18 <mihgen> sorry, trigger a cache flush I meant
16:18:42 <kozhukalov> mihgen: i mean we probably need kinda debug mode which will read some data from disk and compare them with some patterns
16:18:48 <mihgen> kozhukalov: did you see what was actually corrupted?
16:19:19 <kozhukalov> have not had a chance to go deep into this
16:19:56 <kozhukalov> any ideas what command should i run to trigger disk cache?
16:20:29 <mihgen> can we temporarily at least replace sysrq with shutdown/reboot?
16:21:07 <mihgen> google says: echo 3 | sudo tee /proc/sys/vm/drop_caches
16:21:11 <kozhukalov> ok, i am going to try to reproduce this on my lab, and do some research on this, but any assistance is really welcome
16:21:57 <kozhukalov> mihgen: ok, nice, will read about this
16:22:02 <mihgen> kozhukalov: http://stackoverflow.com/questions/9551838/how-to-purge-disk-i-o-caches-on-linux
16:23:22 <kozhukalov> ok, i think right now we can not end up with something working
16:23:27 <agordeev> not sure if this bug is related to virtual envs only. Have anyone able to reproduce it on real hw?
16:23:56 <kozhukalov> agordeev: scale lab
16:24:02 <kozhukalov> 3 times today
16:25:10 <kozhukalov> they are able to reproduce, but unfortunately it is extremely inconvenient to do research on their lab
16:26:11 <kozhukalov> due to long response delay
16:26:18 <kozhukalov> ok moving on
16:26:40 <kozhukalov> #topic Weekly meeting format: suggestions, ideas, whatever
16:26:42 <angdraug> scale lab is not the right place for research :9
16:27:18 <kozhukalov> like i sad many people are not happy with our current meeting format
16:27:31 <kozhukalov> s/sad/said/
16:27:37 <angdraug> sad, too :)
16:28:11 <kozhukalov> and the idea is to stop forcing people to attend and share their development statuss
16:28:43 <kozhukalov> let's try making our meeting more like hacking forum
16:29:19 <bookwar> I would add couple of weekly items which will be on agenda if there is nothing more, something like "hot topics of the week" and "bug trends of the week"
16:29:23 <kozhukalov> i mean any news, features, other interesting points
16:29:49 <bookwar> This will create the initial impulse for further discussions
16:29:49 <angdraug> I'm afraid this will deteriorate this meeting even further, you'll end up talking to yourself
16:29:55 <angdraug> well maybe to me :p
16:30:03 <kozhukalov> bookwar: ok, and who is going to prepare this?
16:31:06 <bookwar> Script
16:31:35 <kozhukalov> angdraug: what exactly? i mean it impossible to understand which statement you are trying to argue
16:32:00 <angdraug> I mean, if we stop discussing development status
16:33:04 <kozhukalov> yes, but currently we are trying to discuss dev status with couple people
16:33:26 <kozhukalov> no one interested in reporting their status here
16:33:42 <angdraug> maybe we should fix the reason why they're not interested?
16:33:49 <kozhukalov> cause we have plenty of other channels
16:34:10 <kozhukalov> what is the reason in your opinion?
16:34:15 <angdraug> instead of avoiding the problem and trying to invent meaning for what you aptly called a cargo cult
16:34:32 <mihgen> if everyone feature lead would report their status here, then we would need more that 1 hour
16:34:43 <mihgen> and no one will be interested in reading all
16:34:44 <angdraug> we do have plenty of communication channels, and for some reason we're preferring not to use IRC
16:35:18 <mihgen> so it might make sense to keep meeting for smaller group actually
16:35:18 <barthalion> and somehow we use slack…
16:35:36 <mihgen> when everyone is involved in the discussion
16:35:37 <alex_didenko> should we move status updates to ML then?
16:35:44 <kozhukalov> mihgen: what if we focus on those features which are more likely interesting to community
16:36:02 <xarses> alex_didenko: that could be helpful
16:36:22 <mihgen> kozhukalov: for instance. but again, if it's gonna be broad -
16:36:39 <mihgen> it won't fly, as some people will be sitting un-interested in reading all
16:36:40 <angdraug> barthalion: you're right, I think introducing slack has contributed to the shift away from irc
16:37:04 <angdraug> still, we had the same problem with skype before:
16:37:06 <mihgen> slack, irc, skype, hangout, email - of course it's too much
16:37:34 <mihgen> angdraug: I think it's not the main reason though
16:37:41 <angdraug> despite many calls to use open channels for everything that doesn't contain secrets and passwords, people stuck to the closed communications channels
16:37:49 <mihgen> main reason is audience. Now it's like for all about nothing
16:38:03 <barthalion> it's one of the reasons, a lot of things is discussed internally even if it could take place on freenode
16:38:17 <angdraug> audience is a very good way to highlight the problem
16:38:23 <angdraug> we shouldn't using irc "for audience"
16:38:24 <mihgen> so if we, let's say, focus on one certain area - let's say underlay networking
16:38:26 <barthalion> so when we start to actually have an audience, nobody will even care about irc
16:38:40 <mwhahaha> perhaps the irc meeting should be more focused on bugs, community/impacting issues and conversations around feature changes?
16:39:15 <angdraug> my favorite idea was discussion about what's new in openstack community and how it may affect or may be integrated into fuel
16:39:21 <barthalion> also, would be great if we could actually move new bug notifications from #fuel-dev
16:39:24 <mihgen> mwhahaha: again, most of attendees will be silent all the time, as we have so broad expertise
16:39:44 <angdraug> still, I maintain that changing the meeting format is secondary to figuring out its purpose
16:39:59 <kozhukalov> angdraug: +1 for news in Openstack and how it affects us
16:40:00 <barthalion> it generates useless noise that nobody pays attention to
16:40:10 <alex_didenko> barthalion: +1, move all the bug notification to slack, for example. To keep IRC chat clean
16:40:18 <mihgen> barthalion: I would agree actually, I think bug reports introduce too much noise in fuel-dev now.
16:40:26 <mihgen> separate channel in IRC is Ok
16:40:28 <docaedo> One of the reasons IRC meetings started in the first place for open source projects
16:40:31 <docaedo> (as far as I can tell) was to gather together folks who were normally spread all
16:40:34 <docaedo> around in different time zones, to quickly come to consensus on issues. When we need
16:40:37 <docaedo> that now, it happens either on openstack-dev ML, or on internal mailing lists.
16:40:54 <mihgen> docaedo: again to gather around single issue
16:41:12 <mihgen> we ended up with having too broad topics at the same place at the same time
16:42:01 <mihgen> for openstack news, you would need people preparing the news
16:42:01 <docaedo> I think part of the reason why there is low participation on IRC is that nobobdy waits around to discuss stuff here. When there's an issue, we discuss on email and get the attention it needs
16:42:19 <xarses> I still think the goal of the meetings is to discuss issues / blockers / live chat on things that are not carrying well on the ML. Its ok that we don't need the full hour, but It should not limit participation
16:43:42 <kozhukalov> xarses: the problem is that when we have an issue we can not wait until next meeting
16:44:10 <kozhukalov> we usually ping a right person in slack or skype or email
16:44:43 <bookwar> I strongly believe that organizing meeting for some community is wrong. We _are_ the community, and we need to organize meeting for _us_ first, and then if there will be external part - that's OK, but if not - so be it. Talking about phantom community just kills all the real motivation, because you start to consider the meeting to be the showroom and not the real thing.
16:45:09 <mihgen> bookwar: +1
16:45:11 <kozhukalov> mihgen: actually we have weekly news reports
16:45:15 <docaedo> what I'm wondering about is how to determine what should "wait" for IRC meeting. This is great for situations that need real-time participation, but in general, nobody waits to discuss things, we just reach out and start the conversation...
16:45:16 <barthalion> you don't start, it became a showroom already
16:45:24 <barthalion> people are pasting their statuses
16:45:26 <kozhukalov> but they are more wide then just openstack
16:45:47 <mihgen> we have people spread out across world right now
16:46:01 <mihgen> we don't have a meeting where all fuel-library folks would be at the same time, for instance
16:46:24 <mihgen> so if we focus around smaller area, like fuel-library, and get everyone once a week in a text mode, that might be good
16:46:41 <barthalion> also, said that, participation isn't the easiest thing when meeting is starting 6PM, and I finish work 2 hours earlier
16:46:42 <angdraug> pasting the status if fine, it saves time, and is useful as a starting point of a discussion, IF people are interested
16:47:28 <barthalion> but I guess it's price of being international
16:48:14 <kozhukalov> ok, let's try to summarize
16:48:19 <mihgen> I thought current timing is actually pretty convenient for everyone whom we have (UA, RU, PL, US-east/west)
16:48:23 <barthalion> angdraug: it saves time, and makes the whole thing somehow creepy, as it looks like meeting with my manager
16:48:39 <barthalion> it's not only my opinion
16:48:48 <mkwiek> mihgen: I don't think there is a convenient time for all our time zones :)
16:48:50 <barthalion> also, the other thing, people often have nothing to say
16:48:54 <angdraug> I'll let your manager know you think he's creepy ;p
16:49:17 <barthalion> all meetings are creepy when you come from non-business open source :p
16:49:55 <barthalion> I'm not a feature lead, so I don't really know what topics I'm supposed to put in agenda
16:50:20 <kozhukalov> 1) we are community and this meeting is not showroom
16:50:39 <kozhukalov> 2) reporting status makes meeting more like meeting with managers
16:51:08 <xarses> so something that the opnfv folks where doing was running irc for links, but audio for their actuall meeting
16:51:08 <kozhukalov> 3) potentially we could discuss news (fuel and openstack related)
16:51:24 <akasatkin> we could report statuses just to openstack-dev
16:51:45 <angdraug> +1 for reporting statuses to openstack-dev
16:51:59 <angdraug> if a particular status kicks off a discussion it can be carried over into irc meeting
16:52:06 <xarses> this news thing seems not useful
16:52:46 <kozhukalov> 4) meeting is good for syncronous discussions but usually can not wait until next meeting if we have something to discuss
16:52:47 <docaedo> +1 angdraug on discussing something in IRC if it seems like a lot of people have much to say on something
16:53:04 <docaedo> and +10 on reporting statuses on openstack-dev
16:54:16 <barthalion> I want also to point out that usually questions are left unanswered on #fuel-dev
16:54:29 <xarses> and #fuel
16:54:36 <angdraug> barthalion: xarses: +1
16:54:41 <barthalion> I don't think it should happen, but I don't really have knowledge to help
16:54:49 <angdraug> that's what I was trying to say in the beginning: we're not using irc as much as we should
16:55:04 <kozhukalov> ok, what do you guys think, do we need this meeting at all, or maybe it is better to take a break and see if we will miss it?
16:55:11 <barthalion> another problem I see is that we have too many channels
16:55:13 <xarses> it seems most people shuffled to slack and have forgotten irc
16:55:40 <barthalion> #fuel-{devops,osci} should really be merged with #fuel-dev
16:55:52 <mihgen> barthalion: in both ways?
16:56:02 <barthalion> what do you mean?
16:56:16 <angdraug> yes, both in irc and slack
16:56:43 <barthalion> no idea about slack
16:57:00 <angdraug> yes, we have too many channels in irc and even more in slack
16:57:01 <alex_didenko> not sure about slack, but definitely +1 for merging in IRC and removing bot from there
16:57:10 <kozhukalov> #startvote Do  we need to take a break and stop running this meeting?
16:57:11 <openstack> Begin voting on: Do  we need to take a break and stop running this meeting? Valid vote options are Yes, No.
16:57:12 <openstack> Vote using '#vote OPTION'. Only your last vote counts.
16:57:13 <barthalion> but if I'd be external contributor, I'd go postal when someone would tell me I need to go to #fuel-foobar to achieve something
16:57:28 <kozhukalov> #vote yes
16:57:32 <barthalion> #vote yes
16:57:35 <alex_didenko> #vote No
16:57:55 <barthalion> or actually…
16:58:00 <barthalion> #vote no
16:58:01 <xarses> #vote no
16:58:10 <daniel3_> #vote no
16:58:10 <angdraug> #vote no
16:58:11 <prmtl> #vote no
16:58:15 <mwhahaha> #vote no
16:58:25 <mkwiek> #vote no
16:58:37 <mihgen> #vote no
16:58:39 <docaedo> #vote no
16:58:59 <mihgen> )
16:59:09 <kozhukalov> ok, looks like majority of attendees are for continuing this meeting
16:59:37 <mihgen> I'm still insisting on having themes per meeting
16:59:44 <angdraug> I think we need to fix our openness (including this meeting), not abandon it
16:59:45 <kozhukalov> so, let's make it more interesting, don't hesitate to add your topics to agenda next week
16:59:57 <kozhukalov> time is over
16:59:58 <mihgen> let's say , one week is about networking, another week is about plugins, etc.
17:00:03 <barthalion> let's discuss it on mailing list
17:00:06 <kozhukalov> thanks everyone
17:00:10 <mihgen> thx
17:00:10 <kozhukalov> closing
17:00:14 <barthalion> bye
17:00:17 <kozhukalov> #endmeeting