15:02:52 <ihrachys> #startmeeting neutron_upgrades
15:02:53 <openstack> Meeting started Mon Mar 21 15:02:52 2016 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:57 <ihrachys> o/
15:02:58 <openstack> The meeting name has been set to 'neutron_upgrades'
15:03:00 <ihrachys> hi everyone!
15:03:01 <korzen> hello
15:03:06 <sayalilunkad> hello!
15:03:11 <jlibosva> hi
15:03:12 <electrocucaracha> hey
15:03:12 <saisriki> hi
15:03:49 <ihrachys> the previous meeting was canceled due to the code sprint we had
15:03:57 <ihrachys> so we should have stuff to go thru :)
15:04:00 <ihrachys> #topic Announcements
15:04:27 <ihrachys> so as I already mentioned, we had a code sprint for the upgrades team focusing on objects in Brno Mar 14-16
15:04:40 <ihrachys> thanks everyone for joining and pushing the agenda :)
15:04:57 <ihrachys> thanks everyone who helped us with patches remotely too :)
15:05:02 <korzen> thanks all for productive time and patch merged!
15:05:17 <ihrachys> we have a tiny report of mine at the mailing list:
15:05:18 <ihrachys> #link http://lists.openstack.org/pipermail/openstack-dev/2016-March/089769.html
15:05:27 <ihrachys> please chime in if I missed something
15:05:46 <sayalilunkad> thanks ihrachys
15:05:53 <ihrachys> apart from objects, we discussed other upgrades related matters
15:06:00 <ihrachys> some of that is captured at:
15:06:01 <ihrachys> #link https://etherpad.openstack.org/p/code-sprint-neutron-objects-brno
15:07:12 <ihrachys> anyhow, the main focus was objects, that you can see in lines 63- in the etherpad
15:07:27 <ihrachys> you see that we mapped resources and started filling in gaps
15:07:57 <ihrachys> we currently target port resource, but a lot of work done lays foundation for next objects too
15:08:25 <ihrachys> there were some complications with getting patches merged during the event due to Mitaka freeze, but anyway, we landed some pieces
15:08:31 <ihrachys> overall, good work team
15:09:06 <ihrachys> we need to follow up on patches we already have and get the plan we have [which is optimistic] reality
15:09:21 <ihrachys> the plan would be getting db code in neutron repo objectified till end of N
15:09:39 <ihrachys> #topic Objects
15:09:51 <rossella_s> sorry for joining late
15:09:52 * ihrachys should probably have switched the topic a while ago :)
15:10:16 <ihrachys> for everyone unaware, we track all objects patches at:
15:10:17 <ihrachys> #link https://review.openstack.org/#/q/topic:ovo
15:10:25 <ihrachys> please use the topic if you upload more
15:10:49 <ihrachys> I don't think we want to go thru specific patches right now, let's leave it to open discussion if there is need for that
15:11:00 <rossella_s> agreed
15:11:00 <ihrachys> any question on objects strategy?
15:11:37 <ihrachys> btw since we are now in N, we may want to start integrating objects into the db code :)
15:11:38 <jlibosva> the db code objectified means using objects in ml2 plugin?
15:11:40 <ihrachys> ok, let's move on
15:11:43 <ihrachys> #topic Partial Multinode Grenade
15:11:57 <ihrachys> there are several things to follow up here
15:12:11 <ihrachys> one is: we have the legacy l3 job in check queue, but it's non voting
15:12:53 * ihrachys searches for the link to graphite...
15:15:09 <ihrachys> it's amazing how I fail to find it :)
15:15:18 <jlibosva> not sure what you mean, this one? http://docs.openstack.org/developer/neutron/dashboards/check.dashboard.html
15:15:44 <ihrachys> YES
15:16:39 <ihrachys> meh, not sure that's the one. we had some other graph with grenade multinode
15:16:49 <ihrachys> anyhow...
15:17:15 <ihrachys> so the last time I saw the graph, it was failing a bit more often than general multinode jobs
15:17:38 <ihrachys> which may indicate some issues in stable branches, like valuable patches not backported or smth
15:17:46 <ihrachys> we would need to dig into it before we enable voting
15:18:09 <ihrachys> ok, that's the link
15:18:11 <ihrachys> #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=6&fullscreen
15:18:39 <ihrachys> note that it's around 13% failure rate for the job
15:18:57 <ihrachys> comparing to 7% for full multinode tempest
15:19:09 <ihrachys> we need to tackle the gap
15:19:27 <ihrachys> anyone willing to scratch existing logs for the job to identify most common reasons for failures?
15:19:29 <rossella_s> for me it takes ages to load, is it just me?
15:19:40 <sayalilunkad> rossella_s: no its same for me
15:19:40 <ihrachys> rossella_s: it does, yes. but it will happen :)
15:19:41 <korzen> it did not load to me
15:19:48 <ihrachys> just wait
15:19:56 <rossella_s> ihrachys, I can investigate
15:20:03 <rossella_s> I can report back next week
15:20:03 * ihrachys waits until everyone sees the graph
15:20:11 <ihrachys> rossella_s: cool!
15:20:34 <ihrachys> #action rossella_s to investigate major reasons for grenade multinode job to fail with 15% rate
15:20:57 <ihrachys> ok, another job we have is the dvr flavour for the same job
15:21:03 <jlibosva> note that grenade runs tempest twice, so if there is a 7% chance to fail tempest, it will likely double in grenade
15:21:20 <ihrachys> jlibosva: yeah, but it does not execute all tests, just smoe
15:21:21 <ihrachys> *smoke
15:22:16 <ihrachys> anyway... so speaking about dvr flavour, korzen, can you give an update?
15:22:50 <korzen> well, I have run it once or twice and it was successfull, I did not dig into it deeper
15:23:35 <ihrachys> korzen: ok, I guess we need some plan to get it into check queue, and maybe replacing the legacy one
15:24:15 <korzen> what is the requirement for the job beeing included into the check>?
15:25:26 <ihrachys> korzen: I guess it's a matter of setting priorities. if the project considers rolling upgrade a priority to care on daily basis, then the argument can be done for the job. otherwise, if it's a niche feature, it may not have a slot there.
15:25:49 <ihrachys> with all the work that the team handles, I *hope* we are in the former category
15:26:03 <ihrachys> that should be discussed with PTL though
15:26:49 <ihrachys> somewhat relevant, there is that mailing thread with concern about jobs taking too much time
15:26:52 <ihrachys> #link http://lists.openstack.org/pipermail/openstack-dev/2016-March/089966.html
15:27:16 <ihrachys> we may consider adding the job depending on what the conclusion there will be
15:28:41 <ihrachys> ok, one additional thing to note on dvr job is it does not execute dvr tests :)
15:28:47 <rossella_s> :O
15:28:48 <ihrachys> and those tests are in neutron tree
15:28:57 <ihrachys> so we need to pull them from there somehow
15:29:05 <ihrachys> f.e. thru tempest plugin
15:29:22 <ihrachys> there is a patch to add such a plugin
15:29:24 <ihrachys> #link https://review.openstack.org/#/c/274023/
15:29:42 <ihrachys> though I am not completely aware about its implications
15:29:51 <ihrachys> more eyes on that one would be great to have
15:30:34 <ihrachys> rossella_s: it sounds crazy, but in a way, we may test dvr implicitly, creating routers. [as long as we don't enforce legacy thru API; that I am not sure about]
15:30:58 <ihrachys> anyhow, it does not run explicit dvr tests, and that's a thing to care about
15:31:00 <rossella_s> ihrachys, it's worth considering
15:31:36 <ihrachys> speaking of grenade job... do we have a reason to execute dhcp agent on controller [upgraded] node?
15:31:51 <ihrachys> I guess we could catch some RPC issues if we would run the old dhcp agent, right?
15:32:10 <korzen> yes
15:32:28 <korzen> the perfect idea would be to have network node
15:32:54 <ihrachys> yes, I believe you mentioned before devstack-gate is ready for 3 nodes?
15:33:04 <korzen> current grenade would not check the RPC interoperability between L3/DHCP agents with upgraded neutron server
15:33:42 <korzen> ihrachys, the infra doc is talking about 3 node setup
15:33:44 <ihrachys> korzen: yes. but dvr flavour would cover for L3
15:33:47 <ihrachys> but not for DHCP
15:34:12 <korzen> but I've never seen anyone using the third one
15:34:12 <ihrachys> even though strictly speaking legacy L3 agent speaks different RPC language than DVR..
15:35:09 <ihrachys> korzen: ok, I guess we may keep it in a further compartment in our brains :)
15:35:35 <ihrachys> I would first target getting DVR flavour voting, then looking into what we do with DHCP
15:35:53 <rossella_s> ihrachys, +1
15:37:20 <ihrachys> ok, I guess we have a plan for the next week: rossella_s will take a look at failure logs, and we talk to PTL and others about getting new grenade jobs voting.
15:37:47 <ihrachys> #topic Open discussion
15:38:02 <ihrachys> before you start sending questions team's way, I have one thing
15:38:11 <ihrachys> here in Europe, we have Easter Monday next Monday
15:38:32 <ihrachys> I suspect it's quite common in the team to have a day-off that day.
15:38:34 <ihrachys> amirite?
15:38:43 <korzen> yes
15:39:06 <ihrachys> if that's the case, we may want to cancel the next meeting.
15:39:13 * ihrachys waits for others to comment.
15:39:21 <sayalilunkad> ihrachys: yes
15:39:27 <electrocucaracha> +1
15:39:29 * jlibosva won't be online
15:39:33 <korzen> we can cancel
15:40:18 <ihrachys> ok, let's cancel then. the next meeting will be April 4th
15:40:56 <ihrachys> I presume that not having the next meeting won't stop anyone from getting more patches and work done even than before ;)
15:41:09 <rossella_s> really? :p
15:41:32 <ihrachys> absolutely! I count on everyone! :P
15:41:50 <rossella_s> ;)
15:42:01 <ihrachys> ok, anyone has anything else to raise?
15:42:03 <sayalilunkad> I need this patch for securitygroups https://review.openstack.org/#/c/295275/
15:42:03 <korzen> Anyone checked if do not have broken upgrades in Mitaka RC1?
15:42:33 <rossella_s> korzen, I haven't checked
15:42:37 <ihrachys> korzen: nope. Doesn't the job we have show that at least in some way it works? :)
15:42:42 <ihrachys> even if just for smole
15:42:44 <ihrachys> *smoke
15:42:57 <ihrachys> sayalilunkad: will take a look right after the meeting
15:43:02 <sayalilunkad> ihrachys: thanks
15:43:32 <rossella_s> sayalilunkad, I will take a look later today too
15:43:48 <korzen> I guess that a sanity check can be done
15:43:48 <sayalilunkad> thanks rossella_s
15:44:29 <korzen> I will take a look at db migration and RPC interface just to make sure, noone has introduce the incompatible change in the API
15:45:05 <ihrachys> #action korzen to look at db migration and RPC interface to make sure no incompatible changes landed in Mitaka
15:45:08 <ihrachys> korzen: thanks!
15:45:57 <ihrachys> gotta call it a day in 3...
15:46:04 <ihrachys> 2...
15:46:10 <ihrachys> 1...
15:46:15 <rossella_s> thanks!!
15:46:16 <ihrachys> sold
15:46:18 <ihrachys> #endmeeting