15:01:35 <ihrachys> #startmeeting neutron_upgrades
15:01:37 <openstack> Meeting started Mon Mar  7 15:01:35 2016 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:39 <rossella_s> hi
15:01:40 <openstack> The meeting name has been set to 'neutron_upgrades'
15:01:45 <korzen> hello
15:01:46 <slunkad> hello
15:01:52 <ihrachys> hi :)
15:02:05 <ihrachys> #topic Announcements
15:02:36 <ihrachys> so we have the code sprint next week March 14-16 in Brno. I guess everyone who planned to go already booked hotels/flights.
15:03:00 <ihrachys> it's worth we work on the etherpad for the sprint before the next week
15:03:01 <ihrachys> #link https://etherpad.openstack.org/p/code-sprint-neutron-objects-brno
15:03:21 <rossella_s> right :)
15:03:23 <korzen> I have added a few points worth discussing
15:03:35 <ihrachys> korzen is cool, so he started a draft of agenda for the sprint, line 42+
15:03:58 * ihrachys is not cool, so he did not
15:04:12 <enriquetaso> rossella_s: hello
15:04:20 <ihrachys> korzen: discussing here or at the event?
15:05:01 <mhickey> Hello, sorry i'm late
15:05:02 <korzen> we can go quickly right now
15:05:10 <ihrachys> korzen: ok, please lead :)
15:05:45 <rossella_s> enriquetaso, welcome
15:05:52 <korzen> ok, so the first one the obvious: Port all database entities to OVO
15:06:23 <korzen> I guess that it would be nice to have the list and share the items with volountiers
15:06:42 <ihrachys> yay. I guess we should come with some short-term plan for the sprint that is more realistic.
15:06:46 <korzen> so if we have new people we can point to the TODO list
15:07:23 <ihrachys> let's say we at least solve port and subnets on the sprint and document everything.
15:07:33 <korzen> in the objectification it would be also nice to have performance comparison before OVO and after
15:07:34 <rossella_s> ihrachys, agreed
15:07:49 <ihrachys> korzen: rally jobs should give some insight
15:08:23 <korzen> As we discussed earlier - we can define use cases for OVO
15:08:34 <korzen> it means, where the OVO would be used
15:08:58 <rossella_s> ihrachys, did you hear back from armax, is it confirmed that Newton will be open?
15:09:16 <ihrachys> rossella_s: not yet. I think it was Fri the last time we talked.
15:09:27 <korzen> and how to go forward with online schema migration and online schema migration
15:09:34 <armax> rossella_s: newton will be open as soon as stable/mitaka is open, that should happen at the end of this week
15:09:47 <armax> rossella_s: or first thing next week
15:09:55 <rossella_s> armax, great, thanks a lot!
15:10:05 <korzen> thanks armax, great news
15:10:13 <ihrachys> armax++
15:10:20 <mhickey> armax, welcome! :)
15:10:34 <rossella_s> so ihrachys can we aim at having port and subnet merged and working in Newton after the sprint? :)
15:10:39 <ihrachys> so we should be ready to land stuff on the sprint
15:10:45 <armax> no problem folks :) keep up the good work!
15:10:51 <ihrachys> rossella_s: I think we should, yes. :)
15:11:25 <ihrachys> that would be a lot of stuff already, and we'll have many foundational blocks already set in tree for quick proceeding on remaining resources
15:11:37 <slunkad> will port also include the port extensions?
15:11:40 <rossella_s> yep that would be perfect
15:11:47 <ihrachys> remember, one thing that should come up from the first objects we'll land is a detailed documentation on transition process.
15:11:47 <rossella_s> slunkad, yes
15:12:05 <ihrachys> slunkad: I think that's a requirement to land port, yes
15:12:07 <rossella_s> ihrachys, good reminder
15:12:37 <korzen> I guess that would be enough work for 3 days ;)
15:12:40 <slunkad> cool
15:13:01 <mhickey> korzen: no time for food or beer! :)
15:13:03 <ihrachys> korzen: :-) but I guess we'll need to touch other topics at least to set everyone on the same page for Newton
15:13:25 <korzen> ok, just joking, what can we do more is Grenade testing
15:13:27 <ihrachys> korzen: you had some more topics in the etherpad. dare to elaborate?
15:14:04 <korzen> Grenade or upgrade testing lacks of dataplane connectivity test in CI
15:14:05 <ihrachys> + on grenade. I think we made a huge progress in Mitaka, but it's not enough, we should strive for more :)
15:14:36 <korzen> we should work on solution enabling the flow testing are not dropped
15:15:05 <ihrachys> korzen: actually, for agent restart, I saw a patch from hmlnarik lately that had a functional test that validates just that. right rossella_s ?
15:15:19 <ihrachys> rossella_s: I saw you were reviewing that test. [unless I halucinate]
15:15:38 <ihrachys> it's not exactly upgrade, but could be a good start.
15:15:40 <rossella_s> ihrachys, i don't know how you can keep up with so many things, that's right
15:15:58 <ihrachys> that thing:
15:16:00 <ihrachys> #link https://review.openstack.org/#/c/284639/8/neutron/tests/functional/agent/test_l2_ovs_agent.py
15:16:06 <rossella_s> ihrachys, right it's a good start, it not about upgrade but about restarting the agent
15:16:35 <ihrachys> rossella_s: huh. Hynek sits right to me in the office, that's how :P
15:16:51 <korzen> the question is, can we take advantage of it in Grenade tests?
15:16:57 <ihrachys> long story short, the test simulates agent restarts and checks that ping works.
15:17:41 <rossella_s> ihrachys, oh really? :D I had a conversation with him regarding the test...
15:17:57 <ihrachys> korzen: for grenade, I guess no. we would need to come up with some background process running during l2 agent upgrade (that would go to usual, 'non-partial' grenade flavor)
15:18:42 <ihrachys> korzen: ok, anything else on your agenda for sprint?
15:19:10 <korzen> I guess we can also talk about the RPC callback being used for Port or smth
15:19:27 <korzen> definitly for Newton release
15:20:26 <ihrachys> korzen: yes, we may need to start thinking of how we make RPC layer more strict. I don't think we'll spend much time actually coding anything there on the sprint, but it's good to have a brief discussion on the next steps.
15:20:50 <korzen> and also the reference architecture for rolling upgrade scenario in Neutron
15:21:08 <rossella_s> ihrachys, I agree, it definitely needs discussion. We can start during the sprint but I think we will have to talk about it again at the summit
15:21:27 <ihrachys> korzen: can you elaborate on the last one? is it some detailed description of upgrade process and what's supported somewhere in user visible docs?
15:22:25 <korzen> I'm seeing the ref implementation as an description of what Neutron components we are testing
15:22:33 <korzen> and procedure as well
15:22:40 <korzen> procedure of upgrades
15:22:58 <ihrachys> korzen: something in networking-guide? what's the audience?
15:23:35 <korzen> audence should be operators
15:25:10 <korzen> I'm not sure if it is worth dicussing on the code sprint but just my thought
15:25:11 <ihrachys> korzen: ok, so I guess it's networking-guide material
15:25:39 <ihrachys> I guess we'll discuss a lot of stuff, that's good to collect ideas, even if small :)
15:26:05 <ihrachys> ok great. all in all, I encourage everyone to go the etherpad and build agenda on top of what korzen contributed.
15:26:11 <ihrachys> korzen: thanks for doing it.
15:26:20 <korzen> ihrachys, np.
15:26:30 <mhickey> korzen:++
15:26:44 <saisriki> hi, sorry i'm late
15:26:44 <ihrachys> let's move to more general topics (even though we touched some of them already)
15:26:51 <ihrachys> #topic Object implementation
15:27:03 <ihrachys> for that, there was not much progress merging patches due to Mitaka freeze
15:27:10 <ihrachys> as we discussed already, master will be open next week.
15:27:16 <saisriki> attending bug hackathon in the castle
15:27:16 <ihrachys> so we'll be able to break the world :)
15:27:42 <ihrachys> I suspect there is not much need to cover specific patches for objects.
15:27:52 <korzen> my patch got merged: https://review.openstack.org/#/c/275790/ composite primary key
15:27:59 <ihrachys> yay
15:27:59 <ihrachys> overall, the only comment I have is that
15:28:04 <mhickey> korzen: well done
15:28:08 <saisriki> korzen: great
15:28:16 <ihrachys> ...if you have patches in review, you better get them in shape for sprint time
15:28:17 <electrocucaracha> korzen, :)
15:28:26 <korzen> I think the most important one now is: https://review.openstack.org/#/c/283711
15:28:26 <korzen> Handle synthetic fields in NeutronDbObject
15:28:28 <ihrachys> so that we don't lag on merging/reviewing them ;)
15:29:34 <korzen> yes, so I would work on SUbnet and maybe network patches to get up to date :)
15:29:35 <rossella_s> I am updating the patch according to korzen's comment
15:29:38 <ihrachys> korzen: will need to wait till Mitaka. we could land some custom type patches this week since they are isolated, but that's it.
15:29:39 <mhickey> rossella_s: maybe set topic to ov for https://review.openstack.org/#/c/283711?
15:29:46 <mhickey> *ovo
15:30:17 <mhickey> ihrachys; custom type patches seem to be locked down?
15:30:19 <rossella_s> mhickey, right
15:30:35 <ihrachys> mhickey: locked down as in '-2 from Armando'?
15:31:05 <mhickey> ihrachys: yes, https://review.openstack.org/#/c/277558/
15:31:14 <ihrachys> ok, if something of those patches is really ready to land, we can talk it thru with Armando case by case
15:31:49 <mhickey> strangely enough, cidr is not: https://review.openstack.org/#/c/285349/
15:32:18 <korzen> yay :)
15:32:30 <mhickey> korzen: lol
15:32:35 <ihrachys> a glitch I guess. or the fact that I +2 on the first one, so there was a risk it will land with no approval.
15:32:49 <ihrachys> ok, let's solve it off the meeting
15:32:57 <ihrachys> #topic Partial Multinode Grenade
15:33:01 <mhickey> ihrachys: maybe not, I might have upset armax at the mid-cycle! :)
15:33:26 <ihrachys> for partial, I guess we are waiting for N to open to enable voting for the job
15:33:29 <armax> mhickey: far from it!
15:33:35 <ihrachys> we also landed DVR experimental job
15:33:40 <mhickey> armax: ciao
15:33:50 <armax> mhickey: buongiorno!
15:34:00 <armax> mhickey: scusa, buonasera
15:34:07 <ihrachys> and I heard Sean made a huge progress on that one, making it passing the vote in the experimental queue
15:34:21 <armax> ihrachys: the dvr one?
15:34:24 <ihrachys> sadly, Sean is not here right now (he was planning to travel by train, that's why)
15:34:25 <korzen> yes, the multinode DVR job was merged last week, I have done the initial run: http://logs.openstack.org/50/281850/7/experimental/gate-grenade-dsvm-neutron-dvr-multinode/7b81449/
15:34:42 <ihrachys> armax: yes, dvr passed with some patch from Sean, as per Sean himself
15:34:49 <armax> ihrachys: nice
15:34:58 <ihrachys> armax: I thought we will get details here, but probably not right now
15:35:07 <ihrachys> armax: it is huge, not just nice.
15:35:34 <armax> ihrachys: you’re younger and more enthusiastic than me
15:35:38 <ihrachys> also, armax proposed a patch to add rolling-upgrade tag to neutron
15:35:39 <armax> ihrachys: it’s understandable
15:35:40 <ihrachys> #link https://review.openstack.org/#/c/286817/
15:35:43 <ihrachys> armax: :P
15:36:01 <ihrachys> that said, armax seems to be unconvinced what we have justifies the tag right now
15:36:23 <ihrachys> if I understand you correctly, you want us to evaluate *aas gates for partial job too, right?
15:36:55 <armax> ihrachys: I am mulling over the idea, I am not entirely convinced
15:37:05 <armax> ihrachys: but ultimately that’s down to what the stadium looks like
15:37:27 <armax> ihrachys: because if we turn out to remove all projects but neutron, then even the question goes away
15:37:50 <ihrachys> I guess one thing we may want to deliver to the project after the sprint is some high level description of what we consider a proper subset of upgrade  scenarios that would reflect real deployments and won't require a dozen of new jobs.
15:38:26 <ihrachys> even if not *aas in the scope, then we have e.g. mixed L3 HA agent versions scenario (raised by Sean today)
15:38:54 <ihrachys> any thoughts folks on where we should claim 'done'?
15:38:56 <korzen> ihrachys, can you elaborate more on the L3 HA scenario?
15:39:38 <armax> ihrachys: I don’t think the L3 HA scenario should be added to the mix, but ideally we could simply consider a multinode
15:39:41 <ihrachys> korzen: if you run HA router served by two separate nodes, you may want to make sure that it's still working (VRRP talking) when you upgrade a node per step
15:39:41 <armax> DVR+HA
15:39:58 <armax> I think ultimately the most intereting multinode testing
15:40:09 <armax> the others, perhaps we can afford to have them set up as periodic
15:40:41 <ihrachys> armax: one more general question to tackle is when we talk about rolling scenarios, do we envision per-service upgrade or per-node upgrade?
15:40:55 <armax> ihrachys: what do you mean?
15:40:57 <ihrachys> the difference is that in the former case you would want to test e.g. new nova-compute with old l2 agent.
15:41:19 <ihrachys> and in the latter, you would assume a compute node runs the same major version of all components of all projects
15:41:29 <armax> ihrachys: I think it’s safe to assume the latter
15:41:37 <ihrachys> and the same would go about 'networking' node that would run l3 agent and dhcp agent and whatnot
15:41:42 <armax> ihrachys: but we can double check with the nova guys
15:42:05 <ihrachys> yes, latter should be safe. also it should reflect reality. [at least until major use case is containerized]
15:42:06 <armax> typically you’d refresh all deps on a single node
15:42:12 <armax> ihrachys: right
15:42:35 <ihrachys> armax: I talked to dansmith about that, he was actually pretty happy we DON'T assume per-service upgrades
15:42:43 <armax> ihrachys: that said, it should be entirely possible to have nova N to work with Neutron N-1
15:43:12 <ihrachys> armax: of course. the question is more about testing matrix than about what we claim to support.
15:43:14 <armax> ihrachys: we don’t in fact
15:43:46 <armax> ihrachys: there’s not any amount of testing that can replace a good and intelligent review
15:43:52 <armax> from a human being
15:43:55 <ihrachys> amen on that
15:44:27 <korzen> are there any teams already doing the rolling upgrades in scale env?
15:44:50 <korzen> not the dev setup, the large servers with 200+ VMs on it
15:45:54 <korzen> it would be nice to have tests done in semi automated way testing the project upgrade during last day of dev cycle
15:45:55 <ihrachys> not that I know. we have very limited rolling scenario testing in-house
15:46:39 <korzen> because what we have to do now, is to setup M-3 and check the liberty to M-3 or Mitaka RC1 upgrade
15:47:10 <korzen> to be sure before release that nothing critcal got merdeg during Mitaka
15:47:27 <korzen> s/merdeg/merged
15:48:05 <rossella_s> korzen, I don't think we have anything like that
15:48:22 <ihrachys> I hear you. it always comes with the question on who is going to do the testing (and documentation of the process for later upgrade checks)
15:49:25 <rossella_s> ihrachys, korzen are we a bit OT now?
15:49:27 <ihrachys> overall, that raises the question of neutron not having any major release check list that PTL or someone in charge could go thru and validate that all usual release stuff is covered.
15:49:37 <ihrachys> yeah, we probably are :)
15:49:41 <ihrachys> let's move on
15:49:53 <ihrachys> actually, I have nothing so...
15:49:55 <rossella_s> we can discuss that after the meeting
15:50:05 <ihrachys> #topic Open discussion
15:50:42 <korzen> talking about test the L3 HA in CI
15:50:43 <rossella_s> I just wanted to say, let's brush up the ovo patches these week so that we can easily merge them during the sprint
15:50:53 <korzen> we would need to have 3 node setup right?
15:50:58 <ihrachys> rossella_s++
15:51:00 <mhickey> rossella_s: Agreed ++
15:51:44 <mhickey> clearing these patches in progress, could clear the way for whats next TBD
15:51:50 <ihrachys> korzen: probably yes. controller (running l3 agent) + networking node (the agent too) + compute
15:52:32 <korzen> currently there are 2 nodes used
15:52:55 <korzen> it would be hard work to setup 3 nodes
15:53:10 <ihrachys> first we should discuss whether it's what we want, then we may look into what's missing on devstack-gate side
15:53:18 <korzen> mostly in devstack-gate
15:53:19 <ihrachys> I believe devstack-gate supports more nodes
15:53:31 <ihrachys> I saw its code referring to subnode arrays
15:54:11 <ihrachys> I am sure there are devils in details though
15:54:29 <korzen> but configuration is written up to handle controller and subnode
15:54:47 <korzen> it woudl require more 'if's
15:54:54 <korzen> to add third node :P
15:55:05 <ihrachys> more spaghetti, yes. as if devstack-gate does not have enough of it.
15:55:12 <johnthetubaguy> I am curious, why would three nodes be useful?
15:55:12 <slunkad> is someone already working on the Port security?
15:55:34 <slunkad> since we decided to brush up all the ovo patches.. I am not sure we have a patch for that
15:56:09 <rossella_s> slunkad, enriquetaso is working on it
15:56:14 <korzen> johnthetubaguy, the thee node is useful when you have testing the multiple L3 agents
15:56:14 <ihrachys> johnthetubaguy: which nodes are going to run l3 agent? or do you suggest we should land l3 agent on compute node to save a node?
15:56:23 <slunkad> oh awesome enriquetaso
15:56:28 <slunkad> :)
15:57:02 <johnthetubaguy> I guess I was thinking run that on one of the existing nodes, yeah
15:57:03 <slunkad> btw enriquetaso is an applicant for this round of  the Outreachy internship
15:57:22 <slunkad> for those who don't know :)
15:57:33 <johnthetubaguy> ihrachys: so yeah, +1 your suggestion, but that might complicate some of the routes, etc, I guess, to that might be a bad idea
15:57:36 <enriquetaso> enriquetaso: I'm working on it haha
15:57:49 <ihrachys> enriquetaso: great!
15:58:38 <ihrachys> johnthetubaguy: we may think indeed about landing it on compute. as long as other compute services are running old code, that can give us proper coverage.
15:58:41 <ihrachys> need to think about it.
15:59:10 <ihrachys> ok folks, we need to wrap up. please make sure all OVO patches are in good shape for the next week sprint, and keep up the good work!
15:59:19 <johnthetubaguy> ihrachys: ack
15:59:21 <rossella_s> thanks ihrachys
15:59:25 <slunkad> yep! bye :)
15:59:29 <korzen> thanks, bye
15:59:32 <enriquetaso> thanks ihrachys
15:59:33 <saisriki> thank you
15:59:37 <ihrachys> btw I will try to arrange a video conf option for the code sprint, will post details later.
15:59:37 <mhickey> thanks, bye
15:59:41 <ihrachys> #endmeeting