Tuesday, 2020-03-10

*** bcafarel has joined #openstack-meeting00:04
*** vishalmanchanda has joined #openstack-meeting00:05
*** tetsuro has joined #openstack-meeting00:06
*** macz_ has quit IRC00:06
*** jamesmcarthur has joined #openstack-meeting00:11
*** tetsuro has quit IRC00:18
*** tetsuro has joined #openstack-meeting00:18
*** jamesmcarthur has quit IRC00:19
*** jamesmcarthur has joined #openstack-meeting00:21
*** tetsuro has quit IRC00:21
*** tetsuro has joined #openstack-meeting00:22
*** dklyle has quit IRC00:22
*** jamesmcarthur has quit IRC00:26
*** diablo_rojo has quit IRC00:42
*** jamesmcarthur has joined #openstack-meeting00:44
*** yamamoto has joined #openstack-meeting00:54
*** jamesmcarthur has quit IRC01:08
*** jamesmcarthur has joined #openstack-meeting01:12
*** jamesmcarthur has quit IRC01:24
*** jamesmcarthur has joined #openstack-meeting01:24
*** hongbin has joined #openstack-meeting01:31
*** ricolin_ has joined #openstack-meeting01:39
*** armax has quit IRC01:43
*** enriquetaso has quit IRC01:51
*** yamamoto has quit IRC02:01
*** jamesmcarthur has quit IRC02:06
*** jamesmcarthur has joined #openstack-meeting02:07
*** yamamoto has joined #openstack-meeting02:07
*** masahito has joined #openstack-meeting02:12
*** masahito has quit IRC02:13
*** brinzhang_ has quit IRC02:14
*** brinzhang_ has joined #openstack-meeting02:15
*** mhen has quit IRC02:20
*** nicolasbock has joined #openstack-meeting02:26
*** enriquetaso has joined #openstack-meeting02:29
*** yaawang has quit IRC02:47
*** yaawang has joined #openstack-meeting02:50
*** mahatic has joined #openstack-meeting02:51
*** yamamoto has quit IRC02:52
*** yamamoto has joined #openstack-meeting02:54
*** vishalmanchanda has quit IRC03:05
*** enriquetaso has quit IRC03:05
*** nicolasbock has quit IRC03:06
*** yaawang has quit IRC03:07
*** yaawang has joined #openstack-meeting03:08
*** jamesmcarthur has quit IRC03:21
*** vishalmanchanda has joined #openstack-meeting03:31
*** hongbin has quit IRC03:47
*** tpatil has joined #openstack-meeting03:55
*** suzhengwei_ has joined #openstack-meeting03:58
tpatil#startmeeting Masakari04:00
openstackMeeting started Tue Mar 10 04:00:14 2020 UTC and is due to finish in 60 minutes.  The chair is tpatil. Information about MeetBot at http://wiki.debian.org/MeetBot.04:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.04:00
*** openstack changes topic to " (Meeting topic: Masakari)"04:00
openstackThe meeting name has been set to 'masakari'04:00
tpatilHello Masakari members04:00
tpatilToday Sampath san is joining late04:00
tpatilRoll call?04:00
suzhengwei_tpatil: hi04:02
tpatilsuzhengwei_: Hi04:02
tpatil#topic Critical Bugs04:03
*** openstack changes topic to "Critical Bugs (Meeting topic: Masakari)"04:03
tpatil#link : https://bugs.launchpad.net/masakari/+bug/185876204:03
openstackLaunchpad bug 1858762 in masakari "“UNKNOWN” host_status notification may cause unsafe evacuation" [Critical,Confirmed] - Assigned to Shilpa Devharakar (shilpasd)04:03
tpatilAny update?04:03
*** rcernin has quit IRC04:04
*** rcernin has joined #openstack-meeting04:04
suzhengwei_Is Shilpa here?04:06
tpatilThis bug seems like a simple fix, need to check the host status and execute the workflow04:06
tpatilMoving ahead04:06
tpatilAny other bugs you want to bring up for discussion now?04:07
tpatilsuzhengwei_: I have checked your comment and it does  makes sense to me04:09
tpatilI have discussed about it with Sampath san and looking forward for his comment on the same.04:10
suzhengwei_thx. another one, https://review.opendev.org/#/c/702328/04:10
suzhengwei_it need some review04:11
tpatilRegarding the first patch, could you please add unit tests for your changes? I  know the changes are simple, but we need unit tests for bug fixes04:12
suzhengwei_ok, I will try to add some unit tests later.04:13
tpatilsuzhengwei_: Thanks04:13
tpatilhttps://review.opendev.org/#/c/702328/, I will review this patch in this week. Sorry lately I'm busy working on another project so I couldn't spend time on this project.04:13
suzhengwei_This is a simple bug, too04:14
tpatilsuzhengwei_: Agree, will review this patch as well in this week.04:15
tpatil#topic: Ussuri work items04:16
*** openstack changes topic to ": Ussuri work items (Meeting topic: Masakari)"04:16
tpatil#link : https://etherpad.openstack.org/p/masakari-u-workitems04:16
tpatilEnable/Disable evacuation segment wise04:17
tpatilSpec: https://review.opendev.org/#/c/705893/04:18
tpatilsuzhengwei_: I have seen you have uploaded a new PS. Will review it soon.04:19
tpatilsuzhengwei_: You also have implemented the spec, so we will plan to plan this feature in U release.04:20
suzhengwei_yes, I will finish it before U release.04:21
*** kiyofujin has joined #openstack-meeting04:21
tpatilsuzhengwei_: Thanks04:22
tpatil# link: https://review.opendev.org/#/c/704811/04:22
tpatilsuzhengwei_: I have seen your comment on above patch. Also, discussed with Sampath san offline. Unfortunately, he is not available today yet04:23
tpatilIMHO, it's ok to add a new API GET /hosts and GET /hosts/<hostuuid>04:24
suzhengwei_You can leave some comment or message under the patch.04:24
tpatilQuestion arise, whether  it's ok to continue existing API GET /segments/{segment_id}/hosts04:25
tpatilor deprecated it04:25
suzhengwei_Maybe we can talk about it sometime later, when more people are here.04:26
tpatilbecause supporting both these API would be redundant04:26
tpatilsuzhengwei_: Agree04:27
tpatilsuzhengwei_ : Do you have anything else to discuss about U work items?04:28
*** yaawang has quit IRC04:28
*** yaawang has joined #openstack-meeting04:29
suzhengwei_this is a simple issue, it is not a bp. I just want to draw some notices.04:30
tpatilFrom masakari point of view, I think what's important is nova-compute process and for the rest of the processes, masakari doesn't take any actions04:31
suzhengwei_yes, but it will try to restart the failed processes in monitor.04:32
tpatilThe current code is not suitable if you are monitoring too many processes.04:32
suzhengwei_it is not good in line flow when two or more process is failed.04:33
tpatilI don't think you need a spec for this change04:33
suzhengwei_No, it doesn't.04:34
tpatilYou can allocate a pool of thread to do the job. definitely we cannot create one thread per process. just  in case operator adds too many processes for monitoring.04:35
tpatilsuzhengwei_: Feel free to work on these changes. No bp/spec need for this change.04:36
suzhengwei_The monitor process list contain nova-compute, libvirt and ovs in my env.04:37
suzhengwei_OK, I will give a patch directly.04:38
tpatilsuzhengwei_: Thanks04:38
*** ianychoi has quit IRC04:39
*** ianychoi has joined #openstack-meeting04:40
tpatilThat's all I wanted to bring up in this meeting04:41
tpatilIf no other topics left for discussion let's close this meeting early04:42
tpatilsuzhengwei_: Thank you for joining the meeting04:43
tpatilSee you all then in the next meeting. Thank you04:44
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"04:44
openstackMeeting ended Tue Mar 10 04:44:42 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)04:44
openstackMinutes:        http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-03-10-04.00.html04:44
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-03-10-04.00.txt04:44
openstackLog:            http://eavesdrop.openstack.org/meetings/masakari/2020/masakari.2020-03-10-04.00.log.html04:44
*** jmasud_ has joined #openstack-meeting04:51
*** masahito has joined #openstack-meeting04:52
*** jmasud has quit IRC04:54
*** tpatil has quit IRC04:56
*** kiyofujin has quit IRC04:57
*** yamamoto has quit IRC04:57
*** suzhengwei_ has quit IRC05:27
*** yamamoto has joined #openstack-meeting05:27
*** lajoskatona has joined #openstack-meeting05:42
*** TrevorV has joined #openstack-meeting05:58
*** ociuhandu has joined #openstack-meeting06:07
*** ijw_ has quit IRC06:08
*** ociuhandu has quit IRC06:12
*** links has joined #openstack-meeting06:15
*** ricolin_ has quit IRC06:23
*** ricolin has joined #openstack-meeting06:23
*** ijw has joined #openstack-meeting06:33
*** ijw has quit IRC06:38
*** e0ne has joined #openstack-meeting07:17
*** e0ne has quit IRC07:22
*** ociuhandu has joined #openstack-meeting07:28
*** tetsuro has quit IRC07:29
*** Lucas_Gray has joined #openstack-meeting07:30
*** suzhengwei has joined #openstack-meeting07:32
*** yamamoto has quit IRC07:33
*** ociuhandu has quit IRC07:35
*** yamamoto has joined #openstack-meeting07:35
*** tetsuro has joined #openstack-meeting07:43
*** Lucas_Gray has quit IRC07:45
*** Lucas_Gray has joined #openstack-meeting07:46
*** suzhengwei has quit IRC07:47
*** Wryhder has joined #openstack-meeting07:51
*** Lucas_Gray has quit IRC07:51
*** Wryhder is now known as Lucas_Gray07:52
*** slaweq has joined #openstack-meeting07:56
*** yasufum has joined #openstack-meeting07:56
*** witek has joined #openstack-meeting07:58
*** tpatil has joined #openstack-meeting08:01
*** takahashi-tsc has joined #openstack-meeting08:03
*** ociuhandu has joined #openstack-meeting08:05
*** JangwonLee has joined #openstack-meeting08:06
*** ijw has joined #openstack-meeting08:08
*** nitinuikey has joined #openstack-meeting08:08
*** tesseract has joined #openstack-meeting08:11
*** ijw has quit IRC08:13
*** ociuhandu has quit IRC08:18
*** yaawang has quit IRC08:20
*** yaawang has joined #openstack-meeting08:21
*** pbhole has joined #openstack-meeting08:22
*** rubasov has joined #openstack-meeting08:26
*** tpatil has quit IRC08:31
*** masahito has quit IRC08:35
*** ijw has joined #openstack-meeting08:39
*** rpittau|afk is now known as rpittau08:43
*** ijw has quit IRC08:44
*** priteau has joined #openstack-meeting08:44
*** TrevorV has quit IRC08:45
*** nitinuikey has quit IRC08:45
*** ralonsoh has joined #openstack-meeting08:50
*** Wryhder has joined #openstack-meeting08:50
*** Lucas_Gray has quit IRC08:51
*** Wryhder is now known as Lucas_Gray08:51
*** ociuhandu has joined #openstack-meeting08:53
*** jmasud_ has quit IRC08:55
*** jmasud has joined #openstack-meeting08:56
*** pbhole has quit IRC08:56
*** yasufum has quit IRC08:58
*** njohnston has quit IRC09:00
*** yamamoto has quit IRC09:02
*** yamamoto has joined #openstack-meeting09:06
*** ociuhandu has quit IRC09:09
*** ijw has joined #openstack-meeting09:10
*** ijw has quit IRC09:15
*** priteau has quit IRC09:28
*** tetsuro has quit IRC09:31
*** ijw has joined #openstack-meeting09:42
*** ijw has quit IRC09:46
*** gibi has quit IRC10:06
*** gibi has joined #openstack-meeting10:07
*** yamamoto has quit IRC10:13
*** ociuhandu has joined #openstack-meeting10:17
*** e0ne has joined #openstack-meeting10:36
*** ijw has joined #openstack-meeting10:44
*** ijw has quit IRC10:49
*** yamamoto has joined #openstack-meeting10:57
*** Lucas_Gray has quit IRC11:04
*** yamamoto has quit IRC11:07
*** Lucas_Gray has joined #openstack-meeting11:08
*** masahito has joined #openstack-meeting11:12
*** ianychoi has quit IRC11:16
*** rpittau is now known as rpittau|bbl11:16
*** ianychoi has joined #openstack-meeting11:16
*** njohnston has joined #openstack-meeting11:17
*** jamesmcarthur has joined #openstack-meeting11:19
*** yamamoto has joined #openstack-meeting11:24
*** jamesmcarthur has quit IRC11:24
*** Lucas_Gray has quit IRC11:26
*** Lucas_Gray has joined #openstack-meeting11:28
*** masahito has quit IRC11:33
*** yamamoto has quit IRC11:37
*** Luzi has joined #openstack-meeting11:37
*** ociuhandu has quit IRC11:40
*** ociuhandu has joined #openstack-meeting11:40
*** ociuhandu has quit IRC11:46
*** nicolasbock has joined #openstack-meeting11:54
*** rh-jelabarre has joined #openstack-meeting11:59
*** raildo has joined #openstack-meeting12:07
*** Lucas_Gray has quit IRC12:12
*** nicolasbock has quit IRC12:13
*** nicolasbock has joined #openstack-meeting12:15
*** Lucas_Gray has joined #openstack-meeting12:19
*** raildo has quit IRC12:19
*** raildo has joined #openstack-meeting12:20
*** jamesmcarthur has joined #openstack-meeting12:21
*** enriquetaso has joined #openstack-meeting12:24
*** ociuhandu has joined #openstack-meeting12:29
*** rh-jelabarre has quit IRC12:36
*** jamesmcarthur has quit IRC12:36
*** ociuhandu has quit IRC12:43
*** ociuhandu has joined #openstack-meeting12:44
*** ociuhandu has quit IRC12:49
*** ianychoi has quit IRC12:49
*** ijw has joined #openstack-meeting12:49
*** ianychoi has joined #openstack-meeting12:50
*** jmasud has quit IRC12:53
*** rpittau|bbl is now known as rpittau12:54
*** ijw has quit IRC12:54
*** jmasud has joined #openstack-meeting12:56
*** eharney has quit IRC12:59
*** yamamoto has joined #openstack-meeting12:59
*** Lucas_Gray has quit IRC13:07
*** JangwonLee_ has joined #openstack-meeting13:08
*** JangwonLee has quit IRC13:08
*** yamamoto has quit IRC13:11
*** Lucas_Gray has joined #openstack-meeting13:13
*** yamamoto has joined #openstack-meeting13:19
*** yamamoto has quit IRC13:19
*** ijw has joined #openstack-meeting13:21
*** Lucas_Gray has quit IRC13:21
*** yamamoto has joined #openstack-meeting13:23
*** Lucas_Gray has joined #openstack-meeting13:25
*** ijw has quit IRC13:25
*** lajoskatona has left #openstack-meeting13:26
*** lajoskatona has joined #openstack-meeting13:27
*** yamamoto has quit IRC13:30
*** Wryhder has joined #openstack-meeting13:37
*** Lucas_Gray has quit IRC13:38
*** Wryhder is now known as Lucas_Gray13:38
*** eharney has joined #openstack-meeting13:43
*** brinzhang has joined #openstack-meeting13:44
*** ociuhandu has joined #openstack-meeting13:44
*** yamamoto has joined #openstack-meeting13:50
*** dklyle has joined #openstack-meeting13:51
*** mlavalle has joined #openstack-meeting13:59
slaweq#startmeeting networking14:00
openstackMeeting started Tue Mar 10 14:00:28 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: networking)"14:00
openstackThe meeting name has been set to 'networking'14:00
*** Liang__ has quit IRC14:00
slaweq#topic Announcements14:01
*** openstack changes topic to "Announcements (Meeting topic: networking)"14:01
slaweqaccording to https://releases.openstack.org/ussuri/schedule.html next important dates are:14:01
slaweqFinal release for non-client libraries - week of March 30th,14:01
slaweqUssuri-3 (feature freeze) - week of April 6th14:02
slaweqso we have only few more weeks to finish implementation of some RFEs/BPs14:02
slaweqone more thing related to calendar14:03
slaweqaccording to the https://governance.openstack.org/election/ PTL nomination period starts on Mar 24th14:03
slaweqso if someone wants to run for neutron PTL for Victoria cycle, nominations should be send between Mar 24th and Mar 31st14:04
*** _mlavalle_2 has joined #openstack-meeting14:05
slaweqI want to continue serving as PTL to this great team so I will send my nomination too14:05
* _mlavalle_2 got diconnected for a few seconds14:06
slaweqand as last announcement, short reminder14:06
slaweqVictoria PTG planning etherpad: https://etherpad.openstack.org/p/neutron-victoria-ptg14:06
*** rh-jelabarre has joined #openstack-meeting14:06
slaweqplease add Your topics there - anything You want to discuss related to Neutron is fine14:06
*** mlavalle has quit IRC14:07
slaweqany other announcements from the team?14:07
*** takahashi-tsc has quit IRC14:07
bcafarelis it time yet to congratulate lajoskatona?14:07
slaweqbcafarel: almost, I hope that I will announce it on our next meeting :)14:08
bcafarelack, I will keep the confettis on hand :)14:09
slaweqbcafarel: :)14:09
slaweqif there is no other announcements, lets move on14:10
slaweq#topic Blueprints14:10
*** openstack changes topic to "Blueprints (Meeting topic: networking)"14:10
*** ianychoi has quit IRC14:10
slaweqlist of BPs for U-3 is at https://launchpad.net/neutron/+milestone/ussuri-314:10
slaweqI would like to clarify status of some of them now14:10
slaweqfirst of all https://blueprints.launchpad.net/neutron/+spec/neutron-ovn-merge14:11
*** ianychoi has joined #openstack-meeting14:11
slaweqI wanted to ask haleyb|away what we are still missing there to consider it as done14:11
slaweqbut haleyb|away is not here today so I will ask him offline about that14:11
slaweqnext one14:11
slaweq_mlavalle_2: will You propose patches for openstack sdk and OSC for that?14:12
slaweqas this is last remaining part IIRC, right?14:12
*** Lucas_Gray has quit IRC14:12
_mlavalle_2and api-def14:12
slaweqby api-def You mean documentation?14:12
slaweqso we are almost done with this one :)14:13
slaweqthx _mlavalle_214:13
_mlavalle_2tempest test merged last night14:13
slaweqyep, that I saw :)14:13
slaweqok, next one14:14
slaweqit's also almost done, there is SDK patch in progress14:14
slaweqand OSC patch is missing14:14
slaweqbut I found some issues with it when I was reviewing sdk patch14:14
slaweqdetails are in https://bugs.launchpad.net/neutron/+bug/186656014:15
openstackLaunchpad bug 1866560 in neutron "FIP Port forwarding description API extension don't work" [High,In progress] - Assigned to Slawek Kaplonski (slaweq)14:15
slaweqI hope it will be fixed soon14:15
_mlavalle_2is there a patch proposed yet?14:16
_mlavalle_2to keep it in my radar screen14:16
slaweqrelated to https://bugs.launchpad.net/neutron/+bug/1866560 ?14:16
openstackLaunchpad bug 1866560 in neutron "FIP Port forwarding description API extension don't work" [High,In progress] - Assigned to Slawek Kaplonski (slaweq)14:16
slaweqyes, there is https://review.opendev.org/#/c/711888/14:17
slaweqI need to rebase it on top of https://review.opendev.org/#/c/711602/14:17
slaweqand it also needs https://review.opendev.org/#/c/711887/14:18
*** yamamoto has quit IRC14:18
slaweqthx _mlavalle_2 for taking look into that :)14:18
slaweqnext one14:18
*** Lucas_Gray has joined #openstack-meeting14:18
slaweqnjohnston: what is current status of this one?14:18
slaweqdo You need any help with that maybe?14:18
njohnstonI have not made good progress14:19
njohnstonIf anyone is interested in helping out, please ping me14:20
njohnstonBut it is a lower priority for me at the moment14:20
slaweqok, I can help with that one this week14:20
slaweqnext one for today14:21
slaweqI recently sent email to _mlavalle_2 and haleyb|away about it14:21
_mlavalle_2I haven't made progress with it14:21
_mlavalle_2still in my plans14:22
slaweqhaleyb|away's opinion is that we should go with this spec on our own, without doing any RFC or something like that14:22
slaweqmy opinion is similar to be honest14:22
slaweq_mlavalle_2: if You want I can try to reach out to cloud-init maintainers this week to see what they think about our spec14:23
slaweqand then implementation on our side should be easy probably14:23
*** ijw has joined #openstack-meeting14:23
slaweqok, so I will do that14:23
slaweq_mlavalle_2: rubasov I will ping You when I will have anything new related to that14:24
slaweqand the last one on my list for today is: https://blueprints.launchpad.net/neutron/+spec/enginefacade-switch14:25
slaweqaccording to that I think we are not doing any progress on it since some time14:25
slaweqI know that there is this "check" patch https://review.opendev.org/#/c/545501/14:26
slaweqwhere all is red now14:26
slaweqI will try also to help with this one more during next few weeks14:26
ralonsohthere are is one patch still under review https://review.opendev.org/#/q/topic:bp/enginefacade-switch+(status:open)14:27
ralonsohthere is*14:27
slaweqif there is anyone else who would like to help, that would be great14:27
ralonsohsure, of course I can help with this14:27
_mlavalle_2I'll do my best to continue working on it14:27
slaweqthx _mlavalle_2 and ralonsoh14:27
*** ijw has quit IRC14:28
slaweqok, and that's all Blueprints on the list for today14:28
slaweqanyone wants to add/ask something about blueprints?14:28
slaweqif not, lets move on14:29
slaweqnext topic for today14:29
slaweq#topic Community goals14:29
*** openstack changes topic to "Community goals (Meeting topic: networking)"14:29
slaweqProject Specific PTL and Contributor Documentation - I think that patch https://review.opendev.org/#/c/707486/ is ready for review now14:30
*** nicolasbock has quit IRC14:30
slaweqso please add it to Your review queue :)14:30
slaweqand thanks in advance for any comments there14:31
slaweqregarding old goal "Support IPv6-Only Deployments"14:31
slaweqI just rebased neutron-tempest-plugin patch https://review.opendev.org/686043 and will try to continue work on it,14:31
*** links has quit IRC14:32
slaweqon the list of opened patches: https://review.opendev.org/#/q/topic:ipv6-only-deployment-and-testing+(status:open+OR+status:merged)+(project:%255Eopenstack/neutron.*+OR+project:%255Eopenstack/networking-.*) there are also some patchrs for networking-bagpipe/bgpvpn and networking-odl14:32
slaweqI think that would be good if maintainers of those projects could take a look at them14:32
bcafarelI thought we had nothing to do mostly for that ipv6 goal? added that patch to review queue14:32
*** Luzi has quit IRC14:32
slaweqthx bcafarel14:32
slaweqand that's all regarding community goals for Ussuri cycle (and olders)14:33
lajoskatonaslwaeq: I check bagpipe/bgpvpn and odl to see the status14:33
slaweqanything else You want to add about community goals?14:33
lajoskatonaslaweq -^14:33
slaweqthx a lot lajoskatona :)14:33
slaweqok, lets move on14:35
slaweqnext topic14:35
slaweq#topic Bugs14:35
*** openstack changes topic to "Bugs (Meeting topic: networking)"14:35
slaweqrubasov was on bug deputy. Summary http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013177.html14:35
slaweqrubasov: anything You want to hightligh from there?14:36
rubasovall bugs have assignees14:36
rubasovor attempted fixes14:36
*** puffc has joined #openstack-meeting14:37
rubasovthis one is incomplete, waiting for info from the reporter, but maybe it rings a bell for somebody: https://bugs.launchpad.net/neutron/+bug/186644514:37
openstackLaunchpad bug 1866445 in neutron "br-int bridge in one compute can't learn MAC addresses of VMs in other compute nodes" [Undecided,Incomplete]14:37
ralonsohbut should this happen? I don't think so14:38
*** jamesmcarthur has joined #openstack-meeting14:39
rubasovI think it shouldn't14:39
ralonsohbr-int can learn another host MAC address14:39
slaweqralonsoh: exactly, the only "learning" I'm aware of is with l2pop enabled14:39
*** brinzhang has quit IRC14:39
slaweqso this seems for me more like proposal of new RFE to add such "learning" mechanism, but is that even possible somehow with ovs?14:40
ralonsohI think the problem they are hitting is, maybe, that br-int is not learning the VM port addresses14:42
rubasovI'm not sure I follow, ovs NORMAL action should learn source MACs, shouldn't it?14:42
ralonsohyes if they are using hybrid firewall14:43
ralonsohthat's the point14:43
ralonsohbr-int should not learn other host MACs14:43
ralonsohbut maybe the internal macs are not in the MAC table and the broadcast traffic is excesive14:43
rubasovI mean we clearly don't have enough info to reproduce the problem yet, I was just fishing if somebody have seen a similar problem before to ask better questions from the reporter14:45
slaweqthx rubasov14:45
slaweqand thank You for the summary email14:46
*** jamesmcarthur has quit IRC14:46
slaweqour bug deputy this week is bcafarel so we are still in good hands14:46
*** ijw has joined #openstack-meeting14:46
slaweqand next week is my turn14:46
bcafarelso we should also be fine!14:46
slaweqso I will make sure that I'm aware of it :P14:46
slaweqany other bugs You want to discuss today?14:47
slaweqok, I guess that this means "no"14:48
slaweqnext topic should be "Networking OVN and ML2+OVS+DVR Convergence"14:48
slaweqbut as there is no Lucas, Maciej and haleyb|away today, I think we can skip it14:48
slaweqso next topic is14:49
openstackRemoving item from minutes: #topic Bugs14:49
slaweq#topic neutron-lib14:49
*** openstack changes topic to "neutron-lib (Meeting topic: networking)"14:49
slaweqI just wanted to say that new version of neutron-lib was just released few minutes ago14:50
slaweqit's version 2.2.014:50
slaweqanything else regarding neutron-lib?14:51
*** armax has joined #openstack-meeting14:51
slaweqif not, lets move on to the last topic14:52
slaweq#topic On Demand Agenda14:52
*** openstack changes topic to "On Demand Agenda (Meeting topic: networking)"14:52
slaweqralonsoh: You have one topic there14:52
ralonsohlet me check...14:52
ralonsohdo you have the link?14:53
ralonsohahh  yes14:53
openstackLaunchpad bug 1476527 in neutron "[RFE] Add common classifier resource" [Wishlist,Triaged] - Assigned to Igor D.C. (igordcard)14:53
ralonsohso the question is: are we going to merge the n-lib patch?14:53
ralonsohand the DB changes14:54
ralonsohnobody, so far, is going to continue with the implementation14:54
ralonsohin OVS, OVN or LB14:54
slaweqIMHO no if we don't have anyone who will propose backend implementation for it14:54
*** jamesmcarthur has joined #openstack-meeting14:54
bcafarelralonsoh: is it https://review.opendev.org/#/c/670049/ ?14:55
slaweqbut I would love to see what others think about it14:55
ralonsohthis is the n-lib patch, the first one14:55
ralonsohand the DB change is here14:55
ralonsoh#link https://review.opendev.org/#/c/670050/14:55
bcafarelI fear we do not have anyone working on it :(14:56
lajoskatonaralonsoh, slaweq: as I see a lot of work is in these patches, and adds possibility to later improvements14:56
*** jamesmcarthur has quit IRC14:56
*** jamesmcarthur has joined #openstack-meeting14:56
ralonsohlajoskatona, I was working with David in Intel when they started this spec, 3 years ago14:57
ralonsohthere is a lot of work there14:57
ralonsohbut there is not going to be any implementation14:57
lajoskatonaWhat I don't see why it is better to integrate it to neutron instead of using x/neutron-classifier14:57
*** ayoung has joined #openstack-meeting14:57
*** yamamoto has joined #openstack-meeting14:57
* _mlavalle_2 has to leave. have a great day14:57
*** _mlavalle_2 has quit IRC14:58
ralonsohbecause the DB changes were needed14:58
slaweqlajoskatona: I remember that there were some issues with db upgrade and dependencies between neutron and x/neutron-classifier projects14:58
ralonsohto make any QoS policy dependant on a CG14:58
lajoskatonaok, I see14:58
slaweqok, we are running out of time now14:59
slaweqwe can get back to it on next meeting14:59
bcafareland maybe find volunteers by then :)14:59
slaweqor ralonsoh You can send an email about that and we can continue discussion there14:59
slaweqthx for attending14:59
ralonsohof course14:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"14:59
openstackMeeting ended Tue Mar 10 14:59:55 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-03-10-14.00.html14:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-03-10-14.00.txt14:59
openstackLog:            http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-03-10-14.00.log.html15:00
ralonsoh#startmeeting neutron_qos15:00
openstackMeeting started Tue Mar 10 15:00:07 2020 UTC and is due to finish in 60 minutes.  The chair is ralonsoh. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: neutron_qos)"15:00
openstackThe meeting name has been set to 'neutron_qos'15:00
ralonsohyou can stay if you want here15:00
ralonsohthe agenda is very light today15:00
*** davidsha has joined #openstack-meeting15:01
ralonsohso let's start15:01
ralonsoh#topic RFEs15:01
*** openstack changes topic to "RFEs (Meeting topic: neutron_qos)"15:01
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/147652715:01
openstackLaunchpad bug 1476527 in neutron "[RFE] Add common classifier resource" [Wishlist,Triaged] - Assigned to Igor D.C. (igordcard)15:01
ralonsohI'll send a mail to the ML to ask for developers15:01
ralonsohfor now, this RFE is frozen15:02
ralonsohnext one15:02
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/185861015:02
openstackLaunchpad bug 1858610 in neutron "[RFE] Qos policy not supporting sharing bandwidth between several nics of the same vm." [Undecided,New]15:02
ralonsohwe agreed in the last drivers meeting to wait for a SPEC or POC15:03
ralonsohthis feature is quite complex so a PoC could help to decide if the architecture change is correct15:03
ralonsohthere are no more RFEs in the list15:04
*** yamamoto has quit IRC15:04
ralonsohdo you have something else to add here?15:04
ralonsohplease, go on15:04
ralonsohdavidsha, do you have something to add here?15:06
davidshaI think they only way this could make sense to me is if it's applied at the instance, so it would need a lot of collaboration with Nova15:06
ralonsohdavidsha, you mean to implement the QoS on the instance?15:06
davidshafor this particular use case yes15:07
ralonsohbut this is beyond the orchestrator15:07
davidshaOtherwise you need a QoS rule thats an aggregate of specific ports15:07
ralonsohthe orchestrator should not interfere on the instance15:07
ralonsohyes, that's the point15:07
ralonsohhow this qos could be implemented in the backend15:07
ralonsohin LB they are proposing to use the IFB block15:07
ralonsohthe same as I used (with no success) in the min BW implementation for LB15:08
ralonsohbut as commented in the last drivers meeting, we'll wait for a spec/poc15:08
davidshaSo you need to change the typical relationship between rule -> policy -> port to rule wrapping multiple ports?15:09
ralonsohyes, that's another topic15:09
ralonsohmore related to the server: how are you going to handle this new rule15:09
ralonsohjust for the VM ports? for all the ports related to this rule?15:09
ralonsohnothing has been decided yet15:09
davidshaya, I''m thinking API at the moment15:09
*** Lucas_Gray has quit IRC15:10
ralonsohok, let's move to the next topic15:11
ralonsoh#topic Bugs15:11
*** openstack changes topic to "Bugs (Meeting topic: neutron_qos)"15:11
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/186603915:11
openstackLaunchpad bug 1866039 in neutron "[OVN] QoS gives different bandwidth limit measures than ml2/ovs" [High,In progress] - Assigned to Maciej Jozefczyk (maciej.jozefczyk)15:11
ralonsohI'm looking for the patch...15:11
ralonsoh#link https://review.opendev.org/#/c/711048/15:11
ralonsohthere is a small transient measuring the BW15:12
ralonsohthe BW at the beginning is higher15:12
ralonsohnot to avoid but to reduce the impact of this transient on the average BW15:12
ralonsohthe file size to transfer will be proportional to the BW speed15:12
ralonsohthis way, the transmission time will be always the same, 10 secs aprox15:13
ralonsohthis will improve all backends15:13
ralonsoh(will improve all backends tests)15:13
ralonsohplease, review the patch15:13
ralonsohnext one15:14
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/186463015:14
openstackLaunchpad bug 1864630 in neutron "Hard Reboot VM with multiple port lost QoS " [Undecided,In progress] - Assigned to Nguyen Thanh Cong (congnt95)15:14
*** mattw4 has joined #openstack-meeting15:14
ralonsohI've +W the patch 1 hour ago15:14
ralonsoh#link https://review.opendev.org/#/c/709687/15:14
*** Lucas_Gray has joined #openstack-meeting15:14
ralonsohgood catch, thanks for reporting this bug15:14
ralonsohand now we have a series of bugs, all of them related15:15
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/186398715:15
openstackLaunchpad bug 1863987 in neutron "[OVN] Remove dependency on port_object" [Medium,In progress] - Assigned to zhanghao (zhanghao2)15:15
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/186385215:15
openstackLaunchpad bug 1863852 in neutron "[OVN]Could not support more than one qos rule in one policy" [Medium,In progress] - Assigned to Taoyunxiang (taoyunxiang)15:15
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/186289315:15
openstackLaunchpad bug 1862893 in neutron "[OVN]Updating a QoS policy for a port will cause a KeyEerror" [Low,In progress] - Assigned to zhanghao (zhanghao2)15:15
ralonsohthe problem of those patches are that the patches are disconnected among them15:16
ralonsohwe need to refactor the OVN QoS driver15:16
ralonsohto handle the QoS as a driver extension15:17
*** ociuhandu has quit IRC15:17
ralonsohwe can't use the ML2 agent extension model because this is NOT an agent15:17
ralonsohI'm still working on this: https://review.opendev.org/#/c/711317/15:17
*** ociuhandu has joined #openstack-meeting15:17
ralonsohthis patch will move the QoS handling to this new extension15:18
ralonsohthe main problem I'm facing now is how to handle the QoS policy update15:18
ralonsohin the agents, the policies and the rules where stored in a in-memory mapping15:19
ralonsohbut I'm very reluctant to do the same in the server15:19
davidshaThis is the Neutron server you're talking about?15:19
ralonsohin HA we can't guarantee this mapping will be shared between instances15:19
davidshaNeutron server has access to the DB, could you not pull from that?15:20
ralonsohNeutron server, sorry, in opposition to Neutron agent (sriov, LB, OVS)15:20
ralonsohthat's the point15:20
ralonsohwe receive this call from the qos_plugin15:20
ralonsohself.driver_manager.call(qos_consts.UPDATE_POLICY, context, policy)15:20
ralonsohat this point, the policy (and the rules, this method is also called when a policy rule is updated)15:21
ralonsohstored in the DB is the new one15:21
ralonsohand we lost any track of the previous one...15:21
ralonsoh--> how can we "delete" the OVN QoS registers if we don't have the information of the old policy and rules?15:22
davidshaIy's the delta you need to have15:22
ralonsohand I'm squeezing my brain to find a solution15:22
*** ociuhandu has quit IRC15:23
davidshaIs there some kind of history for modifications to DB objects that we can reference?15:23
ralonsohanyway, I'll find something15:24
davidshaYa, OvN may need a mirror of the QoS tables.15:24
ralonsohI don't know if I can store this info in the OVN DB15:25
ralonsohno problem, I'll find something there15:25
ralonsohwe still have another bug in the list15:25
ralonsoh#link https://bugs.launchpad.net/networking-sfc/+bug/185317115:25
openstackLaunchpad bug 1853171 in neutron "Deprecate and remove any "ofctl" code in Neutron and related projects " [Medium,In progress]15:25
ralonsohdavidsha, do you have any update?15:26
davidshaYes, I've had a go at the refactor, but I've run out of time: https://review.opendev.org/#/c/711949/15:26
ralonsohno problem15:27
ralonsohonce you stop working on this one, ping me15:27
davidshaThere seems to be some issue with ct_state and ct_mark, I'm not sure am I using it wrong, but the documentation implies it should work.15:27
ralonsohI'll try to reproduce it locally15:27
davidshaThe dynamic flow creation code in rules.py is only half migrated as well.15:28
ralonsohmaybe this is going to be more complex than expected...15:29
davidshaNXActions should be able to do conjunctions from what I read, I just haven't gotten that far to really know.15:29
davidshaIt shouldn't be too bad from here, my only concern is the ct_mark and ct_state issue, there could be a bug in Ryu and OS-Ken15:30
davidshaIt expects to recieve ints for those values, but the documentation says strings15:30
ralonsohdo you know what part of the osken code is handling this?15:31
davidshaI was looking at the code, I think it was in the 1.3 parser. the ct_state and ct_mark are part of nx_actions I believe?15:32
*** armax has quit IRC15:33
davidshaYa it's listed as an int here15:33
ralonsohbut you said before strings15:34
ralonsohCT_MARK_* are string15:35
ralonsohhmmm maybe there you have your problem!15:35
davidshaSo the constants need to be changed from strings to ints15:36
davidshato the corresponding ints*15:36
ralonsohmaybe heheheh15:36
davidshaI had to make a similar change for the ethertypes15:36
davidshaThey were being passed in as strings but needed to be ints @.@15:37
ralonsohping me once you have next version15:38
davidshaThats my last version I'm afraid :/15:39
ralonsohahhhh ok15:39
ralonsohI understand15:39
davidshanp, sorry I didn't get it all the way :P15:39
ralonsohok, thanks a lot davidsha15:40
ralonsohdo we have something else in the bug section?15:41
ralonsoh#topic Open Discussion15:41
*** openstack changes topic to "Open Discussion (Meeting topic: neutron_qos)"15:41
ralonsohI have nothing in the agenda15:41
ralonsohsomething to add here?15:41
*** TrevorV has joined #openstack-meeting15:41
ralonsohthank you all and see you in two weeks15:42
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:42
openstackMeeting ended Tue Mar 10 15:42:10 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:42
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_qos/2020/neutron_qos.2020-03-10-15.00.html15:42
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_qos/2020/neutron_qos.2020-03-10-15.00.txt15:42
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_qos/2020/neutron_qos.2020-03-10-15.00.log.html15:42
*** gyee has joined #openstack-meeting15:43
*** puffc has quit IRC15:44
*** macz_ has joined #openstack-meeting15:46
*** macz_ has quit IRC15:47
*** macz_ has joined #openstack-meeting15:48
*** ociuhandu has joined #openstack-meeting15:48
*** lajoskatona has left #openstack-meeting16:00
*** lajoskatona has quit IRC16:00
*** vishalmanchanda has quit IRC16:05
*** yamamoto has joined #openstack-meeting16:10
*** ianychoi has quit IRC16:11
*** ianychoi has joined #openstack-meeting16:12
*** yamamoto has quit IRC16:15
*** trident has quit IRC16:34
*** trident has joined #openstack-meeting16:36
*** rubasov has quit IRC16:40
*** yamamoto has joined #openstack-meeting16:47
*** ianychoi has quit IRC16:47
*** ianychoi has joined #openstack-meeting16:48
*** jmasud has quit IRC16:51
*** davidsha has quit IRC16:51
*** yamamoto has quit IRC16:52
*** armax has joined #openstack-meeting16:55
*** tesseract has quit IRC17:07
*** rpittau is now known as rpittau|afk17:10
*** ociuhandu has quit IRC17:10
*** jamesmcarthur has quit IRC17:20
*** witek has quit IRC17:24
*** ianychoi has quit IRC17:26
*** mattw4 has quit IRC17:26
*** mattw4 has joined #openstack-meeting17:27
*** jamesmcarthur has joined #openstack-meeting17:27
*** ianychoi has joined #openstack-meeting17:27
*** e0ne has quit IRC17:42
*** jamesmcarthur has quit IRC17:43
*** jmasud has joined #openstack-meeting17:53
*** jamesmcarthur has joined #openstack-meeting17:56
*** jmasud has quit IRC17:57
*** Lucas_Gray has quit IRC18:03
*** ijw_ has joined #openstack-meeting18:05
*** ijw has quit IRC18:07
*** trident has quit IRC18:08
*** ianychoi has quit IRC18:10
*** armax has quit IRC18:15
*** ianychoi has joined #openstack-meeting18:17
*** trident has joined #openstack-meeting18:18
*** diablo_rojo has joined #openstack-meeting18:28
*** Shrews has joined #openstack-meeting18:30
*** jmasud has joined #openstack-meeting18:34
*** AJaeger has joined #openstack-meeting18:48
*** brinzhang has joined #openstack-meeting18:51
*** e0ne has joined #openstack-meeting18:53
*** brinzhang_ has quit IRC18:54
clarkbanyone else here for the infra meeting? we'll get started shortly18:59
* corvus is here with dst-compatible sandwich19:00
clarkbI just finished my sandwich19:00
fungimy sandwiches are decidedly non-dst-compatible19:00
fungi(and also consumed hours ago_19:01
openstackclarkb: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee'19:01
clarkbbah I did that last week too19:01
clarkb#startmeeting infra19:01
openstackMeeting started Tue Mar 10 19:01:11 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
*** openstack changes topic to " (Meeting topic: infra)"19:01
* mordred has fewer sandwiches than he should19:01
openstackThe meeting name has been set to 'infra'19:01
clarkb#link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006608.html Our Agenda19:01
clarkb#topic Announcements19:01
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:01
clarkbFirst up a friendly reminder that for many of us in north america DST switch has happened. Careful with your meeting schedule :)19:01
*** ijw has joined #openstack-meeting19:01
clarkbEurope and others happen at the end of the month so we'll get round two with a different set of people19:01
clarkbThis meeting is at 1900UTC Tuesdays regardless of DST or timezone19:02
clarkb#link http://lists.openstack.org/pipermail/foundation/2020-March/002852.html OSF email on 2020 events19:02
clarkbSecond we've got an email from the Foundation with an update on 2020 events and what the current status is.19:02
clarkbIf you've got feedback for that I know mark and others are more than happy to receive it19:02
clarkbit went to the foundation list so I wanted to make sure everyone saw it19:03
clarkb#topic Actions from last meeting19:04
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)"19:04
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-03-03-19.02.txt minutes from last meeting19:04
clarkbThere were no actions recorded19:04
clarkb#topic Specs approval19:04
*** openstack changes topic to "Specs approval (Meeting topic: infra)"19:04
*** ijw_ has quit IRC19:05
clarkbI approved the python cleanup spec and the website activity stats spec. I think I've seen progress on both since then (I pushed a change today to add a job that runs goaccess and ianw landed the glean reparenting to python3 -m venv in dib)19:05
clarkbPlease add those review topics to your review lists19:05
clarkbtopic:website-stats topic:cleanup-test-image-python19:06
clarkbWe've also got the xwiki spec19:06
clarkb#link https://review.opendev.org/#/c/710057/ xwiki for wikis19:06
clarkbzbr: thank you for the review on that one.19:07
clarkbfrickler and corvus I think you had good feedback on IRC, would be great if you could double check that your concerns were addressed (or not) in the spec itself19:07
*** ralonsoh has quit IRC19:07
*** eharney has quit IRC19:08
clarkbAny other input on the specs work that has happened recently?19:08
*** jmasud has quit IRC19:09
clarkb#topic Priority Efforts19:09
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:09
*** igordc has joined #openstack-meeting19:09
clarkb#topic OpenDev19:09
*** openstack changes topic to "OpenDev (Meeting topic: infra)"19:09
clarkb#link https://review.opendev.org/#/c/710020/ Split OpenDev out of OpenStack governance.19:10
clarkbThat change seems to be getting proper review this time around (which is good)19:10
clarkbit also appears that we can continue to roll forward and make progress if I'm parsing the feedback there19:10
clarkb#link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006603.html OpenDev as OSF pilot project19:10
clarkbI'll try to see if jbryce has time to follow up with our responses on that mailing list thread too19:10
clarkblast week we discovered that we had missed replication events on at least one gitea backend as well as a different backend OOMing and causing some jobs that don't use the local git cache to fail19:11
AJaegerInfra Manual has been updated and is now the "OpenDev Manual", a few changes still up for review19:11
clarkbAJaeger: thanks for the reminder /me needs to make sure he is caught up on those change reviews19:12
clarkbon the missed replication events problem mordred and I brainstormed a bit and we think we can force Gerrit to retry by always stopping the ssh daemon in the gitea setup first19:12
clarkbthat way replication either actually succeeds or will have tcp errors. This is a good thing because Gerrit replication plugin will retry forever if it notices an error replicating a ref19:13
mordredWIP patch for that: https://review.opendev.org/#/c/711130/19:13
mordredwhich is incomplete19:13
clarkbfor the second thing (OOM) I think we should begin thinking about deploying bigger servers and/or more servers. My concern with more servers is that since we pin by IP any busiy IP (NAT perhaps) can still overload a single server19:14
mordredclarkb: while we should certainly avoid OOMing, should we also be trying to make those jobs use the git cache more better?19:15
clarkbmordred: yes jrosser said they would work on that aspect of it19:15
fungiit's not necessarily our jobs which are the bulk of the cause though19:15
fungiremember that our nodes don't go through nat to reach our gitea farm19:16
clarkbfungi: but we do often start many jobs all at once that can by chance hash to the same backend node19:17
fungiso in theory if we have a big batch of jobs kick off which all clone nova, we should see that impact spread somewhat evenly across the pool already19:17
clarkb(also these jobs were cloning nova which is our biggest repo iirc)19:17
fungithe times we've seen this go sideways, it's generally been one backend hit super hard and little or no noticeable spike on the other backends around teh same timeframe19:17
fungiwe talked about some options for using weighting in haproxy based on system resource health checks too, though that was kinda hand-wavey on the details19:18
corvusthe other approach would be to change the lb algorithm to least busy, but the issue there is that backends are sometimes slightly out of sync?  is it the case the only resolution there is shared storage?19:18
clarkbcorvus: ya I think shared storage solves that problem properly. Its possible that least busy would mostly work now that our gitea backends are largely in sync with each other and packing on a similar schedule19:19
fungieven shared storage doesn't necessarily fix it if every backend's view of that storage can be slightly out of sync19:19
clarkbthe old cgit setup used least busy and didn't have the problems with git clients moving around. I think possibly because the git repos were similar enough19:19
corvusclarkb: oh, what changed to get them mostly in sync?19:19
clarkbcorvus: when we first spun up gitea we didn't do any explicit packing so they were somewhat randomly packed based on usage (git will decide on its own to pack things periodically iirc)19:20
corvusfungi: i don't think that is possible with cephfs?19:20
fungiright, using an appropriate shared storage can solve that19:20
fungi(not all shared storage has those properties)19:20
clarkbthinking out loud here, it may be worth flipping the switch back to least busy. Centos7 git in particular was the one unhappy with slightly different backend state iirc and its possible that as centos7 is used less and we are keeping things more in sync we can largely avoid the problem?19:21
clarkbmaybe try that for a bit before deciding to make more/bigger servers?19:21
corvussounds like it's worth it since the state may have improved19:22
fungidefinitely worth a try, sure19:22
fungii expect we can fairly quickly correlate any new failures resulting from that19:22
clarkbfungi: ya and we can run git updates and stuff locally as well to look for issues19:23
fungi(presumably it'll manifest as 404 or similar errors fetching specific objects)19:23
*** yamamoto has joined #openstack-meeting19:23
*** mugsie has quit IRC19:23
clarkb#link https://review.opendev.org/#/q/project:openstack/infra-manual+topic:opendev+status:open Infra manual opendev updates19:23
clarkbAJaeger and I (mostly AJaeger) have been reworking the infra manual to make it more opendev and less openstack19:24
clarkbhopefully this will be more welcoming to new users19:24
clarkbif you've got time reviews on those changes are much appreciated19:24
clarkbAnything else on OpenDev? or should we move on?19:24
clarkb#topic Update Config Management19:26
*** openstack changes topic to "Update Config Management (Meeting topic: infra)"19:26
*** mugsie has joined #openstack-meeting19:26
clarkb#link https://review.opendev.org/#/q/topic:nodepool-legacy+status:open19:26
clarkbianw has a stack there that will deploy nodepool builders with containers19:26
clarkbThis worksaround problems with the builder itself not having things it needs like newer decompression tools because we can get them into the container image19:27
ianwyeah just the main change really left, and then we can test19:27
*** yamamoto has quit IRC19:28
clarkbIt doesn't totally solve the problems we've had as I'm not sure any single container image could have all the right combo of tools necessary to build all the things, but its closer than where we were before19:28
ianwthis makes things better for fedora, but worse for suse, as this image doesn't have the tools to build that19:28
clarkbianw: we could in theory have an image that did do both or use two different images possibly?19:29
corvuswhat do we need for suse?19:29
ianwwe have discussed longer term moving the minimal builds away from host tools, in the mean time we can have a hetrogenous situation19:29
clarkbcorvus: `zypper` the package manager19:29
corvusah that19:29
fungiwhich was temporarily dropped from debian, and thus ubuntu19:30
corvusis it back in debian?19:30
*** jamesmcarthur has quit IRC19:30
fungiit's been reintroduced, but not long enough ago to appear in ubuntu/bionic19:30
ianwoh interesting, because the container isn't ubuntu19:30
mordredcould we potentially add it via bionic-backports?19:30
clarkbfungi: our nodepool-builder images are debian based though so may be able to pull that in19:30
fungi#link https://packages.debian.org/zypper19:31
fungiit made it back in time for the buster release19:31
mordredcool. then we should be able to just add it19:31
ianwthat's somewhat good news, i can look into that avenue19:31
ianwit's still a fragile situation, but if it works, it works19:31
corvusalso we had the 'containerfile' idea for images19:32
corvusstalled at https://review.opendev.org/70008319:32
clarkbright that is the longer term solution taht should fix this more directly for dib itself19:32
corvusi don't actually understand dib, so it's going to take me a while to figure out why those tests are failing19:33
mordredianw: I have verified that zypper is installable in python-base19:33
fungibasically accepting that no one platform will every have all the right tools available to build all the kinds of images we want to build19:33
corvusif anyone else wants to take a peek at that and see if they have any hints, that might help move things along19:33
ianwfor step 0 though, i think getting a working image out of the extant container, and seeing how we manage it's lifespan etc will be the interesting bits for the very near future19:34
clarkbianw: ya I think it will also be good to have someone start using that image too as we've had zuul users be confused about how to get nodepool builder running19:35
clarkbmordred: any updates on gerrit and jeepyb and docker/podman?19:35
mordrednope. last week got consumed - I'm about to turn my attention back to that19:36
*** armax has joined #openstack-meeting19:36
clarkbAnything else before we move on to the next topic?19:37
clarkb#topic General Topics19:38
*** openstack changes topic to "General Topics (Meeting topic: infra)"19:38
clarkbstatic.openstack.org is basically gone at this point right? ianw has shut it down to shake out any errors (and we found one for zuul-ci? did we check starlingx too?)19:38
ianwthe old host is shutdown, along with files0219:38
fungidocs.starlingx.io still seems to be working19:39
ianwi haven't heard anything about missing sites ...19:39
clarkbfungi: thanks for checking19:39
clarkbI mentioned starglinx because it has a similar setup to zuul-ci.org19:39
clarkbthough dns for it is managed completely differently and that was where we had the miss I think19:39
fungi(the main starlingx.io site isn't hosted by us, the osf takes care of it directly)19:40
ianwyeah, i updated that in RAX IIRC19:40
clarkbAnything else we need to do before cleaning up the old resources? do we want to keep any of the old cinder volumes around after deleting servers (as a psuedo backup)19:40
fungii already deleted all but one of them before a scheduled hypervisor host maintenance19:40
ianwhttps://review.opendev.org/#/c/709639/ would want one more review to remove the old configs19:41
fungipvmove and lvreduce and so on19:41
clarkb#link https://review.opendev.org/#/c/709639/ Clean up old static and files servers19:41
fungiso at this point the old static.o.o cinder volumes are down to just main0119:41
*** brinzhang_ has joined #openstack-meeting19:41
clarkbfungi: gotcha, I suppose we can hang onto that volume for a bit longer once the servers are deleted if we feel that is worthwhile19:42
clarkbthe servers themselves shouldn't have any other data on them that matters19:42
ianwnot much left in the story19:42
clarkbanything else to bring up on this subject?19:43
fungii did notice, in looking over our mirroring configs, that we're mirroring tarballs.openstack.org and not tarballs.opendev.org19:43
clarkbfungi: in afs?19:44
clarkboh no the in region mirrors19:44
fungiapache proxy cache, but yeah i suppose that's now obsolete19:44
fungiif we serve them from afs anyway19:44
clarkbI didn't realize we did that for tarballs at all. But ya could serve it "direclty" or update the cache pointer19:44
fungi(though maybe we need to plumb that in the vhost config to make it available)19:45
*** brinzhang has quit IRC19:45
ianwwant me to add a task to look into it?19:46
clarkbianw: ++19:46
clarkbOk as a time check we have less than 15 minutes and a few more items to get through so we'll move on19:46
fungiit doesn't necessarily have to get lumped into the static.o.o move, but can't hurt i guess19:46
fungiit's sort of cleanup19:46
fungicool, next topic19:47
clarkbnext up I wanted to remind everyone we've started to reorg our IRC channels.19:47
clarkbWe've dropped openstacky things from #opendev. But before we actually split the channels we need to communicate the switch19:47
clarkb#link https://review.opendev.org/#/c/711106/1 if we get quorum on that change we can announce the migration then start holding people to it after an agreed on date.19:47
clarkband before we send out mass comms some agreement on the change at ^ would be great19:48
clarkbits a quick review. Mostly about sanity checking the division19:48
clarkband if we get agreement on that I can send out emails to the various places explaining that we'd like to be more friendly to non openstack users and push more of the opendev discussion into #opendev19:48
clarkbthen actually make that switch on #date19:48
clarkbNext up: FortNebula is no more. Long live Open Edge cloud19:49
clarkbeffectively what we've done is s/FortNebula/Open Edge/ donnyd is still our contact there19:49
clarkbI mentioned I could remove the fn grafana dashboard, but have not done that yet /me makes note on todo list for that19:49
fungithough he rebuilt the environment from scratch, new and improved (hopefully)19:50
fungichanged up various aspects of storage there19:50
clarkbThis is largely just a heads up so no one is confused about where thsi new cloud has come from19:50
fungiand the name change signals a shift in funding for it, or something like that19:50
clarkbAnd that takes us to project rename planning19:51
clarkbwe've got a devstack plugin that the cinder team wants to adopt and move from x/ to openstack/ and the openstack-infra/infra-manual repo should ideally be opendev/infra-manual19:51
clarkbmordred: ^ I think I'd somewhat defer to your thoughts on scheduling this since you are also trying to make changing in the gerrit setup19:52
fungii think it's currently openstack/infra-manual actually, not openstack-infra/infra-manual19:52
clarkbfungi: correct (my bad)19:52
fungibasically we chose wrong (or failed to predict accurately) when deciding which namespace to move it into during the mass migration19:52
smcginnisHow far away are we from being able to do a gerrit upgrade? (while we're near the subject)19:53
fungiso i'm viewing that one more as fixing a misstep from the initial migration19:53
donnydclarkb: fungi you are correct - Open Edge is likely to be around for quite a lot longer than FN was going to be able to be sustained for19:53
clarkbsmcginnis: mordred ran into an unexpected thing that needs accomodating so I don't think we have a good idea yet19:53
mordredwell - step one is just doing a restart on the same version but with new deployment19:53
smcginnisclarkb: OK, thanks.19:53
fungithanks again donnyd!!! it's been a huge help (and so have you)19:53
mordredclarkb: I don't expect manage-projects to take more than a day as long as I can actually work on it19:54
clarkbmordred: ok19:54
clarkblooking at ussuri scheduling we are kind of right in the middle of all the fun https://releases.openstack.org/ussuri/schedule.html19:54
clarkbnext week doesn't look bad but then after that we may have to wait until mid april?19:54
clarkb(also I'm supposed to be driving to arizona a week from friday, but unsure if those plans are still in play)19:55
mordredwhy don't we schedule for next week - make that a target for also restarting for the container19:55
mordredI think that's a high doable goal gerrit-wise19:55
clarkbmordred: I'm fine with that, though depending on how things go I may or may not be driving a car and not much help19:55
mordred(to be clear - this isn't upgrade - is jst container)19:55
mordredclarkb: totes19:55
clarkbif I'm not driving the car I'm happy to help19:56
fungiyeah, i expect to be on hand, so any time around then wfm19:56
clarkbmordred: do you want to say friday the 20th?19:56
*** eharney has joined #openstack-meeting19:56
mordredclarkb: seems reasonable19:56
clarkbany suggestions on a time frame (I can send out email that it is our intent to do that soon)19:57
clarkbfungi: mordred: what works best for you since it sounds like you may be most around (judging on current volunteering)19:57
corvusi'd like to help, but i'm not sure i can commit to the 20th right now, will let mordred know asap19:57
fungifridays work well for me more generally, also any time which works for mordred should be fine for me as well since i live an hour in his future anyway19:58
clarkbwell we are just about at time. Maybe we can sort out a timeframe between now and this friday and I'll announce stuff19:58
mordredyeah - any time on the 20th works for me too19:58
clarkbI'll open the floor for any last quick things19:58
clarkb#topic Open Discussion19:59
*** openstack changes topic to "Open Discussion (Meeting topic: infra)"19:59
clarkbSounds like that may be it. Thank you everyone and we'll see you here next week19:59
clarkbfeel free to continue discussion in #openstack-infra or on the mailing list20:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"20:00
openstackMeeting ended Tue Mar 10 20:00:11 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-03-10-19.01.html20:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-03-10-19.01.txt20:00
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-03-10-19.01.log.html20:00
*** AJaeger has left #openstack-meeting20:00
fungithanks, clarkb!20:00
*** slaweq has quit IRC20:03
*** jamesmcarthur has joined #openstack-meeting20:17
*** ianychoi has quit IRC20:17
*** e0ne has quit IRC20:17
*** ianychoi has joined #openstack-meeting20:18
*** e0ne has joined #openstack-meeting20:19
*** e0ne_ has joined #openstack-meeting20:21
*** e0ne has quit IRC20:21
*** brinzhang has joined #openstack-meeting20:26
*** rh-jelabarre has quit IRC20:28
*** brinzhang_ has quit IRC20:29
*** brinzhang_ has joined #openstack-meeting20:37
*** brinzhang has quit IRC20:40
*** mattw4 has quit IRC20:40
*** rcernin has quit IRC20:53
*** Lucas_Gray has joined #openstack-meeting20:54
*** e0ne_ has quit IRC20:59
*** e0ne has joined #openstack-meeting21:18
*** e0ne has quit IRC21:21
*** brinzhang has joined #openstack-meeting21:22
*** ykatabam has quit IRC21:25
*** brinzhang_ has quit IRC21:26
*** jamesmcarthur has quit IRC21:28
*** jamesmcarthur has joined #openstack-meeting21:28
*** ociuhandu has joined #openstack-meeting21:37
*** jamesmcarthur has quit IRC21:44
*** Lucas_Gray has quit IRC21:45
*** jamesmcarthur has joined #openstack-meeting21:45
*** TrevorV has quit IRC21:48
*** jamesmcarthur has quit IRC21:53
*** verdurin has quit IRC21:58
*** andrebeltrami has joined #openstack-meeting22:04
*** jamesmcarthur has joined #openstack-meeting22:05
*** jamesmcarthur has quit IRC22:10
*** ianychoi has quit IRC22:19
*** ianychoi has joined #openstack-meeting22:20
*** ianychoi has quit IRC22:41
*** ociuhandu has quit IRC22:41
*** ianychoi has joined #openstack-meeting22:42
*** igordc has quit IRC22:58
*** ykatabam has joined #openstack-meeting23:20
*** rcernin has joined #openstack-meeting23:26
*** jmasud has joined #openstack-meeting23:28
*** hyunsikyang__ has quit IRC23:29
*** armax has quit IRC23:32
*** macz_ has quit IRC23:35
*** jmasud has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!