| *** arne_wiebalck has quit IRC | 00:03 | |
| *** yonglihe has quit IRC | 00:03 | |
| *** yonglihe has joined #openstack-meeting | 00:03 | |
| *** arne_wiebalck has joined #openstack-meeting | 00:03 | |
| *** macz_ has quit IRC | 00:18 | |
| *** gyee has quit IRC | 01:02 | |
| *** macz_ has joined #openstack-meeting | 01:11 | |
| *** jmasud has quit IRC | 01:12 | |
| *** macz_ has quit IRC | 01:16 | |
| *** ociuhandu has joined #openstack-meeting | 01:28 | |
| *** Liang__ has joined #openstack-meeting | 01:30 | |
| *** macz_ has joined #openstack-meeting | 01:33 | |
| *** ociuhandu has quit IRC | 01:33 | |
| *** macz_ has quit IRC | 01:37 | |
| *** jmasud has joined #openstack-meeting | 01:43 | |
| *** dmacpher has quit IRC | 01:57 | |
| *** dmacpher_ has joined #openstack-meeting | 01:57 | |
| *** jmasud has quit IRC | 02:15 | |
| *** Liang__ has quit IRC | 02:33 | |
| *** Liang__ has joined #openstack-meeting | 02:37 | |
| *** rcernin has quit IRC | 02:48 | |
| *** rcernin has joined #openstack-meeting | 02:50 | |
| *** psachin has joined #openstack-meeting | 03:46 | |
| *** jmasud has joined #openstack-meeting | 04:03 | |
| *** dsariel has joined #openstack-meeting | 04:17 | |
| *** dsariel has quit IRC | 04:21 | |
| *** macz_ has joined #openstack-meeting | 04:34 | |
| *** macz_ has quit IRC | 04:39 | |
| *** psachin has quit IRC | 05:27 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-meeting | 05:33 | |
| *** Liang__ has quit IRC | 06:21 | |
| *** Liang__ has joined #openstack-meeting | 06:22 | |
| *** vishalmanchanda has joined #openstack-meeting | 06:26 | |
| *** macz_ has joined #openstack-meeting | 06:28 | |
| *** macz_ has quit IRC | 06:33 | |
| *** jmasud has quit IRC | 07:25 | |
| *** dklyle has quit IRC | 07:35 | |
| *** ralonsoh has joined #openstack-meeting | 07:54 | |
| *** slaweq has joined #openstack-meeting | 08:03 | |
| *** rcernin has quit IRC | 08:05 | |
| *** macz_ has joined #openstack-meeting | 08:16 | |
| *** macz_ has quit IRC | 08:20 | |
| *** rcernin has joined #openstack-meeting | 08:37 | |
| *** tosky has joined #openstack-meeting | 08:42 | |
| *** e0ne has joined #openstack-meeting | 08:51 | |
| *** rcernin has quit IRC | 08:56 | |
| *** rpittau|afk is now known as rpittau | 09:05 | |
| *** ociuhandu has joined #openstack-meeting | 09:09 | |
| *** ralonsoh has quit IRC | 09:16 | |
| *** ociuhandu has quit IRC | 09:19 | |
| *** ralonsoh has joined #openstack-meeting | 09:19 | |
| *** larainema has joined #openstack-meeting | 09:23 | |
| *** macz_ has joined #openstack-meeting | 09:24 | |
| *** macz_ has quit IRC | 09:29 | |
| *** ociuhandu has joined #openstack-meeting | 09:32 | |
| *** soniya29|afk is now known as soniya29 | 09:37 | |
| *** ociuhandu has quit IRC | 09:38 | |
| *** Liang__ has quit IRC | 09:51 | |
| *** ociuhandu has joined #openstack-meeting | 09:53 | |
| *** rcernin has joined #openstack-meeting | 09:59 | |
| *** macz_ has joined #openstack-meeting | 10:00 | |
| *** macz_ has quit IRC | 10:05 | |
| *** rcernin has quit IRC | 10:13 | |
| *** ociuhandu has quit IRC | 10:37 | |
| *** ociuhandu has joined #openstack-meeting | 10:37 | |
| *** ociuhandu has quit IRC | 10:42 | |
| *** ociuhandu has joined #openstack-meeting | 10:49 | |
| *** lpetrut has joined #openstack-meeting | 10:55 | |
| *** oneswig has joined #openstack-meeting | 10:59 | |
| oneswig | #startmeeting scientific-sig | 11:00 |
|---|---|---|
| openstack | Meeting started Wed Nov 4 11:00:11 2020 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 11:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 11:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 11:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 11:00 |
| oneswig | knock knock | 11:00 |
| *** ociuhandu has quit IRC | 11:01 | |
| oneswig | No formal agenda has been set for today but a couple of points to raise | 11:01 |
| verdurin | Morning. I'm sort-of here. | 11:03 |
| oneswig | aren't we all. | 11:04 |
| oneswig | Hello verdurin | 11:04 |
| oneswig | Had you seen the EGI conference going on currently? | 11:04 |
| oneswig | #link EGI Conference 2020 https://indico.egi.eu/event/5000/timetable/#20201104 | 11:05 |
| verdurin | Parts of it. Can't spare the time to watch much of it live this week. | 11:05 |
| oneswig | Me neither. John from our team is presenting later on preemption and reservation - Cloud computing part 2 session IIRC. | 11:06 |
| oneswig | Interesting to see the "Global Open Science Cloud" that b1airo referred to. | 11:07 |
| oneswig | Have you been presenting recently verdurin? | 11:08 |
| verdurin | No. I really should, though. | 11:11 |
| oneswig | It was a somewhat leading question... there was some discussion recently on recording short presentations for the SIG. | 11:12 |
| oneswig | I think other groups are planning to do this too. | 11:12 |
| oneswig | #link recordings of presentations to SIG meetings https://etherpad.opendev.org/p/os-recordings | 11:12 |
| verdurin | Will have a think. | 11:15 |
| oneswig | Likewise. | 11:15 |
| oneswig | I don't think I had anything else for today. Anything more from you? | 11:16 |
| verdurin | Not really. Haven't even caught up with OpenInfra Summit videos yet... | 11:17 |
| oneswig | No, me neither, but I'm hoping to peruse the new edition in the archive | 11:17 |
| oneswig | Shall we close the meeting? | 11:18 |
| oneswig | In the absence of AOB, let's finish | 11:22 |
| oneswig | #endmeeting | 11:22 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 11:22 | |
| openstack | Meeting ended Wed Nov 4 11:22:40 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 11:22 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-11-04-11.00.html | 11:22 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-11-04-11.00.txt | 11:22 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-11-04-11.00.log.html | 11:22 |
| oneswig | Thanks verdurin | 11:22 |
| *** e0ne has quit IRC | 11:26 | |
| *** oneswig has quit IRC | 11:26 | |
| *** ociuhandu has joined #openstack-meeting | 11:32 | |
| *** ociuhandu has quit IRC | 11:40 | |
| *** masahito_ has joined #openstack-meeting | 11:45 | |
| *** e0ne has joined #openstack-meeting | 11:58 | |
| *** rh-jelabarre has quit IRC | 12:06 | |
| *** raildo has joined #openstack-meeting | 12:10 | |
| *** e0ne has quit IRC | 12:14 | |
| *** ociuhandu has joined #openstack-meeting | 12:50 | |
| *** rh-jelabarre has joined #openstack-meeting | 13:01 | |
| *** ociuhandu has quit IRC | 13:01 | |
| *** ociuhandu has joined #openstack-meeting | 13:02 | |
| *** ociuhandu has quit IRC | 13:07 | |
| *** rfolco has joined #openstack-meeting | 13:07 | |
| *** macz_ has joined #openstack-meeting | 13:09 | |
| *** macz_ has quit IRC | 13:14 | |
| *** e0ne has joined #openstack-meeting | 13:17 | |
| *** ociuhandu has joined #openstack-meeting | 13:17 | |
| *** Luzi has joined #openstack-meeting | 13:19 | |
| *** ZhuXiaoYu has joined #openstack-meeting | 14:01 | |
| *** liuyulong has joined #openstack-meeting | 14:01 | |
| liuyulong | #startmeeting neutron_l3 | 14:02 |
| openstack | Meeting started Wed Nov 4 14:02:17 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:02 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:02 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:02 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:02 |
| liuyulong | Sorry, a bit late. | 14:02 |
| *** ociuhandu has quit IRC | 14:02 | |
| liuyulong | It's about a month that we have no L3 meeting. | 14:03 |
| liuyulong | OK, seems I'm the only one here. | 14:03 |
| liuyulong | #topic Announcements | 14:03 |
| *** openstack changes topic to "Announcements (Meeting topic: neutron_l3)" | 14:03 | |
| *** ralonsoh has left #openstack-meeting | 14:04 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018489.html | 14:04 |
| *** ociuhandu has joined #openstack-meeting | 14:05 | |
| liuyulong | This is the neutron PTG summary, thanks to slaweq for this nice work. | 14:05 |
| liuyulong | It is really useful for those who have not attend the meetings. | 14:06 |
| liuyulong | #link http://kaplonski.pl/images/Virtual_PTG_October_2020/neutron_ptg_team_photo_october_2020.png | 14:07 |
| liuyulong | This is the team photo. | 14:08 |
| liuyulong | #link http://kaplonski.pl/blog/virtual_ptg_october_2020_summary/ | 14:08 |
| liuyulong | There was a horse in the meeting, haha. | 14:09 |
| slaweq | hi | 14:10 |
| *** ociuhandu has quit IRC | 14:10 | |
| slaweq | I have another internal meeting so I will be just lurking here | 14:10 |
| liuyulong | The L3 meeting, IMO, is still necessary for the neutron team. Since we have lots of bugs/rfes, this should be a nice chance to discuss here. | 14:11 |
| liuyulong | But, seems now we have less attendees, maybe I should send another mail to inform the meeting schedule. | 14:12 |
| liuyulong | slaweq, Hi | 14:12 |
| liuyulong | OK | 14:12 |
| liuyulong | Let's review the bug list. | 14:12 |
| liuyulong | #topic Bugs | 14:13 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:13 | |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1899967 | 14:14 |
| openstack | Launchpad bug 1899967 in neutron "L3HA Router in primary state on all nodes" [Critical,In progress] - Assigned to Slawek Kaplonski (slaweq) | 14:14 |
| liuyulong | There is a revert for this. | 14:14 |
| liuyulong | #link https://review.opendev.org/#/c/759679/2/doc/source/admin/config-ovsfwdriver.rst | 14:14 |
| liuyulong | So the original patch about the multicast traffic for 224.0.0.X should be allowed in such case. | 14:15 |
| liuyulong | OK | 14:16 |
| liuyulong | Next one | 14:16 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1901992 | 14:16 |
| openstack | Launchpad bug 1901992 in neutron "Original default route in DVR SNAT namespace is not recovered after creating and deleting /0 extraroute" [Medium,Triaged] - Assigned to gao yu (gaoyublack) | 14:16 |
| *** ociuhandu has joined #openstack-meeting | 14:17 | |
| liuyulong | The bug is confirmed. | 14:17 |
| liuyulong | The 0.0.0.0/0 means the default route, but if user set this intentionally, some functions of DVR may not work. | 14:18 |
| haleyb | right, they have broken themselves | 14:19 |
| liuyulong | When user removes such route, the L3 agent should recover its connection by repair the default route. | 14:19 |
| liuyulong | Yes, I'm OK with that we can allow the user set the 0.0.0.0/0 route. | 14:20 |
| liuyulong | But L3 agent should repair after the removal. | 14:20 |
| haleyb | yes, agreed | 14:21 |
| liuyulong | I have left some comments in the bug and the patch there. | 14:21 |
| liuyulong | #link Intentionally | 14:21 |
| liuyulong | #link https://review.opendev.org/#/c/760314/ | 14:21 |
| liuyulong | No comments from the bug reporter for now. | 14:21 |
| liuyulong | Our local deployment have done some work like the patch did, disable the API for "0.0.0.0/0". | 14:22 |
| liuyulong | So we have no patch to fix the bug in agent side. | 14:23 |
| liuyulong | OK, next one | 14:25 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1902211 | 14:25 |
| openstack | Launchpad bug 1902211 in neutron "Router State standby on all l3 agent when create" [Undecided,Incomplete] | 14:25 |
| liuyulong | I guess the bug report had pasted some information that the VPN related code was stuck. | 14:26 |
| liuyulong | I have some thougths that maybe the BIG lock of these function is the root cause. | 14:27 |
| liuyulong | Something like we did the work in L3 agent extensions, we have narrowed down the lock granularity for those. | 14:29 |
| haleyb | restarting l3-agent usually means it's a race condition, and this is train, so wonder if he is missing a fix? just seems like something we've fixed | 14:29 |
| liuyulong | The lock is set to some funcion that has some time-consuming, so the later router update actions just wait, then the routers are all in stand-by state. | 14:30 |
| liuyulong | I don't know. | 14:34 |
| liuyulong | Nguyen Thanh Cong pasted a log file. | 14:34 |
| liuyulong | #link https://pikab.in/f688029f37 | 14:34 |
| liuyulong | It shows the router VPN extension has done the sync work: "StrongSwanDriver method sync called with arguments". | 14:35 |
| liuyulong | But there is no router update action end log for the update id: 201d36bf-1560-441c-a944-a9c70bf241ed. | 14:35 |
| liuyulong | So, maybe there are some other L3 agent extensions which are time consuming or locked down... | 14:36 |
| liuyulong | Maybe disable all L3 agent extensions to see if the router can be set properly, then enable the extensions one by one. | 14:37 |
| *** TrevorV has joined #openstack-meeting | 14:37 | |
| haleyb | and no traceback | 14:38 |
| liuyulong | OK, no more bugs from me now. | 14:38 |
| liuyulong | Alright, let's move on. | 14:41 |
| liuyulong | #topic On demand agenda | 14:41 |
| *** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)" | 14:41 | |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1877301 | 14:42 |
| openstack | Launchpad bug 1877301 in neutron "[RFE] L3 Router support ndp proxy" [Wishlist,In progress] - Assigned to yangjianfeng (yangjianfeng) | 14:42 |
| liuyulong | #link https://review.opendev.org/#/c/747523/ | 14:42 |
| liuyulong | #undo | 14:42 |
| openstack | Removing item from minutes: #link https://review.opendev.org/#/c/747523/ | 14:42 |
| liuyulong | #link https://review.opendev.org/#/c/728628/ | 14:42 |
| liuyulong | I think that the spec is good enough to get merged except moving to W folder. | 14:43 |
| *** masahito_ has quit IRC | 14:43 | |
| liuyulong | I will conntact jianfeng to move on. | 14:44 |
| liuyulong | OK, thank you guys. Let's end here. | 14:47 |
| liuyulong | #endmeeting | 14:47 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 14:47 | |
| openstack | Meeting ended Wed Nov 4 14:47:57 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:47 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-11-04-14.02.html | 14:47 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-11-04-14.02.txt | 14:48 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-11-04-14.02.log.html | 14:48 |
| *** macz_ has joined #openstack-meeting | 15:06 | |
| *** macz_ has quit IRC | 15:11 | |
| *** liuyulong has quit IRC | 15:17 | |
| *** dklyle has joined #openstack-meeting | 15:23 | |
| *** Luzi has quit IRC | 15:24 | |
| *** abhishekk is now known as abhishekk|dinner | 15:24 | |
| *** macz_ has joined #openstack-meeting | 15:36 | |
| *** lpetrut has quit IRC | 15:42 | |
| *** jmasud has joined #openstack-meeting | 15:48 | |
| *** e0ne has quit IRC | 16:07 | |
| *** ociuhandu has quit IRC | 16:09 | |
| *** ociuhandu has joined #openstack-meeting | 16:10 | |
| *** ociuhandu has quit IRC | 16:14 | |
| *** ociuhandu has joined #openstack-meeting | 16:15 | |
| *** abhishekk|dinner is now known as abhishekk | 16:40 | |
| *** ociuhandu_ has joined #openstack-meeting | 17:08 | |
| *** armstrong has joined #openstack-meeting | 17:10 | |
| *** ociuhandu has quit IRC | 17:11 | |
| *** ociuhandu_ has quit IRC | 17:13 | |
| *** tosky has quit IRC | 17:15 | |
| *** jmasud has quit IRC | 17:31 | |
| *** jmasud has joined #openstack-meeting | 17:32 | |
| *** rpittau is now known as rpittau|afk | 17:35 | |
| *** vishalmanchanda has quit IRC | 17:38 | |
| *** ZhuXiaoYu has quit IRC | 17:57 | |
| *** e0ne has joined #openstack-meeting | 17:59 | |
| *** mlavalle has quit IRC | 18:08 | |
| *** mugsie has quit IRC | 18:18 | |
| *** mugsie has joined #openstack-meeting | 18:21 | |
| *** jmasud has quit IRC | 18:35 | |
| *** mlavalle has joined #openstack-meeting | 18:50 | |
| *** mlavalle has quit IRC | 18:59 | |
| *** mlavalle has joined #openstack-meeting | 19:19 | |
| *** _mlavalle_1 has joined #openstack-meeting | 19:21 | |
| *** mlavalle has quit IRC | 19:24 | |
| *** _mlavalle_2 has joined #openstack-meeting | 19:25 | |
| *** _mlavalle_1 has quit IRC | 19:27 | |
| *** e0ne has quit IRC | 20:31 | |
| *** slaweq has quit IRC | 20:32 | |
| *** tosky has joined #openstack-meeting | 20:35 | |
| *** patchbot has joined #openstack-meeting | 20:59 | |
| *** zaitcev has joined #openstack-meeting | 20:59 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Nov 4 21:00:06 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| kota_ | hi | 21:00 |
| seongsoocho | o/ Hi ! | 21:00 |
| mattoliverau | o/ | 21:00 |
| rledisez | hi o/ | 21:00 |
| timburke | rledisez! we missed you last week | 21:01 |
| timburke | good to see you again :-) | 21:01 |
| rledisez | yeah, sorry I missed that :( i hope you had great talk | 21:02 |
| zaitcev | o/ | 21:02 |
| *** acoles has joined #openstack-meeting | 21:02 | |
| zaitcev | rledisez: I thought it was going to be all ex-OpenIO at OVH from now on, with Swift and Ceph commiserating the in the dustbin. | 21:03 |
| clayg | ugh, *already* | 21:03 |
| zaitcev | rledisez: Because both you and Alex missed, so it was a bit more than just someone could not make it. | 21:03 |
| acoles | o/ | 21:03 |
| timburke | all right -- first up | 21:04 |
| timburke | #topic PTG | 21:04 |
| *** openstack changes topic to "PTG (Meeting topic: swift)" | 21:04 | |
| timburke | thanks to those that cameto the ptg last week, and those that didn't, we all missed you | 21:05 |
| timburke | notmyname even made a cameo appearance or two :-) | 21:05 |
| clayg | #throwback | 21:05 |
| timburke | we covered a bunch of great topics, and tried to takenotes in the etherpad as we did so | 21:06 |
| timburke | #link https://etherpad.opendev.org/p/swift-ptg-wallaby | 21:06 |
| timburke | a quick overview: | 21:06 |
| timburke | clayg was interested in havig a new handoff table in the ring, used only on reads | 21:07 |
| timburke | people generally seemed enthusiastic about the idea, and we'll see what code ends up looking like | 21:08 |
| clayg | "or if it gets written" 🤷♂️ | 21:08 |
| timburke | i was looking at memcached error limiting, particularly when you only have a single mecached server configured in the proxy | 21:08 |
| timburke | i wound up writing a new patch to make the existing, hard-coded error limiting more tunable | 21:09 |
| timburke | #link https://review.opendev.org/#/c/761029/ | 21:09 |
| patchbot | patch 761029 - swift - memcache: Make error-limiting values configurable - 1 patch set | 21:09 |
| timburke | ... which i think will satisfy my needs without going the more-complicated route in the chain starting at https://review.opendev.org/759183 | 21:10 |
| patchbot | patch 759183 - swift - memcache: Refuse to error limit the last available... - 2 patch sets | 21:10 |
| timburke | zaitcev and dsariel continue workig on audit watchers | 21:11 |
| timburke | #link https://review.opendev.org/#/c/706653/ | 21:11 |
| patchbot | patch 706653 - swift - Let developers/operators add watchers to object au... - 38 patch sets | 21:11 |
| timburke | i think it's getting pretty darn close to merging | 21:11 |
| zaitcev | No, actually, it's done. I addressed the comments that came up at PTG like moving to a separate directory. | 21:11 |
| zaitcev | Although leaving a debugging logger.info in there was pretty embarrassing. | 21:12 |
| timburke | yeah,i suppose i should say, "i needto go have another look and merge it" :-) | 21:12 |
| timburke | we talked a bit about the default and recommended configs, and came away with a few different things we wanted to do | 21:12 |
| timburke | cschwede will look at improving our recommendations | 21:13 |
| acoles | zaitcev: not as embarassing as me leaving a debugging print ! | 21:13 |
| timburke | mattoliverau will look at trimming our manpages (to mostly just point to online docs iirc) | 21:14 |
| timburke | and acoles will pull the long tables i the deployment guide out to separate pages (hopefully making the whole thing a bit more readable) | 21:15 |
| mattoliverau | zaitcev: I'll try and get back to audit watchers this week. Just been distracted. | 21:16 |
| timburke | clayg is excited about some of the recent proxy-logging changes and how they let you slice your metrics; he'll likely propose a change to the sample cofigs to have two separately-named proxy-logging middlewares, each with their own namespacing | 21:17 |
| clayg | yeah, i really should do that - and also fix whatever is wrong with the byte-enforcing code | 21:18 |
| timburke | and there was a arathon ops feedback session with a lot of good commentary from ormandj | 21:18 |
| *** raildo has quit IRC | 21:19 | |
| timburke | (i feel a little bad that that took up so much of our time, but i also feel like it's always one of the most-valuable things we can do when we're all together) | 21:19 |
| timburke | for ore on that, see | 21:19 |
| timburke | #link https://etherpad.opendev.org/p/swift-wallaby-ops-feedback | 21:19 |
| mattoliverau | +1 ormandji and the marathon ops feedback was awesome. | 21:19 |
| clayg | for sure all good swift ❤️ ops | 21:20 |
| timburke | that's my quick recap of the ptg; did i miss (or misrepresent) anything major? | 21:20 |
| zaitcev | ALO? | 21:21 |
| acoles | mattoliverau: talked us through all his great work on eliminating overlapping shard ranges | 21:22 |
| timburke | oh yeah -- i keep writing it off sincei haven't actually written any code for it yet ;-) but hopefully people have a better feel for the problems we've seen with trying to use SLOs for s3 MPUs, and why a new type of large object might be useful/necessary | 21:22 |
| timburke | all right, moving on | 21:24 |
| timburke | #topic gate failures | 21:24 |
| *** openstack changes topic to "gate failures (Meeting topic: swift)" | 21:24 | |
| timburke | so lately, i've had this feeling like our gate has been in particularly bad shape | 21:25 |
| mattoliverau | And acoles came up with a great alternative shard audit with gaps algorithm I want to now write into code. | 21:25 |
| timburke | i think the guy that finally pushed meover the edge was https://review.opendev.org/#/c/759790/ -- 10 rechecks for a one-line change to drop an unused package from lower-constraints | 21:26 |
| patchbot | patch 759790 - swift - Remove the unused coding style modules - 1 patch set | 21:26 |
| mattoliverau | Only 10 rechecks :p | 21:27 |
| timburke | so i started writing some tooling to get build info from zuul, pull down logs or subunit results, and parse out failures, looking for which jobs (and which individual *tests*) fail most often | 21:27 |
| zaitcev | Suspense intensifies | 21:28 |
| timburke | i don't have parsing for all job types yet, but it's already been able to help me find some particularly bad/annoying patterns | 21:28 |
| mattoliverau | Nice | 21:29 |
| timburke | for instance, of 258 individual probe test failures, 149 of them were the result of resetswift failing because the loopback device was busy | 21:30 |
| acoles | :'( | 21:30 |
| timburke | hopefully that failure mode will go away with https://review.opendev.org/#/c/761439/ | 21:30 |
| patchbot | patch 761439 - swift - saio: Stop processes more forcefully in resetswift - 1 patch set | 21:30 |
| timburke | i also found that the func tests were pretty nice to deal with, since they emit testrepository.subunit files | 21:31 |
| timburke | so i figured i'd try switching probe tests to use ostestr, too: https://review.opendev.org/#/c/761459/ (and we'll just see whether the file shows up; it's all magic to me :-/) | 21:32 |
| patchbot | patch 761459 - swift - probe: Use ostestr as test runner - 1 patch set | 21:32 |
| mattoliverau | Nice work timburke | 21:33 |
| clayg | mattoliverau: +1 timburke is a gate hero! | 21:33 |
| timburke | there will probably be more information i glean from all of this (and more patches i write as a result), but wanted to share what i've been working on so far with it | 21:33 |
| timburke | because i'm *so* tired of starting (and often ending) my day with a slew of rechecks :-( | 21:34 |
| timburke | any questions or comments? | 21:34 |
| acoles | good stuff timburke | 21:34 |
| tosky | timburke: if I may - at this point you may try stestr | 21:35 |
| tosky | ostestr is meant to be deprecated | 21:36 |
| tosky | not sure it was already considered and tried in the past, maybe it was, so feel free to ignore me on this :) | 21:36 |
| timburke | so i *did* try that originally! but forsome unknown reason it caused the partition numbers used in the probe tests' rings to come out different -- no idea why | 21:36 |
| tosky | uhm | 21:37 |
| timburke | i'm certainly interested in making sure we have maintained software for the test runner, but i figured i'd start with just seeing whether using the same runner that we do in func tests gets me the test artifacts i'm looking for | 21:38 |
| *** zaitcev is now known as zaitcev|doc | 21:38 | |
| *** zaitcev|doc is now known as zaitcev|dr | 21:38 | |
| timburke | it's very strange, though. i'll keep digging, see if i can get some kind of repro/explanation/bug report for it | 21:39 |
| timburke | #topic replication lock | 21:40 |
| *** openstack changes topic to "replication lock (Meeting topic: swift)" | 21:40 | |
| timburke | #link https://review.opendev.org/#/c/754242/ | 21:40 |
| patchbot | patch 754242 - swift - Fix a race condition in case of cross-replication - 6 patch sets | 21:40 |
| timburke | rledisez, i'm sorry to say, i still haven't reviewed it :-( | 21:40 |
| timburke | i even promised i would, too. sorry | 21:40 |
| rledisez | that's ok, everybody is busy. it will be reviewed eventually ;) | 21:41 |
| *** rcernin has joined #openstack-meeting | 21:41 | |
| timburke | i'd still like to get a test ev up such that i actually repro the problem and see the patch fix it, but at the same time, (1) you're already runing it in prod, (2) it's definitely making your clusters better, and (3) i want *everybody's* clusters to run as well as rledisez's | 21:43 |
| timburke | somaybe i should just run it through my mental python parser (clayg always tells me it's a pretty good one) | 21:43 |
| mattoliverau | So that's your secret :) | 21:44 |
| clayg | timburke: your brain is amazing | 21:44 |
| clayg | mattoliverau: and he can do py2 and py3 at the same time!!! | 21:44 |
| clayg | rledisez: did the rsync fix get squashed into the ssync change? I was pretty happy with the strategy for locking the REPLICATION requests; but never quite followed what you were thinking for rsync? | 21:45 |
| timburke | i'll try again this week; we've got a pretty big rebalance coming up, it'd probably be good for us to have that patch | 21:45 |
| timburke | i don't think we have an rsync fix yet | 21:46 |
| rledisez | clayg: no rsync fix. i have the idea pretty clear but I lack time | 21:46 |
| timburke | all right, that's all i've got for the agenda | 21:47 |
| timburke | #topic open discussion | 21:47 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:47 | |
| timburke | speaking of replication, rledisez, i'd be curious about your take on https://review.opendev.org/#/c/758636/ | 21:48 |
| patchbot | patch 758636 - swift - Add option to REPLICATE to just invalidate hashes - 5 patch sets | 21:48 |
| timburke | (though you might be more interested in just ripping out post-(s)sync replicate calls; see https://bugs.launchpad.net/swift/+bug/1818709) | 21:49 |
| openstack | Launchpad bug 1818709 in OpenStack Object Storage (swift) "object replicator update_deleted post ssync REPLICATE request considered harmful" [Undecided,New] | 21:49 |
| rledisez | yeah, I sometimes disable it (like after a relink, it helps a lot). i'll have a look at the patch | 21:50 |
| timburke | thanks | 21:50 |
| tosky | I have a quick note about some legacy jobs (I'm the coordinator for the "no legacy jobs" community goal) | 21:51 |
| tosky | I've just noticed a few legacy jobs I originally missed (thanks for porting basically all of them long time ago!) | 21:51 |
| tosky | they don't use devstack-gate, so they are not so problematic, but still: they are in pyeclib | 21:52 |
| tosky | so if you could convert those as well, and backport them to stable/victoria (and if you want also to older branches), that would be nice! | 21:52 |
| timburke | tosky, thanks for the reminder! i'll look into it (hopefully this week?) | 21:53 |
| tosky | oh, I see now, it doesn't have the openstack stable branches, so I guess master is fine | 21:53 |
| tosky | thank you! | 21:53 |
| mattoliverau | Yay, already easier :) | 21:53 |
| timburke | i suspect they could largely be switched to the openstack-tox-... jobs | 21:53 |
| tosky | most likely | 21:54 |
| tosky | the official gerrit topic is "native-zuulv3-migration" | 21:54 |
| timburke | 👍 | 21:55 |
| tosky | sorry for the late ping, I totally missed them originally; the devstack-gate jobs had (and have, the few left) an higher priority | 21:55 |
| timburke | makes sense -- and pyeclib sees few enough patches, it likely wouldn't show up on any list of recently-run jobs | 21:56 |
| timburke | all right | 21:57 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:57 |
| timburke | #endmeeting | 21:57 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:57 | |
| openstack | Meeting ended Wed Nov 4 21:57:20 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:57 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-11-04-21.00.html | 21:57 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-11-04-21.00.txt | 21:57 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-11-04-21.00.log.html | 21:57 |
| *** patchbot has left #openstack-meeting | 21:57 | |
| *** acoles has quit IRC | 21:58 | |
| *** acoles has joined #openstack-meeting | 21:59 | |
| *** acoles has left #openstack-meeting | 22:03 | |
| *** rcernin has quit IRC | 22:09 | |
| *** rcernin has joined #openstack-meeting | 22:09 | |
| *** TrevorV has quit IRC | 22:29 | |
| *** rfolco has quit IRC | 22:30 | |
| *** jmasud has joined #openstack-meeting | 22:37 | |
| *** rcernin has quit IRC | 23:04 | |
| *** jmasud has quit IRC | 23:07 | |
| *** rcernin has joined #openstack-meeting | 23:13 | |
| *** bbowen has quit IRC | 23:13 | |
| *** bbowen has joined #openstack-meeting | 23:15 | |
| *** rcernin has quit IRC | 23:16 | |
| *** rcernin has joined #openstack-meeting | 23:16 | |
| *** jmasud has joined #openstack-meeting | 23:31 | |
| *** jmasud has quit IRC | 23:43 | |
| *** zaitcev|dr is now known as zaitcev | 23:48 | |
| *** zaitcev has left #openstack-meeting | 23:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
