| *** yasufum has joined #openstack-meeting | 00:02 | |
| *** yasufum has quit IRC | 00:02 | |
| *** yasufum has joined #openstack-meeting | 00:02 | |
| *** hyunsikyang has quit IRC | 00:04 | |
| *** tosky has quit IRC | 00:04 | |
| *** hyunsikyang has joined #openstack-meeting | 00:04 | |
| *** yasufum has quit IRC | 00:07 | |
| *** yasufum has joined #openstack-meeting | 00:08 | |
| *** jmasud has joined #openstack-meeting | 00:50 | |
| *** jmasud has quit IRC | 00:55 | |
| *** macz_ has quit IRC | 00:58 | |
| *** rcernin_ has joined #openstack-meeting | 01:09 | |
| *** rcernin has quit IRC | 01:10 | |
| *** mlavalle has quit IRC | 01:30 | |
| *** _mlavalle_1 has joined #openstack-meeting | 01:30 | |
| *** kevinz has joined #openstack-meeting | 01:39 | |
| *** jmasud has joined #openstack-meeting | 01:58 | |
| *** jmasud has quit IRC | 02:04 | |
| *** yasufum has quit IRC | 02:05 | |
| *** jmasud has joined #openstack-meeting | 02:08 | |
| *** jmasud has quit IRC | 02:10 | |
| *** yasufum has joined #openstack-meeting | 02:16 | |
| *** dasp has quit IRC | 02:20 | |
| *** timburke has quit IRC | 02:21 | |
| *** dasp has joined #openstack-meeting | 02:21 | |
| *** yasufum_ has joined #openstack-meeting | 02:34 | |
| *** benj_- has joined #openstack-meeting | 02:35 | |
| *** yasufum has quit IRC | 02:35 | |
| *** yasufum_ is now known as yasufum | 02:35 | |
| *** benj_ has quit IRC | 02:35 | |
| *** benj_- is now known as benj_ | 02:35 | |
| *** rcernin_ has quit IRC | 02:46 | |
| *** macz_ has joined #openstack-meeting | 02:55 | |
| *** macz_ has quit IRC | 03:00 | |
| *** rfolco has joined #openstack-meeting | 03:11 | |
| *** manpreet has joined #openstack-meeting | 03:18 | |
| *** jamesden_ has quit IRC | 03:29 | |
| *** rfolco has quit IRC | 03:32 | |
| *** gyee has quit IRC | 03:43 | |
| *** macz_ has joined #openstack-meeting | 03:46 | |
| *** macz_ has quit IRC | 03:50 | |
| *** yasufum has quit IRC | 04:09 | |
| *** ociuhandu has joined #openstack-meeting | 04:13 | |
| *** ociuhandu has quit IRC | 04:17 | |
| *** yasufum has joined #openstack-meeting | 04:53 | |
| *** vishalmanchanda has joined #openstack-meeting | 05:02 | |
| *** jmasud has joined #openstack-meeting | 05:08 | |
| *** dmacpher__ has joined #openstack-meeting | 05:25 | |
| *** dmacpher_ has quit IRC | 05:27 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-meeting | 05:33 | |
| *** yasufum has quit IRC | 05:39 | |
| *** ricolin has joined #openstack-meeting | 05:59 | |
| *** yamamoto has quit IRC | 06:48 | |
| *** yasufum has joined #openstack-meeting | 06:49 | |
| *** yasufum has quit IRC | 07:17 | |
| *** yasufum_ has joined #openstack-meeting | 07:17 | |
| *** yamamoto has joined #openstack-meeting | 07:29 | |
| *** yamamoto has quit IRC | 07:39 | |
| *** dklyle has quit IRC | 07:41 | |
| *** ociuhandu has joined #openstack-meeting | 07:50 | |
| *** ociuhandu has quit IRC | 07:54 | |
| *** slaweq has joined #openstack-meeting | 08:00 | |
| *** rpittau|afk is now known as rpittau | 08:18 | |
| *** ociuhandu has joined #openstack-meeting | 08:31 | |
| *** tosky has joined #openstack-meeting | 08:33 | |
| *** jmasud has quit IRC | 08:35 | |
| *** jawad_axd has joined #openstack-meeting | 08:43 | |
| *** ociuhandu has quit IRC | 08:51 | |
| *** e0ne has joined #openstack-meeting | 08:54 | |
| *** jmasud has joined #openstack-meeting | 09:27 | |
| *** jawad_axd has quit IRC | 09:30 | |
| *** jawad_axd has joined #openstack-meeting | 09:30 | |
| *** vishalmanchanda has quit IRC | 09:36 | |
| *** jmasud has quit IRC | 09:39 | |
| *** tosky_ has joined #openstack-meeting | 09:47 | |
| *** tosky is now known as Guest24372 | 09:49 | |
| *** tosky_ is now known as tosky | 09:49 | |
| *** Guest24372 has quit IRC | 09:50 | |
| *** lpetrut has joined #openstack-meeting | 09:57 | |
| *** vishalmanchanda has joined #openstack-meeting | 10:07 | |
| *** yamamoto has joined #openstack-meeting | 10:12 | |
| *** baojg has quit IRC | 10:18 | |
| *** baojg has joined #openstack-meeting | 10:18 | |
| *** lbragstad has quit IRC | 10:31 | |
| *** lbragstad has joined #openstack-meeting | 10:31 | |
| *** masazumi-ota has quit IRC | 10:40 | |
| *** ociuhandu has joined #openstack-meeting | 10:41 | |
| *** yasufum_ has quit IRC | 10:41 | |
| *** yasufum has joined #openstack-meeting | 10:43 | |
| *** ociuhandu_ has joined #openstack-meeting | 10:44 | |
| *** ociuhandu_ has quit IRC | 10:46 | |
| *** ociuhandu_ has joined #openstack-meeting | 10:47 | |
| *** ociuhandu has quit IRC | 10:47 | |
| *** ociuhandu_ has quit IRC | 10:52 | |
| *** ociuhandu has joined #openstack-meeting | 10:52 | |
| *** jmasud has joined #openstack-meeting | 10:53 | |
| *** ociuhandu has quit IRC | 10:53 | |
| *** oneswig has joined #openstack-meeting | 11:00 | |
| oneswig | #startmeeting scientific-sig | 11:00 |
|---|---|---|
| openstack | Meeting started Wed Dec 16 11:00:46 2020 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 11:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 11:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 11:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 11:00 |
| oneswig | What a week | 11:01 |
| *** macz_ has joined #openstack-meeting | 11:18 | |
| *** macz_ has quit IRC | 11:23 | |
| oneswig | #endmeeting | 11:23 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 11:23 | |
| openstack | Meeting ended Wed Dec 16 11:23:58 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 11:23 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-12-16-11.00.html | 11:24 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-12-16-11.00.txt | 11:24 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-12-16-11.00.log.html | 11:24 |
| *** oneswig has quit IRC | 11:24 | |
| *** yamamoto has quit IRC | 11:32 | |
| *** yasufum has quit IRC | 11:44 | |
| *** raildo has joined #openstack-meeting | 11:53 | |
| *** ociuhandu has joined #openstack-meeting | 11:55 | |
| *** yamamoto has joined #openstack-meeting | 11:55 | |
| *** ociuhandu has quit IRC | 11:59 | |
| *** yamamoto has quit IRC | 12:00 | |
| *** rfolco has joined #openstack-meeting | 12:01 | |
| *** baojg has quit IRC | 12:09 | |
| *** baojg has joined #openstack-meeting | 12:09 | |
| *** yamamoto has joined #openstack-meeting | 12:20 | |
| *** baojg has quit IRC | 12:42 | |
| *** baojg has joined #openstack-meeting | 12:43 | |
| *** ociuhandu has joined #openstack-meeting | 12:58 | |
| *** ociuhandu has quit IRC | 13:12 | |
| *** ociuhandu has joined #openstack-meeting | 13:13 | |
| *** ociuhandu has quit IRC | 13:18 | |
| *** ociuhandu has joined #openstack-meeting | 13:21 | |
| *** ociuhandu has quit IRC | 13:24 | |
| *** ociuhandu has joined #openstack-meeting | 13:24 | |
| *** thgcorrea has joined #openstack-meeting | 13:48 | |
| *** jmasud has quit IRC | 13:51 | |
| *** ociuhandu has quit IRC | 13:54 | |
| *** lbragstad has quit IRC | 13:54 | |
| *** lbragstad has joined #openstack-meeting | 13:57 | |
| *** lajoskatona has joined #openstack-meeting | 13:57 | |
| *** lbragstad has quit IRC | 13:57 | |
| *** lbragstad has joined #openstack-meeting | 13:58 | |
| *** liuyulong has joined #openstack-meeting | 13:59 | |
| liuyulong | #startmeeting neutron_l3 | 14:00 |
| openstack | Meeting started Wed Dec 16 14:00:46 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:00 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:00 |
| lajoskatona | Hi | 14:00 |
| liuyulong | Hi | 14:01 |
| rubasov | hi | 14:02 |
| liuyulong | No announcement from me, so let's go througth the bug directly. | 14:02 |
| liuyulong | #topic Bugs | 14:02 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:02 | |
| *** lbragstad_ has joined #openstack-meeting | 14:03 | |
| liuyulong | We have 3 lists | 14:04 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019154.html | 14:04 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019414.html | 14:04 |
| *** ociuhandu has joined #openstack-meeting | 14:04 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019244.html | 14:04 |
| haleyb | hi | 14:04 |
| *** ociuhandu has quit IRC | 14:05 | |
| *** ociuhandu has joined #openstack-meeting | 14:05 | |
| liuyulong | First one: | 14:05 |
| *** lbragstad has quit IRC | 14:05 | |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1906375 | 14:06 |
| openstack | Launchpad bug 1906375 in neutron "[L3] router HA port concurrently deleting" [Low,In progress] - Assigned to LIU Yulong (dragon889) | 14:06 |
| liuyulong | It was reported by me. | 14:06 |
| liuyulong | During a local testing procedure, I found this exception. | 14:06 |
| liuyulong | Basically it has no harm to the neutron, but raised some noise LOGs during the test. | 14:06 |
| slaweq | hi | 14:07 |
| liuyulong | I've summitted the patch: https://review.opendev.org/c/openstack/neutron/+/764913 | 14:07 |
| liuyulong | slaweq, Hi | 14:07 |
| slaweq | I have another meeting in same time but I will be lurking here | 14:07 |
| liuyulong | The patch just catch the port not exist related error, but left all other exceptions raised as it is. | 14:08 |
| liuyulong | It is simple. | 14:08 |
| *** ociuhandu has quit IRC | 14:08 | |
| slaweq | liuyulong: what about oleg's comment? | 14:09 |
| slaweq | are You going to add debug message there? | 14:09 |
| *** ociuhandu has joined #openstack-meeting | 14:09 | |
| liuyulong | Yes, I saw that. The code is same to the code in l3_hamode_db, there is no log. | 14:10 |
| liuyulong | So if you guys insist, I will add it. | 14:10 |
| slaweq | for me it's fine, if such log would be useful we can add it in both places in follow-up patch | 14:11 |
| liuyulong | https://github.com/openstack/neutron/blob/master/neutron/db/l3_hamode_db.py#L721-L729 | 14:11 |
| liuyulong | OK, next one | 14:12 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1907175 | 14:12 |
| openstack | Launchpad bug 1907175 in neutron " intermittently ALL VM's floating IP connection is disconnected, and can be reconnected after 5-6 minutes " [Undecided,New] | 14:12 |
| liuyulong | I've looked at this bug these days. | 14:13 |
| slaweq | for me this one looks more like some ovs bug maybe | 14:14 |
| liuyulong | This is, IMO, not pretty sure, a kernel bug of the bond mode 4. | 14:14 |
| slaweq | as they said that it is fixed when they restart openvswitch | 14:14 |
| slaweq | or kernel bug | 14:14 |
| slaweq | but not neutron issue really | 14:14 |
| liuyulong | I have a kernel patch which is trying to fix that, but it really similar to this. My case is bond mode 6. | 14:15 |
| liuyulong | Wait a minute... | 14:15 |
| liuyulong | #link https://marc.info/?l=linux-netdev&m=160430387811073&w=2 | 14:17 |
| liuyulong | There are some information about the topology in this kernel patch. | 14:18 |
| liuyulong | The main issue is because the slave NIC is trying to send the IPv6 related traffic with run mac which make the physical world failed to find the way back. | 14:19 |
| liuyulong | The LP bug reportor said they use the spine-leaf in their DC as well. | 14:21 |
| liuyulong | So maybe they can try to find the issue in such way. | 14:21 |
| liuyulong | tcpdump the slave NIC to see if it will send some packets out, and monitor the MAC table in the switch to see if the learnt entry is refrshed to disturb the traffic. | 14:23 |
| liuyulong | This is really a relly complicated issue, my suggestion for the user is to invite their switch manufacturer to work together to find out the real issue. | 14:25 |
| liuyulong | Sometimes, the switch forwarding protocol may be different in implementation which will result some unexcepted behavior. | 14:26 |
| liuyulong | OK, no comments. | 14:29 |
| liuyulong | Next one | 14:29 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1905295 | 14:29 |
| openstack | Launchpad bug 1905295 in neutron "[RFE] Allow multiple external gateways on a router" [Wishlist,New] - Assigned to Bence Romsics (bence-romsics) | 14:29 |
| *** ociuhandu has quit IRC | 14:29 | |
| liuyulong | In Last L3 meeting and last driver meeting, this had beed disscussed. | 14:30 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1905295/comments/9 | 14:30 |
| liuyulong | Bence had replied a long comment. : ) | 14:30 |
| rubasov | I hope I made some progress with the alternative we discussed | 14:33 |
| liuyulong | So my response can be: if you do not need floating IP, but you want multiple external interface. You may do it like this. | 14:33 |
| liuyulong | Change your mind of that "external network" to "provider network" which should be a common use. | 14:34 |
| liuyulong | So if you have multiple "provider network", then you can create VMs on it. And as many interface as you want. | 14:34 |
| *** TrevorV has joined #openstack-meeting | 14:34 | |
| rubasov | I guess I would have the provider networks as external, not putting VMs directly on them | 14:35 |
| liuyulong | The external network are basically for L3 floating IP(NAT) which is for neutron to support some product in users' view, aka Elastic Public IP. | 14:36 |
| rubasov | the change in neutron-dynamic-routing seems way simpler than making multiple external gw-s possible in neutron | 14:36 |
| rubasov | I think I will explore what exaclty that change in neutron-dynamic-routing would be | 14:37 |
| liuyulong | Yes, you can do that, but external network is sometimes not visible to common user. | 14:37 |
| rubasov | but overall I like this alternative | 14:38 |
| rubasov | looks like it spares us from some significant amount of work | 14:38 |
| lajoskatona | Just a question here: for me it seems strange (perhasp doc issue) that we can add external net to router as interface which is more "internal" | 14:39 |
| liuyulong | Cool, it's glad to see that we do not need to involve changing all L3 related stuff. | 14:39 |
| lajoskatona | So perhaps it is just how I understand the API doc | 14:39 |
| liuyulong | But can be done in some magic actions. | 14:39 |
| rubasov | if the neutron team agrees I'd definitely like to propose a change to the api-ref clearing this up | 14:40 |
| liuyulong | lajoskatona, you can consider the external network is a common one. | 14:40 |
| *** ociuhandu has joined #openstack-meeting | 14:40 | |
| liuyulong | lajoskatona, so it has subnets, then it can be attach to a router. | 14:40 |
| lajoskatona | liuyulong: ok, it's really good | 14:41 |
| liuyulong | lajoskatona, but is is currently not available for common user, it is admin only action. | 14:41 |
| lajoskatona | liuyulong: yeah, that's fair, as it is more an infrastructure thing | 14:41 |
| rubasov | (that api-ref piece blocked my thinking about this alternative for quite some time) | 14:41 |
| liuyulong | rubasov, my thougth is not to use the name "external network" in such case. Because the code has many "external_xxx", "external_yyy". My suggestion is to use "provider network". | 14:43 |
| liuyulong | And the deployment guide in doc.openstack.org for neutron, has a provider network only option. | 14:44 |
| rubasov | technically it's a decision done years ago I guess, the network already has an attribute called "external" | 14:44 |
| liuyulong | #link https://docs.openstack.org/install-guide/launch-instance-networks-provider.html | 14:46 |
| rubasov | though I'm not sure if that's bit is used anywhere when attaching a subnet of the external=true network as a router interface | 14:46 |
| liuyulong | And there is a routed provider networks guide: | 14:46 |
| liuyulong | #link https://docs.openstack.org/neutron/victoria/admin/config-routed-networks.html | 14:46 |
| liuyulong | Totally no floating IP concept in these guide. | 14:47 |
| liuyulong | rubasov, my idea is not to attach a network's subnet with external=true. | 14:47 |
| liuyulong | I remember there are some bugs related to such action... | 14:48 |
| liuyulong | Some routers added external gateway to the network. Some added the subnet. | 14:48 |
| rubasov | but then what is the meaning of that external bit, if I shoud set it to False on a network that's actually external? | 14:49 |
| liuyulong | IMO, it really not a good option which may case the DVR, routing table wich some unexcepted hehaviors. | 14:49 |
| liuyulong | rubasov, remember the L3 toplogy | 14:50 |
| liuyulong | rubasov, user's network needs to go through the "Router" then to the external network. | 14:50 |
| liuyulong | rubasov, the external gateway of a router is the leg to the outsider world. | 14:51 |
| *** ociuhandu has quit IRC | 14:51 | |
| rubasov | that's exactly my point in my LP comment, that the API today allows an external net's subnet as a router interface | 14:52 |
| lajoskatona | but this is only a flag (for the user/admin) on the API or really makes something different on the backend? | 14:52 |
| rubasov | so can that be my router's north leg instead of the router gw? | 14:52 |
| liuyulong | rubasov, you can, but just routing, no NAT. And again, in order to make things clear, we should can such network as "provider network" to distinguish the concepts. | 14:55 |
| rubasov | lajoskatona: that's a very good question which is hard to very hard to answer by grepping through the code | 14:55 |
| liuyulong | lajoskatona, same answer. No NAT, but only routing in the backend. | 14:55 |
| lajoskatona | ok thanks | 14:56 |
| lajoskatona | that's enough for me | 14:56 |
| rubasov | ok, I think we are on the same page | 14:56 |
| liuyulong | OK, thanks then, no bugs from me now. | 14:58 |
| liuyulong | We still have 2 mins. | 14:58 |
| liuyulong | #topic On demand agenda | 14:58 |
| *** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)" | 14:58 | |
| liuyulong | One min now. | 14:58 |
| liuyulong | Any thing we need to take care? | 14:59 |
| lajoskatona | liuyulong: https://review.opendev.org/c/openstack/neutron-specs/+/767337 | 14:59 |
| lajoskatona | liuyulong: this is (draft) spec for bfd support, if you could check it would be really helpful | 14:59 |
| liuyulong | Got it, I will review it recent days. | 14:59 |
| lajoskatona | of course others' comments as well really helpful | 14:59 |
| lajoskatona | thanks | 14:59 |
| liuyulong | OK time is up. | 15:00 |
| liuyulong | #endmeeting | 15:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:00 | |
| openstack | Meeting ended Wed Dec 16 15:00:29 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-12-16-14.00.html | 15:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-12-16-14.00.txt | 15:00 |
| lajoskatona | Bye | 15:00 |
| rubasov | thank you | 15:00 |
| rubasov | bye | 15:00 |
| liuyulong | Thank you guys, bye. | 15:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-12-16-14.00.log.html | 15:00 |
| *** belmoreira has joined #openstack-meeting | 15:01 | |
| *** ralonsoh has quit IRC | 15:15 | |
| *** ralonsoh has joined #openstack-meeting | 15:15 | |
| *** ricolin_ has joined #openstack-meeting | 15:15 | |
| *** lpetrut has quit IRC | 15:23 | |
| *** lajoskatona has left #openstack-meeting | 15:28 | |
| *** ociuhandu has joined #openstack-meeting | 15:41 | |
| *** liuyulong has quit IRC | 15:42 | |
| *** _mlavalle_1 has quit IRC | 15:46 | |
| *** ociuhandu has quit IRC | 15:53 | |
| *** macz_ has joined #openstack-meeting | 16:04 | |
| *** ociuhandu has joined #openstack-meeting | 16:08 | |
| *** mlavalle has joined #openstack-meeting | 16:16 | |
| *** jawad_axd has quit IRC | 16:23 | |
| *** ricolin_ has quit IRC | 16:31 | |
| *** ociuhandu_ has joined #openstack-meeting | 16:49 | |
| *** ociuhandu has quit IRC | 16:52 | |
| *** dklyle has joined #openstack-meeting | 16:52 | |
| *** ociuhandu has joined #openstack-meeting | 16:53 | |
| *** ociuhandu_ has quit IRC | 16:57 | |
| *** ralonsoh is now known as ralonsoh|afk | 17:00 | |
| *** ociuhandu_ has joined #openstack-meeting | 17:02 | |
| *** ociuhandu has quit IRC | 17:05 | |
| *** ociuhandu_ has quit IRC | 17:07 | |
| *** gyee has joined #openstack-meeting | 17:07 | |
| *** rpittau is now known as rpittau|afk | 17:11 | |
| *** ociuhandu has joined #openstack-meeting | 17:18 | |
| *** ociuhandu has quit IRC | 17:23 | |
| *** dklyle has quit IRC | 17:30 | |
| *** dklyle has joined #openstack-meeting | 17:30 | |
| *** belmoreira has quit IRC | 17:34 | |
| *** jmasud has joined #openstack-meeting | 17:59 | |
| *** jmasud has quit IRC | 18:56 | |
| *** jmasud has joined #openstack-meeting | 19:11 | |
| *** jmasud has quit IRC | 19:30 | |
| *** jmasud has joined #openstack-meeting | 19:32 | |
| *** armstrong has joined #openstack-meeting | 19:43 | |
| *** e0ne has quit IRC | 19:52 | |
| *** lbragstad_ is now known as lbragstad | 19:54 | |
| *** ociuhandu has joined #openstack-meeting | 20:35 | |
| *** ociuhandu has quit IRC | 20:40 | |
| *** timburke has joined #openstack-meeting | 20:41 | |
| *** jmasud has quit IRC | 20:43 | |
| *** timburke has quit IRC | 20:48 | |
| *** SpamapS has quit IRC | 20:53 | |
| *** rfolco has quit IRC | 20:56 | |
| *** timburke has joined #openstack-meeting | 20:56 | |
| *** vishalmanchanda has quit IRC | 20:56 | |
| *** acoles has joined #openstack-meeting | 20:57 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Dec 16 21:00:16 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| kota_ | hi | 21:00 |
| seongsoocho | o/ | 21:00 |
| acoles | hi | 21:00 |
| rledisez | hi o/ | 21:00 |
| mattoliverau | o/ (only for a short while) | 21:01 |
| timburke | thank you all for coming -- i may be a little in-and-out; handling some childcare duties again | 21:02 |
| *** zaitcev has joined #openstack-meeting | 21:02 | |
| timburke | as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | first up | 21:02 |
| clayg | o/ | 21:02 |
| timburke | #topic end-of-year meeting schedule | 21:02 |
| *** openstack changes topic to "end-of-year meeting schedule (Meeting topic: swift)" | 21:02 | |
| clayg | how about: this is it. this is the last meeting of 2020. see you later 2020. | 21:03 |
| timburke | yeah, that :-) | 21:03 |
| mattoliverau | Lol, damn you 2020 :p | 21:03 |
| zaitcev | Red Hat enters the Christmas shutdown until January. | 21:04 |
| zaitcev | But we could have one last meeting if we wanted. | 21:04 |
| timburke | nah -- next meeting as Jan 6 seems perfectly reasonable | 21:04 |
| timburke | next topic | 21:05 |
| timburke | #topic audit watchers | 21:05 |
| *** openstack changes topic to "audit watchers (Meeting topic: swift)" | 21:05 | |
| timburke | we're so close! | 21:05 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/706653 | 21:06 |
| zaitcev | Well, the df was the last principal problem, I think. | 21:06 |
| zaitcev | Now, even if we get back to independent processees in the future, we can. | 21:06 |
| timburke | sorry that i haven't done another pass since my review last week | 21:06 |
| zaitcev | So, I'm honestly content with the final revision. | 21:07 |
| mattoliverau | I'll take another look this week, and hopefully add my +2 again | 21:07 |
| timburke | i am still a bit worried about the need to distinguish start/end for different workers when you've got more than one | 21:07 |
| zaitcev | mattoliverau: I added the doc that you asked. And it includes the Dark Data part after all. At first, I hoped to sweep it under the carpet and only use it in case of emergency at customer clusters. | 21:08 |
| mattoliverau | Nice | 21:08 |
| mattoliverau | timburke: I thought it added their name to the logger | 21:09 |
| zaitcev | Yes, you were right. Doing what I meant is how tribal memory is generated and its' wrong. | 21:09 |
| zaitcev | Yes, logs have prefixes | 21:09 |
| timburke | i think the device_key/worker-id was my last major concern, and i think we could remedy that with a new arg to end (and maybe start? i'm not actually sure how important it is there...) | 21:09 |
| zaitcev | And in fact, using watcher_name is better because that comes from proxy-server.conf, and is not the name of the Python class. | 21:10 |
| zaitcev | It's easy to add new arguments thanks to Sam's foresight. I was way more concerned about letting df stuck in there. But if you want to add some, easy to do in a follow-up. | 21:10 |
| timburke | mattoliverau, we get watcher prefixes, but we still don't have a way to distinguish between the same watcher spread across multiple workers | 21:11 |
| zaitcev | True, but why is it needed? | 21:11 |
| timburke | if you've got, say, 24-bay chassis and 4 workers per node (so each worker is responsible for 6 disks), when you go to dump stats to recon (say), you don't want to have all four workers writing a quarter of the full stats to the same keys | 21:12 |
| zaitcev | Workers are ephymeral, so... even if you know PID, all you can is kill the whole auditor and maybe restart it. Thinking as an operator here, the most important is to know which object triggered issues. | 21:12 |
| zaitcev | Oh, that way. | 21:13 |
| *** jmasud has joined #openstack-meeting | 21:13 | |
| timburke | i was realizing it as i was thinking through https://review.opendev.org/c/openstack/swift/+/766640 (watchers: Add a policy-stat watcher) | 21:13 |
| mattoliverau | Good spot. So we need to append a worker I'd or something? | 21:14 |
| timburke | i think so. soemthing along the lines of the device_key from https://github.com/openstack/swift/blob/2.26.0/swift/obj/auditor.py#L98-L103 | 21:15 |
| *** baojg has quit IRC | 21:15 | |
| zaitcev | Right... of course you can do os.getpid() now safely, but eh... My mental model was that you just increment all stats in some center place like memcached or Prometheus, and reset them at wall clock moment, like midnight on Mondays, rather than when auditor starts. That would give you comparable counts to watch trends. | 21:15 |
| *** baojg has joined #openstack-meeting | 21:16 | |
| zaitcev | If you insist on recon specifically, than a key is needed. | 21:16 |
| zaitcev | But I think you can add it in a follow-up. | 21:16 |
| timburke | good thought on os.getpid() -- forgot about that... might be sufficient | 21:17 |
| zaitcev | os.getpid() changes when auditor restarts, so you'll have a ton of old recon files in /var | 21:17 |
| zaitcev | well, if you reboot | 21:17 |
| timburke | i could also re-work it so that everything's always aggregated by device, and i write things to recon based on that. saves the same problem when worker count changes | 21:18 |
| zaitcev | Hmm. We never have 2 workers crawling the same device? | 21:18 |
| timburke | shouldn't; not for the same audit-type, anyway | 21:19 |
| *** jmasud has quit IRC | 21:19 | |
| *** baojg has quit IRC | 21:19 | |
| timburke | (seems like it'd make for more disk-thrashing) | 21:20 |
| *** baojg has joined #openstack-meeting | 21:20 | |
| zaitcev | Okay. I still think it's good for your and Matt's final review pass. | 21:21 |
| timburke | oh, i also need to think about the resumability of auditors... if they get interrupted, they pick up again more or less where they left off, right? hmm... | 21:21 |
| timburke | all right, i'll make sure to review it again within the next three weeks, and it sounds like mattoliverau will try to do the same | 21:21 |
| zaitcev | More or less. They write that json thing checkpoint. | 21:21 |
| timburke | #topic py3 fixes | 21:22 |
| *** openstack changes topic to "py3 fixes (Meeting topic: swift)" | 21:22 | |
| timburke | i was noticing that we've got a few py3 fixes that i wanted to raise attention for | 21:23 |
| zaitcev | see https://wiki.openstack.org/wiki/Swift/PriorityReviews | 21:23 |
| acoles | timburke: I'll volunteer to review https://review.opendev.org/c/openstack/swift/+/759075 if you like | 21:24 |
| timburke | thanks! it could use a test, but i know i've seen https://bugs.launchpad.net/swift/+bug/1900770 while running tests in my aio | 21:24 |
| openstack | Launchpad bug 1900770 in OpenStack Object Storage (swift) "py3 comparison troubles" [High,In progress] | 21:24 |
| *** larainema has quit IRC | 21:24 | |
| acoles | yup, maybe I'll put a test together, will do me good to re-educate myself about bad buckets | 21:25 |
| timburke | https://review.opendev.org/c/openstack/swift/+/765204 has been observed in the wild: https://bugs.launchpad.net/swift/+bug/1906289 | 21:25 |
| openstack | Launchpad bug 1906289 in OpenStack Object Storage (swift) "Uploading a large object (SLO) in foreign language characters using S3 browser results in 400 BadRequest - Error in completing multipart upload" [High,Confirmed] | 21:25 |
| * mattoliverau needs to take the car in for a service. | 21:26 | |
| mattoliverau | Gotta run, have a great one all o/ | 21:26 |
| zaitcev | mattoliverau: later | 21:27 |
| timburke | and https://review.opendev.org/c/openstack/swift/+/695781 is one that i'd mostly forgotten about, but can let bad utf-8-decoded-as-latin-1-encoded-as-utf-8 out to the client | 21:28 |
| zaitcev | right... are there any more besides these 3 | 21:28 |
| timburke | probably. those are the three i could remember ;-) | 21:29 |
| timburke | i *really* want to get to the point that i can feel confident in moving my prod clusters to py3 | 21:30 |
| timburke | moving on | 21:31 |
| timburke | #topic finishing sharding | 21:31 |
| *** openstack changes topic to "finishing sharding (Meeting topic: swift)" | 21:31 | |
| timburke | i came in late last week, so i wanted to check if there was any more discussion needed here, or if we've got a pretty good idea of what would be involved | 21:32 |
| zaitcev | I don't, but I sent David to investigate and teach me :-) | 21:33 |
| * zaitcev manages | 21:33 | |
| * zaitcev shuffles some more documents | 21:33 | |
| acoles | my summary was: 1. be able to recover from whatever could go wrong with auto-sharding (split brain) 2. do our best to prevent split-brain autosharding 3. get more confident about auto-shrinking | 21:34 |
| acoles | and suggested some current patches as a good starting place to get involved | 21:35 |
| timburke | sounds like a great plan :-) | 21:35 |
| acoles | e.g. the chain starting with https://review.opendev.org/c/openstack/swift/+/741721 | 21:35 |
| timburke | i won't worry then | 21:35 |
| acoles | haha | 21:35 |
| timburke | one last-minute topic | 21:35 |
| timburke | #topic stable gate | 21:35 |
| *** openstack changes topic to "stable gate (Meeting topic: swift)" | 21:35 | |
| acoles | BTW I updated priority reviews because I have squashed a couple of patches into https://review.opendev.org/c/openstack/swift/+/741721 | 21:36 |
| timburke | currently, things are fairly broken. mostly to do with pip-on-py2 trying to drag in a version of bandit that's py3-only | 21:36 |
| timburke | there are some patches to pin bandit, and at least some of them are mergeable, but it looks like there are some other requirements issues going on that complicate some branches | 21:38 |
| acoles | didn't a bandit fix merge? | 21:38 |
| timburke | i'm going to keep working on getting those fixed, just wanted to keep people apprised | 21:38 |
| acoles | https://review.opendev.org/c/openstack/swift/+/765883 ? | 21:39 |
| timburke | yeah, that at least got master moving. i might be able to do that for one or two of the more-recent branches, too | 21:40 |
| acoles | OIC there's a bunch of backport patches | 21:40 |
| timburke | someone proposed a fix back on pike through stein like https://review.opendev.org/c/openstack/swift/+/766495 | 21:41 |
| acoles | ok, is py2-constraints the right way though? | 21:41 |
| timburke | not all branches have a py2-constraints. though maybe we could introduce that? | 21:42 |
| timburke | fwiw, a cap in test-requirements.txt hits failures like https://zuul.opendev.org/t/openstack/build/284cdb5099114af685a4bfeb53b0d2ff/log/job-output.txt#520-522 on some branches | 21:43 |
| zaitcev | no ;python_version=='2.7 though for that bandit, I wonder whyt | 21:43 |
| *** TrevorV has quit IRC | 21:43 | |
| timburke | another option would be to just drop bandit from test-requirements.txt on (some?) stable branches -- we don't backport *that* much, and i'm not sure how much value we get from running bandit checks on stable | 21:44 |
| timburke | that's all i've got | 21:45 |
| timburke | #topic open discussion | 21:45 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:45 | |
| timburke | anything else we should bring up this week? | 21:45 |
| acoles | thanks timburke for all your work on the gate issues, it's incredibly valuable | 21:45 |
| timburke | anything i can do so you guys can focus on making swift great! | 21:46 |
| zaitcev | I don't understand how bandit even gets invoked. There's a [bandit] in tox.ini, but it's not in the list at the top or in any zuul jobs. | 21:46 |
| zaitcev | oh. maybe it's not no master branch. | 21:47 |
| acoles | its part of the pep8 tox env | 21:48 |
| timburke | iirc it's a flake8 plugin -- just install it and it'll start getting run as well | 21:48 |
| zaitcev | Oh, right. | 21:48 |
| zaitcev | Okay. I don't have anything else to discuss. Michelle managed to push through that patch for swift-init, but I have no idea if he's going to stick around. | 21:49 |
| timburke | oh yeah! looking at the bug report (https://bugs.launchpad.net/swift/+bug/1079075), i'm not actually sure that the title was really accurate... | 21:50 |
| openstack | Launchpad bug 1079075 in OpenStack Object Storage (swift) "swift-init should check if binary exists before starting" [Low,In progress] - Assigned to Michele Valsecchi (mvalsecc) | 21:50 |
| zaitcev | how so? He wanted not to have extra messages. | 21:51 |
| zaitcev | So, there's no change in function. | 21:51 |
| zaitcev | By "he" I mean the original reporter. | 21:51 |
| timburke | but the reason processes didn't start up wasn't actually missing binaries (afaict) | 21:52 |
| timburke | "fails because some *configuration files* are not existent" | 21:52 |
| zaitcev | well yeah | 21:53 |
| zaitcev | Someone removed both configurations and binaries | 21:53 |
| zaitcev | You know, I used to try that crap too. It was a mistake. But our RPM packages used to be very fine-grained like that. | 21:53 |
| timburke | *shrug* if it's still a problem, we'll get a new bug report ;-) | 21:54 |
| zaitcev | But then we started to share a bunch of code across types of services. For example, GET on accounts and containers use a function that's not in common code, but in container IIRC. So, when someone installs just one type of service, it blows up. | 21:54 |
| zaitcev | I had to give up and create a common package that contains all of the code, no matter where it belongs. | 21:55 |
| zaitcev | So the logic was, if swift-init starts checking for binaries, it would not attempt to run something that has no configuration. | 21:55 |
| zaitcev | see | 21:55 |
| zaitcev | So, I think it was an appropriate patch and it was okay for us to include it. | 21:56 |
| zaitcev | Well, its value was very low. Only helps people who do this fine-grained installation. | 21:57 |
| timburke | cool. yeah, i'm not worried about the patch; i do think it makes swift better. just thinking about whether the bug should be closed or not | 21:57 |
| zaitcev | Oh, Tim | 21:57 |
| zaitcev | Yeah, of course close it. | 21:57 |
| zaitcev | One question: when are we going to drop py2? | 21:57 |
| timburke | great question! | 21:58 |
| timburke | i don't know! | 21:58 |
| * zaitcev backrolls in nagare kaiten | 21:58 | |
| seongsoocho | lol | 21:58 |
| timburke | i feel like with train/ussuri we saw a decent number of new clusters stood up running py3-only | 21:59 |
| timburke | and more recently in victoria/wallaby we're seeing clusters that were on py2 migrate to py3 | 21:59 |
| zaitcev | I'm sure projects other than bandit are going to put pressure on us. I think eventlet is the worst of them. | 21:59 |
| timburke | yup -- it's a growing worry for me too -- see https://github.com/eventlet/eventlet/pull/665 for their deprecation (i don't think they've dropped it yet, but it's just a matter of time) | 22:01 |
| zaitcev | Red Hat offers 7 years on some of the supported releases, but they have a controlled set of packages + backported patches. But in the trunk it's kind of a pain. | 22:01 |
| timburke | thinking mostly selfishly, i'll say "not until i've migrated off of py2 myself" ;-) | 22:02 |
| zaitcev | so, you have trunk on py2? | 22:02 |
| timburke | my prod clusters run py2, yes | 22:03 |
| zaitcev | What's the OS? Some kind of old Ubuntu I presume. | 22:03 |
| timburke | (home cluster's py3 though!) | 22:03 |
| timburke | centos7, mainly | 22:03 |
| timburke | i think we've got some legacy customers still on ubuntu | 22:03 |
| zaitcev | Right, that is py2. | 22:03 |
| zaitcev | OK thanks for the answer. | 22:03 |
| timburke | we package our own python; system python is a pain | 22:04 |
| *** dklyle has quit IRC | 22:04 | |
| timburke | all right, sorry, i let us go over time. thank you all for coming, and thank you for working on swift! | 22:04 |
| timburke | #endmeeting | 22:04 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:04 | |
| openstack | Meeting ended Wed Dec 16 22:04:37 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:04 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-12-16-21.00.html | 22:04 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-12-16-21.00.txt | 22:04 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-12-16-21.00.log.html | 22:04 |
| *** zaitcev has left #openstack-meeting | 22:04 | |
| *** acoles has left #openstack-meeting | 22:05 | |
| *** ralonsoh|afk has quit IRC | 22:09 | |
| *** timburke has quit IRC | 22:10 | |
| *** zaitcev has joined #openstack-meeting | 22:14 | |
| *** rcernin has joined #openstack-meeting | 22:17 | |
| *** dklyle has joined #openstack-meeting | 22:21 | |
| *** raildo has quit IRC | 22:30 | |
| *** slaweq has quit IRC | 22:42 | |
| *** armstrong has quit IRC | 23:02 | |
| *** haleyb is now known as haleyb|away | 23:18 | |
| *** baojg has quit IRC | 23:25 | |
| *** baojg has joined #openstack-meeting | 23:26 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!