Wednesday, 2020-04-08

*** rbudden has quit IRC00:08
*** ttsiouts has joined #openstack-meeting00:52
*** jmasud has quit IRC00:55
*** jmasud has joined #openstack-meeting00:56
*** gyee has quit IRC01:07
*** Liang__ has joined #openstack-meeting01:13
*** ysandeep|away is now known as ysandeep|rover01:17
*** ttsiouts has quit IRC01:25
*** ricolin has joined #openstack-meeting01:58
*** ysandeep|rover is now known as ysandeep|brb02:10
*** apetrich has quit IRC02:11
*** yamamoto has joined #openstack-meeting02:14
*** moguimar has quit IRC02:14
*** ykatabam has quit IRC02:51
*** ykatabam has joined #openstack-meeting02:57
*** brinzhang_ has quit IRC02:59
*** brinzhang_ has joined #openstack-meeting02:59
*** rh-jelabarre has quit IRC03:00
*** ttsiouts has joined #openstack-meeting03:22
*** psachin has joined #openstack-meeting03:28
*** ysandeep|brb is now known as ysandeep|rover03:42
*** yamamoto has quit IRC03:42
*** yamamoto has joined #openstack-meeting03:44
*** yaawang__ has quit IRC03:53
*** vishalmanchanda has joined #openstack-meeting03:53
*** ttsiouts has quit IRC03:56
*** andrebeltrami has quit IRC04:26
*** yamamoto has quit IRC04:42
*** yamamoto has joined #openstack-meeting04:47
*** ociuhandu has joined #openstack-meeting04:47
*** zaneb has quit IRC04:48
*** zaneb has joined #openstack-meeting04:49
*** ociuhandu has quit IRC04:51
*** jmasud has quit IRC04:53
*** jmasud has joined #openstack-meeting04:56
*** brinzhang has joined #openstack-meeting05:08
*** brinzhang_ has quit IRC05:11
*** armax has quit IRC05:27
*** armax has joined #openstack-meeting05:28
*** igordc has quit IRC05:32
*** armax has quit IRC05:33
*** ttsiouts has joined #openstack-meeting05:53
*** yaawang has joined #openstack-meeting05:58
*** masahito has joined #openstack-meeting06:05
*** armax has joined #openstack-meeting06:13
*** armax has quit IRC06:18
*** links has joined #openstack-meeting06:20
*** ttsiouts has quit IRC06:27
*** gtema has joined #openstack-meeting06:30
*** gtema has quit IRC06:38
*** gtema has joined #openstack-meeting06:40
*** gtema has quit IRC06:45
*** ttsiouts has joined #openstack-meeting06:45
*** gtema has joined #openstack-meeting06:45
*** dklyle has quit IRC06:48
*** yamamoto has quit IRC06:52
*** slaweq_ has joined #openstack-meeting06:57
*** maciejjozefczyk has joined #openstack-meeting07:00
*** yaawang has quit IRC07:04
*** yaawang has joined #openstack-meeting07:05
*** yamamoto has joined #openstack-meeting07:05
*** lpetrut has joined #openstack-meeting07:09
*** belmoreira has joined #openstack-meeting07:16
*** rpittau|afk is now known as rpittau07:17
*** ysandeep|rover is now known as ysandeep|lunch07:24
*** ralonsoh has joined #openstack-meeting07:33
*** e0ne has joined #openstack-meeting07:38
*** ociuhandu has joined #openstack-meeting07:44
*** gtema has quit IRC07:44
*** brinzhang_ has joined #openstack-meeting07:48
*** brinzhang has quit IRC07:51
*** gtema has joined #openstack-meeting07:58
ttxNB: large scale SIG meeting in #openstack-meeting-3 now08:01
*** dmellado has quit IRC08:01
ttxmasahito: belmoreira ^08:01
masahitoI completely forgot we changed the time slot :-)08:02
*** ociuhandu has quit IRC08:02
ttxSorry we had to change the channel too08:02
ttxso -> #openstack-meeting-308:03
masahitogot it.08:03
*** ociuhandu has joined #openstack-meeting08:03
ttxjiaopengju1: ^08:03
*** gtema has quit IRC08:04
*** dmellado has joined #openstack-meeting08:05
ttxetp: ^08:05
*** gtema has joined #openstack-meeting08:07
*** ociuhandu has quit IRC08:08
*** brinzhang has joined #openstack-meeting08:10
*** witek has joined #openstack-meeting08:11
*** brinzhang_ has quit IRC08:13
*** brinzhang_ has joined #openstack-meeting08:14
*** brinzhang has quit IRC08:15
*** rcernin has quit IRC08:27
*** ysandeep|lunch is now known as ysandeep|rover08:29
*** dmacpher has quit IRC08:32
*** dmacpher has joined #openstack-meeting08:33
*** witek has quit IRC08:33
*** slaweq_ is now known as slaweq08:36
*** ociuhandu has joined #openstack-meeting08:45
*** yaawang has quit IRC08:48
*** yaawang has joined #openstack-meeting08:50
*** ociuhandu has quit IRC08:50
*** ociuhandu has joined #openstack-meeting08:52
*** jmasud has quit IRC08:54
*** jmasud has joined #openstack-meeting08:56
*** ociuhandu has quit IRC09:04
*** ociuhandu has joined #openstack-meeting09:04
*** brinzhang has joined #openstack-meeting09:06
*** brinzhang_ has quit IRC09:09
*** ociuhandu has quit IRC09:09
*** nitinuikey has joined #openstack-meeting09:11
*** yaawang has quit IRC09:14
*** yaawang has joined #openstack-meeting09:16
*** dmacpher has quit IRC09:20
*** dmacpher has joined #openstack-meeting09:23
*** ricolin_ has quit IRC09:29
*** nitinuikey has quit IRC09:30
*** ricolin_ has joined #openstack-meeting09:30
*** ricolin has quit IRC09:30
*** yaawang has quit IRC09:35
*** yaawang has joined #openstack-meeting09:36
*** Liang__ has quit IRC09:38
*** ociuhandu has joined #openstack-meeting09:40
*** ykatabam has quit IRC09:45
*** brinzhang_ has joined #openstack-meeting09:52
*** brinzhang has quit IRC09:55
*** ricolin_ has quit IRC09:57
*** ricolin_ has joined #openstack-meeting09:57
*** ricolin has joined #openstack-meeting10:00
*** ricolin_ has quit IRC10:02
*** masahito has quit IRC10:04
*** links has quit IRC10:08
*** yamamoto has quit IRC10:13
*** slaweq has quit IRC10:23
*** rpittau is now known as rpittau|bbl10:24
*** slaweq has joined #openstack-meeting10:25
*** ysandeep|rover is now known as ysandeep|afk10:30
*** links has joined #openstack-meeting10:31
*** yamamoto has joined #openstack-meeting10:33
*** yamamoto has quit IRC10:34
*** masahito has joined #openstack-meeting10:37
*** brinzhang has joined #openstack-meeting10:37
*** ricolin has quit IRC10:39
*** oneswig has joined #openstack-meeting10:39
*** brinzhang_ has quit IRC10:40
*** masahito has quit IRC10:41
*** yamamoto has joined #openstack-meeting10:53
*** ykatabam has joined #openstack-meeting10:55
*** brinzhang_ has joined #openstack-meeting10:57
*** e0ne has quit IRC10:58
*** e0ne has joined #openstack-meeting10:58
*** janders has joined #openstack-meeting11:00
oneswig#startmeeting scientific-sig11:00
openstackMeeting started Wed Apr  8 11:00:12 2020 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.11:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"11:00
openstackThe meeting name has been set to 'scientific_sig'11:00
*** dh3 has joined #openstack-meeting11:00
oneswig#topic greetings11:00
*** openstack changes topic to "greetings (Meeting topic: scientific-sig)"11:00
oneswighi janders11:00
*** brinzhang has quit IRC11:00
oneswigWhat's new with you?11:00
jandersinteresting cx6-eth breakages for AOB :)11:01
oneswigHi dh311:01
oneswigalready looking forward to it...11:01
oneswiggreetings verdurin11:02
oneswigLet's get the show on the road11:02
oneswig#topic Kolla user group11:02
*** openstack changes topic to "Kolla user group (Meeting topic: scientific-sig)"11:02
oneswigOK, only new item on this week's agenda11:02
oneswigFor Kolla users there's a new effort to get users and operators talking with developers11:03
oneswig#link Kolla user group thread
*** gtema has left #openstack-meeting11:03
oneswigIf you're using Kolla, hopefully it will be a worthwhile discussion11:04
verdurinYes, hoping to attend.11:06
oneswigIs there something similar for TripleO - I know there are feedback sessions at the summits11:06
jandersI call it tripleno :P11:07
janders(can be made work though :)  )11:07
oneswigOK, good to know.11:08
oneswigI've been to quite a few operator feedback sessions in different forms, hopefully this will be productive.11:09
oneswigMove on?11:09
oneswig#topic AOB11:09
*** openstack changes topic to "AOB (Meeting topic: scientific-sig)"11:09
jandersok... i promised some cx6-eth stories11:10
janderswe used to have an issue getting VPI to work. All IB config *just worked* but eth-ib less so11:10
jandersa FW version came out some time back that was supposed to fix it, we started testing this a couple weeks back11:10
janderscatch: it seems for VPI to work, eth needs to be on port1 and ib on port211:10
jandersand we're wired the other way round and have limited DC access due to lockdown so it's tricky to swap it over11:11
verdurinjanders: ???11:11
jandersin any case - it does malfunction in an interesting way11:11
janderseth link comes up no worries11:11
janderstill... you try to use it with LB or ovs11:11
janderswhen it just drops any traffic with MAC address not matching physical point11:11
jandersour friends at Mellanox are looking at it but that's where things are at11:12
jandersive seen a few issues with VPI but not this one :)11:12
jandershave you guys seen anything like that?11:12
oneswigI have... kind of...11:12
janderswhat card was that on?11:13
jandersalso a cx6?11:13
jandersor something different?11:13
oneswigConnectX-5, probably different issue.11:13
oneswigThis is SRIOV over a bonded LAG11:13
*** janders has quit IRC11:13
*** janders has joined #openstack-meeting11:14
janderssorry got dropped out11:14
janders(not VPIs fault :)  )11:14
jandersoneswig: what was the problem on cx5?11:14
oneswigAha, well it was related to VF-LAG and OVS11:14
belmoreirahi, just joined11:15
oneswigI got into a situation where I can receive broadcast traffic but not unicast11:15
oneswigHi belmoreira11:15
oneswigAre you using OVS 2.12?11:15
janderswas that only specific to VFs, or would it impact traffic across the board?11:15
oneswigSaw it first in VFs.  When I installed OVS 2.12 it affected both11:16
*** apetrich has joined #openstack-meeting11:16
oneswigHaven't investigated in sufficient detail yet but it might not be related11:16
janders2.9.0 is the version11:17
oneswigBTW The RDO build of OVS 2.12 apparently is quite old compared to recent code on that branch11:17
jandersthis is OSP13/Queens based project11:19
jandersso it may be worth re-testing with latest ovs11:19
jandershas upgrading ovs helped in your case?11:19
oneswigOVS troubleshooting tools are a dark art all to their own11:19
jandersyeah we may need to sacrifice a packet or two to the ovs gods...11:20
oneswigjanders: not yet, 2.11->2.12 caused many problems on first attempt11:20
oneswigNeed to go back and do it again, with better preparation11:20
oneswigbelmoreira: was talking with someone recently about external hardware inventories and Ironic.  I was thinking about Arne's work on that.  Did CERN settle on a hardware inventory management system?11:22
belmoreira not yet11:22
oneswigbelmoreira: so what's new at CERN?11:24
belmoreira:) related with ironic, we are now moving into conductor groups11:24
*** janders has quit IRC11:25
belmoreirathis allows us to split the ironic infrastructure more or less like cells11:25
oneswighow many nodes are you managing now?11:25
*** janders has joined #openstack-meeting11:27
oneswignice work :-)11:28
oneswigHow do you size the number of nodes managed by a conductor group?11:29
*** ykatabam has quit IRC11:30
belmoreiraWe introduced the first conductor group with ~600 nodes11:30
belmoreirathe metric that we use is the time that the resource tracker takes to run11:30
*** ysandeep|afk is now known as ysandeep|rover11:31
oneswigHave you reduced how often it runs?  I think I remember you changed it to run every few hours?11:32
belmoreirayes, but with the latest versions it impacts node state updates11:33
belmoreirawith 600 nodes it takes around ~15 minutes to run11:33
*** ociuhandu has quit IRC11:34
oneswigOn a related subject, someone in our team mentioned the software-raid deployment has improved flexibility now.11:34
belmoreirawe are discussing in having conductor groups with 500 nodes11:34
*** ociuhandu has joined #openstack-meeting11:35
belmoreiraI think the raid work is already merged upstream. For details Arne is the best person to contact11:36
oneswiggreat, thanks.11:37
*** ociuhandu has quit IRC11:39
oneswigOn a different subject - dh3: has there been any further development on your work integrating Lustre with OpenStack clients?11:40
dh3oneswig: "a bit" - it is fighting for our attention with other projects, and we were hitting some weird Lustre errors which made upgrading the client version more urgent (we are settling on 2.12.4 now) but the users still want it11:42
dh3the general approach of building an image with Lustre already in it is working though, we have several groups using it as poc11:44
dh3(can't escape posix :)  )11:44
oneswigdh3: I saw you guys are now famous - some familiar faces here
*** links has quit IRC11:44
*** links has joined #openstack-meeting11:45
*** rh-jelabarre has joined #openstack-meeting11:45
dh3haha, I knew there was something in the works but I hadn't seen that. we are in contract (re)negotiation land now, different kind of fun11:45
oneswiggood luck with that! :-)11:46
*** brinzhang has joined #openstack-meeting11:47
dh3thanks :)11:47
oneswigdh3: there has been some work recently with Cambridge Uni on improved Ansible automation for Lustre.  Might be worth sharing notes with you on that.11:49
dh3oneswig: we'd be interested. We don't let our main site Ansible touch Lustre servers much at the moment, trying to keep them as "black box" - only to install stats scripts and (soon) to control iptables11:50
*** brinzhang_ has quit IRC11:50
*** ociuhandu has joined #openstack-meeting11:51
oneswigdh3: ok sounds good.11:51
oneswigI didn't have more for today - anyone else?11:52
*** raildo has joined #openstack-meeting11:53
oneswigOK y'all, good to see everyone, thanks for joining :-)11:54
*** openstack changes topic to "OpenStack Meetings ||"11:54
dh3take care11:54
openstackMeeting ended Wed Apr  8 11:54:41 2020 UTC.  Information about MeetBot at . (v 0.1.4)11:54
openstackMinutes (text):
oneswigsame to you!11:54
dh3bye for now11:55
*** janders has quit IRC11:57
*** dh3 has quit IRC12:01
*** ociuhandu has quit IRC12:10
*** ociuhandu has joined #openstack-meeting12:11
*** ociuhandu has quit IRC12:16
*** brinzhang has quit IRC12:19
*** JangwonLee_ has joined #openstack-meeting12:21
*** rpittau|bbl is now known as rpittau12:23
*** brinzhang has joined #openstack-meeting12:23
*** brinzhang has quit IRC12:23
*** JangwonLee__ has quit IRC12:25
*** andrebeltrami has joined #openstack-meeting12:40
*** ttsiouts has quit IRC12:53
*** jmasud has quit IRC12:54
*** jmasud has joined #openstack-meeting12:56
*** ttsiouts has joined #openstack-meeting12:57
*** ricolin has joined #openstack-meeting12:57
*** oneswig has quit IRC13:00
*** JangwonLee__ has joined #openstack-meeting13:02
*** JangwonLee_ has quit IRC13:05
*** ysandeep|rover is now known as ysandeep|away13:25
*** eharney has joined #openstack-meeting13:28
*** ociuhandu has joined #openstack-meeting13:33
*** yamamoto has quit IRC13:50
*** jraju__ has joined #openstack-meeting13:50
*** Liang__ has joined #openstack-meeting13:50
*** links has quit IRC13:52
*** liuyulong has joined #openstack-meeting13:59
liuyulong#startmeeting neutron_l314:00
openstackMeeting started Wed Apr  8 14:00:50 2020 UTC and is due to finish in 60 minutes.  The chair is liuyulong. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: neutron_l3)"14:00
*** Liang__ is now known as LiangFang14:00
openstackThe meeting name has been set to 'neutron_l3'14:00
* slaweq will be lurking here due to some internal meeting running in parallel14:01
*** belmoreira has quit IRC14:02
* haleyb in similar situation14:02
liuyulongOK, let's start14:02
liuyulong#topic Announcements14:03
*** openstack changes topic to "Announcements (Meeting topic: neutron_l3)"14:03
liuyulongActually no announcement from me today.14:04
liuyulongthe team announcement yesterday.14:05
liuyulongOK, then let14:05
liuyulongskip this section.14:05
liuyulong#topic Bugs14:05
*** openstack changes topic to "Bugs (Meeting topic: neutron_l3)"14:05
liuyulong^ the bug list from our deputy14:06
*** ttsiouts has quit IRC14:06
*** ttsiouts has joined #openstack-meeting14:06
liuyulongMost of them have been discussed last week, so the first one is14:07
openstackLaunchpad bug 1870313 in neutron ""send_ip_addr_adv_notif" can't use eventlet when called from "keepalived_state_change"" [High,Fix released] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)14:07
liuyulongGood news, it is also fixed.14:07
*** ykatabam has joined #openstack-meeting14:08
liuyulongIt's a really quick one, 5 days between the bug reporting and fix. : )14:09
openstackLaunchpad bug 1869887 in neutron "L3 DVR ARP population gets incorrect MAC address in some cases" [High,Fix released] - Assigned to Slawek Kaplonski (slaweq)14:09
liuyulongThis one is also fixed, and stable backports are also on the way now.14:10
liuyulongNext one:14:11
openstackLaunchpad bug 1869808 in neutron "reboot neutron-ovs-agent introduces a short interrupt of vlan traffic" [Low,New]14:11
liuyulongIt is something related to floating IP network.14:11
liuyulongWe meet some packet drops/lose during ovs agent restart.14:13
liuyulongBut from our test result, it is acceptable。14:13
slaweqliuyulong: but in bug report there is nothing about FIPs14:14
slaweqit seems for me like it is problem with fixed ips14:14
liuyulong"ovs-ofctl dump-flows br-floating"14:14
liuyulongIt has a bridge name br-floating.14:14
slaweqliuyulong: sure, name of bridge is like that14:15
slaweqbut look above, ping to fixed ip is shown there14:15
liuyulongTCP has retransmission mechanism and UDP is unreliable. So I may say one or two packets lose is not too much bad.14:16
liuyulongslaweq, maybe it is just a fake floating address.14:16
slaweq22d55077-b1b5-452e-8eba-cbcd2d1514a8 | test-1-1 | ACTIVE | vlan105= | Cirros 0.4.0 64-bit | m1.tiny |14:16
liuyulongThe testing is icmp14:17
slaweqit's from this bug report14:17
slaweqthis is fixed ip in vlan network14:17
*** yamamoto has joined #openstack-meeting14:17
slaweqthere is nothing related to L3 and floating IPs there IMO14:17
haleybi didn't think 1-2 packets lost was an issue either...14:18
liuyulongOne min, let me check another fix. Maybe it is related.14:19
liuyulongMaybe their local deployment miss this fix.14:22
liuyulongI will leave this link to the LP bug.14:23
haleybliuyulong: that's a good find :)14:23
openstackLaunchpad bug 1870114 in neutron "Trunk subports aren't treated as dvr serviced ports" [Medium,Fix released] - Assigned to Slawek Kaplonski (slaweq)14:26
liuyulongThis one is also fixed, and all stable backports are started now.14:26
liuyulongIt's also a quick one.14:27
liuyulongOK, no more bugs from deputy bug list.14:30
liuyulongLet's have a quick look of the LP list.14:30
liuyulongAlright, looks like a quiet week, all bugs are mentioned here.14:32
liuyulongAnyone has some other bugs?14:33
liuyulongOK, let's move on.14:34
liuyulong#topic OVN_L314:34
*** openstack changes topic to "OVN_L3 (Meeting topic: neutron_l3)"14:34
liuyulongAllow me to continue the topic last week14:34
liuyulongFor one OVN DB node, aka just run ovn-northd, the devstack local conf is not working.14:35
maciejjozefczykliuyulong, can you point us the devstack local.conf that you're using?14:36
liuyulongJust set "enable_service ovn-northd" to the devstack local.conf could not start a node with ovn-northd only.14:36
liuyulongI have 5 nodes, my plan is to run "one node with ovn-northd only, two chassises for compute, two for gateway".14:38
maciejjozefczykhmm, I *think* that also 'Q_AGENT=ovn' needs to be set in order to start using OVN14:38
liuyulongmaciejjozefczyk, ^ the local.conf14:40
ralonsohliuyulong, this is the one I use14:40
ralonsoh(there are many lines commented, just for testing)14:40
maciejjozefczykliuyulong, hmm, that strange. I use as reference for nodes that don't have NBDB installed14:41
liuyulongYes, that's my question, I do not need to run ovn-controller in a DB only node.14:41
maciejjozefczykliuyulong, can you please fill a bug for it? I think that we can easly find it14:41
maciejjozefczykliuyulong, in your configuration you need to specify also: OVN_SB_REMOTE=tcp:$SERVICE_HOST:664214:42
liuyulongBut the I guess something in the ovs installation scripts may have some miss condition.14:42
maciejjozefczykwhere SERVICE_HOST is the NBDB host14:42
liuyulongmaciejjozefczyk, the error is not that, it's something like "could not find the ovsdb service".14:43
liuyulongSomething like that, ralated to ovsdb, maybe this file "/opt/stack/neutron/devstack/lib/ovs"14:44
liuyulongOr this file "/opt/stack/neutron/devstack/lib/ovn_agent", it has a start ovs function.14:45
maciejjozefczykliuyulong, what service shows the 'could not find the ovsdb service' error?14:45
*** igordc has joined #openstack-meeting14:45
liuyulongI have not deeply looked into it. But just add "ovn-controller" to the local.conf, then everything works fine.14:45
*** liuyulong has left #openstack-meeting14:45
*** liuyulong has joined #openstack-meeting14:46
maciejjozefczykliuyulong, hmm, please fill a bug and I'll check it okey?14:46
liuyulongmaciejjozefczyk, just run "", it will be failed, and error logs in that.14:46
maciejjozefczykwith the example you showed14:46
maciejjozefczykliuyulong, ok14:46
liuyulongmaciejjozefczyk, sure14:46
*** lpetrut has quit IRC14:46
liuyulong^ yes, I run devstack to test this. : )14:47
maciejjozefczykTerry has some potential improvements for this, but it will be proposed in next patches, to limit the number of additional calculations14:48
maciejjozefczykBut for now its broken... so we decided to merge what we have now14:48
maciejjozefczykIt is also connected to ongoing improvements in Core OVN, that are under investigation, so thats why14:51
liuyulongUse "ovn-nbctl list Logical_Router_Port" to check the reschedule, right?14:51
maciejjozefczykor ovn-nbctl lrp-get-gateway-chassis <LRP>14:53
maciejjozefczykthat will list candidates and priorities for the gateway14:53
*** ykatabam has quit IRC14:55
liuyulongCool, this command has a more clear outputs. Otherwise I need to list chassis to compare the "gateway_chassis" column.14:56
liuyulongOK, thank you, no more ovn related stuff from me14:57
liuyulongAny updates?14:58
maciejjozefczykIn related to L3 - no14:58
*** moguimar has joined #openstack-meeting14:59
liuyulongok, let's move on14:59
liuyulong#topic On demand agenda14:59
*** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)"14:59
liuyulongOne minute left...14:59
*** dklyle has joined #openstack-meeting15:00
liuyulongLet's just have a cup of tea now...15:00
liuyulongOK, thank you guys for attending.15:00
*** openstack changes topic to "OpenStack Meetings ||"15:00
openstackMeeting ended Wed Apr  8 15:00:36 2020 UTC.  Information about MeetBot at . (v 0.1.4)15:00
openstackMinutes (text):
*** armax has joined #openstack-meeting15:01
*** LiangFang has quit IRC15:01
*** TrevorV has joined #openstack-meeting15:02
*** priteau has joined #openstack-meeting15:06
*** yamamoto has quit IRC15:06
*** gyee has joined #openstack-meeting15:13
*** ykatabam has joined #openstack-meeting15:14
*** yamamoto has joined #openstack-meeting15:15
*** yamamoto has quit IRC15:15
*** yamamoto has joined #openstack-meeting15:55
*** ttsiouts has quit IRC16:00
*** yamamoto has quit IRC16:09
*** ociuhandu has quit IRC16:16
*** ociuhandu has joined #openstack-meeting16:17
*** ociuhandu has quit IRC16:22
*** rpittau is now known as rpittau|afk16:23
*** psachin has quit IRC16:24
*** ykatabam has quit IRC16:50
*** jmasud has quit IRC16:54
*** jmasud has joined #openstack-meeting16:55
*** bbowen has quit IRC16:56
*** bbowen has joined #openstack-meeting16:56
*** ociuhandu has joined #openstack-meeting17:10
*** priteau has quit IRC17:15
*** ociuhandu has quit IRC17:16
*** hagridaaron has joined #openstack-meeting17:34
*** ralonsoh has quit IRC18:01
*** diablo_rojo has quit IRC18:02
*** diablo_rojo has joined #openstack-meeting18:06
*** e0ne has quit IRC18:21
*** ociuhandu has joined #openstack-meeting18:26
*** diablo_rojo has quit IRC18:53
*** slaweq_ has joined #openstack-meeting18:59
*** slaweq has quit IRC19:00
*** hagridaaron has quit IRC19:17
*** maciejjozefczyk has quit IRC19:24
*** vishalmanchanda has quit IRC19:32
*** Trevor_V has joined #openstack-meeting19:48
*** Lucas_Gray has joined #openstack-meeting19:50
*** TrevorV has quit IRC19:51
*** yamamoto has joined #openstack-meeting20:07
*** yamamoto has quit IRC20:13
*** igordc has quit IRC20:17
*** Lucas_Gray has quit IRC20:18
*** igordc has joined #openstack-meeting20:18
*** Lucas_Gray has joined #openstack-meeting20:19
*** hagridaaron has joined #openstack-meeting20:23
*** Lucas_Gray has quit IRC20:33
*** moguimar has quit IRC20:39
*** jmasud has quit IRC20:54
*** jmasud has joined #openstack-meeting20:56
timburke#startmeeting swift21:00
openstackMeeting started Wed Apr  8 21:00:03 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
*** patchbot has joined #openstack-meeting21:00
*** Lucas_Gray has joined #openstack-meeting21:01
claygi may have to duck out early21:01
timburkeas usual, the agenda's at
clayglucky I'm only here for the lawlz21:01
timburkeno worries21:01
timburkeyeah, "updates" may be short ;-)21:02
timburkei know *i'm* having a hard time getting much accomplished21:02
timburkeso, some announcements!21:02
timburke#topic virtual PTG21:02
*** openstack changes topic to "virtual PTG (Meeting topic: swift)"21:02
timburkei set up an etherpad to collect topics21:02
timburkethat's the extent of what planning i've done, though -- i'm having a hard enough time thinking two days into the future, much less two months21:03
timburkei know there are mailing list threads and etherpads going around trying to aggregate openstack's collective wisdom around doing virtual hackathons, though21:04
timburke#topic TC election21:04
*** openstack changes topic to "TC election (Meeting topic: swift)"21:04
timburkethere's more candidates than open seats this time!21:04
timburkeso please go read people's stances and vote21:05
mattoliverauOh sorry I'm late, daylight savings changed in Oz which threw me, things are now an hour earlier. o/21:05
timburkenot to worry!21:06
mattoliverauI voted yesterday. Get in early so I don't forget :)21:06
claygtimburke: I'm scared and confused by the TC vote - I have enough hard time just figuring out my texas rail road commissioner - can you just tell us who to vote for?21:06
claygmattoliverau: scratch that - can YOU tell me who to vote for?21:06
timburkemattoliverau, smart -- i still need to get step one done (read candidacy emails)21:06
*** jraju__ has quit IRC21:07
timburkei'll keep you posted clayg ;-)21:07
mattoliverauYeah, there is a link to those. IE in hit which made it easier21:07
mattoliverauIf I was fully awake and near a computer I'd find the links21:07
mattoliverau*in git21:08
timburkethe top section after timetables has links to candidates' announcements21:08
timburke(speaking of elections, there was a snafu so my candidacy isn't acutally listed there, but i will be continuing as ptl)21:09
timburkethanks for the reminders kota_ and notmyname, but i didn't quite get everything sorted in time ;-)21:10
timburke#topic ironic as a top-level opendev project21:10
*** openstack changes topic to "ironic as a top-level opendev project (Meeting topic: swift)"21:10
mattoliverauOh I was going to mention this21:11
timburkethere's an interesting thread about ironic considering moving toward being more independent from openstack21:11
timburkesoemthing worth following, for sure21:11
claygtimburke: thanks!21:12
timburkeok, on to a couple more swift-specific things21:13
timburke#topic python-swiftclient gate21:13
*** openstack changes topic to "python-swiftclient gate (Meeting topic: swift)"21:13
timburkeit's broken!21:13
timburkewe haven't noticed, because we so rarely propose changes there ;-)21:13
timburkei *think* should fix it, though21:14
patchbotpatch 718309 - python-swiftclient - Blacklist stestr 3.0.0 - 2 patch sets21:14
timburkein general, the further along the py3-only path everything else goes, the more breakage we'll likely see21:14
timburkethough afaict, *swift's* current gate woes aren't actually related to that21:15
timburkei feel like i've been typing "recheck" a lot lately, though :-(21:15
rlediseztimburke: how long do you think it's gonna be sustainable to maintain a py2 code? is there any risk we have to drop py2 before the date we decided?21:16
timburkebiggest risk is in py3-only bugs21:17
timburkethings like kinda scare the bejesus out of me21:17
patchbotpatch 707312 - swift - py3: stop barfing on message/rfc822 Content-Types (MERGED) - 2 patch sets21:17
claygrledisez: getting rid of py2 support is the only interesting thing about supporting py3 - I only want to support both while we're working on switching our prod over to py321:19
timburkewell, no -- *really*, scares me. the stupid rfc822 thing was mostly annoying21:19
patchbotpatch 682173 - swift - proxy: Don't trust Content-Length for chunked tran... (MERGED) - 4 patch sets21:19
claygthe DAY after we finally get py3 working I want to remove py2 🤣21:19
claygmaybe we *should* draw a line in the sand and say "swift can't support py2 anymore because py2 is dead; support will be removed soon"21:19
claygeven if we know we can't remove it until at least WE manage to get off it21:19
claygit's not optional anymore - we waited as long as we could21:20
timburkei should make sure we have a question related to py2 vs py3 in the user survey...21:20
claygbut py3 is going to be amazing - we'll always have bugs to fix - the sooner we're running in prod the sooner we find them 💪21:20
rledisezclayg: totally agree. i'm planning on deplying some py3 on prod in the next few months. but it's gonna be a very slow process…21:20
timburkei look forward to the bug reports ;-)21:20
claygrledisez: same, very slow - it's not the upgrade - it's the eco-system of code that goes along with it 😢21:20
timburkeunwinding the stack a little, speaking of how difficult it's been lately to get patches through...21:21
timburke#topic ussuri releases (both client and server)21:21
*** openstack changes topic to "ussuri releases (both client and server) (Meeting topic: swift)"21:21
timburkei should start getting releases together *now*, so there's some hope of having them actually happen by their deadlines :P21:22
timburkeif there's anything you feel should be included, let me know! or just review and merge it ;-)21:23
timburkei'm going to try to get the versioning support at least into swiftclient, then probably cut its release not long after that21:24
kota_sounds good21:24
timburkei think that about does it for announcements... any other comments or questions?21:24
timburkeall right, on to updates!21:25
timburke#topic waterfall EC21:25
*** openstack changes topic to "waterfall EC (Meeting topic: swift)"21:25
timburkeclayg, any chance to make progress on this, or are you still busy with other stuff for the most part?21:25
timburkei'll take that as a no, too busy ;-) thanks for doing some reviews lately!21:28
timburkemaybe i should drop the topic for now, and clayg can add it back when he's ready...21:28
timburke#topic lots of small files21:29
*** openstack changes topic to "lots of small files (Meeting topic: swift)"21:29
claygno problem - i'd be happy to squeeze in some reviews21:29
claygI did add waterfall to the list of topics21:29
rledisezso alecuyer is not here, but he gave me some informations21:29
*** TrevorV has joined #openstack-meeting21:29
timburkerledisez, yeah, you guys have been busy lately :D21:29
rledisezhe posted today a patch that is meant to cleanup some things in LOSF and get back to normal (hashes.pkl)21:30
patchbotpatch 718445 - swift (feature/losf) - Use hashes.pkl files for LOSF and remove list_part... - 1 patch set21:30
rledisezhe should post tomorrow an other cleanup patch about protobuf arguments21:30
rledisezafter that, he will start to send the new features (new key format, new volume selection algorithm and metadata stored in the KV)21:31
*** Trevor__V has joined #openstack-meeting21:31
rledisezthat's it for losf21:31
timburke#topic cors21:31
*** openstack changes topic to "cors (Meeting topic: swift)"21:31
timburkeso i've got a dirty tree locally that's partway through re-writing the js tests... i should finish that rewrite, but haven't found time21:32
*** Trevor_V has quit IRC21:32
timburkei suppose i should mark it WIP for now21:33
timburkethat's about it, though21:33
timburke#topic open discussion21:33
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:33
timburkeanything else we ought to talk about?21:33
rledisezjust a note about that random failure I've been hitting on
patchbotpatch 704892 - swift - proxy: stop sending frags to PyECLib with a Queue - 2 patch sets21:34
rledisezi found the bad test. I don't understand why, but if I disable it, no more random failure21:34
*** TrevorV has quit IRC21:34
rledisezif anybody have a clue, i'm taking it :)21:34
timburkeright! that weird eventlet issue!21:34
claygis it the GET with swapped frags test?21:35
rledisezit's this one:
patchbotpatch 704892 - swift - proxy: stop sending frags to PyECLib with a Queue - 2 patch sets21:35
claygrledisez: I'll take a look tomorrow!  💪21:35
rledisezclayg: thx a lot, you're awesome!21:35
rledisez(you all are!)21:36
rledisezI'll add a comment to show how to reproduce the issue21:36
rledisezotherwise, I'm still doing some profiling. I'll start looking at out use of len() (it counts, for real :D) and also i'll try to play with buffer protocol to see if we can save some memory copy21:37
*** slaweq_ has quit IRC21:38
*** slaweq_ has joined #openstack-meeting21:40
timburkeanybody have strong opinions on metric naming? or experience with fallout from trying to split one metric into several? i'm looking at in particular...21:40
patchbotpatch 716016 - swift - s3api: Add stats for s3api requests - 1 patch set21:40
timburkei'm trying to judge how important it is for us to get it right the first time, vs needing to do a split by bucket vs key later21:41
mattoliverauWas it the object replicator in recon where we wanted to rename some metrics, we're now been doubling up on metrics there for a while.21:42
mattoliverauI make them the same a the db replicator.21:42
rledisezso, the patch comes from us. we were originally interested in seeing the adoption of s3. but I like the idea of object vs bucket requests21:43
timburkerledisez, yeah, the adoption metrics make a lot of sense21:43
timburkei can wait for another patchset then21:44
*** slaweq_ has quit IRC21:45
timburkeall right, i think i'm'a call it21:46
timburkethank you all for coming, and thank you for working on swift!21:46
*** openstack changes topic to "OpenStack Meetings ||"21:46
openstackMeeting ended Wed Apr  8 21:46:17 2020 UTC.  Information about MeetBot at . (v 0.1.4)21:46
openstackMinutes (text):
*** patchbot has left #openstack-meeting21:46
*** ykatabam has joined #openstack-meeting21:50
*** Lucas_Gray has quit IRC22:01
*** slaweq_ has joined #openstack-meeting22:13
*** Trevor__V has quit IRC22:16
*** TrevorV has joined #openstack-meeting22:17
*** rcernin has joined #openstack-meeting22:32
*** gyee has quit IRC22:49
*** TrevorV has quit IRC22:50
*** hagridaaron has quit IRC23:08
*** gary_perkins has quit IRC23:26
*** gary_perkins has joined #openstack-meeting23:32

Generated by 2.15.3 by Marius Gedminas - find it at!