| *** rfolco has quit IRC | 00:06 | |
| *** macz has quit IRC | 00:32 | |
| *** ricolin has joined #openstack-meeting | 00:37 | |
| *** anastzhyr has quit IRC | 00:52 | |
| *** Liang__ has joined #openstack-meeting | 01:13 | |
| *** Liang__ is now known as LiangFang | 01:14 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:37 | |
| *** jamesmcarthur has quit IRC | 01:44 | |
| *** hongbin has joined #openstack-meeting | 02:02 | |
| *** hongbin has quit IRC | 02:02 | |
| *** hongbin has joined #openstack-meeting | 02:03 | |
| *** hongbin has quit IRC | 02:03 | |
| *** hongbin has joined #openstack-meeting | 02:04 | |
| *** hongbin has quit IRC | 02:04 | |
| *** hongbin has joined #openstack-meeting | 02:06 | |
| *** mhen has quit IRC | 02:18 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:25 | |
| *** jamesmcarthur_ has joined #openstack-meeting | 02:28 | |
| *** jamesmcarthur has quit IRC | 02:28 | |
| *** macz has joined #openstack-meeting | 02:30 | |
| *** macz has quit IRC | 02:35 | |
| *** rmk has quit IRC | 03:32 | |
| *** rmk has joined #openstack-meeting | 03:35 | |
| *** gyee has quit IRC | 03:56 | |
| *** jamesmcarthur_ has quit IRC | 04:08 | |
| *** jamesmcarthur has joined #openstack-meeting | 04:10 | |
| *** hongbin_ has joined #openstack-meeting | 04:12 | |
| *** hongbin has quit IRC | 04:15 | |
| *** macz has joined #openstack-meeting | 04:30 | |
| *** macz has quit IRC | 04:34 | |
| *** hongbin_ has quit IRC | 04:37 | |
| *** jamesmcarthur has quit IRC | 04:43 | |
| *** jamesmcarthur has joined #openstack-meeting | 04:44 | |
| *** jamesmcarthur has quit IRC | 04:50 | |
| *** jamesmcarthur has joined #openstack-meeting | 05:15 | |
| *** jamesmcarthur has quit IRC | 05:23 | |
| *** etp has left #openstack-meeting | 05:38 | |
| *** jamesmcarthur has joined #openstack-meeting | 05:57 | |
| *** jamesmcarthur has quit IRC | 06:02 | |
| *** davee___ has joined #openstack-meeting | 06:04 | |
| *** davee_ has quit IRC | 06:06 | |
| *** mugsie has quit IRC | 06:11 | |
| *** mugsie has joined #openstack-meeting | 06:14 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:36 | |
| *** jamesmcarthur has quit IRC | 06:42 | |
| *** dmsimard has quit IRC | 06:50 | |
| *** dmsimard has joined #openstack-meeting | 06:54 | |
| *** apetrich has quit IRC | 07:11 | |
| *** masahito has joined #openstack-meeting | 07:12 | |
| *** davee__ has quit IRC | 07:16 | |
| *** davee_ has joined #openstack-meeting | 07:16 | |
| *** masahito has quit IRC | 07:17 | |
| *** ralonsoh has joined #openstack-meeting | 07:27 | |
| *** jamesmcarthur has joined #openstack-meeting | 07:38 | |
| *** jamesmcarthur has quit IRC | 07:43 | |
| *** e0ne has joined #openstack-meeting | 07:54 | |
| *** e0ne has quit IRC | 07:54 | |
| *** e0ne has joined #openstack-meeting | 07:54 | |
| *** anastzhyr has joined #openstack-meeting | 07:56 | |
| *** belmoreira has joined #openstack-meeting | 07:57 | |
| *** tesseract has joined #openstack-meeting | 07:58 | |
| *** tesseract has quit IRC | 07:58 | |
| *** tesseract has joined #openstack-meeting | 07:59 | |
| *** maciejjozefczyk has joined #openstack-meeting | 08:06 | |
| *** slaweq has joined #openstack-meeting | 08:09 | |
| *** e0ne has quit IRC | 08:11 | |
| *** e0ne has joined #openstack-meeting | 08:11 | |
| *** Luzi has joined #openstack-meeting | 08:14 | |
| *** e0ne has quit IRC | 08:15 | |
| *** rpittau|afk is now known as rpittau | 08:22 | |
| *** jamesmcarthur has joined #openstack-meeting | 08:39 | |
| *** jamesmcarthur has quit IRC | 08:45 | |
| *** dmellado has quit IRC | 08:51 | |
| *** mdelavergne has joined #openstack-meeting | 08:54 | |
| *** dmellado has joined #openstack-meeting | 08:55 | |
| *** oneswig has joined #openstack-meeting | 08:57 | |
| *** etp has joined #openstack-meeting | 08:59 | |
| belmoreira | #startmeeting large_scale_sig | 09:00 |
|---|---|---|
| openstack | Meeting started Wed Jan 15 09:00:20 2020 UTC and is due to finish in 60 minutes. The chair is belmoreira. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:00 |
| *** openstack changes topic to " (Meeting topic: large_scale_sig)" | 09:00 | |
| openstack | The meeting name has been set to 'large_scale_sig' | 09:00 |
| amorin | hello! | 09:00 |
| belmoreira | Hello everyone. Welcome to the Large Scale SIG meeting. | 09:00 |
| oneswig | hello! | 09:00 |
| belmoreira | #topic Rollcall | 09:00 |
| *** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)" | 09:00 | |
| oneswig | o/ | 09:00 |
| belmoreira | o/ | 09:01 |
| etp | o/ | 09:01 |
| amorin | o/ | 09:01 |
| belmoreira | great. let's start with our first topic | 09:02 |
| belmoreira | #topic "Scaling within one cluster" goal | 09:02 |
| *** openstack changes topic to ""Scaling within one cluster" goal (Meeting topic: large_scale_sig)" | 09:02 | |
| belmoreira | #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling | 09:02 |
| belmoreira | There's no much progress in the etherpad | 09:03 |
| oneswig | Afraid not. Over here we are making slow progress towards improving instrumentation. | 09:03 |
| belmoreira | oslo.metrics blueprint is a great idea | 09:04 |
| belmoreira | let's wait for masahito feedback in the draft blueprint | 09:05 |
| oneswig | I think it ties in well with a fishbowl session from Shanghai, advocating exposing metrics from each service. | 09:05 |
| belmoreira | But this only focus in collecting metrics | 09:06 |
| belmoreira | I'm sure we can expand this more | 09:06 |
| belmoreira | I'm thinking about interaction with other services. Like Placement and how it's deployed | 09:07 |
| belmoreira | does it make sense? | 09:07 |
| oneswig | belmoreira: can you elaborate? | 09:08 |
| belmoreira | in my opinion scaling one nova cell is more than just rabbit and DB. For example, Nova interacts with placement and neutron and in my view we should focus in those scaling limits as well. | 09:10 |
| *** masahito has joined #openstack-meeting | 09:11 | |
| belmoreira | also, being only one cell, how the different components are deployed (shared rabbit or not - for example) will affect the scalability of the deployment | 09:11 |
| masahito | Sorry to be late. My IRC client doesn't work well :-( | 09:11 |
| oneswig | hi masahito | 09:11 |
| belmoreira | welcome masahito | 09:11 |
| masahito | hello o/ | 09:12 |
| belmoreira | oneswig does it make sense? | 09:12 |
| oneswig | belmoreira: is there any study on the growth rate of placement as a function of number of hypervisors? It could be polynomial (ie really bad)... | 09:12 |
| oneswig | belmoreira: makes sense to me. | 09:13 |
| mdelavergne | (hi, sorry for being late) I think it is definitely relevant to consider the placement and network | 09:13 |
| belmoreira | for example: placement is linear, but the number of requests that it can handle will change considering how it's deployed and options | 09:14 |
| oneswig | #link discussion from Shanghai on common monitoring via oslo https://etherpad.openstack.org/p/PVG-bring-your-crazy-idea | 09:14 |
| oneswig | unfortunately there are no actions or follow-up in that etherpad | 09:15 |
| belmoreira | it's not that crazy to me. It would be very cool. | 09:16 |
| *** lucasagomes has joined #openstack-meeting | 09:16 | |
| belmoreira | masahito do you have any update regarding the oslo.metric blueprint | 09:17 |
| masahito | Not much update. | 09:17 |
| masahito | because of some outage in our cluster :-( | 09:17 |
| *** lucasagomes has left #openstack-meeting | 09:17 | |
| masahito | But I want to finish to write the first draft by end of January. | 09:18 |
| oneswig | was thinking of your talk about LINE yesterday - had a rabbitmq issue on a system here | 09:18 |
| belmoreira | Thanks masahito. Let us know. | 09:19 |
| belmoreira | oneswig related with scalability? | 09:19 |
| oneswig | not this time, only moderate scale | 09:20 |
| oneswig | would have been timely otherwise :-) | 09:20 |
| belmoreira | anyone as something else related to this topic? | 09:20 |
| belmoreira | Let's continue to update the etherpad with our experiences in scaling one nova cell. | 09:21 |
| *** apetrich has joined #openstack-meeting | 09:21 | |
| belmoreira | moving to the next topic in the agenda | 09:22 |
| belmoreira | #topic Progress on "Documenting large scale operations" goal | 09:22 |
| *** openstack changes topic to "Progress on "Documenting large scale operations" goal (Meeting topic: large_scale_sig)" | 09:22 | |
| belmoreira | amorin started a thread in the mailing list to document the particular configuration that makes important for large scale deployments | 09:22 |
| belmoreira | thanks | 09:22 |
| amorin | yes | 09:22 |
| belmoreira | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011820.html | 09:22 |
| belmoreira | #link https://etherpad.openstack.org/p/large-scale-sig-documentation | 09:22 |
| amorin | I got an answer from slawek (neutron PTL) | 09:23 |
| amorin | here created a neutron bug: https://bugs.launchpad.net/neutron/+bug/1858419 | 09:23 |
| openstack | Launchpad bug 1858419 in neutron "Docs needed for tunables at large scale" [Medium,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | 09:23 |
| amorin | just like mriedman did for nova | 09:23 |
| amorin | so we can collect the tunings over there | 09:23 |
| amorin | they also notify us about rabbit and DB params | 09:23 |
| amorin | which I believe are important also | 09:24 |
| slaweq | amorin: hi, yes and we have some input from liuyulong who is neutron core and has a lot of large scale deployment experience also | 09:24 |
| slaweq | so we need to start working on some patches based on those comments | 09:24 |
| amorin | yup | 09:24 |
| slaweq | and also if You have anything to add there, feel free to write comments or send patches related to this bug :) | 09:24 |
| oneswig | sounds great! | 09:25 |
| amorin | I am pretty sure I can find some tunings based on OVH experience, I will try to do that | 09:26 |
| belmoreira | This is great. Thank you amorin slaweq | 09:26 |
| etp | sounds good | 09:26 |
| amorin | Also, I proposed to do some documentation change to identify parameters which could affect large scale | 09:26 |
| slaweq | amorin: I know You will :) | 09:26 |
| amorin | what do you think about this? | 09:27 |
| etp | btw has anyone compared different generally available neutron plugins ovs/ovn/...? | 09:27 |
| amorin | I havnt, on our side we use custom plugins based on OVS | 09:28 |
| amorin | maybe slaweq know if OVN is able to scale? | 09:28 |
| oneswig | Only for performance. | 09:28 |
| slaweq | amorin: I know there were some tests of ovn on scale done by networking-ovn team | 09:28 |
| belmoreira | etp definitely that is interesting. It would be great to know more | 09:28 |
| slaweq | I can try to search for them if You want to check them | 09:29 |
| etp | i noted that there's spec in Ussuri to move reference from ovs to ovn | 09:29 |
| slaweq | etp: no, for Ussuri are only moving networking-ovn code to be in-tree neutron driver | 09:29 |
| etp | ah, maybe I misread it :) | 09:30 |
| slaweq | but in the future we will probably want to switch our "default" backend to be ovn instead of ovs-agent | 09:30 |
| slaweq | but not in Ussuri for sure | 09:30 |
| belmoreira | I just added in the etherpad a presentation about CERN network deployment and the configuration options that we use. Will cross check to what is proposed. In our case we use linux bridge | 09:31 |
| amorin | ack | 09:31 |
| etp | slaweq: we are also looking in to same direction, when it happens remains to be seen | 09:31 |
| slaweq | etp: yes | 09:32 |
| etp | belmoreira: tnx | 09:32 |
| slaweq | If I will have link to any comparison between ovn and other backends I will place it in etherpad https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling | 09:32 |
| slaweq | ok? | 09:32 |
| belmoreira | please do | 09:33 |
| amorin | yes, thanks | 09:33 |
| belmoreira | should we have the same approach with other projects? like keystone, placement, glance? | 09:34 |
| belmoreira | have a bug open for relevant options for large deployments? | 09:34 |
| amorin | I think it could be nice, but it should come from them, we can force them | 09:35 |
| amorin | we cant* | 09:35 |
| amorin | sorry | 09:35 |
| oneswig | I had an action item to poll the Scientific SIG for data points. I'll add it to today's meeting (at 1100 UTC) | 09:36 |
| belmoreira | amorin: ok, let's try to signal issues first in the etherpad | 09:37 |
| belmoreira | oneswig thanks | 09:37 |
| amorin | yes | 09:37 |
| belmoreira | anything else related to this topic? | 09:37 |
| *** tonyb has joined #openstack-meeting | 09:38 | |
| belmoreira | moving on | 09:38 |
| belmoreira | #topic AOB | 09:38 |
| *** openstack changes topic to "AOB (Meeting topic: large_scale_sig)" | 09:38 | |
| belmoreira | Is there something else that you would like to discuss? | 09:39 |
| amorin | not on my side | 09:40 |
| oneswig | not for me | 09:40 |
| belmoreira | #topic Next Meeting | 09:40 |
| *** openstack changes topic to "Next Meeting (Meeting topic: large_scale_sig)" | 09:40 | |
| masahito | not from my side | 09:41 |
| *** jamesmcarthur has joined #openstack-meeting | 09:41 | |
| belmoreira | If we follow the 2 weeks rule the next meeting will be on January 29. Is this OK? | 09:41 |
| oneswig | +1 | 09:41 |
| masahito | +1 | 09:41 |
| amorin | +1 | 09:41 |
| masahito | ls | 09:41 |
| mdelavergne | +1 | 09:42 |
| belmoreira | #agreed next meeting: January 29, 9utc #openstack-meeting | 09:42 |
| belmoreira | Anything else before we close the meeting? | 09:42 |
| oneswig | have to go, thanks everyone | 09:42 |
| *** pcaruana has joined #openstack-meeting | 09:42 | |
| belmoreira | Thanks everyone | 09:42 |
| oneswig | ps it's on the agenda now https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_January_15th_2020 | 09:43 |
| mdelavergne | thanks | 09:43 |
| masahito | thanks, all | 09:43 |
| etp | thanks all | 09:43 |
| belmoreira | #endmeeting | 09:43 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 09:43 | |
| openstack | Meeting ended Wed Jan 15 09:43:21 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:43 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-01-15-09.00.html | 09:43 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-01-15-09.00.txt | 09:43 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-01-15-09.00.log.html | 09:43 |
| amorin | thanks! | 09:43 |
| *** jamesmcarthur has quit IRC | 09:46 | |
| *** mdelavergne has left #openstack-meeting | 09:55 | |
| *** LiangFang has quit IRC | 09:58 | |
| *** e0ne has joined #openstack-meeting | 10:07 | |
| *** jamesmcarthur has joined #openstack-meeting | 10:42 | |
| *** masahito has quit IRC | 10:43 | |
| *** simon-AS5591 has joined #openstack-meeting | 10:45 | |
| *** mattia has joined #openstack-meeting | 10:45 | |
| *** jamesmcarthur has quit IRC | 10:46 | |
| *** dh3 has joined #openstack-meeting | 10:59 | |
| *** janders has joined #openstack-meeting | 10:59 | |
| oneswig | #startmeeting scientific-sig | 11:00 |
| openstack | Meeting started Wed Jan 15 11:00:15 2020 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 11:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 11:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 11:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 11:00 |
| oneswig | SIG time! | 11:00 |
| *** witek has joined #openstack-meeting | 11:00 | |
| oneswig | #link Today's agenda https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_January_15th_2020 | 11:00 |
| oneswig | greetings all | 11:00 |
| janders | g'day all | 11:00 |
| dh3 | morning! | 11:01 |
| janders | Happy New Year 2020 | 11:01 |
| oneswig | janders: saw your presentation from kubecon last week, nice work! | 11:01 |
| janders | oneswig: thank you! :) | 11:01 |
| janders | it was a great conference | 11:01 |
| oneswig | clarification - kubecon last november, saw it last week | 11:01 |
| oneswig | did you get many questions in follow-up? | 11:01 |
| janders | just a few.. now I'm thinking if I replied :) was pretty hectic end of the year | 11:02 |
| oneswig | #link Janders at Kubecon https://www.youtube.com/watch?v=sPfZGHWnKNM | 11:02 |
| oneswig | So many moving parts though! Does it hold together? | 11:03 |
| mattia | hello everyone | 11:03 |
| oneswig | Hi mattia, welcome | 11:03 |
| janders | it does - and it's getting better :) waiting for some kit to arrive and hope to test 100GE version and HDR200 version of RDMA-K8s | 11:04 |
| janders | it's still early days though | 11:04 |
| oneswig | Have you looked at options for PMIX so MPI jobs can sit on top? | 11:04 |
| janders | no, not yet | 11:05 |
| janders | I was mostly thinking connecting to non-k8s storage and GPU workloads. I'm sure MPI workloads will bubble up too though | 11:06 |
| oneswig | It seems a lot less straightforward than putting containers into a Slurm environment. But with the energy going into it I'm sure a workable solution will appear | 11:07 |
| oneswig | #link PMIX and kubernetes hinted to here https://pmix.org/wp-content/uploads/2019/04/PMIxSUG2019.pdf | 11:07 |
| oneswig | janders: big numbers for your storage bandwidth at the end of the presentation | 11:08 |
| janders | it's still work in progress but in a scenario where a fair chunk of workloads is cloud-native and MPI is a small fraction I think this approach makes good sense | 11:08 |
| *** melwitt has quit IRC | 11:09 | |
| oneswig | yes - that looks like the use case | 11:09 |
| oneswig | ... I'm sure we had an agenda somewhere ... oops | 11:09 |
| janders | that was actually from our OpenStack system, but when I get more kit I hope to see these in k8s containers as well | 11:09 |
| oneswig | be interesting to know where you take it next | 11:10 |
| janders | we're getting close to going live with the cyber system so my bosses aren't to keen with me testing on the same kit | 11:10 |
| janders | i have some interesting (I hope) questions on that - but let's leave that for AOB | 11:10 |
| oneswig | curse those production obligations! | 11:10 |
| janders | haha! :) | 11:10 |
| oneswig | OK, time to start | 11:10 |
| janders | I keep saying ops focused guys they are no fun | 11:10 |
| oneswig | #topic OpenStack board elections | 11:11 |
| *** openstack changes topic to "OpenStack board elections (Meeting topic: scientific-sig)" | 11:11 | |
| oneswig | Just a quick reminder to vote for individual directors this week | 11:11 |
| oneswig | I'd share a link but I think they are all somehow individual | 11:12 |
| *** pcaruana has quit IRC | 11:12 | |
| oneswig | That's all on that :-) | 11:12 |
| oneswig | #topic Large-Scale SIG survey | 11:12 |
| *** openstack changes topic to "Large-Scale SIG survey (Meeting topic: scientific-sig)" | 11:12 | |
| oneswig | Earlier this morning there was a Large Scale SIG meeting | 11:13 |
| oneswig | There don't appear to be minutes for it on Eavesdrop... | 11:13 |
| oneswig | aha http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-01-15-09.00.html | 11:14 |
| oneswig | #link request for data on large-scale configuration http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011820.html | 11:14 |
| dh3 | Thanks, I missed that (feeling I'm spread a bit thin at the moment) - not that we are *so* large as some | 11:15 |
| *** anastzhyr has quit IRC | 11:15 | |
| oneswig | Hi dh3, everything is relative I guess! | 11:16 |
| oneswig | The Large Scale SIG are hoping for input on scaling issues faced in real-world experiences. | 11:17 |
| oneswig | (and what tools to tackle them) | 11:17 |
| oneswig | #link Large Scale SIG etherpad https://etherpad.openstack.org/p/large-scale-sig-documentation | 11:17 |
| janders | DB, AMQP, Neutron... guessed 3/3 and the order, too :) | 11:19 |
| oneswig | janders: low-hanging fruit that :-) | 11:20 |
| janders | with other APIs it's probably easier to just deploy more API nodes than to optimise... to a degree that is - and that assumes DB scaling is sorted out separately :) | 11:20 |
| oneswig | One interesting starting point was the idea to create an Oslo module for standardising telemetry from OpenStack services, and making it available as prometheus or statsd | 11:21 |
| janders | very prom(eth)ising | 11:21 |
| oneswig | oh, good :-) | 11:21 |
| oneswig | I don't think there's a spec for review yet but I think its underway | 11:22 |
| janders | I think scalable telemetry was one of the PITA/WTF areas of OpenStack for quite a while so this sounds awesome | 11:23 |
| janders | I got quite far with just switching it off but that's not good either | 11:23 |
| dh3 | is the intention that this data is for developers or operators? (or both) and if devs, has anyone thought about "phone home" reporting like Ceph added recently? | 11:24 |
| oneswig | janders: indeed. | 11:24 |
| oneswig | dh3: I think it's for operators principally, but I expect there's a symbiosis where operators have the pain points and use this as evidence for developers, perhaps. | 11:25 |
| oneswig | If people are interested they can encourage this effort by adding their support to the etherpad! | 11:26 |
| janders | RHATs are pumping a fair bit of effort into SAF | 11:26 |
| dh3 | OK. For operators there is a downstream process where - having received some data or telemetry trend or errors - they need to decide what to do about it (not necessarily being an expert in tuning MySQL or RabbitMQ or ...) | 11:26 |
| janders | ("Service Assurance Framework") | 11:27 |
| dh3 | I will try to get some coherent thoughts onto the etherpad. | 11:27 |
| janders | I wonder if the two directions could be combined | 11:27 |
| oneswig | janders: what's that? | 11:27 |
| oneswig | The lauchpad tickets on gathering config parameters that are important for scale may be worth a look | 11:27 |
| janders | OpenShift/K8S powered, Prometheus based OpenStack monitoring | 11:27 |
| dh3 | embarrassed to say I had not met SAF and we have RHOSP | 11:27 |
| oneswig | Nova https://bugs.launchpad.net/nova/+bug/1838819 Neutron https://bugs.launchpad.net/neutron/+bug/1858419 | 11:28 |
| openstack | Launchpad bug 1838819 in OpenStack Compute (nova) "Docs needed for tunables at large scale" [Undecided,Confirmed] | 11:28 |
| openstack | Launchpad bug 1858419 in neutron "Docs needed for tunables at large scale" [Medium,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | 11:28 |
| janders | it's on my TODO list but due to resourcing constraints our first monitoring solution for the cyber system is gonna be Nagios | 11:28 |
| janders | SAF looks quite good though | 11:28 |
| oneswig | got a link? | 11:28 |
| janders | and its getting a lot of good press from the Brisbane support guys who are real OpenStack gurus, hats off to them | 11:29 |
| janders | if they say its good im totally gonna try it | 11:29 |
| janders | (looking it up) | 11:29 |
| dh3 | I found https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/service_assurance_framework/introduction-to-service-assurance-framework_assembly (not sure if there's a free/OSS side?) | 11:29 |
| oneswig | looks like it's purely a RHOSP thing | 11:29 |
| janders | yes it is but I'd be surprised if it doesn't have a good upstream | 11:30 |
| oneswig | The data at source does sound like it would be the same though | 11:30 |
| janders | I cant find public link - would you like doco from KB over email? | 11:30 |
| dh3 | janders: anything you can share will be interesting (though we are likely to move off RH at some point so wary of lockin) | 11:31 |
| oneswig | Thanks janders, one to watch | 11:31 |
| witek | https://github.com/redhat-service-assurance/telemetry-framework | 11:31 |
| oneswig | Hi witek, welcome! | 11:31 |
| oneswig | and thanks for the link | 11:31 |
| witek | they have presented it first time at the summit in Berlin | 11:31 |
| janders | dh3: what's the best email to send it to? | 11:32 |
| dh3 | janders: dh3@sanger.ac.uk please | 11:32 |
| dh3 | witek: thanks, I will look up the talk | 11:32 |
| janders | oneswig: dh3: done | 11:33 |
| dh3 | thanks, on first skim it looks the same as the access.redhat.com docs, I will find a quiet corner to read it this afternoon | 11:35 |
| janders | yeah that's the source | 11:36 |
| janders | I reviewed this with my monitoring expert and we figured it's worth a go, might be a good solution going forward | 11:36 |
| oneswig | If more telemetry data is made available via standard protocols, everyone wins | 11:36 |
| janders | but it's too complex to be deployed as the initial monitoring solution for our cyber system | 11:36 |
| witek | it follows similar architecture as Monasca | 11:38 |
| oneswig | I should perhaps rtfm but I wonder what SAF are doing for the limitations around eg retention policies in prometheus | 11:38 |
| oneswig | janders: do they have a story for logs and notifications? | 11:39 |
| janders | speaking of OSP, 16 is out in beta | 11:39 |
| janders | notifications - yes, I believe so | 11:39 |
| janders | logs - in what sense? | 11:39 |
| oneswig | you know, like syslog | 11:40 |
| janders | logstash/elasticsearch kind of stuff? | 11:40 |
| oneswig | yep | 11:40 |
| janders | I think they do this via collectd somehow but this part I'm less familiar with | 11:40 |
| oneswig | sounds like when they get there they will have reinvented much of what monasca does today ;-) | 11:41 |
| janders | :) | 11:41 |
| janders | as long as the wheel is still round, a little reinvention is tolerable even if not optimal | 11:41 |
| oneswig | anyway, should move on I guess. Final comments on large scale? | 11:42 |
| janders | nothing specific just big +1 for the initiative | 11:42 |
| dh3 | likewise | 11:42 |
| janders | I think there was a need for this for a very long time | 11:42 |
| oneswig | sure, I think so too | 11:43 |
| *** jamesmcarthur has joined #openstack-meeting | 11:43 | |
| oneswig | #topic events for 2020 | 11:43 |
| *** openstack changes topic to "events for 2020 (Meeting topic: scientific-sig)" | 11:43 | |
| janders | on that... do you guys know what's the CFP timeframe for Vancouver? | 11:43 |
| oneswig | OK y'all, lets get some links out for events coming up... | 11:43 |
| oneswig | janders: not heard, nor checked | 11:43 |
| janders | i looked around but haven't managed to find any info | 11:44 |
| dh3 | we were talking about conferences in the office this morning. wondering when the European summit dates/location might be announced. and a bit sad there are no upcoming OpenStack Days listed | 11:44 |
| janders | it feels like that's typically Jan-Feb for the May conferences | 11:44 |
| janders | this year NA event is bit later so perhaps so is CFP | 11:44 |
| oneswig | #link UKRI Coud Workshop (London) https://cloud.ac.uk/ | 11:44 |
| oneswig | CFP closing date for that is end of Jan I think | 11:45 |
| janders | is there anything interesting coming up for ISC2020? I was asked by my boss if I wanna go | 11:46 |
| oneswig | I know the group who did SuperCompCloud at SC2019 have a similar workshop planned (CSCS in Switzerland, Indiana University, maybe others) | 11:46 |
| dh3 | we had the Arista conference ("Innovate") and Lustre LUG/LAD on our radar | 11:47 |
| janders | at least OIS Vancouver and ISC20 don't overlap (but are close enough to make it tricky for us Aussies to attend both) | 11:47 |
| *** jamesmcarthur has quit IRC | 11:48 | |
| oneswig | janders: I'm expecting to go to ISC 2020, hopefully see you there. | 11:50 |
| janders | that would be great! :) | 11:50 |
| janders | are you likely headed to Vancouver as well? | 11:50 |
| oneswig | I expect so, provided they aren't the same week :-) | 11:51 |
| janders | looks like they are not :) | 11:51 |
| oneswig | Not planning on Cephalocon though - Seoul in March https://ceph.io/cephalocon/ | 11:51 |
| oneswig | #link HPCAC Swiss Workshop in April is also good https://www.hpcadvisorycouncil.com/events/2020/swiss-workshop/ | 11:52 |
| oneswig | dh3: where do cloud-oriented bioinformaticions go for conferences? | 11:53 |
| dh3 | oneswig: I'm not sure that the cloud emphasis affects the choice of conference but I will ask around | 11:55 |
| janders | as we're nearing the end of the hour I have some AOB | 11:56 |
| oneswig | #topic AOB | 11:56 |
| dh3 | other than some of platform-oriented things like Kubecon maybe have a bit more of an application focus than say the OpenStack summit | 11:56 |
| *** openstack changes topic to "AOB (Meeting topic: scientific-sig)" | 11:56 | |
| oneswig | well spotted :-) | 11:56 |
| janders | do you have much experience with cinder volume transfers? | 11:56 |
| janders | I'm thinking using this as a tool for devsecops guys to make malware samples available to cyber researchers | 11:56 |
| dh3 | we tried it several releases back and it "just worked" but we didn't have a use case/user story for it | 11:57 |
| dh3 | for small scale (malware samples are small right) I might look at image sharing instead? | 11:57 |
| janders | 1) people upload encrypted malware to secure dropbox 2) devsecops guys have a VM that sits on the same provider network as the dropbox and they create a volume, attach it and copy encrypted malware in 3) they give the volume to the researcher | 11:58 |
| janders | I need a more fancy backend setup than glance can offer | 11:58 |
| *** dviroel has joined #openstack-meeting | 11:58 | |
| janders | with cinder I can have separate pools for normal and disco volumes | 11:58 |
| janders | i tried transfers today and it *just worked* | 11:59 |
| oneswig | I've used rbd export to effectively dd out a running volume before | 11:59 |
| janders | if you guys have no horror stories how it breaks terribly in some corner cases I think I will endorse this solution | 11:59 |
| oneswig | not used a cinder method before though | 11:59 |
| janders | ok! thanks oneswig | 12:00 |
| dh3 | rbd export works for us (in "trying circumstances" too) | 12:00 |
| oneswig | good luck with it janders! | 12:00 |
| janders | thanks and I will report back | 12:00 |
| oneswig | ok, time to close, thanks everyone | 12:00 |
| janders | thank you all | 12:00 |
| oneswig | #endmeeting | 12:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 12:00 | |
| openstack | Meeting ended Wed Jan 15 12:00:40 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-01-15-11.00.html | 12:00 |
| dh3 | thanks, bye | 12:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-01-15-11.00.txt | 12:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-01-15-11.00.log.html | 12:00 |
| *** oneswig has quit IRC | 12:02 | |
| *** simon-AS559 has joined #openstack-meeting | 12:03 | |
| *** rfolco has joined #openstack-meeting | 12:06 | |
| *** simon-AS5591 has quit IRC | 12:06 | |
| *** janders has quit IRC | 12:18 | |
| *** ociuhandu has joined #openstack-meeting | 12:26 | |
| *** jamesmcarthur has joined #openstack-meeting | 12:44 | |
| *** pcaruana has joined #openstack-meeting | 12:49 | |
| *** jamesmcarthur has quit IRC | 12:49 | |
| *** enriquetaso has joined #openstack-meeting | 12:55 | |
| *** ociuhandu has quit IRC | 12:59 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:15 | |
| *** witek has quit IRC | 13:20 | |
| *** eharney has quit IRC | 13:20 | |
| *** jamesmcarthur has quit IRC | 13:24 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:24 | |
| *** ociuhandu has joined #openstack-meeting | 13:27 | |
| *** ykatabam has quit IRC | 13:32 | |
| *** jamesmcarthur has quit IRC | 13:36 | |
| *** liuyulong has joined #openstack-meeting | 13:41 | |
| *** Liang__ has joined #openstack-meeting | 13:45 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:47 | |
| *** mnaser has quit IRC | 13:54 | |
| *** mnaser has joined #openstack-meeting | 13:55 | |
| liuyulong | #startmeeting neutron_l3 | 14:00 |
| *** rfolco has quit IRC | 14:00 | |
| openstack | Meeting started Wed Jan 15 14:00:37 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:00 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:00 |
| liuyulong | hi there | 14:01 |
| slaweq | hi | 14:01 |
| *** Liang__ is now known as LiangFang | 14:01 | |
| *** rfolco has joined #openstack-meeting | 14:01 | |
| liuyulong | #topic Announcements | 14:01 |
| *** eharney has joined #openstack-meeting | 14:01 | |
| *** openstack changes topic to "Announcements (Meeting topic: neutron_l3)" | 14:01 | |
| *** sfernand has joined #openstack-meeting | 14:02 | |
| liuyulong | All things were mentioned during neutron team meeting yesterday. | 14:03 |
| liuyulong | So I guess we can move on then? | 14:03 |
| *** rcernin has quit IRC | 14:04 | |
| liuyulong | OK, let's move on. | 14:04 |
| liuyulong | #topic Bugs | 14:04 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:04 | |
| *** andrebeltrami has joined #openstack-meeting | 14:04 | |
| liuyulong | [neutron] Bug deputy report - Jan 06 to 12 | 14:05 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011955.html | 14:05 |
| liuyulong | Let's revisit this bug first. | 14:05 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1774459 | 14:06 |
| openstack | Launchpad bug 1774459 in neutron "Update permanent ARP entries for allowed_address_pair IPs in DVR Routers" [High,In progress] - Assigned to Brian Haley (brian-haley) | 14:06 |
| ralonsoh | hi | 14:06 |
| *** rsimai_away is now known as rsimai | 14:06 | |
| liuyulong | #link https://review.opendev.org/#/c/601336/ | 14:06 |
| liuyulong | ^ this is the fix of the bug. | 14:06 |
| liuyulong | #link https://review.opendev.org/#/c/653883/ | 14:07 |
| liuyulong | This should be related to the bug and fix. | 14:07 |
| liuyulong | I've tested this with some new based on the problem we met locally. So I created a new bug https://bugs.launchpad.net/neutron/+bug/1859638 | 14:08 |
| openstack | Launchpad bug 1774459 in neutron "duplicate for #1859638 Update permanent ARP entries for allowed_address_pair IPs in DVR Routers" [High,In progress] - Assigned to Brian Haley (brian-haley) | 14:08 |
| *** vkmc has quit IRC | 14:09 | |
| *** vkmc has joined #openstack-meeting | 14:09 | |
| haleyb | hi | 14:10 |
| liuyulong | And back to fix, I've left some comments. Some of them are technical issues, IMO, it should be addressed. | 14:11 |
| haleyb | imho #link https://review.opendev.org/#/c/653883/ is ready to merge | 14:12 |
| liuyulong | haleyb, yes, looks good to me. | 14:13 |
| haleyb | i see your comments on the other one, i've had no time to get back to it recently | 14:14 |
| *** dh3 has quit IRC | 14:14 | |
| liuyulong | haleyb, sure, we can continue the details in the gerrit. | 14:15 |
| haleyb | or if anyone has more cycles... i was hoping it was a quick update but doesn't look it | 14:16 |
| liuyulong | Besides I also have an idea to fix the bug when we have no GARP from users VM. | 14:16 |
| liuyulong | L2pop and ARP responder will not be considered, ARP request and reply will be flooded to tunnels... (Bad news...) | 14:19 |
| *** jungleboyj has quit IRC | 14:20 | |
| liuyulong | ARP reply will be sent to local qr-device and flood to tunnels at same time. Then all routers will be aware of the allowed address mac. | 14:21 |
| *** jungleboyj has joined #openstack-meeting | 14:21 | |
| liuyulong | OK, I will summary these comments to the LP bug or gerrit. | 14:27 |
| liuyulong | Next one: | 14:28 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1858661 | 14:28 |
| openstack | Launchpad bug 1858661 in neutron "Error when creating a Linux interface" [High,Confirmed] | 14:28 |
| liuyulong | slaweq, ralonsoh any updates about this ? | 14:28 |
| ralonsoh | liuyulong, no sorry, I didn't have time | 14:29 |
| ralonsoh | maybe tomorrow | 14:29 |
| slaweq | me neighter | 14:29 |
| liuyulong | It blocks the CI in high failure rate, or not? | 14:30 |
| ralonsoh | sometimes yes, but this is not a repetitive error | 14:31 |
| liuyulong | Alright, not 100% is good news. And we also have this from the deputy bug list: | 14:32 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1858642 | 14:32 |
| openstack | Launchpad bug 1858642 in neutron "paramiko.ssh_exception.NoValidConnectionsError error cause dvr scenario jobs failing" [High,Confirmed] | 14:32 |
| ralonsoh | yes, I'm on it now | 14:32 |
| liuyulong | OK, all these issues can be discussed during CI meeting in details later. | 14:36 |
| liuyulong | Next one | 14:36 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1859163 | 14:36 |
| openstack | Launchpad bug 1859163 in neutron "Create port with fixed ipv6 address for non-router ports is not allowed anymore" [Medium,Fix committed] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez) | 14:36 |
| liuyulong | Just updated the status of the bug, it is fixed. | 14:37 |
| liuyulong | #link https://review.opendev.org/#/c/701965/ | 14:37 |
| ralonsoh | cool! | 14:37 |
| ralonsoh | waiting for the gate | 14:37 |
| liuyulong | Next one | 14:37 |
| liuyulong | #link sometimes yes, but this is not a repetitive error | 14:37 |
| liuyulong | #undo | 14:38 |
| openstack | Removing item from minutes: #link sometimes | 14:38 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1858419 | 14:38 |
| openstack | Launchpad bug 1858419 in neutron "Docs needed for tunables at large scale" [Medium,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | 14:38 |
| *** jamesmcarthur has quit IRC | 14:40 | |
| liuyulong | Again, I'm now revisit those patches of scale issues have the config options. | 14:40 |
| liuyulong | #link https://review.opendev.org/#/c/364793/ | 14:40 |
| liuyulong | This can be an example. | 14:40 |
| slaweq | thx liuyulong for working on this | 14:41 |
| liuyulong | slaweq, np | 14:42 |
| *** artom has joined #openstack-meeting | 14:42 | |
| liuyulong | That's all from the deputy list. Let's shot a quick scan of LP bug list. | 14:42 |
| *** lseki has quit IRC | 14:44 | |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1858086 | 14:44 |
| openstack | Launchpad bug 1858086 in neutron "qrouter's local link route cannot be restored " [Medium,Confirmed] | 14:44 |
| *** diablo_rojo_phon has quit IRC | 14:44 | |
| *** lseki has joined #openstack-meeting | 14:44 | |
| liuyulong | Assigned to me now, I will try to fix it in neutron-server API to disable such route update. | 14:45 |
| slaweq | if You are changing API You should add new api extension to make this change discoverable | 14:45 |
| ralonsoh | even if this is an error? | 14:45 |
| slaweq | ralonsoh: I think so | 14:45 |
| ralonsoh | (just asking) | 14:45 |
| ralonsoh | perfect | 14:46 |
| slaweq | as we are changing API behaviour | 14:46 |
| slaweq | so e.g. we will not be able to test it in tempest properly | 14:46 |
| slaweq | (potentially) | 14:46 |
| slaweq | so it should be probably some shim extension but still we will need that | 14:46 |
| liuyulong | Sure | 14:47 |
| liuyulong | No more bugs from me now. Any updates? | 14:47 |
| ralonsoh | no | 14:47 |
| slaweq | I have one new | 14:48 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1859832 | 14:48 |
| openstack | Launchpad bug 1859832 in neutron "L3 HA connectivity to GW port can be broken after reboot of backup node" [Medium,New] - Assigned to Slawek Kaplonski (slaweq) | 14:48 |
| slaweq | reported just before this meeting | 14:48 |
| slaweq | I'm working on some fix for this | 14:48 |
| slaweq | but for now only fix which I have in mind requires changes in both L3 and ovs agent | 14:48 |
| *** diablo_rojo_phon has joined #openstack-meeting | 14:49 | |
| *** electrofelix has joined #openstack-meeting | 14:49 | |
| slaweq | and also second thing | 14:50 |
| slaweq | I yesterday sent patch https://review.opendev.org/#/c/702547/ which is (I hope) fix for old issue with dvr routers | 14:50 |
| slaweq | please review it when You will have some time | 14:50 |
| ralonsoh | sure | 14:50 |
| slaweq | and that's all from my side | 14:50 |
| liuyulong | Looks like the GARP issue for bug 1859832. | 14:50 |
| openstack | bug 1859832 in neutron "L3 HA connectivity to GW port can be broken after reboot of backup node" [Medium,New] https://launchpad.net/bugs/1859832 - Assigned to Slawek Kaplonski (slaweq) | 14:50 |
| slaweq | liuyulong: not GARP but MLDv2 packets | 14:51 |
| slaweq | as it's related to IPv6 strictly | 14:51 |
| liuyulong | yes, no matter which type of packets, the gateway mac will be refreshed to "backup" node in physical switch. | 14:52 |
| slaweq | liuyulong: exactly | 14:53 |
| slaweq | I was trying to describe as clear as I possible the issue in bug report | 14:53 |
| slaweq | I hope it will be clear for You what's going on there | 14:53 |
| slaweq | but if not, I can try to explain it e.g. on IRC if You will have any questions | 14:54 |
| liuyulong | I have a simple way to verify this issue. | 14:54 |
| liuyulong | When you have centralized floating IPs in the snat gateway, you can use "neutron l3-agent-router-add" to add a new HA member to the router, to see if the floating IP traffic is broken. | 14:56 |
| slaweq | liuyulong: but it's race and it can happen only sometimes | 14:57 |
| slaweq | so it's possible that on not overloaded system when You are adding only one router, it will be fine | 14:57 |
| liuyulong | The way I mentioned can 100% reproduce the problem. | 14:57 |
| *** macz has joined #openstack-meeting | 14:57 | |
| slaweq | but in general it's true what You said, it can break in such case | 14:57 |
| liuyulong | So you have any idea to fix the issue? | 14:58 |
| slaweq | liuyulong: yes | 14:58 |
| slaweq | I want to create port in br-int and add some additional flag to say to ovs agent to not remove dead_vlan_tag | 14:59 |
| slaweq | and then, when l3 agent will do all with this port, update ovs port to remove this flag | 14:59 |
| slaweq | and than ovs agent will change tag to proper one | 14:59 |
| slaweq | that's long story short | 15:00 |
| slaweq | but I'm still working on it | 15:00 |
| slaweq | I hope to send something to gerrit this week | 15:00 |
| liuyulong | All right, time is up. Let's move to our channel. | 15:00 |
| liuyulong | slaweq, thanks. | 15:00 |
| slaweq | k | 15:00 |
| slaweq | o/ | 15:00 |
| liuyulong | #endmeeting | 15:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:00 | |
| openstack | Meeting ended Wed Jan 15 15:00:43 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-01-15-14.00.html | 15:00 |
| ralonsoh | bye | 15:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-01-15-14.00.txt | 15:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-01-15-14.00.log.html | 15:00 |
| *** ociuhandu has quit IRC | 15:01 | |
| *** rfolco is now known as rfolco|bbl | 15:03 | |
| *** Luzi has quit IRC | 15:04 | |
| *** apetrich has quit IRC | 15:07 | |
| *** jamesmcarthur has joined #openstack-meeting | 15:22 | |
| *** ociuhandu has joined #openstack-meeting | 15:28 | |
| *** ociuhandu has quit IRC | 15:39 | |
| *** ociuhandu has joined #openstack-meeting | 15:40 | |
| *** LiangFang has quit IRC | 15:46 | |
| *** eharney has quit IRC | 15:48 | |
| *** eharney has joined #openstack-meeting | 15:52 | |
| *** efried is now known as efried_afk | 15:56 | |
| *** melwitt has joined #openstack-meeting | 16:00 | |
| *** witek has joined #openstack-meeting | 16:01 | |
| *** gyee has joined #openstack-meeting | 16:03 | |
| *** armax has joined #openstack-meeting | 16:07 | |
| *** carloss has joined #openstack-meeting | 16:12 | |
| *** simon-AS559 has quit IRC | 16:15 | |
| *** sfernand has quit IRC | 16:31 | |
| *** andrebeltrami has quit IRC | 16:34 | |
| *** ociuhandu has quit IRC | 16:39 | |
| *** ociuhandu has joined #openstack-meeting | 16:39 | |
| *** pcaruana has quit IRC | 16:45 | |
| *** clayg has quit IRC | 16:49 | |
| *** clayg has joined #openstack-meeting | 16:49 | |
| *** beisner has quit IRC | 16:50 | |
| *** beisner has joined #openstack-meeting | 16:50 | |
| *** simon-AS559 has joined #openstack-meeting | 16:51 | |
| *** simon-AS5591 has joined #openstack-meeting | 16:52 | |
| *** simon-AS559 has quit IRC | 16:56 | |
| *** rsimai is now known as rsimai_away | 16:57 | |
| *** ociuhandu has quit IRC | 16:59 | |
| *** maciejjozefczyk has quit IRC | 17:04 | |
| *** maciejjozefczyk has joined #openstack-meeting | 17:04 | |
| *** simon-AS5591 has left #openstack-meeting | 17:06 | |
| *** ociuhandu has joined #openstack-meeting | 17:08 | |
| *** gmann has quit IRC | 17:09 | |
| *** gmann has joined #openstack-meeting | 17:10 | |
| *** rpittau is now known as rpittau|afk | 17:15 | |
| *** tesseract has quit IRC | 17:26 | |
| *** njohnston has quit IRC | 17:28 | |
| *** njohnston has joined #openstack-meeting | 17:29 | |
| *** ociuhandu has quit IRC | 17:33 | |
| *** jgriffith has quit IRC | 17:40 | |
| *** jgriffith has joined #openstack-meeting | 17:42 | |
| *** e0ne has quit IRC | 18:03 | |
| *** electrofelix has quit IRC | 18:07 | |
| *** ociuhandu has joined #openstack-meeting | 18:14 | |
| *** carloss has quit IRC | 18:18 | |
| *** ociuhandu has quit IRC | 18:19 | |
| *** eharney has quit IRC | 18:26 | |
| *** rfolco|bbl is now known as rfolco | 18:28 | |
| *** TheJulia has quit IRC | 18:29 | |
| *** rpittau|afk has quit IRC | 18:30 | |
| *** seongsoocho has quit IRC | 18:30 | |
| *** masayukig has quit IRC | 18:30 | |
| *** _erlon_ has quit IRC | 18:30 | |
| *** evgenyl has quit IRC | 18:30 | |
| *** coreycb has quit IRC | 18:30 | |
| *** donnyd has quit IRC | 18:30 | |
| *** csatari has quit IRC | 18:30 | |
| *** jamespage has quit IRC | 18:30 | |
| *** tdasilva has quit IRC | 18:30 | |
| *** walshh_ has quit IRC | 18:30 | |
| *** lamt has quit IRC | 18:30 | |
| *** ildikov has quit IRC | 18:31 | |
| *** arne_wiebalck_ has joined #openstack-meeting | 18:31 | |
| *** mnaser has quit IRC | 18:31 | |
| *** xyang has quit IRC | 18:31 | |
| *** donnyd has joined #openstack-meeting | 18:31 | |
| *** arne_wiebalck has quit IRC | 18:31 | |
| *** arne_wiebalck_ is now known as arne_wiebalck | 18:31 | |
| *** jamespage has joined #openstack-meeting | 18:32 | |
| *** datasundae has quit IRC | 18:32 | |
| *** diablo_rojo_phon has quit IRC | 18:32 | |
| *** TheJulia has joined #openstack-meeting | 18:32 | |
| *** datasundae has joined #openstack-meeting | 18:32 | |
| *** seongsoocho has joined #openstack-meeting | 18:32 | |
| *** coreycb has joined #openstack-meeting | 18:33 | |
| *** tdasilva has joined #openstack-meeting | 18:33 | |
| *** ildikov has joined #openstack-meeting | 18:33 | |
| *** csatari has joined #openstack-meeting | 18:33 | |
| *** johnsom has quit IRC | 18:33 | |
| *** mnaser has joined #openstack-meeting | 18:34 | |
| *** johnsom has joined #openstack-meeting | 18:35 | |
| *** masayukig has joined #openstack-meeting | 18:35 | |
| *** walshh_ has joined #openstack-meeting | 18:36 | |
| *** csatari has quit IRC | 18:38 | |
| *** eharney has joined #openstack-meeting | 18:38 | |
| *** TheJulia has quit IRC | 18:40 | |
| *** rcernin has joined #openstack-meeting | 18:40 | |
| *** jamespage has quit IRC | 18:40 | |
| *** donnyd has quit IRC | 18:40 | |
| *** donnyd has joined #openstack-meeting | 18:42 | |
| *** tdasilva has quit IRC | 18:43 | |
| *** jamesmcarthur has quit IRC | 18:44 | |
| *** slaweq has quit IRC | 18:47 | |
| *** xyang has joined #openstack-meeting | 18:55 | |
| *** efried_afk is now known as efried | 18:55 | |
| *** witek has quit IRC | 19:08 | |
| *** dklyle has quit IRC | 19:36 | |
| *** ociuhandu has joined #openstack-meeting | 19:36 | |
| *** dklyle has joined #openstack-meeting | 19:37 | |
| *** dklyle has quit IRC | 19:37 | |
| *** dklyle has joined #openstack-meeting | 19:38 | |
| *** jamespage has joined #openstack-meeting | 19:52 | |
| *** jamespage has quit IRC | 19:52 | |
| *** e0ne has joined #openstack-meeting | 19:57 | |
| *** e0ne has quit IRC | 20:04 | |
| *** TheJulia has joined #openstack-meeting | 20:08 | |
| *** _erlon_ has joined #openstack-meeting | 20:11 | |
| *** enriquetaso has quit IRC | 20:13 | |
| *** enriquetaso has joined #openstack-meeting | 20:13 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:19 | |
| *** eharney has quit IRC | 20:36 | |
| *** slaweq has joined #openstack-meeting | 20:36 | |
| *** vishalmanchanda has quit IRC | 20:44 | |
| *** enriquetaso has quit IRC | 20:45 | |
| *** patchbot has joined #openstack-meeting | 20:46 | |
| *** ralonsoh has quit IRC | 20:57 | |
| *** alecuyer has joined #openstack-meeting | 20:57 | |
| *** seongsoocho_ has joined #openstack-meeting | 20:59 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Jan 15 21:00:43 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| *** tdasilva has joined #openstack-meeting | 21:00 | |
| timburke | who's here for the swift meeting? | 21:00 |
| alecuyer | o/ | 21:00 |
| tdasilva | eu | 21:01 |
| kota_ | hi | 21:01 |
| seongsoocho_ | o/ | 21:01 |
| timburke | as usual, the agenda's at | 21:01 |
| timburke | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:01 |
| timburke | i've been out of town for much of the last week, so i don't really have much more than updates to bring up | 21:02 |
| timburke | #topic versioning | 21:02 |
| *** openstack changes topic to "versioning (Meeting topic: swift)" | 21:02 | |
| timburke | i've been taking a look, and tdasilva has been busy addressing the last few issues we've found | 21:03 |
| timburke | if you have any concerns about the new API, now's the time to check it out! | 21:03 |
| timburke | #link https://review.opendev.org/#/c/682382/ | 21:03 |
| patchbot | patch 682382 - swift - New Object Versioning mode - 72 patch sets | 21:03 |
| timburke | #link https://review.opendev.org/#/c/673682/ | 21:03 |
| patchbot | patch 673682 - swift - s3api: Implement object versioning API - 44 patch sets | 21:03 |
| timburke | i hope to have both of those landed within the next week or so, with any lingering issues being addressed as follow-up patches | 21:04 |
| timburke | tdasilva, anything i forgot to mention about that? | 21:04 |
| timburke | and everyone else, anything else you'd like to know about it? | 21:04 |
| tdasilva | nope, hoping to review a bit more of the swift client too, it would be nice to get that merged as closed together with the swift patches as possible | 21:05 |
| timburke | agreed | 21:05 |
| *** jamesmcarthur has quit IRC | 21:06 | |
| kota_ | sounds reasonable | 21:06 |
| timburke | i'll also go back and abandon some of the alternate approaches we looked at on our way to this solution | 21:06 |
| timburke | oh yeah, and the client patch is | 21:07 |
| timburke | #link https://review.opendev.org/#/c/691877/ | 21:07 |
| patchbot | patch 691877 - python-swiftclient - object versioning features - 9 patch sets | 21:07 |
| timburke | #topic bulk regression | 21:07 |
| *** openstack changes topic to "bulk regression (Meeting topic: swift)" | 21:07 | |
| timburke | #link https://launchpad.net/bugs/1857546 | 21:07 |
| openstack | Launchpad bug 1857546 in OpenStack Object Storage (swift) "Auto-extract looses X-Delete-At and X-Delete-After headers" [High,Confirmed] | 21:07 |
| timburke | i saw rledisez put up an alternate fix | 21:08 |
| timburke | #link https://review.opendev.org/#/c/701736/ | 21:08 |
| patchbot | patch 701736 - swift - Allow bulk to fwd some headers at tar extraction - 2 patch sets | 21:08 |
| timburke | has anyone had a chance to look at either of the proposed fixes? | 21:08 |
| *** zaitcev has joined #openstack-meeting | 21:09 | |
| kota_ | that option 2 looks good to me. | 21:10 |
| kota_ | just looking now | 21:10 |
| timburke | i was just about to ask if anyone had a preference for which way we fix it ;-) | 21:10 |
| *** jamesmcarthur has joined #openstack-meeting | 21:10 | |
| *** dviroel has quit IRC | 21:10 | |
| zaitcev | If we're talking about the versioning prompted by S3, I haven't reviewed it, sorry. | 21:11 |
| timburke | i guess my only concern is all the other allowed_headers at https://github.com/openstack/swift/blob/2.23.0/etc/object-server.conf-sample#L135 | 21:11 |
| tdasilva | I haven't looked at either yet, but I'll try to take a look today | 21:12 |
| kota_ | ah, it's in object-server config... | 21:13 |
| timburke | zaitcev, this is the bulk-upload fix -- we've now got two solutions proposed: p 700652 (which mostly backs out the subrequest change that caused the regression but still fixes logging) and p 701736 (which starts copying over a subset of headers to the cleaned-up subrequest env) | 21:13 |
| patchbot | https://review.opendev.org/#/c/700652/ - swift - Revert "bulk: Use make_subrequest to make subreque... - 1 patch set | 21:13 |
| patchbot | https://review.opendev.org/#/c/701736/ - swift - Allow bulk to fwd some headers at tar extraction - 2 patch sets | 21:13 |
| *** lseki has quit IRC | 21:14 | |
| *** jamesmcarthur has quit IRC | 21:15 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:15 | |
| kota_ | hmm... | 21:16 |
| tdasilva | is allowed headers available in /info? | 21:16 |
| timburke | nope; wrong layer :-( | 21:16 |
| timburke | proxy vs object | 21:17 |
| tdasilva | i guess it would not be if it's a object server config | 21:17 |
| tdasilva | right | 21:17 |
| *** slaweq has quit IRC | 21:19 | |
| timburke | i'm not sure i like the hard-coded list -- seems like it'd be better as a configurable list... but then also allowing whole prefixes? hmm... trying to remember if we have some prior art for that... | 21:20 |
| kota_ | good point | 21:20 |
| timburke | or have object meta hardcoded, then configurable list for extra headers *shrug* | 21:21 |
| timburke | anyway, i suppose we can argue approaches more on the reviews | 21:22 |
| timburke | one extra topic to inject: | 21:22 |
| timburke | #topic next swift release | 21:22 |
| *** openstack changes topic to "next swift release (Meeting topic: swift)" | 21:22 | |
| timburke | it's been a while since we did a release, i think it's about time for a new one! | 21:22 |
| zaitcev | I lean towards Romain's approach, BTW | 21:23 |
| *** luyao has quit IRC | 21:23 | |
| timburke | i'll look to see what else we could merge for it, but i'm looking at three things, mainly: | 21:23 |
| timburke | versioning (ideally, with s3 support) | 21:23 |
| timburke | bulk fix | 21:23 |
| timburke | and py3 formpost fix | 21:23 |
| timburke | #link https://launchpad.net/bugs/1858259 | 21:24 |
| openstack | Launchpad bug 1858259 in OpenStack Object Storage (swift) "Upload failed if filename is unicode string in formpost (python3)" [High,Confirmed] | 21:24 |
| timburke | seongsoocho_ already looked at https://review.opendev.org/#/c/701497/ (thanks!) | 21:24 |
| patchbot | patch 701497 - swift - py3: Fix formpost unicode filename issues - 2 patch sets | 21:24 |
| seongsoocho_ | timburke: yey !!! o/ | 21:24 |
| timburke | if anyone's got a moment to give it a spin, that'd be great | 21:24 |
| tdasilva | I think p 697739 might also be able to make it to that list | 21:25 |
| patchbot | https://review.opendev.org/#/c/697739/ - swift - Have slo tell the object-server that it wants whol... - 6 patch sets | 21:25 |
| timburke | sounds good | 21:25 |
| timburke | *maybe* even https://review.opendev.org/700056 | 21:25 |
| patchbot | patch 700056 - swift - Middleware that allows a user to have quoted Etags - 5 patch sets | 21:25 |
| timburke | i'll get a fresh version up soon that actually has the right names everywhere ;-) | 21:26 |
| timburke | that's all i've got for today | 21:26 |
| timburke | #topic open discussion | 21:26 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:26 | |
| zaitcev | What about client 3.8.2 or something? | 21:26 |
| timburke | anything else people want to bring up? | 21:27 |
| timburke | zaitcev, yeah, that sounds good too | 21:27 |
| timburke | if we can get the versioning patch for the client, all the better :-) | 21:27 |
| zaitcev | On the dark data patch, or, rather auditor plugin patch https://review.opendev.org/212824, I had a change of heart and decided to agitate against the process separation. | 21:28 |
| patchbot | patch 212824 - swift - Let developers/operators add watchers to object audit - 20 patch sets | 21:28 |
| tdasilva | there's a also a client link patch that would be great to have, not sure its state tho | 21:28 |
| *** igordc has joined #openstack-meeting | 21:29 | |
| tdasilva | p 694211 | 21:29 |
| patchbot | https://review.opendev.org/#/c/694211/ - python-swiftclient - Support uploading Swift symlinks without content. - 7 patch sets | 21:29 |
| zaitcev | This is going to be a week of catching up on reviews. | 21:32 |
| tdasilva | zaitcev: was just thinking the same :) | 21:32 |
| timburke | zaitcev, i'd love that :D | 21:32 |
| timburke | all right, that's all i've got | 21:32 |
| zaitcev | Anyway, Sam and Alistair are not with us anymore. | 21:33 |
| zaitcev | So, I guess I own the auditor API if nobody objects. | 21:33 |
| zaitcev | Romain had some input... | 21:33 |
| alecuyer | Ill have to read up.. missed that patch, but that's a really nice idea | 21:33 |
| timburke | right after this i'll update the priority reviews wiki to reflect the release plan | 21:33 |
| alecuyer | (I see there's some history there :) ) | 21:33 |
| timburke | alecuyer, just a little ;-) | 21:33 |
| timburke | well, let's let seongsoocho_, kota_, and tdasilva grab breakfast if they haven't yet | 21:35 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:35 |
| kota_ | thx | 21:35 |
| timburke | #endmeeting | 21:35 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:35 | |
| openstack | Meeting ended Wed Jan 15 21:35:42 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:35 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-01-15-21.00.html | 21:35 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-01-15-21.00.txt | 21:35 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-01-15-21.00.log.html | 21:35 |
| *** alecuyer has left #openstack-meeting | 21:36 | |
| *** zaitcev has left #openstack-meeting | 21:36 | |
| *** tdasilva has quit IRC | 21:36 | |
| *** vishakha has quit IRC | 21:38 | |
| *** rcernin has quit IRC | 21:42 | |
| *** eharney has joined #openstack-meeting | 21:48 | |
| *** ociuhandu has quit IRC | 21:54 | |
| *** jamesmcarthur has quit IRC | 22:06 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:16 | |
| *** jamesmcarthur has quit IRC | 22:19 | |
| *** maciejjozefczyk has quit IRC | 22:35 | |
| *** dosaboy has quit IRC | 22:36 | |
| *** dmacpher has quit IRC | 22:37 | |
| *** dklyle has quit IRC | 22:40 | |
| *** diablo_rojo has quit IRC | 22:45 | |
| *** enriquetaso has joined #openstack-meeting | 22:47 | |
| *** dklyle has joined #openstack-meeting | 22:49 | |
| *** seongsoocho_ has quit IRC | 22:52 | |
| *** dosaboy has joined #openstack-meeting | 22:53 | |
| *** patchbot has left #openstack-meeting | 22:54 | |
| *** rcernin has joined #openstack-meeting | 22:55 | |
| *** igordc has quit IRC | 23:09 | |
| *** igordc has joined #openstack-meeting | 23:10 | |
| *** jamespage has joined #openstack-meeting | 23:27 | |
| *** tdasilva has joined #openstack-meeting | 23:37 | |
| *** tobiash has quit IRC | 23:38 | |
| *** tobiash has joined #openstack-meeting | 23:39 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!