| *** mattw4 has quit IRC | 00:03 | |
| *** ociuhandu has joined #openstack-meeting | 00:04 | |
| *** ociuhandu has quit IRC | 00:09 | |
| *** eharney has quit IRC | 00:15 | |
| *** nicolasbock has quit IRC | 00:18 | |
| *** jmasud has joined #openstack-meeting | 00:26 | |
| *** eharney has joined #openstack-meeting | 00:27 | |
| *** brinzhang has joined #openstack-meeting | 00:29 | |
| *** takamatsu has quit IRC | 00:35 | |
| *** macz_ has quit IRC | 00:36 | |
| *** takamatsu has joined #openstack-meeting | 00:52 | |
| *** rh-jelabarre has quit IRC | 00:58 | |
| *** ykatabam has joined #openstack-meeting | 01:00 | |
| *** brinzhang_ has joined #openstack-meeting | 01:04 | |
| *** brinzhang_ has quit IRC | 01:05 | |
| *** rh-jelabarre has joined #openstack-meeting | 01:05 | |
| *** brinzhang_ has joined #openstack-meeting | 01:06 | |
| *** brinzhang has quit IRC | 01:08 | |
| *** masahito has joined #openstack-meeting | 01:40 | |
| *** ykatabam has quit IRC | 01:42 | |
| *** yamamoto has joined #openstack-meeting | 01:47 | |
| *** ykatabam has joined #openstack-meeting | 01:52 | |
| *** ociuhandu has joined #openstack-meeting | 02:06 | |
| *** yamamoto has quit IRC | 02:07 | |
| *** yamamoto has joined #openstack-meeting | 02:10 | |
| *** ykatabam has quit IRC | 02:10 | |
| *** ociuhandu has quit IRC | 02:11 | |
| *** gyee has quit IRC | 02:13 | |
| *** jmasud has quit IRC | 02:22 | |
| *** ijw has quit IRC | 02:23 | |
| *** brinzhang has joined #openstack-meeting | 02:33 | |
| *** brinzhang_ has quit IRC | 02:36 | |
| *** ykatabam has joined #openstack-meeting | 02:56 | |
| *** ykatabam has quit IRC | 02:59 | |
| *** ykatabam has joined #openstack-meeting | 02:59 | |
| *** ijw has joined #openstack-meeting | 03:07 | |
| *** rh-jelabarre has quit IRC | 03:36 | |
| *** manuvakery has joined #openstack-meeting | 03:37 | |
| *** ykatabam has quit IRC | 03:40 | |
| *** ijw has quit IRC | 03:42 | |
| *** hyunsikyang__ has joined #openstack-meeting | 03:45 | |
| *** hyunsikyang has quit IRC | 03:48 | |
| *** hongbin has joined #openstack-meeting | 03:49 | |
| *** brinzhang_ has joined #openstack-meeting | 03:59 | |
| *** brinzhang_ has quit IRC | 04:01 | |
| *** brinzhang_ has joined #openstack-meeting | 04:01 | |
| *** brinzhang has quit IRC | 04:02 | |
| *** brinzhang_ has quit IRC | 04:03 | |
| *** brinzhang_ has joined #openstack-meeting | 04:03 | |
| *** brinzhang_ has quit IRC | 04:04 | |
| *** brinzhang_ has joined #openstack-meeting | 04:05 | |
| *** brinzhang has joined #openstack-meeting | 04:23 | |
| *** brinzhang has quit IRC | 04:25 | |
| *** brinzhang has joined #openstack-meeting | 04:25 | |
| *** brinzhang_ has quit IRC | 04:26 | |
| *** hongbin has quit IRC | 04:39 | |
| *** yaawang has joined #openstack-meeting | 04:44 | |
| *** brinzhang_ has joined #openstack-meeting | 04:48 | |
| *** brinzhang_ has quit IRC | 04:50 | |
| *** brinzhang_ has joined #openstack-meeting | 04:50 | |
| *** brinzhang has quit IRC | 04:51 | |
| *** brinzhang_ has quit IRC | 04:52 | |
| *** brinzhang_ has joined #openstack-meeting | 04:52 | |
| *** brinzhang_ has quit IRC | 04:54 | |
| *** brinzhang_ has joined #openstack-meeting | 04:54 | |
| *** brinzhang_ has quit IRC | 04:56 | |
| *** brinzhang_ has joined #openstack-meeting | 04:56 | |
| *** brinzhang_ has quit IRC | 04:58 | |
| *** brinzhang_ has joined #openstack-meeting | 04:59 | |
| *** brinzhang_ has quit IRC | 05:12 | |
| *** brinzhang_ has joined #openstack-meeting | 05:12 | |
| *** brinzhang_ has quit IRC | 05:14 | |
| *** brinzhang_ has joined #openstack-meeting | 05:14 | |
| *** brinzhang_ has quit IRC | 05:15 | |
| *** brinzhang_ has joined #openstack-meeting | 05:16 | |
| *** brinzhang_ has quit IRC | 05:17 | |
| *** brinzhang_ has joined #openstack-meeting | 05:18 | |
| *** brinzhang_ has quit IRC | 05:18 | |
| *** brinzhang_ has joined #openstack-meeting | 05:19 | |
| *** tetsuro has quit IRC | 05:20 | |
| *** tetsuro has joined #openstack-meeting | 05:22 | |
| *** links has joined #openstack-meeting | 05:25 | |
| *** vishalmanchanda has joined #openstack-meeting | 05:34 | |
| *** tetsuro has quit IRC | 05:39 | |
| *** manuvakery has quit IRC | 05:47 | |
| *** larainema has joined #openstack-meeting | 05:51 | |
| *** e0ne has joined #openstack-meeting | 05:59 | |
| *** jmasud has joined #openstack-meeting | 05:59 | |
| *** ykatabam has joined #openstack-meeting | 06:06 | |
| *** kozhukalov has joined #openstack-meeting | 06:08 | |
| *** ociuhandu has joined #openstack-meeting | 06:08 | |
| *** ociuhandu has quit IRC | 06:13 | |
| *** e0ne has quit IRC | 06:15 | |
| *** e0ne has joined #openstack-meeting | 06:16 | |
| *** ykatabam has quit IRC | 06:19 | |
| *** ykatabam has joined #openstack-meeting | 06:19 | |
| *** brinzhang_ has quit IRC | 06:20 | |
| *** brinzhang_ has joined #openstack-meeting | 06:21 | |
| *** brinzhang_ has quit IRC | 06:22 | |
| *** brinzhang_ has joined #openstack-meeting | 06:23 | |
| *** brinzhang has joined #openstack-meeting | 06:24 | |
| *** brinzhang has quit IRC | 06:25 | |
| *** brinzhang has joined #openstack-meeting | 06:25 | |
| *** brinzhang has quit IRC | 06:26 | |
| *** brinzhang has joined #openstack-meeting | 06:27 | |
| *** brinzhang has quit IRC | 06:27 | |
| *** brinzhang_ has quit IRC | 06:28 | |
| *** brinzhang has joined #openstack-meeting | 06:28 | |
| *** brinzhang has quit IRC | 06:29 | |
| *** brinzhang has joined #openstack-meeting | 06:30 | |
| *** e0ne has quit IRC | 06:30 | |
| *** tetsuro has joined #openstack-meeting | 06:31 | |
| *** brinzhang has quit IRC | 06:31 | |
| *** brinzhang has joined #openstack-meeting | 06:32 | |
| *** brinzhang has quit IRC | 06:33 | |
| *** brinzhang has joined #openstack-meeting | 06:34 | |
| *** brinzhang has quit IRC | 06:35 | |
| *** brinzhang has joined #openstack-meeting | 06:36 | |
| *** brinzhang has quit IRC | 06:39 | |
| *** brinzhang has joined #openstack-meeting | 06:40 | |
| *** brinzhang has quit IRC | 06:42 | |
| *** brinzhang has joined #openstack-meeting | 06:42 | |
| *** rcernin has quit IRC | 06:47 | |
| *** masahito has quit IRC | 06:49 | |
| *** ykatabam has quit IRC | 06:50 | |
| *** ociuhandu has joined #openstack-meeting | 07:02 | |
| *** ociuhandu has quit IRC | 07:07 | |
| *** Lucas_Gray has joined #openstack-meeting | 07:13 | |
| *** Lucas_Gray has quit IRC | 07:18 | |
| *** Lucas_Gray has joined #openstack-meeting | 07:22 | |
| *** lajoskatona has joined #openstack-meeting | 07:33 | |
| *** slaweq_ has joined #openstack-meeting | 07:42 | |
| *** ociuhandu has joined #openstack-meeting | 07:42 | |
| *** masahito has joined #openstack-meeting | 07:42 | |
| *** slaweq_ is now known as slaweq | 07:45 | |
| *** Lucas_Gray has quit IRC | 07:46 | |
| *** Lucas_Gray has joined #openstack-meeting | 07:48 | |
| *** ociuhandu has quit IRC | 07:48 | |
| *** ykatabam has joined #openstack-meeting | 07:54 | |
| *** brinzhang_ has joined #openstack-meeting | 08:00 | |
| *** brinzhang_ has quit IRC | 08:02 | |
| *** brinzhang_ has joined #openstack-meeting | 08:02 | |
| *** brinzhang has quit IRC | 08:03 | |
| *** Lucas_Gray has quit IRC | 08:05 | |
| *** ociuhandu has joined #openstack-meeting | 08:05 | |
| *** tesseract has joined #openstack-meeting | 08:06 | |
| *** jawad_axd has joined #openstack-meeting | 08:07 | |
| *** Lucas_Gray has joined #openstack-meeting | 08:07 | |
| *** Lucas_Gray has quit IRC | 08:11 | |
| *** ociuhandu has quit IRC | 08:11 | |
| *** jmasud has quit IRC | 08:13 | |
| *** jmasud has joined #openstack-meeting | 08:15 | |
| *** brinzhang has joined #openstack-meeting | 08:21 | |
| *** yaawang has quit IRC | 08:21 | |
| *** yaawang has joined #openstack-meeting | 08:21 | |
| *** brinzhang has quit IRC | 08:21 | |
| *** brinzhang has joined #openstack-meeting | 08:22 | |
| *** rcernin has joined #openstack-meeting | 08:22 | |
| *** brinzhang_ has quit IRC | 08:23 | |
| *** brinzhang has quit IRC | 08:23 | |
| *** brinzhang has joined #openstack-meeting | 08:24 | |
| *** brinzhang has quit IRC | 08:26 | |
| *** brinzhang has joined #openstack-meeting | 08:26 | |
| *** Lucas_Gray has joined #openstack-meeting | 08:27 | |
| *** brinzhang has quit IRC | 08:28 | |
| *** brinzhang has joined #openstack-meeting | 08:29 | |
| *** brinzhang has quit IRC | 08:31 | |
| *** brinzhang has joined #openstack-meeting | 08:31 | |
| *** brinzhang has quit IRC | 08:33 | |
| *** brinzhang has joined #openstack-meeting | 08:33 | |
| *** brinzhang has quit IRC | 08:35 | |
| *** brinzhang has joined #openstack-meeting | 08:35 | |
| *** brinzhang has quit IRC | 08:37 | |
| *** brinzhang has joined #openstack-meeting | 08:38 | |
| *** brinzhang has quit IRC | 08:38 | |
| *** brinzhang has joined #openstack-meeting | 08:39 | |
| *** Lucas_Gray has quit IRC | 08:40 | |
| *** brinzhang has quit IRC | 08:40 | |
| *** brinzhang has joined #openstack-meeting | 08:41 | |
| *** masahito has quit IRC | 08:41 | |
| *** masahito has joined #openstack-meeting | 08:41 | |
| *** oneswig has joined #openstack-meeting | 08:51 | |
| *** mdelavergne has joined #openstack-meeting | 08:53 | |
| *** ralonsoh has joined #openstack-meeting | 08:53 | |
| *** lajoskatona has left #openstack-meeting | 08:58 | |
| amorin | hello | 09:00 |
|---|---|---|
| ttx | #startmeeting large_scale_sig | 09:00 |
| openstack | Meeting started Wed Feb 26 09:00:09 2020 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:00 |
| *** openstack changes topic to " (Meeting topic: large_scale_sig)" | 09:00 | |
| openstack | The meeting name has been set to 'large_scale_sig' | 09:00 |
| ttx | Hi! | 09:00 |
| ttx | #topic Rollcall | 09:00 |
| *** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)" | 09:00 | |
| amorin | LO | 09:00 |
| ttx | amorin: how are things? | 09:00 |
| oneswig | hello | 09:00 |
| amorin | pretty well | 09:00 |
| ttx | Agenda for today at: | 09:01 |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-meeting | 09:01 |
| ttx | Let's wait a couple minutes for our friends in Asia | 09:02 |
| masahito | o/ | 09:02 |
| amorin | ack | 09:02 |
| ttx | jiaopengju1: around? | 09:03 |
| ttx | Ok, let's get started | 09:03 |
| ttx | #topic Progress on "Documenting large scale operations" goal | 09:03 |
| *** openstack changes topic to "Progress on "Documenting large scale operations" goal (Meeting topic: large_scale_sig)" | 09:03 | |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-documentation | 09:03 |
| ttx | amorin: any progress on documenting configuration defaults for large scale ? | 09:03 |
| ttx | any blocker / help needed? | 09:04 |
| amorin | yes | 09:04 |
| amorin | I have been working on that in the past few days | 09:04 |
| amorin | as you can see in the link you pasted | 09:04 |
| amorin | I collected some params that need to be tweaked on nova / neutron | 09:04 |
| amorin | some of them, it's just a beginning | 09:05 |
| amorin | but I'd like to go further with that now | 09:05 |
| amorin | so I have some questions for all of you :p | 09:05 |
| ttx | re: oslo.privsep on Neutron, last time I looked it was not there yet | 09:06 |
| amorin | first is, should we create a wiki page somewhere? | 09:06 |
| amorin | ah yes, thanks | 09:06 |
| ttx | It's mostly cinder/nova at this point | 09:06 |
| ttx | (and os-brick) | 09:06 |
| amorin | ok | 09:06 |
| *** lennyb has quit IRC | 09:06 | |
| amorin | so neutron and others are still using rootwrap daemon | 09:07 |
| amorin | that's the kind of info I want to share on the wiki page | 09:07 |
| amorin | do you know how I can create that? | 09:07 |
| ttx | and even there, most of the time it's a thin wrapper rather than a reimplementation of functions for higher security | 09:07 |
| amorin | (I havn't checked at all) | 09:07 |
| *** lennyb has joined #openstack-meeting | 09:07 | |
| ttx | so from a performance perspective, equivalent to rootwrap-daemon | 09:07 |
| *** ociuhandu has joined #openstack-meeting | 09:08 | |
| amorin | ack | 09:08 |
| ttx | only Nova did serious usage of privsep | 09:08 |
| ttx | resulting in stability, security and performance benefits | 09:08 |
| amorin | that's what came out from talk with matt | 09:08 |
| amorin | I dont want to disperse, but maybe the large scale group could also help on pushing privsep in other projects | 09:09 |
| ttx | re:wiki, I think it would be great to create a page under https://wiki.openstack.org/wiki/Large_Scale_SIG | 09:09 |
| ttx | amorin: yeah, could be a collaboration with the Security SIG... or some pop-up team to push for privsep in a specific direction | 09:10 |
| ttx | I have a long-standing TODO to work on privsepping osbrick | 09:10 |
| amorin | ok :p | 09:10 |
| oneswig | ttx: where do the performance benefits come in with privsep? | 09:10 |
| ttx | since it sounded like a fun project | 09:10 |
| ttx | oneswig: mostly in using narrow python functions rather than calling on IPC... gain depends on function called though | 09:11 |
| ttx | in some rare cases I could see it being actually slower, but overall there should be a small gain. | 09:11 |
| ttx | But the real gain is security | 09:12 |
| oneswig | OK, that makes sense thanks | 09:12 |
| ttx | amorin: so +1 on creating a wiki page | 09:12 |
| amorin | I will do then | 09:13 |
| amorin | and register the link on https://wiki.openstack.org/wiki/Large_Scale_SIG | 09:13 |
| ttx | ++ | 09:13 |
| amorin | that was the first point, | 09:13 |
| amorin | then, after the wiki page is done | 09:13 |
| amorin | I'd like to contribute to documentation, on nova for example | 09:13 |
| amorin | with a small change on some parameters docs | 09:13 |
| amorin | to show a link to this page | 09:14 |
| amorin | what do you think? | 09:14 |
| amorin | #action amorin do a wiki page for large scale documentation | 09:14 |
| ttx | it sounds like a good idea. To be discussed with Nova team I guess | 09:15 |
| *** lucasagomes has joined #openstack-meeting | 09:15 | |
| amorin | yes, I will do a proposal that could be used to start talking | 09:15 |
| ttx | maybe they would prefer making it direct part of doc, but at least at a first stage the content of that wiki page are likely to be a moving target | 09:15 |
| ttx | so until it stabilizes it really makes sense to expose it through a small doc patch | 09:16 |
| amorin | ok | 09:16 |
| ttx | amorin: anything else on that topic? | 09:16 |
| amorin | #action amorin start a patch on documentation for nova | 09:17 |
| amorin | last is, if you have some params on your side that could be identified in that documentation | 09:17 |
| amorin | feel free to add them :p | 09:17 |
| amorin | and that's all from me! | 09:17 |
| amorin | the scaling story here: | 09:17 |
| amorin | #link https://etherpad.openstack.org/p/scaling-stories | 09:17 |
| ttx | #idea pop up team between Large Scale SIG / Security SIG to push privsep in specific areas | 09:18 |
| amorin | is an amazing souce of info | 09:18 |
| ttx | yes, I wish there were more, really | 09:18 |
| amorin | I'd love to have more like that | 09:18 |
| slaweq | amorin: ttx: we are moving with many things to e.g. pyroute2 library to use privsep | 09:18 |
| amorin | +1 | 09:18 |
| amorin | slaweq, nice! | 09:18 |
| slaweq | but it's still not finished and in some places we are still using rootwrap | 09:18 |
| ttx | slaweq: great to see progress there ! | 09:18 |
| ttx | OK... Looking at the etherpad I don't think we had any new doc links added, so let's skip that | 09:19 |
| ttx | #topic Progress on "Scaling within one cluster" goal | 09:19 |
| *** openstack changes topic to "Progress on "Scaling within one cluster" goal (Meeting topic: large_scale_sig)" | 09:19 | |
| ttx | Re: https://etherpad.openstack.org/p/scaling-stories , no recent story posted, so we should carry that over | 09:19 |
| ttx | I was thinking of sending a periodic reminder | 09:20 |
| amorin | +1 | 09:20 |
| ttx | but ideally we'd have scaling stories from this group, to make it look more like a living resource | 09:20 |
| amorin | at OVH we can add some, I'll try to gather the info internally | 09:20 |
| ttx | Also I was thinking of doing a live exercise on this at the Large Scale usage of infrastructure track at OpenDev | 09:20 |
| ttx | (more on that later) | 09:20 |
| ttx | amorin: can be anecdotes, does not have to be as comprehensive as the first one posted | 09:21 |
| amorin | I am pretty sure CERN can also add some very interesting things | 09:21 |
| ttx | Like "I remember when we did this and had to do this to get it to work" | 09:21 |
| ttx | We got a couple more reviews on oslo.metrics blueprint at https://review.opendev.org/#/c/704733/ | 09:22 |
| amorin | yes | 09:22 |
| ttx | masahito: Based on bnemec's comment, I'd say we should not overthink it... Maybe post a new patchset to take comments into account and run for approval ? | 09:22 |
| masahito | Thanks for the comments. | 09:22 |
| *** ociuhandu has quit IRC | 09:22 | |
| masahito | ah, yes. I plan to update this in tomorrow. | 09:23 |
| ttx | Nice! | 09:23 |
| oneswig | sounds great | 09:23 |
| amorin | great, for me it's quite ok, I think we must first focus on simplicity | 09:23 |
| amorin | one of my concern was about the fact that oslo metrics was sending the data somewhere | 09:24 |
| amorin | are we sure we want to do that? | 09:24 |
| ttx | yes and getting the code out there. If there was a fatal flaw in the idea, we'd have had more comments on that spec :) | 09:24 |
| ttx | amorin: you mean central collection? what would the alternative be? | 09:24 |
| *** psachin has joined #openstack-meeting | 09:25 | |
| masahito | And as Ben commented it's better to public our codes for a first look. But I need to check which doesn't have sensitive information in our company. | 09:25 |
| amorin | IMHO, oslo.metrics should collect the metrics and leave it locally on an unix socket | 09:25 |
| ttx | masahito: of course, take your time :) | 09:25 |
| amorin | the central collection could be done by antoher tool | 09:26 |
| amorin | but if your implementation is already doing it, then it's ok :p | 09:26 |
| masahito | amorin: Good point. In our cases, oslo.metrics sends data. But if that's not good for other company, I don't want to push our cases. | 09:26 |
| amorin | in my company, we already have some kind of collectors on hosts, | 09:27 |
| amorin | so I'd like to reuse those | 09:27 |
| ttx | amorin: could that be one config option? | 09:27 |
| amorin | of course | 09:27 |
| amorin | I wasnt aware that an implemetnation was already done | 09:27 |
| ttx | maybe we can propose that once the original code is posted | 09:28 |
| amorin | make sense | 09:28 |
| ttx | amorin: it's what masahito presented in Shanghai, running at LINE | 09:28 |
| masahito | amorin: Not a much implementation. We just started. | 09:28 |
| amorin | ah ok! | 09:28 |
| amorin | perfect then ;p | 09:28 |
| *** Lucas_Gray has joined #openstack-meeting | 09:28 | |
| ttx | If there is one thing I learned about telemetry and metrics collection, it's that you need to be flexible to adapt to every case | 09:29 |
| ttx | and that simple is better than complex for that reason | 09:29 |
| ttx | ok, anything else to discuss re: "Scaling within one cluster" goal ? | 09:30 |
| oneswig | I added something | 09:30 |
| ttx | #action masahito to post new patchset of spec | 09:30 |
| oneswig | on bare metal scaling. Is it time for that? | 09:30 |
| ttx | yes, in a sec | 09:31 |
| ttx | #topic Other topics | 09:31 |
| *** openstack changes topic to "Other topics (Meeting topic: large_scale_sig)" | 09:31 | |
| ttx | oneswig: go for it | 09:31 |
| oneswig | ah right | 09:31 |
| oneswig | so I was thinking this might be another front for discussion | 09:31 |
| oneswig | how big can bare metal clusters go, and how are they managed in extremis? | 09:32 |
| oneswig | We have experience with working on the rate at which nodes can be deployed concurrently | 09:32 |
| ttx | I feel like that's definitely aligned with the "scaling within one cluster" goal, for a specific definition of cluster | 09:33 |
| oneswig | One limitation can be the processing rate of networking-generic-switch, if we aren't reverting to flat networking | 09:33 |
| oneswig | ttx: think so too :-) | 09:33 |
| ttx | oneswig: it could start as a scaling story | 09:34 |
| oneswig | I don't have much to contribute on this yet but I'm happy to write and maintain a scaling story for this based on our experiences. | 09:34 |
| ttx | I don;t think we have enough data points to start making separate categories | 09:34 |
| amorin | that could be nice | 09:34 |
| amorin | you eventually found some tunings to be done for your story, no? | 09:34 |
| oneswig | cool, sign me up. | 09:34 |
| ttx | Once we have plenty of stories we'll have to think on how to best present them, but for now the big etherpad is enough | 09:35 |
| oneswig | Well, the most recent work has been a combination of tunables and refactoring of n-g-s | 09:35 |
| ttx | #action oneswig to contribute a scaling story on bare metal cluster scaling | 09:35 |
| oneswig | nb, might take a couple of months | 09:35 |
| *** tesseract has quit IRC | 09:35 | |
| ttx | oneswig: Oath / Verizon Media has a massive footprint of bare metal clusters, would be nice to hear from them | 09:35 |
| ttx | which brings us to our next point | 09:36 |
| oneswig | for sure! | 09:36 |
| ttx | belmoreira and masahito volunteered to join the programming committee for the "Large-scale Usage of Open Source Infrastructure Software" track | 09:36 |
| ttx | at OpenDev event in Vancouver, June 8-11 | 09:36 |
| ttx | It's our opportunity to shape the discussion so that it aligns with the group's objectives | 09:36 |
| ttx | And recruit new members | 09:36 |
| masahito | We also have tons of baremetal but not managed by OpenStack ;-< | 09:36 |
| *** tesseract has joined #openstack-meeting | 09:36 | |
| ttx | I'm already involved with another track, so Allison Price will be the OSF contact to organize that track. | 09:36 |
| ttx | I'll follow closely still, and we can discuss it in this meeting | 09:37 |
| amorin | unfortunately I wont be able to attend | 09:37 |
| ttx | amorin: that's unfortunate | 09:37 |
| ttx | A track has several subthemes... | 09:37 |
| oneswig | I expect to be there and will participate provided it doesn't conflict with the scientific sig | 09:37 |
| ttx | So we could have one around collecting scaling stories from the participants | 09:37 |
| ttx | oneswig: it does not | 09:37 |
| masahito | I plan to be there as ttx said | 09:37 |
| ttx | Opendev tracks are in the mornings and PTG in the afternoons (+ last day) | 09:38 |
| ttx | We could also have a specific theme around scaling bare metal clusters | 09:38 |
| ttx | The format is pretty open, the idea is to prefer direct engagement over formal talks | 09:38 |
| oneswig | ttx: sounds good to me. | 09:39 |
| ttx | so as you are contacted and start discussing the contents, please push our SIG's agenda ! | 09:39 |
| oneswig | +1 | 09:40 |
| ttx | ideally we'd recruit new members, but at the very least we should collect information | 09:40 |
| ttx | I'd really like a scaling-stories session where people can informally talk about scaling anecdotes, and then we'd take notes and follow up with them to submit something more detailed | 09:41 |
| ttx | #topic Next meeting | 09:41 |
| *** openstack changes topic to "Next meeting (Meeting topic: large_scale_sig)" | 09:41 | |
| ttx | everyone still OK with a biweekly frequency? | 09:41 |
| amorin | it's ok for me | 09:41 |
| oneswig | yes | 09:41 |
| masahito | yes | 09:41 |
| *** ociuhandu has joined #openstack-meeting | 09:42 | |
| ttx | #info next meeting: March 11 | 09:42 |
| ttx | Anything else to mention before we close ? | 09:42 |
| oneswig | nothing here | 09:43 |
| ttx | #idea have a scaling-stories session where people can informally talk about scaling anecdotes, and then we'd take notes and follow up with them to submit something more detailed | 09:43 |
| ttx | ok I think I captured everything | 09:43 |
| ttx | Thanks everyone | 09:44 |
| ttx | #endmeeting | 09:44 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 09:44 | |
| openstack | Meeting ended Wed Feb 26 09:44:07 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:44 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-02-26-09.00.html | 09:44 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-02-26-09.00.txt | 09:44 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-02-26-09.00.log.html | 09:44 |
| masahito | Thanks all! | 09:44 |
| amorin | thanks | 09:44 |
| oneswig | thanks | 09:44 |
| *** witek has joined #openstack-meeting | 09:46 | |
| *** witek has quit IRC | 10:00 | |
| *** oneswig has quit IRC | 10:03 | |
| *** witek has joined #openstack-meeting | 10:08 | |
| *** ociuhandu has quit IRC | 10:11 | |
| *** witek has quit IRC | 10:15 | |
| *** ociuhandu has joined #openstack-meeting | 10:21 | |
| *** masahito has quit IRC | 10:25 | |
| *** ociuhandu has quit IRC | 10:31 | |
| *** e0ne has joined #openstack-meeting | 10:37 | |
| *** e0ne has quit IRC | 10:38 | |
| *** Lucas_Gray has quit IRC | 10:48 | |
| *** e0ne has joined #openstack-meeting | 10:54 | |
| *** Lucas_Gray has joined #openstack-meeting | 10:54 | |
| *** noggin143 has joined #openstack-meeting | 10:57 | |
| *** brinzhang_ has joined #openstack-meeting | 10:57 | |
| *** brinzhang_ has quit IRC | 10:59 | |
| *** oneswig has joined #openstack-meeting | 10:59 | |
| *** Lucas_Gray has quit IRC | 10:59 | |
| *** brinzhang_ has joined #openstack-meeting | 10:59 | |
| oneswig | #startmeeting scientific-sig | 11:00 |
| openstack | Meeting started Wed Feb 26 11:00:13 2020 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 11:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 11:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 11:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 11:00 |
| oneswig | greetings | 11:00 |
| *** Lucas_Gray has joined #openstack-meeting | 11:00 | |
| *** brinzhang has quit IRC | 11:00 | |
| oneswig | #link today's agenda (such as it is) https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_February_26th_2020 | 11:01 |
| *** priteau has joined #openstack-meeting | 11:01 | |
| noggin143 | lo | 11:02 |
| noggin143 | got the ironic team at CERN this week on the midcycle F2F | 11:03 |
| oneswig | Hi noggin143 | 11:03 |
| oneswig | how is it going? | 11:03 |
| *** janders has joined #openstack-meeting | 11:03 | |
| verdurin | Morning. | 11:04 |
| janders | g'day! | 11:04 |
| oneswig | Hello verdurin janders | 11:04 |
| janders | sorry finishing off some work on my 4wd still :) | 11:04 |
| oneswig | hardware hacking? | 11:04 |
| noggin143 | and a chance to tour some of the facilities | 11:05 |
| oneswig | noggin143: I joined the large-scale sig meeting this morning and suggested a user story for large-scale baremetal | 11:05 |
| oneswig | There was discussion there about a session track at the Vancouver PTG | 11:06 |
| ttx | Opendev actually but yes | 11:06 |
| oneswig | Ah, hi ttx | 11:06 |
| oneswig | thanks :-) | 11:06 |
| oneswig | also a hope that the CERN team might contribute some experiences to https://etherpad.openstack.org/p/scaling-stories | 11:07 |
| *** ociuhandu has joined #openstack-meeting | 11:07 | |
| janders | that would be a very interesting read! noggin143 +1 | 11:08 |
| noggin143 | will have a look... | 11:08 |
| *** tesseract has quit IRC | 11:08 | |
| oneswig | thanks noggin143, share and enjoy! | 11:08 |
| *** tesseract has joined #openstack-meeting | 11:11 | |
| oneswig | Martial has requested half a day (or slightly more) for the scientific sig at Vancouver | 11:11 |
| oneswig | I haven't seen the scheduling yet | 11:11 |
| *** ociuhandu has quit IRC | 11:12 | |
| *** janders has quit IRC | 11:13 | |
| oneswig | Another useful development was that the oslo.metrics blueprint is getting a good reception - https://review.opendev.org/#/c/704733/ | 11:15 |
| oneswig | verdurin: are you involved with the workshop for https://cloud.ac.uk/ this year? | 11:20 |
| noggin143 | We'll have a couple of folk there too (Theodoros and Joao) | 11:23 |
| oneswig | noggin143: great - isn't Theodoros presenting? | 11:23 |
| noggin143 | yes, both are | 11:24 |
| oneswig | I know Bharat from our team is looking forward to meeting Theodoros | 11:24 |
| noggin143 | Theodoros on the current state / ongoing activities and Joao on the EU ARCHIVER project using public cloud storage for science | 11:25 |
| verdurin | oneswig: Yes, I am one of the organisers, and I'm chairing a session. | 11:26 |
| oneswig | It looks like it will be another good year, hope it goes smoothly verdurin | 11:27 |
| verdurin | There was a bit of competition to chair the restricted data session, and I lost. | 11:27 |
| *** e0ne has quit IRC | 11:27 | |
| oneswig | I was somewhat alarmed to follow someone from the Alan Turing Institute - was intending to present details on the paper they published on this myself... | 11:28 |
| oneswig | ... I hastily redrafted my content | 11:28 |
| oneswig | OK well I don't think there's much to cover this week, let's do AOB | 11:29 |
| oneswig | #topic AOB | 11:29 |
| *** openstack changes topic to "AOB (Meeting topic: scientific-sig)" | 11:29 | |
| oneswig | I'm getting to the bottom of doing SR-IOV on top of 2-port bonded NICs, am hoping to write up the findings within a week or two | 11:30 |
| *** janders has joined #openstack-meeting | 11:30 | |
| oneswig | should be interesting but the question of "was it worth the trouble" is harder to answer | 11:31 |
| janders | in case of fitting the winch and radios on the 4wd - for sure! :) | 11:31 |
| noggin143 | we're looking into a tough ceph issue from last week, current suspicion is that LZ4 compression may have problems | 11:31 |
| noggin143 | but nothing for sure | 11:31 |
| oneswig | noggin143: ouch. How do you isolate that? | 11:31 |
| *** ianychoi_ is now known as ianychoi | 11:32 | |
| noggin143 | had to rebuild the osd-maps. Dan will write up something once the analysis is complete | 11:32 |
| oneswig | sounds interesting, thanks. | 11:32 |
| oneswig | Do you know how the seastar messenger work is progressing btw? | 11:32 |
| noggin143 | I don't know the details, was due to get an update at cephalacon but it got cancelled (South Korea not the best place to visit at the moment)) | 11:33 |
| *** Lucas_Gray has quit IRC | 11:34 | |
| oneswig | I'm having not very much success in finding the point in the chain where packets are being discarded, when it appears to be set up for end-to-end flow control. Very niggly. | 11:34 |
| oneswig | OK, any more AOB to share? | 11:36 |
| janders | nothing from my side | 11:36 |
| verdurin | Nor from me. | 11:36 |
| oneswig | In which case I shall return to my dark cave of LAGs, SRIOV and troubleshooting packet drops... | 11:37 |
| oneswig | thanks all, see you next week verdurin | 11:38 |
| oneswig | #endmeeting | 11:38 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 11:38 | |
| openstack | Meeting ended Wed Feb 26 11:38:11 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 11:38 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-02-26-11.00.html | 11:38 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-02-26-11.00.txt | 11:38 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-02-26-11.00.log.html | 11:38 |
| *** noggin143 has quit IRC | 11:38 | |
| *** oneswig has quit IRC | 11:38 | |
| *** janders has quit IRC | 11:40 | |
| *** brinzhang has joined #openstack-meeting | 11:46 | |
| *** brinzhang_ has quit IRC | 11:49 | |
| *** yamamoto has quit IRC | 11:49 | |
| *** ociuhandu has joined #openstack-meeting | 11:51 | |
| *** ociuhandu has quit IRC | 11:55 | |
| *** brinzhang_ has joined #openstack-meeting | 12:02 | |
| *** nicolasbock has joined #openstack-meeting | 12:02 | |
| *** brinzhang_ has quit IRC | 12:03 | |
| *** brinzhang_ has joined #openstack-meeting | 12:04 | |
| *** brinzhang has quit IRC | 12:05 | |
| *** yamamoto has joined #openstack-meeting | 12:09 | |
| *** jmasud has quit IRC | 12:12 | |
| *** Lucas_Gray has joined #openstack-meeting | 12:13 | |
| *** jmasud has joined #openstack-meeting | 12:14 | |
| *** enriquetaso has joined #openstack-meeting | 12:19 | |
| *** mdelavergne has quit IRC | 12:24 | |
| *** psachin has quit IRC | 12:25 | |
| *** ociuhandu has joined #openstack-meeting | 12:30 | |
| *** apdibbo has joined #openstack-meeting | 12:34 | |
| *** witek_ has joined #openstack-meeting | 12:36 | |
| *** ociuhandu has quit IRC | 12:40 | |
| *** witek_ has quit IRC | 12:47 | |
| *** yamamoto has quit IRC | 12:51 | |
| *** brinzhang has joined #openstack-meeting | 12:54 | |
| *** rh-jelabarre has joined #openstack-meeting | 13:04 | |
| *** e0ne has joined #openstack-meeting | 13:04 | |
| *** psachin has joined #openstack-meeting | 13:05 | |
| *** ykatabam has quit IRC | 13:05 | |
| *** ociuhandu has joined #openstack-meeting | 13:06 | |
| *** brinzhang has quit IRC | 13:27 | |
| *** brinzhang has joined #openstack-meeting | 13:29 | |
| *** brinzhang has quit IRC | 13:30 | |
| *** brinzhang__ has joined #openstack-meeting | 13:30 | |
| *** brinzhang has joined #openstack-meeting | 13:31 | |
| *** brinzhang_ has quit IRC | 13:32 | |
| *** brinzhang has quit IRC | 13:32 | |
| *** brinzhang has joined #openstack-meeting | 13:33 | |
| *** brinzhang has quit IRC | 13:34 | |
| *** Lucas_Gray has quit IRC | 13:34 | |
| *** brinzhang has joined #openstack-meeting | 13:35 | |
| *** Lucas_Gray has joined #openstack-meeting | 13:35 | |
| *** brinzhang has quit IRC | 13:36 | |
| *** brinzhang has joined #openstack-meeting | 13:37 | |
| *** ysandeep is now known as ysandeep|away | 13:38 | |
| *** nicolasbock has quit IRC | 13:40 | |
| *** nicolasbock has joined #openstack-meeting | 13:40 | |
| *** e0ne has quit IRC | 13:43 | |
| *** witek_ has joined #openstack-meeting | 13:44 | |
| *** brinzhang has quit IRC | 13:46 | |
| *** brinzhang has joined #openstack-meeting | 13:52 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:53 | |
| *** Liang__ has joined #openstack-meeting | 13:53 | |
| *** Liang__ is now known as LiangFang | 13:54 | |
| *** anastzhyr has joined #openstack-meeting | 13:58 | |
| *** Lucas_Gray has quit IRC | 13:59 | |
| *** Lucas_Gray has joined #openstack-meeting | 14:00 | |
| *** e0ne has joined #openstack-meeting | 14:01 | |
| *** sfernand has joined #openstack-meeting | 14:01 | |
| *** slaweq has quit IRC | 14:05 | |
| *** Lucas_Gray has quit IRC | 14:05 | |
| *** Lucas_Gray has joined #openstack-meeting | 14:07 | |
| *** apdibbo_ has joined #openstack-meeting | 14:07 | |
| *** geguileo has joined #openstack-meeting | 14:07 | |
| *** dasp has quit IRC | 14:08 | |
| *** slaweq has joined #openstack-meeting | 14:09 | |
| *** apdibbo has quit IRC | 14:10 | |
| *** witek_ has quit IRC | 14:12 | |
| *** enriquetaso has quit IRC | 14:16 | |
| *** enriquetaso has joined #openstack-meeting | 14:16 | |
| *** maciejjozefczyk has joined #openstack-meeting | 14:18 | |
| *** yamamoto has joined #openstack-meeting | 14:22 | |
| *** jamesmcarthur has quit IRC | 14:27 | |
| *** apdibbo__ has joined #openstack-meeting | 14:28 | |
| *** apdibbo_ has quit IRC | 14:29 | |
| *** jamesmcarthur has joined #openstack-meeting | 14:30 | |
| *** jamesmcarthur has quit IRC | 14:32 | |
| *** links has quit IRC | 14:36 | |
| *** ociuhandu has quit IRC | 14:37 | |
| *** yamamoto has quit IRC | 14:48 | |
| *** yamamoto has joined #openstack-meeting | 14:50 | |
| *** yamamoto has quit IRC | 14:55 | |
| *** TrevorV has joined #openstack-meeting | 15:01 | |
| *** psachin has quit IRC | 15:05 | |
| *** yamamoto has joined #openstack-meeting | 15:10 | |
| *** ociuhandu has joined #openstack-meeting | 15:11 | |
| *** TrevorV has quit IRC | 15:11 | |
| *** TrevorV has joined #openstack-meeting | 15:11 | |
| *** Lucas_Gray has quit IRC | 15:14 | |
| *** Lucas_Gray has joined #openstack-meeting | 15:15 | |
| *** ociuhandu has quit IRC | 15:16 | |
| *** mugsie has quit IRC | 15:20 | |
| *** mugsie has joined #openstack-meeting | 15:22 | |
| *** brinzhang has quit IRC | 15:22 | |
| *** brinzhang has joined #openstack-meeting | 15:23 | |
| *** Lucas_Gray has quit IRC | 15:32 | |
| *** Lucas_Gray has joined #openstack-meeting | 15:34 | |
| *** ociuhandu has joined #openstack-meeting | 15:41 | |
| *** ijw has joined #openstack-meeting | 15:44 | |
| *** psachin has joined #openstack-meeting | 15:46 | |
| *** Trevor_V has joined #openstack-meeting | 15:46 | |
| *** ijw has quit IRC | 15:48 | |
| *** TrevorV has quit IRC | 15:49 | |
| *** dasp has joined #openstack-meeting | 15:52 | |
| *** jawad_axd has quit IRC | 15:56 | |
| *** psachin has quit IRC | 15:57 | |
| *** e0ne has quit IRC | 15:59 | |
| *** TrevorV has joined #openstack-meeting | 15:59 | |
| *** e0ne has joined #openstack-meeting | 15:59 | |
| *** Trevor_V has quit IRC | 16:03 | |
| *** e0ne has quit IRC | 16:10 | |
| *** apdibbo has joined #openstack-meeting | 16:16 | |
| *** eharney has quit IRC | 16:19 | |
| *** apdibbo__ has quit IRC | 16:19 | |
| *** maciejjozefczyk has quit IRC | 16:21 | |
| *** mattw4 has joined #openstack-meeting | 16:33 | |
| *** apdibbo has quit IRC | 16:34 | |
| *** brinzhang has quit IRC | 16:35 | |
| *** jamesmcarthur has joined #openstack-meeting | 16:36 | |
| *** brinzhang has joined #openstack-meeting | 16:36 | |
| *** psachin has joined #openstack-meeting | 16:36 | |
| *** phrobb has joined #openstack-meeting | 16:36 | |
| *** brinzhang has quit IRC | 16:37 | |
| *** brinzhang has joined #openstack-meeting | 16:38 | |
| *** ijw has joined #openstack-meeting | 16:38 | |
| *** brinzhang has quit IRC | 16:38 | |
| *** yamamoto has quit IRC | 16:39 | |
| *** rubasov has quit IRC | 16:47 | |
| *** tesseract has quit IRC | 16:50 | |
| *** gyee has joined #openstack-meeting | 16:52 | |
| *** eharney has joined #openstack-meeting | 16:55 | |
| *** lucasagomes has quit IRC | 17:01 | |
| *** ociuhandu has quit IRC | 17:03 | |
| *** mattw4 has quit IRC | 17:03 | |
| *** mattw4 has joined #openstack-meeting | 17:04 | |
| *** yamamoto has joined #openstack-meeting | 17:11 | |
| *** psachin has quit IRC | 17:12 | |
| *** yamamoto has quit IRC | 17:17 | |
| *** ociuhandu has joined #openstack-meeting | 17:30 | |
| *** mattw4 has quit IRC | 17:34 | |
| *** mattw4 has joined #openstack-meeting | 17:44 | |
| *** vishalmanchanda has quit IRC | 17:47 | |
| *** e0ne has joined #openstack-meeting | 17:48 | |
| *** jamesmcarthur has quit IRC | 17:58 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:00 | |
| *** ociuhandu has quit IRC | 18:05 | |
| *** jamesmcarthur has quit IRC | 18:05 | |
| *** mattw4 has quit IRC | 18:05 | |
| *** ociuhandu has joined #openstack-meeting | 18:05 | |
| *** ociuhandu has quit IRC | 18:11 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:21 | |
| *** yamamoto has joined #openstack-meeting | 18:22 | |
| *** ijw has quit IRC | 18:27 | |
| *** priteau has quit IRC | 18:32 | |
| *** mattw4 has joined #openstack-meeting | 18:36 | |
| *** igordc has joined #openstack-meeting | 18:37 | |
| *** maciejjozefczyk has joined #openstack-meeting | 18:42 | |
| *** igordc has quit IRC | 18:43 | |
| *** jmasud has quit IRC | 18:45 | |
| *** ociuhandu has joined #openstack-meeting | 18:46 | |
| *** mahatic has joined #openstack-meeting | 18:51 | |
| *** ociuhandu has quit IRC | 18:51 | |
| *** maciejjozefczyk has quit IRC | 18:57 | |
| *** Lucas_Gray has quit IRC | 18:57 | |
| *** mattw4 has quit IRC | 19:09 | |
| *** mattw4 has joined #openstack-meeting | 19:17 | |
| *** jamesmcarthur has quit IRC | 19:31 | |
| *** yamamoto has quit IRC | 19:31 | |
| *** ociuhandu has joined #openstack-meeting | 19:34 | |
| *** eharney has quit IRC | 19:38 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:40 | |
| *** ociuhandu has quit IRC | 19:40 | |
| *** eharney has joined #openstack-meeting | 19:51 | |
| *** jamesmcarthur has quit IRC | 19:58 | |
| *** yamamoto has joined #openstack-meeting | 20:02 | |
| *** e0ne has quit IRC | 20:06 | |
| *** yamamoto has quit IRC | 20:08 | |
| *** ociuhandu has joined #openstack-meeting | 20:08 | |
| *** ociuhandu has quit IRC | 20:12 | |
| *** rsimai has quit IRC | 20:13 | |
| *** rsimai has joined #openstack-meeting | 20:21 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:21 | |
| *** Lucas_Gray has joined #openstack-meeting | 20:25 | |
| *** ociuhandu has joined #openstack-meeting | 20:28 | |
| *** kozhukalov has quit IRC | 20:33 | |
| *** kozhukalov has joined #openstack-meeting | 20:36 | |
| *** mattw4 has quit IRC | 20:42 | |
| *** mattw4 has joined #openstack-meeting | 20:42 | |
| *** Lucas_Gray has quit IRC | 20:50 | |
| *** ykatabam has joined #openstack-meeting | 20:51 | |
| *** alecuyer has joined #openstack-meeting | 20:51 | |
| *** Lucas_Gray has joined #openstack-meeting | 20:56 | |
| *** ykatabam has quit IRC | 20:57 | |
| clayg | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Feb 26 21:00:14 2020 UTC and is due to finish in 60 minutes. The chair is clayg. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| clayg | anyone here for the swift meeting today? | 21:00 |
| seongsoocho | o/ | 21:00 |
| alecuyer | hi | 21:00 |
| *** eharney has quit IRC | 21:00 | |
| kota_ | hi | 21:01 |
| rledisez | hi o/ | 21:01 |
| *** zaitcev has joined #openstack-meeting | 21:01 | |
| mattoliverau | o/ | 21:01 |
| clayg | tdasilva: if i'm remember where you're at it's getting kinda late for you? | 21:01 |
| *** ykatabam has joined #openstack-meeting | 21:03 | |
| clayg | Ok, so Tim is busy - let's go ahead and do a quick meeting | 21:03 |
| *** larainema has quit IRC | 21:03 | |
| clayg | Agenda is at https://wiki.openstack.org/wiki/Meetings/Swift | 21:03 |
| clayg | #topic vancouver | 21:03 |
| *** openstack changes topic to "vancouver (Meeting topic: swift)" | 21:03 | |
| clayg | I think Tim just wants to reminder everyone this is coming up | 21:03 |
| clayg | we're still in planning and I guess we still have some time | 21:04 |
| clayg | If anyone has any questions ... probably best to email timburke :D | 21:04 |
| mattoliverau | lol | 21:04 |
| alecuyer | hehe | 21:04 |
| clayg | #topic concurrent gets for EC | 21:05 |
| *** openstack changes topic to "concurrent gets for EC (Meeting topic: swift)" | 21:05 | |
| mattoliverau | I assume this is the waterfall topic on gerrit? | 21:05 |
| clayg | I peeled this one off our queue - my first attempt didn't work out 😢 | 21:06 |
| mattoliverau | :( | 21:06 |
| clayg | mattoliverau: yeah, waterfall-ec https://review.opendev.org/#/c/709326/ | 21:06 |
| clayg | I thought I saw a way to life the concurrency context up one level where EC could manage it across all the ResumingGetters... but I don't think it's gunna work 😞 | 21:07 |
| *** ykatabam has quit IRC | 21:07 | |
| clayg | basically we DO want to to have ndata responses, but picking which "frag lane" to spawn the extra requests in is going to be ... maybe impossible? | 21:08 |
| mattoliverau | I was things do we want that? does it matter if we look at a few hand offs that might actaully have some ndata one them | 21:09 |
| mattoliverau | (just thinking out loud) | 21:09 |
| alecuyer | you mean we could start a few extra requests, in case some of them fail? | 21:10 |
| clayg | So I think it's time to put the GetOrHeadHandler back together and let it work as intended for replicated requests then just extend they way we're already spawning and joining requests in the ECObjectController so it works like replicate concurrent_gets | 21:10 |
| clayg | then maybe in the end it's possible to pull out something smaller or simpler that can be re-used - but for now I'm just trying to get both cases working the way we want | 21:10 |
| clayg | mattoliverau: I think replicated (and ec) should not open a request to a handoff until at least one primary comes back with a timeout or error | 21:11 |
| clayg | alecuyer: yes, the config options should *allow* an operator to spawn up to ec_n_unquie_frags (data+parity) requests concurrently and take the first ndata | 21:12 |
| clayg | alecuyer: unfortunately when testing that case I found the original design didn't degrade gracefully into that working well | 21:12 |
| alecuyer | clayg: that sounds good (to me at least? :D) | 21:12 |
| alecuyer | ok | 21:12 |
| clayg | alecuyer: in the general sense I think we'll open connections to the first ndata sorted by timing - then after a small timeout (500ms) open another connection (or more?) to the other primaries to make up the difference for those yet to come back | 21:13 |
| clayg | ... but definately as soon as we have ndata responses we can start to iter out the fragarchives and return a response | 21:13 |
| alecuyer | right | 21:13 |
| clayg | any questions? | 21:13 |
| mattoliverau | what can we do to help? need anything from us at this point. Or should I just review the topic? | 21:14 |
| zaitcev | So, reconstruct every time, even if systematic scheme could possibly deliver data in a few more milliseconds? | 21:14 |
| clayg | zaitcev: I don't think we have a natural prefernce to node order currently - default sort order is timing - so parity frags might already be coming back | 21:15 |
| clayg | zaitcev: in general we assume the rebuild is cheap compared to all the other work of getting bytes off disk and across a network | 21:15 |
| clayg | mattoliverau: I was hoping I would need some help - but it seems i'm back to the drawing board | 21:15 |
| rledisez | zaitcev: is there an algorithmic difference between only data frag and data+parity frags? | 21:16 |
| clayg | I'd like to keep moving - hopefully I'll have some more to sa ynext week | 21:16 |
| rledisez | zaitcev: I always assumed it was the same math | 21:16 |
| mattoliverau | :( | 21:16 |
| clayg | #topic forced client backoff on 503 | 21:16 |
| *** openstack changes topic to "forced client backoff on 503 (Meeting topic: swift)" | 21:16 | |
| clayg | just this morning in a side conversation with Ed on the proxyfs team we ended up thinking about forcing clients to backoff by throwing some sleep on our 503 responses | 21:17 |
| mattoliverau | rledisez: data can just be basically appended because it's the data, parity need to be be computed before appendings. So getting ndata should be more effiecent. AFAIK | 21:17 |
| mattoliverau | oh interesting | 21:18 |
| kota_ | mattoliverau: correct | 21:18 |
| rledisez | mattoliverau: thx, very instructive | 21:18 |
| clayg | the issue came up in a thundering heard scenario where an artificial load generator would do like 200 requests slam a bunch of disks then wait a fixed amount of time before doing it again | 21:18 |
| clayg | the "solution" was to make do some random dithering to the retries - but he pointed out we could do that server side | 21:18 |
| clayg | I honestly hadn't considered it | 21:19 |
| clayg | Has anyone ever done anything like this in middleware or on other services? | 21:19 |
| mattoliverau | ok so backing off at while point? from client to proxy for proxy to the storage nodes. proxy is stateless so how do we deem its time to backoff or are we thinking another middleware? | 21:19 |
| clayg | i assume it could all be operator configuable... just threw it on the agenda to get some feedback | 21:19 |
| clayg | mattoliverau: yeah it might not be a backoff where it gets longer and longer - but it could sleep a random amount | 21:20 |
| clayg | mattoliverau: i'm not sure would might be useful or expected | 21:20 |
| clayg | most of the time we already hit a timeout which is kinda long, so maybe 0-500ms extra sleep isn't a big deal... | 21:20 |
| clayg | but I'm also not sure how useful it would be | 21:21 |
| alecuyer | too long would probably hold too many ports open ? (slowloris like issue?) | 21:21 |
| *** jmasud has joined #openstack-meeting | 21:21 | |
| *** rcernin has quit IRC | 21:21 | |
| clayg | alecuyer: yeah if the client can reliably generate a 5xx it's a vehicle for attack, maybe could be mitigated with some limit or bounds | 21:22 |
| mattoliverau | like adding a back off window. potentailly adding it to the rate limitting middleware. But the proxy being stateless might make tracking the data harder. | 21:22 |
| *** ralonsoh has quit IRC | 21:22 | |
| zaitcev | Don't we already have a rate limiting middleware? | 21:22 |
| zaitcev | Right, that one. | 21:23 |
| clayg | mattoliverau: I was thinking more the simple case or just adding some jitter, but I could see... | 21:23 |
| *** Lucas_Gray has quit IRC | 21:23 | |
| clayg | I think if we started doing exponential backoff (via ratelimiting middleware) that could get really akward for well behaved clients | 21:23 |
| *** bnemec has quit IRC | 21:23 | |
| clayg | I think rate limit currently only effects PUT/DELETE (it was specifically for container update limiting) | 21:24 |
| mattoliverau | yeah, true. jitter is a good start.. starting simple is always a good starting point. | 21:24 |
| clayg | but it'd probably be a good place to look | 21:24 |
| clayg | ok, thanks for the feedback | 21:24 |
| clayg | #topic open discussion | 21:24 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:24 | |
| clayg | https://wiki.openstack.org/wiki/Swift/PriorityReviews | 21:24 |
| clayg | looks like python-swiftclient went out maybe? https://github.com/openstack/python-swiftclient/releases | 21:25 |
| clayg | I think we did a pretty good job with Tim gone! | 21:27 |
| zaitcev | I feel like I promised some improvements to dark data and I forgot about it. | 21:27 |
| clayg | Fully expect everything to be back to normal next week | 21:27 |
| zaitcev | ok | 21:27 |
| clayg | zaitcev: hah | 21:27 |
| zaitcev | I only remember that I tried to get Romain's buy-in. | 21:28 |
| *** priteau has joined #openstack-meeting | 21:28 | |
| *** Lucas_Gray has joined #openstack-meeting | 21:28 | |
| rledisez | yeah, I remember that, need to go back to that review | 21:28 |
| *** kozhukalov has quit IRC | 21:29 | |
| clayg | I might have to keep working on https://bugs.launchpad.net/swift/+bug/1863841 | 21:30 |
| openstack | Launchpad bug 1863841 in OpenStack Object Storage (swift) "container listing with limit=N is returning additional containers" [High,Confirmed] | 21:30 |
| clayg | Tim and I are not really sure what we're gunna do about https://review.opendev.org/#/c/333331/ | 21:30 |
| clayg | I think both of those have functests now tho, so that's progress | 21:30 |
| clayg | the query param thing is for https://bugs.launchpad.net/swift/+bug/1553714 | 21:31 |
| openstack | Launchpad bug 1553714 in OpenStack Object Storage (swift) "SLO multipart-manifest=put ignored with temp-url" [Medium,In progress] | 21:31 |
| kota_ | interesting | 21:31 |
| zaitcev | That query parameters thing looks awfully familiar in theme to something I liked before. | 21:31 |
| *** slaweq has quit IRC | 21:32 | |
| clayg | ok, well that's it from me - can we call it early? | 21:33 |
| mattoliverau | we can :) then kota_ seongsoocho and I can get breakfast ;) | 21:34 |
| clayg | thanks for making swift awesome! | 21:34 |
| clayg | #endmeeting | 21:34 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:34 | |
| openstack | Meeting ended Wed Feb 26 21:34:21 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:34 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-02-26-21.00.html | 21:34 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-02-26-21.00.txt | 21:34 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-02-26-21.00.log.html | 21:34 |
| mattoliverau | nice work clayg, great job! | 21:34 |
| alecuyer | thanks clay | 21:34 |
| *** zaitcev has left #openstack-meeting | 21:34 | |
| kota_ | thx clayg | 21:34 |
| *** jamesmcarthur has quit IRC | 21:36 | |
| *** ociuhandu has quit IRC | 21:36 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:36 | |
| *** ociuhandu has joined #openstack-meeting | 21:37 | |
| *** ociuhandu has quit IRC | 21:41 | |
| *** jamesmcarthur has quit IRC | 21:44 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:45 | |
| *** ociuhandu has joined #openstack-meeting | 21:50 | |
| *** enriquetaso has quit IRC | 21:57 | |
| *** jamesmcarthur has quit IRC | 21:59 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:00 | |
| *** jamesmcarthur has quit IRC | 22:01 | |
| *** ociuhandu has quit IRC | 22:02 | |
| *** ociuhandu has joined #openstack-meeting | 22:02 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:05 | |
| *** ociuhandu has quit IRC | 22:07 | |
| *** priteau has quit IRC | 22:11 | |
| *** rfolco has quit IRC | 22:12 | |
| *** jamesmcarthur has quit IRC | 22:29 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:30 | |
| *** eharney has joined #openstack-meeting | 22:31 | |
| *** hyunsikyang__ has quit IRC | 22:31 | |
| *** JangwonLee has quit IRC | 22:31 | |
| *** jamesmcarthur has quit IRC | 22:35 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:35 | |
| *** ijw has joined #openstack-meeting | 22:37 | |
| *** TrevorV has quit IRC | 22:38 | |
| *** enriquetaso has joined #openstack-meeting | 22:39 | |
| *** ociuhandu has joined #openstack-meeting | 22:40 | |
| *** ykatabam has joined #openstack-meeting | 22:40 | |
| *** enriquetaso has quit IRC | 22:41 | |
| *** ociuhandu has quit IRC | 22:45 | |
| *** rcernin has joined #openstack-meeting | 22:49 | |
| *** jamesmcarthur has quit IRC | 22:49 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:54 | |
| *** brinzhang__ has quit IRC | 23:05 | |
| *** JangwonLee has joined #openstack-meeting | 23:11 | |
| *** hyunsikyang has joined #openstack-meeting | 23:11 | |
| *** jamesmcarthur has quit IRC | 23:15 | |
| *** bnemec has joined #openstack-meeting | 23:18 | |
| *** ysandeep|away is now known as ysandeep | 23:42 | |
| *** hyunsikyang has quit IRC | 23:45 | |
| *** JangwonLee has quit IRC | 23:45 | |
| *** JangwonLee has joined #openstack-meeting | 23:47 | |
| *** hyunsikyang has joined #openstack-meeting | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!