| *** epei_ has quit IRC | 00:00 | |
| *** mattw4 has quit IRC | 00:02 | |
| *** epei has joined #openstack-meeting | 00:03 | |
| *** ijw has joined #openstack-meeting | 00:07 | |
| *** ijw has quit IRC | 00:07 | |
| *** epei has quit IRC | 00:08 | |
| *** ijw has joined #openstack-meeting | 00:08 | |
| *** ijw has quit IRC | 00:08 | |
| *** ijw has joined #openstack-meeting | 00:10 | |
| *** bnemec has quit IRC | 00:14 | |
| *** jawad_axd has joined #openstack-meeting | 00:14 | |
| *** jawad_axd has quit IRC | 00:18 | |
| *** armax has joined #openstack-meeting | 00:27 | |
| *** yamamoto has joined #openstack-meeting | 00:32 | |
| *** jawad_axd has joined #openstack-meeting | 00:34 | |
| *** ykatabam has quit IRC | 00:35 | |
| *** ykatabam has joined #openstack-meeting | 00:38 | |
| *** jawad_axd has quit IRC | 00:39 | |
| *** rodrigods has quit IRC | 00:39 | |
| *** brinzhang has joined #openstack-meeting | 00:44 | |
| *** gyee has quit IRC | 00:47 | |
| *** jawad_axd has joined #openstack-meeting | 00:55 | |
| *** jawad_axd has quit IRC | 01:00 | |
| *** artom has quit IRC | 01:08 | |
| *** Liang__ has joined #openstack-meeting | 01:09 | |
| *** jawad_axd has joined #openstack-meeting | 01:16 | |
| *** jawad_axd has quit IRC | 01:21 | |
| *** yamamoto has quit IRC | 01:34 | |
| *** jawad_axd has joined #openstack-meeting | 01:37 | |
| *** ijw has quit IRC | 01:40 | |
| *** ijw has joined #openstack-meeting | 01:40 | |
| *** jawad_axd has quit IRC | 01:41 | |
| *** ijw has quit IRC | 01:45 | |
| *** diablo_rojo has quit IRC | 01:55 | |
| *** jawad_axd has joined #openstack-meeting | 01:57 | |
| *** jawad_axd has quit IRC | 02:01 | |
| *** dklyle has quit IRC | 02:03 | |
| *** david-lyle has joined #openstack-meeting | 02:03 | |
| *** yamamoto has joined #openstack-meeting | 02:04 | |
| *** ykatabam has quit IRC | 02:06 | |
| *** mriedem has quit IRC | 02:09 | |
| *** armax has quit IRC | 02:12 | |
| *** jawad_axd has joined #openstack-meeting | 02:18 | |
| *** ykatabam has joined #openstack-meeting | 02:18 | |
| *** jawad_axd has quit IRC | 02:22 | |
| *** jawad_axd has joined #openstack-meeting | 02:39 | |
| *** jawad_axd has quit IRC | 02:43 | |
| *** yamamoto has quit IRC | 02:44 | |
| *** apetrich has quit IRC | 03:09 | |
| *** yamamoto has joined #openstack-meeting | 03:11 | |
| *** gouthamr has quit IRC | 03:14 | |
| *** brinzhang_ has joined #openstack-meeting | 03:18 | |
| *** brinzhang_ has quit IRC | 03:20 | |
| *** brinzhang has quit IRC | 03:21 | |
| *** brinzhang_ has joined #openstack-meeting | 03:22 | |
| *** yaawang has joined #openstack-meeting | 03:23 | |
| *** gouthamr has joined #openstack-meeting | 03:24 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:29 | |
| *** ykatabam has quit IRC | 03:29 | |
| *** psachin has joined #openstack-meeting | 03:39 | |
| *** jamesmcarthur has quit IRC | 03:44 | |
| *** brinzhang has joined #openstack-meeting | 03:49 | |
| *** brinzhang_ has quit IRC | 03:51 | |
| *** yamamoto has quit IRC | 03:58 | |
| *** brinzhang_ has joined #openstack-meeting | 04:24 | |
| *** brinzhang has quit IRC | 04:27 | |
| *** masahito has joined #openstack-meeting | 04:33 | |
| *** osmanlicilegi has joined #openstack-meeting | 04:34 | |
| *** masahito has quit IRC | 04:38 | |
| *** yamamoto has joined #openstack-meeting | 04:39 | |
| *** epoojad1 has joined #openstack-meeting | 04:49 | |
| *** yamamoto has quit IRC | 04:49 | |
| *** Lucas_Gray has joined #openstack-meeting | 04:50 | |
| *** brinzhang_ has quit IRC | 04:54 | |
| *** Lucas_Gray has quit IRC | 04:58 | |
| *** Lucas_Gray has joined #openstack-meeting | 05:00 | |
| *** ykatabam has joined #openstack-meeting | 05:11 | |
| *** pcaruana has joined #openstack-meeting | 05:24 | |
| *** Lucas_Gray has quit IRC | 05:27 | |
| *** yonglihe has joined #openstack-meeting | 05:29 | |
| *** ricolin has joined #openstack-meeting | 05:29 | |
| *** links has joined #openstack-meeting | 05:34 | |
| *** ayva has joined #openstack-meeting | 05:38 | |
| *** zaneb has quit IRC | 05:53 | |
| *** hyunsikyang has quit IRC | 05:58 | |
| *** ociuhandu has joined #openstack-meeting | 06:07 | |
| *** ociuhandu has quit IRC | 06:11 | |
| *** masahito_ has joined #openstack-meeting | 06:13 | |
| *** masahito_ has quit IRC | 06:17 | |
| *** masahito has joined #openstack-meeting | 06:18 | |
| *** masahito has quit IRC | 06:22 | |
| *** masahito has joined #openstack-meeting | 06:25 | |
| *** epei has joined #openstack-meeting | 06:28 | |
| *** masahito has quit IRC | 06:30 | |
| *** epei has quit IRC | 06:33 | |
| *** masahito has joined #openstack-meeting | 06:44 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:46 | |
| *** janki has joined #openstack-meeting | 06:48 | |
| *** jamesmcarthur has quit IRC | 06:51 | |
| *** ykatabam has quit IRC | 06:55 | |
| *** rcernin has quit IRC | 06:56 | |
| *** epei has joined #openstack-meeting | 07:05 | |
| *** epei has quit IRC | 07:09 | |
| *** lpetrut has joined #openstack-meeting | 07:26 | |
| *** hyunsikyang has joined #openstack-meeting | 07:41 | |
| *** masahito has quit IRC | 07:45 | |
| *** kozhukalov has joined #openstack-meeting | 07:52 | |
| *** kopecmartin|off is now known as kopecmartin | 07:53 | |
| *** slaweq has joined #openstack-meeting | 08:03 | |
| *** vesper11 has quit IRC | 08:09 | |
| *** jawad_axd has joined #openstack-meeting | 08:09 | |
| *** ricolin has quit IRC | 08:11 | |
| *** vesper11 has joined #openstack-meeting | 08:14 | |
| *** masahito has joined #openstack-meeting | 08:16 | |
| *** ykatabam has joined #openstack-meeting | 08:17 | |
| *** ricolin has joined #openstack-meeting | 08:18 | |
| *** epei has joined #openstack-meeting | 08:22 | |
| *** yaawang has quit IRC | 08:25 | |
| *** epei has quit IRC | 08:27 | |
| *** slaweq has quit IRC | 08:36 | |
| *** ralonsoh has joined #openstack-meeting | 08:40 | |
| *** yamamoto has joined #openstack-meeting | 08:48 | |
| *** slaweq has joined #openstack-meeting | 08:50 | |
| *** oneswig has joined #openstack-meeting | 08:51 | |
| *** yamamoto has quit IRC | 08:52 | |
| *** mdelavergne has joined #openstack-meeting | 08:57 | |
| *** kozhukalov has quit IRC | 08:58 | |
| *** rpittau|afk is now known as rpittau | 08:59 | |
| ttx | #startmeeting large_scale_sig | 09:00 |
|---|---|---|
| openstack | Meeting started Wed Dec 18 09:00:01 2019 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:00 |
| *** openstack changes topic to " (Meeting topic: large_scale_sig)" | 09:00 | |
| openstack | The meeting name has been set to 'large_scale_sig' | 09:00 |
| ttx | #topic Rollcall | 09:00 |
| *** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)" | 09:00 | |
| oneswig | hi | 09:00 |
| ttx | who is around? | 09:00 |
| amorin | hey all | 09:00 |
| ttx | oneswig, amorin o/ | 09:00 |
| oneswig | greetings \o | 09:00 |
| masahito | o/ | 09:01 |
| ttx | While we wait for more, here is our agenda: | 09:01 |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-meeting | 09:01 |
| ttx | Feel free to add other topics ! | 09:01 |
| ttx | jiaopengju, Dinesh_Bhor: around? | 09:02 |
| mdelavergne | hi | 09:02 |
| ttx | etp: too | 09:02 |
| etp | o/ | 09:02 |
| ttx | ok let's get started | 09:03 |
| ttx | #topic Last meeting actions | 09:03 |
| *** openstack changes topic to "Last meeting actions (Meeting topic: large_scale_sig)" | 09:03 | |
| ttx | We have minutes for our last meeting at: | 09:03 |
| ttx | #link http://eavesdrop.openstack.org/meetings/large_scale_sig/2019/large_scale_sig.2019-11-27-09.00.html | 09:03 |
| ttx | We had the following actions: | 09:03 |
| ttx | - ttx to propose large scale SIG creation changes to openstack-sigs repository | 09:03 |
| ttx | That's done, our SIG is now official at https://governance.openstack.org/sigs/ ! | 09:03 |
| ttx | We also have a basic information page on the wiki at https://wiki.openstack.org/wiki/Large_Scale_SIG | 09:04 |
| ttx | - ttx to send meeting summary and create two etherpads to further refine the two initial goals | 09:04 |
| amorin | yay! | 09:04 |
| masahito | Thanks for the patch and organization :-) | 09:04 |
| ttx | I did that too, and we'll dive in the etherpads for the rest of the meeting | 09:04 |
| ttx | #topic Objective: Scaling within one cluster, and instrumentation of the bottlenecks | 09:05 |
| *** openstack changes topic to "Objective: Scaling within one cluster, and instrumentation of the bottlenecks (Meeting topic: large_scale_sig)" | 09:05 | |
| ttx | with the following volunteers: masahito, oneswig, YusukeTatsumi, jiaopengju | 09:05 |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling | 09:05 |
| ttx | To kick off the discussion I suggested two subgoals on that one | 09:05 |
| ttx | - Document "average" single-cluster scaling limits today, and what usually breaks first | 09:05 |
| ttx | - Measurement of MQ behavior through oslo.metrics | 09:06 |
| ttx | Last I looked, nobody jumped on the first one. Does it sound like a valuable goal ? Or should we focus on the second for now? | 09:06 |
| ttx | My thought was that describing what currently happens when we scale a cluster up, and what tends to fail first and at which limit, gives us a good base. | 09:06 |
| ttx | (I personally have no idea because I don;t have that experience, only second-hand accounts) | 09:07 |
| oneswig | I think it is a valuable goal. The difficulty is finding someone who does this regularly enough. | 09:07 |
| amorin | what we can without so much effort is collecting the experiences from everybody in the group | 09:07 |
| ttx | And with your collective experience, you're well positioned to describe that. | 09:07 |
| ttx | yes... Does not have to be a formal documentation, we could set it up as a mailing-list thread where each would contribute their experience... | 09:07 |
| ttx | Or an etherpad with a call for participation... | 09:07 |
| amorin | yes, and maybe add a link to this etherpad or the mail on a wiki page | 09:08 |
| amorin | so we can find it easily | 09:08 |
| masahito | Is the first goal like an userstory? | 09:08 |
| ttx | masahito: yes... the goal is to describe what currently breaks first when we scale up. I bet each case is a bit different... but | 09:09 |
| ttx | I also bet there are general issues | 09:09 |
| masahito | If so, I could quickly write the problem we hit before. I thought it targeted some technical document, like actual configuration for scaling. | 09:09 |
| ttx | Like we all know rabbit is (one of the) first to break | 09:09 |
| ttx | No, I was thinking more like "tell us your story when you scaled up, what broke" | 09:10 |
| ttx | (and at which numbers, if possible) | 09:10 |
| ttx | do you prefer etherpad or an email thread as a way to cllect that information? | 09:11 |
| amorin | +1 for etherpad | 09:11 |
| *** altk has joined #openstack-meeting | 09:11 | |
| ttx | In both cases we should be ready to seed the discussion, to encourage others to talk | 09:11 |
| amorin | we can do both actually | 09:11 |
| ttx | and by we I mean you, since I don;t have experience to contribute in that area | 09:11 |
| ttx | amorin: not a bad idea | 09:11 |
| amorin | an etherpad and a mail to let everybody aware | 09:11 |
| ttx | post email, and collect the results in an etherpad | 09:12 |
| *** jiaopengju1 has joined #openstack-meeting | 09:12 | |
| masahito | +1 | 09:12 |
| amorin | +1 | 09:12 |
| etp | +1 | 09:12 |
| ttx | OK, so how about... you all prepare a short description of what usually happens to you when you scale up a single cluster up to a point. We wait until after new year. I create an etherpad and send a call for stories early January | 09:13 |
| ttx | and then you reply to that thread with your prepared stories, hopefully encouraging others to chime in | 09:13 |
| ttx | we collect all the stories in a single etherpad | 09:13 |
| ttx | then reduce that etherpad to common trends | 09:14 |
| amorin | good plan | 09:14 |
| etp | works for me | 09:14 |
| jiaopengju1 | +1, sorry for my late | 09:14 |
| masahito | I'm okay | 09:14 |
| ttx | #action all prepare a short description of what happens (what breaks first) when scaling up a single cluster | 09:15 |
| ttx | #action ttx to prepare etherpad and send ML thread asking for user scaling stories, to be posted after the end-of-year holidays | 09:15 |
| ttx | ok, we have a plan | 09:15 |
| ttx | <--- happy | 09:15 |
| ttx | - Measurement of MQ behavior through oslo.metrics... | 09:16 |
| ttx | we have some early steps documented | 09:16 |
| ttx | Who is interested in writing that oslo.metric blueprint? | 09:16 |
| ttx | (or collaborate on one) | 09:16 |
| ttx | I think LINE is well advanced on that | 09:16 |
| masahito | yes, I wrote it on the etherpad | 09:17 |
| ttx | maybe you can post the first draft and we can help you review it? | 09:17 |
| masahito | Yes. | 09:17 |
| ttx | #action masahito to produce first draft for the oslo.metric blueprint | 09:18 |
| ttx | ok.. anything more to do immediately on that subgoal? | 09:18 |
| masahito | I don't have much time to work for writing the draft right now. I will be able to push the bp in beggining of next year. | 09:19 |
| ttx | sure, no hurry :) | 09:19 |
| ttx | Just documenting next steps | 09:19 |
| ttx | oneswig added another subgoal: Instrumentation for "golden signals" | 09:20 |
| ttx | I think that would be very valuable. But it feels more like a second-stage goal | 09:21 |
| oneswig | I did... I think the terminology in this book might be useful to adopt. | 09:21 |
| oneswig | ttx: it's more abstract than specific cases such as rabbit | 09:21 |
| ttx | Once we identify what the current state is | 09:21 |
| ttx | I think golden signals will emerge | 09:21 |
| ttx | unless you think we can start documenting that already? | 09:22 |
| amorin | what do you mean by golden signals? | 09:22 |
| oneswig | Not sure. I can start in early 2020. | 09:22 |
| ttx | or maybe as a first step, learn about that concept | 09:22 |
| oneswig | golden signals are tell-tales, strong indicators for symptoms of trouble. | 09:23 |
| etp | do you mean like nova api response time increasing suddenly/steadily? | 09:23 |
| ttx | I know! it's when my phone vibrates | 09:23 |
| oneswig | etp: yes, exactly that kind of thing - that would be a latency measurement | 09:24 |
| amorin | ok thanks | 09:24 |
| etp | those do tend to reveal interesting things :) | 09:24 |
| oneswig | apologies I must leave in a few minutes - diary conflict | 09:25 |
| ttx | ok moving on quickly | 09:25 |
| ttx | I propose as action that we read a bit more about golden signals between now and next meeting | 09:26 |
| ttx | #action all learn more about golden signals concept as described in https://landing.google.com/sre/book.html | 09:26 |
| ttx | OK, let's move to the other goal | 09:26 |
| ttx | #topic Objective: Document large scale configuration and tips &tricks | 09:27 |
| *** openstack changes topic to "Objective: Document large scale configuration and tips &tricks (Meeting topic: large_scale_sig)" | 09:27 | |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-documentation | 09:27 |
| ttx | Again I did suggest two subgoals based on our discussion last week | 09:27 |
| ttx | err... last meeting | 09:27 |
| ttx | The first one is probably a group exercise... If you find doc or blogposts that can help, add them here. There may not be that many. | 09:27 |
| ttx | Searching planet.openstack.org may help. | 09:27 |
| *** apetrich has joined #openstack-meeting | 09:27 | |
| ttx | #action all add links to relevant articles around large scale openstack (if you find any) to the etherpad | 09:28 |
| oneswig | With a Scientific SIG hat on, it might be a useful group for gathering data from. I can follow up on that. | 09:28 |
| oneswig | have to go, sorry but do set an action iif you agree. | 09:28 |
| *** oneswig has quit IRC | 09:28 | |
| ttx | yes, scientific communities are generally more used to sharing | 09:28 |
| ttx | #action oneswig to follow up with Scientific community to find such articles | 09:29 |
| ttx | The second subgoal is around documenting configuration values | 09:30 |
| ttx | amorin suggested several tactics there | 09:30 |
| ttx | Patching existing docs, producing a new doc, or a mixture of both... | 09:30 |
| amorin | yes | 09:30 |
| ttx | What do others think? | 09:30 |
| ttx | My gut feeling is that this should appear in regular doc, if it's good and current enough to support that. If not good/current, a new doc might be preferable | 09:30 |
| amorin | I think the best is my last proposition | 09:30 |
| amorin | a mix between patching current doc and eventually have an external pages with details | 09:31 |
| ttx | yeah that would probably work best | 09:31 |
| ttx | allows to discover settings, without making the original doc too complex | 09:31 |
| amorin | yup | 09:32 |
| amorin | I have the feeling that it would help | 09:32 |
| amorin | at least, help me :p | 09:32 |
| etp | +1, external page would be useful linked to golden metrics | 09:32 |
| ttx | OK, so what would be the next step for this? | 09:33 |
| amorin | I dont know how we should proceed | 09:34 |
| amorin | but maybe we should ask the PTL of each project | 09:34 |
| ttx | #agree we should patch existing doc to point to a separate new doc | 09:34 |
| amorin | what they thing about our plan? | 09:34 |
| amorin | maybe they will have ideas | 09:34 |
| ttx | 'OK, so the simplest would be to post a ML thread with [largescale-sig] and [ptl] | 09:35 |
| ttx | If that does not work, I can help with engaging more directly | 09:35 |
| amorin | ok | 09:36 |
| ttx | but having the thread as a base we can link to will be useful in all cases | 09:36 |
| amorin | I can start this thread if you want | 09:36 |
| ttx | that would be perfect. Again I think we should avoid the end-of-year, because that's when threads go ignored | 09:36 |
| *** Liang__ has quit IRC | 09:36 | |
| amorin | ok, I can start it in january | 09:37 |
| ttx | A lot of people just declare ML bankrupcy after the holidays :) | 09:37 |
| ttx | #action amorin to start a thread on documenting configuration defaults for large scale, introduce the "mixture of both" tactic | 09:37 |
| ttx | Alright, we have next steps here too... anything else on that topic? | 09:38 |
| ttx | Any other topic you'd like to discuss? | 09:39 |
| ttx | #topic Any other business | 09:40 |
| *** openstack changes topic to "Any other business (Meeting topic: large_scale_sig)" | 09:40 | |
| ttx | I noticed we have a new group member, mdelavergne | 09:40 |
| ttx | maybe you can introduce yourself? | 09:40 |
| mdelavergne | Yep :) | 09:40 |
| ttx | What would you like to achieve in this group, and do you find our early goals in line with that | 09:41 |
| mdelavergne | Hi everyone, I'm am a PhD student at Inria, I work with the discovery initiative which tried to evaluate Openstack at the edge (so large scale and wanwide), and I'm trying to help with pointers to what we did, and trying to learn from your experiences mostly ! | 09:42 |
| ttx | perfect! welcome | 09:42 |
| mdelavergne | thanks :) | 09:42 |
| amorin | welcome! | 09:42 |
| etp | welcome! | 09:43 |
| masahito | welcome :) | 09:43 |
| ttx | Time to discuss next meeting date... | 09:43 |
| ttx | I think it's safe to skip the next two weeks. Do you prefer January 8 or January 15 for next meeting? | 09:43 |
| masahito | I'm not available January 8. | 09:43 |
| ttx | I feel like we won't have made much progress by January 8... Maybe the 15th would be better? Just before Chinese new year | 09:44 |
| ttx | As long as we start those threads before we should be good | 09:44 |
| jiaopengju1 | 15th is ok for me | 09:44 |
| masahito | 15th works for me. | 09:44 |
| etp | +1 for 15th | 09:44 |
| mdelavergne | 15th is fine by me | 09:44 |
| amorin | 15th is ok for me | 09:44 |
| ttx | Alright looks like a winner. Hopefully belmiro will be able to join us on that date | 09:45 |
| ttx | #agreed next meeting: January 15, 9utc #openstack-meeting | 09:45 |
| ttx | I'll send the meeting summary and the list of actions. | 09:45 |
| ttx | We have a bunch of threads to start, and the week of January 6 is probably the best for that | 09:46 |
| *** rsimai_away is now known as rsimai | 09:46 | |
| ttx | Alright... anything else before we close? | 09:46 |
| ttx | Thanks again to everyone for your participation in the group, and have a great end of year | 09:47 |
| ttx | #endmeeting | 09:47 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 09:47 | |
| openstack | Meeting ended Wed Dec 18 09:47:38 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:47 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/large_scale_sig/2019/large_scale_sig.2019-12-18-09.00.html | 09:47 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2019/large_scale_sig.2019-12-18-09.00.txt | 09:47 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/large_scale_sig/2019/large_scale_sig.2019-12-18-09.00.log.html | 09:47 |
| masahito | Thanks. Have a good end of year! | 09:47 |
| mdelavergne | Thanks and have a nice holidays! | 09:48 |
| amorin | thanks all | 09:48 |
| etp | Thanks all | 09:49 |
| *** apetrich has quit IRC | 09:58 | |
| *** ociuhandu has joined #openstack-meeting | 09:59 | |
| *** masahito has quit IRC | 10:06 | |
| *** e0ne has joined #openstack-meeting | 10:11 | |
| *** ociuhandu has quit IRC | 10:34 | |
| *** janki has quit IRC | 10:35 | |
| *** ociuhandu has joined #openstack-meeting | 10:36 | |
| *** ociuhandu has quit IRC | 10:40 | |
| *** ociuhandu has joined #openstack-meeting | 10:42 | |
| *** ociuhandu has quit IRC | 10:48 | |
| *** tbarron_ has joined #openstack-meeting | 10:59 | |
| *** ociuhandu has joined #openstack-meeting | 11:16 | |
| *** ociuhandu has quit IRC | 11:22 | |
| *** ociuhandu has joined #openstack-meeting | 11:23 | |
| *** ociuhandu has quit IRC | 11:24 | |
| *** ociuhandu has joined #openstack-meeting | 11:25 | |
| *** ociuhandu has quit IRC | 11:29 | |
| *** yaawang has joined #openstack-meeting | 11:31 | |
| *** Lucas_Gray has joined #openstack-meeting | 11:34 | |
| *** Lucas_Gray has quit IRC | 11:38 | |
| *** ociuhandu has joined #openstack-meeting | 11:40 | |
| *** apetrich has joined #openstack-meeting | 11:42 | |
| *** Lucas_Gray has joined #openstack-meeting | 11:42 | |
| *** ralonsoh has quit IRC | 11:48 | |
| *** ociuhandu has quit IRC | 11:50 | |
| *** ralonsoh has joined #openstack-meeting | 11:51 | |
| *** ociuhandu has joined #openstack-meeting | 11:51 | |
| *** masahito has joined #openstack-meeting | 11:52 | |
| *** brinzhang has joined #openstack-meeting | 11:54 | |
| *** brinzhang_ has joined #openstack-meeting | 11:56 | |
| *** jiaopengju2 has joined #openstack-meeting | 11:57 | |
| *** epoojad1 is now known as epoojad1|afk | 11:59 | |
| *** brinzhang has quit IRC | 11:59 | |
| *** jiaopengju1 has quit IRC | 12:00 | |
| *** altk has quit IRC | 12:00 | |
| *** jiaopengju1 has joined #openstack-meeting | 12:01 | |
| *** ociuhandu has quit IRC | 12:03 | |
| *** epoojad1|afk has quit IRC | 12:04 | |
| *** raildo has joined #openstack-meeting | 12:05 | |
| *** masahito has quit IRC | 12:20 | |
| *** ociuhandu has joined #openstack-meeting | 12:25 | |
| *** brinzhang has joined #openstack-meeting | 12:25 | |
| *** brinzhang_ has quit IRC | 12:28 | |
| *** brinzhang_ has joined #openstack-meeting | 12:28 | |
| *** ociuhandu has quit IRC | 12:28 | |
| *** ociuhandu has joined #openstack-meeting | 12:29 | |
| *** ociuhandu has quit IRC | 12:30 | |
| *** brinzhang_ has quit IRC | 12:30 | |
| *** ociuhandu has joined #openstack-meeting | 12:30 | |
| *** brinzhang_ has joined #openstack-meeting | 12:30 | |
| *** brinzhang has quit IRC | 12:31 | |
| *** brinzhang_ has quit IRC | 12:32 | |
| *** brinzhang_ has joined #openstack-meeting | 12:33 | |
| *** brinzhang_ has quit IRC | 12:34 | |
| *** ociuhandu has quit IRC | 12:34 | |
| *** ociuhandu has joined #openstack-meeting | 12:34 | |
| *** brinzhang_ has joined #openstack-meeting | 12:35 | |
| *** brinzhang_ has quit IRC | 12:36 | |
| *** Lucas_Gray has quit IRC | 12:36 | |
| *** brinzhang_ has joined #openstack-meeting | 12:36 | |
| *** brinzhang has joined #openstack-meeting | 12:38 | |
| *** brinzhang_ has quit IRC | 12:41 | |
| *** bbowen has quit IRC | 12:44 | |
| *** brinzhang_ has joined #openstack-meeting | 12:45 | |
| *** brinzhang has quit IRC | 12:48 | |
| *** senrique_ has joined #openstack-meeting | 12:52 | |
| *** bbowen has joined #openstack-meeting | 13:00 | |
| *** jiaopengju2 has quit IRC | 13:04 | |
| *** jiaopengju2 has joined #openstack-meeting | 13:04 | |
| *** Shrews has left #openstack-meeting | 13:05 | |
| *** electrofelix has joined #openstack-meeting | 13:05 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:05 | |
| *** ociuhandu has quit IRC | 13:06 | |
| *** ociuhandu has joined #openstack-meeting | 13:07 | |
| *** ociuhandu has quit IRC | 13:11 | |
| *** jamesmcarthur has quit IRC | 13:12 | |
| *** ociuhandu has joined #openstack-meeting | 13:12 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:13 | |
| *** ociuhandu has quit IRC | 13:18 | |
| *** brinzhang has joined #openstack-meeting | 13:19 | |
| *** ykatabam has quit IRC | 13:19 | |
| *** brinzhang_ has quit IRC | 13:22 | |
| *** jamesmcarthur has quit IRC | 13:29 | |
| *** brinzhang_ has joined #openstack-meeting | 13:30 | |
| *** ociuhandu has joined #openstack-meeting | 13:32 | |
| *** brinzhang has quit IRC | 13:33 | |
| *** brinzhang_ has quit IRC | 13:33 | |
| *** brinzhang_ has joined #openstack-meeting | 13:34 | |
| *** brinzhang_ has quit IRC | 13:35 | |
| *** ociuhandu has quit IRC | 13:39 | |
| *** njohnston is now known as njohnston|xmas | 13:40 | |
| *** mriedem has joined #openstack-meeting | 13:43 | |
| *** ociuhandu has joined #openstack-meeting | 13:47 | |
| *** Liang__ has joined #openstack-meeting | 13:47 | |
| *** Liang__ is now known as LiangFang | 13:48 | |
| *** LiangFang has quit IRC | 13:49 | |
| *** Liang__ has joined #openstack-meeting | 13:50 | |
| *** jiaopengju2 has quit IRC | 13:50 | |
| *** senrique_ is now known as enriquetaso | 13:50 | |
| *** jiaopengju1 has quit IRC | 13:50 | |
| *** Liang__ is now known as LiangFang | 13:50 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:55 | |
| *** jamesmcarthur has quit IRC | 13:59 | |
| *** ociuhandu has quit IRC | 14:00 | |
| *** jamesmcarthur has joined #openstack-meeting | 14:03 | |
| *** dviroel has joined #openstack-meeting | 14:04 | |
| *** ociuhandu has joined #openstack-meeting | 14:04 | |
| *** sfernand has joined #openstack-meeting | 14:06 | |
| *** bnemec has joined #openstack-meeting | 14:10 | |
| *** priteau has joined #openstack-meeting | 14:17 | |
| *** ociuhandu has quit IRC | 14:26 | |
| *** ociuhandu has joined #openstack-meeting | 14:27 | |
| *** ociuhandu has quit IRC | 14:31 | |
| *** LiangFang has quit IRC | 14:33 | |
| *** Liang__ has joined #openstack-meeting | 14:34 | |
| *** Liang__ is now known as LiangFang | 14:34 | |
| *** ociuhandu has joined #openstack-meeting | 14:47 | |
| *** mriedem has quit IRC | 14:51 | |
| *** david-lyle is now known as dklyle | 14:51 | |
| *** mriedem has joined #openstack-meeting | 14:51 | |
| *** ociuhandu has quit IRC | 14:56 | |
| *** jawad_axd has quit IRC | 14:58 | |
| *** jawad_axd has joined #openstack-meeting | 15:02 | |
| *** ociuhandu has joined #openstack-meeting | 15:02 | |
| *** LiangFang has quit IRC | 15:03 | |
| *** jawad_ax_ has joined #openstack-meeting | 15:06 | |
| *** jawad_axd has quit IRC | 15:06 | |
| *** jawad_ax_ has quit IRC | 15:10 | |
| *** links has quit IRC | 15:10 | |
| *** anastzhyr has joined #openstack-meeting | 15:15 | |
| *** ociuhandu has quit IRC | 15:19 | |
| *** ociuhandu has joined #openstack-meeting | 15:20 | |
| *** ociuhandu has quit IRC | 15:21 | |
| *** ociuhandu has joined #openstack-meeting | 15:21 | |
| *** lpetrut has quit IRC | 15:23 | |
| *** yamamoto has joined #openstack-meeting | 15:24 | |
| *** eharney has quit IRC | 15:29 | |
| *** yamamoto has quit IRC | 15:29 | |
| *** panda has quit IRC | 15:31 | |
| *** panda has joined #openstack-meeting | 15:33 | |
| *** yamamoto has joined #openstack-meeting | 15:54 | |
| *** jawad_axd has joined #openstack-meeting | 15:56 | |
| *** rsimai is now known as rsimai_away | 15:56 | |
| *** jawad_axd has quit IRC | 16:01 | |
| *** woojay has joined #openstack-meeting | 16:03 | |
| *** gyee has joined #openstack-meeting | 16:03 | |
| *** rbudden has joined #openstack-meeting | 16:04 | |
| *** woojay has left #openstack-meeting | 16:08 | |
| *** armax has joined #openstack-meeting | 16:09 | |
| *** epei has joined #openstack-meeting | 16:11 | |
| *** jawad_axd has joined #openstack-meeting | 16:17 | |
| *** jawad_axd has quit IRC | 16:21 | |
| *** yamamoto has quit IRC | 16:22 | |
| *** eharney has joined #openstack-meeting | 16:24 | |
| *** mdelavergne has quit IRC | 16:25 | |
| *** ociuhandu has quit IRC | 16:30 | |
| *** eharney_ has joined #openstack-meeting | 16:33 | |
| *** eharney has quit IRC | 16:37 | |
| *** epei has quit IRC | 16:56 | |
| *** epei has joined #openstack-meeting | 16:57 | |
| *** yamamoto has joined #openstack-meeting | 16:58 | |
| *** kopecmartin is now known as kopecmartin|off | 16:58 | |
| *** epei_ has joined #openstack-meeting | 17:00 | |
| *** epei has quit IRC | 17:01 | |
| *** sfernand has quit IRC | 17:06 | |
| *** dtrainor has quit IRC | 17:08 | |
| *** dtrainor has joined #openstack-meeting | 17:08 | |
| *** yamamoto has quit IRC | 17:11 | |
| *** artom has joined #openstack-meeting | 17:17 | |
| *** jawad_axd has joined #openstack-meeting | 17:18 | |
| *** diablo_rojo has joined #openstack-meeting | 17:18 | |
| *** rpittau is now known as rpittau|afk | 17:21 | |
| *** jawad_axd has quit IRC | 17:23 | |
| *** dklyle has quit IRC | 17:26 | |
| *** david-lyle has joined #openstack-meeting | 17:26 | |
| *** david-lyle has quit IRC | 17:26 | |
| *** david-lyle has joined #openstack-meeting | 17:26 | |
| *** mattw4 has joined #openstack-meeting | 17:35 | |
| *** panda has quit IRC | 17:36 | |
| *** epei_ has quit IRC | 17:37 | |
| *** panda has joined #openstack-meeting | 17:39 | |
| *** psachin has quit IRC | 17:39 | |
| *** ociuhandu has joined #openstack-meeting | 17:51 | |
| *** dviroel has quit IRC | 17:53 | |
| *** ociuhandu has quit IRC | 17:55 | |
| *** rubasov has quit IRC | 17:57 | |
| *** enriquetaso has quit IRC | 17:59 | |
| *** rfolco has quit IRC | 18:07 | |
| *** electrofelix has quit IRC | 18:09 | |
| *** priteau has quit IRC | 18:19 | |
| *** ralonsoh has quit IRC | 18:25 | |
| *** e0ne has quit IRC | 18:39 | |
| *** eharney_ has quit IRC | 18:40 | |
| *** rubasov has joined #openstack-meeting | 18:42 | |
| *** enriquetaso has joined #openstack-meeting | 18:43 | |
| *** rfolco has joined #openstack-meeting | 18:52 | |
| *** jamesmcarthur has quit IRC | 19:03 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:26 | |
| *** dviroel has joined #openstack-meeting | 19:32 | |
| *** jamesmcarthur has quit IRC | 19:38 | |
| *** e0ne has joined #openstack-meeting | 19:58 | |
| *** david-lyle is now known as dklyle | 20:00 | |
| *** enriquetaso has quit IRC | 20:08 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:09 | |
| *** anastzhyr has quit IRC | 20:15 | |
| *** rf0lc0 has joined #openstack-meeting | 20:15 | |
| *** rfolco has quit IRC | 20:17 | |
| *** e0ne has quit IRC | 20:25 | |
| *** slaweq has quit IRC | 20:36 | |
| *** rf0lc0 has quit IRC | 20:41 | |
| *** slaweq has joined #openstack-meeting | 20:44 | |
| timburke | #startmeeting swift | 21:01 |
| openstack | Meeting started Wed Dec 18 21:01:32 2019 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:01 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:01 | |
| openstack | The meeting name has been set to 'swift' | 21:01 |
| timburke | who's here for the swift meeting? | 21:01 |
| rledisez | o/ | 21:02 |
| timburke | quiet. maybe this'll be a fast one | 21:03 |
| timburke | as always, agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:04 |
| timburke | #topic next meeting | 21:04 |
| *** openstack changes topic to "next meeting (Meeting topic: swift)" | 21:04 | |
| tdasilva | o/ | 21:04 |
| timburke | just a reminder: no meeting next week (Dec 25) or the week after (Jan 1) | 21:04 |
| timburke | so next meeting will be Jan 8 | 21:04 |
| tdasilva | ack | 21:05 |
| timburke | that's all i've got, really -- on to updates! | 21:05 |
| timburke | #topic versioning | 21:06 |
| *** openstack changes topic to "versioning (Meeting topic: swift)" | 21:06 | |
| mattoliverau | o/ (sorry I'm late) | 21:06 |
| timburke | tdasilva, how's it going? | 21:06 |
| timburke | mattoliverau, no worries :-) | 21:06 |
| timburke | i can also talk about what *i've* found as a reviewer ;-) | 21:07 |
| timburke | i've been poking at getting a probe test for sharding a new-style versioned container | 21:07 |
| tdasilva | we are continuing to make good progress I think. timburke left comments on the swift versioning so we are working through some of those | 21:07 |
| tdasilva | sorry, i type slow | 21:07 |
| timburke | it's still early for you ;-) | 21:08 |
| timburke | (in particular, i'm looking to shard the backing, reserved-name container) | 21:08 |
| tdasilva | just wanted to add we added some todo items to the bottom of the ehterpad: https://etherpad.openstack.org/p/swift-object-versioning | 21:08 |
| tdasilva | kind of summarizes last few items we are currently aware of and working on | 21:09 |
| timburke | sounds good | 21:09 |
| timburke | any other input we need from people, or is it mostly a matter of cranking through the remaining issues? | 21:10 |
| tdasilva | so yeah, we are pushing it hard to get it over the finish line soon | 21:10 |
| clayg | I’m working on a version of the brain splitter that can use internal client so I can have reconciler probe tests for the null namespace. | 21:10 |
| timburke | clayg, *i* was thinking i need that, too! | 21:10 |
| clayg | Oh. Well today’s been crazy. Maybe tomorrow. | 21:11 |
| *** yamamoto has joined #openstack-meeting | 21:11 | |
| *** zaitcev has joined #openstack-meeting | 21:12 | |
| timburke | i'll see how far i can get; throw up whatever i've got at EOD, anyway | 21:12 |
| *** ayoung has joined #openstack-meeting | 21:12 | |
| timburke | am i right in thinking that we know what needs to happen, and there aren't really any decisions we need to make regarding it? | 21:13 |
| tdasilva | oh | 21:13 |
| tdasilva | i think we said last week we would discuss again about `enabled by default` ? | 21:13 |
| tdasilva | or am i confused? | 21:13 |
| timburke | sorry, i'd forgotten. i think i'm of the opinion that i *wouldn't* recommend it by default yet (for upstream, anyway) | 21:15 |
| tdasilva | actually, yeah, nevermind: "anyway, something to think about. i don't think we're likely to enable-by-default before the next time we all see each other face-to-face, anyway ;-)" | 21:15 |
| *** raildo has quit IRC | 21:16 | |
| timburke | maybe i'll feel differently once i've got probe tests for sharding/reconciler/expirer ;-) | 21:16 |
| tdasilva | especially given the probe tests ...right! | 21:16 |
| timburke | sounds like we've got what we need to keep moving | 21:16 |
| timburke | #topic profiling/md5 optimization | 21:17 |
| *** openstack changes topic to "profiling/md5 optimization (Meeting topic: swift)" | 21:17 | |
| timburke | rledisez, how's it going? | 21:17 |
| *** yamamoto has quit IRC | 21:17 | |
| rledisez | well, I didn't di much on that. I mostly think about how I would implement it. I don't want to redo the whole world, so i'll try to be smart in integrating that. but no real code for now | 21:18 |
| timburke | 👍 | 21:18 |
| rledisez | the different combination (crypto/replica, crypto/EC, EC, replica) make it a bit hard to do something really optimized (in the way it's designed). i'm thinking about that too | 21:19 |
| *** alecuyer has joined #openstack-meeting | 21:20 | |
| rledisez | that's all for me about that | 21:20 |
| timburke | i wonder how crazy it'd be to always use footers and have someone pretty far left (like, gatekeeper or something) be responsible for validating the client data and storing the etag... then have everything else punt to a crc or something lighter-weight... | 21:21 |
| rledisez | it's exactly what i'm building. I wrote it that way in the etherpad. | 21:22 |
| timburke | we need more hours in the day! at least, that's usually *my* problem ;-) | 21:22 |
| timburke | #topic RFC-compliant etags | 21:23 |
| *** openstack changes topic to "RFC-compliant etags (Meeting topic: swift)" | 21:23 | |
| timburke | so i've got a patch to enable it cluster-wide, but it sounds like rledisez might have a separate middleware that would allow finer-grained control... | 21:23 |
| timburke | i haven't seen that patch yet, though ;-) | 21:24 |
| rledisez | timburke: I totally forgot to push it. i'll put a reminder for tomorrow | 21:24 |
| timburke | thanks! it doesn't have to be perfect -- i'd be happy to help polish docs or tests or what-have-you :D | 21:25 |
| rledisez | it's "run-in-prod" polished, no more ;) | 21:25 |
| timburke | i just had one other thing i wanted to draw attention to, this time with a little more time for discussion :-) | 21:25 |
| timburke | #topic ranged SLO reads | 21:26 |
| *** openstack changes topic to "ranged SLO reads (Meeting topic: swift)" | 21:26 | |
| *** rf0lc0 has joined #openstack-meeting | 21:26 | |
| *** rf0lc0 has quit IRC | 21:26 | |
| timburke | so we had a customer complaining about latency variations | 21:26 |
| timburke | as i was helping look into it, i noticed that ranged SLO reads have this funny request pattern: | 21:27 |
| timburke | proxy sends the range request down to the object server | 21:27 |
| timburke | object server just has a manifest, so it responds 416 (that range is way off the end of the thing!) | 21:27 |
| zaitcev | I presume this is replicated, but go on. | 21:27 |
| timburke | (good point; yes, though the range manipulations we do for EC mean you get much the same behavior) | 21:28 |
| timburke | proxy says, well, no, actually i want that *whole manifest*! | 21:28 |
| timburke | object server sends it back | 21:28 |
| timburke | proxy finds the appropriate segment, requests *that* | 21:29 |
| timburke | object server sends it back, and we can actually start serving users | 21:29 |
| timburke | i set out to get rid of that 416-refetch behavior and just have the object-server send the whole manifest from the get-go | 21:30 |
| zaitcev | Wait, how does proxy know that it's an SLO object to begin with. | 21:30 |
| zaitcev | Of course it asks for a range first. | 21:30 |
| timburke | i think i designed my solution fairly smartly; it adds a new, not-slo-specific header to tell the object server "hey, if you see any of these pieces of metadata, ignore the range header i sent" | 21:31 |
| zaitcev | oh, nice | 21:31 |
| timburke | then has slo add X-Static-Large-Object to that list | 21:31 |
| zaitcev | or.... you could _always_ ignore range for manifests | 21:32 |
| timburke | i was thinking that i don't really want object-server to need to know *more* about large objects -- we've already seen how much trouble it is that they have to know about swift_bytes, for example | 21:32 |
| timburke | i was mostly inspired by the x-backend-etag-is-at behavior we already have | 21:33 |
| timburke | there is a wart, though, that i'm not sure i like: the EC obj controller (currently) pops off the Range header so that swob behaves itself when it makes the response | 21:34 |
| timburke | so i guess i'm wondering: has anyone else seen this behavior or had users complain about it? | 21:36 |
| timburke | does this design seem reasonable? | 21:36 |
| timburke | and are we ok with that wart? (even in this patch, it means that SLO has to grab the range itself before it calls down to the app, just so it can replace it later...) | 21:37 |
| zaitcev | I did not notice, but that's because I barely have any SLOs at all. It sounds completely plausible for the reasons of a-priori information that I just mentioned. | 21:37 |
| mattoliverau | I like the approach, re: generic header and middleware can add to it, smart. I guess I need to look to understand the wart some more | 21:38 |
| timburke | i tried to capture it in the commit message; you can also see the consequence of it in https://review.opendev.org/#/c/697739/3/swift/common/middleware/slo.py | 21:39 |
| mattoliverau | Cool, I'll check it out | 21:40 |
| timburke | thanks! i'll see about getting that save/restore logic out to the EC controller, too... i feel like that should be doable, and it'd remove my only real concern | 21:41 |
| timburke | that's all i've got | 21:41 |
| timburke | #topic open discussion | 21:41 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:41 | |
| *** dviroel has quit IRC | 21:41 | |
| timburke | what else would people like to bring up? | 21:41 |
| timburke | all right, looks like we can let mattoliverau, tdasilva, and seongsoocho grab some breakfast ;-) | 21:43 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:44 |
| timburke | talk to you all next year :-) | 21:44 |
| timburke | #endmeeting | 21:44 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:44 | |
| openstack | Meeting ended Wed Dec 18 21:44:19 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:44 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2019/swift.2019-12-18-21.01.html | 21:44 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2019/swift.2019-12-18-21.01.txt | 21:44 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2019/swift.2019-12-18-21.01.log.html | 21:44 |
| mattoliverau | Thanks timburke | 21:44 |
| *** zaitcev has left #openstack-meeting | 21:49 | |
| *** dtroyer has joined #openstack-meeting | 21:52 | |
| *** pcaruana has quit IRC | 21:54 | |
| *** alecuyer has left #openstack-meeting | 22:03 | |
| *** jamesmcarthur has quit IRC | 22:04 | |
| *** bbowen has quit IRC | 22:05 | |
| *** mattw4 has quit IRC | 22:07 | |
| *** mattw4 has joined #openstack-meeting | 22:10 | |
| *** haleyb has quit IRC | 22:22 | |
| *** haleyb has joined #openstack-meeting | 22:26 | |
| *** rbudden has quit IRC | 22:31 | |
| *** bbowen has joined #openstack-meeting | 22:34 | |
| *** mriedem has quit IRC | 22:51 | |
| *** mattw4 has quit IRC | 22:51 | |
| *** mattw4 has joined #openstack-meeting | 22:59 | |
| *** rcernin has joined #openstack-meeting | 23:10 | |
| *** rbudden has joined #openstack-meeting | 23:25 | |
| *** mattw4 has quit IRC | 23:40 | |
| *** mattw4 has joined #openstack-meeting | 23:40 | |
| *** mattw4 has quit IRC | 23:45 | |
| *** yamamoto has joined #openstack-meeting | 23:46 | |
| *** mattw4 has joined #openstack-meeting | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!