| *** mlavalle has quit IRC | 00:56 | |
| *** vesper11 has joined #openstack-meeting-3 | 01:09 | |
| *** hemanth_n has joined #openstack-meeting-3 | 02:22 | |
| *** psahoo has joined #openstack-meeting-3 | 06:15 | |
| *** psahoo has quit IRC | 07:26 | |
| *** psahoo has joined #openstack-meeting-3 | 07:31 | |
| *** lpetrut has joined #openstack-meeting-3 | 07:38 | |
| *** slaweq has joined #openstack-meeting-3 | 07:56 | |
| *** belmoreira has joined #openstack-meeting-3 | 08:03 | |
| *** tosky has joined #openstack-meeting-3 | 09:05 | |
| *** e0ne has joined #openstack-meeting-3 | 10:57 | |
| *** raildo has joined #openstack-meeting-3 | 11:50 | |
| *** e0ne has quit IRC | 12:40 | |
| *** haleyb has joined #openstack-meeting-3 | 12:46 | |
| *** vesper11 has quit IRC | 12:53 | |
| *** Luzi has joined #openstack-meeting-3 | 13:03 | |
| *** psahoo has quit IRC | 13:06 | |
| *** psahoo has joined #openstack-meeting-3 | 13:11 | |
| *** hemanth_n has quit IRC | 13:23 | |
| *** e0ne has joined #openstack-meeting-3 | 13:34 | |
| *** haleyb has quit IRC | 13:46 | |
| *** Luzi has quit IRC | 13:51 | |
| *** liuyulong has joined #openstack-meeting-3 | 13:57 | |
| *** haleyb has joined #openstack-meeting-3 | 14:13 | |
| *** raildo has quit IRC | 14:19 | |
| *** genekuo_ has joined #openstack-meeting-3 | 14:27 | |
| *** raildo has joined #openstack-meeting-3 | 14:27 | |
| *** mdelavergne has joined #openstack-meeting-3 | 14:53 | |
| ttx | o/ | 15:00 |
|---|---|---|
| ttx | #startmeeting large_scale_sig | 15:00 |
| openstack | Meeting started Wed Dec 2 15:00:32 2020 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: large_scale_sig)" | 15:00 | |
| openstack | The meeting name has been set to 'large_scale_sig' | 15:00 |
| ttx | #topic Rollcall | 15:00 |
| *** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)" | 15:00 | |
| mdelavergne | Hi o/ | 15:00 |
| ttx | Who is here for the Large Scale SIG meeting ? | 15:00 |
| genekuo_ | Hi o/ | 15:00 |
| ttx | amorin is probably busy. If not, he should be | 15:01 |
| ttx | belmoreira: maybe around? | 15:01 |
| belmoreira | o/ | 15:02 |
| belmoreira | thanks for the ping | 15:02 |
| ttx | hoping to see imtiaz soon | 15:02 |
| ttx | Alright, lets get started | 15:02 |
| ttx | Our agenda for today is at: | 15:02 |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-meeting | 15:02 |
| *** imtiazc has joined #openstack-meeting-3 | 15:03 | |
| ttx | checking for late-added agenda items | 15:03 |
| ttx | #topic Review previous meetings action items | 15:03 |
| *** openstack changes topic to "Review previous meetings action items (Meeting topic: large_scale_sig)" | 15:03 | |
| ttx | "ttx to refactor wiki and other etherpads into that "journey" FAQ view" | 15:03 |
| ttx | So... I did reorganize all pages under: | 15:03 |
| ttx | #link https://wiki.openstack.org/wiki/Large_Scale_SIG | 15:03 |
| ttx | If you have a look... you can see, I split everything into 4 subpages, one for each stage of the journey | 15:04 |
| ttx | For each subpage there is a FAQ, a few resources links | 15:04 |
| ttx | + a section on other SIG work pertaining to that stage | 15:04 |
| jpward | o/ | 15:04 |
| genekuo_ | Yeah I saw that, I'll be adding something to those pages before the next meeting | 15:04 |
| ttx | jpward: hi! welcome | 15:04 |
| liuyulong | Hi | 15:05 |
| ttx | yes, please feel free to add questions, answers or links relevant to the stage | 15:05 |
| ttx | liuyulong: hi! | 15:05 |
| genekuo_ | Hi, seeing new people | 15:05 |
| ttx | I'll finish reviewing previous point and we'll do an introduction round | 15:05 |
| ttx | At our next meeting we'll review each stage contents, so please spend some time reviewing those in the next two weeks! | 15:05 |
| ttx | #action all to review pages under https://wiki.openstack.org/wiki/Large_Scale_SIG in preparation for next meeting | 15:06 |
| ttx | Also let me know if you have trouble logging into the wiki. | 15:06 |
| ttx | The goal being, anyone in the SIG should feel free to update those pages, not just me | 15:06 |
| ttx | comments on that point? | 15:06 |
| genekuo_ | The structure looks pretty good to me | 15:07 |
| imtiazc | The 4 categories look good. Could we add "upgrade" story as well? | 15:07 |
| ttx | imtiazc: hi! I feel like like upgrade is an orthogonal concern at each stage | 15:07 |
| ttx | unless we add it as a 5th stage | 15:08 |
| ttx | like, once you have scaled out, how the hell do you upgrade? | 15:08 |
| imtiazc | exactly. It does become quite challenging. | 15:08 |
| genekuo_ | I probably can add that maybe next year, we're planning on doing that | 15:08 |
| genekuo_ | Should be a lot of pain | 15:09 |
| ttx | I like the idea of a 5th stage. does that make sense to you mdelavergne, belmoreira ? | 15:09 |
| ttx | I mean, if upgrading at high scale has unique constraints, it makes sense for us to document it | 15:10 |
| mdelavergne | mmh yep ! | 15:10 |
| mdelavergne | upgrade/maintain ? | 15:10 |
| ttx | OK, I'll create a skeleton page | 15:10 |
| ttx | "rinse, repeat" | 15:10 |
| ttx | #action ttx to add 5th stage around upgrade and maintain scaled out systems in operation | 15:11 |
| liuyulong | configuration tunning means you need to restart/reload/respawn the processes/servies/agents and so on, so looks like a upgrade already. : ) | 15:11 |
| ttx | ok, next action item from last meeting was... | 15:11 |
| ttx | "genekuo to review/approve https://review.opendev.org/#/c/755069/" | 15:11 |
| ttx | That's done | 15:11 |
| ttx | then... "ttx to set up release jobs and request a 0.1 release" | 15:11 |
| ttx | (for oslo.metrics) | 15:12 |
| imtiazc | By upgrade, I meant moving from one OpenStack release to another. | 15:12 |
| ttx | I did that: | 15:12 |
| ttx | #link https://review.opendev.org/c/openstack/project-config/+/763986 | 15:12 |
| ttx | #link https://review.opendev.org/c/openstack/releases/+/764631 | 15:12 |
| genekuo_ | I've checked the PR seems one of them is blocked by CI | 15:12 |
| ttx | release should be processed very soon now, maybe today | 15:12 |
| ttx | err, failed again. Will have a look :) | 15:12 |
| ttx | #action ttx to make sure oslo.metrics 0.1 is released | 15:13 |
| ttx | "ttx to reschedule meeting to be biweekly on Wednesdays, 15utc" | 15:13 |
| ttx | That's done: http://eavesdrop.openstack.org/#Large_Scale_SIG_Meeting | 15:13 |
| genekuo_ | I'll start working on oslo.messaging code once 0.1 is released | 15:13 |
| ttx | and finally... "all to think about how to improve how we collect feedbackPTG/Summit postmortem" | 15:13 |
| ttx | we'll discuss that in this meeting after the round of intros | 15:13 |
| ttx | #topic Introducing new SIG members | 15:14 |
| *** openstack changes topic to "Introducing new SIG members (Meeting topic: large_scale_sig)" | 15:14 | |
| ttx | I see two new faces, would be good to introduce ourselves and our interest in the Large Scale SIG | 15:14 |
| ttx | I'll start | 15:14 |
| ttx | I'm Thierry Carrez, VP Engineering at the now Open Infrastructure Foundation. I'm helping drive this group because I have an interest in getting large users to contribute their experience running openstack, and receive lots of questions from users that worry about the scaling journey and would very much like that we have great answers to that | 15:14 |
| ttx | (yes, I copied last week's intro) | 15:15 |
| liuyulong | Hi, my name is LIU Yulong, I'm the core of Neutron project. So I just want to see how many scale issue/pains you guys see on Neutron. : ) | 15:15 |
| genekuo_ | Hi, I'm Gene Kuo, working at LINE as Infrastructure engineer. Our team have been developing and operating OpenStack based private clouds to run our services. | 15:15 |
| ttx | liuyulong: neutron is definitely a hot topic around here | 15:15 |
| ttx | especially since rabbitMQ started to behave a bit more sanely lately | 15:15 |
| genekuo_ | I also copied last weeks intro. | 15:15 |
| genekuo_ | I probably cannot give a lot of feedback regarding neutron as we implemented our own plugins | 15:16 |
| mdelavergne | Hi, I'm Marie Delavergne, PhD student working on large scale Openstacks :) | 15:16 |
| ttx | Neutron is often the first scaling pain point, so we appreciate you visiting! | 15:16 |
| liuyulong | Yes, I know that. | 15:16 |
| liuyulong | I'm working at China Unicom now. We have some large deployment for public cloud. | 15:17 |
| ttx | jpward: care to introduce yourself and tell us what you're interested in? | 15:17 |
| jpward | I'm John Ward, and I work for Global InfoTek, we have a number of different openstack deployments, the largest one that I am working on is a 15k core but looking to continue scaling. | 15:18 |
| ttx | nice! I suspect it's already made of multiple clusters? | 15:18 |
| ttx | or cells or.. | 15:18 |
| jpward | glad to be here, I bring some experience working on even larger cloud Rackspace public cloud is my previous experience | 15:19 |
| jpward | currently we don't have cells implemented, but that is on the road map | 15:19 |
| ttx | Nice, very interested in hearing how your scaling went so far ! | 15:20 |
| genekuo_ | Cool, nice to meet you | 15:20 |
| ttx | So, for today's meeting we planned to discuss how to best collect feedback from experienced operators | 15:20 |
| ttx | #topic How to best collect feedback from experienced operators? | 15:20 |
| *** openstack changes topic to "How to best collect feedback from experienced operators? (Meeting topic: large_scale_sig)" | 15:20 | |
| ttx | as I said a couple of weeks ago, during Victoria cycle we tried to use etherpads to collect scaling stories, then curate them onto a wiki page | 15:21 |
| ttx | That was not very successful. | 15:21 |
| ttx | (understatement of 2020) | 15:21 |
| ttx | In contrast, we had several people sharing at our Opendev and Forum sessions around scaling | 15:21 |
| ttx | So I was wondering if we should not change our strategy there | 15:21 |
| ttx | Rather than run opendev and forum sessions about scaling, in hope that people will join the SIG and share more... | 15:21 |
| imtiazc | I am Imtiaz Chowdhury. I am the Cloud Architect for Workday. At Workday, we have 45 clusters running over 9K hypervisors and now close to 500K core. The deployment size is expected to double next year. | 15:22 |
| ttx | Maybe we should run events specifically to collect those experiences, and not expect people to join the SIG afterwards | 15:22 |
| ttx | or fill an etherpad | 15:22 |
| ttx | What's your view on that? | 15:22 |
| genekuo_ | Yes that what I also think | 15:22 |
| genekuo_ | It's hard to get people fill out etherpad after work | 15:22 |
| ttx | (i mean, if people jion the SIG as regular members, that's awesome, but we should collect their scaling story without expecting them to join first) | 15:23 |
| genekuo_ | I think an event with in OpenInfra summit and OpsMeetup is the best place to gather information | 15:23 |
| ttx | Also it's hard to write and easier to just discuss | 15:23 |
| imtiazc | Etherpad or any tool that allows collaborative editing: Google docs, Wiki | 15:23 |
| ttx | So how about... | 15:24 |
| genekuo_ | People tend to get more active when there's event or deadline | 15:24 |
| ttx | We build a schedule of regular Large Scale SIG events (think ~ every 2 months) | 15:24 |
| ttx | piggybacking on existing events (forum, opendev, ops meetup) if available, or running our own if not | 15:24 |
| ttx | and use that to ask specific questions and collect output | 15:25 |
| genekuo_ | I will suggest piggybacking at first | 15:25 |
| genekuo_ | And advertise it to some event you can join even if you are not currently running large scale | 15:25 |
| ttx | genekuo_: yes but there isn;t much planned in the coming months. I have to doublecheck what the OpsMettup has planned | 15:25 |
| ttx | Other suggestions included: | 15:26 |
| ttx | - Leverage superuser nominations to extract knowledge | 15:26 |
| genekuo_ | but planning to scale in the future | 15:26 |
| ttx | - Reach out to past speakers that spoke on scaling | 15:26 |
| ttx | - Engage with Chinese users | 15:26 |
| ttx | so, more direct or narrow outreach | 15:26 |
| liuyulong | Neutron team has a mechanism that will enable a deputy each week to collect/filter the bugs. | 15:26 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018782.html | 15:27 |
| liuyulong | For instance ^ | 15:27 |
| imtiazc | I like those suggestions. | 15:27 |
| liuyulong | So maybe this SIG can add such routine for collecting the scale related infromations. | 15:27 |
| liuyulong | And feedback a mail to the community. | 15:27 |
| ttx | liuyulong: that sounds good | 15:27 |
| ttx | Superuser nominations, next round is a bit far away | 15:27 |
| imtiazc | Is there a way we could facilitate connecting different large scale operators? | 15:28 |
| ttx | but we could try to identify past summit talks on scaling | 15:28 |
| ttx | and reach out to speakers | 15:28 |
| genekuo_ | Previous superuser nominations also will work | 15:28 |
| ttx | (in addition to actually extractign info from the video content) | 15:28 |
| *** mlavalle has joined #openstack-meeting-3 | 15:28 | |
| genekuo_ | I can help out reaching directly to those user if man power is needed. | 15:29 |
| ttx | Re: old scaling presentations, I'll create an etherpad where we can dump out findings, organized per event | 15:29 |
| ttx | I think that's a good resource to link to in our various stages page anyway | 15:30 |
| ttx | #link https://etherpad.opendev.org/p/large-scale-sig-scaling-videos | 15:30 |
| genekuo_ | I can pick up some of those videos once the list is complete | 15:31 |
| genekuo_ | I'll pick up listing for Shanghai | 15:33 |
| genekuo_ | Will do others if I have additional time | 15:33 |
| ttx | If you have a few cycles, please assign yourself one of the summits and do a quick search for scale-related presentations | 15:34 |
| ttx | If you watch any, feel free to drop notes and remarks on the etherpad too | 15:34 |
| ttx | #action all to help in filling out https://etherpad.opendev.org/p/large-scale-sig-scaling-videos | 15:34 |
| ttx | I'll check out the Ops meetups future plans | 15:35 |
| ttx | #action ttx to check out Ops meetups future plans | 15:35 |
| imtiazc | I shall look at Virtual summit and Denver. I shall also add the past scaling presentation we did from Workday | 15:35 |
| ttx | imtiazc: great, thanks | 15:35 |
| ttx | as far as Chinese users go, I'll defer to Chinese contributors. It feels like we get a lot of engagement when we use China-specific social media | 15:36 |
| ttx | so I was wondering if we could use that to ask simple questions from large scale deployments in China | 15:36 |
| ttx | am open to suggestions on how to best proceed theer | 15:37 |
| ttx | there* | 15:37 |
| imtiazc | I am not sure about that. Could we get some help from Jonathan Bryce or Mark C here? They seem to at least have contacts of the large operators and sponsors from China | 15:38 |
| genekuo_ | hmm, I can ask Rico if he can help | 15:38 |
| genekuo_ | ricolin | 15:38 |
| ttx | That suggestion was actually from Rico :) | 15:39 |
| genekuo_ | yeah I know | 15:39 |
| ttx | OK, that sounds like great first steps. Any other suggestions? | 15:40 |
| ttx | Alright then, moving on to next topic | 15:41 |
| ttx | #topic Next meeting | 15:41 |
| *** openstack changes topic to "Next meeting (Meeting topic: large_scale_sig)" | 15:41 | |
| ttx | Our next meeting will be December 16. | 15:41 |
| ttx | (Then we'll skip, and have the one after that on January 13) | 15:41 |
| *** lpetrut has quit IRC | 15:41 | |
| ttx | The main topic for that next meeting will be to review all stages, and identify simple tasks to do a first pass at improving those pages | 15:41 |
| jpward | same time on the 16th? | 15:41 |
| ttx | yes | 15:41 |
| genekuo_ | I'm ok | 15:42 |
| mdelavergne | ok | 15:42 |
| ttx | So between now and then, please check out the base content at https://wiki.openstack.org/wiki/Large_Scale_SIG and think a bit on how we can do a first pass at improving that | 15:42 |
| ttx | I bet there are a few easy question/answers we could add | 15:43 |
| genekuo_ | yep | 15:43 |
| ttx | Like, put yourself back into that stage of your own scaling story, and answer one of your own early questions you had | 15:43 |
| ttx | #topic Open discussion | 15:44 |
| *** openstack changes topic to "Open discussion (Meeting topic: large_scale_sig)" | 15:44 | |
| ttx | That is all we had on the agenda... Is there anything else you would like to discuss? | 15:44 |
| genekuo_ | Nope :) | 15:44 |
| jpward | nice meeting everyone, nothing else from me | 15:44 |
| imtiazc | I have a question on deployment story | 15:44 |
| ttx | If not, I'll wrap up now and post the summary. We have a bunch of actions to work on between now and next meeting | 15:44 |
| ttx | imtiazc: yes? | 15:45 |
| imtiazc | What deployment tools work best for large scale deployment? We are aware of limitations of TripleO but not so sure about Kolla. | 15:45 |
| genekuo_ | We currently write our own Ansible script and separate compute nodes to different host groups | 15:47 |
| ttx | I heard good things of OpenStack-Ansible, but i don't run a deployment myself. What do you all use, if anything? | 15:47 |
| ttx | jpward, belmoreira, liuyulong: any specific tooling? | 15:48 |
| jpward | We are using salt for our deployment currently, I have used OSA and TripleO in the past | 15:48 |
| imtiazc | We are currently using community forked Chef based tools with some home grown tools. | 15:48 |
| ttx | wow lots of homegrown tools | 15:48 |
| ttx | I thought there was more convergence toward community deployment tools, but maybe I imagined things | 15:50 |
| imtiazc | ttx: Do you think "deployment" story could be added as stage zero to the list of categories? | 15:50 |
| genekuo_ | I personally use kolla-ansible for my own cluster before and have good experience with it, but the scale is very small | 15:50 |
| ttx | jpward: I had an unrelated question, how did you learn about the SIG? | 15:51 |
| jpward | I run across the wiki site one day when searching for something else | 15:52 |
| ttx | ah? funny | 15:53 |
| mdelavergne | unexpected | 15:53 |
| ttx | yeah usually people can't find anything in the wiki | 15:54 |
| *** slaweq has quit IRC | 15:54 | |
| ttx | alright, if nothing else... | 15:54 |
| ttx | Let's continue the discussion at our next meeting | 15:54 |
| mdelavergne | I like the idea of deployment as stage zero! | 15:54 |
| belmoreira | sorry... couldn't follow the meeting... fighting some fires | 15:55 |
| ttx | mdelavergne: we'll end up writing a complete guide to openstack :) | 15:55 |
| mdelavergne | ahah | 15:55 |
| ttx | belmoreira: it's ok, you can catch up with the logs | 15:55 |
| ttx | belmoreira: I should know that, but do you use a specific deployment tooling to handle the CERRN deployment? | 15:56 |
| *** slaweq has joined #openstack-meeting-3 | 15:56 | |
| ttx | CERN* | 15:56 |
| belmoreira | for configuration management we use puppet | 15:56 |
| belmoreira | we deploy OpenStack using puppet | 15:57 |
| ttx | using the openstack-puppet upstream stuff? | 15:57 |
| liuyulong | Our operators use ansible, but the templates are written by them, not the openstack ansible. | 15:57 |
| belmoreira | yes, openstack-puppet | 15:57 |
| ttx | alright, thanks everyone, time to move to.. another meeting | 15:57 |
| ttx | #endmeeting | 15:57 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:57 | |
| genekuo_ | Thank you all today | 15:57 |
| openstack | Meeting ended Wed Dec 2 15:57:55 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:57 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-12-02-15.00.html | 15:57 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-12-02-15.00.txt | 15:58 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-12-02-15.00.log.html | 15:58 |
| mdelavergne | thanks everyone, see you on the 16th! | 15:58 |
| jpward | thanks everyone | 15:58 |
| *** liuyulong has quit IRC | 15:58 | |
| *** mdelavergne has quit IRC | 15:58 | |
| *** macz_ has joined #openstack-meeting-3 | 16:33 | |
| *** e0ne has quit IRC | 16:51 | |
| *** psahoo has quit IRC | 17:16 | |
| *** artom has quit IRC | 17:38 | |
| *** e0ne has joined #openstack-meeting-3 | 17:41 | |
| *** lpetrut has joined #openstack-meeting-3 | 18:09 | |
| *** belmoreira has quit IRC | 18:34 | |
| *** elod has left #openstack-meeting-3 | 18:54 | |
| *** artom has joined #openstack-meeting-3 | 19:15 | |
| *** e0ne has quit IRC | 19:16 | |
| *** e0ne has joined #openstack-meeting-3 | 19:50 | |
| *** e0ne has quit IRC | 20:11 | |
| *** vesper11 has joined #openstack-meeting-3 | 20:15 | |
| *** vesper has joined #openstack-meeting-3 | 20:19 | |
| *** vesper11 has quit IRC | 20:19 | |
| *** lpetrut has quit IRC | 20:29 | |
| *** vesper has quit IRC | 21:06 | |
| *** vesper11 has joined #openstack-meeting-3 | 21:30 | |
| *** raildo has quit IRC | 21:40 | |
| *** slaweq has quit IRC | 23:02 | |
| *** macz_ has quit IRC | 23:16 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!