15:00:24 <belmoreira> #startmeeting large_scale_sig
15:00:25 <openstack> Meeting started Wed Mar 10 15:00:24 2021 UTC and is due to finish in 60 minutes.  The chair is belmoreira. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:28 <openstack> The meeting name has been set to 'large_scale_sig'
15:00:56 <belmoreira> hello everyone. Welcome to the Large Scale SIG meeting
15:01:03 <belmoreira> #topic Rollcall
15:01:20 <mdelavergne> Hi!
15:01:20 <genekuo> o/
15:01:31 <belmoreira> o/
15:01:39 <imtiazc> Hi
15:02:38 <belmoreira> hello. ttx can't join today, so I'm charing the meeting
15:02:50 <genekuo> Thanks!
15:02:58 <belmoreira> Our agenda for today is at:
15:03:03 <belmoreira> #link https://etherpad.openstack.org/p/large-scale-sig-meeting
15:03:13 <belmoreira> #topic Action items review
15:03:27 <belmoreira> ttx is not here but he left comments on etherpad - agenda
15:03:48 <belmoreira> Leaving them here for the minutes. Let me know if you have any comment
15:03:57 <belmoreira> * ttx to contact OpenStack TC re: OSarchiver and see which option sounds best
15:04:07 <belmoreira> From the etherpad: "Done, see logs at:"
15:04:14 <belmoreira> #link http://eavesdrop.openstack.org/meetings/tc/2021/tc.2021-02-18-15.01.log.html#l-166"
15:04:27 <belmoreira> * ttx to summarize "how many compute nodes in your typical cluster" thread into the wiki
15:04:34 <belmoreira> From the etherpad: "Done at:"
15:04:41 <belmoreira> #link https://wiki.openstack.org/wiki/Large_Scale_SIG/ScaleUp
15:04:50 <belmoreira> * ttx to create a wiki page for Ceph-related questions and seed it with SG questions
15:04:58 <belmoreira> From the etherpad: "Done at:"
15:05:04 <belmoreira> #link https://wiki.openstack.org/wiki/Large_Scale_SIG/CephAndOpenStack
15:05:58 <belmoreira> if no one has comments, let's move to the next topic
15:06:07 <belmoreira> #topic Discussing CentOS Stream
15:06:19 <imtiazc> We all know that the OpenStack technical group and governing body has decided to go with CentOS 8 Stream after CentOS 8 hit EOL. The questions we have are:
15:06:35 <imtiazc> 1. Stability: As an operator how do I know which version of Stream is "stable". How are you all planning to test and perform updates?
15:06:50 <imtiazc> 2. Hardware compatibility: Hardware support usually comes from RedHat. How do ensure certain
15:06:50 <imtiazc> Version of the Stream is compatible.
15:07:05 <imtiazc> 3. Security: Since there is no SLO for when fix for a CVE will be release on the Stream, what plan should we follow?
15:07:20 <imtiazc> Does anyone have any test methodology of framework?
15:07:55 <belmoreira> These are defenitly good questions and a concern for everyone running CentOS
15:08:05 <genekuo> I know we decided to go CentOS Stream in the future, but OS, security stuff is handle by other teams so I currently don't know the details
15:08:48 <belmoreira> CERN statement regarding CentOS 8 end-of-life and CentOS stream can be found at:
15:08:56 <belmoreira> #link https://linux.web.cern.ch/centos8/
15:09:00 <genekuo> Let me ask other team about these question to see how they handle these issue
15:09:37 <imtiazc> @belmoreira: How are you planning to test the snapshots  ?
15:11:44 <belmoreira> imtiazc I'm not directly involved with the Operating Systems team. But there are several discussions on how to move forward. For CERN this doesn't affects the Cloud Infrastructure but all the organization, including the experiments infrastructures and software
15:12:24 <jangutter> belmoreia: is your cloud infra based on CentOS 7?
15:12:27 <belmoreira> ... doesn't affects only the ...
15:13:11 <genekuo> LINE infra is currently based on CentOS 7
15:13:11 <imtiazc> Do you know what stable OpenStack release comes out on which version of the CentOS Stream?
15:13:25 <belmoreira> jangutter all the compute nodes are in CentOS 7 and the control plane started be be migrated to CentOS 8 before the end-of-life change
15:13:51 <jangutter> belmoreia: thanks!
15:15:37 <belmoreira> imtiazc RDO is moving to stream
15:15:41 <jangutter> (OpenStack TC published the tested runtimes here: https://governance.openstack.org/tc/reference/runtimes/xena.html. It's still uncertain if Wallaby is going to be retrofitted to Stream, but the RDO project has already indicated so).
15:16:57 <belmoreira> imitiazc from a talk in FOSDEM, Wallaby will be released for CentOS Stream 8
15:17:11 <belmoreira> #link: https://wiki.centos.org/Events/Dojo/FOSDEM2021?action=AttachFile&do=view&target=openstack_pena_moraejo.pdf
15:17:54 <belmoreira> and early releases will be tested on Stream as well
15:18:59 <belmoreira> imtiazc do you already have a plan? or still considering the options
15:19:50 <imtiazc> We are moving ahead with our plan to deploy with Victoria on CentOS 8. We now have to consider moving to Stream after 8 EOL
15:21:24 <belmoreira> for the compute nodes you upgraded from CentOS 7 or they are new clusters?
15:21:53 <imtiazc> These will be new clusters
15:22:24 <imtiazc> Which release is CERN is using now?
15:23:08 <jangutter> I guess my concern is the added burden of maintaining older versions of OpenStack on newer versions of Stream. Stable branches are going to have a lot more churn than in the past.
15:23:56 <belmoreira> imtiazc we are running Stein for Nova. Other projects are in Ussuri or Victoria
15:25:21 <imtiazc> Thanks for that.
15:25:51 <belmoreira> jangutter true. But at least CentOS 7 end-of-life dind't change which also is important because the python 2/3 change
15:27:18 <belmoreira> something else regarding this topic?
15:28:22 <imtiazc> I guess not. Can you keep us posted on how we can agree on a common strategy or test methodology?
15:28:23 <belmoreira> I think we will learn more about Stream and gain more confidence over time
15:29:33 <imtiazc> We shall share our experience as well :)
15:30:53 <belmoreira> "common strategy or test methodology" I think is very particular to each environment. For example for CERN one of the main concerns is all the operational and scientific software for experiments. Extensive tests will be required.
15:31:15 <belmoreira> I will keep the SIG posted if we have any update
15:31:51 <belmoreira> #topic Video meetings
15:32:05 <belmoreira> We had our first video meeting 2 weeks ago.
15:32:20 <belmoreira> More than 30 people participated. I think it was a great discussion.
15:32:33 <belmoreira> What's your opinion?
15:32:41 <mdelavergne> Yep, great discussion, great to see a lot of people in the SIG :)
15:32:53 <mdelavergne> Presentation was nice too ;)
15:33:07 <genekuo> More people join than I expected
15:33:58 <belmoreira> thanks mdelavergne
15:34:02 <imtiazc> I love the video format as well. I thought it was a great discussion!
15:34:23 <belmoreira> I think we should try to have another video meeting in 2 weeks
15:34:30 <imtiazc> +1
15:34:53 <belmoreira> We collected some ideas few meetings ago:
15:34:59 <belmoreira> #link https://etherpad.opendev.org/p/large-scale-sig-lightning-talks
15:35:26 <belmoreira> I would prefer to not do the introduction presentation next time
15:36:12 <belmoreira> What should be the discussion topic and who would like to do the introduction presentation?
15:37:18 <mdelavergne> Everyone is suddenly hidden in bushes
15:38:35 <genekuo> I'm trying to find out which topic is better to present using our case
15:38:36 <belmoreira> in the agenda "Piortr" asked about multitenant Neutron scaling
15:40:01 <genekuo> Maybe I can introduce how we do multi-tenant using SRv6
15:40:26 <genekuo> But we're currently only running it in a small cluster, not exactly sure how it will scale
15:40:58 <genekuo> And the plugin isn't under upstream repo yet
15:41:47 <belmoreira> genekuo For this topic I was thinking more about Neutron scalability issues... and the workarounds
15:42:25 <belmoreira> is anyone willing to talk about the experience?
15:42:56 <belmoreira> From the last meeting I think 10min presentation is enough to pace the discussion after
15:43:32 <genekuo> I can probably talk about how we setup and what metrics we monitor for rabbitmq
15:43:38 <belmoreira> other topic that can be very interesting is RabbitMQ scalling
15:43:42 <genekuo> but I'll have to ask for permission first
15:44:03 <imtiazc> Our neutron experience is very specific to Contrail.
15:44:51 <belmoreira> genekuo that would be great
15:45:10 <genekuo> We're using our plugin similar to Calico that uses BGP and without multi-tenancy, so the neutron topic might not fit our use case.
15:45:59 <genekuo> belmoreira, let me ask my manager for clearance tomorrow
15:47:01 <belmoreira> #action genekuo will inform us if he can do a short presentation about "Scalling RabbitMQ" in the next SIG video meeting
15:47:32 <belmoreira> genekuo, thanks let us know
15:47:42 <genekuo> I will
15:48:12 <belmoreira> I think this will be a great topic
15:48:30 <belmoreira> let's move on...
15:48:33 <belmoreira> #topic OSarchiver
15:48:55 <belmoreira> I think amorin is not around
15:49:45 <mdelavergne> Probably not, considering what happened at OVH
15:49:57 <genekuo> Yeah
15:50:12 <belmoreira> yes
15:50:58 <belmoreira> This was discussed in the TC meeting. Should be discussed with amorin in a future meeting
15:51:27 <belmoreira> #topic Progress/Blockers on documenting the Large Scale Journey
15:51:38 <belmoreira> * Stage 1 - Configure (amorin)
15:51:44 <belmoreira> #link https://wiki.openstack.org/wiki/Large_Scale_SIG/Configure
15:52:07 <belmoreira> we skip the report
15:52:12 <belmoreira> * Stage 2 - Monitor (genekuo)
15:52:17 <belmoreira> #link https://wiki.openstack.org/wiki/Large_Scale_SIG/Monitor
15:52:36 <genekuo> I've added a short answer to 2 of the questions
15:52:50 <belmoreira> great
15:52:58 <belmoreira> * Stage 3 - Scale Up (ttx)
15:53:03 <belmoreira> #link https://wiki.openstack.org/wiki/Large_Scale_SIG/ScaleUp
15:53:10 <belmoreira> we skip the report
15:53:14 <belmoreira> # Stage 4/5 - Scale Out, upgrade & maintain (belmoreira)
15:53:19 <belmoreira> #link https://wiki.openstack.org/wiki/Large_Scale_SIG/ScaleOut
15:53:24 <belmoreira> #link https://wiki.openstack.org/wiki/Large_Scale_SIG/UpgradeAndMaintain
15:53:40 <belmoreira> I didn't include nothing new since the last meeting
15:53:50 <genekuo> I think it will be good to add the link to video in that page
15:54:03 <genekuo> recording of last video meeting
15:54:16 <belmoreira> fell free to add all your comments, questions/answers in this knowledge base
15:55:01 <belmoreira> genekuo the video link is the main SIG page. I will add the slides later
15:55:27 <genekuo> Ah, ok, just saw it
15:55:28 <belmoreira> #topic Next meeting
15:55:40 <belmoreira> Our next meeting should be in two weeks.
15:55:47 <belmoreira> keep in mind that US will have moved to DST and Europe not yet
15:56:15 <belmoreira> and we will try to have another video meeting.
15:56:28 <belmoreira> #info March 24th, 15UTC
15:57:22 <belmoreira> #info Next meeting will be a Zoom meeting (usual time, link to be posted a few days before) with topic: "Scaling RabbitMQ". genekuo will check if he can do the intro talk
15:57:34 <belmoreira> #topic Open discussion
15:58:07 <belmoreira> is there something else to discuss?
15:58:15 <genekuo> None from me
15:58:29 <mdelavergne> Me neither
15:59:17 <belmoreira> thanks a lot. See you in 2 weeks for the next video discussion
15:59:26 <mdelavergne> Thanks!
15:59:36 <genekuo> THanks!
15:59:40 <belmoreira> #endmeeting