Wednesday, 2020-07-22

*** ricolin has joined #openstack-meeting-302:01
*** baojg has quit IRC03:10
*** psachin has joined #openstack-meeting-303:31
*** maciejjozefczyk_ has joined #openstack-meeting-305:43
*** halali_ has quit IRC06:10
*** e0ne has joined #openstack-meeting-306:27
*** halali_ has joined #openstack-meeting-306:34
*** ralonsoh has joined #openstack-meeting-307:29
*** markvoelker has joined #openstack-meeting-307:47
*** dparkes has joined #openstack-meeting-307:48
*** markvoelker has quit IRC07:52
ttx#startmeeting large_scale_sig08:00
openstackMeeting started Wed Jul 22 08:00:03 2020 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.08:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.08:00
*** openstack changes topic to " (Meeting topic: large_scale_sig)"08:00
openstackThe meeting name has been set to 'large_scale_sig'08:00
ttx#topic Rollcall08:00
*** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)"08:00
ttxHi everyone! Who is here for the Large Scale SIG meeting ?08:00
ttxOur agenda for today is at:08:00
ttx#link https://etherpad.openstack.org/p/large-scale-sig-meeting08:00
ttx(please feel free to add to it)08:00
*** t0rrant has joined #openstack-meeting-308:02
ttxamorin, dparkes: hi!08:02
t0rrantgood morning!08:02
dparkesHi, hello08:02
ttxt0rrant: welcome08:02
* t0rrant thanks08:03
*** psahoo has joined #openstack-meeting-308:03
ttxwaiting a minute for more people to join08:03
ttxhalali_: hi!08:04
ttxOK let's start, I'm not sure we'll have more people, this is summer time08:04
ttx#topic Welcome newcomers08:04
*** openstack changes topic to "Welcome newcomers (Meeting topic: large_scale_sig)"08:04
ttxA few weeks ago we had a great Opendev virtual event around large scale deployments of open infrastructure08:05
ttxAs a result of that event we have a number of new people joining us today08:05
ttxSo I'd like to take the time to welcome new attendees, and spend some time discussing what they are interested in08:05
ttxt0rrant: care to introduce yourself and what you're interested in?08:05
t0rrantsure08:06
*** mdelavergne has joined #openstack-meeting-308:07
ttxmdelavergne: hi!08:07
t0rrantMy name is Manuel Torrinha and I work with INESC-ID in Portugal. I collaborate with the IT services in a university setting which have a multi-region Openstack deployment. That said I'm here today to learn from your larger-scale examples overall and am specially interested in discussing different control plane architectures in order to improve our own08:07
mdelavergnehi, sorry for being late08:07
ttxt0rrant: welcome. I'm Thierry Carrez, VP of Engineering for the OpenStack Foundation, helping with running the SIG08:08
ttxMy interest is in getting operators of large scale deployments to collaborate together and share practices and tools08:09
ttxdparkes: how about you?08:09
dparkesyes, sure08:09
ttxother regular members of the SIG are mdelavergne, amorin, masahito, belmoreira... but most of them are not around today08:11
ttxdparkes: could you quickly introduce yourself?08:12
dparkesHi, I'm Daniel Parkes and work for services at Red Hat, I work with OSP, and have several large deployments that I work with were we fight daily with performance and operational issues, so I'm here to share experiences on what things I see break how we fix them so contribute to user stories, and share also your knowledge and discuss about all these topics08:12
ttxnice!08:12
ttxSo when the SIG started, we tried to avoid boiling the ocean, and came up with two short-term objectives08:13
ttxtrying to keep it reasonable, as inflated expectations can quickly kill a group like this08:14
ttxWe'll now review progress on those two goals08:14
ttxBut obviously the SIG just goes where its members push it, so if there is a workstream you'd like to pursue, we are open08:14
ttxhalali_: are you around?08:15
ttxI guess not, we can come back to intros if he shows up later08:15
ttx#topic Progress on "Documenting large scale operations" goal08:15
*** openstack changes topic to "Progress on "Documenting large scale operations" goal (Meeting topic: large_scale_sig)"08:15
ttx#link https://etherpad.openstack.org/p/large-scale-sig-documentation08:15
ttxSo this is one of our current goals for the SIG - produce better documentation to help operators setting up large deployments.08:15
halali_Hello, Sorry for that, was away from the desk, discuss something with team, will join later08:15
ttxsure no pb :)08:16
ttxIn particular, this goal is around documenting better configuration values when you start hitting issues with default values08:16
ttxIf you look at https://etherpad.openstack.org/p/large-scale-sig-documentation you can see the various things we are pushing in that direction08:16
ttxI'll push back amorin's TODO from last meeting since he is in vacation those days08:17
ttx#action amorin to add some meat to the wiki page before we push the Nova doc patch further08:17
ttxWe had another work item on collecting metrics/billing stories08:17
ttxThat points to one critical activity of this SIG:08:17
ttxIt's all about sharing your experience operating large scale deployments of OpenStack08:17
ttxso that we can derive best practices and/or fix common issues08:17
ttxOnly amorin contributed the story for OVH on the etherpad so far (line 34+), so please add to that08:18
ttxif you have any experience with such setups08:18
ttxI'll log an action item to everyone on that08:18
ttx#action all to describe briefly how you solved metrics/billing in your deployment in https://etherpad.openstack.org/p/large-scale-sig-documentation08:18
ttxFinally, we had an action on discussing how to upstream osarchiver, OVH's internal tool for database cleanup08:19
ttxamorin raised a thread about it:08:19
ttx#link http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015970.html08:19
ttxI did reply as planned to discuss how to best land it:08:19
ttx#link http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015978.html08:19
ttxNot much replies on that yet... so I'll escalate it to get a response08:19
ttx#action ttx to escalate OSops revival thread for osarchiver hosting08:19
dparkesYes I have had interest in the osarchiver in the past, and did some testing, it would be a great addition08:20
ttxThat's it for status updates on this goal... did you have comments on this goal, does that sound like a good thing to pursue, any additional action you'd like to suggest in that area ?08:20
ttx(the trick being, it's hard to push for best practices until we reach a critical mass of experience feedback)08:21
t0rrantI wasn't aware of osarchiver but it looks like a very useful tool for large scale and even small scale deployments08:21
ttxyes, we are trying to revive the "OSops" concept and land it there. OSops was an operator-led collection of small tools, with low bar to entry08:22
t0rrantwe don't use mistral for example, but I guess it goes through all the services08:22
ttxok, moving on to the other SIG goal...08:23
ttx#topic Progress on "Scaling within one cluster" goal08:23
*** openstack changes topic to "Progress on "Scaling within one cluster" goal (Meeting topic: large_scale_sig)"08:23
ttx#link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling08:23
ttxThis is the other early goal of the SIG - identify, measure and push back common scaling limitations within one cluster08:23
ttxTo that effect we collect "scaling stories", like what happened when you started adding up nodes, what breaks first08:24
ttxWe collect the stories in https://etherpad.opendev.org/p/scaling-stories08:24
ttxand then publish them to https://wiki.openstack.org/wiki/Large_Scale_Scaling_Stories for long-term storage08:24
ttxOne common issue when adding nodes is around RabbitMQ falling down08:25
ttxAnd so this SIG has worked to produce code to instrument oslo.messaging calls and get good metrics from them08:25
ttxThis resulted in https://opendev.org/openstack/oslo.metrics -- a PoC that we hope to turn into a full-fledged oslo library to do taht instrumentation08:25
ttxNext step is to add basic tests, so that we are reasonably confident we do not introduce regressions08:25
ttxLet me know if you are interested in looking into that08:26
t0rrantone thing we are seeing in our deployment is timeouts on the DB side, and probably MQ bottlenecks. We have still metrics collection to set up to be sure, but I can make it fail with a not so large rally test08:26
t0rrantthat tool would be very helpful08:26
*** baojg has joined #openstack-meeting-308:27
ttxyes, and the idea is to expand it beyond oslo.messaging, to oslo,db, which would allow to capture those DB timeouts08:27
t0rrant+108:27
ttxLINE used the feedback from that tool to efficiently shard its RabbitMQ setup08:27
ttxallowing to push back the size of an individual cluster and not create too many of those08:28
dparkesBecause osprofiler doesn't have much traction these days?08:28
ttxless traction, but also more scope08:28
ttxthis is more targeted08:28
dparkeswe sometimes find with the need of tracing to see where the issue is, so somethis like oslo.metrics would be great08:28
ttxfor me osprofiler is a dev tool, while this is an operational tool08:29
ttxobviously there is overlap08:29
dparkesttx yes, something light, easy for ops, but that can give you and idea of where your spending your time08:29
ttxdparkes: interested in your scaling (horror) stories. Not sure how much you can share from RH OSP customers, but even anonymized repotrs would be super-useful08:30
ttxThe idea being to identify what breaks first and focus on that, and gradually raise the number of hosts we can have in a given cluster08:32
ttxOK, anything else on that topic?08:32
ttxdparkes, t0rrant: do those two goals sound good to you? Or was there something completely different you were interested in pursuing?08:33
t0rrantthose goals seem reasonable to me, one thing I would like to discuss is advice on control plane architecture,m although I don't know if this meeting is the most appropriate :P08:34
dparkesttx yes I will go through the notes and try to add to the user stories, things that break ,etc08:34
ttxdparkes: cool, thanks08:34
ttxt0rrant: we rae not really at that level of detail yet, but we could come to it08:35
ttxok, moving on...08:35
t0rrantsure thing08:35
ttx#topic Discuss a US-friendly meeting time08:35
*** openstack changes topic to "Discuss a US-friendly meeting time (Meeting topic: large_scale_sig)"08:35
ttxFollowing the Opendev event we have a couple more people interested in joining08:35
ttxBut they are based on US Pacific TZ, so our current meeting time is not working for them :)08:35
ttx(1am)08:35
ttxThe SIG members are currently mostly in EU, with a couple in APAC08:36
ttxGiven the SIG membership, I was thinking we could alternate between a APAC-EU time and a EU-US time.08:36
ttxFor example, have next meeting in two weeks at 16utc, then in four weeks, back to 8utc08:36
ttxWould that work for you all? Obviously just attend the meetings you can attend :)08:36
dparkesyes, sounds fair08:36
t0rrantlooks like a good compromise yes08:37
ttxSince the goal of the SIG is really to collect and share experiences, I feel like we'll maximize input that way08:37
mdelavergnefine by me, but 16utc every time is also fine08:37
ttxeven if that will make my work communicating each meeting output a bit more critical :)08:37
ttxI'll confirm with our US-based prospects that 16utc every 4 weeks on wednesdays is ok, and update the meeting info.08:38
ttx#action ttx to set alternating US-EU / EU-APAC meetings08:38
ttx#topic Next meeting08:38
*** openstack changes topic to "Next meeting (Meeting topic: large_scale_sig)"08:38
ttxSo... I'll be taking time off in two weeks, at when would be our first US-EU meeting... So I'd rather move it off one week08:38
ttxLike first US-EU meeting on August 12, 16utc, then next EU-APAC meeting on August 26, 8utc.08:38
ttxHow does that sound?08:38
t0rrantsounds good to me08:39
ttxWe have reduced attendance in summer months anyway08:39
mdelavergneyep, sounds good08:39
ttxOK, I'll keep you posted on openstack-discuss mailing-list as always08:39
mdelavergnethanks :)08:39
ttxwe are using the [largescale-sig] prefix for all things SIG-related08:40
ttxI announce the meetings, and post summaries after the fact there08:40
ttx#info next meetings: Aug 12, 16:00UTC, Aug 26, 8:00UTC08:40
ttx#topic Open discussion08:40
*** openstack changes topic to "Open discussion (Meeting topic: large_scale_sig)"08:40
ttxAnything else you'd like to discuss today?08:41
ttxThanks again for joining the SIG and helping making OpenStack better!08:41
t0rrantThank you!08:42
ttxAlright, let's close this. Thanks everyone for attending today08:42
mdelavergneWelcome to the newcomers, and thanks everyone08:42
ttx#endmeeting08:42
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"08:42
openstackMeeting ended Wed Jul 22 08:42:54 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)08:42
openstackMinutes:        http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-07-22-08.00.html08:42
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-07-22-08.00.txt08:42
openstackLog:            http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-07-22-08.00.log.html08:42
dparkesThanks!08:43
mdelavergnesee you in august then!08:43
t0rrantseen then08:43
t0rrant*see you08:43
*** mdelavergne has quit IRC08:43
*** t0rrant has left #openstack-meeting-309:04
*** halali_ has quit IRC10:43
*** ricolin has quit IRC10:48
*** halali_ has joined #openstack-meeting-311:07
*** raildo has joined #openstack-meeting-311:29
*** baojg has quit IRC11:49
*** baojg has joined #openstack-meeting-311:51
*** maciejjozefczyk_ has quit IRC12:01
*** maciejjozefczyk has joined #openstack-meeting-312:01
*** baojg has quit IRC12:11
*** baojg has joined #openstack-meeting-312:12
*** tosky has joined #openstack-meeting-312:17
*** artom has quit IRC12:24
*** e0ne has quit IRC12:33
*** e0ne has joined #openstack-meeting-312:37
*** e0ne has quit IRC12:39
*** artom has joined #openstack-meeting-313:21
*** baojg has quit IRC13:23
*** baojg has joined #openstack-meeting-313:24
*** bnemec has joined #openstack-meeting-313:30
*** baojg has quit IRC13:41
*** baojg has joined #openstack-meeting-313:43
*** ricolin has joined #openstack-meeting-313:43
*** e0ne has joined #openstack-meeting-314:00
*** e0ne has quit IRC14:22
*** baojg has quit IRC14:22
*** e0ne has joined #openstack-meeting-314:23
*** psachin has quit IRC14:24
*** baojg has joined #openstack-meeting-314:25
ralonsoh#startmeeting neutron_ci15:02
openstackMeeting started Wed Jul 22 15:02:07 2020 UTC and is due to finish in 60 minutes.  The chair is ralonsoh. Information about MeetBot at http://wiki.debian.org/MeetBot.15:02
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:02
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:02
openstackThe meeting name has been set to 'neutron_ci'15:02
ralonsohhello, sorry for the delay15:02
ralonsohhmmmm summer time, I guess15:03
ralonsohok, just for the records, I'll do a quick agenda review15:04
ralonsoh#topic Actions from previous meetings15:04
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:04
ralonsohmaciejjozefczyk to check neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers in ovn jobs15:04
ralonsohI don't know if you had time to review it15:04
ralonsohok, that was the only item15:05
ralonsohon't know if you had time to review it15:05
ralonsohsorry15:05
ralonsoh#topic Stadium projects15:05
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:05
ralonsohtwo of them with active patches15:05
ralonsohnetworking-odl15:05
ralonsoh#link https://review.opendev.org/#/c/725647/15:06
ralonsohand Neutron OVN15:06
ralonsoh#link https://review.opendev.org/#/c/729591/15:06
ralonsohboth are passing15:06
ralonsohabout IPv6-only CI, no activity in the community15:07
ralonsoh#link https://etherpad.openstack.org/p/neutron-stadium-ipv6-testing15:07
ralonsohall patches are still WIP15:07
ralonsohok, next topic15:07
ralonsoh#topic Switch to Ubuntu Focal15:07
*** openstack changes topic to "Switch to Ubuntu Focal (Meeting topic: neutron_ci)"15:07
ralonsoh#link https://etherpad.opendev.org/p/neutron-victoria-switch_to_focal15:07
ralonsoh#link https://review.opendev.org/#/c/734304/15:07
ralonsohthe pending patch was merged15:08
ralonsohI rebased it to check if works now15:08
ralonsohnext one https://review.opendev.org/#/c/737370/15:08
ralonsohstill waiting for https://review.opendev.org/#/c/734700/15:09
ralonsohand for ODL15:09
ralonsoh#link https://review.opendev.org/#/c/736703/15:09
ralonsohWIP15:09
ralonsoh#topic Stable branches15:09
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)"15:09
ralonsohUssuri dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=115:10
ralonsohTrain dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=115:10
ralonsohBut I can make any statement about both dashboards, I didn't review them15:10
ralonsoh#topic fullstack/functional15:10
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"15:10
ralonsohThe problems we had on Monday due tohttps://review.opendev.org/#/c/739306/, were solved in https://review.opendev.org/#/c/741957/15:11
ralonsohI don't recall any other "big" problem in FT or fullstack tests15:11
ralonsoh#topic Tempest/Scenario15:11
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"15:11
ralonsoh#link https://review.opendev.org/#/c/739955/ is already merged, increasing the tempest jobs timeout15:12
ralonsohthe problem with neutron-ovn-tempest-full-multinode-ovs-master15:12
ralonsohreported in https://bugs.launchpad.net/neutron/+bug/188680715:12
openstackLaunchpad bug 1886807 in neutron "neutron-ovn-tempest-full-multinode-ovs-master job is failing 100% times" [High,Confirmed] - Assigned to Maciej Jozefczyk (maciej.jozefczyk)15:12
ralonsohthat should be fixed in https://review.opendev.org/#/c/741606/15:13
ralonsohsetting both OVS and OVN branches to "master"15:13
ralonsohthis patch is almost merged15:13
ralonsohand that's all this *slow* week15:13
ralonsohif you review the meeting logs and you have any question or problem, do not hesitate to ping me in IRC15:14
ralonsohthanks15:14
ralonsoh#endmeeting15:14
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:14
openstackMeeting ended Wed Jul 22 15:14:37 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:14
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-22-15.02.html15:14
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-22-15.02.txt15:14
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-22-15.02.log.html15:14
*** factor has joined #openstack-meeting-315:26
*** njohnston has joined #openstack-meeting-315:30
*** baojg has quit IRC15:54
*** baojg has joined #openstack-meeting-315:56
*** factor has quit IRC16:08
*** factor has joined #openstack-meeting-316:09
*** e0ne has quit IRC16:15
*** ricolin has quit IRC16:19
*** halali_ has quit IRC16:30
*** baojg has quit IRC16:34
*** factor has quit IRC16:39
*** factor has joined #openstack-meeting-316:40
*** factor has quit IRC16:40
*** halali_ has joined #openstack-meeting-316:46
*** factor has joined #openstack-meeting-317:04
*** halali_ has quit IRC17:05
*** e0ne has joined #openstack-meeting-317:20
*** psahoo has quit IRC17:25
*** _erlon_ has joined #openstack-meeting-318:02
*** icarusfactor has joined #openstack-meeting-318:07
*** icarusfactor has left #openstack-meeting-318:08
*** factor has quit IRC18:09
*** e0ne has quit IRC18:15
*** halali__ has joined #openstack-meeting-318:26
*** ralonsoh has quit IRC18:45
*** bnemec has quit IRC19:03
*** baojg has joined #openstack-meeting-319:07
*** e0ne has joined #openstack-meeting-319:17
*** maciejjozefczyk has quit IRC19:26
*** baojg has quit IRC19:31
*** baojg has joined #openstack-meeting-319:32
*** e0ne has quit IRC19:47
*** e0ne has joined #openstack-meeting-320:04
*** halali__ is now known as halali_20:05
*** e0ne has quit IRC20:12
*** bnemec has joined #openstack-meeting-321:04
*** bnemec has quit IRC21:11
*** bnemec has joined #openstack-meeting-321:18
*** baojg has quit IRC21:22
*** baojg has joined #openstack-meeting-321:23
*** raildo has quit IRC21:28
*** tosky has quit IRC23:03
*** halali_ has quit IRC23:15
*** markvoelker has joined #openstack-meeting-323:27
*** markvoelker has quit IRC23:32

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!