Tuesday, 2020-01-21

*** brinzhang has joined #openstack-meeting00:08
*** brinzhang_ has quit IRC00:12
*** jamesmcarthur has quit IRC00:12
*** jamesmcarthur has joined #openstack-meeting00:12
*** jamesmcarthur has quit IRC00:18
*** njohnston_ has quit IRC00:23
*** jamesmcarthur has joined #openstack-meeting00:24
*** igordc has quit IRC00:25
*** igordc has joined #openstack-meeting00:25
*** brinzhang has quit IRC00:39
*** igordc has quit IRC00:49
*** brinzhang has joined #openstack-meeting00:52
*** brinzhang has quit IRC00:54
*** ayoung has quit IRC00:54
*** ricolin has joined #openstack-meeting00:55
*** ayoung has joined #openstack-meeting00:57
*** artom has quit IRC01:00
*** jamesmcarthur has quit IRC01:04
*** jamesmcarthur has joined #openstack-meeting01:04
*** jamesmcarthur has quit IRC01:06
*** jamesmcarthur has joined #openstack-meeting01:06
*** masahito has joined #openstack-meeting01:07
*** masahito has quit IRC01:07
*** yamamoto has joined #openstack-meeting01:15
*** lseki has quit IRC01:40
*** rfolco has quit IRC01:41
*** Lucas_Gray has quit IRC01:49
*** jamesmcarthur has quit IRC01:56
*** jamesmcarthur has joined #openstack-meeting01:57
*** jamesmcarthur has quit IRC02:02
*** ircuser-1 has joined #openstack-meeting02:02
*** ykatabam has quit IRC02:09
*** ykatabam has joined #openstack-meeting02:11
*** mhen has quit IRC02:12
*** macz has joined #openstack-meeting02:15
*** yamamoto has quit IRC02:19
*** macz has quit IRC02:20
*** jamesmcarthur has joined #openstack-meeting02:24
*** ociuhandu has joined #openstack-meeting02:47
*** ociuhandu has quit IRC02:51
*** jamesmcarthur has quit IRC03:04
*** jamesmcarthur has joined #openstack-meeting03:05
*** dklyle has joined #openstack-meeting03:07
*** apetrich has quit IRC03:10
*** jamesmcarthur has quit IRC03:11
*** yamamoto has joined #openstack-meeting03:14
*** jamesmcarthur has joined #openstack-meeting03:30
*** jamesmcarthur has quit IRC03:33
*** tpatil has joined #openstack-meeting03:56
*** tpatil has quit IRC03:59
*** hongbin has joined #openstack-meeting04:03
*** tetsuro has quit IRC04:18
*** tetsuro has joined #openstack-meeting04:19
*** tetsuro has quit IRC04:23
*** hongbin has quit IRC04:39
*** jamesmcarthur has joined #openstack-meeting04:42
*** links has joined #openstack-meeting04:49
*** tetsuro has joined #openstack-meeting05:03
*** sridharg has joined #openstack-meeting05:09
*** macz has joined #openstack-meeting05:20
*** macz has quit IRC05:22
*** jamesmcarthur has quit IRC05:35
*** jamesmcarthur has joined #openstack-meeting05:37
*** jamesmcarthur has quit IRC05:43
*** ykatabam has quit IRC05:58
*** jamesmcarthur has joined #openstack-meeting06:06
*** epoojad1 has joined #openstack-meeting06:08
*** lpetrut has joined #openstack-meeting06:08
*** lpetrut has quit IRC06:09
*** lpetrut has joined #openstack-meeting06:10
*** jamesmcarthur has quit IRC06:13
*** epoojad1 has quit IRC06:18
*** ykatabam has joined #openstack-meeting06:23
*** lpetrut has quit IRC06:49
*** maohongbo has joined #openstack-meeting07:03
*** jamesmcarthur has joined #openstack-meeting07:09
*** jamesmcarthur has quit IRC07:14
*** keiko-k has joined #openstack-meeting07:24
*** ociuhandu has joined #openstack-meeting07:30
*** ociuhandu has quit IRC07:35
*** cschwede has joined #openstack-meeting07:37
*** lpetrut has joined #openstack-meeting07:38
*** mahatic has joined #openstack-meeting07:38
*** slaweq has joined #openstack-meeting08:00
*** joxyuki has joined #openstack-meeting08:02
joxyuki#startmeeting tacker08:03
openstackMeeting started Tue Jan 21 08:03:45 2020 UTC and is due to finish in 60 minutes.  The chair is joxyuki. Information about MeetBot at http://wiki.debian.org/MeetBot.08:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.08:03
*** openstack changes topic to " (Meeting topic: tacker)"08:03
openstackThe meeting name has been set to 'tacker'08:03
*** takahashi-tsc has joined #openstack-meeting08:04
joxyuki#topic RollCall08:04
*** openstack changes topic to "RollCall (Meeting topic: tacker)"08:04
hyunsikyangHello All.08:04
joxyukihi hyunsikyang08:05
joxyukiok, let's get started...08:06
*** bnemec has joined #openstack-meeting08:06
joxyuki#topic spec: event alarm08:06
*** openstack changes topic to "spec: event alarm (Meeting topic: tacker)"08:06
joxyukiI posted some comments on the spec https://review.opendev.org/#/c/700167/08:07
joxyukitakahashi-tsc, do you have any query or discussion?08:07
takahashi-tscNot here, but there are something to consider. Basically I consider.08:09
*** jamesmcarthur has joined #openstack-meeting08:10
takahashi-tscI replied comment, so I would be grateful if you have additional comments.08:10
hyunsikyangI didn't check yet. I will check it too:) thanks08:11
joxyukiyes, I saw your replay just now. Let's keep discussion on gerrit or IRC>08:11
takahashi-tscOK, thanks.08:11
joxyuki#topic multi-interface for CNF08:12
*** openstack changes topic to "multi-interface for CNF (Meeting topic: tacker)"08:12
joxyukithanks for the patch, hyunsikyang08:12
joxyukido you have any discussion about it?08:12
hyunsikyangyou're welcome. last time, we checked the code and got +1.08:12
hyunsikyangSo now, If you have a time, please review test code.08:12
hyunsikyangand then we can finish it for next one.:)08:13
*** jamesmcarthur has quit IRC08:14
joxyukiI found functional test failed.08:15
hyunsikyangin my code?08:15
joxyukiI don't think your code.08:16
hyunsikyangAh I thought you mentioned my code.08:16
joxyukioh, it's due to your code. You deleted tools/test-setup-default-vim.sh so devstack installation failed in functional test.08:18
joxyukiand also tacker/test/etc/sample/local-vim.yaml.08:19
joxyukiDo you have any reason to delete these files?08:19
hyunsikyangAh AM I delete it? I thought I delete it in my patch08:19
hyunsikyangBecause, last time when I upload code, it include without my intend,, so I delete it.08:20
*** tesseract has joined #openstack-meeting08:20
hyunsikyangSo how can I fix it?08:21
joxyukiNP, can you revive these file on your patch?08:21
hyunsikyangI will add it in the next patch!08:21
joxyukiyou should checkout these file from master branch and submit next patchset.08:22
joxyukimoving next...08:22
joxyukikeiko-k, do you have your topic today?08:23
joxyuki#topic open-discussion08:24
*** openstack changes topic to "open-discussion (Meeting topic: tacker)"08:24
joxyukiwe should suport python3 and  drop python2 support.08:25
joxyukiI recommend you all to make your devstack on python3.08:26
joxyukiNotetaht when you do it, you need https://review.opendev.org/#/c/696227/.08:27
joxyukiYou can get the patch by doing git pull on master branch as it was already merged.08:28
*** priteau has joined #openstack-meeting08:29
joxyukiokey, does anyone have another topic? if not, I will close the meeting.08:29
*** ralonsoh has joined #openstack-meeting08:30
joxyukiseems nothig. Thanks folks.08:31
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"08:31
openstackMeeting ended Tue Jan 21 08:31:11 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)08:31
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tacker/2020/tacker.2020-01-21-08.03.html08:31
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tacker/2020/tacker.2020-01-21-08.03.txt08:31
openstackLog:            http://eavesdrop.openstack.org/meetings/tacker/2020/tacker.2020-01-21-08.03.log.html08:31
*** joxyuki has left #openstack-meeting08:33
*** links has quit IRC08:47
*** keiko-k has quit IRC08:48
*** rpittau|afk is now known as rpittau08:48
*** links has joined #openstack-meeting08:53
*** yamamoto has quit IRC08:53
*** yamamoto has joined #openstack-meeting09:14
*** brinzhang has joined #openstack-meeting09:16
*** yamamoto has quit IRC09:18
*** apetrich has joined #openstack-meeting09:18
*** brinzhang has quit IRC09:22
*** brinzhang has joined #openstack-meeting09:23
*** tetsuro has quit IRC09:56
*** links has quit IRC10:04
*** links has joined #openstack-meeting10:07
*** e0ne has joined #openstack-meeting10:19
*** priteau has quit IRC10:41
*** takahashi-tsc has quit IRC10:45
*** raildo has joined #openstack-meeting11:06
*** raildo_ has joined #openstack-meeting11:14
*** rpittau is now known as rpittau|bbl11:15
*** raildo has quit IRC11:17
*** dviroel has joined #openstack-meeting11:25
*** Lucas_Gray has joined #openstack-meeting11:40
*** rfolco has joined #openstack-meeting12:00
*** njohnston has joined #openstack-meeting12:00
*** priteau has joined #openstack-meeting12:07
*** ociuhandu has joined #openstack-meeting12:12
*** ociuhandu has quit IRC12:13
*** Lucas_Gray has quit IRC12:13
*** Lucas_Gray has joined #openstack-meeting12:17
*** hyunsikyang__ has joined #openstack-meeting12:18
*** hyunsikyang has quit IRC12:22
*** artom has joined #openstack-meeting12:24
*** rcernin has quit IRC12:28
*** Luzi has joined #openstack-meeting12:28
*** Liang__ has joined #openstack-meeting12:46
*** Liang__ is now known as LiangFang12:46
*** nicolasbock has joined #openstack-meeting12:49
*** ociuhandu has joined #openstack-meeting12:52
*** lseki has joined #openstack-meeting12:57
*** sfernand has joined #openstack-meeting12:57
*** ociuhandu has quit IRC13:01
*** priteau has quit IRC13:05
*** rpittau|bbl is now known as rpittau13:05
*** jamesmcarthur has joined #openstack-meeting13:18
*** zbr|drover has quit IRC13:18
*** zbr has joined #openstack-meeting13:19
ricolinself-healing and auto-scaling SIG meeting starts in 30 mins on #openstack-meeting feel free to join13:27
*** whoami-rajat_ has joined #openstack-meeting13:28
*** enriquetaso has joined #openstack-meeting13:29
*** jamesmcarthur has quit IRC13:33
*** Liang__ has joined #openstack-meeting13:37
*** LiangFang has quit IRC13:38
*** ykatabam has quit IRC13:41
*** Lucas_Gray has quit IRC13:42
*** jamesmcarthur has joined #openstack-meeting13:45
*** ociuhandu has joined #openstack-meeting13:47
*** dougsz has joined #openstack-meeting13:48
*** ociuhandu has quit IRC13:53
*** witek has joined #openstack-meeting13:58
ricolin#startmeeting self_healing14:01
openstackMeeting started Tue Jan 21 14:01:02 2020 UTC and is due to finish in 60 minutes.  The chair is ricolin. Information about MeetBot at http://wiki.debian.org/MeetBot.14:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:01
*** openstack changes topic to " (Meeting topic: self_healing)"14:01
openstackThe meeting name has been set to 'self_healing'14:01
ricolinwitek, o/14:01
witekhi ricolin14:01
ricolingot any suggest topics?:)14:02
witekI have tried to look into CI problems with Monasca auto-scaling, but didn't have too much time :)14:03
ricolin#link https://review.opendev.org/#/c/700989/14:03
witekthanks for debugging information provided in review14:03
ricolinthe way I put in https://review.opendev.org/#/c/700989/ is the same code we have for heat14:04
ricolinbut if we can have the actual reason will be perfect14:05
ricolinI mean the root cause14:05
*** cschwede has quit IRC14:06
ricolinWe also need to take action to update the SIG guideline for detail monasca+heat14:06
ricolinI assume most of info we need already in your repo (the one for workshop)14:07
witekbut https://review.opendev.org/#/c/700989/ alone doesn't help, as we can see in https://review.opendev.org/70088614:07
witekwhat kind of information would you like to see and where?14:07
ricolinwitek, yeah, according to the log, codes from python-monasca didn't even shows up14:08
ricolinin some reason, the dependency python-monascaclient patch is ignored14:09
ricolinFor guideline, I only have this story set up https://storyboard.openstack.org/#!/story/200704614:11
ricolinthe detail of guideline is what we can discuss14:11
ricolinAs long as people can read it and do it their own, will be a successful one:)14:12
witek:) is there some document for Telemetry available?14:13
witekwe could reuse and copy some of the material we used in the workshop14:15
witekI think till now we have just linked the reference to the GitHub repo14:15
ricolin+1 on reuse14:16
ricolinI think most people confused when there's to many detail but no docs around14:16
witekyes, let's keep it as simple as possible14:17
ricolinwitek, right, maybe like a workshop doc for a scenario?14:18
witekwhat do you mean?14:18
ricolinWe have collected some scenarios like heat+monasca, maybe we can do the guideline like heat+monasca with cpu loading detect14:20
witekOK, get it14:20
ricolinotherwise separate by project might be another option14:21
*** yamamoto has joined #openstack-meeting14:21
witekI'll try to write up something14:22
ricolinthat's great:)14:22
ricolinI will consult others and see if there's anyway to proper depends on python client patch14:23
witekthanks for helping with that!14:24
*** links has quit IRC14:24
ricolinthe doc and the CI job sounds like all we need at this stage14:24
ricolinif that python client issue resolved, I can send out the CI job in very short time14:25
ricolinI assume that's all we got to works on14:26
ricolinIf nothing else, let's close the meeting:)14:26
witeknothing from me14:27
ricolinOkay, thanks for join meeting witek:)14:27
ricolinlet's keep the action rolling:)14:27
witekthanks rico14:27
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"14:27
openstackMeeting ended Tue Jan 21 14:27:51 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:27
openstackMinutes:        http://eavesdrop.openstack.org/meetings/self_healing/2020/self_healing.2020-01-21-14.01.html14:27
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/self_healing/2020/self_healing.2020-01-21-14.01.txt14:27
openstackLog:            http://eavesdrop.openstack.org/meetings/self_healing/2020/self_healing.2020-01-21-14.01.log.html14:27
*** ayoung has quit IRC14:30
*** yamamoto has quit IRC14:36
ricolinMulti-Arch SIG meeting in 23 mins. feel free to join us:) https://etherpad.openstack.org/p/Multi-Arch-agenda14:37
*** dtroyer has joined #openstack-meeting14:38
*** Luzi has quit IRC14:46
*** sridharg has quit IRC14:47
*** jeremyfreudberg has joined #openstack-meeting14:51
*** yamamoto has joined #openstack-meeting14:59
*** yamamoto has quit IRC14:59
*** yamamoto has joined #openstack-meeting14:59
ricolin#startmeeting multi_arch15:00
openstackMeeting started Tue Jan 21 15:00:21 2020 UTC and is due to finish in 60 minutes.  The chair is ricolin. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: multi_arch)"15:00
openstackThe meeting name has been set to 'multi_arch'15:00
ricolin#topic roll call15:00
*** openstack changes topic to "roll call (Meeting topic: multi_arch)"15:00
ricolinjeremyfreudberg, o/15:00
ricolinjeremyfreudberg, how's going!15:01
*** shintaro has joined #openstack-meeting15:01
jeremyfreudbergricolin: all good, a little busy15:01
jeremyfreudbergbut busy can be good15:01
*** rh-jelabarre has joined #openstack-meeting15:02
jeremyfreudbergricolin: do we expect anyone else at this meeting time? or is everybody attending earlier?15:02
ricolinianw, kevinz, and mrda joined earlier15:03
rh-jelabarrewhat would the earlier time be in EST?  I think 3AM?15:03
ricolinI assume tonyb might be join this one, but not sure15:03
jeremyfreudbergrh-jelabarre: yes, i think 3am15:03
jeremyfreudbergricolin: i did not hear anything specific of tony, but maybe he will join now15:04
*** yamamoto has quit IRC15:04
ricolinrh-jelabarre, Tuesday at 0800 UTC is all I know:)15:04
jeremyfreudbergricolin: rh-jelabarre is in my time zone (and also work with me and tony)15:04
ricolinappear he is traveling15:04
jeremyfreudberghi tonyb15:04
ricolinthere you're!15:04
tonybSo it look slike the calendar is wrong as this is set to biweekly15:04
jeremyfreudbergtonyb: it is biweekely, so what's the problem?15:05
tonybOh nothing I'm just dense15:05
ricolin#chair ricolin tonyb15:06
openstackCurrent chairs: ricolin tonyb15:06
ricolin#topic adding items to agenda15:06
*** openstack changes topic to "adding items to agenda (Meeting topic: multi_arch)"15:06
ricolin#link https://etherpad.openstack.org/p/Multi-Arch-agenda15:06
rh-jelabarretonyb: I resemble that remark...15:06
ricolinAlso here's logs from earlier meeting http://eavesdrop.openstack.org/meetings/multi_arch/2020/multi_arch.2020-01-21-08.00.html15:07
ricolin#topic Follow on Repo and StoryBoard setup15:08
*** openstack changes topic to "Follow on Repo and StoryBoard setup (Meeting topic: multi_arch)"15:09
ricolinsome quick paste info15:09
ricolinOur Repo is there, but I haven't add the doc and zuul structure yet. Will do it this week.15:09
*** Liang__ is now known as LiangFang15:09
ricolin#link https://opendev.org/openstack/multi-arch-sig15:09
ricolinAccording to the plan we got from last meeting, documents will first to put in. I think that can started now, and send out once the repo structure is ready.15:09
ricolinAlso we have core reviewer list15:09
ricolin#link https://review.opendev.org/#/admin/groups/2079,members15:09
ricolinAnd I would like to suggest we add volunteers as core reviewers.15:10
ricolinAnyone volunteer? rh-jelabarre jeremyfreudberg :)15:10
tonybricolin: Works for me15:10
jeremyfreudbergricolin: feel free to add me as core15:10
ricolinjeremyfreudberg, will add you later, or tonyb can add you too as he is also in the list15:11
rh-jelabarrenot sure what a core reviewer would be doing15:11
tonybjeremyfreudberg: done15:11
ricolintonyb, that's quick15:11
tonybrh-jelabarre: probably not a good fit then15:11
tonybrh-jelabarre: We can help you learn "the openstack way"15:11
ricolinrh-jelabarre, we can definitely help you onboard:)15:12
ricolinOur StoryBoard is created,15:12
tonybricolin: you've been busy!15:13
ricolin#link https://storyboard.openstack.org/#!/project/openstack/multi-arch-sig15:13
ricolintonyb, yep!:)15:13
ricolinin the good way:)15:13
tonyb#thanks ricolin For his awesome and rapid setup of the multi-arch-SIG15:14
ricolinWe can start add tasks in Storyboard now15:14
ricolintonyb, :)15:14
ricolinSo if all members can put in their current work in progress task in this story will make others easier to track. We already have some collected WIP patch recorded in https://etherpad.openstack.org/p/Multi-arch which will make sense to move to StoryBoard too.15:15
ricolinjeremyfreudberg, also we need to add the document plan as task in Storyboard15:15
jeremyfreudbergricolin: yes15:16
jeremyfreudbergi'll try to find some time15:16
*** electrofelix has joined #openstack-meeting15:16
ricolincan I put an action to have you add in document tasks in storyboard?15:17
ricolin#action jeremyfreudberg create docs plan as tasks in Storyboard15:17
*** LiangFang has quit IRC15:18
ricolintonyb, I already added an action to ask everyone to put their current tasks (anything that related to Multi-arch) in storyboard if possible15:19
tonybricolin: Sounds good15:19
ricolintonyb, cool!15:19
tonybricolin: It'll be good to get that written down aso we can find ways forward15:20
*** dougsz has left #openstack-meeting15:20
ricolintonyb, I currently only aware of tasks that list in https://etherpad.openstack.org/p/Multi-arch15:20
ricolintonyb, you mean we can add those in?15:21
tonybricolin: There are some ppc64le things I'm working on some of which will have arm64 work also so getting that written down and then looking for ways forward15:22
ricolintonyb, can you help to put those in Storyboard15:23
*** witek has left #openstack-meeting15:23
tonybricolin: For sure15:23
*** shintaro has quit IRC15:24
ricolinIs ppc64le generally supported in OpenStack services?15:24
ricolinI mean core components15:25
tonybricolin: Yes a lot of the work is in deployment tools15:25
*** jamesmcarthur has quit IRC15:25
tonybricolin: It's "reasaonably" easy to build an all in one OpenStack (keystone, glance, swift, nova)15:26
ricolinI learn from ianw that devstack, *might* still not working with multi-arch, that's where I wonder ppc64le will need to check as well15:26
tonybricolin: I'm like 95% certain the same can be said for arm6415:26
ricolintonyb, good to learn that:)15:27
tonybricolin: Hmm I'll check but it *used* to work, I admit I don't use it (devstack) alot15:27
tonybricolin: I15:27
tonybll add it do the Action Items list15:27
ricolinianw also take action to create a general job so we can test with it15:27
ricolintonyb, great to have you can join that action:)15:28
ricolinyou can make the action your self:)15:28
tonybricolin: Will do15:28
ricolinmove on:)15:29
ricolinI think you all aware of the new irc channel :)15:29
*** ociuhandu has joined #openstack-meeting15:30
ricolinfor who don't know, please join that channel:)15:30
rh-jelabarreI am now...15:30
ricolinrh-jelabarre, join!:)15:31
*** electrofelix has quit IRC15:31
ricolinThat's all I got for SIG infra setup15:31
tonybricolin: Thanks again!15:32
ricolintonyb rh-jelabarre jeremyfreudberg , any other infra you think we need to add?15:32
tonybricolin: I want to see where we're at with arm and zuul etc but I don't know that's "setup"15:33
ricolinThat's something we need to proper tracking by storyboard task I assume15:34
ricolinyeah, it's setup for sure:)15:34
ricolinI will working on puting init docs and docs test in repo this week15:35
*** jamesmcarthur has joined #openstack-meeting15:35
*** electrofelix has joined #openstack-meeting15:36
ricolinif we plan for some new zuul job, it can defined under our repo too15:36
*** electrofelix has quit IRC15:36
*** electrofelix has joined #openstack-meeting15:36
tonybricolin: Okay cool.15:38
ricolintonyb, besides adding to storyboard, do you think it's helpful to have an repo on current arm status?15:38
tonybricolin: Yeah I think a report would be really good15:39
tonybricolin: we can probably make status.rst or sumilar in the repo to collect that information15:39
ricolindo you have anything in you mind that the status.rst might looks like?15:40
tonybricolin: Just a high level summary of what we have, what's known to work, what's known to break15:41
tonybricolin: but I don't have a strong idea15:42
*** Lucas_Gray has joined #openstack-meeting15:42
ricolintonyb, got it, let's record it with an action so we don't forgot about that idea:)15:42
tonyb#action tonyb to create a placeholder "status.rst" for discussion15:43
ricolinmove on15:44
ricolin#topic Follow on CI Resources15:44
*** openstack changes topic to "Follow on CI Resources (Meeting topic: multi_arch)"15:44
ricolinFor what I know mnaser mentioned that there might be some ARM resource Vexxhost can donate to community.15:44
ricolinAlso Huawei mentioned that they can donate ARM resources too. I will keep tracking with them and see if anything scheduled.15:44
ricolinianw mentioned from eariler meeting that the second group of node from Linaro is ready https://review.opendev.org/#/c/70353515:45
ricolinso we now have two groups of node15:45
*** lpetrut has quit IRC15:46
ricolinmaybe some other provider can help to donates too15:46
tonybricolin: ideally yes15:47
ricolinwill try to contact more provider to see if they're interested15:48
ricolin#topic Follow on documentation15:48
*** openstack changes topic to "Follow on documentation (Meeting topic: multi_arch)"15:48
ricolinjeremyfreudberg, do you have anything to update on the docs?15:49
jeremyfreudbergricolin: i do not have much update today. as you suggested i will make some stories for what kind of docs we want.15:50
* ricolin didn't have any to update for now except will try to build the init doc infra in repo15:50
ricolinjeremyfreudberg, thanks!15:50
ricolin#topic open discussion15:50
*** openstack changes topic to "open discussion (Meeting topic: multi_arch)"15:50
ricolintonyb, jeremyfreudberg rh-jelabarre anything we should discuss today?15:51
*** ociuhandu has quit IRC15:51
jeremyfreudbergnot from me15:52
rh-jelabarrenot at present15:52
tonybricolin: I don't have anything15:52
ricolinwe already have a very productive starts:)15:53
ricolinlet's see what we can gather on Storyboard:)15:53
tonybricolin: Cool beans15:54
ricolinlet's end this meeting, thanks all for join:)15:55
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:55
openstackMeeting ended Tue Jan 21 15:55:05 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:55
openstackMinutes:        http://eavesdrop.openstack.org/meetings/multi_arch/2020/multi_arch.2020-01-21-15.00.html15:55
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/multi_arch/2020/multi_arch.2020-01-21-15.00.txt15:55
openstackLog:            http://eavesdrop.openstack.org/meetings/multi_arch/2020/multi_arch.2020-01-21-15.00.log.html15:55
*** jeremyfreudberg has left #openstack-meeting15:55
slaweq#startmeeting neutron_ci16:00
openstackMeeting started Tue Jan 21 16:00:16 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"16:00
openstackThe meeting name has been set to 'neutron_ci'16:00
slaweqI think we have our usuall attendees so lets start16:02
slaweq#topic Actions from previous meetings16:02
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"16:02
slaweqfirst one:16:02
slaweqslaweq to remove networking-midonet and tripleo based jobs from Neutron check queue16:02
slaweqI did patches:16:02
slaweq    Midonet: https://review.opendev.org/#/c/703282/116:02
slaweq    TripleO: https://review.opendev.org/#/c/703283/16:03
slaweqI sent separate patches for each job because it will be easier to revert if we will want to bring those jobs back16:03
ralonsoh+2 to both16:03
slaweqthx ralonsoh16:03
slaweqand the next one:16:04
slaweqbcafarel to send cherry-pick of https://review.opendev.org/#/c/680001/ to stable/stein to fix functional tests failure16:04
bcafarelsent and merged! https://review.opendev.org/#/c/702603/16:05
slaweqthx bcafarel16:05
njohnstonFor midonet, I am not sure it is still broken on py3... look at this midonet change, to drop py27, the py3 jobs all work well https://review.opendev.org/#/c/701210/16:05
slaweqnjohnston: yes, but there is no any scenario job there16:05
ralonsohnjohnston, good catch16:05
slaweqonly UT16:05
slaweqproblem is that midonet's CI is using centos and it's not really working on python 3 IIRC16:06
njohnstonOK, I'll look into it and probably +2+W16:07
slaweqthx njohnston16:07
slaweqok, I think we can move on16:08
njohnstonmay need to wait a little, recentl failures fail with the "pip._internal.distributions.source import SourceDistribution\nImportError: cannot import name SourceDistribution\n" error that is now getting cleared up16:08
slaweqnjohnston: is that failure from midonet job or from where?16:08
njohnstonI saw that in networking-midonet-tempest-aio-ml2-centos-7 results, yes16:09
slaweqahh, ok16:10
njohnstonFOr those who have not seen, it was discussed on openstack-discuss16:10
njohnston#link http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012117.html16:10
slaweqif You have some time to play with it to fix it, that would be great, if not - I would still be for removing it for now until it will be fixed16:10
njohnstonslaweq: Yeah, I'll take a quick look, if it gets outside the timebox then I'll approve the change and we can move on :-)16:11
slaweqnjohnston: thx a lot16:11
slaweqok, next topic than?16:11
slaweq#topic Stadium projects16:13
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"16:13
slaweqnjohnston: any updates about dropping py2?16:14
slaweqYou wasn't on yesterday's team meeting so maybe You have something new for today :)16:14
njohnstonso midonet merged it's change16:14
njohnstonwhich I think means the only one left for dropping py27 is neutron-fwaas16:15
bcafarelyeah open reviews list is quite short now16:15
njohnston#link https://review.opendev.org/#/c/688278/16:15
njohnstonif neutron-fwaas is going to be retired then I believe there's no sense in doing the work of fixing it16:16
njohnstonso in short: mission complete!16:16
slaweqnjohnston: I don't think it's going to be retired this cycle16:16
slaweqmaybe in next one16:16
slaweqso IMHO we should merge this patch, but it shouldn't be too much work IMO16:17
bcafarelneutron-fwaas change was in already good shape last time I checked, should be able to merge it soon16:17
njohnstonok, I'll start pushing it16:17
slaweqthx njohnston16:17
slaweqyes, amotoki found some small issues in this patch, except that IMO it's good to go16:18
njohnstonI'll push the fixes after the meeting16:18
slaweqand tempest job failure wasn't related to dropping py2 for sure16:18
slaweqthx njohnston16:18
slaweqok, I think that's all about stadium projects16:19
slaweqso we can move on16:19
slaweq#topic Grafana16:19
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"16:19
slaweq#link http://grafana.openstack.org/dashboard/db/neutron-failure-rate16:19
njohnstonlooking pretty good16:21
ralonsohbut during the last week, the number of rechecks has increased16:22
ralonsohthe gate is almost "closed"16:22
ralonsohFT is failing a lot because ovsdb timeouts16:22
*** ociuhandu has joined #openstack-meeting16:23
slaweqyes, grafana looks pretty good but I also have impression that it's not working very well for some of the jobs16:23
bcafarelyup, same feeling here16:23
slaweqalso my script shown me that last week we had 1 recheck in average to merge patch16:24
slaweqbut I think it's because we didn't merge many patches and some of them were with e.g. docs only, so less jobs are run on them16:24
ralonsohand, for example, the gate now is dropping all jobs16:25
slaweqok, so lets maybe talk about some specific jobs and issues from last week16:25
slaweqralonsoh: why?16:25
slaweqany specific reason?16:25
slaweqor just random failures?16:25
ralonsohstill reviewing it16:26
slaweqok, thx16:26
bcafarelthe recent pip issue?16:26
slaweqahh, yes I saw some email about it today16:26
ralonsohseems that the problem is fixed and released16:27
ralonsohpip 20.0.116:27
*** ociuhandu has quit IRC16:28
slaweqyes, this should be better now16:28
slaweqok, lets move on to the specific jobs now16:29
slaweq#topic fullstack/functional16:29
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"16:29
slaweqI have couple of issues for today16:29
slaweqfirst one is, already mentioned by ralonsoh ovsdbapp timeouts, like e.g.:16:29
slaweq    https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_559/703299/1/gate/neutron-functional/559c0ed/testr_results.html16:29
slaweq    (fullstack) https://1a798a15650a97d81a82-17406e3478c64e603d8ff3ea0aac16c8.ssl.cf1.rackcdn.com/703366/1/check/neutron-fullstack/59e1877/testr_results.html16:29
slaweq    https://0bad6b662fac8347dc41-be430d2f919a8698d2e96141ed3ac146.ssl.cf5.rackcdn.com/687922/15/gate/neutron-functional/b6769d6/testr_results.html16:29
slaweqI think there was old bug reported for that even16:30
ralonsoh#link https://bugs.launchpad.net/neutron/+bug/180264016:30
openstackLaunchpad bug 1802640 in neutron "TimeoutException: Commands [<ovsdbapp.schema.open_vswitch.commands.SetFailModeCommand in q-agt gate failure" [Medium,Confirmed]16:30
slaweqyes, this one16:30
slaweqthx ralonsoh16:30
ralonsohI would like to submit a patch to increase the ovsdbapp log16:30
ralonsohand then, maybe, talk to Terry Wilson to do something there, in the command commit context16:31
slaweqif that may help to debug this issue, I'm all for this :)16:31
*** macz has joined #openstack-meeting16:31
bcafarel+1 if that gives a better image of what it is actually spending time on16:31
bcafarelI did not check the specific failure, but functional has been grumpier recently on stein/trein too - may be similar issue16:32
bcafarel(where rechecks are ping pong between functional and grenade timeouts)16:32
slaweqok, so ralonsoh will increase log level for ovsdbapp in those jobs and we will see then16:33
slaweq#action ralonsoh to increase log level for ovsdbapp in fullstack/functional jobs16:34
slaweqthx ralonsoh for taking care of it16:34
slaweqok, next one which I saw this week was Issue with EVENT OVSNeutronAgentOSKenApp->ofctl_service GetDatapathRequest16:35
*** eharney has quit IRC16:35
slaweqlike e.g. https://f26da45659020db1220c-76fc92e5e7c4e5a091c792a95503ad1d.ssl.cf5.rackcdn.com/701565/5/check/neutron-functional/4db8d3f/controller/logs/dsvm-functional-logs/neutron.tests.functional.agent.l2.extensions.test_ovs_agent_qos_extension.TestOVSAgentQosExtension.test_policy_rule_delete_egress_.txt16:35
slaweqand I remember that I already saw such issue in the past few times16:35
*** ociuhandu has joined #openstack-meeting16:36
slaweqand it failed on setup_physical_bridges because it couldn't get dp_id16:37
slaweqis there anyone who wants to check that?16:37
ralonsohI can take a look at it16:38
ralonsohis there a bug reported?16:38
slaweqralonsoh: nope, but I will open one for it today16:38
slaweq#action slaweq to open bug for issue with get_dp_id in os_ken16:38
slaweqand I will send it to You16:39
*** gyee has joined #openstack-meeting16:39
slaweqok, lets move on to fullstack job than16:39
slaweqfirst issue there:16:39
slaweqI saw it few weeks ago also16:39
slaweqand according to http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20QoS%20register%20not%20found%20with%20queue-num%5C%22 it is happening from time to time recently16:40
ralonsohyes, it's very "delicate" the min-qos feature16:40
ralonsohif you have only one agent, ewverything is ok16:40
ralonsohbut when we are testing, we are introducing several qos/queue configurations16:41
ralonsohthe agent extension is not ready for this16:41
ralonsohbut we fake this in the tests16:41
*** mattw4 has joined #openstack-meeting16:41
ralonsohbut of course, we can have these kind of situations...16:41
ralonsohbut, since the last patch, is much more stable16:42
ralonsohand, btw, this is still pending16:42
slaweqahh, ok16:42
slaweqso this patch should helps with this issue, right?16:43
ralonsohI hope so16:43
ralonsoh(patch you are "my last hope")16:43
slaweqok, next issue with fullstack is16:43
slaweq    Failure on cleanup phase: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_564/702250/6/gate/neutron-fullstack/564244f/testr_results.html16:43
slaweq    Do we really need cleanup in fullstack tests?16:43
slaweqI started thinking today that we maybe don't need cleanup phase in fullstack tests16:44
slaweqfor each test we are spawning new neutron-server, agents and everything else16:44
slaweqand it's always only for this one test16:44
slaweqso what's the point to delete routers, ports, networks, etc. at the end?16:44
slaweqIMO it's just waste of time16:45
slaweqor am I missing something here?16:45
njohnstonto make sure the delete functionality doesn't have bugs, yes?16:45
slaweqnjohnston: sure, but we are testing delete functionality in scenario and api tests also, no?16:45
slaweqfullstack is more for kind of whitebox testing where You can check if things are actually configured on host properly16:46
*** bnemec has quit IRC16:48
njohnstonOK, you could make that argument.  We could spin a change to remove those cleanups and make sure nothing bag happens.16:48
slaweqso I will send such patch to also see how much time we can safe on that job if we will remove cleanups from it16:48
slaweqif it's not too much, maybe we can drop this patch and stay with it as it is now16:49
ralonsohmy only concern are the L1 elements in the host (taps, devices, bridges, etc)16:49
slaweqok for You?16:49
slaweqralonsoh: sure, I'm not talking about things like that16:49
ralonsohwe can try it16:49
slaweqI just want to skip cleaning neutron resources16:49
bcafarelif you run fullstack in a loop and do no start seeing OOM errors it is probably good ;)16:50
slaweqbcafarel: good test indeed :)16:50
slaweq#action slaweq to try to skip cleaning up neutron resources in fullstack job16:51
slaweqok, lets move on quickly to the next topic as we are running out of time16:51
slaweq#topic Tempest/Scenario16:51
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"16:51
njohnstonjust make sure that we also accomodate developers running fullstack on their personal laptops16:51
slaweqnjohnston: sure, I will check if this will be fine in such case16:52
slaweqok, speaking about tempest jobs I saw again this week issue with paramiko:16:52
ralonsoh#link https://review.opendev.org/#/c/702903/16:53
slaweqbasically there is error like "paramiko.ssh_exception.SSHException: No existing session"16:53
ralonsohthat's why I pushed this16:53
ralonsohahhh no, this is another error16:54
ralonsohbut that was solved last week16:54
slaweqI don't think it was solved last week16:54
slaweqthere was another, similar issue but with error like "NoneType object has no attribute session" or something like that16:55
ralonsohpfffff ok then16:55
slaweqthere is bug reported for this new issue, I need to find it16:55
slaweqparamiko.ssh_exception.SSHException: No existing session16:56
openstackLaunchpad bug 1858642 in neutron "paramiko.ssh_exception.NoValidConnectionsError error cause dvr scenario jobs failing" [High,Confirmed]16:56
slaweqif anyone whould have some time to look into it, that would be great16:56
slaweqand it's not only in dvr jobs16:56
slaweqand that's all from me for today16:57
ralonsohI can take a look, but this problem seems to be in the zuul.test.base16:57
slaweqzuul.test.base? what is it?16:57
ralonsohnothing, wrong search16:58
slaweqahh, ok16:58
slaweqok, lets move on quickly16:58
slaweq2 more things16:58
slaweq1. thx ralonsoh for fix mariadb periodic job - it's working fine now16:59
bcafarelthanks ralonsoh!16:59
slaweq2. I would like to ask You about change time of this meeting16:59
slaweqwhat do You say about moving it to wednesday for 2pm utc, just after L3 meeting?16:59
njohnstonthat works fine for me17:00
slaweqas today at same time there is tc call in redhat every 3 weeks and I would like to attend it sometimes17:00
ralonsohyou mean 3pm UTC17:00
slaweqsorry, yes17:00
slaweq2pm is L3 meeting17:00
ralonsohok for me17:00
slaweqI will propose patch and add all of You as reviewers than17:00
slaweqok, that's all for today17:01
bcafareland mail maybe also for interested folks17:01
slaweqthx for attending17:01
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"17:01
openstackMeeting ended Tue Jan 21 17:01:12 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-01-21-16.00.html17:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-01-21-16.00.txt17:01
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-01-21-16.00.log.html17:01
slaweqbcafarel: sure, I will send email also17:01
*** tesseract has quit IRC17:01
*** rpittau is now known as rpittau|afk17:04
*** ricolin has quit IRC17:04
*** ricolin has joined #openstack-meeting17:05
*** raildo_ has quit IRC17:12
*** rfolco is now known as rfolco|brb17:16
*** e0ne has quit IRC17:24
*** eharney has joined #openstack-meeting17:46
*** jamesmcarthur has quit IRC18:00
*** ayoung has joined #openstack-meeting18:02
*** macz has quit IRC18:03
*** ociuhandu_ has joined #openstack-meeting18:13
*** Lucas_Gray has quit IRC18:15
*** ociuhandu has quit IRC18:17
*** ociuhandu_ has quit IRC18:18
*** slaweq_ has joined #openstack-meeting18:26
*** slaweq has quit IRC18:27
*** ralonsoh has quit IRC18:31
*** electrofelix has quit IRC18:34
*** enriquetaso has quit IRC18:36
clarkbwe'll start the infra meeting shortly18:59
clarkb#startmeeting infra19:01
openstackMeeting started Tue Jan 21 19:01:12 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.19:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:01
*** openstack changes topic to " (Meeting topic: infra)"19:01
openstackThe meeting name has been set to 'infra'19:01
clarkb#link http://lists.openstack.org/pipermail/openstack-infra/2020-January/006577.html Our Agenda19:01
clarkb#topic Announcements19:01
*** openstack changes topic to "Announcements (Meeting topic: infra)"19:01
clarkbI did not have any announcements19:01
clarkb#topic Actions from last meeting19:02
*** openstack changes topic to "Actions from last meeting (Meeting topic: infra)"19:02
clarkb#link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-01-14-19.01.txt minutes from last meeting19:02
clarkbI had an action item to start making the opendev governance changes official. This has been done, but lets talk about it during our OpenDev topic19:03
clarkb#topic Priority Efforts19:03
*** openstack changes topic to "Priority Efforts (Meeting topic: infra)"19:03
clarkb#topic OpenDev19:03
*** openstack changes topic to "OpenDev (Meeting topic: infra)"19:03
clarkbwe can jump into that immediatly :)19:03
clarkbThere are two related changes to make the OpenDev governance changes official. The first removes OpenDev from OpenStack Governance. The second creates a new self bootstrapped governance system for OpenDev19:04
clarkb#link https://review.opendev.org/#/c/703134/ Split OpenDev out of OpenStack Governance19:04
clarkb#link https://review.opendev.org/#/c/703488/ Update OpenDev docs with new Governance19:04
clarkbBoth have received a fair amount of review, thank you allfor that. If you haven't yet had a chance to review them your feedback is very welcome19:04
clarkbAre there any questions or concerns at this point that I (or others) can help with?19:05
fungiseems like a fairly straightforward evolution for me19:06
clarkbOne thing to point out is that I think the first change needs to merge before the second19:06
clarkbotherwise there is a governance conflict19:06
fungii guess the main question i have is whether there will also be an openstack-infra governance defined after the opendev governance gets published in the place that used to live19:06
clarkbfungi: ajaeger brought that up too, and I don't htink it needs anything beyond the default openstack system19:06
fungialso we could consider openstack-infra becoming a sig, but one step at a time19:07
clarkbfungi: the repos/tools/systems that remain in openstack are fairly self contained and won't require subteam organization I don't think19:07
corvusyeah, if it end up needing more formal organization, maybe a sig19:08
fungiwell, a sig would be *less* formal organization really19:09
corvusi mean more formal than nothing :)19:09
fungi(less organization than an openstack project team anyway)19:09
fungioh, sure ;)19:09
corvusoh, yes.19:09
corvuson the question of sig vs subteam, sig sounds right, due to the value of cross-subteam collaboration.19:10
corvus(and relative dearth of release deliverables)19:10
*** Shrews has joined #openstack-meeting19:11
clarkbOn the technical side of things jrosser noticed that some jobs running between 1400 and 1500 UTC today had git clone errors. I've suggested that the job instead rely on zuul managed repos (as they come from caches and allow for cross repo testing). I also discovered that about half the total connections to gitea04 (which was the unhappy node at the time) came from huawei IPs19:12
clarkbthose IPs were all in the same /22 subnet19:13
clarkbit looks like a CI system in that way19:13
*** dustinc|PTO is now known as dustinc19:13
clarkbI think we have ~3 options. A) bigger gitea nodes with more memory to avoid swapping and OOMs B) work with huawei to not DDoS us 3) try least connections load balance method again (this is unlikely to be stable)19:13
fungiwe also had a massive load spike on lists.o.o which looked like they could be a pipermail python process pileup19:14
fungibut i didn't find a smoking gun19:14
clarkbat this point I think its motsly a monitor situation. If it continues and or gets worse then we will need to sort out a plan19:14
clarkbbut maybe this is a one off and we'll be ok19:14
corvusclarkb: ++ let's get 2 or 3 data points19:14
fungiand... same recommendation on the lists.o.o front for now19:15
clarkbAnything else opendev related?19:15
funginothing springs to mind19:16
clarkb#topic Update Config Management19:16
*** openstack changes topic to "Update Config Management (Meeting topic: infra)"19:16
clarkbreview-dev is running without puppet at this point \o/19:16
clarkbwe did discover a couple gaps that need to be addressed though. The first is in disabling the old init script for gerrit once converted to podman. The other is configuring a webserver to front gerrit19:17
clarkbI think for the first one we ma yjust make that part of our migration document but the second will need updated to the ansible/container tooling19:17
fungiyeah, extra ansible to delete an initscript seems overkill19:18
clarkbI don't think the webserver change(s) have been written yet and I don't think we have mordred here today19:18
fungior possibly he's just waking up19:18
clarkbI guess keep an eye out for those changes as they are the next step in getting gerrit switched over19:19
clarkbare there any other config management related changes to call out?19:20
*** jamesmcarthur has joined #openstack-meeting19:20
clarkbSounds like no. Let's move on19:21
clarkb#topic General Topics19:21
*** openstack changes topic to "General Topics (Meeting topic: infra)"19:21
clarkbianw: do you want to start with static.opendev.org or arm64 cloud?19:21
ianwi haven't made too much progress on static.o.o19:22
clarkblooks like the changes have landed at least19:22
ianwyep, i can loop back on that and i think starting a new server is probably the next step19:23
ianwthen governance and some others should be publishing to afs and we can move them to be served from there19:23
clarkbrelated to this topic, we threw out last week that maybe having a day or two to focus on getting rid of the last trusty servers would be worthwhile19:23
clarkbsuggestion is to use the 27th and 28th for that (28th and 29th australia days)19:24
clarkbI'm still on board for that and will plan to focus on this work on those days19:24
clarkbeveryone else is more than welcome to join in :)19:24
clarkbianw: we must be getting close to needing to set up dns records as well19:25
clarkbianw: does that need to happen before the server starts (so that LE can configure things properly)19:25
ianwclarkb: umm, yes probably, i will loop back on that19:26
fungii can try to set aside those two days to make progress on wiki.o.o configuration management in hopes we can migrate to a replaceable xenial server with actual puppet-applied configuration19:26
fungi(i think wiki-dev is close-ish)19:26
clarkbsounds good, thanks19:26
clarkbNext up is a new arm64 cloud19:27
clarkb#link https://review.opendev.org/#/q/topic:linaro-us+status:open Changes to deploy new arm64 cloud19:27
clarkbianw: I left a minor -1 on the clouds.yaml changes19:27
corvusooh neat -- we'll have 2 now?19:27
clarkbcorvus: yup. This one looks to be quite a bit larger than the existing one as well19:28
ianwclarkb: yep, will do comments on that small stack asap19:28
clarkb44 test instances (if using 8vcpu per instance)19:28
ianwif we can get that in, i can look at bringing up a mirror, and then ... it should work? :)19:28
clarkbianw: ya I think we are less concerned with performance given it is arm and simply different19:29
clarkbpeople running jobs thee will have to accept it may be slower19:29
ianwkevinz did mention it was ipv6 only which is new territory for arm64 cloud so may be wrinkles there19:29
clarkbthat is a good point.19:30
fungiwell, in #openstack-infra he just said there were a lot fewer ipv4 addresses than quota for test nodes, but yes in practice that means ipv6 global with v4 egress via overload nat19:30
fungi(port address translation)19:31
clarkband dedicated ipv4 for the mirror19:31
fricklerthis likely also means that we should setup some special routing for the v4 mirror like FN has19:32
clarkbfrickler: thats a good point ,we should double check the traceroute for test nodes to the mirror19:32
clarkbin case the ipv6 setup is similar and bottlenecks through the router19:32
fungiyeah, or rather we end up overrunning the nat's port mapping pool with local connections to the mirror server19:33
fungi(remembering that the tcp/udp port range is a mere 65k, so that's not a lot of parallel connections in reality)19:34
clarkbAnything else on the topic of a new arm64 cloud?19:35
ianwwe did have a mutli-arch meeting19:36
ianw#link https://storyboard.openstack.org/#!/project/openstack/multi-arch-sig19:36
ianwplan is to put interesting things to track @ ^19:36
fungii hope that goes somewhere19:38
clarkbI guess that sig will be taking advantage of the new set of resources to do more testing of openstack on more than just x86?19:38
fungiit's apparently in tonyb's interest to have better support for powerpc now19:39
fungiso maybe even more than just x86 and aarch6419:39
ianw:) yep, i think we start with a build-it-and-they-will-come approach19:40
fungiit's worked so far19:40
clarkbAlright anything else?19:41
ianwnot from me19:42
clarkb#topic Open Discussion19:44
*** openstack changes topic to "Open Discussion (Meeting topic: infra)"19:44
clarkbI'll give it a few minutes for other topics19:44
*** jamesmcarthur has quit IRC19:46
clarkbnot hearing any other business. I'll go ahead and call the meeting19:47
clarkbThank you everyone!19:47
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"19:47
openstackMeeting ended Tue Jan 21 19:47:31 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:47
openstackMinutes:        http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-01-21-19.01.html19:47
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-01-21-19.01.txt19:47
openstackLog:            http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-01-21-19.01.log.html19:47
corvusthanks clarkb!19:47
*** jamesmcarthur has joined #openstack-meeting19:50
*** nicolasbock has quit IRC19:57
*** jamesmcarthur has quit IRC20:01
*** jamesmcarthur has joined #openstack-meeting20:01
*** jamesmcarthur has quit IRC20:08
*** yamamoto has joined #openstack-meeting20:11
*** jamesmcarthur has joined #openstack-meeting20:14
*** yamamoto has quit IRC20:16
*** Lucas_Gray has joined #openstack-meeting20:26
*** eharney has quit IRC20:27
*** ykatabam has joined #openstack-meeting20:41
*** rcernin has joined #openstack-meeting20:42
*** rbudden has joined #openstack-meeting20:50
*** oneswig has joined #openstack-meeting20:58
*** whoami-rajat_ has quit IRC20:58
*** Shrews has left #openstack-meeting20:58
*** martial has joined #openstack-meeting20:58
*** jamesmcarthur has quit IRC20:58
*** trandles has joined #openstack-meeting20:59
*** jmlowe has joined #openstack-meeting21:00
oneswig#startmeeting scientific-sig21:00
openstackMeeting started Tue Jan 21 21:00:18 2020 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"21:00
openstackThe meeting name has been set to 'scientific_sig'21:00
oneswigup up and away!21:00
oneswig#topic who's here21:00
*** openstack changes topic to "who's here (Meeting topic: scientific-sig)"21:00
trandlesHi all21:00
oneswighi all21:00
martialthat's a topic!21:01
oneswigThe new informal SIG.  I'm not even wearing a tie tonight :-)21:01
oneswigOK, I forgot to post an agenda on the wiki, apologies.21:02
oneswig#chair martial21:02
openstackCurrent chairs: martial oneswig21:02
oneswig#topic Conferences and CFPs for 202021:03
*** openstack changes topic to "Conferences and CFPs for 2020 (Meeting topic: scientific-sig)"21:03
oneswigSo what's coming up?21:03
oneswigIf you're in London in early March, this free 1-day conference at the Francis Crick Institute is excellent: https://cloud.ac.uk/21:04
martialGTC in March21:04
martialOpenStack Summit in June (CFP coming out soon I was told)21:05
martialPEARC ?21:05
oneswigmartial: where and when for GTC?  You going?21:05
martialGTC, LA, March21:05
martialplanning to21:05
jmloweISC, SuperCompCloud, working on the CFP and program committee (nominations taken) https://sites.google.com/view/supercompcloud21:06
oneswigOnce you're done with GTC, have some downtime lakeside in Ticino, Switzerland: https://www.hpcadvisorycouncil.com/events/2020/swiss-workshop/21:06
jmlowePEARC in Portland at the end of July21:06
martialSC20 in Atlanta ... November?21:07
oneswigjmlowe: you planning a presentation to submit for PEARC this year?21:07
jmlowePossibly, definitely if JS2 gets funded21:07
oneswigI haven't checked yet when the deadline is for OpenStack Vancouver - anyone know?21:10
trandlesCFP hasn't even opened has it?21:11
martialdon't think it is out yet21:11
martialIldiko mentioned it should be soon21:11
trandlesThere's nothing but dates and location on the normal summit page.21:12
oneswigIt's pretty minimal right now - https://www.openstack.org/summit/vancouver-2020/21:12
trandlesOh, looks like an eventbrite registration page is open now. Totally new format by the look of it?21:13
oneswigIt actually looks great - quite a grassroots feel to the way it is presented.21:15
martiallooks very different to what we are used to indeed21:16
trandlesYeah, much less like a conference and more like a big integrated PTG?21:16
martialsure does21:17
oneswigThis plenary opener - "Short kickoff with all attendees to set the goals for the day or discuss the outcomes of the previous day." - sounds a lot more "intimate", for sure.21:17
rbuddenindeed, it looks like the format has drastically changed21:18
martialI think there will still be an upstream academy21:18
trandlesWonder how the various SIGs fit into the structure? Will they still support SIG meetings as part of the program?21:19
trandlesOr maybe we'll have to self-organize21:19
oneswigtrandles: hopefully in a different way than in Shanghai, where it didn't work out at all21:19
martialnot sure yet, once the call for participation is open we can ask21:20
martialyes that sound like it was less what was expected at the time21:20
trandles"PTG: Afternoon working sessions for project teams and SIGs to continue the morning’s discussions."21:20
martialwait it only one day?21:20
trandles4 days21:21
martialokay 8-1121:21
martialmakes more sense21:21
jmloweThey were saying it would be different, I wasn't sure I believed them until now21:21
trandlesIt's 4 days and they call out 4 focus areas, I wonder if each day will be dedicated to each area21:22
oneswigI like it.  I think we can get stuff done here.21:22
trandlesgah - "if each day will be dedicated to one area"21:22
*** armax has joined #openstack-meeting21:22
trandlesIt's certainly a lot cheaper than past summits!21:22
martialwell I look forward to learning new things21:23
martialalthough the question remains about "is it a long PTG"21:23
martialanything else?21:24
oneswigdefinitely scope for an evening social!21:24
oneswigOK, move on?21:24
jmloweI appreciate social events closer to sea level21:25
oneswig#topic large-scale SIG data21:25
*** openstack changes topic to "large-scale SIG data (Meeting topic: scientific-sig)"21:25
oneswigI've been taking part in the large-scale SIG meetings21:25
martialexpand on the topic please?21:26
oneswigThey are looking for operator data on pain points21:26
martialoh right, that is a lot of our SIG users :)21:26
oneswig#link mailing list post http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011820.html21:26
oneswig#link etherpad for information on scaling https://etherpad.openstack.org/p/large-scale-sig-documentation21:27
oneswigCould actually become a handy resource that one.21:27
jmlowetldr; did they define large-scale?21:27
*** diablo_rojo has quit IRC21:27
*** diablo_rojo has joined #openstack-meeting21:28
oneswigNot precisely.  But approaching 1000 hypervisors is definitely in it.  500+ will be showing a lot of the early signs of scaling trouble21:28
*** eharney has joined #openstack-meeting21:28
oneswigOur contribution so far has been around "golden signals" for scaling problems - rates, latencies, errors, and all that.21:30
oneswigto which end, the most useful thing I've seen to date is telemetry from HAProxy21:30
trandlesHAProxy at that scale? The OSA documents explicitly suggest you don't want to use haproxy in production at all.21:31
oneswigIt will give response latencies by endpoint for example.  Very cool.  And a graph of 500 response codes.  Highly useful.21:32
oneswigtrandles: that's surprising.  We build upon it as standard.  Multiple in a deploy, sometimes.21:32
oneswigAs someone who appreciates a tight piece of C code, what's the objection, and indeed the alternative?21:33
jmloweum what!? no HAProxy, everything I have would fall over without it21:33
trandlesLooking for the reference now, but IIRC, it says something like "HAProxy is appropriate for testing but you should use a real hardware director in production"21:33
martialwhat are the alternatives?21:34
jmlowesure, I've got a spare couple of million for a enterprise load balancer21:34
oneswigtrandles: did you heed this guidance? :-)21:35
*** jamesmcarthur has joined #openstack-meeting21:35
jmloweI certainly didn't21:35
trandlesoneswig, I can't even get a deployment to finish. I spend most of my time googling to figure out what configuration variable I need to set that is left out of the OSA guide.21:36
oneswigI don't think I've seen our OpenStack APIs generate enough traffic or connections to be a concern for it.  Famous last words perhaps.21:36
trandlesMaybe Vancouver should have one topic: decent documentation21:36
oneswigtrandles: feel your pain.  My own notes are the same.  I'm like, "thanks, former self"...21:36
trandlesoneswig, I have started a new notebook of nothing but "gotchas and solutions" to deploying OpenStack21:37
oneswigThats awesome.  Next time you do it, it'll sail through!21:37
rbudden+1 on docs. I feel like the HA guides are lacking, and there’s an overall sense in the community that if you are doing production it’s assumed you are doing OSA or something similar which doesn’t really fit all deployment models21:39
martialI feel that there is a book in the making here21:39
trandles# load balancer21:39
trandles# Ideally the load balancer should not use the Infrastructure hosts.21:39
trandles# Dedicated hardware is best for improved performance and security.21:39
jmloweyou're doing something seriously wrong if the openstack api's knock over haproxy for anything other than storing glance image21:39
trandlesIn the OSA config file21:40
oneswigAnyway, for haproxy users, I highly recommend the telemetry.  There's a prometheus endpoint, which we scrape directly or poll and store in Monasca.21:40
trandlesBut that's not exactly the comment I'm remembering21:40
oneswigtrandles: I might read that comment as "dedicated hardware for running haproxy"21:40
trandlesoneswig, I agree21:40
rbuddenindeed. running haproxy on control plane nodes isn’t really a great idea21:41
rbudden@trandles i’d be curious to chat sometime about your experiences with OSA21:41
martial+1 on the dedicated hardware ... but that was a long time ago when we did our first deployment21:41
trandlesoneswig, you said prometheus...did you replace gnocchi with it?21:42
trandlesrbudden, any time21:42
jmlowejust from a security stand point, you greatly reduce your attack surface by only letting haproxy touch the outside and keeping your controllers inside21:43
oneswigI saw it in two configs today - one with Prometheus (and for metric storage), the other with Monasca scraping the same endpoint (ending up in influx).21:43
rbuddenjmlowe: +1 this is rather important for our setups21:44
trandlesrbudden, the reason I'm asking about prometheus/gnocchi is because gnocchi wouldn't install via OSA without serious fiddling and someone in OSA project pointed me to a third-person's ansible for prometheus installation21:44
oneswigInteresting aside, the Red Hat monitoring framework SAF apparently standardises on collectd instead.21:44
rbuddenthe future of Gnocchni and/or it’s replacement is concerning21:45
rbuddenbeing knee deep in a large scale Train deploy at the moment i’m curious if/when there will be a concensus in the community as to the direction of Telemetry21:46
trandlesrbudden, I'm deploying train via OSA, we definitely need to have a call very soon21:46
rbuddensounds like a plan21:47
oneswigprometheus is getting wide adoption but the back-end storage has limitations, such as control over retention period.  We've done a lot with influx (but never paid for the enterprise HA version)21:47
rbuddenwe currently have a custom hybrid Puppet/Ansible model with the intent on moving it entirely to Ansible as time allows21:47
oneswigDoes anyone have good pointers for telemetry that they monitor?21:48
martialfor us it is heavy Prometheus and heavy Ansible (but we do only have six racks)21:49
rbuddenmy understanding was Promethus with Ceilometer was not appropriate for billing data21:50
trandles10 minute warning btw - in case we're in the weeds ;)21:50
oneswigAs a peddlar of scalar telemetry, Prometheus doesn't get notifications, which can be useful for fine-grained timing.21:51
martialI think the next topic is AOB ... so "in the weeds" works :)21:51
oneswig#topic AOB21:51
*** openstack changes topic to "AOB (Meeting topic: scientific-sig)"21:51
oneswigseamless transition21:51
oneswigWanted to know if anyone has been using WekaIO?21:52
martialI want my emojis to high give Stig :)21:52
trandlesoneswig, took me a few months to get WekaIO to stop cold calling me. :P21:52
oneswigtrandles: surely after the first call they knew who you were? :-)21:53
trandlesArtful Dodger kept answering my phone21:53
oneswigI've been working on getting it well integrated with OpenStack clients, in a mutually beneficial way.21:54
oneswigWith help we got a client mounting Weka in "fast mode", ie SR-IOV and DPDK21:55
oneswigDon't have benchmark data yet, got held up building images with very specific versions (CentOS 7.6, MLNX-OFED 4.6). It can be hard to go back a few versions on cloud software :-)21:57
trandlesI did see this today: https://www.hpcwire.com/off-the-wire/openstack-software-adds-native-upstream-support-for-hdr-200-gigabit-infiniband/21:58
martialoneswig: seems like a pain to be honest21:58
oneswigAh, saw that too.  What's the big deal, surely one version of IB is like another?21:58
martialtrandles: that is cool21:58
trandlesoneswig, no idea21:59
oneswigjanders would probably know, but I think he's been doing this for ages anyway!22:00
trandlesI don't know enough about HDR 200 to know if it has to do with virtual functions, verbs-type stuff...??22:00
oneswigDon't know, will keep an eye out.22:00
oneswigWe are on the hour, time to close22:00
oneswigthanks all22:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"22:01
openstackMeeting ended Tue Jan 21 22:01:02 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)22:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-01-21-21.00.html22:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-01-21-21.00.txt22:01
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-01-21-21.00.log.html22:01
rbuddencatch ya’ll later22:01
martialtalk soon :)22:02
trandlesBy all22:02
trandles*bye even22:02
*** oneswig has quit IRC22:02
*** mattw4 has quit IRC22:06
*** eharney has quit IRC22:06
*** mattw4 has joined #openstack-meeting22:06
*** jmlowe has quit IRC22:09
*** mattw4 has quit IRC22:14
*** mattw4 has joined #openstack-meeting22:15
*** slaweq_ has quit IRC22:16
*** trandles has left #openstack-meeting22:16
*** enriquetaso has joined #openstack-meeting22:30
*** masahito has joined #openstack-meeting22:42
*** jamesmcarthur has quit IRC22:42
*** hyunsikyang has joined #openstack-meeting22:50
*** hyunsikyang__ has quit IRC22:54
*** e0ne has joined #openstack-meeting23:00
*** Lucas_Gray has quit IRC23:08
*** masahito has quit IRC23:09
*** Lucas_Gray has joined #openstack-meeting23:09
*** rbudden has quit IRC23:11
*** Liang__ has joined #openstack-meeting23:20
*** armax has quit IRC23:41
*** rfolco|brb is now known as rfolco23:41
*** e0ne has quit IRC23:52
*** dviroel has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!