16:00:03 <mattmceuen> #startmeeting airship
16:00:03 <openstack> Meeting started Tue Jun 18 16:00:03 2019 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:06 <openstack> The meeting name has been set to 'airship'
16:00:09 <mattmceuen> #topic Rollcall
16:00:19 <alexanderhughes> o/
16:00:20 <evgenyl> Hi everyone!
16:00:27 <mattmceuen> GM / GE everyone!   here's our agenda for today, you know the drill: https://etherpad.openstack.org/p/airship-meeting-2019-06-18
16:00:28 <michael-beaver> o/
16:00:29 <nishantkr> o/
16:00:34 <mattmceuen> we'll give it just a minute
16:00:36 <obravo> Hello everyone!
16:00:45 <mattmceuen> o/ obravo welcome
16:01:45 <dwalt> o/
16:02:05 <mattmceuen> Alright, let's go:
16:02:12 <mattmceuen> #topic Election
16:02:54 <mattmceuen> Just a very brief update on what we discussed last week - it took some time to get the airship/elections project created, so I'm planning on sending out an email on that to kick off TC nominations today
16:03:12 <mattmceuen> That's all I have on that
16:04:09 <mattmceuen> PS:  if you visit the election repo (https://opendev.org/airship/election) the dates are all wrong - they were the optimistic ones I authored last week, I'll fix them prior to sending out an email
16:04:29 <mattmceuen> unless any questions, I'll move on:
16:04:53 <mattmceuen> #topic Meeting Time Voting
16:05:23 <mattmceuen> I sent out an email on this yesterday, but the instructions are also in the etherpad (no need to find the email if you missed it)
16:06:14 <mattmceuen> We discussed at the PTG that there's no good time for this IRC meeting that works well for everyone, and we wanted to vote not just on meeting times but also on some potential approaches (e.g. alternating meetings) to try to accomodate more attendees
16:07:00 <mattmceuen> The ask is for everyone (who cares about timing) to give their top five choices for Meeting Time + Meeting Approach by filling in a template on the bottom of the etherpad:
16:07:01 <mattmceuen> https://etherpad.openstack.org/p/airship-meeting-vote-2019
16:07:25 <mattmceuen> More details in the etherpad, and thanks to everyone who's voted so far!
16:07:44 <mattmceuen> We'll leave voting open till EOD next Tuesday so we can remind folks in next Tuesday's meeting
16:07:45 <openstackgerrit> Kudaka Poorna Rajesh proposed airship/promenade master: [DNM] Test probes for calico-etcd  https://review.opendev.org/665900
16:07:58 <mattmceuen> any questions / discussion on the meeting time stuff?
16:09:08 <mattmceuen> Cool, moving on:
16:09:14 <mattmceuen> #topic Tungsten Fabric Integration
16:09:42 <mattmceuen> obravo has been doing some work to integrate Tungsten Fabric into Airship -- AIAB PS: https://review.opendev.org/#/c/664648/
16:09:52 <mattmceuen> obravo, can you please give us some quick overview of this work?
16:11:32 <obravo> Well , it makes AIAB use TF instead of openvswitch . I did some changes into deploy scripts and configs.
16:12:02 <mattmceuen> That's awesome
16:12:03 <obravo> I tried to keep  switching between TF and vSwitch as easy , as possible
16:12:28 <mattmceuen> How is the work progressing / working?
16:13:04 <obravo> it`s tested with TF running and is working good in singlenode deployment
16:13:19 <evgenyl> Should we consider leaving neutron-ovs by default and creating overrides specific to other network providers?
16:13:43 <evgenyl> Also you may start looking into an up to date AIAB configuration https://github.com/airshipit/treasuremap/tree/master/tools/deployment/aiab
16:14:01 <evgenyl> The one in Airship in a Bottle is outdated, and will be retired in the nearest future.
16:14:27 <evgenyl> *The one in Airship in a Bottle repository
16:14:48 <mattmceuen> Yeah, obravo's patchset prompted me to add that onto the agenda - we'll dig into the "which aiab" more in a minute
16:15:17 <evgenyl> Yep, sounds good.
16:15:39 <mattmceuen> obravo:  I know you've been working to get TF support into the upstream OSH Neutron Chart: https://review.opendev.org/#/c/663390/
16:15:55 <mattmceuen> Would it make sense to point your patchset for TF AIAB at that patchset?
16:16:22 <obravo> Yes , this is related review
16:16:40 <openstackgerrit> PRATEEK REDDY DODDA proposed airship/promenade master: [WIP] Coredns: Add pod/container security context  https://review.opendev.org/664662
16:17:39 <mattmceuen> my thought was just that, in `versions.yaml` you could point the neutron chart to that patchset, rather than the github version, to help keep the two in sync.  just a suggestion
16:18:37 <mattmceuen> this is great to see this progressing, and I'll dig in and provide feedback in the PS obravo
16:18:45 <obravo> Oh , i got it. Thank you  , this will he be  more  convenient
16:19:24 <mattmceuen> one thing to keep in mind as evenyl said we're moving aiab into treasuremap.  Part of the driver for that is to keep all of our "reference manifests" aligned and de-duplicated
16:20:17 <obravo> Fine , i`ll take a look into it
16:20:19 <mattmceuen> I'm considering how best to make TF manifests available as a reference -- AIAB is a nice start, but it would be ideal not to constrain them just to the AIAB site
16:20:59 <mattmceuen> This would be a perfect use case for the "service layer" mix-and-match approach we've wanted for a long time, but don't yet have
16:21:00 <jamesgu__> +1 ... I was going to make the same request
16:22:16 <mattmceuen> Using "what we have today", it would be somewhat straightforward to:
16:22:33 <mattmceuen> well, hmm
16:23:12 <mattmceuen> there are multiple tools in our toolbox, including overriding at the type or site level, making new manifests at the global level, or actually implementing something like service layers
16:23:21 <mattmceuen> Can we do this:
16:24:16 <mattmceuen> obravo: I don't want to derail your work -- perhaps you can keep doing what you're doing, perhaps migrating over to treasuremap
16:25:02 <mattmceuen> and then let's chew on the best way to integrate TF reference manifests (and other types of use cases in the future) into treasuremap more generally next week
16:25:34 <mattmceuen> does that sound good to everyone?  Ok to take some "think this through" homework? :)
16:26:19 <jamesgu__> sounds good.. (minus the homework:-))
16:26:21 <obravo> Yes, that`s fine
16:26:29 <mattmceuen> lol jamesgu__
16:26:40 <mattmceuen> ok, great
16:26:47 <mattmceuen> thanks for bringing this up obravo
16:27:09 <mattmceuen> related topic:
16:27:16 <mattmceuen> #topic AIAB -> Treasuremap migration status
16:27:57 <mattmceuen> We've started directing folks (informally) to the new treasuremap single-node AIAB when they hit issues with the single-node demo site in the airship-in-a-bottle repo
16:28:31 <evgenyl> Yeah, as mentioned earlier here is link https://github.com/airshipit/treasuremap/tree/master/tools/deployment/aiab
16:28:32 <mattmceuen> Are we comfortable to make the treasuremap version `the`AIAB single-node deployment yet?
16:28:40 <evgenyl> Please try it out.
16:28:51 <evgenyl> The builds are kind of stable, the only concern is MaaS.
16:29:05 <evgenyl> Sometime it takes more than 20 minutes for maas-import-resource to finish.
16:29:15 <evgenyl> So it may fail due to timeout.
16:29:36 <mattmceuen> how long does maas-import-resource take on the aiab repo version?
16:30:08 <evgenyl> mattmceuen: I have not measured on a previous version, but definitely less than 20 minutes.
16:30:32 <evgenyl> Also the average time of the build between old and new conifguration increased by 20 mins.
16:30:40 <evgenyl> So this looks to be consistent.
16:30:49 <mattmceuen> why the difference in the import time, do you know -  just "recent maas chart" vs "older maas chart"?
16:31:06 <evgenyl> Yep, I don't see any other difference.
16:31:20 <evgenyl> It waits for rack to sync.
16:31:32 <evgenyl> Maybe there is some issue with properly configuring timeouts.
16:31:47 <evgenyl> It would great if we can find some airship-maas expert to look into that.
16:31:51 <mattmceuen> ok.   well, it is what it is, and if we need to increase timeouts, then we need to increase timeouts -- I don't think we should pin to an old maas just for timing's sake
16:32:03 <mattmceuen> and agree, better to improve the new stuff than pin to the old stuff
16:32:16 <nishantkr> This change may be of help - https://review.opendev.org/#/c/664045/
16:32:30 <nishantkr> still in review phase
16:33:13 <evgenyl> The issue with postgresql crashloop is still present by the way.
16:33:23 <evgenyl> But it also happens in the old configuration.
16:33:36 <evgenyl> Let's see if switching to newer charts helps.
16:33:52 <evgenyl> Since they use patroni now for the clustering and configuration.
16:34:09 <mattmceuen> evgenyl - you and I have different PS out there to integrate new patroni, and I think we need some of each of our PS :)  let's sync later today on this
16:34:35 <evgenyl> Sure, let's discuss it.
16:34:48 <mattmceuen> nishantkr - thanks for that one, I added it to the "please review" list
16:35:27 <mattmceuen> evgenyl, should we target hardening & tuning this week, with a goal of formally switching our "advertised" AIAB over next week?
16:36:18 <evgenyl> I think we can start asking people to use it even now, the deployment should be as stable as the new one.
16:36:48 <evgenyl> But AIAB repo will still be there, at least until we move multinode installation over to treasuremap.
16:37:05 <mattmceuen> Yeah, I was just thinking the same thing. I think I do want to try to harden it for a couple days though - when we ask people to switch over, we want to make a good first impression
16:37:09 <evgenyl> *as stable as the old one
16:37:58 <mattmceuen> i.e., let's switch over when it's a clear and above improvement to switch -- if we all beat up on the TM AIAB a bit I bet we can get there this week
16:38:41 <mattmceuen> michael-beaver:  how is work on migrating the multi-node AIAB over to TM going?  Sorry, I've lost line of sight on that
16:39:43 <michael-beaver> It is going alright, but I am stuck at the barbican test in genesis as I am getting some SSL errors when it tries to contact keystone
16:39:57 <evgenyl> michael-beaver: Do you need some help with that?
16:40:02 <evgenyl> I wanted to start looking into that.
16:40:11 <michael-beaver> That would definitely be appreciated, would love to sync up on it
16:40:46 <mattmceuen> that would be awesome - thanks michael-beaver & evgenyl
16:40:54 <evgenyl> Yep, sounds good.
16:41:57 <mattmceuen> Alright!  If nothing else on this one, moving on:
16:42:21 <mattmceuen> #topic consuming openstack services value overrides added for OS distro and openstack release in openstack-helm
16:42:30 <mattmceuen> arunkant:  this one's yours
16:42:56 <arunkant> yes..so I was wondering if there has been discussion in past about this following topic
16:42:57 <mattmceuen> and maybe related to the "how to best add TF to treasuremap" conversation from before!
16:43:05 <mattmceuen> So first thing:
16:43:30 <mattmceuen> Could you please outline what you see as the full set of overrides that would be deployed in this context
16:43:48 <arunkant> So in openstack-helm service charts, there was value overrides were added specific to a OS distro and openstack release as well.
16:43:58 <mattmceuen> Basically "all of them" across airship components and openstack components, right?
16:44:25 <arunkant> Its specifically for openstack services ..
16:44:58 <mattmceuen> so we had talked previously about the more narrowly-scoped "how can we do SUSE Airskiff"
16:45:03 <arunkant> as we store config and other related data in per services values file..which can change across openstack release
16:45:14 <evgenyl> I think I know the answer, but is there anything else other than images overrides?
16:45:58 <arunkant> Yes..there are configuration changes as well. We noticed in neutron some of rootwrap configuration does not work with rocky release as its changed
16:46:11 <evgenyl> Ok, I see.
16:46:23 <arunkant> from the version we have in neutron values.yml
16:47:35 <arunkant> So that's one of the reason these value overrides were added to allow per release changes and then if anything which is different for specific distro, then have override file for that
16:47:35 <mattmceuen> So, in treasuremap, we have three layers defined:  Global (general defaults for charts), Type (tunings for particular use cases), and Site (site-specific overrides)
16:47:41 <obravo> speaking about review with  TF support - overriding is not needed
16:47:43 * redrobot pokes head in at the sound of barbican
16:48:17 <mattmceuen> The challenge is that when we're talking about overrides for *versions* of things, that's kind a different question than "use case" or "site"
16:48:37 <mattmceuen> Would be nice to have a "mixin" of overrides on top of a use case or site
16:48:39 <evgenyl> It is not a serious suggestion, however maybe we should start looking into what kustomize can do and if we can start leveraging it for these use-cases?
16:48:48 <evgenyl> As a part of transition to Airship 2.0.
16:48:59 <mattmceuen> yes, was thinking the same thing
16:49:02 <openstackgerrit> Alexander Noskov proposed airship/treasuremap master: Move Airship Seaworthy pipeline to the folder.  https://review.opendev.org/666048
16:49:30 <mattmceuen> Jury is still out on kustomize I believe, but especially if we switch toolsets for 2.0 then we should do so with a plan for mixin / service layer type functionality
16:49:38 <mattmceuen> o/ redrobot
16:50:21 <evgenyl> It may be as pre or post pegleg processing step for now, until we have a complete (?) replacement.
16:50:52 <arunkant> Yes, this can be something to think for 2.0 . In the interim, was there any discussion how those value overrides can be consumed on airship side.
16:50:55 <mattmceuen> For "SUSE Airskiff", we had talked about applying extra overrides on top of the airskiff site manifests at the little-used fourth layer, "cicd"
16:51:34 <mattmceuen> we could easily follow the same approach for, e.g. the seaworthy site, with the burden of repetition however
16:51:43 <mattmceuen> as we have a strict tree-model
16:51:49 <mattmceuen> for our manifest layers
16:53:07 <arunkant> mattmceuen: So with tree-model or a new layer in-between, are we going to duplicate those override configuration in those layer ?
16:53:12 <mattmceuen> arunkant, I think the answer today is to either create a different `type` which applies all the overrides, unless airskiff is enough from a treasuremap perspective, in which case that fourth layer (cicd) should fit the bill
16:54:41 <arunkant> mattmceuen: I am not aware of cicd layer.. is it documented or sample available somewhere ?
16:55:05 <mattmceuen> The only way to avoid duplication will be to 1) do some kind of pre- or post- processing (sidestepping the declarative model) like evgenyl suggested, or, implement something like service layering
16:55:18 <mattmceuen> the most documentation was when we discussed it a couple weeks ago :)
16:55:32 <mattmceuen> I will drop you a link to the logs arunkant, will need to dig a smidge
16:56:16 <mattmceuen> But this does in fact tie into the same general question that obravo's TF integration raised
16:56:33 <mattmceuen> "orthogonal" - that's the word I was trying to think of
16:56:36 <arunkant> mattmceuen: So if we need to add pre- or post-processing logic to avoid duplication , which airship component is best suited for ?
16:56:43 <arunkant> mattmceuen: So if we need to add pre- or post-processing logic to avoid duplication , which airship component is best suited for that ?
16:57:17 <jamesgu__> airskiff suse gate could be sufficient until we get to 2.0
16:57:28 <mattmceuen> choice of software, operating system, openstack version, SDN choice... these things are someone (not completely) orthogonal to one another, and don't fit into a hierarchy well
16:57:46 <mattmceuen> cool.  jamesgu__ I'll add this to rodolfo's design call agenda for next week
16:57:56 <mattmceuen> to at least make sure it's on the radar
16:58:11 <jamesgu__> k
16:58:39 <mattmceuen> arunkant:  I don't have a really solid idea on that yet, do you evgenyl?
16:59:47 <mattmceuen> in any case - I think you're caught up on the conversation now arunkant :)  more conversation needs to be had
17:00:09 <evgenyl> This should be done somewhere in the gate before or after pegleg collect, but how we can exactly use customize for changing the manifests is not completely clear.
17:00:29 <evgenyl> This requires some additional research and thinking.
17:00:33 <mattmceuen> I've captured in our "homework" for the week:
17:00:37 <mattmceuen> Beat up on the new AIAB in treasuremap
17:00:37 <mattmceuen> Give some thought to how to incorporate e.g. TungstenFabric and SUSE manifests in to treasuremap
17:00:54 <arunkant> mattmceuen: I have not even thought through about this, just want to see if there are any ideas discussed in past.
17:00:54 <mattmceuen> would be awesome if everyone can do some of each!
17:01:12 <mattmceuen> alright guys, we are out of time
17:01:19 <hogepodge> can i pop in with one thing
17:01:19 <mattmceuen> #topic Review Requests
17:01:20 <hogepodge> ?
17:01:26 <mattmceuen> almost!
17:01:29 <mattmceuen> https://review.opendev.org/#/c/664648/ - Tungsten Fabric integration in AIAB
17:01:29 <mattmceuen> https://review.opendev.org/#/c/664993/ - Treasuremap [aiab] Fix a path to scripts directory
17:01:29 <mattmceuen> https://review.opendev.org/#/c/665007/ - Treasuremap [aiab] Add a site linting gate to Zuul
17:01:29 <mattmceuen> https://review.opendev.org/#/c/665228/ - Treasuremap Uplift PostgreSQL
17:01:29 <mattmceuen> https://review.opendev.org/#/c/664045/ - Check sync of only active MaaS rack controllers
17:01:33 <mattmceuen> Please review :) ^
17:01:47 <mattmceuen> #topic Hogepodge gets a word in!
17:01:51 <mattmceuen> go for it hogepodge
17:01:59 <hogepodge> OSF is planning on having a booth at KubeCon NA this November
17:02:32 <hogepodge> We're having our first planning meeting for it today, but I wanted to make you all aware that we want to have an Airhip presence and to start thinking about things we might want to do there.
17:02:35 <hogepodge> Demos, talks, so on.
17:02:43 <mattmceuen> awesome!
17:02:54 <mattmceuen> that's really good to hear
17:03:02 <hogepodge> If you know that you're going to be going, please let me know so I can coordinate with the rest of the staff.
17:03:09 <hogepodge> I'll be there.
17:03:28 <mattmceuen> I don't know about me yet - need to submit some talks :)
17:04:07 <mattmceuen> anything we can do to help at this point (before attempting to show up there) hogepodge?
17:05:21 <hogepodge> Right now knowing who's going and how we want to handle our presence would be the biggest help.
17:05:42 <mattmceuen> sounds like a plan - will work toward that sir
17:05:49 <mattmceuen> thanks for bringing this up
17:06:12 <mattmceuen> Thanks for joining, everyone, we're out of time!
17:06:17 <mattmceuen> have a great week :)
17:06:21 <mattmceuen> #endmeeting