14:00:20 <mattmceuen> #startmeeting airship 14:00:21 <openstack> Meeting started Tue Jan 15 14:00:20 2019 UTC and is due to finish in 60 minutes. The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:24 <openstack> The meeting name has been set to 'airship' 14:00:28 <mattmceuen> #topic rollcall 14:00:42 <mattmceuen> gm everyone! 14:00:44 <dwalt> \o/ 14:00:50 <dwalt> GM!' 14:00:59 <mattmceuen> Here's our agenda for today: https://etherpad.openstack.org/p/airship-meeting-2019-01-15 14:01:11 <mattmceuen> Please add anything you'd like to discuss 14:01:15 <evgenyl> Good morning! 14:01:26 <mattmceuen> and we'll give it a couple mins for folks to filter in 14:01:30 <sthussey> here 14:01:42 <Kaspars> also here 14:01:49 <mattmceuen> hey guys 14:02:29 <portdirect> o/ 14:03:15 <mattmceuen> so the meat of the agenda today is to share/explain a few things that have gotten added in recently 14:03:20 <mattmceuen> #topic podsecuritypolicy chart 14:03:52 <mattmceuen> So you may or may not be familiar with k8s PodSecurityPolicies 14:04:43 <mattmceuen> If you configure your k8s to use them, then the k8s api server will only allow you an actor to schedule pods that meet certain criteria/policy 14:04:56 <mattmceuen> Based on k8s RBAC 14:05:36 <mattmceuen> This is a security feature we wanted to add into airship, but in a way that doesn't break all the privileged actions that are taken across airship and openstack-helm 14:05:46 <mattmceuen> This helm chart was added to openstack-helm-infra: https://github.com/openstack/openstack-helm-infra/tree/master/podsecuritypolicy 14:06:17 <mattmceuen> It specifies by default an incredibly (100%) permissive podsecuritypolity, and sets it up as a default for the cluster 14:06:54 <mattmceuen> It was also added into the Treasuremap reference yamls: https://review.openstack.org/#/c/629686/ 14:07:04 <mattmceuen> You can do a couple things through the chart: 14:07:18 <sthussey> Why was the chart put in a different project if it was intended as a part of Airship? 14:07:41 <mattmceuen> because it's not airship-specific 14:07:46 <mattmceuen> pretty broadly applicable 14:08:24 <mattmceuen> You can use the chart to: 14:08:24 <mattmceuen> 1) add whatever additional podsecuritypolicies you want for your cluster, programmatically, letting helm manage lifecycle 14:08:34 <mattmceuen> 2) change the default(s) 14:09:04 <mattmceuen> Over time, we want to tune the defaults in the chart to be a reasonable non-fully-open set of policy, as much as possible. However, the intent is also that operators fully customize it for their workloads, 14:09:39 <portdirect> Nice work mattmceuen 14:09:59 <mattmceuen> You can set up defaults in the chart individually for serviceacounts, authenticated users, and unauthenticated users via the chart, and/or associate the PSPs to other roles outside of the chart itself 14:10:04 <mattmceuen> that's it - any questions? 14:10:13 <portdirect> Should help make it much easier to manage the security posture of a cluster. 14:10:25 <roman_g> o/ 14:10:37 <hogepodge> Hi 14:10:42 <mattmceuen> o/ guys :) 14:10:56 <mattmceuen> One more note, just having to do with how PSP's work in k8s 14:10:59 <sthussey> What is the expectation of coordinating access to PSPs and the service accounts created by other Helm charts? 14:11:48 <mattmceuen> that'll be on the operator to coordinate; in an airship world, via substitution. That would be a good pattern to construct/demo in treasuremap I'd think 14:12:59 <mattmceuen> so PSPs only affect pods created *after* the PSPs are in place. So, that's why the psp chart was added first in the treasuremap armada manifest 14:13:26 <mattmceuen> alright, next topic: 14:13:35 <mattmceuen> #topic Mini-mirror integration 14:13:41 <mattmceuen> dwalt take it away! 14:14:00 <dwalt> alrighty! 14:14:16 <roman_g> dwalt: Nice job! I like it. Thank you. 14:14:21 <dwalt> One of the challenges we were having in larger scale deployments was the coordination of packages on the host 14:14:26 <dwalt> Thanks roman_g! 14:15:57 <dwalt> i.e. packages changed between the time that we ran test deployment pipelines to the time we actually deployed Airship 14:17:13 <dwalt> Mini-mirror exists as a way to combat that issue. By mirroring Ubuntu package repositories and deploying them into the cluster, we can control what packages exist on the host for a deployment 14:17:55 <powerds0111> o/ 14:18:23 <dwalt> Currently, you can utilize mini-mirror by building an image by specifying the sources/packages you would like to mirror 14:18:25 <dwalt> #link https://github.com/openstack/openstack-helm-images/blob/master/mini-mirror/README.rst 14:19:11 <dwalt> The next step is integrating mini-mirror into the genesis process, with the vision that we may one be able to deploy Airship without internet connectivity 14:19:31 <dwalt> The chart for mini-mirror is also here: 14:19:33 <dwalt> #link https://github.com/openstack/openstack-helm-addons/tree/master/mini-mirror 14:19:37 <dwalt> Any questions? 14:19:57 <Kaspars> this looks really cool, are you thinking to add it to treasuremap any time soon? 14:20:26 <roman_g> At the moment mini-mirror does not contain: docker image archives and k8s binaries; they would need to be baked into some /var/cache inside the image 14:20:30 <dwalt> Kaspars: Thanks! Definitely. As soon as we have a published demo image, we would like to get a full demo 14:21:50 <sthussey> mini-mirror is just a framework. It should contain whatever packages you tell it sourced from whatever source you provide when you build the image. 14:22:05 <dwalt> ++ 14:22:19 <mattmceuen> Yup roman_g those would be good enhancements down the road 14:22:24 <mattmceuen> o/ powerds0111 14:22:48 <mattmceuen> anything else on the mini-mirror front guys? 14:22:57 <mattmceuen> also, hooray for clever names. very catchy 14:23:16 <sthussey> I would suggest if it isn't merged yet that you consider naming it mini-apt or something 14:23:47 <mattmceuen> lol well you got one vote each way there 14:23:52 <mattmceuen> but it's merged 14:24:10 <sthussey> It isn't really part of Airship, so this is just tangential, but there is a merged spec in Airship to support multiple OSes and making mini-mirror Ubuntu specific doesn't really jive with that 14:24:20 <portdirect> There would be nothing stopping adding rpm support 14:24:35 <mattmceuen> totally agree 14:24:35 <sthussey> And I don't think long term adding the complexity to make a single framework support both is worth it 14:24:50 <sthussey> Just pick whether you want mini-apt or mini-rpm and go forward 14:25:02 <portdirect> Or just a value toggel 14:25:18 <portdirect> All it would take is turning on or off, two manifests 14:25:19 <mattmceuen> the *chart* part is really an nginx chart, right dwalt? 14:25:31 <sthussey> Yeah, I don't really care about the chart part 14:25:32 <mattmceuen> and the apt magic is in the docker build stuff 14:25:36 <sthussey> The complexity is in building the image 14:25:44 <dwalt> At it's core, it really is just NGINX 14:25:56 <dwalt> the image itself is just an NGINX image with the mirrored snapshots 14:26:15 <sthussey> Again, just tangential as it isn't part of Airship. But a consideration. 14:26:15 <dwalt> so a similar tool could be used (with a toggle) to mirror rpm repos 14:26:24 <mattmceuen> it might make sense to separate the apt docker build from the rpm docker build, since they'd likely be quite different 14:26:52 <dwalt> This is true. And OpenStack-Helm addons already has the pattern for multiple OS/Dockerfiles 14:27:07 <mattmceuen> I don't think it's tangential, we want to bring multi-os support into Airship 14:27:08 <dwalt> OpenStack-Helm-images* 14:27:10 <mattmceuen> good point 14:27:26 <mattmceuen> anything else on MM? 14:27:55 <portdirect> Maxi-mirror? 14:28:16 <srwilkers> :) 14:28:19 <mattmceuen> Just depends how much stuff you shove in the image portdirect :) 14:28:27 <mattmceuen> #topic Monthly Treasuremap Release 14:28:27 <portdirect> Is there thought toward caching of things like docker images at this point, or that still at design stage? 14:28:41 * mattmceuen jumps the gun 14:28:48 <mattmceuen> #topic maxi-mirror 14:29:23 <mattmceuen> portdirect: I think there's been some discussion around it, but I don't know if any stories are in storyboard or anyone working on it yet 14:29:51 <roman_g> me neither 14:29:52 <sthussey> While possibly branded as mini-mirror, I would expect something like that to basically be completely separate in terms of image build 14:30:32 <portdirect> I was more thinking an adjunt here, to separate the concerns 14:30:45 <sthussey> I think designs right now around image caching are something like squid w/ MITM TLS signing 14:30:48 <portdirect> Looks like mini-mirror allows offline host provisioning 14:31:05 <roman_g> dwalt: would you work on blueprint, or I can take it? 14:31:19 <portdirect> But still leaves the gap of how to get containers into a site without internet or artifactory or summat. 14:31:35 <roman_g> (bake into genesis iso) 14:31:39 <dwalt> roman_g: You can take a stab at it if you'd like! 14:31:45 <roman_g> dwalt: deal 14:31:50 <sthussey> At this point, that wasn't the business value targeted w/ mini-mirror 14:31:50 <portdirect> Nice 14:32:09 <portdirect> sthussey: I understand that, hence my q. 14:32:56 <roman_g> sthussey: portdirect: mattmceuen what is your decision/input? 14:33:23 <sthussey> Decision on what? If you want to draw up a spec for something, go for it. 14:33:40 <mattmceuen> roman_g: if you'd like to take a stab at a spec PS, I think that gives a good opportunity to prompt discussion, even (especially) if that prompts a change in direction 14:33:46 <portdirect> After 3 mins of percolation time, bit early for me to put my stake in the ground on anything roman_g 14:34:01 <portdirect> ++ to mattmceuen s point 14:34:18 <mattmceuen> thanks roman_g 14:34:34 <mattmceuen> Anything else on the multi-mirror discussino? 14:35:00 <mattmceuen> #topic Monthly Treasuremap Release 14:35:08 <mattmceuen> Kaspars, all yours 14:36:00 <Kaspars> great! so we have I would say almost routine monthly tag of airship-treasuremap manifests found located at https://github.com/openstack/airship-treasuremap/releases 14:36:48 <Kaspars> as before - it includes auto-uplifts of components (with a few exclusions this time that include MAAS HA that Scott will talk about later, and armada due to test pod handling) 14:37:17 <Kaspars> there are a few other notable changes as ceph partition handling and update of Docker to much newer version 14:38:22 <Kaspars> as before - it has ran through full airship deployment pipeline (Airship Seaworthy), here are few sentances about it https://airship-treasuremap.readthedocs.io/en/latest/ 14:38:57 <Kaspars> I would say enjoy the latest release candidate for 1.0 14:39:21 <mattmceuen> that's awesome Kaspars 14:39:29 <seaneagan> in-flight fix for Armada test pod handling: https://review.openstack.org/#/c/629313/ 14:39:45 <mattmceuen> thanks for keeping the good stuff coming and the pipelines green :) 14:39:59 <mattmceuen> o/ seaneagan 14:40:30 <mattmceuen> any other discussion on the release candidate? 14:41:18 <mattmceuen> #topic HA MaaS overview 14:41:23 <mattmceuen> sthussey all yours! 14:42:12 <sthussey> Recently there a significant refactor of the Airship chart for the Canonical MAAS product. 14:42:41 <sthussey> Previously it was a bit fragile in terms of where it ran and how it responded to moving to other nodes. 14:43:44 <sthussey> It now should support a highly available configuration for the MAAS components (HA Postgres under it is out of scope) and with that HA support comes mobility across nodes in the cluster 14:44:09 <sthussey> There are some caveats and suppositions for the current solution 14:44:17 <portdirect> thats great sthussey 14:44:39 <portdirect> is the required config in treasuremap and airship in a bottle - so we can take it for a spin? 14:44:41 <sthussey> 1) It expects a flat L2 broadcast domain for the PXE network across all nodes in the cluster 14:45:07 <sthussey> re: config - Can't speak to Treasuremap, but the multinode-gate in AIAB should work with it 14:46:03 <Kaspars> there has not been patchsets submitted for treasuremap, yet 14:46:19 <sthussey> 2) You must be able to route between the PXE booting network and whatever network your cluster ingress runs on. If there is a single network in your cluster for all traffic, that is sufficient. 14:47:05 <sthussey> 3) Currently there is no TLS support for accessing the MAAS API/metadata server. That support will hopefully be in place in the near future 14:47:41 <hogepodge> Somewhat related, is Ironic making progress in the RCs? 14:48:08 <mattmceuen> hogepodge: there was good discussion in rodolfo's zoom call last thursday, if you want to see the recording 14:48:11 <sthussey> 4) If you are using the OSH-infra ingress chart, you must be using nginx-ingress-controller v0.20.0 or better 14:48:29 <hogepodge> (kind of previous topic juggling multiple chat windows) 14:48:30 <sthussey> Otherwise you get false port binding conflicts between cluster ingress and maas ingress 14:48:42 <sthussey> That is all from me. 14:49:01 <mattmceuen> thanks for all the details sthussey - that was a big effort with a lot of tricky bits 14:49:12 <hogepodge> mattmceuen yes please, I’ll coordinate offline thx 14:49:21 <mattmceuen> awesome, sounds good hogepodge 14:49:37 <mattmceuen> sounds like treasuremap is pinned to an old version - I guess the interface to the maas chart isn't backwards compatible, right sthussey? 14:50:34 <sthussey> Not sure, would have to test. The previous overrides certainly won't enable the HA functionality 14:51:28 <Kaspars> the pipeline did fail - so I had to disable auto-uplift for maas/drydock charts.. 14:51:33 <mattmceuen> gotcha. Kaspars, maybe we can coordinate in the chat here to after the meeting to see what the issues you were seeing are, so we can get HA MaaS into TM now that the stable release was cut 14:52:12 <mattmceuen> k we have a few more topics and scant few minutes! 14:52:20 <mattmceuen> #topic PS needing review 14:52:25 <Kaspars> there was a pretty large PS comitted to AIAB - so those changes need to be ported to treasuremap as well 14:52:32 <mattmceuen> I'm just going to paste these with a blurb and then move on! 14:52:44 <Kaspars> to support latest MAAS/drydock - as well as few DNS handling need to be settled 14:52:47 <Kaspars> some work.. 14:52:54 <mattmceuen> Blurb: I want to make sure there's an opportunity for folks to bring up PS weekly that they view as high priority, and/or lost in the shuffle 14:52:58 <mattmceuen> https://review.openstack.org/630749 14:52:59 <mattmceuen> https://review.openstack.org/630752 14:52:59 <mattmceuen> https://review.openstack.org/630755 14:53:15 <mattmceuen> ^^ from dwalt - please give those some eyes soon team 14:53:20 <mattmceuen> any others? 14:54:09 <mattmceuen> One topic not on the agenda: 14:54:33 <mattmceuen> #topic best of luck mark-burnett ! 14:54:44 <hogepodge> I have one regarding ATC codes for Denver Summit 14:55:24 <Kaspars> best of luck, Mark!! 14:55:32 <mattmceuen> Mark is taking another opportunity. He is of course more than welcome to continue participating in the airship community as much as possible! But I also just wanted to share that hey may not have the same bandwidth 14:56:08 <mattmceuen> Mark has been instrumental in forming the airship community, and definitely with him best of luck in the next endeavor! 14:56:25 <mattmceuen> #topic ATC codes for Denver Summit 14:56:30 <mattmceuen> go for it hogepodge 14:57:00 <hogepodge> Congratulations Mark! 14:57:40 <cbcg> congarts Mark! 14:57:47 <hogepodge> we need to put together a list of active contributors to receieve discount codes for the Denver summit 14:58:23 <hogepodge> To help us with that we need a list of repositories to draw from for contributions. 14:58:54 <hogepodge> I’m assuming all repos prefixed airship-, but if there are others we missed getting them to fungi would be a big help 14:59:11 <mattmceuen> I think that is the right set hogepodge 14:59:31 <mattmceuen> thanks for making sure 14:59:37 <hogepodge> Since Airship is so new, we’re looking at contributions back to the launch of Airship as a pilot project in Vancouver for the contribution timeframe 14:59:48 <hogepodge> cool, thanks :-) 14:59:51 <mattmceuen> Sounds reasonable to me! :) 15:00:36 <mattmceuen> Ok guys we are hard outta time -- please vote in the board election, and help out with the deckhand doc builds: 15:00:36 <mattmceuen> Please, assist: airship-deckhand documentation builds are broken since Oct. 30, 2018 15:00:36 <mattmceuen> Bug: https://storyboard.openstack.org/#!/story/2004747 15:00:36 <mattmceuen> Logs: https://readthedocs.org/api/v2/build/8395458.txt 15:01:06 <roman_g> It's mine, and I don't know how to fix it. 15:01:32 <dwalt> roman_g: I will take a look at that sometime today 15:01:34 <mattmceuen> Cool - let's continue the discussion here post-meeting roman_g and get that working 15:01:38 <mattmceuen> #endmeeting