16:00:23 <mattmceuen> #startmeeting Airship
16:00:24 <openstack> Meeting started Tue May 21 16:00:23 2019 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:27 <openstack> The meeting name has been set to 'airship'
16:00:28 <mattmceuen> #topic Rollcall
16:00:32 <alexanderhughes> o/
16:00:34 <aaronsheffield> o/
16:00:34 <mattmceuen> Good morning / evening everyone!
16:00:38 <michael-beaver> o/
16:00:39 <mattmceuen> https://etherpad.openstack.org/p/airship-meeting-2019-05-21
16:00:41 <evgenyl> Hi!
16:00:48 <dwalt> o/ !
16:00:52 <mattmceuen> ^ agenda, let's give folks a couple mins to get here and add stuff to the agenda
16:00:54 <nishantkr> o/
16:01:24 <mattmceuen> dwalt it looks like you're raising your arm in alarm - it's not a stickup, I promise
16:01:31 <arunkant> o/
16:01:39 <roman_g> o/
16:02:02 <alexanderhughes> if it was, wouldn't it be \o/
16:02:18 <mattmceuen> ! \o/ !
16:02:20 <openstack> mattmceuen: Error: "\o/" is not a valid command.
16:02:35 <kskels> o/
16:02:54 <mattmceuen> Great turnout today, thanks for being here everyone :)
16:02:57 <mattmceuen> #topic Email notifications to mailing list for failing, periodic jobs
16:03:07 <mattmceuen> this one is dwalt's - take it away sir
16:03:28 <dwalt> lol - missed the comment above. Not alarmed, just happy to be here :)
16:04:29 <dwalt> So mattmceuen and I have been discussing some jobs we have that would be better suited as periodic jobs. e.g. chart linting against the latest helm-toolkit from osh-infra and a job that would verify our credentials for posting images and publishing images is correct
16:04:32 <dwalt> are* correct
16:05:19 <dwalt> An idea we had was emailing the results of the aforementioned jobs to a new mailing list, perhaps airship-alerts or airship-builds.. something like that
16:06:17 <dwalt> I added this to the agenda in case for a few reasons. 1: for knowledge sharing if anyone has set something up like this before (looks like there is an email template with zuul), 2: for feedback, if anyone has additional ideas for jobs or comments about the idea in general
16:06:46 <mattmceuen> So looking for prior art, here are the openstack mailing lists:  http://lists.openstack.org/cgi-bin/mailman/listinfo
16:06:50 <alexanderhughes> Roman and I had a similar discussion for readthedocs post job, and the major concern we had was "spam" whatever e-mail address this comes from needs to be easily identifiable so it doesn't go to the spam box
16:06:56 <roman_g> failing docs build jobs are now e-mailing to me.
16:06:59 <dwalt> The idea is just to raise awareness about these issues before they strike us too late. For example, we recently learned we haven't published images in 20 days :o
16:07:10 <roman_g> I can set it to e-mail to the mailing list, I think
16:07:14 <dwalt> roman_g: is that directly from RTD or zuul?
16:07:21 <roman_g> that's from RTD
16:07:34 <roman_g> Zuul has no clue that RTD build has failed (or not)
16:07:45 <dwalt> do your patches make it so?
16:07:47 <roman_g> ^^^^^^^^^^^^^^ - working on it
16:07:48 <mattmceuen> I like the idea of creating a new ML for automation-oriented emails, since it may have a different audience of folks who care than the announce and discuss lists
16:07:56 <mattmceuen> but am open to other thoughts on that
16:08:13 <roman_g> no, it's just a setting to e-mail about failed builds on RTD service
16:09:00 <dwalt> I do see the release-job-failures on your link mattmceuen. Sounds like this is common practice, so that's a good thing
16:09:09 <mattmceuen> Yeah, that's the one that seemed most similar to me too
16:09:41 <dwalt> I also like the idea of a separate mailing list. Does anyone have any thoughts on that?
16:10:16 <mattmceuen> `Airship-automation`or `Airship-bot` or something
16:10:26 <dwalt> roman_g: That is definitely a good start. I would propose we make that the first thing to email the new mailing list
16:10:39 <dwalt> Until we can fully detect failures in the docs zuul job
16:11:01 <dwalt> I do like airship-automation. Or airship-integration
16:11:13 <roman_g> OK. Get new mailing list added, and I will set up RTD to report there (need to have access to airship-treasuremap RTD, Stacey holds the keys)
16:11:40 <dwalt> Another thing we can add to the list is the GitHub mirroring, I see we will be discussing that later. SO if no one has any other thoughts on this, we can get to work!
16:11:42 <dwalt> thanks roman_g
16:11:48 <mattmceuen> Cool.  Sounds like this new ML will have multiple use cases and noone has any concerns?
16:12:10 <kskels> I think I have access to RTD also, I'll ping you Roman sometime tomorrow
16:12:11 <alexanderhughes> nope, good to see some automated alerting on things we can't easily see post merge of patches - changed credentials to quay (image builds), rtd etc
16:12:29 <mattmceuen> awesome.  Moving on then:
16:12:33 <mattmceuen> #topic All deckhand integration gate jobs are failing
16:12:33 <roman_g> kskels: great, thank you.
16:12:42 <mattmceuen> Yes - this one is not good
16:13:08 <mattmceuen> I chatted with Dan Crank on this since he is back from vacation, and dug a little bit this morning
16:13:31 <mattmceuen> I'm able to reproduce the issue in a VM using the `integration-tests.sh` script as well in deckhand
16:14:11 <mattmceuen> deployment of the minimal cluster (in minikube) to run deckhand on top of progresses until it stalls on Keystone
16:14:57 <dwalt> how does it install Keystone? If the Airskiff job is passing, may be worthy taking the same approach
16:15:01 <mattmceuen> Keystone deployment fails with:
16:15:07 <mattmceuen> https://www.irccloud.com/pastebin/ndKQJiP6/
16:15:33 <mattmceuen> Simultaneously to that, the API Server is getting errors like:
16:15:35 <mattmceuen> https://www.irccloud.com/pastebin/8XtzhaYE/
16:15:56 <mattmceuen> The OSH-Infra ansible scripts dwalt - may be pinned to a different version though
16:16:18 <mattmceuen> I haven't had time to dig deeper than that, does this jog anyone's memory for a likely root cause?
16:17:15 <mattmceuen> dwalt:  agree, that's a good idea.  the airskiff job is already using the latest and greatest from osh-infra, so piggybacking on it might even be faster than fixing whatever's going wrong here
16:17:18 <dwalt> Nothing from me -- I was wondering about the version
16:17:41 <dwalt> If we could make the Airskiff stuff  a little more resuable, I agree, that would help
16:18:08 <dwalt> it's hard to maintain the Deckhand jobs, because they're not similar to the rest
16:18:23 <mattmceuen> Ok - I will take this as a priority today unless anyone else would like to volunteer
16:19:13 <mattmceuen> Unless anything else on this, moving on:
16:19:29 <mattmceuen> #topic AirshipCTL spec is up for review
16:20:05 <mattmceuen> Just to raise awareness that jezogwza has a WIP spec up for the airship 2.0 `airshipctl` CLI:  https://review.opendev.org/#/c/659808/
16:20:36 <mattmceuen> He's asked for folks to give it a thorough going-over so we can iterate on it and get it in good shape prior to any implementation work
16:21:19 <roman_g> mattmceuen: would like to have short call w/ you on this matter.
16:21:39 <roman_g> will ping you later
16:21:45 <mattmceuen> sure thing roman_g
16:22:17 <mattmceuen> #topic Certificate update (and how to change them easily)
16:22:30 <mattmceuen> I can't see who added this -- lemko, is it yours?
16:23:13 <mattmceuen> lemko was attempting to update certificate overrides for promenade's (calico) etcd chart, and hit issues
16:23:13 <lemko> Yes it is me
16:23:33 <mattmceuen> logs: http://paste.openstack.org/show/751880/
16:25:09 <mattmceuen> I've been told from the folks that tested password rotation that they've cert rotation has been tested too, so I believe this is either an issue with an old version of the chart lemko has deployed perhaps, or, it's a regression
16:25:23 <mattmceuen> Either way I think we need to test cert rotation using the latest charts
16:25:41 <mattmceuen> lemko would you be able to try reproducing using the latest promenade?
16:26:32 <lemko> Ok I'll try using latest promenade. Thanks
16:26:57 <lemko> I'll upgrade my environment
16:27:20 <mattmceuen> ok, thanks - let me know how that goes!
16:28:16 <mattmceuen> alright, unless anything else on cert rotation, moving on:
16:28:37 <mattmceuen> #topic GitHub Mirrors
16:28:59 <mattmceuen> last week, kskels re-set-up our github mirroring (thanks kskels!!)
16:29:19 <kskels> I did submit a first PS for treasuremap repo only for now to test it..
16:29:19 <mattmceuen> https://github.com/airshipit/
16:29:38 <mattmceuen> evgenyl pointed out:     Should github mirrors work already? https://github.com/airshipit/treasuremap (doesn't look to be up-to-date)
16:29:50 <mattmceuen> Do you expect the sync'ing to be in effect yet kskels?
16:29:53 <kskels> I think it should start working (or failing) on the next PS that comes in
16:30:10 <mattmceuen> fantastic
16:30:13 <dwalt> The post jobs are failing  http://zuul.openstack.org/builds?job_name=treasuremap-upload-git-mirror
16:30:14 <kskels> so the way it works is a POST job for merge..
16:30:17 <kskels> but it's till neter tested
16:30:26 <kskels> so might need some tuning..
16:30:49 <dwalt> it said something about a key, I didn't quite understand where that came into play
16:31:01 <openstackgerrit> Sean Eagan proposed airship/promenade master: Fix potential port conflict  https://review.opendev.org/659369
16:31:26 <kskels> https://www.irccloud.com/pastebin/2ya1018p/
16:31:37 <roman_g> airshipbot@github.com: Permission denied (publickey) ?
16:31:40 <kskels> yeah - I will take a look on this tomorrow once I'm back in the office
16:31:50 <kskels> unless someone can already figure this out
16:32:00 <mattmceuen> oh boy, are you on vacation today kskels?
16:32:06 <kskels> sick :(
16:32:09 <mattmceuen> :(
16:32:18 <mattmceuen> Hope you feel better dude
16:33:17 <mattmceuen> Once we get the key stuff sorted out, this'll be great.
16:33:46 <kskels> right.. we will then need to implement this for all repos.
16:33:50 <mattmceuen> Yup
16:34:12 <mattmceuen> Anything else on Github mirroring, folks?
16:34:50 <mattmceuen> #topic Treauremap-based gates capacity
16:34:55 <mattmceuen> this one is yours roman_g
16:35:25 <roman_g> Yep. There are 2 patches to limit versions.yaml updater to only Ubuntu-based images
16:36:00 <roman_g> We are probably not able to run gates x2 times, to test both ubuntu and opensuse images.
16:36:28 <roman_g> Does anyone have objection or opinion in that?
16:36:43 <roman_g> *on
16:36:56 <mattmceuen> this was discussed a bit in the ptg too
16:36:56 <evgenyl> Yep, definitely not for BM, we can return to this questions when we get multinode and aiab moved into treasuremap.
16:37:01 <dwalt> Is an Airskiff job for opensuse images still in the works? I know we discussed at PTG
16:37:30 <mattmceuen> I like your PS roman_g as it makes the updater script parameterizable (you can choose the distro to update), rather than hard coded to ubuntu, right?
16:37:52 <roman_g> mattmceuen: yes. But pipeline is hardcoded at the moment to ubuntu
16:38:04 <roman_g> *will be
16:38:44 <mattmceuen> yup - and that's fine, because opensuse isn't really planned to be used with maas-based provisioning, so an airskiff-type deployment is a better integration setup until we get ironic support in there
16:39:37 <roman_g> So... appreciate the reviews
16:39:50 <mattmceuen> dwalt I'm not sure if anyone's actively working on the opensuse airskiff setup yet
16:40:01 <roman_g> and there is a need to talk to SUSE on creation of OpenSUSE-ased treasuremap
16:40:08 <mattmceuen> roman_g:  cool I added it to the review request list :)
16:40:29 <dwalt> that makes sense since the Deckhand image is not yet published. Just curious :)
16:40:40 <roman_g> arunkant: are you here? Whom to talk to?
16:41:37 <roman_g> dwalt: this is taken care of. If there is no image mathcing the filter, the first latest tag is being used
16:41:41 <mattmceuen> we may have to catch up with arunkant tomorrow
16:41:57 <dwalt> sorry roman_g, was referring to airskiff
16:42:20 <roman_g> from what I've understood, they are not aware that they potentially would need to build own gate
16:42:30 <roman_g> dwalt: np
16:42:39 <jamesgu__> roman_g: what about new gate?
16:42:56 <roman_g> jamesgu__: good you are here :)
16:43:04 <jamesgu__> distracted, but back
16:43:20 <roman_g> Treasuremap gates are running Ubuntu and will use Ubuntu-based images.
16:43:57 <roman_g> There is probably a need to have OpenSUSE-based third-party gates running from Treasuremap repo
16:44:37 <roman_g> mattmceuen, kskels any comments?
16:44:55 <mattmceuen> jamesgu_, dwalt, remind me:  we were planning on running the opensuse airskiff job on the different projects, right?  Or, was it planned as a periodic job?
16:45:27 <mattmceuen> I.e., run it on openstack infra
16:45:28 <dwalt> I understood it to be on each project, alongside the existing Ubuntu jobs
16:45:35 <mattmceuen> yup - that makes sense to me
16:45:54 <jamesgu__> yes, we talked the possibility of 3rd party for BM, but VM gates is not an issue, i.e., can be in Zuul
16:46:13 <mattmceuen> So one way we could do that would be to add e.g. an airskiff-opensuse site, and then update a site-level versions.yaml with overrides for opensuse images, or something
16:47:31 <jamesgu__> mattmceuen: it makes sense to me. do we need to start with a spec? sounds like you suggest we start with Airskiff gate?
16:48:07 <jamesgu__> treasuremap uplifting gate is the one that needs BM, right?
16:48:18 <roman_g> no spec is probably needed. ideally the only difference between sites should be versions.yaml
16:48:26 <mattmceuen> jamesgu_ :  may be a small enough thing to design for that we don' tneed a spec - do you have anyone who could focus on that, if we discussed it in next week's meeting?
16:48:36 <evgenyl> jamesgu__: We will be able to do uplift using a virtual multinode gate, once it is moved.
16:48:55 <jamesgu__> evgeny1: that's awesome
16:49:02 <roman_g> yes, treasuremap repository. there are 3 sites defined there, one of them is BM seaworthy, others are virtual
16:49:13 <kskels> airsloop is also bare-metal
16:49:18 <jamesgu__> mattmceuen: we will ahve someone to work on that in a couple of weeks I hope.
16:49:19 <roman_g> oh, yes
16:49:27 <roman_g> airsloop is BM
16:49:49 <mattmceuen> jamesgu__ awesome -- whenever that person is able to join, let's have the conversation then so that they can be a part of it, sound good?
16:49:57 <jamesgu__> we can discuss in next week's meeting for sure to get it started
16:50:01 <mattmceuen> perfect
16:50:10 <mattmceuen> we'll plan on that
16:50:29 <mattmceuen> Ok!
16:50:35 <mattmceuen> #topic Requests for Review
16:50:42 <mattmceuen> https://review.opendev.org/#/c/656988/ - Pegleg, decrypt command bugfix
16:50:42 <mattmceuen> https://review.opendev.org/#/c/658380/ - Pegleg, base64 encoding in passphrase catalog (needed for k8s)
16:50:42 <mattmceuen> https://review.opendev.org/660429 Add tag filter parameter to the updater tool
16:50:42 <mattmceuen> https://review.opendev.org/660441 Update Jenkinsfile's to use Ubuntu-based images
16:50:42 <mattmceuen> https://review.opendev.org/#/c/659808/  - airshipctl
16:50:52 <mattmceuen> Some of these have already been discussed here today
16:51:02 <mattmceuen> Please try to give them some review today or tomorrow
16:51:15 <mattmceuen> any others with a big need for review?
16:51:51 <mattmceuen> #topic Roundtable
16:52:07 <mattmceuen> Open mic  - anything else to bring up?
16:52:11 <mattmceuen> Karaoke?
16:52:18 <alexanderhughes> core reviewer updates?
16:53:07 <mattmceuen> oh!  good point - I need to send out an email as well, the one-week deadline snuck up on me :)
16:53:35 <alexanderhughes> time flies on the airship
16:53:36 <mattmceuen> I'd like to welcom alexanderhughes as a core reviewer for sphyglass and pegleg, and ian-pittwood as a core reviewer for spyglass
16:53:41 <mattmceuen> it sure does :)
16:54:12 <mattmceuen> happy to have you guys' continued focus on these projects from a core reviewer perspective!  I will update the necessary checkboxes today
16:54:29 <ian-pittwood> Thanks mattmceuen!
16:54:32 <alexanderhughes> thanks :)
16:54:38 <dwalt> congrats! Thanks for all the hard work, looking forward to working with you both :)
16:54:42 <roman_g> \o/
16:55:13 <mattmceuen> \o/ !
16:55:46 <mattmceuen> Any other items to bring up today, team?
16:56:52 <mattmceuen> in that case, have a great rest of your day / week!  thanks for your time & discussion today
16:57:06 <mattmceuen> #endmeeting