14:01:10 <mark-burnett> #startmeeting airship
14:01:11 <openstack> Meeting started Tue Oct  2 14:01:10 2018 UTC and is due to finish in 60 minutes.  The chair is mark-burnett. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:14 <openstack> The meeting name has been set to 'airship'
14:01:15 <mark-burnett> Well played scott
14:01:39 <aaronsheffield> o/
14:01:43 <seaneagan> o/
14:01:44 <mark-burnett> So we have a light agenda again this week, please feel free to add to it: https://etherpad.openstack.org/p/airship-meeting-2018-10-02
14:01:47 <dwalt> o/
14:02:55 <portdirect> o/
14:03:24 <srwilkers> o/
14:04:47 <mark-burnett> I'm trying to help someone get in irc to talk about the wiki, sorry :)
14:04:51 <alanmeadows> o/
14:05:06 <mark-burnett> Should we talk about armada retries first?
14:05:11 <mark-burnett> #topic Armada Retries
14:05:32 <mark-burnett> Can whoever added that give an overview of the issue?
14:07:59 <portdirect> sure
14:08:55 <portdirect> when running site update, or initial deploy - armada runs helm test, if this fails then armada will remove and redeploy the chart
14:09:22 <mark-burnett> Are you sure?
14:09:27 <mark-burnett> I don't think I've seen that
14:09:29 <portdirect> though sometimes we end up in a situation were armada moves on, as it sees the chart has been deployed on the 2nd run
14:09:51 <seaneagan> if the install/upgrade times out, then helm marks the release as FAILED
14:10:02 <alanmeadows> I believe there is only a removal when there is a helm (tiller) timeout - not on test fail
14:10:09 <seaneagan> in that case on a retry armada will remove and reinstall the release
14:10:20 <portdirect> ah ok
14:10:20 <seaneagan> but shouldn't on a helm test failure
14:10:38 <mark-burnett> Yeah, I have definitely seen both install/upgrade failures resulting in removal and also armada "moving on" the 2nd run
14:10:43 <portdirect> though the end result is the same - effectivly helm test is being bypassed
14:11:08 <seaneagan> here's one idea for how armada can deal better with the retry: https://storyboard.openstack.org/#!/story/2003897
14:11:11 <mark-burnett> So I guess one obvious factor here is that we have historically relied on retries from prom & sy
14:11:42 <mark-burnett> Yeah, that's a nice idea seaneagan
14:11:59 <alanmeadows> Sean should you also paste a link to your statefulness spec? Perhaps its the link above
14:12:07 <portdirect> seaneagan: whats proposed above would improve things in one respect, though i worry that run time would increase a lot?
14:12:17 <mark-burnett> What do you do if something has installed correctly and passed, but then later fails?
14:12:22 <seaneagan> means unnecessary test runs sometimes though, yes
14:12:42 <sthussey> I think rerunning the tests should be a solid default behavior, modified by supplied enabling and blacklisting/whitelisting
14:13:28 <mark-burnett> I mean the test.* section could be updated to include controlling configuration for when the test should be run
14:13:29 <sthussey> Right. If you are in a case where confirming software health is not needed, you can disable testing or whitelist only what you care about
14:13:39 <mark-burnett> So if there are charts with very slow tests then they can be controlled differently
14:14:27 <sthussey> I think in most site update scenarios, positive test results across all deployed charts would be a Good Thing(tm)
14:14:33 <seaneagan> if armada had memory that a particular chart failed previously, then it could only rerun tests/waits in that case
14:14:42 <alanmeadows> Im not sure i fully understand though—this seems like duct tape around not having state (e.g record of it previously passing)
14:15:36 <sthussey> I'd disagree since tests are just point-in-time
14:15:51 <alanmeadows> Rerunning every test every time while potentially something someone would want sounds more like an edge case desire—state tracking seems like the right economical solution (in the currency of run time)
14:15:52 <sthussey> So rerunning them regularly seems like a good operational goal
14:16:21 <openstackgerrit> Kaspars Skels proposed openstack/airship-treasuremap master: Uplift latest charts/images except ceph  https://review.openstack.org/605331
14:16:30 <hogepodge> o/
14:16:37 <portdirect> i think if we were to re-run the regularly - we would also need to improve how the logs from the tests are handled
14:16:42 <alanmeadows> That just seems like a mode armada should offer to your point scott - not as a solution to this problem though
14:16:53 <portdirect> ideally storing them somewhere, that would allow us to retreive them
14:17:05 <sthussey> Though a second avenue of maintaining state of previously failed release tests is fine as well
14:17:08 <sthussey> Isn't that LMA?
14:17:14 <portdirect> rather than relying on the pod deletetion as part of an upgrade hook currently
14:17:44 <portdirect> it in in some cases, but that relys on lma being there, and operational
14:17:53 <portdirect> which wont be the case for many users
14:18:41 <srwilkers> yep - really depends when LMA's deployed, and cases where the LMA components are the releases with failed tests
14:19:08 <seaneagan> this would slightly improve matters for deleting test pods at the right time: https://storyboard.openstack.org/#!/story/2003896
14:19:56 <seaneagan> though agree with the direction of using something like LMA
14:19:59 <sthussey> Seems that is the price you pay with a self-hosted monitoring solution. You have that issue across all logs prior to LMA being deployed. I'm not sure how re-running tests changes this issue significantly.
14:22:04 <sthussey> But to the original topic, it seems the solutions are test-result persistence or rerunning tests with every 'apply'
14:22:42 <sthussey> Does it make sense for Sean to take this issue and these possible solutions offline to write a spec that references other outstanding specs?
14:23:25 <seaneagan> yes, i did make one for state management previously
14:24:06 <seaneagan> https://review.openstack.org/#/c/586582/
14:24:24 <seaneagan> so could be an update to that one or a new one
14:25:35 <mark-burnett> Ok, maybe we can review this offline.
14:25:45 <mark-burnett> #topic New Wiki
14:25:57 <mark-burnett> So we have a new wiki: https://wiki.openstack.org/wiki/Airship
14:26:11 <mark-burnett> Unfortunately, the person who has put the most work into this is having issues getting into irc today
14:26:37 <mark-burnett> I know he wanted to share some points about it, but perhaps it will have to wait until next time
14:27:44 <mark-burnett> Anyway, please take a look and contribute!
14:27:53 <portdirect> looks great
14:27:57 <b-str> This looks nice - a great add.
14:28:05 <mark-burnett> Yeah, it really is a great start :)
14:28:32 <mark-burnett> Ok, looks like there is one more item
14:28:37 <mark-burnett> #topic Marketing 1 Pager
14:28:46 <portdirect> i may have to borrow Lesley to pimp the osh one
14:29:04 <mark-burnett> :)
14:29:24 <mark-burnett> @hogepodge did you want to comment on the 1 pager?
14:29:49 <hogepodge> yes
14:29:54 <hogepodge> We have just two requests.
14:30:19 <hogepodge> The first is clarifying some of the language about the underlying k8s/openstack layer and how that works with the openstack and other application delivery layer
14:30:39 <hogepodge> There are comments in the working doc identifying the two confusing sections.
14:31:10 <mark-burnett> Can you link the working doc again please?
14:31:22 <hogepodge> The second is that we'd like to have a small sidebar that talks about AT&T usage and the origin of the project, but that's a bit more work and we understand that would require AT&T approval.
14:31:52 <hogepodge> link https://docs.google.com/document/d/1BHFZaKDCuoXKjbsRcAOtb2RFuaOThemXCoFMVxXkAMU/edit?usp=sharing Airship One Pager
14:31:56 <hogepodge> #link https://docs.google.com/document/d/1BHFZaKDCuoXKjbsRcAOtb2RFuaOThemXCoFMVxXkAMU/edit?usp=sharing Airship One Pager
14:32:00 <mark-burnett> Thanks :)
14:32:33 <hogepodge> The first is critical, the second is a nice to have if we can get it.
14:32:37 <hogepodge> thank you!
14:33:03 <mark-burnett> Thank you, this is really shaping up :)
14:33:42 <hogepodge> Yeah, everything is basically there. We're producing one for StarlingX also, which is what gives a bit of breathing room to tidy up the final content.
14:33:49 <hogepodge> Thanks to everyone who's contributed to it.
14:34:02 <hogepodge> Questions?
14:35:57 <mark-burnett> Ok, let's close out I guess
14:36:01 <mark-burnett> #topic Roundtable
14:36:02 <openstackgerrit> Andrey Volkov proposed openstack/airship-in-a-bottle master: Bump Shipyard version  https://review.openstack.org/607203
14:36:03 <alanmeadows> Is there an example of the use case sidebar for AT&T usage - e.g. like the kata 1 pager
14:36:05 <mark-burnett> Any other last minute topics?
14:36:07 <mark-burnett> Ah
14:37:03 <portdirect> alanmeadows: the one mattmceuen did for the containers handout was pretty good
14:37:27 <portdirect> back then is made reference to ucp, but i think coul form the basis quite well?
14:38:03 <alanmeadows> sure, material isn't a problem I just meant an example of what the foundation is looking for for this -- for this 1 pager there was a skeleton to clone
14:38:53 <alanmeadows> we can take this one offline, don't need to hold everyone up
14:41:36 <openstackgerrit> Andrey Volkov proposed openstack/airship-treasuremap master: Bump Shipyard version  https://review.openstack.org/607212
14:43:46 <mark-burnett> Sorry for cutting off the conversation.  Anything else?
14:48:45 <mark-burnett> Ok, thanks for coming
14:48:47 <mark-burnett> #endmeeting