15:00:10 <mattmceuen> #startmeeting openstack-helm
15:00:12 <openstack> Meeting started Tue Apr 17 15:00:10 2018 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:15 <openstack> The meeting name has been set to 'openstack_helm'
15:00:16 <mattmceuen> #topic rollcall
15:00:35 <cristicalin> o/
15:00:49 <mattmceuen> o/ cristi
15:00:56 <jayahn> o/
15:01:01 <jayahn> I am back. :)
15:01:03 <cristicalin> hey matt & all
15:01:06 <mattmceuen> https://etherpad.openstack.org/p/openstack-helm-meeting-2018-04-17 <- agenda for today, please add things you'd like to discuss
15:01:06 <srwilkers> o/
15:01:09 <mattmceuen> Hey jayahn!
15:02:05 <lamt> o/
15:02:06 <jayahn> sorry, i was overwhelmed by some management crab (but important ones)
15:02:13 <jayahn> crap
15:02:21 <mattmceuen> I feel your pain man
15:02:55 <mattmceuen> #topic Active/Passive MariaDB
15:03:05 <srwilkers> you just gave me a proper name for the crab i just got tattooed on my arm jayahn
15:03:08 <srwilkers> "management crab"
15:03:15 <mattmceuen> +1
15:03:27 <jayahn> lol
15:03:34 <mattmceuen> criticalin brought this up last time -- A/P MariaDB (not crabs)
15:03:53 <mattmceuen> I don't think portdirect is able to join today unfortunately
15:04:04 <mattmceuen> But he's planning on digging into MariaDB resiliency this week
15:04:14 <cristicalin> hmm, that's gonna be kind of one-sided then
15:04:27 <jayahn> including mariadb deadlock?
15:04:41 <cristicalin> I've had a look on what he was saying about the ipvs implementation of services
15:04:44 <mattmceuen> His plan is to fix Galera clustering -- we've had a PS out for quite a while but it hasn't gotten to mergability
15:04:51 <cristicalin> but I don't think that's gonna help
15:05:12 <cristicalin> jayahn,  the deadlock is due to the random nature of the k8s services
15:05:19 <mattmceuen> What is your concern w/ that approach cristicalin?
15:05:21 <cristicalin> all writes need to land on the same node
15:05:34 <cristicalin> we use haproxy in front but we don't run maria in k8s
15:05:53 <jayahn> we also used haproxy in front at "legacy" openstack
15:05:55 <cristicalin> we tried with proxysql but turned out to be too complicated
15:06:16 <cristicalin> I like the oracle sql router approach which uses the db to elect the master
15:06:28 <cristicalin> though I don't think it works with mariadb
15:06:45 <cristicalin> also probably just an 8.0+ feature
15:07:01 <portdirect> we need to scope where these deadlocks occur - as far as im aware only nova and heat suffer from the potential for deadlocks. Im exploring both ipvs (via keepalived) and a proxy approach
15:07:17 <cristicalin> o/ portdirect
15:07:23 <jayahn> o/
15:07:33 <cristicalin> actually neutron is the most affected by deadlocks
15:07:36 <portdirect> o/ sorry snowed under today, so may be a bit terse ;)
15:07:52 <cristicalin> nova is not so affected unless you throw a ton of workload at it
15:08:16 <portdirect> well then - we need to get them all fixed :)
15:08:19 <cristicalin> heat might have some convergence issues, though I haven't tested how far it breaks
15:08:40 <portdirect> you can increment a counter and blow it up 100% of the time
15:08:49 * jayahn think prodirect is great english teacher. I just leanrd "snowed under" today.
15:09:38 <portdirect> currently im thinking that an ingress controller with a custom template may be the cleanest solution
15:10:19 <cristicalin> I'm not so familiar with using ingresses
15:10:31 <cristicalin> we don't use them at all in our environments
15:10:39 <portdirect> it would provide the same result as std a/p haproxy
15:10:58 <portdirect> but use k8s for backend healthchecks
15:11:29 <cristicalin> and what would the ing be based on ?
15:11:31 <cristicalin> haproxy
15:11:38 <portdirect> i think so
15:11:52 <portdirect> nginx streams could work
15:12:04 <portdirect> but i think haproxy seems the safest choice here
15:12:15 <cristicalin> the most tested at least
15:12:24 <portdirect> though I'd like to test both
15:12:25 <cristicalin> I never used nginx in front of mysql
15:12:34 <srwilkers> im less concerned about what's most tested, more concerned about what we can put up and see how it works
15:12:48 <srwilkers> i agree with portdirect:  would be nice to try both and see what works
15:14:44 <cristicalin> I'd be happy to ask our guys to help in testing
15:15:28 <portdirect> cristicalin: that would be awesome
15:15:44 <portdirect> soon as i get a wip up I'll add you and jay as reviewer
15:15:46 <portdirect> s
15:16:03 <mattmceuen> It may be helpful to spec this out to ensure the solution is understood (including alternate approaches like nginx vs haproxy) and what/how we're solving the deadlock issue -- i.e. list out the alternatives and why we arrived at what we think is an optimal solution?
15:16:18 <jayahn> okay. I will ask our guys as well.
15:17:19 <mattmceuen> that said - I don't want to spin on a spec for long, as this is clearly functionality we all want in place as soon as possible :)
15:20:22 <cristicalin> any interest to support alternative mysql implementations ?
15:20:54 <cristicalin> like percona or oracle mysql
15:21:30 <portdirect> cristicalin: yes
15:21:44 <portdirect> I'm planning on adding a vitess chart when i find time
15:22:04 <cristicalin> hmm, interesting, you mentioned that in the PTG
15:22:12 <cristicalin> did you do any tests with vitess yet ?
15:22:59 <portdirect> just very basic ones - but looks promising
15:23:12 <mattmceuen> neat
15:24:06 <mattmceuen> alright - anything else on the DB front?
15:25:26 <mattmceuen> #topic LMA values
15:25:45 <mattmceuen> jayahn - I haven't dug into the link you sent yet, but it sounded like a good discussion to have here
15:25:59 <mattmceuen> Quoting
15:26:00 <mattmceuen> Example armada-manifests skt is using in CI - https://github.com/sktelecom-oslab/armada-manifests/blob/master/taco-ha-example.yaml
15:26:00 <mattmceuen> srwilkers: "syslog gathering" is enabled in this manifest. what things we want to include upstream lma values? If not, do we want at least a documentation on some lma use cases?
15:26:23 <jayahn> I just lost power in my computer.. typing on mobile.
15:26:38 <mattmceuen> d'oh
15:26:46 <mattmceuen> perfect timing :)
15:27:25 <jayahn> my macbook does not act normal.
15:27:28 <mattmceuen> jayahn we definitely need to document some basic LMA operations
15:27:50 <srwilkers> i think getting the kubelet logs via syslog makes sense here.  it's a suitable default in my eyes
15:28:47 <srwilkers> im currently working on getting elasticsearch to play nice with the s3 api for radosgw -- wanting to get that sorted out so we can also provide some useful defaults or guidance on maintaining indices long term
15:29:12 <jayahn> okay. furthermore, sungil has a question on preparing some reference vlaue sets besides default, something like fluent-bit only case.
15:29:49 <jayahn> we think it will help others. but, question is how? documentation? gating?
15:30:25 <mattmceuen> jayahn, would it make sense to propose the syslog gathering and any other good defaults as a PS to the fluent chart?  Then we can discuss any issues around them as defaults and get 'em merged
15:30:31 <srwilkers> documentation makes sense, and maybe explore having a place to put some reference overrides?
15:30:46 <jayahn> srwilkers: yeap. that
15:30:52 <jayahn> mattmceuen: sure
15:31:53 <jayahn> I can type good on mobile. :)
15:32:22 <mattmceuen> Agree -- there are many opportunities to add additional reference overrides
15:32:42 <mattmceuen> This is flexible software after all :)
15:33:37 <mattmceuen> #topic Vancouver
15:33:38 <jayahn> documentation might be enough, but we concerns that it will outdated if there is no check running against thoes ovverides.
15:33:44 <mattmceuen> d'oh I jumped the gun
15:33:58 <mattmceuen> There needs to be a countdown for these topic changes
15:34:17 <jayahn> that was just my last comment. :)
15:34:32 <mattmceuen> #topic one-last-LMA
15:34:41 <jayahn> I will back with PS, we can do more discussion next week
15:34:42 <mattmceuen> can we gate them? :)
15:34:56 <jayahn> not everyone for sure. :)
15:35:13 <jayahn> so we did not have our answer in team discussion.
15:35:14 <mattmceuen> what about experimental gating that gets done periodically
15:35:25 <jayahn> probably possible
15:35:50 <portdirect> Id really like to see lma leveraged in the gates
15:36:20 <portdirect> Should be the best way to get logs out... rather than my shanky xargs crazyness
15:36:41 * mattmceuen FYI jayahn I don't think shanky is a word, ignore professor portdirect
15:36:57 <jayahn> i was googling.. again
15:37:09 <portdirect> https://www.merriam-webster.com/dictionary/skanky
15:38:12 * jayahn I heard my team members were discussing how creative professor portdirct's english is.
15:38:21 <mattmceuen> distinctly not the same thing :)  I'm sure we can invent a good software engineering for shanky, I'll add it to a future agenda item
15:38:29 <mattmceuen> *term
15:38:57 <mattmceuen> alright - going once
15:39:03 <mattmceuen> going twice
15:39:13 <mattmceuen> #topic Vancouver-for-reals
15:39:38 <mattmceuen> Jay you added this as a "future topic" so we don't have to have the full discussion today
15:39:42 <jayahn> it was a teaser for the next week meeting.
15:39:42 <portdirect> Do we have a spec for the vms yet jayahn ?
15:40:03 <portdirect> As that really deterimes what we can do
15:40:15 <jayahn> vms.. try to recall what it is.. (sorry)..
15:40:42 <jayahn> pls help me. vms..
15:40:54 <portdirect> The Virtual machines that your going to provide for the workshop
15:41:01 <jayahn> ah..
15:41:53 <jayahn> okay. I will prepare the spec for the next meeting. it can be two specs with / without lma.
15:42:42 <mattmceuen> cool
15:42:54 <mattmceuen> Thanks jayahn
15:43:09 <mattmceuen> One more thing to whet our appetites -- FYI lots of OSH talks in vancouver :)  https://www.openstack.org/summit/vancouver-2018/summit-schedule/global-search?t=openstack-helm
15:44:33 <mattmceuen> #topic PS Needing Review
15:44:38 <mattmceuen> https://review.openstack.org/#/c/498929/
15:45:07 <mattmceuen> ^ srwilkers would be really nice to get Tempest merged in!  What additional review do *you* feel you need on this
15:45:30 <srwilkers> just eyes on it really
15:46:18 <mattmceuen> Cool.  Please help with that team and give it a spin for yourself.  Can't have too much testing :)
15:46:22 <mattmceuen> What else guys
15:46:45 <mattmceuen> Other PS that have been languishing?
15:47:42 <mattmceuen> 3
15:47:43 <mattmceuen> 2
15:47:44 <mattmceuen> 1
15:47:50 <mattmceuen> #topic Roundtable
15:48:05 <mattmceuen> Anything else on your minds in the world of OSH or otherwise?
15:49:46 <mattmceuen> alright guys - I'll give you 10 mins back
15:50:01 <mattmceuen> Thanks for time & efforts!
15:50:03 <jayahn> thanks.
15:50:11 <mattmceuen> #endmeeting