15:00:10 #startmeeting openstack-helm 15:00:12 Meeting started Tue Apr 17 15:00:10 2018 UTC and is due to finish in 60 minutes. The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:13 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:15 The meeting name has been set to 'openstack_helm' 15:00:16 #topic rollcall 15:00:35 o/ 15:00:49 o/ cristi 15:00:56 o/ 15:01:01 I am back. :) 15:01:03 hey matt & all 15:01:06 https://etherpad.openstack.org/p/openstack-helm-meeting-2018-04-17 <- agenda for today, please add things you'd like to discuss 15:01:06 o/ 15:01:09 Hey jayahn! 15:02:05 o/ 15:02:06 sorry, i was overwhelmed by some management crab (but important ones) 15:02:13 crap 15:02:21 I feel your pain man 15:02:55 #topic Active/Passive MariaDB 15:03:05 you just gave me a proper name for the crab i just got tattooed on my arm jayahn 15:03:08 "management crab" 15:03:15 +1 15:03:27 lol 15:03:34 criticalin brought this up last time -- A/P MariaDB (not crabs) 15:03:53 I don't think portdirect is able to join today unfortunately 15:04:04 But he's planning on digging into MariaDB resiliency this week 15:04:14 hmm, that's gonna be kind of one-sided then 15:04:27 including mariadb deadlock? 15:04:41 I've had a look on what he was saying about the ipvs implementation of services 15:04:44 His plan is to fix Galera clustering -- we've had a PS out for quite a while but it hasn't gotten to mergability 15:04:51 but I don't think that's gonna help 15:05:12 jayahn, the deadlock is due to the random nature of the k8s services 15:05:19 What is your concern w/ that approach cristicalin? 15:05:21 all writes need to land on the same node 15:05:34 we use haproxy in front but we don't run maria in k8s 15:05:53 we also used haproxy in front at "legacy" openstack 15:05:55 we tried with proxysql but turned out to be too complicated 15:06:16 I like the oracle sql router approach which uses the db to elect the master 15:06:28 though I don't think it works with mariadb 15:06:45 also probably just an 8.0+ feature 15:07:01 we need to scope where these deadlocks occur - as far as im aware only nova and heat suffer from the potential for deadlocks. Im exploring both ipvs (via keepalived) and a proxy approach 15:07:17 o/ portdirect 15:07:23 o/ 15:07:33 actually neutron is the most affected by deadlocks 15:07:36 o/ sorry snowed under today, so may be a bit terse ;) 15:07:52 nova is not so affected unless you throw a ton of workload at it 15:08:16 well then - we need to get them all fixed :) 15:08:19 heat might have some convergence issues, though I haven't tested how far it breaks 15:08:40 you can increment a counter and blow it up 100% of the time 15:08:49 * jayahn think prodirect is great english teacher. I just leanrd "snowed under" today. 15:09:38 currently im thinking that an ingress controller with a custom template may be the cleanest solution 15:10:19 I'm not so familiar with using ingresses 15:10:31 we don't use them at all in our environments 15:10:39 it would provide the same result as std a/p haproxy 15:10:58 but use k8s for backend healthchecks 15:11:29 and what would the ing be based on ? 15:11:31 haproxy 15:11:38 i think so 15:11:52 nginx streams could work 15:12:04 but i think haproxy seems the safest choice here 15:12:15 the most tested at least 15:12:24 though I'd like to test both 15:12:25 I never used nginx in front of mysql 15:12:34 im less concerned about what's most tested, more concerned about what we can put up and see how it works 15:12:48 i agree with portdirect: would be nice to try both and see what works 15:14:44 I'd be happy to ask our guys to help in testing 15:15:28 cristicalin: that would be awesome 15:15:44 soon as i get a wip up I'll add you and jay as reviewer 15:15:46 s 15:16:03 It may be helpful to spec this out to ensure the solution is understood (including alternate approaches like nginx vs haproxy) and what/how we're solving the deadlock issue -- i.e. list out the alternatives and why we arrived at what we think is an optimal solution? 15:16:18 okay. I will ask our guys as well. 15:17:19 that said - I don't want to spin on a spec for long, as this is clearly functionality we all want in place as soon as possible :) 15:20:22 any interest to support alternative mysql implementations ? 15:20:54 like percona or oracle mysql 15:21:30 cristicalin: yes 15:21:44 I'm planning on adding a vitess chart when i find time 15:22:04 hmm, interesting, you mentioned that in the PTG 15:22:12 did you do any tests with vitess yet ? 15:22:59 just very basic ones - but looks promising 15:23:12 neat 15:24:06 alright - anything else on the DB front? 15:25:26 #topic LMA values 15:25:45 jayahn - I haven't dug into the link you sent yet, but it sounded like a good discussion to have here 15:25:59 Quoting 15:26:00 Example armada-manifests skt is using in CI - https://github.com/sktelecom-oslab/armada-manifests/blob/master/taco-ha-example.yaml 15:26:00 srwilkers: "syslog gathering" is enabled in this manifest. what things we want to include upstream lma values? If not, do we want at least a documentation on some lma use cases? 15:26:23 I just lost power in my computer.. typing on mobile. 15:26:38 d'oh 15:26:46 perfect timing :) 15:27:25 my macbook does not act normal. 15:27:28 jayahn we definitely need to document some basic LMA operations 15:27:50 i think getting the kubelet logs via syslog makes sense here. it's a suitable default in my eyes 15:28:47 im currently working on getting elasticsearch to play nice with the s3 api for radosgw -- wanting to get that sorted out so we can also provide some useful defaults or guidance on maintaining indices long term 15:29:12 okay. furthermore, sungil has a question on preparing some reference vlaue sets besides default, something like fluent-bit only case. 15:29:49 we think it will help others. but, question is how? documentation? gating? 15:30:25 jayahn, would it make sense to propose the syslog gathering and any other good defaults as a PS to the fluent chart? Then we can discuss any issues around them as defaults and get 'em merged 15:30:31 documentation makes sense, and maybe explore having a place to put some reference overrides? 15:30:46 srwilkers: yeap. that 15:30:52 mattmceuen: sure 15:31:53 I can type good on mobile. :) 15:32:22 Agree -- there are many opportunities to add additional reference overrides 15:32:42 This is flexible software after all :) 15:33:37 #topic Vancouver 15:33:38 documentation might be enough, but we concerns that it will outdated if there is no check running against thoes ovverides. 15:33:44 d'oh I jumped the gun 15:33:58 There needs to be a countdown for these topic changes 15:34:17 that was just my last comment. :) 15:34:32 #topic one-last-LMA 15:34:41 I will back with PS, we can do more discussion next week 15:34:42 can we gate them? :) 15:34:56 not everyone for sure. :) 15:35:13 so we did not have our answer in team discussion. 15:35:14 what about experimental gating that gets done periodically 15:35:25 probably possible 15:35:50 Id really like to see lma leveraged in the gates 15:36:20 Should be the best way to get logs out... rather than my shanky xargs crazyness 15:36:41 * mattmceuen FYI jayahn I don't think shanky is a word, ignore professor portdirect 15:36:57 i was googling.. again 15:37:09 https://www.merriam-webster.com/dictionary/skanky 15:38:12 * jayahn I heard my team members were discussing how creative professor portdirct's english is. 15:38:21 distinctly not the same thing :) I'm sure we can invent a good software engineering for shanky, I'll add it to a future agenda item 15:38:29 *term 15:38:57 alright - going once 15:39:03 going twice 15:39:13 #topic Vancouver-for-reals 15:39:38 Jay you added this as a "future topic" so we don't have to have the full discussion today 15:39:42 it was a teaser for the next week meeting. 15:39:42 Do we have a spec for the vms yet jayahn ? 15:40:03 As that really deterimes what we can do 15:40:15 vms.. try to recall what it is.. (sorry).. 15:40:42 pls help me. vms.. 15:40:54 The Virtual machines that your going to provide for the workshop 15:41:01 ah.. 15:41:53 okay. I will prepare the spec for the next meeting. it can be two specs with / without lma. 15:42:42 cool 15:42:54 Thanks jayahn 15:43:09 One more thing to whet our appetites -- FYI lots of OSH talks in vancouver :) https://www.openstack.org/summit/vancouver-2018/summit-schedule/global-search?t=openstack-helm 15:44:33 #topic PS Needing Review 15:44:38 https://review.openstack.org/#/c/498929/ 15:45:07 ^ srwilkers would be really nice to get Tempest merged in! What additional review do *you* feel you need on this 15:45:30 just eyes on it really 15:46:18 Cool. Please help with that team and give it a spin for yourself. Can't have too much testing :) 15:46:22 What else guys 15:46:45 Other PS that have been languishing? 15:47:42 3 15:47:43 2 15:47:44 1 15:47:50 #topic Roundtable 15:48:05 Anything else on your minds in the world of OSH or otherwise? 15:49:46 alright guys - I'll give you 10 mins back 15:50:01 Thanks for time & efforts! 15:50:03 thanks. 15:50:11 #endmeeting