15:00:16 <gagehugo> #startmeeting openstack-helm
15:00:17 <openstack> Meeting started Tue Jul 14 15:00:16 2020 UTC and is due to finish in 60 minutes.  The chair is gagehugo. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:20 <openstack> The meeting name has been set to 'openstack_helm'
15:00:34 <gagehugo> #link https://etherpad.opendev.org/p/openstack-helm-weekly-meeting agenda
15:00:36 <stevthedev> Hello
15:00:39 <gagehugo> o/
15:01:23 <andrii_ostapenko> o/
15:01:32 <lamt> \o
15:02:19 <gagehugo> we will start in a couple mins
15:04:54 <gagehugo> ok
15:05:04 <gagehugo> #topic congrats andrii
15:05:12 <stevthedev> \o/
15:05:23 <andrii_ostapenko> thank you :)
15:05:24 <gagehugo> the OSH core team would like to welcome andrii as a core reviewer, thanks andrii for all your hard work!
15:07:01 <gagehugo> and I believe andrii has more things to discuss today too :)
15:07:02 <lamt> \o/
15:07:06 <gagehugo> #topic Improvements in jobs execution time
15:07:16 <gagehugo> andrii_ostapenko: the floor is yours
15:08:49 <andrii_ostapenko> yes. Thank you, gagehugo. I started with small think for now and sorry for commit message. I'm introducing a mechanism to parallelize gate scripts execution that at this point saves 10-20% of execution time. https://review.opendev.org/739659 https://review.opendev.org/739891
15:09:20 <andrii_ostapenko> if we're going in this direction i have several more optimizations in my mind
15:10:01 <andrii_ostapenko> like script refactoring and running all tests in parallel at the end
15:10:36 <gagehugo> The gates could use cleanup like this, so I'm in favor of something like this
15:10:41 <andrii_ostapenko> also looking to switch back to kubeadm but move kubeadm-aio image build to osh images and reuse it in osh during the build
15:11:19 <andrii_ostapenko> this should be evaluated if we can save couple mins there, so rnd for now
15:11:26 <andrii_ostapenko> also pre-pulling some images in parallel
15:11:55 <andrii_ostapenko> for now only https://review.opendev.org/739659 https://review.opendev.org/739891 - i'll prepare commits for review later
15:12:34 <andrii_ostapenko> so if you have some other ideas - please share
15:12:36 <gagehugo> I think there was discussion previously about hosting some images in the infra mirror
15:12:46 <gagehugo> idk what happened there though
15:12:55 <andrii_ostapenko> my goal is to have minimum time for compute-kit ~30 mins
15:13:09 <gagehugo> That would be amazing
15:13:22 <lamt> ++
15:14:02 <andrii_ostapenko> i don't feel we can have an average 30 mins, but at least we should move in this direction
15:15:05 <gagehugo> I'm for it
15:15:06 <andrii_ostapenko> also i noticed it's a bit worse on 8GB nodes. maybe we have memory consumption spikes due to parallel execution. still it's faster than consequent one
15:15:30 <gagehugo> There may be things we can reduce there as well
15:15:34 <gagehugo> replica counts
15:15:43 <andrii_ostapenko> i'll prepare some comparison based on statistics
15:15:45 <gagehugo> min memory values
15:16:00 <andrii_ostapenko> yes. looking at individual charts is the next step
15:16:30 <gagehugo> ok cool
15:17:15 <andrii_ostapenko> minute here, minute there - every minute matters
15:17:36 <stevthedev> ++
15:17:40 <gagehugo> yup, might just be a bunch of small tweaks that add up to larger gains
15:17:49 <gagehugo> #topic osh-infra-images testing
15:18:00 <gagehugo> andrii_ostapenko: this is you as well I believe
15:18:19 <andrii_ostapenko> https://review.opendev.org/#/q/topic:test-osh-images
15:18:41 <andrii_ostapenko> it's a mechanism to reuse osh jobs in osh-images on post-review
15:19:19 <andrii_ostapenko> so what is suggest is to build and upload images during the gates on osh-images and test it there as well
15:20:22 <andrii_ostapenko> also in future I want periodic builds for loci images, testing and promotion for loci images. we often have a situation when the last latest image is more then 1 month old
15:20:41 <andrii_ostapenko> because we build them only on commits to corresponding code in osh-images
15:20:48 <gagehugo> yeah, we've already hit issues where some images are old and out-of-date with upstream changes
15:21:06 <gagehugo> I'm in favor of this as well
15:21:13 <gagehugo> it's been some pain points
15:21:43 <andrii_ostapenko> this topic is ready for review. it provides mechanism and implements it for loci stein and train on gates
15:21:50 <stevthedev> If we build on each commit, why do we need to build periodically?
15:22:19 <andrii_ostapenko> we build on each commit to osh-images to corresponding to particular image code
15:22:35 <andrii_ostapenko> but don't build for new openstack component merges
15:22:52 <stevthedev> Ah okay
15:23:03 <stevthedev> I don't think I run into this problem much with the LMA images
15:23:19 <andrii_ostapenko> yes. we need periodic one only for loci
15:23:25 <gagehugo> It's been a bit annoying with things like the requirements image and stable images for openstack
15:23:39 <stevthedev> Thanks for clarifying for me
15:23:43 <gagehugo> branches get updated with bug fixes or reqs change, but the image is months behind
15:23:52 <stevthedev> Makes sense
15:24:08 <gagehugo> and suddenly images fail because the reqs match up game breaks
15:24:11 <andrii_ostapenko> periodic ones will be available if i manage to merge https://review.opendev.org/#/c/740560/
15:24:35 <andrii_ostapenko> but now we can at least run test deployment on gates and topic is ready for review
15:24:52 <andrii_ostapenko> i'm going to implement this logic for other images, not only loci
15:25:09 <andrii_ostapenko> if we have a corresponding test job in osh or osh-infra
15:25:30 <stevthedev> I think thats a good idea. I remember Steve W wanting more robust testing between osh-infra and osh-images too
15:25:51 <andrii_ostapenko> e.g. for some changes in kibana image we can run osh-infra-logging job
15:26:02 <stevthedev> ++
15:26:53 <andrii_ostapenko> also i want to get rid of some duplication across the repos https://review.opendev.org/#/c/740530/
15:27:23 <andrii_ostapenko> and inherit from infra job and reuse osh-infra-gate-runner instead of having it in both repos
15:27:26 <stevthedev> I will go through the chain of patches and take a look
15:27:31 <andrii_ostapenko> thank you
15:28:05 <gagehugo> sounds good!
15:28:14 <gagehugo> thanks andrii!
15:28:24 <gagehugo> #topic Open Discussion/Reviews
15:28:37 <gagehugo> #link https://review.opendev.org/#/c/740825/ node-down detector
15:28:46 <gagehugo> #undo
15:28:47 <openstack> Removing item from minutes: #link https://review.opendev.org/#/c/740825/
15:28:50 <stevthedev> I want to pass on a Q from my team
15:28:53 <stevthedev> Radhika has been working on a dockerfile for the node-problem-detector. The dockerfile merged, and she is following that up with a zuul job for the image (https://review.opendev.org/#/c/740825/). The job log shows the image was built successfully, but we don't see it on dockerhub (https://hub.docker.com/u/openstackhelm/)
15:28:55 <gagehugo> #link https://review.opendev.org/#/c/740825/ node problem detector
15:29:17 <gagehugo> yeah it may need to get added
15:29:26 <stevthedev> Is there a missing component, does the repo need to be created manually, will this get fixed when PS merges?
15:29:30 <andrii_ostapenko> it will be uploaded after merge only
15:29:31 <gagehugo> but also there's credentials I believe in the repo
15:29:57 <andrii_ostapenko> actually it's being uploaded on post-review
15:30:19 <andrii_ostapenko> with temp tag
15:30:28 <andrii_ostapenko> and promoted to defined tag after the merge
15:31:25 <andrii_ostapenko> image is not uploaded to dockerhub in untrusted pre-review pipeline in case if zuul-jobs *-docker-image jobs are used
15:32:24 <stevthedev> Upon merging https://review.opendev.org/#/c/740825/ do we have to make a change to the dockerfile to trigger the build? Or will it run once the zuul addition is merged?
15:33:45 <stevthedev> to rephrase that - is it merge to the image directory that triggers the build, or any change to osh-images
15:34:04 <andrii_ostapenko> - node-problem-detector/.*
15:34:10 <andrii_ostapenko> - zuul.d/node-problem-detector.yaml
15:34:23 <andrii_ostapenko> so it's defined in zuul.d/node-problem-detector.yaml
15:34:35 <andrii_ostapenko> when to run it
15:34:42 <stevthedev> Ah I see now. ty
15:38:01 <gagehugo> Anyone else have anything for this week?
15:38:13 <andrii_ostapenko> that actually have a relation to testing an image
15:38:31 <andrii_ostapenko> we'll be able to do it on post-review for this image as well
15:39:16 <andrii_ostapenko> if there's any job in osh or osh-infra that deploys this chart
15:39:17 <stevthedev> Yep :)
15:39:31 <andrii_ostapenko> but not in pre-review
15:39:46 <stevthedev> I'm not sure if it's been included in an osh-infra job yet. But I'll have to double check
15:39:51 <stevthedev> might be in monitoring?
15:40:52 <andrii_ostapenko> https://review.opendev.org/#/c/668754/
15:41:11 <andrii_ostapenko> aio-monitoring
15:44:20 <gagehugo> Thanks everyone have a good rest of the week
15:44:24 <gagehugo> #endmeeting