19:01:05 <clarkb> #startmeeting infra
19:01:05 <openstack> Meeting started Tue Jul  9 19:01:05 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:08 <openstack> The meeting name has been set to 'infra'
19:01:10 <mordred> o/
19:01:23 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-July/006410.html Meeting Agenda
19:01:38 <clarkb> #topic Actions from last meeting
19:01:44 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-07-02-19.01.txt minutes from last meeting
19:02:05 <clarkb> mordred: any progress with the github account creation and -infra org cleanup?
19:02:16 <mordred> nope
19:02:40 <fungi> jroll sent a followup agreeing with ttx's suggestion about repository descriptions
19:02:50 <fungi> (on the openstack-discuss ml thread)
19:02:57 * mordred missed that - goes to look
19:03:09 <jroll> apologies for that dragging, am just looking for volunteers to help at this point
19:03:27 <clarkb> #action mordred create opendevadmin github account
19:03:34 <clarkb> #action mordred clean up openstack-infra github org
19:03:42 <fungi> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007607.html Assuming control of GitHub organizations
19:04:28 <fungi> jroll: yes, volunteers! (and if there are none, maybe that opens the door for alternative choices about whether mirroring there is even important?)
19:04:50 <jroll> indeed
19:05:32 <AJaeger> seeing how often people link to github on #openstack-infra: Many are used to it - but seems it might not be important enough to take action
19:05:57 <fungi> they can always get used to linking something else ;)
19:06:20 <AJaeger> exactly
19:07:49 <clarkb> seems like we usually get volunteers when we put it in those terms
19:08:02 <clarkb> I guess if no one shows up in the near future we can always bring that up as the alternative
19:08:11 <clarkb> (as a non github user I'm not terribly interested myself :) )
19:08:36 <clarkb> #topic Priority Efforts
19:08:42 <clarkb> #topic OpenDev
19:09:14 <clarkb> I'm not aware of anything new on the opendev side of things since last week. Anyone else have something to share here?
19:10:05 <fungi> the mirror stuff but that's got its own topic
19:11:14 <clarkb> #topic Update Config Management
19:11:49 <clarkb> corvus: has been improving our job setups and has a change into zuul which will allow us to avoid testing everythign in system-config when zuul.yaml updates
19:12:04 <clarkb> #link https://review.opendev.org/669752 Test job updates without explicit file matcher
19:12:34 <clarkb> This should help us move more quickly on our ansible changes as we won't end up testing the world (I think it will help with some of the other efforst we have to add jobs as well)
19:12:51 <corvus> yeah, with that we can really scale up the number of system-config jobs
19:13:56 <mordred> ++
19:15:01 <clarkb> will reduce the number of test nodes we need for system-config changes too which should make everyone happy
19:16:50 <clarkb> Any other config management improvement updates?
19:17:26 <corvus> mordred: let's you and i sync up after the meeting and figure out next steps for the gitea project creation change?
19:17:34 <corvus> i think we said we'd revisit that this week
19:17:48 <corvus> one of us should start on that soon :)
19:18:26 <mordred> corvus: kk
19:19:06 <clarkb> #topic Storyboard
19:19:13 <clarkb> fungi: diablo_rojo any udpates to share?
19:19:37 <diablo_rojo> clarkb, got some of our migration tracking stuff up to day
19:19:39 <diablo_rojo> date
19:19:52 <diablo_rojo> poked at a few more teams to see if they want to migrate, havent gotten responses yet
19:19:57 <fungi> i took a vacation ;)
19:20:17 <diablo_rojo> Would be great if we can get some help from mordred on the database indexing and whatnot to speed search up
19:20:31 <diablo_rojo> Since our intern got a job and left us :'(
19:21:45 <AJaeger> diablo_rojo: might want to followup on https://review.opendev.org/651942 as well
19:22:14 <diablo_rojo> AJaeger, thanks for the heads up. Will do!
19:22:29 <diablo_rojo> clarkb, thats all I have really
19:22:44 <mordred> diablo_rojo: oh no! (although awesome for the intern)
19:23:56 <diablo_rojo> mordred, yes, happy for her also, selfishly, very sad for StoryBoard and the amount of effort I put in to get an intern
19:24:13 <mordred> yeah
19:24:30 <clarkb> #topic General Topics
19:24:37 <clarkb> First up checking in on our trusty life support situation
19:24:49 <clarkb> fungi: were you able to get your change to pass tests after we switch to the legacy base jobs?
19:25:01 <clarkb> and if so has that change been merge yet?
19:25:38 <fungi> yeah, it worked and has merged
19:26:02 <fungi> and just a bit ago i cleaned out the weirdly miscloned files where vcsrepo wants to put its repos
19:26:10 <clarkb> cool I guess its rerun and see what the next errors are?
19:26:12 <fungi> so waiting to see if puppet will dtrt now
19:27:03 <clarkb> Ok next on the agenda was an update on the afs and mirrors situation. Sounds like dhowells is one vacation for a bit as is ianw so we've stopped using the kafs mirror.
19:27:07 <fungi> looks like the redirect was not the problem for those though
19:27:14 <clarkb> :(
19:27:19 <fungi> so still hunting for an explanation
19:27:27 <clarkb> could be submodule related?
19:28:03 <clarkb> I did want to ask if we think we should go ahead and replace all the mirrors with bionic + 1.8.3 at this point. That setup seems stable and will get us LE certs and tls
19:28:27 <AJaeger> nit: ianw mentioned 1.8.4 AFAIR in his email
19:28:58 <clarkb> ah well whatever is in the infra PPA :)
19:29:14 <fungi> seems fine to me
19:29:19 <clarkb> I'm not sure this is urgent but does potentially open the door to more interesting job activities
19:29:23 <AJaeger> 1.8.3 - link is http://lists.openstack.org/pipermail/openstack-infra/2019-July/006411.html
19:29:27 <clarkb> (they can more strongly trust our mirrors if so)
19:29:47 <AJaeger> so, I misremembered
19:30:06 <corvus> it might be worth figuring out how to build these packages institutionally, rather than the ianw pushes to ppa approach?  i get the feeling we may be dependening on these for a while, and we may need to update them.
19:30:16 <fungi> most of my memories are misrememberies
19:31:09 <fungi> i think the goal was to convince ubuntu package maintainers to either update the prerelease they shipped in bionic, or get it added to bionic-backports?
19:31:35 <corvus> we never have come up with a good way to do that though.  maybe we could have a zuul post job push the config to the ppa?  i've forgotten how ppas work though.
19:31:49 <fungi> afs folks seem a bit unnerved by ubuntu deciding to distribute a prerelease in their lts distro
19:32:09 <clarkb> fungi: ya I think the two ideas ianw had were to either use kafs or have ubuntu update their clearly broken package(s)
19:32:10 <corvus> it's not ideal
19:32:39 <clarkb> we've found kafs needs more work (which we are helping with but still needs time) and I'm never sure how possible updates to ubuntu packaging is
19:33:09 <fungi> i think the only real benefit to the ppa is that we can have it build arm64 packages
19:33:56 <corvus> working on a way to maintain the packages as a team rather than an individual effort gives us a backup plan in case ubuntu doesn't jump when we say, and gives us options if we need to update versions.
19:34:04 <clarkb> corvus: ++
19:34:26 <corvus> maybe a topic for next week's meeting, assuming ianw is back?
19:34:28 <fungi> running debuild on a test node, running apt-ftparchive over that, generating a detached release file on the executor and pushing that somewhere wouldn't be complicated, and we do have arm64 nodes too
19:34:42 <clarkb> thinking out loud here we should probably do both things then? ubuntu should ideally have working openafs and we can also manage the ppa more programatically
19:35:02 <clarkb> corvus: and ya maybe we let ianw chime in when back and then go from there
19:35:06 <corvus> yeah, ubuntu having working afs is definitely the ideal
19:35:13 <corvus> i just don't want to be caught without a backup plan
19:36:23 <fungi> makes sense, for sure
19:37:03 <clarkb> sounds like a plan we'll bring it up with ianw and work towards sustainable bionic openafs
19:37:16 <clarkb> The last thing I had on the agenda was an update of the fortnebula cloud addition
19:37:38 <clarkb> I had hoped we'd be "done" at this point but have run into glean + centos/fedora + network manager + ipv6 only cloud problems
19:37:52 <clarkb> gentoo images also don't work but for different reasons
19:38:21 <corvus> (what's the gentoo problem in 6 words or less?)
19:38:36 <clarkb> Where I am at in the debugging is I have nodepool's test centos image with new glean unit file and old unit file ready to upload into fortnebula to boot test nodes there. And have also held a nodepool test job so I can figure out why our supposed fix doesn't work in testing
19:38:51 <clarkb> the gentoo problem is similar to the centos problem. basically no ipv6 addr configured on the interface
19:39:00 <corvus> okay, just different tooling
19:39:05 <clarkb> but gentoo uses systemd-networkd and not networkmanager so the mechanism of failure should be different
19:39:07 <clarkb> yup
19:39:23 <fungi> that's more than six words ;)
19:39:26 <clarkb> :P
19:39:30 <mordred> fungi: you're more than six words
19:39:41 <fungi> yup yup yup yup yup yup yup
19:39:45 <clarkb> what I'd like to do with the images I'm uploading to fn is confirm the fix works there so I can turn my focus to the test jobs
19:39:54 <clarkb> and I guess I'll find what I find
19:40:14 <clarkb> I did test all of the other images one by one and they all seemed to work (ubuntu, opensuse, debian)
19:41:21 <fungi> and the fix which isn't working in testing did work when booting the nodes there directly (outside of our nodepool test job), right?
19:41:58 <clarkb> fungi: yes, but that was on modified images in place (i updated the unit file, reloaded the systemd daemon config, deleted the /etc/sysconfig/network-scripts/ifcfg-eth0 file then rebooted)
19:42:32 <clarkb> its possible that something else related to booting a second time is causing it to work which is why I'm now going to double check with fresh images directly from the test job (I copied the image out of the test job)
19:43:02 <fungi> sure
19:43:27 <clarkb> so ya I'll be poking at that today and into the near future probably
19:43:35 <clarkb> And that takes us to the end of our agenda
19:43:40 <clarkb> #topic Open Discussion
19:43:48 <clarkb> Anything else?
19:45:29 <AJaeger> FYI: fungi has a change up to switch to ansible-lint 4 in zuul-jobs, see https://review.opendev.org/667695. I plan to +2A by end of week if there are no objections.
19:45:58 <clarkb> thank you for the heads up
19:46:55 <fungi> pay attention to the child changes of that one, since they do give us some indication of where ansible-lint devs are headed and whether it will be an increasingly steep treadmill over time
19:47:13 <clarkb> fungi: maybe we are in a position to give feedback to those devs?
19:47:35 <clarkb> "hey here is a bunch of real world ansible and it doesn't beneift from x or y but could really use a check for z" type of feedback maybe
19:47:45 <AJaeger> one of those child changes needs a fix in ansible-lint, others might have a -1
19:48:18 <fungi> sure, or to decide whether ansible-lint is (or even wants to be) an effective tool for this particular use case
19:49:28 <corvus> yeah, i'm unconvinced that linting for code style is a useful thing.  we've wasted hundreds of person-hours on it.  but i've tried to give those changes a fair evaluation in the spirit of compromise.
19:49:31 <fungi> i split the "fixes" for each new rule into separate changes so each can be evaluated on their own merits and included or abandoned as we choose
19:49:45 <corvus> if someone wanted to remove it entirely, i would happily +2 those patches.
19:49:58 <corvus> and i think we should all be aware that it may come to that depending on the direction that ansible-lint takes.
19:50:08 <clarkb> ya what I'd really like are linter checks for deprecated behavior in ansible
19:50:19 <clarkb> but we've got a plan for finding and addressing those by running ansible instead
19:50:23 <mordred> yeah. and things that are actually catching gotcha behavior
19:50:29 <corvus> yes.  love that.
19:50:55 <mordred> "I don't think that means what you think it means"
19:51:02 <mordred> helpful
19:51:07 <mordred> "I don't like the color of your shirt"
19:51:08 <mordred> not helpful
19:51:11 <corvus> ++
19:51:22 <fungi> inconceivable
19:51:32 * corvus is wearing a *black* shirt, so i assume that's okay ;)
19:51:51 * mordred is not wearing a shirt - given it's the carribean and there is no A/C where he's sitting
19:52:00 <corvus> this is why we irc
19:52:05 <mordred> corvus: ++
19:53:02 <clarkb> its actually cool and rainy here
19:53:02 <corvus> clarkb: you said something about a short meeting? :)
19:53:11 <clarkb> corvus: ya we better stop now to keep to that :)
19:53:14 <clarkb> thanks everyone!
19:53:18 * fungi wore shorts to the meeting
19:53:24 * AJaeger too
19:53:24 <corvus> haha
19:53:29 <clarkb> find us in #openstack-infra or at openstack-infra@lists.openstack.org
19:53:32 <clarkb> #endmeeting