19:03:42 <clarkb> #startmeeting infra
19:03:43 <openstack> Meeting started Tue Aug 22 19:03:42 2017 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:03:44 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:03:46 <openstack> The meeting name has been set to 'infra'
19:03:51 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:04:03 <clarkb> #topic Announcements
19:04:15 <clarkb> PTG is quickly approaching.
19:05:08 <clarkb> Pike is right in the middle of release fun as release candidates come out and all that (so we should be careful to be slushy and avoid breaking things)
19:05:25 <clarkb> That is all I had. Anyone have anything to add?
19:06:05 <ianw> clarkb: just that mirror nodes should be Xenial now, so if anyone has issues we're not aware of, let us know
19:06:42 <clarkb> ya I've added that to general discussion so that we can talk about it a bit more
19:06:46 <clarkb> #topic Actions from last meeting
19:06:59 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-08-15-19.03.txt Minutes from last meeting
19:07:17 <cmurphy> o/
19:07:29 <clarkb> ianw: I'm guessing that mirror update hasn't been updated to xenial yet. Have you heard from the release team on whether or not we can move forward on that?
19:08:52 <clarkb> I completely failed to send the gerrit upgrade email last week so shoudl get that done nowish. And I haven't seen word of infracloud switchport counts
19:08:58 <clarkb> #action fungi get switchport counts for infra-cloud
19:09:08 <clarkb> #action clarkb send reminder of Gerrit upgrade after the PTG to the dev mailing list
19:09:30 <ianw> i also completely failed on that, sorry
19:09:41 <clarkb> #action ianw upgrade mirror-update server and bandersnatch
19:10:09 <clarkb> that is all we had from last meeting. So moving on
19:10:11 <clarkb> #topic Specs approval
19:10:41 <clarkb> #link https://review.openstack.org/#/c/462207/ is an update to the zuulv3 spec around zuul's handling of ssh keys
19:10:53 <clarkb> jeblair: SpamapS ^ is that generally ready for approval?
19:11:39 <jeblair> clarkb: yes, sorry!
19:11:52 <clarkb> I think we may also want to add in the way we switch out the ifnra key for the job key?
19:12:34 <clarkb> aiui pre playbook removes infra key from agent and adds in a per job key? Not sure if you want that to be a separate spec update though. In any case I think getting that updated as necessary would be good
19:12:43 <jeblair> clarkb: we could -- though that's already implemented so we don't really need to drive consensus on it.  may be better to find a good place to document that for posterity.
19:12:55 <jeblair> maybe expand our base job documentation
19:13:00 <clarkb> wfm
19:13:19 <clarkb> The other spec I wanted to bring up was mentioned last week.
19:13:40 <clarkb> #link https://review.openstack.org/#/c/492287/ mark gerrit contact store spec as completed
19:13:47 <pabelanger> o/
19:13:49 <clarkb> It has a couple depends on that could use reviews from infra-roots
19:14:14 <clarkb> basically all cleanup at this point, pretty straightforward.
19:14:38 <clarkb> Are there any other specs that we should bring up today?
19:16:28 <clarkb> sounds like not. Would be great to get 462207 reviewed and I expect that fungi will merge that when he gets back (I don't have approval bits in that repo yet). And reviews to finish up the contact store priority effort much appreciated as well
19:16:33 <clarkb> #topic Priority Efforts
19:16:52 <clarkb> I don't think there is much else to add on the gerrit upgrade or contact store front.
19:17:00 <clarkb> jeblair: anything from zuulv3?
19:17:18 <jeblair> nope, still chugging along
19:18:09 <clarkb> #topic General topics
19:18:40 <clarkb> One of the big things that happened last week was we upgraded our region local mirror instances to xenial from trusty in an effort to get more reliable http servers and afs
19:19:03 <clarkb> We weren't sure if it would fix all the problems but ianw pointed out debugging a current lts is far more beneficial than the last one
19:19:12 <pabelanger> Ya, switching to xenial has been an improvement from what I see
19:19:47 <pabelanger> also the patch for htcacheclean 4096 directories is also working great
19:19:59 <ianw> clarkb: only one in the list still is OVS, is that ready?
19:20:13 <clarkb> yup I agree. Though I did notice that yesterday we had 4 cases of pip hash mismatches in rax-dfw all within a minute of each other so I think we may still have some minor issues around afs (assuming it was an afs problem in the first place)
19:20:13 <ianw> "the list" being https://etherpad.openstack.org/p/mirror-xenial for reference
19:20:38 <clarkb> #link http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22THESE%20PACKAGES%20DO%20NOT%20MATCH%20THE%20HASHES%20FROM%20THE%20REQUIREMENTS%20FILE.%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1&from=864000s for query on pip hash mismatch
19:20:50 <clarkb> #link https://etherpad.openstack.org/p/mirror-xenial for finishing up mirror upgrades
19:21:09 <clarkb> ianw: I didn't really want to touch them until we got the account situation sorted out there just to avoid potentially causing more problems
19:21:11 <ianw> hmm, i'll look into that, because a cluster is consistent with the segfaults we saw
19:21:22 <pabelanger> ianw: clarkb: the ttl for proxypass also seems to be an improvement for 502 proxy errors. I haven't see any issues from buildlogs.centos.org today
19:21:28 <cmurphy> hmm i noticed that pycrypto doesn't build properly when using the mirrors, possibly related to the mirror update? https://review.openstack.org/#/c/496027/1
19:22:09 <ianw> cmurphy: hmm, that shouldn't be different ...looking
19:22:25 <clarkb> pabelanger: that is good to know thanks.
19:23:05 <clarkb> jeblair: you haven't heard anything back from jean daniel have you? Checking my email I don't have anything
19:23:19 <cmurphy> ianw: those tests try and fail to install gerritbot which fails building pycrypto, but it seems to work fine on the same vm without the pip.conf changes
19:24:17 <jeblair> clarkb: negative
19:24:43 <mordred> whoops. /me waves
19:24:47 <ianw> cmurphy: that seems to be a problem with the host side, rather than the mirror side?
19:25:21 <clarkb> #action clarkb follow up on ovh account situation and make sure mirrors get rebuilt there as xenial
19:25:33 <ianw> cmurphy: ill comment
19:25:34 <clarkb> I'll go ahead and try to track that down now that we expect people to be back from vacation aiui
19:26:18 <clarkb> The other bigish thing that happened recently was we no longer allow glean to update dns resolvers based on dhcp content in our images
19:26:42 <clarkb> we were having dns resolution problems in clouds like rax and it turned out we weren't using our local unbound install, we were hitting the cloud resolvers instead
19:26:45 <pabelanger> Yes! That has helped bigtime for centos
19:26:52 <clarkb> pabelanger did a bunch of work in glean to fix that.
19:26:59 <pabelanger> now that we are using unbound, DNS failures are basically gone
19:27:10 <clarkb> except for on suse where we now have the problem of using unbound :)
19:27:24 <clarkb> #link https://review.openstack.org/#/c/496310/ should fix unbound for suse if people can review that today
19:27:36 <Shrews> git branch
19:27:38 <clarkb> #link https://review.openstack.org/#/c/496341/1 is alternate suse fix
19:27:43 <Shrews> oops. sorry
19:27:59 <clarkb> there ended up being two changes pushed for that, if we prefer one more than the other I will be sure to abandoned the second one
19:28:08 <pabelanger> k, I'll look at suse fixes after meeting
19:28:17 <clarkb> thanks
19:28:53 <clarkb> Overall I think these changes are doing quite a bit to make test jobs more reliable so thank you to everyone that helped get them in place
19:29:15 <pabelanger> Ya, tripleo gate pipeline hasn't reset in 24 hours now
19:30:09 <clarkb> The last item I had for general discussion was something that happened this morning. nose-html-output is still in use by a couple projects (horizon and swift I think?) Its basically unsupported by us due nose being highly discouraged as a test runner and lack of subunit. As a result I have added amotoki to the core list as they have an interest in keeping the tool running at least for the short
19:30:11 <clarkb> term
19:30:27 <clarkb> if there are other individuals interested in supporting that project I think we should just go ahead and add them to the core group
19:30:52 <clarkb> (good news is sounds like there will be new investigation of getting horizon to emit subunit so yay)
19:31:13 <pabelanger> nice
19:31:54 <clarkb> #topic Open discussion
19:32:21 <clarkb> Anything else?
19:32:31 <ianw> anyone mind if i get to a bindep release today?
19:32:42 <ianw> there's some rpm changes that will be helpful for testing python3.6
19:32:46 <pabelanger> can we remove fedora-25 DIB?
19:33:19 <clarkb> re bindep any concern about that affecting the release? IIRC the unittest are pretty robust since its basically just a language parser so probably fine
19:33:24 <pabelanger> also, somebody asked if we could remove debian-jessie too. Since openstack-deb team is getting disbanded
19:33:28 <ianw> pabelanger: not quite, but i can do some cleanup today to get us there.  fedora-26 is ok with devstack now
19:33:43 <clarkb> pabelanger: ya I had asked about debian but fungi pointed out projects like dib are using the image so I think we can keep it around
19:34:00 <dmsimard> For open discussion, just wanted to mention I officially got the green light from Red Hat to spend time on upstream/infra and help you guys out. My first work item is to improve devstack-gate bash/ansible things.
19:34:04 <ianw> clarkb: i think, looking at the changes, low risk, mostly just the changes to add "--whatprovides" to rpm calls
19:34:10 <cmurphy> i have a mass approval request if anyone wants some internet points https://review.openstack.org/#/q/status:open+topic:refactor-infra-beaker the idea is to hopefully reduce the need for mass updates
19:34:12 <pabelanger> k, the reason I ask, is we only have 100Mbps link on nb03.o.o, so it does take some time for new images to upload.
19:34:40 <pabelanger> rackspace is crazy fast, like 30mins to upload 80GB
19:34:41 <ianw> i don't think dib is testing on debian
19:34:42 <jeblair> dmsimard: yay!  thanks!
19:34:56 <clarkb> dmsimard: nice! and thank you
19:35:18 <pabelanger> dmsimard: ^5
19:35:20 <clarkb> ianw: did that change to add the output of all deps get merged? I +2'd it but I think the submitter would like it if it was in the next release
19:35:48 <clarkb> it did merge so thats good
19:36:13 <clarkb> #link https://review.openstack.org/#/q/status:open+topic:refactor-infra-beaker mass approval revuest to reduce the need for mass updates
19:36:47 <ianw> clarkb: https://review.openstack.org/#/c/492693/ ? yep
19:36:58 <clarkb> ianw: ya
19:37:21 <ianw> the only outstanding changes are refactor ones ... bit iffy on code motion without other reasons for it
19:37:26 <clarkb> pabelanger: probably the next step is do a job listing for the debian image and see if its just our bindep fallback test and the debian packaging jobs
19:38:35 <pabelanger> clarkb: agree
19:39:00 <clarkb> cmurphy: I'll try to review that after lunch
19:39:10 <cmurphy> ty clarkb
19:41:27 <clarkb> for those of us going to the ptg is there interest in trying to have a team dinner again? anyone interested in lining that up?
19:42:02 <clarkb> that might be better off as a ml thread
19:42:27 <pabelanger> +1
19:43:36 <clarkb> I'll give it to 19:45UTC before ending the meeting but beginning to sound like that is it
19:45:24 <clarkb> ok thank you everyone. You can find us in #openstack-infra if we missed anything or you remember something important
19:45:32 <clarkb> #endmeeting