19:01:18 <clarkb> #startmeeting infra
19:01:18 <openstack> Meeting started Tue Nov 20 19:01:18 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:22 <openstack> The meeting name has been set to 'infra'
19:01:35 <gary_perkins> o/ is here for the meeting :)
19:02:31 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:42 <clarkb> #topic Announcements
19:03:13 <fungi> i missed adding a reminder about the largeish ml merge i enacted yesterday
19:03:19 <clarkb> This week is a big holiday week in the USA. This means in addition to jet lag lag you'll likely find that many of us in this country will be ignoring our computers later this week
19:03:22 <frickler> o/
19:03:26 <rfolco> o/
19:03:33 <ssbarnea|bkp2> o/
19:03:35 <clarkb> fungi: that seems like it is worth a reminder here though
19:03:50 <fungi> note: i'm not ignoring my computer, i'm just ignoring people i don't like (kidding!)
19:03:52 <clarkb> fungi did the openstack-dev/etc into openstack-discuss switch as discussed for a few months since the PTG
19:04:06 <clarkb> if you haven't joined the -discuss list now is an excellent time to do so
19:04:25 <fungi> we're almost at 350 subscribers now
19:04:33 <AJaeger> o/
19:04:53 <clarkb> I'm personally intending to try and be around today and tomorrow with various errands interspersed in today (to convince my body that it shouldn't fall asleep at 5pm). But then I'm afk Thursday to Monday
19:05:24 <fungi> i'll be offline some of my usual thursday to meet friends for chow
19:05:44 <fungi> otherwise i should mostly be around
19:05:45 * AJaeger is not celebrating Thanksgiving but travelling all of next week...
19:05:48 <anteaya> happy thanks chow
19:05:53 <fungi> as much as i ever am the week after a summit anyway
19:06:12 <clarkb> #topic Actions from last meeting
19:06:20 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-11-06-19.01.txt minutes from last meeting
19:06:39 <clarkb> No explicit actions called out and considering we have a lot ot talk about post summit I'm inclined to push forward
19:06:55 <clarkb> #topic Priority Efforts
19:07:06 <clarkb> #topic Update Config Management
19:07:21 <clarkb> I believe the changes that ianw wanted to get in to make dockering possible have happened
19:07:43 <clarkb> other than that I'm not sure we've made much progress on this effort since we last met since summit and associated demands were happening
19:08:15 <ianw> https://review.openstack.org/#/c/605585/ is ready for review, that installs docker on control plane
19:08:22 <corvus> sorry i'm late
19:08:22 <clarkb> mordred: ianw Mathias Gutehall mentioned on the infra mailing list that there was interest in helping with the dockering. I've suggested that you two may be the best points of contact for that
19:08:36 <clarkb> I'm happy to help there as I can but most likely as a reviewer
19:08:50 <ianw> yes, we've had a brief chat.  i think the next thing to do is to get a hello-world type thing working ontop of 605585
19:08:55 <clarkb> #link https://review.openstack.org/#/c/605585/ install docker on the control plane (ready for review)
19:09:34 <ianw> then get an ipv6 hello-world working too ... this will i think be harder, but required for anything we deploy
19:09:35 <corvus> ianw: awesome!
19:09:37 <clarkb> ah I know this change. I'll rereview it before I take off for errands
19:10:17 <ianw> clarkb: it's a unix system, i know this!
19:10:27 <clarkb> exactly
19:10:37 <clarkb> velociraptors don't stand a chance
19:11:01 <corvus> why don't the iptables tests work?
19:11:26 <clarkb> corvus: I had to update the iptables tests to accomodate the stuff docker adds
19:11:28 <corvus> like -- i get that dockers doing other weird stuff, but shouldn't the rules we check for still be there?
19:11:52 <clarkb> corvus: yup they are still there, but the order changes (docker prepends iirc)
19:12:14 <clarkb> oh latest patchset removed the test
19:12:19 <corvus> gotcha... maybe we could reduce the scope of the iptables test then, so we're still at least checking that, say, snmp is open
19:12:32 <clarkb> ianw: I had updated it so that the existing test would work with docker I thought. But doing somethign like ^ is probably good enough too
19:12:56 <fungi> granted if the block rule came first and we weren't checking at least some ordering, that could be problematic
19:13:09 <clarkb> I don't think docker adds any block rules
19:13:30 <clarkb> in any case we should be able to test the iptables + docker and iptables + !docker case
19:13:32 <corvus> fungi: yeah, we could assert that snmp is open and our block rule is after that
19:14:00 <clarkb> Are there other config mgmt update changes or items we'd like to talk about before moving on?
19:14:20 <ianw> ok, make review comments and i'll look at it :)
19:14:29 <corvus> clarkb: yep, and i think testing is worthwhile to prevent a regression; seems easy to do here
19:15:14 <corvus> ianw: done :)
19:15:42 <clarkb> #topic Storyboard
19:16:05 <fungi> we yakked about some storyboard stuff at the forum last week
19:16:12 <clarkb> There was a good storyboard migration blockers session in Berlin. Seemed like it mostly boiled down to improving performance and error handling as well as some specific needs by various projects
19:16:25 <clarkb> the other item that came up was easy mode file a bug tooling
19:16:45 <clarkb> as is the current story creation process is quite involved and makes more sense on the backend for a developer planning work
19:16:50 <fungi> #link https://etherpad.openstack.org/p/BER-Contrib-Portal-Feedback
19:17:43 <clarkb> I don't think there were any real surprises though which is good
19:17:44 <corvus> why is that called 'contrib portal'?
19:18:10 <fungi> i think it must have gotten repurposed from a contributor portal session
19:18:17 <fungi> i wondered that myself
19:18:46 <corvus> ok.  it's not relevant, that's the main thing to know i guess.  :)
19:18:55 <corvus> (i thought i mispercieved what was happening)
19:19:07 <fungi> there's also a BER-Contrib-Portal pad which i think was for that
19:19:38 <fungi> guessing it was a cut-n-paste error and we were all too jetlagged on tuesday to notice
19:20:01 <clarkb> fungi: diablo_rojo_phon anything else worth calling out from that session?
19:20:18 <clarkb> Looks like we had pretty decent note taking :)
19:20:19 <fungi> not that i'm aware of
19:20:42 <clarkb> Moving on then
19:20:49 <clarkb> #topic General Topics
19:21:34 <clarkb> I'll start with a quick summit recap. One of the sessions was on dealing with timezones and language barriers and a suggestion that came out of that was leaning on meeting agendas more explicitly and only having meetings if we have an agenda
19:21:49 <diablo_rojo_phon> I definitely messed up the Etherpads and rather than lose the edit history just kept the confusing name.
19:22:00 <diablo_rojo_phon> Just got into SeaTac :)
19:22:03 <clarkb> the goal here is to allow people in less convenient timezones for the meeting to sleep/have dinner/etc if the meeting doesn't need to happen or doesn't have an agenda that affects them
19:22:11 <fungi> thanks for the deets diablo_rojo_phon!
19:22:19 <clarkb> Also gives people a chance to prepare their thoughts on topics
19:22:39 <anteaya> clarkb: was the timezones and language barriers aimed at infra? or was it openstack wide?
19:22:43 <clarkb> I've proposed to the infra mailing list that we require agenda updates 24 hours before the start of the infra meeting and if there is no update at that time then cancel the meeting
19:22:45 <clarkb> anteaya: was openstack wide
19:23:09 <clarkb> do we think this proposal is worthwhile? so far the reception on the mailing list has been positive
19:23:27 <frickler> I think it's a good proposal
19:23:31 <AJaeger> +1
19:23:38 <anteaya> I'd like to add that we have at least one meeting every four weeks
19:23:44 <clarkb> I'm happy to do my best to send notes to the mailing list signalling if (or not) we will have a meeting around 1900UTC monday
19:23:51 <gary_perkins> I'm in the UK, so +1
19:23:59 <anteaya> so if we have an agenda lull we at least have a cadence for some sort of meeting
19:24:22 <clarkb> anteaya: can probably do that informally with a catch up agenda item if we otherwise wouldn't meet
19:24:26 <fungi> i'm just not a fan of holding meetings we don't need, so also in favor ;)
19:24:39 <clarkb> anteaya: I expect we'll end up having meetings closer to every other week
19:24:54 <corvus> clarkb: ++ plan; ++ anteaya's idea
19:24:58 <clarkb> we can also tweak it as we go
19:25:09 <anteaya> sure, I'd just like something in the proposal, so that in years to come it is written down
19:25:16 <anteaya> and thanks
19:25:29 <corvus> i think in practice, it'll be at least every other week.  we often have something to talk about here :)
19:25:46 <fungi> usually someone at least adds something to the agenda, yes
19:26:01 <fungi> i wouldn't be surprised if we still hold it most weeks
19:26:02 <anteaya> oh I agree, I don't think we will only meet once per month
19:26:08 <anteaya> I just like a backstop
19:26:11 <clarkb> ok how about this for an ammended proposal " THe infra team requires an updated meeting agenda 24 hours prior to the meeting time otherwise the meeting will be cancelled. If we don't have a meeting for 4 weeks we should have a catch up agenda item added by the chair"
19:26:13 <fungi> but we'll need to be more diligent about adding things to the agenda sooner
19:26:35 <anteaya> clarkb: +1
19:26:52 <fungi> that wfm. also not necessary to spell out in the agenda but if you have an agenda item to add in the final 24 hours it probably goes on the backlog for the next week's agenda
19:26:56 <anteaya> yes, this is where I miss pleia2
19:27:04 <anteaya> she was very attentive to meeting things
19:27:22 <fungi> i miss pleia2 for many, many reasons. she's still lurking in here though!
19:27:36 <clarkb> We can start with that process next week then. I'll do my best monday morning to check in on agenda and send email as necessary from the beach house
19:27:38 <ianw> i'm not about this cancelling but I'll respond on list, seems a better place to discuss it
19:27:40 <anteaya> yes, not the only thing I miss her for
19:27:51 * corvus waves at pleia2
19:27:54 <clarkb> ianw: ya I'll follow up there to
19:28:24 <clarkb> the other summit session that impacts us is the opendev feedback session.
19:28:27 <corvus> ianw: was that last missing a word?
19:28:33 <anteaya> clarkb: beach house has a lovely ring to it
19:28:38 <clarkb> If I had to sum that one up the tl;dr would be communicate communicate communicate
19:28:59 <fungi> good thing that session was also a means of communicating!
19:29:09 <clarkb> There didn't seem to be any major sky is falling concerns but there was the ask that we keep bodies like the TC informed of planning. I think we've managed to communicate well so far, we just have to keep at it
19:29:51 <anteaya> I will say again I hope the team will continue to be available in the openstack-infra channel
19:30:12 <fungi> i don't think we intend to cease having an irc channel
19:30:15 <clarkb> anteaya: yup I don't expect we'll move soon and if/when we do move we can set things up with freenode to redirect people to the correct location
19:30:42 <anteaya> I hope it has infra in the name of the channel if the openstack is dropped
19:30:45 <anteaya> and thank you
19:31:14 <clarkb> our attentiveness and willingness to help was called out. People don't want that to go away and I don't expect it to
19:31:32 <clarkb> the TC is particularly interested in any plans for reorganizing git repos in gerrit
19:31:42 <anteaya> when you say called out do you mean some find it lacking currently?
19:31:54 <clarkb> anteaya: no they found it helpful today and don't want that to change
19:32:10 <anteaya> ah, maintainance of status quo
19:32:14 <clarkb> (I don't think we intend for that to change)
19:32:24 * fungi doesn't intend to try to change it
19:32:25 <anteaya> yeah, I didn't think so
19:32:39 <anteaya> called out has many meanings for me, I guess
19:32:43 <fungi> except finding ways to get even better at it, if possible
19:32:56 <anteaya> I'm not a fan of cloning
19:33:31 <clarkb> I'm trying to think if there were any other sessions that have a direct impact on the infra team
19:34:11 <clarkb> any other summit recap related items others want to bring up? or questions from those that weren't able to attend?
19:34:24 <fungi> the summit feedback session sort of impacts us, insofar as one of our infra-root sysadmins should see if we can figure out how to roll back an edit to the pad for it
19:34:35 <anteaya> I hope you got to a beirgarten?
19:34:56 <fungi> someone autotranslated the first half the pad into japanese after the session ended
19:34:58 <clarkb> fungi: if the pad isn't broken corrupt I think we can rell back to an older version with the api
19:35:08 <fungi> yep, i was just digging up the api docs
19:35:15 <clarkb> *roll
19:35:48 <clarkb> ssbarnea|bkp2: corvus: re gertty is that still something we need/want to discuss in this meeting? or did that get sorted out?
19:36:07 <ssbarnea|bkp2> sorted.
19:36:26 <clarkb> alright the next agenda item is having an infra topic for the new -discuss list
19:36:37 <clarkb> #link https://etherpad.openstack.org/p/common-openstack-ml-topics openstack-discuss topics listing
19:36:41 <clarkb> corvus: ^ I think you added this
19:37:05 <corvus> yep.  i don't have a position to advocate, mostly wanted to see if folks thought we should
19:37:16 <corvus> or how we should use the new list to communicate to folks about infra changes
19:37:18 <anteaya> I think we should
19:37:23 <fungi> seems like a fine addition. [infra] works for that case
19:37:33 <frickler> isn't "infra" a team name already like "qa"?
19:37:41 <clarkb> yes, I think it will be helpful since the alternative is likely to be filtered into the bitbucket
19:37:54 <clarkb> and then people will miss our updates
19:38:06 <corvus> [infra] for "hey should we tweak a thing?" and [infra][all] for "gerrit is being upgraded" ?
19:38:09 <fungi> frickler: future status as an openstack team is probably up in the air
19:38:09 <clarkb> frickler: it is technically "infrastructure" I think
19:38:18 <clarkb> corvus: ++
19:38:27 <fungi> corvus: i like
19:38:40 <corvus> of course "all" is now more than just "all developers" it's "all people interested in openstack"
19:38:49 <fungi> adding [all] in the case of announcements basically
19:39:19 <corvus> hrm.  maybe we should refrain from [all] except in the most highly impactful cases, and use [infra][dev] for "gerrit is being upgraded"?
19:39:26 <fungi> there are plenty of people in the community who don't engage in software development but probably still care about bits of the community infrastructure
19:39:54 <anteaya> I'm a fan of using [all] judiciously
19:40:01 <corvus> fungi: indeed; i certainly think it's widely applicable, but i don't have a read on how much.
19:40:10 <fungi> not sure any of us do yet
19:40:20 <fungi> we're all still figuring this out
19:40:22 <clarkb> corvus: I still feel like gerrit upgrades are an [all]? simply due to the large impact it is
19:40:25 <clarkb> *impact it has
19:40:31 <clarkb> and maybe we scale that back if we get asked to?
19:40:58 <fungi> openstack-discuss [all] is still not on the same level as posting to openstack-announce at least
19:40:59 <anteaya> I'd go the other way
19:41:18 <corvus> i can buy the argument for [all]
19:41:24 <anteaya> use [infra][dev] to start and go to all if the feedback requests
19:42:00 <clarkb> as fungi points out we are all learning and probably best thing is to try something and change if that doesn't work
19:42:13 <anteaya> and put gerrti upgrade in the subject line
19:42:21 <corvus> we can also add [all] to, say, an initial announcement of a major change (such as a gerrit upgrade) and drop down to [dev] for followup scheduling messages, etc
19:42:22 <frickler> +1 to anteaya, folks will just unsubscribe instead of complaining about too many mails
19:42:31 <clarkb> corvus: that might be a good compromise
19:42:41 <fungi> yeah, all fine ideas in my opinion, we just need to be flexible (especially here at the beginning) until we work out a viable pattern
19:42:52 <anteaya> yup
19:42:59 <anteaya> just suggestions
19:43:28 <corvus> we can ask for feedback after our first message:  "did you enjoy receiving this message?"  "do you wish you had received this message?"  ;)
19:43:33 <fungi> i don't really have enough of a feel yet for which will work better, so use [infra] and apply your best judgement in adding other tags?
19:43:39 <anteaya> ha ha ha
19:43:41 <clarkb> fungi: ++
19:44:19 <corvus> sounds good.  i'm glad we thought a bit about it.
19:44:25 <clarkb> we've also only got about ~16 minutes left and a couple important topics left so I think we should move on. I think we are all aware of how careful we should be using [all] and we can learn to see what works as we go
19:44:41 <clarkb> Next on the list is an opendev update
19:44:51 <clarkb> I believe we are waiting on dns registrar to be happy with dnssec?
19:45:16 <corvus> yeah, jamesmcarthur has been very helpful in making requests to the registrar
19:45:57 <corvus> apparently they are confused about our request to add dnskey records.  we're trying to convince them that the data we sent are correct, or at least find out more info about why they think it isn't.
19:46:31 <fungi> and i have to assume we haven't heard any more because he hasn't heard any more yet
19:46:35 <clarkb> can (should) we set up the domain without dnssec to start?
19:46:41 <corvus> the glue records are said to be in place now, however, i think making requests to them in this manner is a bit like talking to a jinn -- i don't think we actually asked them to change the ns records to *use* the glue records.
19:46:54 <corvus> so i reckon we need to explicitly ask for that too :)
19:47:16 <corvus> clarkb: technically, yes, but i have to say i'm really not interested in it.
19:47:39 <corvus> i *really* want sshkey records
19:47:55 * fungi too
19:48:06 <fungi> i use them with my domains, very handy
19:48:18 <clarkb> in that case I guess we wait for registrar to say more words about the problem?
19:48:39 <clarkb> fungi and I may be able to followup with them and jimmy if necesary too (I think email is their primary form of communication)
19:49:44 <clarkb> I'll check with jimmy if he thinks we can do anything else to help as well
19:49:49 <clarkb> he has some experience with this group
19:49:53 <fungi> very confused e-mail, if my previous experiences with them are representative
19:50:04 <corvus> it'd be one thing if requests like this took a day or two, but the fact that it's taking *weeks* is, honestly, pretty unworkable.
19:50:20 <jamesmcarthur> clarkb: I'm waiting to hear back from them after rebutting their last assertion that the DNSSEC key is wrong.
19:50:26 <jamesmcarthur> And corvus: I agree 100%.
19:50:27 <clarkb> jamesmcarthur: roger
19:50:31 <fungi> thanks jamesmcarthur!
19:50:36 <jamesmcarthur> It's top of our list to move off to a different registrar.
19:50:43 <corvus> jamesmcarthur: yeah, i think we're on the same page on that :)
19:50:45 <jamesmcarthur> Just one of those "we need some time" things
19:50:59 <jamesmcarthur> These guys are absolutely useless.
19:51:08 <clarkb> once dns is in place I expect we can start pushing snowballs downhill quickly
19:51:23 <jamesmcarthur> lsell: and I have already been communicating about a plan to move to another registrar
19:51:24 <clarkb> Anything else before that snowball is rolling? I'd like to get to ianw's topic before we run out of time
19:51:46 <fungi> we have a few snowballs already perched atop the avalanche
19:52:14 <clarkb> Alright ianw would like to talk about cleaning up our pypi afs volume
19:52:42 <clarkb> for those that maybe don't know we serve pypi via caching proxy now beacuse pypi itself has become quite large and is growing more quickly every day
19:52:54 <clarkb> this has been mostly working
19:53:19 <ianw> yes, our afs volume is basically full, the f29 updates are pushing it.  removing the old pypi volume obviously frees up most of it
19:53:20 <clarkb> and I'm not sure we'd be able to realistically go back to mirroring even if we wanted to. But if we did creating a new voluming and building up without packages that are huge and we don't want is possible
19:53:41 <clarkb> I'm in favor of removing the pypi volume given that doing ^ is feasible if we need to go back to the old setup
19:53:44 <fungi> also switching to a new enough version of bandersnatch to successfully clean up that mirror needs python from ubuntu bionic or later
19:54:05 <ianw> yeah, the idea we could re-use the data is probably fantasy
19:54:33 <ianw> what i would like to happen is people +2 on the removal of the docs etc with
19:54:36 <ianw> #link https://review.openstack.org/#/c/618326/
19:54:37 <fungi> we certainly may want to switch back to bandersnatch at some point, but by the time we can do that we're better off just starting a fresh mirror from scratch anyway
19:54:50 <ianw> and i'll take that as an indication to clean up
19:55:20 <ianw> the f29 mirror is pushing it (but i'll ping in #infra about removing f27, which will relieve some of the pressure)
19:55:32 <clarkb> done
19:56:18 <ianw> that's all on that
19:56:25 <AJaeger> thanks, ianw
19:56:33 <corvus> a note on the commit message -- we can work around the 2tb limit by sharding.  but i think the other issues are compelling and we should stick with proxy.
19:57:08 <clarkb> corvus: ya the fact that the growth is only getting worse as time goes on makes it a treadmill that will continue to be hard to be on
19:57:13 <fungi> yeah, we're not using it anyway, so the complexity of sharding that isn't worth the effort to maintain
19:57:19 <ianw> corvus: true, but we'd have to sort of invent the sharding, an effort i'm not sure any of us want to go to
19:58:13 <clarkb> We have just a couple minutes left
19:58:16 <clarkb> #topic Open Discussion
19:58:22 <clarkb> Any last minute items before our hour is up?
19:58:23 <corvus> it's not difficult; just do a volume for each directory under packages/
19:58:57 <corvus> (so ~256 volumes for the 00 -- ff directories) plus some more
19:59:13 <corvus> i mostly mention this as a general pattern it's useful to keep in mind
19:59:22 <corvus> afs can handle hundreds of thousands of volumes easily
19:59:40 <anteaya> thanks for the meeting clarkb
19:59:42 <ssbarnea|bkp2> considering the size  (and number of changes) of pypi i suspect we are better off with a proxy than a mirror, but that's only my "gut feeling".
19:59:43 <corvus> (this is the way home directories are set up at the big universities)
19:59:47 <clarkb> the biggest issue is the pypi package size cost is concentrated in a few packages so we'd still need to ignore them in bandersnatch
19:59:53 <anteaya> happy too much food and shopping those who do
20:00:03 <corvus> clarkb: yes, and keep adding packages to the list as we see them
20:00:19 <clarkb> ssbarnea|bkp2: yup mine too. Basically current system works and is low maintenance
20:00:30 <clarkb> I think if we decide to go back to afs for pypi we'd be rebuiilding things anyway
20:00:37 <clarkb> so its fine to move fowrard on ianw's plan for now
20:00:40 <clarkb> and with that we are at time
20:00:42 <clarkb> #endmeeting