19:01:23 <clarkb> #startmeeting infra
19:01:24 <openstack> Meeting started Tue Apr  3 19:01:23 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:27 <openstack> The meeting name has been set to 'infra'
19:01:52 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:15 <clarkb> #topic Announcements
19:02:31 <dmsimard> \o
19:02:33 <clarkb> Zuul v3 released last week after much hard work by many in this group and others
19:02:53 <clarkb> so raise a glass of your favorite beverage if you haven't already done so (or do it again!)
19:02:53 <corvus> thanks everyone!
19:02:58 <fungi> [and there was much rejoicing]
19:03:06 <anteaya> congratulations
19:03:57 <corvus> i will be happy to raise a drink again if it be the wish of this group
19:03:58 <clarkb> The OpenStack Summit and the colocated OpenDev (with focus on CI/CD) events are coming up in about 6 weeks or so. Apparently laws/regulations that affect airbnb just chagned in vancouver, BC so double check if you planned to stay with them
19:04:33 <clarkb> (I don't actually know how big an impact that will have but it was mentioned by someone that we should warn conference attendees to double check)
19:05:24 <clarkb> anything else we want to announce before digging into the agenda proper?
19:05:26 <anteaya> corvus, yes, do
19:06:24 <fungi> it's been suggested that as opendev is a more grassroots sort of event, encouraging people to pass the word along to their respective professional circles of non-summitgoers about it would be good
19:07:12 <clarkb> right, we hope to see you and your friends there, I expect it will be a good time
19:07:25 <clarkb> #topic Actions from last meeting
19:07:26 <corvus> lots of collaborative space/time scheduled
19:07:37 <clarkb> oh should I undo to keep talking opendev?
19:07:55 <corvus> nah
19:08:04 <fungi> let's keep this bus in drive
19:08:04 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-03-27-19.03.txt
19:08:26 <clarkb> I wasn't able to follow lsat week's meeting due to needing to have lunch on tight conf schedule but I don't see any actions after a quick grep
19:08:42 <clarkb> #topic Specs Approval
19:09:00 <clarkb> There are a couple updates to existing specs that I'd like to get in soon.
19:09:01 <anteaya> I agreed to post a patch, which I did: https://review.openstack.org/#/c/557979
19:09:16 <clarkb> and anteaya ^ has patch for survey spec
19:09:26 * clarkb linsk them all via meetbot
19:09:35 <anteaya> thanks
19:09:41 <clarkb> #link https://review.openstack.org/#/c/557772/ mark zuul v3 spec as done
19:09:51 <fungi> oh, yeah i should have mentioned to you that i put the "Amend top-level project hosting spec" change up for council vote
19:10:03 <clarkb> #link https://review.openstack.org/#/c/555104/ Amend minor detail on git repo hosting for top level project hosting
19:10:16 <clarkb> #link https://review.openstack.org/#/c/557979 implementation of survey spec
19:10:22 <fungi> (by being a poor stand-in meeting chair i help ensure i won't have to do it often!)
19:10:56 <clarkb> fungi: heh its fine. I was already planning on moving ahead with that. Do we feel like the zuul v3 spec being marked done needs to go through formal council vote? if so are we good with giving that until thursday afternoon pacific time?
19:11:24 <clarkb> I'll likely go ahead and get the minor top level project hosting amendment in today since it was put up for council vote last week
19:11:40 <fungi> i'm ambivalent on whether shuffling entries around in the index file needs formal voting
19:11:55 <anteaya> I have no opinion
19:11:59 <clarkb> I'm hopign that the presence of the 3.0.0 tag on zuul makes that one non controverial
19:12:09 <fungi> yup
19:12:11 <clarkb> corvus: ^ do you think it needs any more review than it has received?
19:12:14 <fungi> it has my vote, at any rate
19:12:22 <corvus> nah
19:13:02 <clarkb> alright then I'll work toget those approved after the meeting. And anyone interested in survey tooling should review that third change from anteaya
19:13:06 <fungi> once it merges, we're down to one long-standing priority spec
19:13:22 <clarkb> which takes us to the next agenda item
19:13:30 <clarkb> #topic Priority Efforts
19:13:56 <clarkb> I've put on the agenda that now might be a good time to reevaluate current efforts and decide if any of them should be bumped up to priority effort status.
19:14:03 <fungi> for our one remaining priority effort, i finally got the 4-byte utf-8 support into production over the weekend and successfully imported stories for tripleo-ui
19:14:10 <clarkb> fungi: nice
19:14:24 <anteaya> fungi, congratulations
19:14:35 <fungi> my next big task there is figuring out why gerrit commenting on stories broke around the time we upgraded to gerrit 2.13
19:14:59 <anteaya> gerrit paid attention to my new patch
19:15:03 <clarkb> fungi: as the person that helped drive the 2.13 upgrade my failure to catch that means I'm happy to help debug it now. Let me know if I can be of assistance
19:15:13 <anteaya> it changed the status when I uploaded
19:15:26 <anteaya> sorry storyboard paid attention*
19:15:28 <corvus> yeah, i've noticed status tracking working
19:15:31 <fungi> it probably involves me making sure pabelanger is done testing things with review-dev01 so i can try some hunches
19:15:42 <corvus> what specifically is broken?
19:16:11 <pabelanger> fungi: I have not yet, just gerrit testing
19:16:11 <fungi> when you add a story footer in your commit message, the its-storyboard plugin for gerrit is supposed to leave a comment on the story with a url to the change
19:16:29 <corvus> oh, so it's changing task status but not leaving words
19:16:34 <fungi> yep
19:16:34 <anteaya> ah
19:16:41 <anteaya> yes, there were no words
19:16:47 <corvus> fascinating failure mode
19:17:00 <fungi> indeed. and i'm hoping it won't involve a foray into java
19:17:07 <corvus> i'm accustomed to things breaking much more completely than that :)
19:17:33 <fungi> there's a possibility it's just a side effect of changes in how we're doing commentlinks in the gerrit configuration since its plugins seem to rely on that to infer some configured behaviors
19:17:46 <clarkb> and those did change with 2.13
19:18:02 <fungi> zaro__ gave me a few suggestions to dig into there
19:18:26 <fungi> anyway, we've had some projects say they want to hold off migrating to sb until that's fixed
19:19:14 <fungi> and i really didn't want to monopolize the priority efforts discussion, so let's get on with discussing adding some new ones ;)
19:19:55 <clarkb> since we are about to remove one of the long standing efforts that has taken a bunch of our time I thought it may be worthwhile to decide if any other in progress efforts (or possibly new ones) should be bumped up to priority status
19:20:26 <clarkb> on the wiki agenda I threw out some ideas. I don't expect we'll decide this today, but wanted people to start thinking about what is going to be important to the infra team for the next few months or so we can take it from there
19:20:49 <fungi> not to keep bringing up storyboard, but if people do have spare cycles and want to help hack on it, they're very welcome to do so ;)
19:21:04 <anteaya> I don't think the survey spec is a priority or will take that much effort, I care about but I don't think it should be elevated
19:21:10 <fungi> presumably it's staying on the priority list at least
19:21:14 <fungi> (sb i mean)
19:21:17 <clarkb> fungi: ya I don't expect we'll remove it
19:21:31 <pabelanger> agree, we still have some trusty systems we need to move to xenial that we didn't get to last virtual sprint
19:21:42 <pabelanger> would be open to having another sprint to finish that off
19:21:47 <clarkb> I just wanted to spark the conversation and idea here but will probably move discussion to the infra ml so that we can get broader reach and input and decide together there
19:21:51 <fungi> i do agree with the things on the suggested list
19:22:01 <pabelanger> +1
19:22:13 <fungi> i'd _love_ for someone who isn't me to be interested in picking up the wiki work i dropped on the floor
19:22:27 <fungi> because clearly i'm not doing well at getting around to it
19:22:46 <clarkb> so start thinking about it and I'll get a thread started shortly where we can hear all the great ideas
19:23:00 <fungi> but i'm very happy to provide a high-bandwidth sync up of where it's gotten to for anyone who's interested
19:23:22 <clarkb> fungi: thanks!
19:24:10 <clarkb> #topic General Topics
19:24:29 <clarkb> I'm going to flip the order on these two because one is time sensitive
19:24:50 <clarkb> pabelanger has spun up a new review01.openstack.org running xenial which means we now have IP addresses we can warn people about changing to in the future
19:25:00 <clarkb> I think the last remaining question is the scheduling of that server migration
19:25:10 <pabelanger> indeed
19:25:13 <clarkb> pabelanger: have you asked the release team if R-17 works for them?
19:25:38 <pabelanger> clarkb: yah, looks like kubecon EU is that week and all of the release team is going to it
19:25:47 <clarkb> pabelanger: cool I guess that makes it a perfect time?
19:25:58 <pabelanger> yah, I think so
19:26:06 <clarkb> infra-root does May 02, 2018 work for migrating the gerrit server?
19:26:11 <clarkb> I can eb around for that myself
19:26:19 <fungi> sounds fine. i have high hopes it'll be a brief outage. mostly just mechanical stuff like moving the cinder volume, changing dns and stopping/starting services
19:26:26 <clarkb> fungi: yup
19:26:34 <pabelanger> yah, https://etherpad.openstack.org/p/review01-xenial-upgrade is the steps I am expecting
19:26:36 <dmsimard> It's sufficiently early that I can suppose I'll be around
19:26:43 <pabelanger> which was done on review01-dev
19:26:47 <clarkb> pabelanger: assuming it works for you and you've got fungi and I on board and now dmsimard I think we can commit to that date
19:26:58 <fungi> yeah, i'm happy to help with it
19:27:08 <pabelanger> clarkb: ++
19:27:24 <pabelanger> I'll start spamming the ML then with info
19:27:44 <clarkb> 2000UTC is the current proposed time on the therpad for the ml email
19:27:50 <clarkb> which wfm
19:27:53 <fungi> #link https://etherpad.openstack.org/p/review01-xenial-upgrade Ubuntu Xenial move for production Gerrit
19:28:10 <ianw> wfm as someone who can jump in later if required
19:28:19 <clarkb> pabelanger: sounds good, thanks for putting this together. One fewer gerrit todo on the list marked off
19:28:46 <pabelanger> np!
19:28:46 <fungi> just to confirm, may 2 is a wednesday
19:28:54 <pabelanger> yes
19:29:09 <mordred> clarkb: may 2 wfm
19:29:31 <fungi> 2000z is right after the end of the storyboard meeting, so looks fine as far as my availability is concerned
19:29:38 <clarkb> #agreed Upgrade review.o.o's control plane server to Xenial at 2000UTC May 2, 2018
19:30:07 <ianw> p.s. it might be worth looking at https://review.openstack.org/#/c/555633 before this, so that the gerrit jobs start working again, in case a new .war or something is required
19:30:16 <clarkb> ianw: ++ thanks
19:30:33 <pabelanger> ack
19:30:35 <clarkb> #link https://review.openstack.org/#/c/555633 Get the gerrit jobs working again
19:31:12 <clarkb> The next topic is dmsimard's. Our meetbot can be abused during times of internet trolling
19:31:23 <dmsimard> yeah
19:31:44 <clarkb> in particular On April 1st which happened to also be Easter there was at least one webclient nick starting meetings and calling them maintenance then telling people in channel they all had to leave
19:31:44 <dmsimard> It's probably the same person that has done the few occurrences I've seen
19:31:59 <fungi> to be fair we've known about this for years, and yesterday was the first time i've ever seen it happen in the wild (and consider the date)
19:32:23 <clarkb> fungi: I think it has happened at least once before with this startmeeting and tell people to leave story
19:32:25 <dmsimard> I didn't realize the date -- I was mostly concerned about someone potentially putting up offensive topics and stuff like that.
19:32:27 <corvus> we could revert https://review.openstack.org/505669 and ask people to opt-in allowing meetbot to set topics
19:32:39 <fungi> they hit a couple dozen of our channels, and i managed to get the vast majority of those cleaned up a little over an hour later when the meetings expired for them
19:33:07 <jlvillal> Can meetbot ensure they are logged in?
19:33:26 <fungi> er, i guess not yesterday it was sunday. anyway, april 1
19:33:36 * fungi has lost all sense of time any more
19:34:04 <clarkb> corvus: I think one of the problems with that is meetbot is also our channel logger
19:34:14 <clarkb> corvus: so even if people don't want meetings it is desireable to have logs?
19:34:15 <corvus> jlvillal: i'm not sure if it supports that, but could probably be added easily
19:34:21 <corvus> clarkb: that's not what that change does
19:34:28 <corvus> clarkb: that only allows meetbot to set /topic
19:34:32 <clarkb> ah
19:34:33 <jlvillal> And also if the person who starts the meeting leaves then stop meeting.
19:34:46 <fungi> jlvillal: yeah i don't know if anyone's looked into checking whether their nick has identified with nickserv. we've had suggestions in the past of doing that, or relying on +v in channels, or even whitelisting specific chair nicks in advance for specific meetings. all of these come with certain drawbacks
19:35:13 <clarkb> my personal feeling on it is regardless of the bots in use people can troll public channels and generally be annoying. Addressing this in the bot itself won't change that
19:35:18 <anteaya> jlvillal, I've seen many times where people who chair lose connection
19:35:28 <anteaya> but they want the meeting to continue
19:35:42 <fungi> revoking +t for the meetbot except in channels where people want to hold meetings (and in fact they can still do that even without having it change the channel topic) seems like a low-cost workaround
19:35:52 <anteaya> the project working group is one group I have seen that happen with
19:36:05 <clarkb> which means maybe educating users and having channel local ops may be a good approach
19:36:14 <clarkb> (then ops can kick the offender and they can stop the meeting)
19:36:50 <fungi> if meetbot supports allowing anyone who is a chanop to #endmeeting at any time, that also might be a nice alternative
19:36:51 <clarkb> I do wonder how much this behavior is tied to being able to set the topic
19:37:01 <clarkb> fungi: I'm not sure if does but it would make sense as a feature
19:37:07 <Regget> And you said this meetbot abuse has happened before?
19:37:23 <Regget> when?
19:37:27 <clarkb> at least once, I think it coincided with another holiday like US thanksgiving or something
19:37:32 <fungi> sounds like dmsimard and clarkb have seen prior occurrences. those didn't come to my attention
19:37:41 <Regget> ok
19:37:43 <dmsimard> I put up links in the agenda
19:37:56 <dmsimard> #link http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-04-01.log.html#t2018-04-01T16:39:49
19:38:02 <dmsimard> #link http://eavesdrop.openstack.org/irclogs/%23openstack-operators/%23openstack-operators.2018-04-01.log.html#t2018-04-01T16:59:48
19:38:15 <dmsimard> Those are the two I've seen, there may be others. Just happened to walk on those.
19:38:19 <fungi> yeah, irc vandals seem to like to pop up on weekends and popular holidays when they think channel operators (and probably server operators for the network) will be otherwise occupied
19:38:28 <Regget> Yeah I saw those logs from Sunday
19:38:32 <pabelanger> I've only recently started seeing it more, it is a side effect of 505669 corvus linked above?
19:38:39 <dmsimard> Need to step away to pick up kids, be back in a bit
19:38:52 <corvus> pabelanger: on what other dates have you seen it?
19:38:58 <fungi> dmsimard: oh, those were both part of the same incident. they hit somewhere between 20-30 of our channels (i stopped counting and just cleaned them up)
19:39:48 <pabelanger> corvus: I'd have to go back into logs and see, but thought I remember doing endmeeting in some channels to clear the topic.
19:39:58 <pabelanger> may have just been openstakc-meeting too
19:40:06 <corvus> pabelanger: because someone vandalized it?
19:40:21 <fungi> anyway, if it's an infrequent nuisance, i'm inclined to prefer keeping our tools flexible and just dealing with the result (i mean, we made it so anyone can #endmeeting after an hour anyway)
19:40:26 <pabelanger> corvus: yah, some random user did startmeeting test IIRC
19:40:30 <pabelanger> then left
19:40:39 <corvus> pabelanger: i'd hardly consider that vandalism :)
19:40:47 <fungi> that sounds less malicious and more like someone didn't know what they were doing
19:40:51 <corvus> i mean, *i've* done "#startmeeting test" :)
19:40:58 <ianw> is it possible it's some sort of markov chain thing that has happened across something that works?  it's ... odd
19:41:12 <Regget> and what do you think the topic of maintenance has to do with it
19:41:21 <Regget> since that is the name of those meetings
19:41:24 <fungi> there was a similar incident last where where someone did "#startmeeting kolla" in #openstack-dev (i think) because they thought that was how they were supposed to summon people for discussion
19:41:31 <clarkb> Regget: I think that was the pretense used to say people should leave teh channel bceause it is under maintenance now
19:41:32 <fungi> er, similar incident last year
19:41:42 <Regget> oh
19:41:57 <corvus> fungi: i agree
19:42:00 <clarkb> I agree with fungi though. It is currently infrequent and relatively easy to address when it happens
19:42:17 <Regget> So how do we know it was not a maintenance person doing his work on weekends so he doesn’t interfere with meetings
19:42:23 <clarkb> if we need to educate our users a bit more on why/how it happens and what they can do about it (local hcannel ops etc) then we can do that
19:42:40 <anteaya> Regget, because the maintenance people are in this meeting
19:42:42 <clarkb> Regget: because it was an unauth'd nick from a webclient
19:42:47 <Regget> ok
19:42:50 <clarkb> Regget: and not a chanop
19:42:54 <anteaya> Regget, and that is not how we do mantainance
19:42:59 <Regget> ok
19:43:18 <fungi> Regget: i'm probably overestimating what constitutes common sense with respect to irc networks, but the only people doing maintenance on the irc servers should be freenode staff and they wouldn't use people's random bots to change channel topics to signal that
19:43:28 <Regget> ok
19:43:32 <Regget> I see now
19:43:46 <clarkb> fungi: thee is a separte question of maintenance on the bot itself but no one should have to leave the channel for that
19:43:56 <fungi> indeed
19:45:24 <clarkb> In general we can educate people that yes public communications channels do get trolled occasionally. If you have questions or concerns you can ask an chanop of the channel(s) you lurk in
19:45:29 <fungi> we _do_ use bots to change channel topics so as to let people know about maintenance work we're doing on _other_ systems, but those maintenance activities wouldn't require people to leave any irc channels (to the contrary, it's where we're communicating about them)
19:46:49 <anteaya> hope he comes back, he was asking good questions
19:46:50 <corvus> if anyone is interested in improving our ability to respond to trolls in general, this spec needs volunteers: http://specs.openstack.org/openstack-infra/infra-specs/specs/irc.html
19:46:58 <anteaya> he/she
19:46:59 <mordred> fungi: perhaps we should communicate that if we use a bot to ask people to take an action, that action will be documented somewhere on an openstack.org url so someone can verify whether or not a bot message with an action is actually from us?
19:47:31 <clarkb> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/irc.html This spec needs volunteers which will help us be in a better position to update our IRC bots to address these issues when they happen
19:48:06 <clarkb> mordred: using the status page on the wiki I guess?
19:48:36 <fungi> mordred: yeah, the first place which springs to mind is the infrastructure status wiki, but referring people to a wiki any random person on the 'net can edit may not be any better
19:48:59 <clarkb> fungi: ya that was going to be my next concern and not wanting to suggest twitter as an official of record location either
19:49:07 <mordred> yah. I was more thinking some non-wiki location - but I haven't thought through the whole thing deeply yet
19:49:09 <anteaya> perhaps a link to the status page should be included in maintenance annoucments
19:49:17 <anteaya> as in verify here: link
19:49:31 <clarkb> anteaya: I think the problem is then a troll could just send a verify here link to anywhere
19:49:39 <clarkb> if its advertised out of band its a bit more controlled
19:50:25 <clarkb> in any case I think the two steps forward here are 1) improving our irc bot implementation as per corvus' spec. Volunteers needed. and 2) Better docuemntation/education for users of IRC to avoid confusion when trolls are crafty enough to maybe look legit
19:50:29 <fungi> for the specific "leave the channel" scenario, the only situations we've had where we wanted people to exit a channel are when it's being renamed/forwarded. and for that we just give them a heads up (with a link to an ml thread about it generally) and then kick the stragglers eventually
19:51:11 <clarkb> I can work on drafting an email to the dev ml with info on what infra and freenode would do as far as maintenance goes and who to contact if you have questions
19:51:26 <anteaya> clarkb, true
19:51:39 <clarkb> does that seem like a reasonable place to start?
19:51:42 <fungi> we also used to try and do that to reclaim channel founder privs, but it's usually close to impossible unless there are just a few people in there so tends to be easier to reach out to freenode staff and get them to override
19:51:45 <clarkb> can point to the irc bot infra spec there too
19:52:00 <fungi> yeah, that sounds like a great idea
19:52:47 <jlvillal> corvus, FYI: I might be interested in assisting with the IRC bot.
19:53:00 <anteaya> clarkb, yes and also perhaps add the text to the IRC services area in the docs
19:53:11 <clarkb> anteaya: ya we can work it into the docs too
19:53:29 <clarkb> ok we've a few minutes left for the what did we miss open discussion topic
19:53:34 <clarkb> #topic Open Discussion
19:53:40 <anteaya> David did a thing
19:53:48 <anteaya> but I think he said he is picking up kids
19:53:56 <jlvillal> gerritbot review request: https://review.openstack.org/#/c/545469/  Some cleanup/refactoring and adding unit tests. Has one +2 :)
19:54:00 <corvus> jlvillal: awesome!  i'm happy to work with you on that and pitch in
19:54:10 <clarkb> #link https://www.youtube.com/watch?v=6gTsL7E7U7Q&t=1697 dmsimard did a talk on how to help infra
19:54:12 <jlvillal> corvus, cool :)
19:54:22 <clarkb> feel free to share the video if you know people who are interested
19:54:34 * dmsimard back from picking up kids
19:54:41 <pabelanger> https://review.openstack.org/#/q/topic:emit-job-header-nodes is a way we could start printing nodepool info in jobs again, if people are interested
19:55:00 <anteaya> dmsimard, how did the talk go?
19:55:11 <fungi> good-sized crowd?
19:55:40 <dmsimard> It went great, less people than expected showed up but since it was recorded it was still worth it. Still got a few good questions throughout the talk.
19:55:55 <pabelanger> https://review.openstack.org/#/q/topic:fedora-27 is also needed to migrate jobs from fedora-26 to fedora-27, would like some help landing that too
19:56:01 <anteaya> sometimes small groups can be more effective
19:56:06 <anteaya> yay for questions
19:56:22 <fungi> and we're getting a bunch of new infra contributors from montreal as a result, right? ;)
19:56:35 <dmsimard> The format of the talk was very informal on purpose and was more or less a ~1hr live demo with recursive and meta references. It was fun.
19:56:35 <anteaya> oh this was in Montreal?
19:56:53 <dmsimard> Yeah, we have regular OpenStack meetups in Montreal.
19:57:08 <anteaya> nice, I think I was at one in the past
19:57:15 <anteaya> when I went to a pycon
19:57:25 <dmsimard> ah yeah there was a pycon about 2-3 years back :)
19:57:29 <anteaya> bit of a drive to go every month
19:57:31 <dmsimard> fungi: we'll see :D
19:57:40 * fungi crosses his fingers
19:58:21 <dmsimard> My hope is that the talk remains relevant and a form of informal documentation since it was recorded
19:59:48 <dmsimard> pabelanger: I'll help with those reviews
20:00:07 <clarkb> alright thanks everyone
20:00:11 <anteaya> thank you
20:00:13 <clarkb> #endmeeting