22:01:54 <corvus> #startmeeting zuul
22:01:56 <openstack> Meeting started Mon Mar 26 22:01:54 2018 UTC and is due to finish in 60 minutes.  The chair is corvus. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:01:57 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:01:59 <openstack> The meeting name has been set to 'zuul'
22:02:01 <corvus> #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:02:11 <corvus> #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2018/zuul.2018-03-19-22.03.html
22:02:21 <corvus> #topic Action items
22:02:25 <corvus> corvus attempt to help with storyboard private story notification fix
22:02:30 <corvus> so i haven't gotten very far with that
22:02:39 <Shrews> we didn't encourage you enough
22:02:48 <corvus> er, i should clarify
22:02:52 <corvus> i tried again really hard
22:02:56 <corvus> and still didn't get anywhere
22:03:32 <jlk> o/
22:03:36 <corvus> the test infrastructure around this in storyboard is not functioning the way i expect, and i'm at a loss for how to fix it
22:03:55 <jlk> (only half paying attention, please ping me if I'm needed)
22:04:05 <corvus> so we need to decide if we're okay living with the fact that we won't get email about critical security bugs
22:04:14 <corvus> or if we want to do something else
22:04:18 <fungi> that's depressing. i'd like it to not be the case, but am dealing with my own "why did tests not catch this?" moment in sb now as well
22:04:44 <corvus> yeah, i'd also love to have the time to work on a systemic fix to the issue i'm running into, but it would be a *large* change
22:05:01 <corvus> i know how to do that, but don't have time.  i do have time to make a small change, but don't know how.
22:06:01 <Shrews> what's our other option? or is there none?
22:06:01 <corvus> we could try to peridocally poll storyboard with a cron.  however, in pure coincidence, the cron i have running for the zuulv3 dashboard just started timing out.  i don't know if the same would be true for a query for private stories.
22:06:07 <corvus> i could try it and see.
22:06:30 <corvus> that's one option.  another might be to create a mailing list for folks to send security bug reports to.  either the entire report, or just a notification after they create a bug.
22:06:51 <corvus> honestly, if we can work around this with a cron job, that's *probably* the slightly safer/friendler path.
22:07:00 <corvus> (for reporters)
22:07:09 <corvus> it's probably a bit more work on our part to make sure the cronjob doesn't bitrot
22:07:22 <corvus> (also, could be a periodic job in zuul, doesn't need to be cron ;)
22:07:54 <fungi> i'm wondering if the recent api slowness is limited to searches by story tag since that's where i've mainly noticed it so far. more of a topic for #storyboard but it may mean that searching on other criteria is not currently slow
22:08:07 <corvus> fungi: good point
22:08:19 <corvus> so maybe i should try writing an automated query for this and report back?
22:08:32 <fungi> however, i also haven't looked to determine whether you can query by private-ness
22:08:52 <corvus> fungi: hopefully 'list all zuul stories' doesn't become prohibitively expensive :)
22:09:16 <pabelanger> crontab seems like a good next step
22:09:27 <corvus> #action corvus write automated query for private zuul stories (to find security issues)
22:09:42 <corvus> the other action item from last week is: corvus create some remote tests for zuul-stream
22:09:54 <corvus> i did that.  i think they merged!
22:10:08 <corvus> #topic  Why not tagging the current master, e.g. release early, release often? (tristanC)
22:10:09 <fungi> fair, if zuul has _that_ many stories before sb performance improves, then maybe we're doing something wrong ;)
22:10:30 <corvus> tristanC is not here
22:10:48 <corvus> but i guess the topic is self-explanatory :)
22:11:47 <clarkb> a beta release wouldn't be the worst thing imo (especially if github thing is fixable at this point)
22:11:51 <corvus> my straightforward answer to that question is that as soon as we are able to release, we will.
22:11:56 <clarkb> but ya I realize ^
22:12:03 <fungi> could do zuul 3.0.0.0rc1 or something i suppose
22:12:07 <corvus> i know of two issues:
22:12:34 <corvus> 1) javascript -- mordred has some changes that he thinks we should merge now because otherwise, migrating folks later will be painful/impossible
22:13:04 <corvus> (as in, we've got at least one more case where we must merge a change then go do manual things to make it work)
22:13:26 <corvus> we don't want to be landing those kinds of changes in the future
22:13:29 * mordred hopes to have that in shape for folks by tomorrow morning
22:13:48 * mordred currently on very bad in0-flight wifi
22:14:15 <corvus> 2) github3.py -- we're still in the position where if we get a working release of this before item #1 clears, great!  otherwise, we need to vendor it before release.
22:14:32 <corvus> either way, i think those both preclude releasing master at this point
22:14:57 <pabelanger> wfm
22:15:11 <clarkb> corvus: it was mentioned by tobiash that the github3.py issue for license api thing is fixed? I guess it just needs a release now?
22:15:16 <corvus> however, once we do release, we need to do one more thing, then we can return to our pattern of releasing frequently.  like, maybe every few weeks.
22:15:25 <corvus> clarkb: oh i missed that!
22:15:29 <corvus> maybe it's time to ping jlk :)
22:15:30 <clarkb> jlk can probably confirm
22:15:44 <jlk> So.
22:15:54 <jlk> I think we fixed at least some of the issue tobiash found, but maybe not all
22:16:05 <corvus> #info release is blocked by some pending javascript patches (mordred to prepare ASAP)
22:16:14 <jlk> and I've been really bad about being able to set up something to validate, so it'd be helpful if we could get a fresh attempt by tobiash
22:16:32 <corvus> #action mordred propose release-blocking javascript patches
22:16:33 <jlk> There hasn't been a release, so he'd have to test from develop branch. If we can confirm, I can do the work to make a release happen
22:16:46 <corvus> #action tobiash test most recent github3.py to see if it's ready to release
22:17:20 <corvus> sweet!  hopefully, with the magic of timezones, he can do that while i'm asleep :)
22:18:14 <jlk> There have been other GitHub Enterprise folks testing and they've found at least one other issue that has an open PR, so I'm not sure if that effects tobiash or not
22:18:22 <corvus> #info ping jlk with github3.py test results to make release
22:18:38 <corvus> jlk: have a link handy?
22:19:40 <jlk> digging itup
22:19:56 <jlk> https://github.com/sigmavirus24/github3.py/pull/813 is the pr
22:20:10 <jlk> https://github.com/sigmavirus24/github3.py/issues/794 is the (closed perhaps too early) issue
22:20:17 <fungi> good ol' sigmavirus24
22:20:32 * prometheanfire misses sigmavirus
22:20:34 <corvus> #link https://github.com/sigmavirus24/github3.py/issues/794 closed github enterprise issue
22:20:50 <corvus> #link https://github.com/sigmavirus24/github3.py/pull/813 potential github enterprise issue?
22:20:55 <fungi> electrofelix reported that issue?
22:21:00 <fungi> small world
22:21:10 <corvus> both in fact :)
22:22:28 <pabelanger> nice
22:23:33 <corvus> i think that's it for this subject...
22:23:43 <corvus> #topic Open Discussion
22:23:45 <jlk> Do we know electrofelix?
22:24:12 <corvus> jlk: oh, yes, he's a contributor to jenkins-job-builder, and some other openstack-infra-related projects
22:24:50 <corvus> that's his nick in irc; can often be found in #openstack-infra or #zuul
22:25:26 <fungi> a.k.a. pelix
22:25:45 <corvus> oh that name is new to me
22:25:50 <fungi> jjb core reviewer
22:26:11 <fungi> he was pelix for a while beforee electrofelix
22:26:48 <corvus> in other news, i think i'm pretty close to getting git.zuul-ci.org up and running
22:26:57 <pabelanger> I have an awkward patch up for ipv6 support in zuul: https://review.openstack.org/556327/ would love to discuss that this week.
22:27:23 <corvus> it will be especially nice to be able to tell people to use "https://git.zuul-ci.org/zuul-jobs"
22:27:36 <pabelanger> yah, that would be good
22:27:39 <pabelanger> and cool
22:27:42 <Shrews> pabelanger: is there an equivalent patch for nodepool?
22:28:11 <jlk> corvus: oh! I wonder then if he's another zuul + github user
22:28:16 <pabelanger> Shrews: yah, that merged this morning https://review.openstack.org/556209/
22:28:30 <Shrews> ah, ok. missed that
22:29:15 <fungi> though biggest impact for zuul-{base-,}jobs since it goes in everyone's config if they want to cd our stdlib
22:29:47 <corvus> jlk: if not currently, i'm given to understand, he'd like to be :)
22:29:58 <corvus> i missed that too
22:30:04 <corvus> i'd really like it if we stopped merging changes without docs
22:30:08 <corvus> in fact, i'm going to propose a revert of that
22:30:52 <corvus> we have past the point where we should merge a change without tests and docs
22:31:04 <clarkb> thats awkward that statsd has to have an explicit ipv6 flag
22:31:17 <corvus> yeah, i assume it's because it's udp?
22:31:29 <clarkb> maybe instead of a flag we just detect if it is an ipv4 addr or ipv6 and do the right thing?
22:31:48 <Shrews> clarkb++
22:31:48 <pabelanger> we can do that, but no way to stop statsd from then detecting it again
22:32:10 <corvus> clarkb: we'd need to know if the ipv6 address is routable from us, right?
22:32:39 <clarkb> corvus: yes, I think assuming it is if you have global ipv6 address is mostly safe
22:32:46 <pabelanger> https://github.com/jsocol/pystatsd/blob/master/statsd/client.py#L152 is how it works today in statsd. If we want to propose a change, I'm okay witht hat
22:32:48 <clarkb> (this seems to be behavior of most applications)
22:33:05 <corvus> clarkb: yeah, i'd prefer that approach too
22:33:09 <fungi> yeah, if both are global then fine, otherwise that's not entirely trivial to determine
22:33:36 <corvus> so basically, do a lookup of the host, see if it has a global ipv6 address, if so, see if we do, if so, set the ipv6 flag...
22:33:43 <corvus> (why doesn't py-statsd do that?)
22:34:04 <fungi> and "happy eyeballs" approaches don't work so well for connectionless datagrams
22:35:39 <corvus> if someone had an internal (not global) statsd server only on ipv6, this scheme wouldn't work for them.  they'd need a flag.
22:36:25 <corvus> oh, there's one more thing related to releases i should mention --
22:36:33 <clarkb> its unfortunate that is even a thing considering ipv6's available address space being there to avoid that problem entirely
22:36:43 <fungi> but yeah, if you're hoping to take a non-global route to a v6 destination they can't reach via v4, then maybe the onus is on them to make dns only resolve the thing they can reach or use an address literal?
22:37:00 <corvus> fungi: yeah, that might be enough of a workaround
22:37:43 <corvus> after we do release 3.0.0, and before we release 3.0.1 -- we should land support for reno.  i don't want to do it before 3.0.0 (because i don't want to even pretend we can write those release notes).  but going forward, i'd like zuul and nodepool to have real release notes.
22:37:55 <corvus> so that basically means that's the first post 3.0.0 thing we need to land
22:38:22 <pabelanger> so, use socket.getaddrinfo() again to detect if ipv6, then setup flag on statsd client. wfm
22:38:50 <corvus> pabelanger: yeah, let's give that a shot.  because i think we all agree this option is weird. :)
22:39:05 <pabelanger> indeed
22:39:49 <fungi> pabelanger: not quite as simple as all that, need to check that you have a global route i think? but basically yes
22:40:36 <corvus> fungi: check the route, or check to see if we have a global ipv6 address?
22:41:11 <pabelanger> fungi: okay, I was going to use the same logic for fingergw we did for ipv6, I don't think we checked global route there
22:41:54 <fungi> well one or the other should work, just pointing out it's not sufficient to check that socket.getaddrinfo() works via v6
22:43:03 <pabelanger> fungi: okay, maybe after meeting we can look at https://review.openstack.org/551015/ and see if we'd have the same issue
22:43:34 <fungi> yep, we can hash it out outside the meeting for sure
22:44:03 <corvus> i had an action item a while back to follow up on the meeting time change.  i did not do that last week, but have started doing it this week.
22:44:36 <corvus> i don't think there's a way we can make everyone happy, so i'll probably end up making an executive decision one way or the other.  however, there's another alternative i'd like us to consider:
22:45:24 <corvus> end the practice of regular meetings and focus more on mailing list discussions for wide dissemination/collection of information, and ad-hoc irc discussions for when that's not practical
22:45:39 <corvus> as the folks who randomly showed up to today's meeting, what do you think about that? :)
22:46:07 <Shrews> wfm
22:46:20 <pabelanger> I admit, i like the structure IRC meetings, but happy to try something new if people would like
22:46:34 <fungi> as a completely non-representative sample, i think it would be nice to venture headlong into this new governance unencumbered by the baggage of weekly meetings
22:47:26 <clarkb> I know monty struggles with mailing lists based on some recent conversations. But I imagine ad hoc conversations can get him pointed to relevant threads as needed
22:47:52 <fungi> culturally the openstack community has used scheduled meetings as a crutch to avoid discussing over mailing lists
22:48:20 <fungi> would be nice to see if not having scheduled meetings reduced that tendency
22:48:32 <clarkb> I also have a really hard time with threads that get outlooked, but maybe I just need a better tactic for dealing with those
22:49:09 <corvus> it's really hard to get to a decision on a mailing list.  so it will require we develop some different skills.
22:49:32 <fungi> go from "i should add that to the agenda for some upcoming meeting which may be a week away and might already be too overbooked to get to my topic" to "i'll post to the mailing list about this now"
22:49:38 <corvus> and yeah, meetings provide a structure, timeboxed by their nature, which we will have to be cognizant of.  (bi-?)weekly summaries may help with that.
22:49:50 <jlk> I'm pretty bad about email for this project, but I can respond to random channel pings for data
22:50:19 <clarkb> corvus: re making decisions and lack of time boxes maybe once a thread has had reasonable conversation we set a N day timer on completing the decision making
22:50:24 <pabelanger> yah, summaries might help with that
22:50:24 <corvus> also, we have a dedicated mailing list, which may make it a bit easier for folks to deal with than, for instance, openstack-dev.
22:50:24 <clarkb> similar to specs I uess
22:51:39 <pabelanger> +1 to trying MLs
22:51:58 <corvus> okay, maybe i'll reply to the ml thread about the meeting time with the suggestion we consider terminating it.
22:52:20 <corvus> i'll indicate in this meeting, at least, we saw some value to the meeting, but there wasn't widespread revolt
22:53:01 <fungi> i don't find the meetings useless, but concede they may be an antipattern
22:53:42 <corvus> probably the biggest thing is that it's an hour where we all actually plan on talking about zuul.  for those of us with multiple hats, that can be surprisingly important.
22:53:51 <corvus> ^ (for me)
22:53:55 <fungi> fair!
22:54:20 <pabelanger> ++
22:54:38 <corvus> but it's an hour that several other people plan on being asleep during, so... there you have it :)
22:55:03 <corvus> anything else, or should we wrap it up?
22:55:10 <fungi> i'm good
22:56:11 <Shrews> fwiw, ML discussions have been extremely valuable (e.g., the zk retry stuff), but was surprisingly lacking on much participation. not sure if an IRC meeting would be much better (or worse)
22:57:07 <Shrews> just more food (mmmm, food) for thought
22:57:43 <ianw> personally i like the meetings, as someone quite remote it's often very vaulable to get decisions made in them
22:58:55 <corvus> ianw: if we do keep it, maybe we could be more strict about the agenda (so people can more reliably anticipate if they should attend), and cancel it if there's nothing on it.
22:59:26 <corvus> ianw: otoh, we could ping the folks active on an email thread and say "hey, so how about we decide on that thing we were emailing about"
23:00:11 <corvus> thanks all!
23:00:12 <corvus> #endmeeting