22:01:44 <corvus> #startmeeting zuul
22:01:45 <openstack> Meeting started Mon Feb 19 22:01:44 2018 UTC and is due to finish in 60 minutes.  The chair is corvus. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:01:46 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:01:48 <openstack> The meeting name has been set to 'zuul'
22:01:50 <corvus> #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:02:05 <corvus> let's talk about v3 blockers
22:02:13 <corvus> #topic Release blockers
22:02:21 <corvus> #link roadmap https://storyboard.openstack.org/#!/board/53
22:02:30 <corvus> there are 4 items on that list
22:02:39 <corvus> the first is the zuul_json fix
22:02:49 <corvus> dmsimard is working on that
22:02:50 <dmsimard> hi
22:02:58 <fungi> awesome
22:03:01 <corvus> dmsimard: do you need any help on that?
22:03:38 <dmsimard> I'm working on that *right now*, I sent a patch that should reproduce the issue and if it does I'll see from there (might toggle keep or autohold)
22:04:09 <corvus> dmsimard: oh you mean you're trying to reproduce it in infra's zuul?
22:04:37 <dmsimard> corvus: I'm unable to reproduce it locally by exercising zuul_stream and zuul_json by themselves (through the test-logs tool script)
22:05:24 <dmsimard> https://review.openstack.org/#/c/545994/ should hopefully trigger it
22:06:11 <mordred> dmsimard: if it does, is the next step to figure out what needs to happen with that same test playbook in a zuul unittest to be able to cause the error there?
22:06:59 <dmsimard> mordred: the next step is to see if https://review.openstack.org/#/c/543633/ fixes it somehow
22:07:23 <corvus> well, we need a test reproduction to see if it fixes it
22:07:37 <dmsimard> as I recall, that patch on it's own wasn't enough to fix everything because the feature/zuulv3 version of that patch (same code) wasn't enough to fix it at one point, see the finger URLs here: https://review.openstack.org/#/c/504238/
22:08:25 <dmsimard> This may have changed with ansible dot releases or whatever might have happened since then, I'm just not confident to let that merge without additional validation first
22:08:38 <dmsimard> so validation is being done
22:08:44 <mordred> yah
22:08:51 <corvus> dmsimard: well, you said the test in the change didn't fail without the fix
22:08:58 <corvus> so it's not actually a test of the fix
22:09:01 <mordred> right
22:09:12 * rbergeron sneaks into a seat ever so inconspicuously
22:09:23 <dmsimard> corvus: right, so I might need help understanding why that is once we figure out how to properly reproduce it
22:09:46 <dmsimard> I didn't need more than that to validate the fix for ara
22:10:14 <dmsimard> https://github.com/openstack/ara/commit/6fd7a95ab940d7a78ade0040232a1ffdaf41ccfc
22:10:17 <corvus> so if there's still a problem (apparently there is, since we saw this, what, 1 or 2 weeks ago?) we need a test case to show the problem, so that we can confirm the fix and verify it doesn't regress in the future
22:10:23 <mordred> dmsimard: well, let me know if you hit a wall on making it break again
22:10:28 <mordred> corvus: agree
22:10:29 <dmsimard> mordred: ack
22:10:34 <dmsimard> corvus: yes, we need some sort of test indeed
22:11:05 <corvus> i'm also happy to help
22:11:23 <corvus> okay, so that's thing 1/4 -- hopefully we'll sort it out soon
22:11:24 <dmsimard> ok, I'll shout once I get more info
22:11:30 <mordred> dmsimard: honestly - give it SpamapS - he can get anything to break :)
22:11:49 <corvus> thing 2/4 is, incidentally, to refactor zuul_stream so we have better testing :)
22:11:50 <dmsimard> mordred: nod, I also need to work on that exit code thing we discussed the other day
22:12:06 <corvus> mordred: i saw patches for that; what's the state?
22:12:20 <mordred> corvus: yah - I started work on part of that, but have shifted to the javascript stack for now
22:12:41 <corvus> okay, so we shouldn't focus on that yet.. javascript first
22:12:44 <corvus> javascript is thing 3/4
22:13:19 <mordred> https://review.openstack.org/#/q/topic:zuul-stream-rework is the work started down the streaming piece of the puzzle
22:13:38 <corvus> mordred: is there anything anyone can do to pitch in on that now? or should we just wait?
22:13:40 <mordred> it needs to be reworked a little bit to have the stream listener be tied to the job and not to the playbook
22:14:10 <dmsimard> I'll take a look at those patches and review
22:14:38 <mordred> corvus: anyone who feels like hacking should be able to take what's there and move forward if they want to - main thing is the lifecycle of the listener needs to be essentially the same as the ssh agent
22:15:10 <mordred> this is because of the controlpersist
22:15:43 <mordred> which can only forward something _once_ - not every time - since subsequent connections just re-use the previous forwarded port
22:16:06 <mordred> so when we forward it the first time, the thign we forward needs to continue to exist for the next times
22:16:20 <corvus> that means the listener can't start in the callback, right? it needs to be started by the executor
22:16:59 <mordred> corvus: that's right. and it probably needs to be started as a subprocess rather than a thread, otherwise we've got a bazillion threads running around
22:17:14 <mordred> corvus: so it may take a few more stabs to find the right thing
22:17:45 <corvus> ok.  so if anyone feels like poking at that, talk to mordred
22:18:01 <mordred> yah - my feelings will not be hurt if someone wants to dive in :)
22:18:10 <corvus> we're blocking the v3 release on this basically because there's so little testing of this right now, if someone comes in with a bug, we can't fix it
22:18:23 <mordred> but also - if nobody else does, I should be able to return to it in a little bit
22:18:58 <corvus> okay, thing 3/4 is javascript
22:19:19 <mordred> that's very close - although I've still got two of the patches marked WIP
22:19:22 <corvus> mordred: we merged 4 pre-req patches today -- does https://review.openstack.org/538099 need any reworking because of that?
22:19:47 <mordred> corvus: yes - it does - that's the next thing on my list
22:20:14 <mordred> that reworking shouldn't take long - I just wanted to be able to verify it against the live status page in addition to the unit tests
22:20:28 <mordred> cause, you know - moving parts
22:20:36 <corvus> ya
22:21:13 <corvus> that's the best way to find out what extra unit tests to add :)
22:21:21 <mordred> EXACTLY :)
22:21:34 <mordred> http://logs.openstack.org/55/545755/1/check/build-javascript-content/0012b09/npm/html/status.html <-- however, is the end of the stack, and you can see it working
22:21:46 <corvus> and speaking of tests, that one now includes the test that we can serve the js files in all 3 ways we know zuul is deployed
22:21:59 <mordred> yup. thanks for that - it's already been helpful :)
22:22:29 <corvus> i'm looking forward to not breaking openstack-infra's zuul everytime we land a web change :)
22:22:43 <clarkb> ++
22:22:46 <mordred> RIGHT?
22:23:04 <fungi> will be nice
22:23:35 <corvus> okay, so if mordred and dmsimard ask for help about any of those 3 efforts, please pitch in -- those are our internal release blockers, and we're so close
22:23:52 <corvus> thing 4/4 on the list is the github3.py release
22:23:52 * mordred is curious on the state of 4 ...
22:23:58 <corvus> jlk: around?
22:24:12 <jlk> in fact, I am.
22:24:16 <corvus> yay!
22:24:26 <pabelanger> I'll read up 2/4 tonight
22:24:41 <jlk> I have no real change in news, other than continued work on teh open issue. https://github.com/sigmavirus24/github3.py/pull/773
22:25:02 <jlk> A pile of my work was merged into that PR, but there is more to do yet it seems.
22:25:47 <corvus> jlk: if folks had time to pitch in on that, are there more units of work to hand out?
22:26:00 <corvus> or it it mostly in the hands of the maintainers at this point?
22:26:50 <jlk> well... not easily no. It's mostly in the hands of the maintainer. I know part of the work that needs to be done, but a lot of this is kinda beyond me too. It's a fairly big restructuring of how the objects interact, but it's mostly opaque to the API users.
22:27:41 <jlk> I think it's down to the last few files to go through, but they all intertwine a bit. I don't know what the maintainer's strategy is tor the rest, but I'll ask.
22:27:53 <corvus> cool
22:28:16 <corvus> we probably aren't going to be ready to release zuul this week, so we still have a bit of time
22:28:43 <corvus> but once we clear the first 3 things on that list, and this is all that remains... i think we need to consider a backup plan
22:28:55 <corvus> which i think is probably to vendor github3.py into zuul temporarily.
22:29:14 <corvus> hopefully it won't have to live there for long
22:29:19 <fungi> vendoring code copies is the way of the world these days
22:29:19 <jlk> I think that's the right thing to do
22:29:25 <mordred> yah. much as I'm not a fan of vendoring things like that, it seems like an ok choice in this instance
22:29:32 <corvus> but i think that's the best way we can get zuul out there without making a big fuss.
22:29:52 <mordred> fungi: so is electing fascists ... but that doesn't mean I want to continue the trend :)
22:30:00 <fungi> fair!
22:30:01 <corvus> mordred: ahem
22:30:06 <jlk> o_O
22:30:09 <mordred> uhoh. /me politicaled
22:30:14 * rbergeron fans herself
22:30:17 * mordred backs slowly out of the room
22:30:35 <jlk> oh now I get the context. I was missing it for a moment.
22:30:41 <corvus> okay, so we'll see how things shake out, and if it comes to that, i'll volunteer to do the vendoring
22:30:59 <mordred> corvus: kk. I will volunteer to review
22:31:15 <corvus> and of course, we'll remove it and release as soon as github3.py releases
22:31:21 <mordred> ++
22:31:32 <fungi> sounds like a reasonable approach
22:32:02 <corvus> okay, that's the whole release-blocker list...  thanks everyone whose working on those, and everyone else who's assisting :)
22:32:09 <fungi> hopefully that doesn't take long enough that we have to start worrying about tracking security bugs in a vendored unreleased depndency
22:32:09 <corvus> #topic open discussion
22:32:14 <mordred> corvus: is it worth going ahead and doing that since we have the git url in our requirements now anyway?
22:32:16 <corvus> fungi: yeah
22:32:35 <clarkb> worth noting a few behavioral changes have merged recently in zuul. We've updated how timeouts work and the status api moved
22:32:57 <mordred> clarkb: ++ ... .json have been removed from rest api calls
22:32:59 * rbergeron haz a thing for open discussions when the moment presents itself :)
22:33:00 <corvus> mordred: i don't think it's going to be very impactful for folks until we're nearly ready to release...
22:33:06 <mordred> corvus: kk
22:33:14 * mordred presents rbergeron a moment
22:33:35 <mordred> oh. wait. that's an old loaf of bread ...
22:33:37 <rbergeron> ooh. neat
22:33:55 <rbergeron> oh, not so neat. I guess i should obtain a moment from corvus instead
22:33:58 <rbergeron> ;)
22:33:59 <rbergeron> anywayyyyyyy
22:34:29 <rbergeron> So: Some of you may have seen the lovely message from mr. rich bowen on the dev list re: video things at the PTG - I chatted with him a few weeks ago and...
22:34:36 <pabelanger> Over the weekend, we had another issue where zuul.o.o lost access to nodepool.o.o (zookeeper) a few times. This results in zuul unlocking nodes, and nodepool mass deleting all in-use nodes, which deletes them under running jobs. Aside from moving to zookeeper cluster, plan for PTG. Is there other way to help protect running jobs from not losing their nodes under them and resulting in a bunch of
22:34:37 <pabelanger> SSH failures
22:34:55 <rbergeron> he was super willing to record some interviews (like, group style, 2 or 3 humans, plus anyone who might be using zuul aside from openstack who wants to talk as well)
22:35:05 <rbergeron> for us, which might be useful / interesting fodder to add to the website.
22:35:08 <rbergeron> http://lists.openstack.org/pipermail/openstack-dev/2018-February/127525.html
22:35:09 <mordred> rbergeron: \o/ that sounds like good
22:35:16 <corvus> i would love more website fodder!
22:35:17 <rbergeron> So i am here to procure wranglings of a few of you to do such things.
22:35:17 <fungi> that'd be neat
22:35:37 <mordred> rbergeron: I'm always happy to sit in front of a camera - especially when I don't have to prepare before-hand
22:35:37 <dmsimard> I can vouch for rbowen doing great work with the interviews, had a few in the past
22:35:50 <dmsimard> He's on #rdo (rbowen) if you want to chat
22:35:57 <rbergeron> (i also said I'd try to wrangle someone into volunteering to also chat about broader infra things, and I should poke dmsimard about talking about ARA, and maybe mordred talking about ansible / shade / modules along with other humans)
22:36:11 <rbergeron> (and, you know, wrangling is my thing, so, yeah)
22:36:15 <dmsimard> rbergeron: spoiler, I already had an interview about ARA
22:36:17 <corvus> tobiash: is usually not around for these meetings, but he will be at the ptg and is a non-openstack user; we should see if he'd be willing to chat
22:36:30 <rbergeron> in any case: lest we all say "yeah yeah" and then find ourselves exhausted on thursday night:
22:36:48 <clarkb> ya I tend to avoid adding too much for ^ that reason
22:36:51 <pabelanger> +1
22:36:59 <rbergeron> he's doing things tues-thurs, though he might make an exception for a monday thing for anyone who is only around on monday (AHEM COUGH JLK)
22:37:26 <jlk> hai. I mean, I could talk about our failed/cancelled CI project, but uh...
22:37:41 * jlk wonders if Clint would be around
22:37:42 <rbergeron> so: I can either voluntold a few of you, but i know infra as well as zuul folks have some cross-project things going on / help sessions etc, so if anyone is willing to pinpoint an afternoon or moning where they feel safe :)
22:37:56 <jlk> oh, I mean IBM is still using it internally with great success as far as I know.
22:37:57 <fungi> jlk: or you could unveil your secret plans for world domination
22:38:31 <rbergeron> jlk: mostly highlighting "yay github works with zuul" is the thing. it's nicer to highlight it with the human who has done a bunch of that work :) but... after, post-thing-via-video, whatever is cool.
22:38:32 <clarkb> I think clint said he won't be making it
22:38:35 <rbergeron> or i can dress up as you
22:38:46 <jlk> oh yeah I can talk about that too.
22:39:27 <rbergeron> and also: it'd be neat if like, you know, not just people employed by my gracious employer overlords
22:39:30 <mordred> yah. also - since jlk works at github now - an interview on how awesome zuul github integration is from him would be neat
22:39:31 <rbergeron> pretty please
22:40:01 <jlk> yup, I think I can toe that line and make it clear I did the work before joining GitHub, and that it's not an official GitHub effort :D
22:40:06 <mordred> then I can take words out of context and say "github endorses using zuul for all CI"
22:40:12 <rbergeron> jlk: but he can do things monday if you have things to say, or if we can pair you up with a human, and maybe we can do another pair of humans in a few days :)
22:40:15 <mordred> jlk: oh - darn - you anticipated my evil
22:40:34 <rbergeron> mordred: you're always evil, is anticipaaaaation really a thing there?
22:40:36 <corvus> rbergeron: cool, with jlk, and maybe asking tobiash tomorrow, we might get 2 non-openstack folks
22:40:40 <mordred> rbergeron: good point
22:40:44 <fungi> mordred: careful editing of his interview in post production can get around it
22:40:52 <corvus> are there any other non-openstack folks going to the ptg?
22:40:56 <clarkb> fungi: the stephen colbert treatment?
22:41:00 <jlk> I should be back in Dublin again on Thursday and could slip away for an hour or so if it doesn't workout otherwise.
22:41:06 <rbergeron> non-openstack or not-just-openstack? :)
22:41:27 <corvus> rbergeron: not-just-openstack -- i guess folks that can give a non-openstack-infra perspective on zuul
22:41:37 <mordred> rbergeron: also - melvin is signed up for an interview about openlab, so that should be another non-redhat-zuul talk
22:41:47 <rbergeron> jlk: ack, I will see what we can do for monday and also keep that in mind.
22:42:01 <jlk> thanks
22:42:03 <rbergeron> mordred: oh good, that was a thing i brained for a moment before the squirrels came earlier today
22:42:14 <corvus> there's the software factory perspective
22:42:28 <dmsimard> there'll be some folks involved in software factory at the PTG
22:42:33 <corvus> and maybe anyone using zuulv3 for openstack third-party testing
22:42:47 <rbergeron> so: should I pick some times on the sheet and propose them on the zuul list -- or just sign up for things and also eyeball the infra plan and hope that if we screw up that it will still be okay? :)
22:42:49 <mordred> rbergeron: have I mentioned that squrrels have literally been trying to eat through our roof above our bedroom?
22:42:59 <rbergeron> dmsimard: ahh, do you know who exactly?
22:43:02 <rbergeron> mordred: TMI
22:43:04 <clarkb> rbergeron: did you see my rough schedule for infra rooms?
22:43:05 <fungi> texas-sized squirrels
22:43:12 <dmsimard> rbergeron: I'll double check and get back to you
22:43:16 <mordred> very loud dubstep scares them away, fwiw
22:43:26 <clarkb> rbergeron: tldr is monday tuesday is more infra office hours for random projects. Expect to have a bit of zuulv3 related stuff there
22:43:39 <rbergeron> clarkb: I did eyeball it, or maybe i only eyeballed the etherpad and not an actual schedule of activities
22:43:42 <pabelanger> dmsimard: rbergeron I know jpena and apevec are also attending from SF
22:43:45 <clarkb> then wednesday and first half of thursday just focus on zuul and transition into more general infra topics thursday afternoon and friday
22:43:47 <rbergeron> clarkb: yep, caught that
22:44:01 <rbergeron> *nod*
22:44:12 <fungi> clarkb: fwiw, i'm expecting monday/tuesday to be a _lot_ of zuul v3 job config q&a
22:44:26 <clarkb> fungi: ya
22:44:28 * fungi predicts a line out the door
22:44:41 <corvus> everything good around here has a line out the door, so ++
22:44:44 <fungi> (okay, hopefully not *that* bad)
22:44:45 * rbergeron will be the gatekeeper with the marshmallows
22:45:06 <rbergeron> ...and bad jokes and puns
22:45:52 <Shrews> what, no calamari?
22:45:53 <rbergeron> okay, so: i'll find slots and do my best to make them work with times and will just blather at y'all on the channel. i think that's probably best (and probably in the morning because wow, i have been typing for like 11 hours)
22:46:09 <corvus> rbergeron: why don't you send something to the zuul list (whether it's "i think you should do these times", or "i signed you up for these times") since it may get better exposure there, especially for the not-just-openstack folks
22:46:28 <mordred> Shrews: nicely played
22:46:35 <rbergeron> corvus: sounds good, and thank you for the "yes, more humans will see that" reminder :)
22:46:49 <rbergeron> (even i forget this here and there despite reminding others of it all the time!)
22:47:20 <rbergeron> shrews: only if you want me to vanish
22:48:03 <rbergeron> #action rbergeron to mail list with times she signed people up for after thorough research and hope for non-conflict to do zuul interviews for yayyyy, content
22:48:34 <rbergeron> unrelated: corvus, can we mail the meeting logs to the zuul list? (I thought this happened automagically but i may be thinking of fedora instead)
22:48:42 <rbergeron> (or maybe they go to infra)
22:48:49 <rbergeron> (i guess it is slightly related as it made me think of this)
22:49:03 <mordred> rbergeron: they go to web
22:49:07 <clarkb> rbergeron: the logs are hosted at http://eavesdrop.openstack.org but aren't emailed otu
22:49:32 <rbergeron> (and with that: that is all the squirrels that i have)
22:49:39 <rbergeron> ah, okay. i may be confuzzled
22:49:40 <corvus> we could (manually) email logs (and agenda reminders) if folks would find that useful...
22:49:51 <mordred> rbergeron: http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-02-13-19.01.html
22:50:00 <rbergeron> (i can see how that would be miserable on openstack-dev, but maybe not on the zuul list)
22:50:01 <corvus> if we do that, we should start with the next meeting cause i totally haven't been #ing
22:50:11 <mordred> corvus: it's possibly not a terrible idea to email log summary and links
22:50:16 <rbergeron> corvus: i have mostly found it useful in that sometimes folks are like "OH HEY NEAT I HAD NO IDEA THERE WERE MEETINGS"
22:50:25 <fungi> we used to have someone doing some of that, but determined that people used their calendars and knew where to find the logs/minutes anyway
22:50:31 <clarkb> also no meeting next week? It will be middle of the night almost right?
22:50:33 <corvus> oh, speaking of next meeting, let's not have one next week?
22:50:38 <rbergeron> "and i can see i might have things to yap about"
22:50:50 <fungi> yeah, it'll be like 10-11pm local
22:50:54 <rbergeron> corvus: ack :) and yes, no meets next week plz
22:50:57 <corvus> also, i won't be around for the meeting after that -- do folks want to have it, or call it a PTG recovery day?
22:51:31 <pabelanger> wfm
22:51:35 <corvus> i'll probably be drinking whiskey to warm up after tramping through a bog or something
22:51:44 <clarkb> I will be traveling so will be ptg recovery day for me
22:51:52 <clarkb> I'll be on a plane
22:51:53 <rbergeron> my friend is in dublin and he saw an actual tanker truck of guinness
22:51:56 <fungi> convenient bit of trivia, while we're in dublin they'll be on utc basically, so i won't need to spend the week converting times to add them to my calendar (for once)
22:52:04 <pabelanger> rbergeron: OMG
22:52:21 <rbergeron> pabelanger: right. like, super cliche yet also awesome
22:52:32 <pabelanger> +1
22:52:39 <corvus> pabelanger: to your question earlier -- no we don't have a way of avoiding that other than "fix whatever communication problem is between zuul and zookeeper"
22:53:07 <corvus> so hopefully this gets better with the cluster
22:53:21 <rbergeron> https://scontent.fphx1-1.fna.fbcdn.net/v/t31.0-8/28061721_10213872342754761_459075879379154238_o.jpg?oh=7b377a0a69bb3df528ad5cac28837917&oe=5B11639B
22:53:30 <pabelanger> corvus: yah, that is what I figured. Wanted to ask
22:54:15 <mordred> ++ cluster
22:54:59 <corvus> okay, i saw no objections to canceling next 2, and some support
22:55:22 <corvus> #info no meeting on feb 26 or mar 5
22:55:35 <corvus> anything else?
22:55:53 <pabelanger> safe travels
22:55:56 <rbergeron> plz to have safe metal tubing
22:56:25 <rbergeron> and respond to pabelanger's infra dinner things!
22:56:41 <corvus> ++.  thanks all, see some of you soon :)
22:56:44 <fungi> thanks!
22:56:45 <corvus> #endmeeting