19:03:08 <fungi> #startmeeting infra
19:03:09 <openstack> Meeting started Tue Dec 13 19:03:08 2016 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:03:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:03:12 <openstack> The meeting name has been set to 'infra'
19:03:13 <olaph> o/
19:03:17 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:22 <fungi> #topic Announcements
19:03:30 <fungi> #info REMINDER: If you want to come hack on Infra things at the PTG a couple months from now in Atlanta, don't forget to sign up!
19:03:36 <fungi> #link https://pikeptg.eventbrite.com/
19:03:37 <fungi> #link http://www.openstack.org/ptg
19:03:44 <fungi> i'm told there is plenty of travel assistance available too--if you need it don't be embarassed to ask for it
19:03:47 <fungi> #link http://www.openstack.org/ptg#tab_travel
19:04:31 <fungi> SpamapS asked if there would be zuul v3 goodness happening there, and unless the next ptl (if it isn't me) decides otherwise, i expect it to be a primary activity
19:04:51 <fungi> i hope to have some semblance of an agendaish thing put together soon
19:05:02 * morgan needs to book a hotel for PTG things
19:05:18 <fungi> but really if there's anything infra-related you want to work on and you think at least one other person will be around to collaborate on it with you, we'll work it in somehow
19:05:44 <fungi> as always, feel free to hit me up with announcements you want included in future meetings
19:05:50 <fungi> #topic Actions from last meeting
19:05:58 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-12-06-19.03.html
19:06:05 <fungi> pabelanger add openstackci::zuul_launcher puppet class
19:06:10 <fungi> that was the only one we had
19:06:16 <fungi> we're using that at this point, yeah?
19:06:39 <fungi> or are we still going through the openstack_project module for it?
19:07:41 <fungi> hrm, yeah i don't see any recent changes of pabelanger's, open or merged, for puppet-openstackci
19:07:53 <clarkb> I think zaro and pabelanger mentioned that is still todo
19:08:13 <pabelanger> o/
19:08:19 <fungi> oh, there's pabelanger!
19:08:23 <pabelanger> fungi: clarkb: yes, on the list to start today
19:08:28 <fungi> should i just push that reminder back on the stack for now?
19:08:32 <pabelanger> sorry for dragging this on
19:08:39 <fungi> no apologies needed
19:08:48 <fungi> #action pabelanger add openstackci::zuul_launcher puppet class
19:08:55 <fungi> we can revisit next week
19:09:07 <pabelanger> Yes, will have patches ready before then
19:09:13 <fungi> awesome. thanks!
19:09:15 <fungi> #topic Specs approval
19:09:24 <fungi> #info APPROVED "Zuul v3: use Zookeeper for Nodepool-Zuul protocol"
19:09:29 <fungi> #link https://review.openstack.org/305506 "Zuul v3: use Zookeeper for Nodepool-Zuul protocol" spec update
19:10:00 <fungi> that's even less scary now that we're actually using zk in nodepool.o.o already ;)
19:10:15 <jeblair> yay!  and i'll shortly clear the zuulv3->master merge so we can start working on it
19:11:46 <fungi> #info "Newton testing on Xenial" spec moved to implemented
19:12:08 <fungi> [and there was much rejoicing]
19:12:16 <fungi> #info "Automate Creating Branches" spec moved to implemented
19:12:54 <fungi> i also have proposed to mark the artifact signing spec as implemented
19:13:17 <fungi> #link https://review.openstack.org/409906 "Artifact signing is now implemented" change
19:13:39 <fungi> though there's one last outstanding documentation change it's waiting on reviews for:
19:13:56 <fungi> #link https://review.openstack.org/409905 "Document documenting rotated signing keys" change
19:14:32 <fungi> if anybody thinks of any other specs in the approved list that can be moved to abandoned or implemented, please give me a heads up
19:14:40 <fungi> #topic Specs approval: PROPOSED "Zuul v3: update with Ansible role information" spec  update (jeblair)
19:14:46 <fungi> #link https://review.openstack.org/381329 "Zuul v3: update with Ansible role information" spec update
19:15:21 <jeblair> i think this is ready for approval; and with it, we should be able to actually write the bit of zuulv3 that runs ansible jobs for real
19:16:05 <rcarrillocruz> i like the fact it leaves the door open for requirements.yaml
19:16:33 <rcarrillocruz> +1 from me
19:17:40 <fungi> yeah, seems like a great fleshing out of the missing bits for the ansible portion of the spec
19:18:02 <fungi> anybody have any immediate questions about it they want to raise here?
19:18:36 <fungi> #info The "Zuul v3: update with Ansible role information" spec update is open for Infra Council vote until 19:00 UTC Thursday, December 15.
19:18:47 <jeblair> thanks!
19:18:56 <fungi> if you come up with questions, obviously follow up in the review linked above
19:19:19 <fungi> #topic Priority Efforts
19:19:57 <fungi> just a side note that the xenial job transition spec which was a priority effort was, as i mentioned, moved to implemented
19:20:09 <clarkb> yes after a long week of staring at yaml this is now done
19:20:17 <fungi> though the publish jobs for the specs site raced, so it doesn't reflect it at the moment
19:20:37 <fungi> i'm going ahead and removing it from the priority efforts section of our agenda now
19:21:11 <fungi> thanks clarkb for driving that, and to everyone who helped write and review changes for it
19:21:24 <jeblair> clarkb: thanks!
19:21:30 <clarkb> its worth noting that there is apparently a libvirtd realloc issue that e-r is tracking on xenial. (This is why we test like this)
19:21:45 <pabelanger> indeed, thanks for that
19:22:13 <fungi> and it's now officially off the agenda ;)
19:22:39 <fungi> i don't see any priority effort updates for this week, aside from the zuul v3/nodepool specs mentioned above
19:22:40 <jeblair> clarkb: that's a "nice" reminder that this is actually a thing, despite all the stuff we pip install, etc.  :)
19:22:56 <fungi> oh, though i guess...
19:23:15 <fungi> #topic Priority Efforts: Nodepool: Use Zookeeper for Workers
19:23:29 <fungi> as of friday i guess? this is more or less working
19:23:49 <clarkb> I think pabelanger said there is a known oustanding issue to be corrected by next daemon restart
19:24:10 <fungi> #link http://lists.openstack.org/pipermail/openstack-infra/2016-December/004972.html Upcoming Nodepool changes
19:24:35 <jeblair> yeah, i've proposed the merge of the zuulv3 branch to master
19:24:45 <jeblair> and after that lands, i will switch the prod machines back to running from master
19:24:46 <fungi> nodepool 0.3.1 (final pre-zk release) was tagged a couple hours ago be jeblair
19:24:48 <fungi> er, by
19:24:50 <pabelanger> clarkb: yup
19:24:57 <clarkb> I =2'd it but didn't approve so that we can check the integration tests first
19:25:00 <jeblair> and after that, we'll restart
19:25:02 <clarkb> but otherwise it looked fine to me
19:25:06 <jeblair> clarkb: good call
19:25:42 <fungi> #link https://review.openstack.org/410357 "Merge feature/zuulv3 into master" change
19:26:26 <fungi> anything else need saying about this?
19:26:50 <fungi> oh, i remember... does the zk merge effectively drop snapshot image support for now?
19:27:17 <jeblair> fungi: yes
19:27:23 <fungi> i assume snapshot images still work in 0.3.1
19:27:29 <clarkb> correct
19:27:42 <fungi> so that'll be a big warning when we release 0.4.0 or whatever i guess
19:27:52 <clarkb> it was integration tested too up to taht point
19:28:01 <clarkb> so pretty confident it was working in 0.3.1
19:29:04 <fungi> i recall there was mention of adding support for telling nodepool to use a pre-built image instead of trying to build one. i'm guessing that's not on the roadmap for 0.4.0, is it?
19:30:07 <jeblair> fungi: i think it's something we should support, but it's not in our critical path for infra atm so isn't on the immediate roadmap.
19:30:28 <clarkb> it would probably be a farily straightforward change though if someone wanted to make it
19:30:28 <fungi> i guess the messaging around it will be that we recommend 0.4.0 for anyone who can use dib-built diskimages, and those who rely on snapshot images should continue to pin to <0.4.0
19:30:42 <clarkb> basically a nodepool.yaml edit to provider images with an image uuid
19:30:51 <clarkb> and teach nodepool to not talk to zk if thats set
19:30:53 <clarkb> or something
19:30:54 <jeblair> clarkb: yep
19:31:09 <jeblair> it's mostly a matter of *not* doing things :)
19:32:08 <fungi> cool, just wanted to make sure i was clear on the point at which we were dropping support for that, since i know it's been mildly contentious
19:32:38 <jeblair> also, if anyone wants to write support for that, it can definitely be done in the new system
19:32:44 <fungi> and if nothing else, we have a recommended feature implementation should someone miss it so much that they want to hack on the alternative solution
19:32:51 <fungi> yeah, that
19:33:05 <jeblair> and the same holds true for snapshot builds
19:33:43 <fungi> anybody want to talk about anything else having to do with the nodepool work on zookeeper-connected workers?
19:34:55 <fungi> thanks jeblair and pabelanger for getting that into place last week (and to everyone else who worked on it up to now as well)
19:35:18 <ianw> i spent a bit of time looking at the centos7 zookeeper story again -> https://etherpad.openstack.org/p/zookeeper-epel7
19:35:23 <ianw> we'll keep working on it
19:35:48 <fungi> oh, right, so that nodepool 0.4.0 and zuul v3 can continue to be packaged?
19:36:04 <fungi> that looked like a crazy dep chain
19:36:32 <ianw> yeah, it's turtles all the way down
19:36:40 <pabelanger> Ya, hoping to help with the zookeeper effort for centos :)
19:37:13 <ianw> i have a "package", that's a zk release stuffed into an rpm
19:37:34 <fungi> presumably it's all been packaged on debian, so should be packagable on centos as well (if there's enough elbow grease on hand)
19:38:17 <clarkb> ya debuntu has packages
19:38:39 <fungi> #link https://packages.debian.org/sid/libzookeeper-java where the dep chain starts to show up
19:39:05 <pabelanger> fedora rawhide has zookeeper RPM, so all the bits are there, just need to get it into epel
19:39:15 <pabelanger> I've been using that locally, works as expected
19:39:35 <fungi> oh, so mostly a backporting challenge then, bot a lot of specfiles that need to be written from scratch?
19:39:42 <fungi> s/bot/not/
19:39:45 <pabelanger> right
19:39:49 <jeblair> bot a lot indeed
19:39:57 <fungi> it's a botload
19:40:41 <fungi> #topic possible hosting for a nova bugs dashboard (auggy)
19:40:47 <fungi> #link http://45.55.105.55:8082/bugs-dashboard.html current nova bugs dashboard
19:41:01 <auggy> yes! we chatted in #openstack-infra about this
19:41:16 <auggy> i just wanted to check on what you all need from me in order to evaluate this request
19:41:22 <fungi> #link http://lists.openstack.org/pipermail/openstack-dev/2016-December/108872.html Useful metrics?
19:41:28 <fungi> (related ml thread)
19:41:58 <auggy> basically we have this dashboard we use that markus wrote and hosted, and we just want it centralized so it's not depending on a single person
19:42:28 <auggy> so i thought i'd ask if it made sense to have you all host it, but if you have other suggestions then I'm open to that too
19:42:54 <fungi> #link https://github.com/markuszoeller/openstack/blob/master/scripts/launchpad/bugs_dashboard.py Creates a HTML file which can be used as a dashboard for cleanup tasks of the bug management.
19:42:57 <auggy> It's a python script that runs via cron on the hour that makes some read only queries to launchpand and then renders the results to html
19:43:05 <auggy> thx fungi !
19:43:29 <auggy> it's also potentially usable by other openstack projects but i'm not sure how custom our bug queries are in that code
19:44:11 <fungi> we have this...
19:44:16 <JayF> auggy: just as a note, Ironic has http://ironic-divius.rhcloud.com -- I don't know where the source is that generates that though. It looks like yours covers our use case + maybe more.
19:44:18 <fungi> #link http://git.openstack.org/cgit/openstack-infra/puppet-bugdaystats/tree/manifests/site.pp
19:44:28 <fungi> could likely call it from an adjacent cron
19:44:53 <jeblair> i think the infrastructure is run by and for all the openstack projects, so these should absolutely be run centrally.  any project should feel free to propose changes to infra systems to add things like this.  :)
19:44:56 <clarkb> my preference would be to incorporate wahtever we do into bugday, we can evolve it if we need to, but seems like every project has their own one of these thigns and if we had to host a different one for each project well thats not really scalable
19:45:12 <jeblair> clarkb: collaboration, if possible would be nice
19:45:19 <clarkb> (neutron had similar things at one point that did get incorporated into bugday fwiw)
19:45:26 <jeblair> clarkb: this seems a little different than bugday though
19:45:38 <fungi> #link https://git.openstack.org/cgit/openstack-infra/bugdaystats
19:45:42 <jeblair> clarkb: are you thinking of the neutron reviewday thing?
19:45:51 <clarkb> jeblair: ya
19:46:07 <fungi> yeah, neutron's thing is a gerrit dashboard url generator run as part of reviewday
19:46:19 <clarkb> ya using bug info as an input iirc
19:46:45 <fungi> well, sure, reviewday itself is also doing lp queries to line reviews up with open bugs, blueprints, et cetera
19:46:57 <auggy> so if you want me to look at bugday or another tool you already have and see about integrating this into that then i can do that and then come back
19:48:10 <fungi> bugday has graphs of bug status changes over time but is currently short-duration and high-frequency updates
19:48:24 <persia> Is the scale issue about execution resources, or maintenance over time?
19:48:29 <clarkb> right, I guess my point is I think its ok to fix deficiencies in bugday so that it works for the larger set of use cases
19:48:37 <clarkb> like neutron has done
19:48:41 <clarkb> and potentially ironic and nova could do
19:48:58 <fungi> persia: more about target audience/use case i think
19:48:58 <jeblair> yeah, any integration with existing tools would be great as collectively we can spend time on making fewer tools better rather than all writing the same tool over
19:49:33 <jeblair> however, i'd rather see us running 5 redundant tools centrally than 5 redundant tools in random hosting places
19:49:46 <persia> So, one tool with per-project views?
19:50:00 <fungi> i think sdague also makes a good point that we have a lot of tooling built around graphite now, and bugday is relatively simple and could stand to be redone in something more inline with our present toolset too
19:50:16 <jeblair> yeah, bugday -> graphite would be pretty easy
19:51:05 <fungi> so maybe this script could seed a new bugstat project of some kind that can repace the bugday use cases as well with some graphs once someone wants to implement that part of it
19:51:59 <auggy> yeah i could work with markus to set it up in an openstack repo under infra or wherever you all think is best
19:52:05 <fungi> with graphite/grafana we can easily accommodate the old bugday use case (what did our squash impact look like in a 24 hour period) along with longer trending bug managers want to see
19:52:34 <persia> With nova workflow, but for selectable project?
19:52:52 <auggy> well that's the work that would have to be done on it
19:53:09 <fungi> it wouldn't be the first time nova's workflow for something got adopted by a ton of other projects ;)
19:53:30 <auggy> i haven't looked to see how specific the lp queries are but potentially one could specify them seperately to match their bugs
19:53:48 <fungi> so i'm not going to push for adding more to bugday unless this is the start of a bugday v2 reimplementation anyway
19:54:33 <auggy> eg, what queries do you want to go with which tabs kind of thing
19:54:33 <auggy> kk how about i work with markus to set this up as an openstack-infra repo project thingy?
19:54:33 <auggy> and then do some work to make it more project universal?
19:54:33 <fungi> but running this from a cron in the bugdaystats::site class and adding a tab on status.o.o for it or whatever seems fine to me
19:55:19 <fungi> auggy: i think that sounds like a fine plan
19:55:29 <fungi> anyone disagree?
19:56:02 <jeblair> sgtm
19:56:16 <pabelanger> ++
19:56:23 <auggy> thanks :)
19:56:24 <fungi> #agreed The current Nova bug dashboard authors are welcome to import their work into a new Infra repo and then work on making it generalized for other projects' use.
19:56:56 <fungi> once that step is done, we can get more into the weeds on how to automate deployment and where to link/display it
19:57:25 <fungi> #topic Open discussion
19:57:31 <fungi> 2.5 minutes to go
19:57:59 <pabelanger> meeting next week? or holidays?
19:58:07 <clarkb> and week after and weke after
19:58:23 <pabelanger> wfm
19:58:30 <EmilienM> I didn't know infra folks would take holidays :-D
19:58:32 * EmilienM runs
19:58:33 <clarkb> I am likely to be around all but the 27th
19:58:36 <fungi> lesee...
19:58:48 * jeblair trips EmilienM before he can get to the door
19:58:51 <fungi> yeah, i'm here at least for the 20th and can chair
19:59:03 <jeblair> i should be around
19:59:11 <ttx> I would appreciate some reviews on https://review.openstack.org/#/c/408284/ and friends, to close that before the holidays
19:59:12 <clarkb> my brother works weekends and has monday's and tuesdays off so that stretch of 25-27 will likely be family time
19:59:33 <fungi> i will likely be in a car for much of the 27th so maybe we should cancel that one unless someone wants to chair for me?
19:59:42 <jeblair> i don't plan on being around from 26-30
19:59:45 <fungi> i'll be available to chair again on january 3
19:59:58 <EmilienM> just a random question: who is here during that time? in case something breaks hard
20:00:06 <fungi> so yeah, let's plan to cancel the meeting for the 27th
20:00:11 <ttx> safe bet
20:00:11 <fungi> oh, and we're out of time!
20:00:16 <fungi> thanks everyone
20:00:21 <fungi> #endmeeting