19:02:22 <fungi> #startmeeting infra
19:02:23 <openstack> Meeting started Tue Jan 24 19:02:22 2017 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:27 <openstack> The meeting name has been set to 'infra'
19:02:29 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:35 <fungi> #topic Announcements
19:02:45 <fungi> i don't have any for this week
19:02:51 <fungi> as always, feel free to hit me up with announcements you want included in future meetings
19:03:09 <clarkb> fungi: we are likely to be mostly afk next week
19:03:19 <jhesketh> o/
19:03:33 <fungi> yep, i was going to make a call for substitute chairs during open discussion
19:04:07 <fungi> as i will almost certainly be unable to run the meeting next tuesday due to quarterly employer-mandated in-person meeting time
19:04:18 <fungi> #topic Actions from last meeting
19:04:31 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-01-17-19.04.html
19:04:47 <fungi> fungi Obtain docs.openstack.org X.509 certificate.
19:04:49 <fungi> fungi Obtain developer.openstack.org X.509 certificate.
19:04:53 <fungi> i've done both of those
19:05:14 <AJaeger> great!
19:05:15 <fungi> they're in the certs directory on puppetmaster.o.o and i'm working on the change to consume them from hiera now
19:05:30 <fungi> while i was at it i updated the soon-expiring ask.o.o and zuul.o.o certs
19:06:00 <fungi> pabelanger: i also got the requested cert for trystack.org, need to figure out how we want to do that redirect
19:06:27 <fungi> right now the trystack domains are incorrectly configured as vhost aliases rather than permanent redirects
19:06:54 <fungi> #action fungi announce the infra ptg pike etherpad to relevant mailing lists.
19:07:05 <fungi> i plan to get to that later today after the tc meeting wraps up
19:07:13 <fungi> jeblair mark http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html implemented
19:07:24 <fungi> that was done and approved yesterday
19:07:25 <jeblair> i did a thing!
19:07:38 <fungi> mordred make the temporary branch for the Nodepool v3 shim.
19:08:01 <fungi> i see a feature/gearman-zk-shim branch in openstack-infra/nodepool
19:08:01 <jeblair> that thing has happened: feature/gearman-zk-shim
19:08:06 <fungi> so that seems to be done, yes
19:08:37 <fungi> #action fungi plumb https for docs.o.o and developer.o.o
19:08:53 <fungi> that ought to be done in the next couple days depending on who reviews
19:09:20 <fungi> #topic Specs approval
19:09:42 <fungi> i have a few to report here... just a sec while i organize urls
19:10:09 <fungi> #info Ethercalc spec approved
19:10:25 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/ethercalc.html Ethercalc spec
19:11:05 <fungi> #info ABANDONED "Nodepool launch and delete workers" spec
19:11:19 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-launch-workers.html "Nodepool launch and delete workers" spec
19:11:50 <pabelanger> fungi: great
19:12:01 <fungi> #info "Nodepool: Image build and upload workers" spec moved to implemented
19:12:16 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-workers.html "Nodepool: Image build and upload workers" spec
19:12:47 <fungi> er, wrong one
19:12:49 <fungi> #undo
19:12:49 <openstack> Removing item from minutes: #link http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-workers.html
19:12:51 <fungi> #undo
19:12:51 <openstack> Removing item from minutes: #info "Nodepool: Image build and upload workers" spec moved to implemented
19:13:01 <jeblair> so many specs
19:13:22 <fungi> #info "Nodepool: Use Zookeeper for Workers" spec moved to implemented
19:13:41 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html "Nodepool: Use Zookeeper for Workers" spec
19:13:46 <fungi> that's the one i meant ;)
19:14:16 <fungi> #topic Specs approval: PROPOSED Update zuulv3 spec to include job repo information (jeblair)
19:14:36 <fungi> #link https://review.openstack.org/410442 Update zuulv3 spec to include job repo information
19:14:55 <fungi> fairly small but important clarification/addition
19:15:02 <jeblair> i realized we'd been talking about how zuul preps all the repos for a job, but didn't actually write that in the spec.  :)
19:15:02 <fungi> already has several rollcall votes
19:15:12 <jeblair> hopefully this is not surprising
19:15:46 <fungi> anybody think it needs to incubate in review longer?
19:15:53 <fungi> seems likely already an unspoken consensus
19:16:34 <fungi> #info Infra Council voting is open on "Update zuulv3 spec to include job repo information" spec update until 19:00 UTC Thursday, January 26.
19:16:59 <fungi> any related questions before i move on?
19:18:10 * jhesketh adds it to his review list
19:18:16 <fungi> #topic Priority Efforts
19:18:33 <fungi> i don't see any updates in here today, though i did clear the
19:18:52 <fungi> "Nodepool: Use Zookeeper for Workers" priority effort from the agenda now that it's implemented
19:19:11 <fungi> #topic Ethercalc Deployment (clarkb)
19:19:31 <clarkb> hello, just wanted to get this in front of people as trying to get this up soonish to be used for ptg related activities
19:19:35 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/ethercalc.html Ethercalc spec
19:19:54 <clarkb> there are two changes I have up that could use review. The first updates the puppet module to run an ethercalc and the other is system-config to actually deploy it
19:20:04 <clarkb> #link https://review.openstack.org/#/c/422921/
19:20:12 <clarkb> #link https://review.openstack.org/#/c/424800/
19:20:28 <fungi> looks like those need their topic set to match the one from the spec
19:20:42 <clarkb> fungi: oh good point
19:20:52 <clarkb> I will do that
19:20:53 <fungi> #link https://review.openstack.org/#/q/status:open+topic:ethercalc
19:21:25 <fungi> (will actually display those changes once topical)
19:21:26 <clarkb> the other thing I wanted to ask about was how robust do we think we need to make ther service. Right now it uses a colocated redis server without authentication (relying on firewall to avoid outsdie connections) and doesn't have redis backups
19:21:57 <fungi> redis is taking the place of mysql in etherpadland?
19:22:03 <clarkb> I think the firewall is sufficient for now rather than password authing. And maybe we don't need backups if we are just testing this out for a single event?
19:22:07 <clarkb> fungi: correct
19:22:13 <fungi> i.e., storing all the spreadsheet data?
19:22:25 <fungi> backups seem useful
19:22:33 <jeblair> i'd like to have backups for that data
19:22:35 <clarkb> the two options are slow direct writes to disk (for testing) or redis for more rpduction like qualities. No mysql option
19:23:04 <clarkb> ok I will bump backups up the priority list as my next task there
19:23:34 <fungi> #link https://www.npmjs.com/package/redis-dump Dump redis database into redis commands or json with command line or node.js
19:24:15 <fungi> oh, wait, is redis implemented in javascript? or just something javascripty people use a lot?
19:24:30 <fungi> though i guess ethercalc is node.js at least
19:24:31 <clarkb> its not a js service just popular with that crowd
19:24:49 <clarkb> looks like redis already snapshots to disk and those snapshots are viable backups so I just need to set up bup
19:25:01 <fungi> oh, simple enough
19:25:01 <jeblair> oh that's cool
19:25:03 <clarkb> rather than needing an explicit db dump type action like you would have with a mysql server
19:25:14 <fungi> in that case
19:25:16 <fungi> #undo
19:25:17 <openstack> Removing item from minutes: #link https://www.npmjs.com/package/redis-dump
19:25:35 <jeblair> clarkb: there are rotating snapshots?
19:25:54 <jeblair> (so if bup runs while snapshot N is being written, N-1 is still untouched and viable (if old)?)
19:26:05 <clarkb> jeblair: yes I think so, but I will confirm
19:26:12 <jeblair> cool
19:26:17 <clarkb> it however doesn't keep N-30, so we'd rely on bup for that
19:26:44 <jeblair> yeah, the ideal thing for bup actually would just be N-1
19:26:58 <jeblair> (so it isn't diffing 30 different backups)
19:27:19 <jeblair> though something...something... git...packing...
19:27:55 <clarkb> thats about all I had, reviews requested and will work on making sure that backups are a thing
19:30:35 <fungi> okay, cool. i guess we don't have any other questions on ethercalc
19:30:47 <fungi> i think i've reviewed those puppet changes already, but will make sure
19:30:58 <fungi> #topic Open discussion
19:31:28 <fungi> as mentioned earlier, i'll be sequestered for most of next week so don't expect me to respond much on irc and mailing lists
19:31:37 <fungi> i'll need a volunteer to chair next week's infra meeting
19:31:51 <clarkb> as will I
19:32:55 <jlvillal> Just curious when a new release of gerritbot will happen.
19:33:38 <jlvillal> Or whatever action is needed for latest gerritbot to be active :)
19:34:11 <clarkb> I believe that is the action, if there is no objection to making a release I can probably do that today
19:34:44 <jlvillal> Cool :)
19:34:46 <fungi> jlvillal: looks like we could stand to tag a new release. looks like we're running 0.2.1.dev13 probably manually installed
19:35:12 <jlvillal> Thanks
19:35:47 <fungi> looks like the puppet manifest just does an ensure=>present right now so isn't auto-upgrading
19:35:54 <fungi> that likely ought to get revisited
19:36:19 <dims> many thanks to the infra and project-config teams, we have a gate-tempest-dsvm-py35-ubuntu-xenial-nv now that seems to be holding up
19:36:48 <fungi> neat!
19:37:13 <clarkb> #action clarkb make a gerritbot release
19:37:19 <ianw> i wouldn't mind another eye or two on https://review.openstack.org/424431 <- run wheel build in parallel.  currently that job is timing out
19:37:44 <clarkb> ianw: I have added to my list (will revie after lunch)
19:38:01 <ianw> also i'm not sure if anyone has an opinion on https://review.openstack.org/423904 <- add bindep virtualenv to wheel slaves
19:38:22 <ianw> i think the concern is that we're installing packages on a long-lived slave from requirements
19:40:27 <clarkb> ya as much as possible I think we want to try and use distro pacakges
19:40:34 <clarkb> which is what bindep does
19:40:52 <clarkb> so kind of a chicken and egg there
19:41:26 <fungi> yeah, the risk is that the requirements reviewers might be able to squeeze in a package that backdoors the server and then the mirrors
19:41:52 * ianw doesn't mention the d-word
19:42:26 <fungi> though that mirror building job already does plenty of other risky things by running a script from the requirements repo and installing packages from pypi into a virtualenv to build up the wheel cache, right?
19:43:02 <fungi> which is why we make those nodes separate from our other static nodes
19:43:05 <ianw> it doesn't run a script, just reads the requirements files
19:43:10 <fungi> well, one of te reasons anyway
19:43:28 <fungi> oh, i thought the job payload called a script in the reqs repo
19:44:13 <clarkb> no it just runs a pip wheel global-requirements --cache-dir /some/path iirc
19:44:16 <fungi> but on second look, those scripts are in project-config
19:44:18 <ianw> fungi:  i can see the sneak attack ... but it would have to *also* be a sneak attack into a distro package?  because we're fundamentally just "yum/apt-get install"
19:45:44 <fungi> anyway, the bigger risk already present is that pip install, even with a virtualenv, is still running arbitrary code from pypi packages (invoking setup.py)
19:46:38 <persia> Distro packages also run arbitrary code from distro packages (maintainer scripts), but more exploration of any of this probably requires publication, and so should be avoided in this forum :)
19:47:05 <fungi> basically a new release of anything listed in upper-constraints.txt could do just about anything (and probably more) than we're risking by using bindep
19:48:28 <fungi> and i do agree there is an unpleasantness to having to constantly maintain parity between the packages listed in the puppet module for deploying that node and the bindep file in the requirements repo
19:49:08 <ianw> pip is at least user-only, as opposed to pkg manager
19:49:31 <fungi> yeah, but the job user on that node has access to the only data of any value on the machine
19:49:47 <fungi> so denying it root access doesn't really buy us anything
19:50:08 <fungi> other than some assurance a job doesn't break the server
19:50:48 <fungi> it's just the first time (to my knowledge) we'll have a job running on a static node installing distro packages at job runtime
19:51:19 <fungi> so reviewers are justifiably concerned and trying to consider possible corner cases
19:52:53 <ianw> yep.  i've been trying to think how we could we do the building on a regular slave somehow
19:53:05 <ianw> which also seems better because it has up-to-date packages
19:53:33 <fungi> the credential handling in zuul v3 probably makes this easier to switch to single-use nodes
19:54:22 <fungi> so i'm leaning toward just bindep'ing in that job given the other safeguards we already have in place and teh existing known risks which are in my opinion at least as serious if not moreso
19:55:06 <ianw> and maybe a longer term goal of either moving it towards any work we do with docker containers or using new stuff to convert to single-use slaves
19:55:46 <fungi> my bigger concern is there exists some slippery slope here
19:56:16 <fungi> just because we're using bindep in the wheel mirror building jobs doesn't mean the same considerations hold for unrelated jobs running on other static nodes
19:56:39 <fungi> so need to make sure it doesn't get used as an unqualified precedent
19:57:18 <fungi> 3 minutes remaining before the next meeting. last call for volunteers to chair this meeting in my absence next week
19:57:33 <ianw> i can call that out in a comment in the puppet
19:57:48 <persia> That it builds a mirror is a useful distinction, because most of the security holes predume access to modify the mirror content anyway.
20:00:03 <fungi> and we're out of time. thanks everyone!
20:00:04 <fungi> #endmeeting