19:02:22 #startmeeting infra 19:02:23 Meeting started Tue Jan 24 19:02:22 2017 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:25 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:27 The meeting name has been set to 'infra' 19:02:29 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:35 #topic Announcements 19:02:45 i don't have any for this week 19:02:51 as always, feel free to hit me up with announcements you want included in future meetings 19:03:09 fungi: we are likely to be mostly afk next week 19:03:19 o/ 19:03:33 yep, i was going to make a call for substitute chairs during open discussion 19:04:07 as i will almost certainly be unable to run the meeting next tuesday due to quarterly employer-mandated in-person meeting time 19:04:18 #topic Actions from last meeting 19:04:31 #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-01-17-19.04.html 19:04:47 fungi Obtain docs.openstack.org X.509 certificate. 19:04:49 fungi Obtain developer.openstack.org X.509 certificate. 19:04:53 i've done both of those 19:05:14 great! 19:05:15 they're in the certs directory on puppetmaster.o.o and i'm working on the change to consume them from hiera now 19:05:30 while i was at it i updated the soon-expiring ask.o.o and zuul.o.o certs 19:06:00 pabelanger: i also got the requested cert for trystack.org, need to figure out how we want to do that redirect 19:06:27 right now the trystack domains are incorrectly configured as vhost aliases rather than permanent redirects 19:06:54 #action fungi announce the infra ptg pike etherpad to relevant mailing lists. 19:07:05 i plan to get to that later today after the tc meeting wraps up 19:07:13 jeblair mark http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html implemented 19:07:24 that was done and approved yesterday 19:07:25 i did a thing! 19:07:38 mordred make the temporary branch for the Nodepool v3 shim. 19:08:01 i see a feature/gearman-zk-shim branch in openstack-infra/nodepool 19:08:01 that thing has happened: feature/gearman-zk-shim 19:08:06 so that seems to be done, yes 19:08:37 #action fungi plumb https for docs.o.o and developer.o.o 19:08:53 that ought to be done in the next couple days depending on who reviews 19:09:20 #topic Specs approval 19:09:42 i have a few to report here... just a sec while i organize urls 19:10:09 #info Ethercalc spec approved 19:10:25 #link http://specs.openstack.org/openstack-infra/infra-specs/specs/ethercalc.html Ethercalc spec 19:11:05 #info ABANDONED "Nodepool launch and delete workers" spec 19:11:19 #link http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-launch-workers.html "Nodepool launch and delete workers" spec 19:11:50 fungi: great 19:12:01 #info "Nodepool: Image build and upload workers" spec moved to implemented 19:12:16 #link http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-workers.html "Nodepool: Image build and upload workers" spec 19:12:47 er, wrong one 19:12:49 #undo 19:12:49 Removing item from minutes: #link http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-workers.html 19:12:51 #undo 19:12:51 Removing item from minutes: #info "Nodepool: Image build and upload workers" spec moved to implemented 19:13:01 so many specs 19:13:22 #info "Nodepool: Use Zookeeper for Workers" spec moved to implemented 19:13:41 #link http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html "Nodepool: Use Zookeeper for Workers" spec 19:13:46 that's the one i meant ;) 19:14:16 #topic Specs approval: PROPOSED Update zuulv3 spec to include job repo information (jeblair) 19:14:36 #link https://review.openstack.org/410442 Update zuulv3 spec to include job repo information 19:14:55 fairly small but important clarification/addition 19:15:02 i realized we'd been talking about how zuul preps all the repos for a job, but didn't actually write that in the spec. :) 19:15:02 already has several rollcall votes 19:15:12 hopefully this is not surprising 19:15:46 anybody think it needs to incubate in review longer? 19:15:53 seems likely already an unspoken consensus 19:16:34 #info Infra Council voting is open on "Update zuulv3 spec to include job repo information" spec update until 19:00 UTC Thursday, January 26. 19:16:59 any related questions before i move on? 19:18:10 * jhesketh adds it to his review list 19:18:16 #topic Priority Efforts 19:18:33 i don't see any updates in here today, though i did clear the 19:18:52 "Nodepool: Use Zookeeper for Workers" priority effort from the agenda now that it's implemented 19:19:11 #topic Ethercalc Deployment (clarkb) 19:19:31 hello, just wanted to get this in front of people as trying to get this up soonish to be used for ptg related activities 19:19:35 #link http://specs.openstack.org/openstack-infra/infra-specs/specs/ethercalc.html Ethercalc spec 19:19:54 there are two changes I have up that could use review. The first updates the puppet module to run an ethercalc and the other is system-config to actually deploy it 19:20:04 #link https://review.openstack.org/#/c/422921/ 19:20:12 #link https://review.openstack.org/#/c/424800/ 19:20:28 looks like those need their topic set to match the one from the spec 19:20:42 fungi: oh good point 19:20:52 I will do that 19:20:53 #link https://review.openstack.org/#/q/status:open+topic:ethercalc 19:21:25 (will actually display those changes once topical) 19:21:26 the other thing I wanted to ask about was how robust do we think we need to make ther service. Right now it uses a colocated redis server without authentication (relying on firewall to avoid outsdie connections) and doesn't have redis backups 19:21:57 redis is taking the place of mysql in etherpadland? 19:22:03 I think the firewall is sufficient for now rather than password authing. And maybe we don't need backups if we are just testing this out for a single event? 19:22:07 fungi: correct 19:22:13 i.e., storing all the spreadsheet data? 19:22:25 backups seem useful 19:22:33 i'd like to have backups for that data 19:22:35 the two options are slow direct writes to disk (for testing) or redis for more rpduction like qualities. No mysql option 19:23:04 ok I will bump backups up the priority list as my next task there 19:23:34 #link https://www.npmjs.com/package/redis-dump Dump redis database into redis commands or json with command line or node.js 19:24:15 oh, wait, is redis implemented in javascript? or just something javascripty people use a lot? 19:24:30 though i guess ethercalc is node.js at least 19:24:31 its not a js service just popular with that crowd 19:24:49 looks like redis already snapshots to disk and those snapshots are viable backups so I just need to set up bup 19:25:01 oh, simple enough 19:25:01 oh that's cool 19:25:03 rather than needing an explicit db dump type action like you would have with a mysql server 19:25:14 in that case 19:25:16 #undo 19:25:17 Removing item from minutes: #link https://www.npmjs.com/package/redis-dump 19:25:35 clarkb: there are rotating snapshots? 19:25:54 (so if bup runs while snapshot N is being written, N-1 is still untouched and viable (if old)?) 19:26:05 jeblair: yes I think so, but I will confirm 19:26:12 cool 19:26:17 it however doesn't keep N-30, so we'd rely on bup for that 19:26:44 yeah, the ideal thing for bup actually would just be N-1 19:26:58 (so it isn't diffing 30 different backups) 19:27:19 though something...something... git...packing... 19:27:55 thats about all I had, reviews requested and will work on making sure that backups are a thing 19:30:35 okay, cool. i guess we don't have any other questions on ethercalc 19:30:47 i think i've reviewed those puppet changes already, but will make sure 19:30:58 #topic Open discussion 19:31:28 as mentioned earlier, i'll be sequestered for most of next week so don't expect me to respond much on irc and mailing lists 19:31:37 i'll need a volunteer to chair next week's infra meeting 19:31:51 as will I 19:32:55 Just curious when a new release of gerritbot will happen. 19:33:38 Or whatever action is needed for latest gerritbot to be active :) 19:34:11 I believe that is the action, if there is no objection to making a release I can probably do that today 19:34:44 Cool :) 19:34:46 jlvillal: looks like we could stand to tag a new release. looks like we're running 0.2.1.dev13 probably manually installed 19:35:12 Thanks 19:35:47 looks like the puppet manifest just does an ensure=>present right now so isn't auto-upgrading 19:35:54 that likely ought to get revisited 19:36:19 many thanks to the infra and project-config teams, we have a gate-tempest-dsvm-py35-ubuntu-xenial-nv now that seems to be holding up 19:36:48 neat! 19:37:13 #action clarkb make a gerritbot release 19:37:19 i wouldn't mind another eye or two on https://review.openstack.org/424431 <- run wheel build in parallel. currently that job is timing out 19:37:44 ianw: I have added to my list (will revie after lunch) 19:38:01 also i'm not sure if anyone has an opinion on https://review.openstack.org/423904 <- add bindep virtualenv to wheel slaves 19:38:22 i think the concern is that we're installing packages on a long-lived slave from requirements 19:40:27 ya as much as possible I think we want to try and use distro pacakges 19:40:34 which is what bindep does 19:40:52 so kind of a chicken and egg there 19:41:26 yeah, the risk is that the requirements reviewers might be able to squeeze in a package that backdoors the server and then the mirrors 19:41:52 * ianw doesn't mention the d-word 19:42:26 though that mirror building job already does plenty of other risky things by running a script from the requirements repo and installing packages from pypi into a virtualenv to build up the wheel cache, right? 19:43:02 which is why we make those nodes separate from our other static nodes 19:43:05 it doesn't run a script, just reads the requirements files 19:43:10 well, one of te reasons anyway 19:43:28 oh, i thought the job payload called a script in the reqs repo 19:44:13 no it just runs a pip wheel global-requirements --cache-dir /some/path iirc 19:44:16 but on second look, those scripts are in project-config 19:44:18 fungi: i can see the sneak attack ... but it would have to *also* be a sneak attack into a distro package? because we're fundamentally just "yum/apt-get install" 19:45:44 anyway, the bigger risk already present is that pip install, even with a virtualenv, is still running arbitrary code from pypi packages (invoking setup.py) 19:46:38 Distro packages also run arbitrary code from distro packages (maintainer scripts), but more exploration of any of this probably requires publication, and so should be avoided in this forum :) 19:47:05 basically a new release of anything listed in upper-constraints.txt could do just about anything (and probably more) than we're risking by using bindep 19:48:28 and i do agree there is an unpleasantness to having to constantly maintain parity between the packages listed in the puppet module for deploying that node and the bindep file in the requirements repo 19:49:08 pip is at least user-only, as opposed to pkg manager 19:49:31 yeah, but the job user on that node has access to the only data of any value on the machine 19:49:47 so denying it root access doesn't really buy us anything 19:50:08 other than some assurance a job doesn't break the server 19:50:48 it's just the first time (to my knowledge) we'll have a job running on a static node installing distro packages at job runtime 19:51:19 so reviewers are justifiably concerned and trying to consider possible corner cases 19:52:53 yep. i've been trying to think how we could we do the building on a regular slave somehow 19:53:05 which also seems better because it has up-to-date packages 19:53:33 the credential handling in zuul v3 probably makes this easier to switch to single-use nodes 19:54:22 so i'm leaning toward just bindep'ing in that job given the other safeguards we already have in place and teh existing known risks which are in my opinion at least as serious if not moreso 19:55:06 and maybe a longer term goal of either moving it towards any work we do with docker containers or using new stuff to convert to single-use slaves 19:55:46 my bigger concern is there exists some slippery slope here 19:56:16 just because we're using bindep in the wheel mirror building jobs doesn't mean the same considerations hold for unrelated jobs running on other static nodes 19:56:39 so need to make sure it doesn't get used as an unqualified precedent 19:57:18 3 minutes remaining before the next meeting. last call for volunteers to chair this meeting in my absence next week 19:57:33 i can call that out in a comment in the puppet 19:57:48 That it builds a mirror is a useful distinction, because most of the security holes predume access to modify the mirror content anyway. 20:00:03 and we're out of time. thanks everyone! 20:00:04 #endmeeting