19:01:46 <clarkb> #startmeeting infra
19:01:46 <openstack> Meeting started Tue Aug 14 19:01:46 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:47 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:49 <openstack> The meeting name has been set to 'infra'
19:02:04 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:15 <clarkb> #topic Announcements
19:02:30 <clarkb> I'm going to be on vacation next week.
19:02:52 <clarkb> fungi: we don't have a stein release key yet do we?
19:03:09 <fungi> nope, but i'm happy to generate one unless someone else wants to take a try at it
19:03:24 <fungi> the process documentation should be very easy to follow
19:03:41 <fungi> and i'm happy to provide support too
19:03:44 <clarkb> fungi: when have you tried to have those generated by relative to the cycle?
19:04:13 <fungi> ideally before release day, but any time in the prior cycle is fine
19:04:24 <ianw> does it happen on puppetmaster or bridge now?
19:04:29 <fungi> as long as the expiration date is set for a sufficient duration
19:04:41 <fungi> it should be doable on bridge
19:04:59 <clarkb> https://releases.openstack.org/rocky/schedule.html says Ideally it would be done next week
19:05:08 <clarkb> or between now and then
19:05:12 <fungi> really it just needs to happen on a secure system to which we have access so we can all double-check the fingerprint of the symmetrically-encrypted master key on disk
19:05:34 <clarkb> anyway it occurred to me that we normally announce the need to go sign the key in the announcement portion of this meeting and realized we hadn't done that here so brought it up
19:05:58 <fungi> makes sense--thanks for the reminder!
19:06:11 <fungi> we also put it into the release schedule so we get a reminder from the release team
19:06:33 * diablo_rojo sneaks in a little late
19:06:38 <clarkb> fungi: ah ok so they should reach out as well? I probably won't commit to it as I'll likely be afk when they want it
19:06:58 <fungi> looks like diablo_rojo just volunteered! ;)
19:07:18 <clarkb> Anything else to announce?
19:07:21 <fungi> (kidding, it needs to be an infra-root volunteer specifically)
19:07:25 <diablo_rojo> Umm..uh..
19:07:28 <diablo_rojo> whew
19:07:48 <fungi> but yeah, i'll plan to do it later this week. if anyone else wants to instead, let me know
19:08:18 <mordred> clarkb: I'm also going to be out most ofnext week
19:08:42 <mordred> clarkb: I leave wednesday for vietnam - so will be traveling for $hours - then the week and a half after that I'll be on vacation
19:09:12 <clarkb> fun! I'm headed to the oregon coast with family while parents are here. Going to try our luck at crabbing
19:09:13 <corvus> sounds like we may be short staffed enough that we should all just take a break :)
19:09:58 <mordred> clarkb: I'll wave across the pacific
19:09:59 <clarkb> #topic Actions from last meeting
19:10:12 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-08-07-19.01.txt Minutes from last meeting
19:10:24 <clarkb> I don't see any actions
19:10:37 <clarkb> #topic Specs approval
19:11:04 <clarkb> ianw: Should we be looking at the letsencrypt spec at this point? You mentioned in IRC earlier that you didn't feel like you had other things to change?
19:11:14 <clarkb> maybe reviews this week and look at putting up for approvla next week?
19:11:48 <ianw> yes please, i've incorporated all the current review comments at this point, so ready for review
19:12:03 <clarkb> #link https://review.openstack.org/#/c/587283/ Letsencrypt spec
19:12:24 <clarkb> Please review and we can try to put it up for approval next week
19:12:31 <fungi> i'm about 2/3 through reading the current version (i think). a slew of minor comments pending in gertty for it, but am generally in favor so far
19:12:40 <clarkb> fungi: thanks
19:12:56 <clarkb> #topic Priority Efforts
19:13:03 <clarkb> #topic Storyboard
19:14:14 <clarkb> Haven't seen anything major here, but projects continue to plan migrations
19:14:19 <fungi> diablo_rojo: any updates?
19:14:25 <diablo_rojo> I just saw on the ML puppet is interested
19:14:35 <fungi> yup, that was encouraging
19:14:39 <diablo_rojo> so I will help the new PTL and the project through stuff.
19:14:44 <fungi> looking to migrate next cycle sounds like
19:14:48 <diablo_rojo> We have a lot of open reviews that need some love.
19:14:49 <diablo_rojo> https://review.openstack.org/#/q/project:openstack-infra/storyboard-webclient+status:open
19:15:04 <diablo_rojo> fungi, next cycle or just whenever they feel comfortable
19:15:26 <fungi> the puppet-openstack ptl appears to have taken the initiative to deploy a copy of storyboard locally and perform an import himself even
19:15:38 <clarkb> #link https://review.openstack.org/#/q/project:openstack-infra/storyboard-webclient+status:open Storyboard reviews
19:15:40 <diablo_rojo> Yeah that was awesome.
19:16:02 <diablo_rojo> Thanks clarkb :) I can never remember to do that unless I am leading the meeting.
19:16:22 * fungi notes diablo_rojo has volunteered to chair future infra team meetings ;)
19:16:32 <clarkb> We do need a volunteer for next week :P
19:16:32 <diablo_rojo> *sigh*
19:17:11 * diablo_rojo averts eyes like she did in highschool to avoid having to answer a question she didn't know the answer to
19:17:23 <fungi> seems like that's it for storyboard updates this week
19:17:32 <clarkb> #topic Config Managment Updates
19:17:32 <diablo_rojo> Just pleading for reviews
19:17:52 <cmurphy> o/
19:17:58 <clarkb> Things are moving here, mordred has bridge.o.o up and running and cmurphy's parade of puppet 4 changes are moving along
19:18:01 <mordred> I thought we were just about ready to do the puppetmaster to bridge cutover ... but then realized there were a couple of special-cases with exim config, and the base playbook has exim stuff in it
19:18:16 <clarkb> topic:puppet-4 and topic:update-cfg-mgmt
19:18:36 <mordred> so hopefully we'll get the special cases handled - once we do - I'll want to do the cutover - at which point we should be able to just do small patches replacing a puppet bit with an ansible bit
19:18:49 <corvus> mordred: i guess we should all strive to help you achieve that before you go on vacation? :)
19:18:51 <cmurphy> mostly just have new tests and then some handpicked nodes to turn the future parser on
19:19:04 <mordred> corvus: yes. it should be TOTALLY achievable this week
19:19:09 <cmurphy> also this xenial fix for askbot https://review.openstack.org/585196
19:19:11 <corvus> (i'd love to avoid having the puppetmaster split across a long period).  cool.
19:19:24 <mordred> it's honestly just those exim patches and making sure we like them
19:19:51 <mordred> corvus: so maybe let's sync up on those after the meeting?
19:19:52 <clarkb> and for those following along the new server will continue to run puppet against most things, we just don't need puppet on the control node to do that
19:20:04 <clarkb> then we'll migrate puppetized things one or a handful at a time
19:20:21 <clarkb> (while concurrently making them use puppet that isn't ancient (because we expect the off puppet migration to take time)
19:20:30 <mordred> yes
19:20:31 <ianw> cmurphy: looks like that falls back to being stuck on https://review.openstack.org/#/c/559178/ (minor fork of puppet-solr?)
19:20:49 <mordred> additionally, this initial migration also handles a chunk of the things in openstack_project::server in ansible
19:21:00 <mordred> so users, initial packages, etc - are all migrated into ansible already
19:21:27 <mordred> so each "run" of puppet will run the base.yaml playbook first, then will run remote_puppet*
19:21:31 <ianw> mordred: have you run it against bridge.o.o itself?  i noticed my key was wrong there
19:21:47 <cmurphy> ianw: i think askbot in general is stuck on that patch but the patch i linked shouldn't be stuck on anything
19:21:56 <mordred> ianw: yes - I think we should have landed the patch to fix your key - but I haven't re-run it since that landed
19:22:56 <ianw> cmurphy: i think it's on top of the current outstanding stack (585196)?
19:22:57 <clarkb> Mostly just need reviews and coordinating hte larger action of deleting puppetmaster.o.o then?
19:23:37 <cmurphy> ianw: oh yes you're right
19:23:53 <cmurphy> i was looking at owner:self derp
19:23:55 <mordred> clarkb: yah
19:24:29 <ianw> mordred: so does puppet currently redo some of the stuff that the new base.yaml does?
19:24:49 <mordred> ianw: it would if we ran the stack as it is landed right now
19:24:56 <mordred> ianw: by the end of the current proposed stack, no
19:25:21 <mordred> ianw: although - with the exception of exim config on 4 hosts - the puppet and ansible are currently producing the same results
19:25:32 <mordred> so the puppet run is a no-op on the things that are the same
19:25:41 <mordred> I did run the base.yaml playbook against review-dev.openstack.org
19:25:57 <mordred> ianw: your ssh key should be fixed now
19:26:23 <ianw> thanks :)
19:26:33 <mordred> one other thing, just so people are aware - bridge is running ansible 2.6 using python3 - and is also using the new openstack inventory _plugin_ instead of the old openstack inventory script
19:26:43 <mordred> plus some other new fancy ansible inventory plugins
19:26:55 <mordred> so if you haven't seen that stuff yet, it's worth looking at it - hopefully it's understandable
19:26:56 <clarkb> mordred: any concern with doing that and wanting to use "random" roles for services?
19:27:14 <ianw> mordred: maybe just so it's in the minutes, and i think related -- what's the thinking on moving kerberos/afs roles into system-config for -> https://review.openstack.org/#/c/590591/
19:27:24 <ianw> (and i presume similar roles)
19:27:24 <clarkb> we add python3 support to them I guess if that becoes a problem
19:27:31 <mordred> clarkb: yah
19:27:38 <mordred> clarkb: we're gonna have to add it at some point :)
19:27:42 <mordred> http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/roles/install-ansible/files/groups.yaml
19:27:58 <mordred> is the replacement for how we add hosts into ansible groups from the hacked-together groups.txt thing
19:28:50 <mordred> ianw: I think we should move them into system-config - I also think roles wants to move up into the root of system-config - but we need to do a dance with role paths for that, and I'm worried about the puppetmaster/bridge split and complexity
19:29:33 <mordred> so I think we should shift roles after we cutover - and add your afs/krb roles as the first thing
19:30:12 <ianw> yeah, i'm kind of missing the why they're better there bit :)
19:30:33 <clarkb> ianw: rather than being in their own repos you mean?
19:30:54 <ianw> or zuul-jobs?  won't that get dragged in?
19:31:10 <mordred> no - we're not running these in zuul - we run ansible on bridge in its own context
19:31:28 <clarkb> mordred: well we also use them in zuul jobs (or could use them) for wheel build jobs to set up afs
19:31:32 <mordred> so if we used them from zuul-jobs we'd wind up adding zuul-jobs roles path to our production role path on system-config - which seems backwards to me
19:31:44 <mordred> but yes - if we put them in system-config, we can add system-config to the roles path for the jobs
19:32:01 <mordred> so that way it's "use the production roles in testing" rather than "use the testing roles in production"
19:32:28 <clarkb> mordred: ya I like that also makes it clear if you change this then productio nchanges
19:32:32 <mordred> yah
19:32:34 <clarkb> which probably won't be so clear if in zuul-jobs
19:33:09 <mordred> eventually having an ansible-role-openafs-client repo would probably be aces - but for now keeping them in system-config while we sort it out seems like less churn
19:33:42 <clarkb> fwiw I kind of like the aggregate roles situation in eg zuul-jobs or ozj better than puppet-.* everything we ended up with
19:33:46 <ianw> ok > zuul-jobs roles path to our production role path on system-config
19:33:59 <ianw> i guess that was my misunderstanding, i thought that would be fine
19:34:17 <ianw> maybe even kind of the idea of zuul-jobs as generic collection of things
19:34:44 <corvus> well, i see it as a collection of jobs that are generally useful
19:35:20 <clarkb> so maybe an infra roles stdlib repo would make sense rather than a repo per role (but not sure how happy ansible is with that construct generally)
19:35:34 <corvus> i don't know how generally applicable our afs mirror wheel building jobs are.  maybe?
19:35:55 <ianw> corvus: this is just generic "setup as a kerberos client" and "install afs client" roles
19:35:58 <clarkb> and I think system-config is an easy place to start so no objections to use it
19:36:04 <mordred> clarkb: ansible is growing the ability for grok a repo of roles - but its not there yet
19:36:25 <mordred> there is a new tool called "mazer" that is being developed to do fancier things with role management
19:36:44 <corvus> ianw: right, but with no job to use them defined there, it's questionable whether that's the right place.
19:36:46 <mordred> so if we did it, it would be fine, but it would involve us putting the git repo on disk and adding it to roles path
19:37:11 <corvus> (we have *some* roles in zuul-jobs without jobs using them, but generally those are ones which are designed to be used in a config repo)
19:37:24 <mordred> clarkb: which, incidentally, is what https://review.openstack.org/#/c/590752/3 does
19:38:16 <clarkb> anything else before we continue on our way down the agenda?
19:38:25 <ianw> ok, in terms of testing ... i have https://review.openstack.org/#/c/589335/ ... i don't think that pulls in system-config roles, i guess that will be ok to do in o-z-j?
19:38:48 <mordred> ianw: yes, I believe so
19:39:14 <ianw> ok, sorry, if i have more questions i'll follow up, but thanks
19:39:40 <mordred> ianw: awesome - and sorry, I keep meaning to try to overlap days with you more to chat about it
19:40:37 <clarkb> #topic General Topics
19:41:13 <clarkb> Really quickly. its been pointed out that we need to continue upgrading off of trusty because it has abou 8 months of life left
19:41:47 <clarkb> I'm trying to push on ethercalc and etherpad as they had a bit of work done around their puppet to support it. I think we are to the point where a new ethercalc can be deployed so will do that today after the meeting
19:41:57 <clarkb> #link https://review.openstack.org/#/c/591017/ Support for digitized etherpad servers
19:42:12 <clarkb> that change is necesary for etherpad. mordred I think it runs into group change stuff that we need to coordinate on
19:42:24 <clarkb> mordred: ^ maybe we wait for bridge cutover before booting new etherpad* nodes
19:42:33 <mordred> yah - that'll probably be easiest
19:42:42 <clarkb> ok I'll WIP it then
19:42:54 <mordred> that way we won't be trying to juggle both boxes - but we'll also push hard on the cutover
19:43:25 <clarkb> There are a number of services that need to be updated to newer distro. I think next month is a great time to work on that as the release will be behind us
19:43:44 <clarkb> When I get back from vacation I'll try to put together some planning for a sprint as I think those have worked well in the past
19:43:54 <clarkb> (likely after ptg given that we'll be juggling that soon enough)
19:44:40 <clarkb> ianw: Want to talk about third party ci spec?
19:44:47 <clarkb> #link https://review.openstack.org/563849 Direction setting for 3rd Party CI
19:44:54 <ianw> ++ it's much easier if we collaborate, as you get stuck with a bunch of puppet patches on repos that nobody really watches
19:45:04 <clarkb> ianw: indeed
19:45:05 <ianw> and then get distracted :)  at least i do :)
19:45:34 <ianw> re 3rd party; just a quick note -- prior reviews out of all the options windmill was pulling ahead
19:45:57 <ianw> i think as stage 2, we should really define what we mean by 3rd party ci -- i.e. what you start with, and where you get to
19:46:06 <ianw> so i've written that out in the spec
19:46:16 <ianw> so i'd like reviews on that
19:46:47 <ianw> i think, once we have those two bits, we have a complete thing we can start actually working on, with criteria for it being "done"
19:46:54 <mordred> ++
19:47:22 <ianw> that's all :)
19:48:30 <clarkb> #topic Open Discussion
19:48:47 <clarkb> It occurs to me I didn't explicitly ask for a volunteer to run for the meeting next weke but we will need one
19:49:08 <clarkb> Let me knowif interested
19:49:27 <pabelanger> I'd like to push on our base job rework if possible, http://lists.openstack.org/pipermail/openstack-infra/2018-August/006032.html for more info
19:49:39 <pabelanger> I likey need to rebase conflicts
19:50:50 <clarkb> pabelanger: why is base-ozj in project-config?
19:51:17 <ianw> pabelanger: i did look, some of the reviews i got a bit scared because they were moving big bits.  a bit more description will help in the changes i think
19:51:20 <clarkb> might just be me but that seems confusing organizationally
19:51:25 <clarkb> ianw: ++
19:51:39 <pabelanger> okay, will update
19:51:48 <pabelanger> but base-ozj is needed for integeation testing of zuul-jobs
19:52:10 <pabelanger> which is currently base-minimal
19:52:13 <clarkb> pabelanger: mostly its weird that base-ozj isn't in ozj
19:52:25 <pabelanger> clarkb: open to suggestions for naming
19:52:36 <pabelanger> jobs in ozj will use it
19:53:38 <fungi> mmm, so when i said i was in favor of the le spec so far, i hadn't gotten to the actual implementation proposal yet. worried it's going to be overengineered, fragile and less secure than the current situation we have (no offense, ianw, i really thought the earlier discussions about "proxying" acme renewals and having a site proxy to bootstrap new systems was meant to be parody)
19:54:12 <clarkb> pabelanger: can followup after the meeting
19:54:30 <clarkb> pabelanger: but I think it would be helpful if we could have ascii art or similar of where we are today and where we want to be
19:54:48 <clarkb> pabelanger: at least for me understanding these organizational changes are aided by diagrams
19:55:19 <pabelanger> okay
19:57:26 <clarkb> I'll let the last 3 minutes run out if there is anything else, but that is all I had
19:57:31 <clarkb> thank you everyone
20:00:14 <clarkb> #endmeeting