19:01:13 <clarkb> #startmeeting infra
19:01:14 <openstack> Meeting started Tue May  8 19:01:13 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:17 <openstack> The meeting name has been set to 'infra'
19:01:18 <mordred> o/
19:01:22 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:01:35 <clarkb> #topic Announcements
19:01:48 <clarkb> Summit/OpenDev is in just under two weeks
19:02:08 <clarkb> If you are attending don't forget to do whatever prep you need :)
19:02:14 * corvus frets
19:02:35 * fungi is so behind
19:02:42 <clarkb> we need a support group
19:03:15 <clarkb> I was silly and decided to take a mini vacation next week for my birthday. Which further cuts into my available time (but I'm excited to go fishing and sit on the (cold) oregon beach)
19:03:28 <fungi> sounds marvellous
19:03:29 <clarkb> Due to ^ we'll need an infra meeting chair volunteer
19:03:29 <anteaya> happy birthday
19:03:51 <clarkb> and I figure we'll cancel the May 22 meeting as many of us will be at the summit
19:04:00 <clarkb> Let me know if interested in chairing on the 15th
19:04:17 <cmurphy> o/
19:04:34 * fungi hopes that's cmurphy volunteering to chair next week's meeting ;)
19:04:41 <cmurphy> lolno
19:04:43 <clarkb> fungi: I totally read it that way too :P
19:04:43 <mordred> that's how I read it
19:04:49 * cmurphy runs
19:05:04 <clarkb> anything else need to be announced?
19:05:14 <fungi> i don't seem to have any conflicts for the 15th so _can_ chair, but totally wouldn't want to take that pleasure away from anyone who wants to volunteer. i've chaired plenty of these already
19:05:33 <anteaya> I think cmurphy should take a turn
19:05:54 <clarkb> I think cmurphy might also be semi afk too so won't push too hard on that :)
19:06:01 <anteaya> ah okay
19:06:06 <anteaya> another time then
19:06:11 <cmurphy> yeah i am in and out for the next couple weeks
19:06:25 <clarkb> #topic Actions from last meeting
19:06:32 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-05-01-19.01.txt minutes from last meeting
19:06:57 <clarkb> There wasn't a specific #action recorded but more of a collective please go review our three config management future specs. Seems like there were comments on the whole set so I think people were reading them
19:07:13 <clarkb> Before we dig into that subject
19:07:16 <clarkb> #topic Specs approval
19:07:23 <clarkb> #link https://review.openstack.org/349831 Survey tool spec
19:07:24 <patchbot> patch 349831 - openstack-infra/infra-specs - Add survey spec
19:07:31 <pabelanger> o/
19:07:51 <clarkb> anteaya: has indicated that ^ should be ready for review. tl;dr is lets run a limesurvey to host the surveys that various groups want to run around openstack
19:08:06 <clarkb> fungi: I think you've been helping out there are you happy for that to be up for approval this week as well?
19:08:39 <fungi> yeah, i think i've already registered a rollcall +1 on it
19:08:44 <clarkb> yup I see that now
19:08:55 <anteaya> and yes he has been helping out a lot
19:08:59 <fungi> the biggest unknown was the self-service signup to create surveys
19:09:04 <anteaya> for which I'm very grateful
19:09:25 <clarkb> if the infra council voters could look over that spec this week it would be appreciated. I'll make a not to look at it probably late on thursday as far as approving it goes
19:09:33 <clarkb> *make a note
19:09:40 <fungi> but digging into configuration now that we have a demo/prototype up, it looks like it should be able to handle webserver auth so we have a good chance of getting apache mod_auth_openid working
19:10:06 <clarkb> cool
19:10:13 <fungi> i'm hoping to give that a shot tomorrow now that i've found all the commented configuration defaults for limesurvey
19:10:24 <anteaya> thank you
19:10:30 <fungi> also it does support pluggable authentication, so we could roll our own if someone gets ambitious
19:10:36 * anteaya ladles on more gratitude
19:11:04 <mordred> anteaya: I read that as gravy
19:11:14 <anteaya> mordred, when did you last eat?
19:11:23 <anteaya> might you be hungry?
19:11:37 <fungi> yeah, now i want to go find food after this meeting
19:11:41 <mordred> anteaya: about 15 minutes ago ... but gravy is always good
19:11:42 <anteaya> sorry
19:11:48 <fungi> good gravy
19:11:50 <clarkb> #topic Priority efforts
19:11:58 * clarkb takes gravy as the cue we are ready for next topic :)
19:12:11 <anteaya> clarkb, good pickup
19:12:11 <fungi> all topics are now gravy
19:12:17 <clarkb> we'll start with storyboard. Looks like migrations continue. Heat is done and loci is in the queue?
19:12:27 <fungi> yeah, heat was a massive import
19:12:41 <fungi> >5k bugs
19:12:46 <fungi> er, >4k bugs
19:12:49 <fungi> still a lot
19:13:05 <fungi> took in the neighborhood of 12-14 hours to import, i think
19:13:22 <mordred> wow
19:13:23 <fungi> loci should be quick by comparison
19:13:39 <diablo_rojo> Poked at the next tripleo squad again too- trying to get them on the schedule of migrations
19:13:40 <anteaya> many things will be quick by comparison
19:14:05 <anteaya> my flight to vancouver will be quick by comparison
19:14:21 <diablo_rojo> mordred, nothing compared to what neutron and nova will be (I tried to do a test of the neutron one and it ran for a week before I stopped it)
19:14:54 <fungi> the model there is sort of weird though. i'm not entirely convinced it will make sense for them to import all tripleo bugs into tripleo-ci but start by only importing the ones tagged as security and then import the others into the same project later
19:15:14 <fungi> er, into tripleo-common
19:15:17 <fungi> not tripleo-ci
19:15:20 <diablo_rojo> fungi, agreed
19:15:28 <diablo_rojo> I think making a repo for them would make more sense
19:15:54 <fungi> any bugs which are already in lp and then after the import they add the security tag... do those need to get closed and reopened in storyboard? no idea
19:16:41 <diablo_rojo> Its definitely messy.
19:16:46 <fungi> the tripleo leadership have expressed concerns that having storyboard tasks tied to repositories isn't compatible with their workflow and they'd rather just have all tasks be for their team
19:17:13 <diablo_rojo> Then they should all go at once rather than squad by squad
19:17:26 <diablo_rojo> probably should at least
19:17:34 <clarkb> fungi: that is how they operate in lp I guess?
19:18:11 <fungi> yeah, there is a "tripleo" project in lp and then they add bug tags to indicate which subteams are working on what bugs, seems like
19:18:52 <clarkb> gotcha
19:19:17 <fungi> so anyway, the closest we have under our current model (there is no "tripleo" repo) is to migrate their bugs to have tasks for the tripleo-common repo
19:19:42 <fungi> but as diablo_rojo notes, their desire to migrate subteam at a time makes this extra weird
19:20:27 <mordred> that seems weird to me as well - and something that seems pretty rife for confusion and stress while it's happening
19:20:57 <clarkb> is that something we should try to have a more direct conversation with the tripleo team about?
19:21:05 <clarkb> (thinking with summit coming up that may be an opportunity there)
19:21:45 <fungi> while that would probably be useful, i think i'm booked solid by this point
19:22:01 <diablo_rojo> We can add it to the agenda for the storyboard forum session monday
19:22:01 <clarkb> ya I feel that way too
19:22:02 <fungi> unless we can somehow have that conversation while i'm asleep
19:22:04 <diablo_rojo> if they can make it
19:22:16 <clarkb> fungi: or corner mwhahaha and EmilienM_PTO in a bar on tuesday
19:22:17 <fungi> i predict i'll be having many conversations while asleep by the last day
19:22:55 * mwhahaha disagrees with the confusion part but whatever
19:23:04 <mordred> fungi: we're allowed to sleep at the summit?
19:23:08 <clarkb> mordred: you aren't
19:23:23 <mwhahaha> i won't be at summit, but i'd be happy to have a chat about it sooner, as we need to figure out the migration to storyboard anyway
19:23:26 <diablo_rojo> mwhahaha, would you be able to make it to the Storyboard session Monday?
19:23:29 <diablo_rojo> Dang
19:23:31 <diablo_rojo> nvm
19:23:59 <mwhahaha> tl;dr, we probably should just mass cutover sooner but i need to sent a note to th eml this week
19:24:07 <mwhahaha> this subteam move probably isn't working out
19:24:48 <clarkb> cool sounds like we should be able to sort something out then
19:25:02 <clarkb> (both parties are motivated etc)
19:25:51 <diablo_rojo> mwhahaha, it worked for a few of the squads, but yeah its probably time if we can swing it with your conributors
19:26:29 <mwhahaha> it's not really working cause it's confusing everyone on where to create and there's no fix except duplicating work
19:27:27 <fungi> that's the sort of confusion i was concerned partial migration would cause
19:29:07 <clarkb> alright are we ready to move on to the next topic?
19:29:30 <clarkb> #topic Modern Config Management
19:29:35 <clarkb> #link https://review.openstack.org/449933 Puppet 4 Infra
19:29:40 <clarkb> #link https://review.openstack.org/469983 Ansible Infra
19:29:45 <clarkb> #link https://review.openstack.org/565550 Containerized infra
19:30:04 <clarkb> Now that we've had a chance to read these through and consider them, what are our thoughts/impressiosn?
19:30:14 <mordred> pabelanger pointed out that ansible and container potentially share a common base
19:30:51 <pabelanger> +1
19:31:06 <clarkb> that is the setup email and users and install docker portion right?
19:31:08 <mordred> so if we like either of the two of them, it might be worth splitting out an ansible-base spec to describe that portion
19:31:11 <mordred> clarkb: yah
19:31:12 <corvus> yeah, and to some extent, we may end up with some hosts where we have some ansible and no containers (eg, afs servers).  it may really be a matter of degree?
19:31:18 <mordred> and how to deal with hiera/secrets
19:31:24 <mordred> corvus: ++
19:31:59 <pabelanger> I am however, liking puppet-4 (I guess 5 because of bionic) as it might be less work
19:32:26 <corvus> i'm leaning toward the container approach, mostly because it completely decouples us from operating system dependencies
19:32:35 <clarkb> my initial impression was that puppet 4 and then 5 feel like less upfront work. Like we can probably get to 4 in a week or two. But then persistent issues like bionic not having puppet 5 packages and the weird renaming of silly things and distros undoing that seems like pain I'd like to avoid
19:32:40 <mordred> I would prefer either ansible or container specs (or some combo of the two) - but obvs sinceI wrote it I prefer the container one
19:32:51 <cmurphy> i'm trying to verify whether puppet 4 will be supported on bionic, i think it will be
19:33:19 <mordred> corvus: ++ - that's my main motivation from it - that and it shifts some burden to the ci system
19:33:26 <mordred> which, it turns out, we hae a good one of
19:33:52 <clarkb> and ya I think containerizing applications gives us some new features that will be useful over time that both puppet and ansible largely lack as we'd use them otherwise
19:34:35 <pabelanger> If I replace containers with the word packaging, I think I'm okay with them :)  mordred did a good job outlining some use cases I was having trouble seeing
19:34:41 <clarkb> (whereas puppet and ansible are roughly equivalent in "power"/utility, puppet even has thair own remote ssh tool now (its literally just ansible written in ruby for puppet))
19:34:43 <anteaya> I'd just like to see whatever the decision is well publicized so that in 2 months when someone says, hey you should all do [thing] someone can point to a url
19:34:47 <fungi> it's still not clear to me that ditching puppet for $next_thing necessarily fixes the need to periodically update our configuration nor the odds of backward-incompatible changes to file paths over time
19:34:53 <mordred> pabelanger: \o/ (and yea, that's how I've been thinking of them)
19:34:54 <corvus> with puppet, we have to have a working version of puppet on every host, and if that version is different (it often is) things get complicated.  with ansible, we only need a working version of ansible on one host (the controller).  and with containers, because so many of our services are things we run from source, we can drop a lot of the work we're doing to deploy those via a config management
19:34:54 <corvus> system.
19:35:25 <fungi> okay, that i find compelling
19:35:32 <clarkb> corvus: aiui you can do that with puppet now, though that might require first getting to puppet 5?
19:35:45 <fungi> by
19:35:51 <clarkb> corvus: they literally implemented ansible remote execution in ruby for puppet resources
19:36:00 <fungi> er, by "that" i mean the ability to not need to install a client/agent on all our various systems
19:36:13 <fungi> so does actually solve the consistency issue
19:36:15 <corvus> clarkb: it's a good model
19:36:29 <clarkb> corvus: yup, I think ansible sort of proved that point to puppet :)
19:36:36 <ianw> as i mentioned in my comments, just because the containers are there, still most of the configuration is/can be in puppet?  i mean it's not a thrilling idea to translate huge blobs of erb config templates into jinja just because
19:36:37 <corvus> the containers idea further minimizes the ansible we will end up with.
19:37:05 <mordred> ianw: the containers plan ends up with puppet going completely away
19:37:13 <mordred> ianw: and using ansible to template out config files
19:37:42 <ianw> ends up with ... but it's not like we need to have a flag day switch?
19:37:44 <pabelanger> yah, as I understand it, we'd write the container as Dockerfile, then use ansible to template and bind-mount a config into it
19:37:45 <mordred> (I agre that translating erb to jinja sounds like unfun)
19:37:47 <fungi> and to make determinations on what package renames happen between distro versions?
19:37:51 <mordred> ianw: right. we don' tneed to flag day
19:38:03 <corvus> mordred, ianw: one server at a time, i'd reckon
19:38:07 <mordred> ++
19:38:12 <clarkb> ya I think any transition to using more ansible would invole puppet_run_all running less "all"
19:38:21 <clarkb> and things being converted being more ansibly
19:38:27 <clarkb> eventuall all will be an empty set
19:38:30 <corvus> (i... actually think translating erb to jinja2 sounds fun)
19:38:49 * mordred assigns corvus the erb to jinja2 tasks
19:38:53 <corvus> but i can at least agree it's busy work, and it is a cost that we should allocate to the ansible options :)
19:39:35 <clarkb> I also think in that scenario we'd likely want to continue to keep pupept on enough life support that it is viable until we are off it. Which likely means puppet 4 is still something we'd do? maybe just with less rigorous testing of each application deployment?
19:39:38 <pabelanger> the part I didn't fully understand, are we moving away from puppetmaster.o.o to say zuul post pipelines? Or new bastion host?
19:40:33 <clarkb> pabelanger: I think beacuse puppet remains until we are off of it the initial transition would require a bastion host remains
19:40:48 <clarkb> then a future spec could have us switch to zuul directly driving the deployment
19:40:52 <mordred> pabelanger: I was thinking we keep puppetmaster for now - but we could certainly start triggering some playbooks on it as a static host in a post pipeline
19:40:54 <clarkb> (basically thats out of scope I think)
19:40:58 <mordred> clarkb: ++
19:41:01 <pabelanger> okay, keep in mind, we do need to rebuild that server (since it is trusty)
19:41:28 <cmurphy> i think moving to puppet 4 is little enough work that we can keep going with it without rejecting future plans
19:41:28 <pabelanger> (but maybe just inplace upgrade)
19:41:59 <corvus> pabelanger: yeah, maybe we just punt for 8 months and if it's still important, do an in-place upgrade
19:42:14 <corvus> (we have a year of trusty remaining)
19:42:20 <pabelanger> wfm
19:42:33 <mordred> cmurphy: yah - and puppet needs to stick around for a while anyway for transition purposes
19:42:40 <clarkb> cmurphy: mordred exactly
19:43:31 <mordred> there's a little handwavey in the container spec about hiera/secrets and inventory that I think we should likely do some POCs of options
19:44:28 <fungi> so it's feeling like we have a little of all three specs we could consider in tandem?
19:44:49 <ianw> cmurphy: ++ i still see having puppet4 in the mix as useful over the medium term
19:44:59 <fungi> continue with puppet 4 uplift, get ansiblification working, possibly tack on containerization?
19:45:23 <clarkb> fungi: ya I get the sense what we'll acutally end up with is life support puppet4 short/medium term, transition to ansible base + container application "packaging" longer term, eventually having zuul do deployments (but this last bit should be its own spec)
19:45:30 <pabelanger> fungi: yah, that order seems right. We'll need ansible in place before containers
19:45:37 <corvus> fungi: yes, though i'd like us to disambiguate that to remove the 'possibly' :)
19:45:39 <mordred> clarkb: ++
19:45:43 <clarkb> having typed that does anyone here think that is a terrible idea just as their initial reaction?
19:45:54 <fungi> sure, having a roadmap is good
19:46:09 <mordred> nope. I think it's good ... a few pending details notwithstanding
19:46:28 <pabelanger> yah, I'm willing to try
19:46:33 <fungi> i'm not opposed to just sticking with puppet (i know it, even if it's changing), though i need to get more familiar with ansible/jinja anyway given zuul's use of it...
19:46:56 <corvus> should we make a new spec which lays out all 3 things in a single document, or revise the specs to build on each other?
19:47:15 <clarkb> corvus: I kinda think one doc will be best for people's understadning if not directly invovled day to day
19:47:35 <clarkb> maybe as a next step I should through my summary out to the infra lst today, and if no one objects we can start on making that the future spec?
19:47:36 * mordred can take a stab at that ... programming is harder than text writing right now
19:47:57 <clarkb> I guess nothing prevents starting to work on that now either if mordred wants to do word typing instead of programming typing
19:47:58 <mordred> reaching for the special keys is the thing that blows the most
19:48:06 <fungi> yeah, i can git behind the short-term puppet 4 and medium-term migrate to ansible (if that's what the others who have time to spend on this are interested in doing), but i'm not yet sold on the longer-term containers piece (no offense, mordred)
19:48:17 <fungi> er, get behind
19:48:21 * fungi sighs at finger memory
19:48:30 <corvus> fungi: i do that all the time
19:48:48 <pabelanger> +1 to migration idea
19:48:56 <mordred> fungi: none taken! having been skeptical myself for a long time I both appreciate and likely agree with at least some if not most of your concerns
19:49:01 <clarkb> ok sounds like we have a plan to make a plan :)
19:49:05 <mordred> woot
19:49:06 <clarkb> I'll send that out after lunch then
19:49:13 <pabelanger> yay
19:49:19 <clarkb> before we run out of time there is a related topic from pabelanger we should at least touch on
19:49:27 <fungi> i liked planning the plan to make a plan
19:49:31 <clarkb> #topic General Topics
19:49:42 <clarkb> pabelanger brings up a second round of xenial upgrades
19:49:59 <clarkb> given the above soft decision do we think we should continue to push towards xenial?
19:50:24 <fungi> smarcet was asking just today about upgrading openstackid-dev and openstackid.org to xenial so that openstackid can take advantage of newer laravel which needs newer php
19:50:27 <corvus> fungi: let me know what i can do to convince you -- i'm keen on it not only because it will reduce our deploy-in-config-management footprint, but i think it's coming at an opportune time when we're also at risk of fallout from pip10 and the like, where we might have to be looking at changing our deployment-from-source strategy anyway
19:50:32 <corvus> [sorry for the topic leak]
19:50:40 <pabelanger> yah, we have 1 year left on trusty and some long lived servers still to do. At this point, I think it is mostly about scheduling outages
19:50:49 <clarkb> my initial thought is since short/medium term we'd be puppet 4ing anyways (likely) that getting to xenial helps us avoid having another factor in the migration process. Basically get it out of the way then migrate applications based on the application not hosting platform
19:51:18 <corvus> [also, wow, turns out that was actually still relevant to this topic]
19:51:25 <clarkb> corvus: :)
19:51:45 <clarkb> also getting to systemd across the baord just makes toehr stuff easier
19:51:56 <clarkb> not because its systemd but because its consistent
19:51:59 <fungi> yeah, i wouldn't want the fact that we decide to do do something different in general which we're not quite ready to start implementing yet hold back server upgrades we need
19:52:02 <clarkb> (don't panic systemd haters :) )
19:52:10 <mordred> clarkb: agree
19:52:31 <corvus> i imagine we don't want to touch openstackid for the next 3 weeks anyway?
19:52:38 <clarkb> corvus: ++
19:52:51 <fungi> corvus: that would be a great question for smarcet &co
19:52:55 <pabelanger> yah, https://ethercalc.openstack.org/osiuhjzjq336 is the propose week for sprints
19:52:59 <pabelanger> which is after summit
19:53:13 <pabelanger> r-11, r-10, r-09
19:53:16 <fungi> i sure hope they're not wanting to switch platforms for it right before the summit, right
19:53:25 <clarkb> #link https://ethercalc.openstack.org/osiuhjzjq336 help select a week for a xenial upgrade sprint
19:53:39 <pabelanger> and https://etherpad.openstack.org/p/infra-sprint-xenial-upgrades-part2 has a list of servers we still need to migrate
19:53:53 <clarkb> #link https://etherpad.openstack.org/p/infra-sprint-xenial-upgrades-part2 servers that still need to migrate to xenial
19:54:40 <clarkb> if interesting in helping out please indicate your availability
19:55:00 <pabelanger> thanks!
19:55:03 <clarkb> I do think it would be wortwhile and helpful
19:55:17 <pabelanger> it was also a great learning experience for new infra-root
19:55:27 <clarkb> indeed
19:55:31 <mordred> ++
19:55:40 <clarkb> alright we have 5 minutes left I'll open the floor to anything else
19:55:45 <clarkb> #topic Open Discussion
19:56:19 <clarkb> I heard a rumor of zuul stickers so keep space for those :P
19:56:30 <clarkb> I also still have more infra ant stickers if you want them
19:57:04 <pabelanger> clarkb: sounds like team dinner is monday night?
19:57:28 <clarkb> oh right I proposed monday night at a brewpub near to the convention center. I think we can swing that without going through too much official paper pushing
19:57:46 <clarkb> (I don't want to pay a deposit on a room then sort out paying with a single card)
19:57:54 <anteaya> clarkb, good call
19:58:01 <fungi> sounds good to me
19:58:23 <clarkb> the proposed location seems quite large, has pool tables and brew their own beer. Food wise they have vegetarian options and gluten free looks like but not vegan or at least nothign screams vegan
19:58:26 <pabelanger> great
19:59:30 <clarkb> if for some reason you don't like this option I'm totally open to alternatives I'll just be asking that you do some of the leg work too :)
19:59:32 <clarkb> let me know
19:59:53 <fungi> i'm up for losing a few rounds of pool
20:00:06 <clarkb> and we are at time. Thanks everyone
20:00:08 <clarkb> #endmeeting