19:01:05 <clarkb> #startmeeting infra
19:01:06 <openstack> Meeting started Tue Mar  5 19:01:05 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:10 <openstack> The meeting name has been set to 'infra'
19:01:18 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-March/006291.html
19:01:40 <clarkb> #topic Announcements
19:02:20 <clarkb> The OpenStack TC election is happening right now. Those of you that have contributed to the infra project in the last year or so should have received a personal ballot in your email inbox. You have until 23:45UTC today to vote if you haven't yet
19:03:10 <clarkb> #link https://docs.openstack.org/infra/system-config/signing.html#attestation sign the Train release key
19:03:33 <fungi> #link https://sks-keyservers.net/pks/lookup?op=vindex&search=0xcdc08088c3cb45a9be08332b2354069e5b504663&fingerprint=on OpenStack Infra (Train Cycle) artifact signing key
19:03:33 <clarkb> fungi has set up a train release gpg key (thank you!) and those of us with root access should follow the steps in the link above to sign it
19:04:05 <clarkb> That is on my todo list for post meeting so hopefully I'll get that done soon. Something to do while waiting for afs to replicate a not small volume :)
19:04:15 <clarkb> #link https://www.openstack.org/summit/denver-2019/call-for-presentations Submit Forum session ideas now
19:04:28 <diablo_rojo> o/
19:04:37 <clarkb> It is also that time of year where we plan to get together and talk about designing software/services
19:04:57 <clarkb> If you've got topics for that feel free to submit them
19:06:24 <clarkb> #topic Actions from last meeting
19:06:31 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-02-26-19.01.txt minutes from last meeting
19:06:50 <clarkb> I set an everyone action to sign up for gitea story tasks. I think that has gone reasonably well and we are making progress on the gitea front
19:07:06 <corvus> yep, all tasks accounted for
19:07:07 <clarkb> Lets bring that up during opendev discussion though
19:07:13 <clarkb> corvus: yay
19:07:30 <clarkb> #topic Specs Approval
19:07:51 <clarkb> We also merged the letsencrypt spec after reviews last week. THank you everyone for that
19:08:05 <clarkb> Mostly just mentioning it here in case you missed it.
19:08:19 <clarkb> #topic Priority Efforts
19:08:52 <clarkb> #topic Storyboard
19:09:25 <clarkb> I ran through running storyboard tests locally and pushed an update to the storyboard docs that appear to have merged
19:09:30 <diablo_rojo> Been slowly getting SotK's attachment patches through
19:09:56 <diablo_rojo> Also still being innundated with prospective outreachy interns
19:09:59 <clarkb> I did that in support of making it easier for outreachy applicants. Hopefully they find it useful
19:10:07 <fungi> yeah, mordred added an alternative solution relying on swift tempurls which will hopefully solve one of the security risks we missed with the previous client auth idea
19:10:25 <clarkb> cool sounds like good progress on the attachments implementation then
19:10:54 <mordred> fungi: oh - crappit - I need to finish that / follow up on those patches
19:11:04 <mordred> fungi: thanks for the reminder
19:11:04 <clarkb> for outreachy are there specific tasks/deadlines we should be aware of as a "hosting project" (I don't know what the actual term is)
19:11:24 <fungi> the outreachy hopefuls are also burning through our new-contributor-friendly stories at am amazing pace
19:11:28 <mordred> (fwiw, I've been saying "tempurl" when I mean "formsubmit" - but yes)
19:11:35 <diablo_rojo> I think the application proposals for interns closes soon.
19:11:41 <fungi> oh, thanks for the clarification mordred
19:11:45 <diablo_rojo> Not sure what day we need to pick an intern by though
19:12:02 <diablo_rojo> fungi, yeah that has been amazing
19:12:12 <fungi> oh, related to interns but not outreachy, hogepodge also has some capstone interns from ndsu he's sticking on some storyboard todo items as well
19:12:13 <diablo_rojo> I guess I will need to go triage some more soon
19:12:21 <diablo_rojo> Oh yeah
19:12:35 <diablo_rojo> specifically amending the migration script to handle bps as well
19:12:43 <diablo_rojo> optionally
19:12:56 <fungi> right, we've had a few teams request lp blueprints be migrated to stories
19:12:59 <diablo_rojo> So we could migrate bps and bugs, or just one or the other
19:13:08 <clarkb> diablo_rojo: we probably want to do a review pass of those changes as part of the application process? (and when I say we I really mean people that know storyboard)
19:13:11 <mordred> diablo_rojo: maybe after meeting let's sync on a plan for the formsubmit swift stuff
19:14:27 <diablo_rojo> mordred, sounds good
19:14:38 <diablo_rojo> SotK, should probably be around too for that
19:14:56 <clarkb> Are there other topics we should cover that aren't outreachy and the attachments implementation?
19:14:57 <diablo_rojo> clarkb, yeah I have reviewed a few so far, but there are more to do if anyone can help out.
19:15:11 <diablo_rojo> Placement will be using sb? Yay!
19:15:13 <clarkb> diablo_rojo: maybe we can tag them all with an outreachy topic?
19:15:26 <clarkb> diablo_rojo: that might help reviewers? not sure how difficult it is to sort them out from the other work
19:15:26 <diablo_rojo> clarkb, oh yeah thats a good call. I can go and do that
19:15:30 <clarkb> great thanks
19:15:45 <fungi> yeah, the last placement meeting talked about sb some, but ended in cdent's e-mail to the openstack-discuss ml
19:15:58 <fungi> so probably doesn't need separate summarizing
19:16:52 <clarkb> ok lets move on then
19:17:00 <clarkb> #topic Update Config Management
19:17:37 <clarkb> We are ready to start deploying puppet 4 in places. https://review.openstack.org/#/c/630391/ Does that for our dev servers but was updated to fix an issue frickler caught before I could approve it
19:17:47 <clarkb> rereviews very welcome and I'm happy to approve that and watch it
19:17:58 <cmurphy> yay
19:18:29 <clarkb> We continue to refine the docker-compose to deploy docker images with ansible process/tooling
19:18:41 <clarkb> we are running 4 services with that now?
19:18:47 <clarkb> corvus: ^ you probably know off the top of your head
19:19:19 <corvus> gitea, haproxy, registry, preview
19:19:24 <clarkb> Also my changes to add a zuulcd user to bridge are merged so I think we can start with Zuul triggering CD jobs again
19:19:50 <clarkb> I guess so far just one chagne and we need to have a discussion about auto updating key data
19:20:08 <clarkb> but that is sufficient for now if we want to start splitting out config management runs from the always run every 15 minutes to run when necessary
19:20:21 <corvus> i think we might be able to pull the git playbook out of base and run it on changes to projects.yaml
19:20:44 <clarkb> zuul-preview deployments likely another good candidate for that (run when zuul-preview updates)
19:20:57 <fungi> i also added some suggestions on the authorized_keys retrieval change
19:20:59 <corvus> yep, we can chain it to follow the promote job
19:21:12 <clarkb> fungi: thanks I'll take a look today
19:21:44 <fungi> as a side note there, is there a reason we can't have it retrieve keys from zuul.opendev.org instead of zuul.openstack.org? i think that solves several of my concerns
19:22:00 <corvus> fungi: we absolutely can.  it was just written before it existed
19:22:04 <clarkb> yup that
19:22:12 <fungi> cool, i thought (hoped) that might be the case
19:22:18 <clarkb> if that will alleviate concern I'll go ahead and make that update
19:22:34 <fungi> i was having trouble putting together a working api url to do that yesterday but that was probably just my fault
19:22:57 <fungi> clarkb: the other remaining concern will be easier to take care of later with letsencrypt
19:23:13 <fungi> and is something we might want to consider doing for all our opendev le certs
19:23:51 <fungi> though i'm not 100% sure the python requests module supports it (yet)
19:24:21 <clarkb> ok I'll take a look today
19:24:40 <clarkb> Are there other config management related changes/items that we want to go over before we zoom in on opendev/gitea?
19:25:07 <fungi> none for which i'm aware
19:25:39 <clarkb> #topic OpenDev
19:25:47 <corvus> #link https://storyboard.openstack.org/#!/story/2004627 opendev tasks
19:26:13 <corvus> since last week, we're basically halfway done with the tasks needed before we are ready to actually move
19:26:38 <clarkb> We now have 8 gitea servers running behind haproxy hosting https://opendev.org
19:26:57 <corvus> i think the most urgent thing is actually a new task not on there, which is to fix the default branch setting for the projects we're replicating now as well as new projects
19:27:14 <corvus> mordred is working on that, with, presumably, clarkb and i helping
19:27:36 <clarkb> yup I'm happy to help on that
19:27:41 <corvus> once replication is done and looks correct, i plan on starting on the last task on the list, which is the announcement email
19:28:10 <corvus> i want to get that out as early as possible so folks know the plan and openstack and starlingx can make any preparations for repo moves
19:28:18 <clarkb> ++
19:28:23 <fungi> sounds great
19:28:31 <corvus> but i also want to wait until we have a fully-functioning site to demo in the email
19:28:59 <fungi> i've just updated the story to include previously missed airship domain name
19:28:59 <corvus> meanwhile, we can work on the rest of the preparatory tasks
19:29:20 <corvus> fungi: thx :)
19:29:22 <clarkb> is anyone stuck or needing help on any of their tasks?
19:30:17 <fungi> i just need to carve out time to knock mine out. shouldn't take long
19:30:48 <ianw> I'll go through devstack plugins looking for git://'s
19:31:04 <frickler> the issues and release tabs don't look correct for some projects, do we have a plan for that?
19:31:13 <frickler> issues always redirect to sb
19:31:32 <ianw> corvus: is there a plan for port 9418 in the interim, to maybe forward to a proxy or something?
19:31:42 <clarkb> frickler: the issue there being we should link to launchpad in some cases?
19:31:43 <corvus> ianw: no, it's not possible to support
19:32:07 <clarkb> frickler: we should be able to fix that by looking up if a project uses storyboard or not in the project.yaml yaml file I expect. Then run the fixup playbook across all the projects once we haev that logic in place
19:32:13 <clarkb> frickler: probably worth a new task to track that though
19:32:26 <frickler> and releases look incomplete, compare https://opendev.org/openstack/designate/releases to https://github.com/openstack/designate/releases
19:32:29 <mordred> yeah.
19:33:01 <mordred> well - the releases tab is a thing we probably need to think about - maybe we should remove it from the ui for now until we have a good plan for it
19:33:17 <clarkb> frickler: looking at that almost appears to be a gitea bug?
19:33:30 <clarkb> the tags are available on the code screen at least
19:33:31 <corvus> can someone tell me what the bug is?
19:33:41 <clarkb> corvus: only the rc releases seem to be listed
19:34:01 <corvus> ah
19:34:02 <clarkb> well rc and beta. None of the actual releases are listed
19:34:03 <mordred> well - the bug for ME is that I don't want us hosting links to things people shouldn't ever download
19:34:08 <corvus> mordred: yeah, i think you were... that. yes.
19:34:12 <frickler> its not consistent it seems, but e.g. for designate 7.0.0 seems to be missing
19:34:17 <corvus> so let's just drop the releases page
19:34:20 <mordred> yeah
19:34:35 <mordred> I think it could be a nice thing to do something with at some point to have releases point to our actual releases
19:34:35 <corvus> i'll make new tasks for all of these things
19:34:36 <clarkb> mordred: the concern being the tarball there isn't a valid python sdist? I agree that is a proble mand removing the tab would be good
19:34:37 <fungi> makes sense to me
19:34:37 <mordred> but that's more work
19:34:43 <mordred> clarkb: yes. that is the problem
19:34:58 <mordred> same bug as github's ui has - a git export isn't always a valid source tarball
19:35:45 <clarkb> this is good feedback and godo for us to catch these things before we announce it more widely :)
19:36:00 <corvus> yep :)
19:36:47 <clarkb> Anything else opendev or gitea related? If not we'll move on to our general topics list
19:36:58 <corvus> story updated with 3 new tasks, 2 of which are unassigned
19:37:45 <corvus> clarkb: none from me
19:38:13 <clarkb> #topic General Topics
19:38:28 <clarkb> #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup Trusty Server Upgrades
19:38:37 <clarkb> This is mostly a status update and thank you to those helping with this
19:38:51 <clarkb> I've written down an afs fileserver upgrade plan on that etherpad (and am currently executing step 1)
19:39:07 <clarkb> if you have a chance to review that the input I've received so far has been helpful
19:39:19 <clarkb> And once again if you are able to upgrade a server or two please volunteer.
19:39:37 <clarkb> As for sprinting on this I haven't heard any responses to my email about that
19:39:53 <clarkb> For my schedule March 11 and 12 work best (next monday and tuesday)
19:40:13 <clarkb> so I'm probably going to go ahead and scribble that in as the sprint days and I'll see you there if you can make it?
19:40:57 <clarkb> Next up is ianw's backups topic
19:41:14 <clarkb> ianw seems we have some cleanup to do on the backup erver?
19:41:34 <ianw> the main problem is : /dev/mapper/main-backups            3.0T  2.8T  579M 100% /opt/backups
19:42:48 <ianw> now we have 3tb in "old" volumes, which we decided to clean up "in a few months" @ http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-10-31-19.01.log.html#l-189
19:43:16 <clarkb> ianw: I would be on board with cleaning that up by deleting the contents of those volumes, but then repurposing the voluming as the target for new backups
19:43:25 <clarkb> then we can do similar cleanup with the current volumes in the future?
19:43:31 <ianw> i actually had a thought that we document our process here as periodically keeping the old volume, and swapping
19:43:42 <clarkb> ianw: ++
19:43:50 <ianw> clarkb: cool, i think we said the same thing :)
19:43:55 <clarkb> since bup is append only having a formal rotation process would be good
19:43:55 <corvus> i like it
19:44:15 <clarkb> ianw: is this something you'll be able to drive? or do we need volunteer(s)?
19:44:25 <ianw> clarkb: i can take an action item to do that
19:44:37 <clarkb> #action ianw rotate backup volumes on backup server
19:44:39 <clarkb> ianw: thanks
19:45:07 <ianw> yeah, i didn't know if we wanted to investigate some sort of automated pruning
19:45:26 <clarkb> Not to completely derail, but at times I wonder if borgbackup would be helpful here. I've been using it for personal backups and my understanding is it can do append only backups with pruning
19:45:35 <fungi> i wonder if we could abuse logrotate to do it
19:45:39 <ianw> personally, i feel like some custom script has a non-trivial chance of wiping out much more than it intends
19:46:11 <clarkb> I don't think you can prune bup backups
19:46:27 <clarkb> at least not in bup itself
19:46:45 <ianw> no, but we could do some sort of automated repo moving type thing
19:46:45 <fungi> yeah, i guess rotating with retention means losing deduplication of the rotated copy
19:47:02 <clarkb> ianw: ya we could move the dest repo out of the way and replace it with a new empty target
19:47:08 <clarkb> then after X time delete the moved repo
19:47:09 <ianw> it's probably spec worthy, maybe the whole backups process could do with a re-look in the opendev world
19:47:17 <clarkb> ++
19:47:31 <corvus> well, pruning with bup is experimental
19:47:35 <clarkb> corvus: ah
19:47:44 <ianw> anyway, i won't sign up to take on a full overhaul :)  but just flagging it for now
19:47:59 <clarkb> ianw: ya the earlier fix should give us breathing room to think it through
19:48:24 <ianw> the other quick thing i noticed was we're not backing up stats on graphite.opendev.org (previously openstack.org)
19:48:30 <clarkb> fwiw I'm likely to suggest borgbackup if we intend to do a big overhaul as I've really enjoyed working with it. You can fuse mount a backup to copy files and it encrypts at the source so you don't have to trust your remote :)
19:48:33 <ianw> not sure if that is a bug or feature
19:49:01 <corvus> i don't think we need to back up stats
19:49:02 <clarkb> ianw: I'm not sure how important it is to backup that data
19:49:13 <clarkb> its nice to have data that we can live without
19:49:32 <ianw> ok, if it's a feature, that's cool :)
19:49:37 <corvus> i agree with every one of the possible ways of reading clarkb's sentence :)
19:50:13 <corvus> (it's nice to have (data that we can live without)) and (it's (nice to have) data that (we can live without))
19:50:34 <clarkb> :)
19:50:54 <clarkb> Anything else backup related?
19:51:13 <ianw> no, that's it for me
19:51:17 <clarkb> #topic Open Discussion
19:52:07 <clarkb> I'm going to be in Seattle on the 19th but I should arrive early enough and be settled enough to run our meeting (especially with DST change this weekend). However the week after that (26th) we are doing a mini road trip for spring break back to the seattle area and I will likely be afk
19:52:20 <clarkb> if anyone really wants to run the meeting in 3 weeks let me know
19:53:04 <ianw> i have no other plans, so can do that if nobody else wants to
19:53:37 <clarkb> Also DST change in the USA (and maybe canada?) is this weekend apparently
19:53:55 <frickler> one other thing regarding gitea: can we clean up the exim paniclogs to reduce the amount of daily mails?
19:54:05 <clarkb> This means that our meeting will happen an hour later in the day that it happened today for those of you in the usa
19:54:49 <fungi> frickler: i think it's a more general bootstrapping problem. we might want to wipe the exim paniclog if present at the end of launch-node.py
19:55:13 <frickler> clarkb: so the meeting won't stay at 19:00 UTC?
19:55:22 <clarkb> frickler: oh sorry it stays at 1900UTC
19:55:25 <fungi> pretty much every new server we've launched has a paniclog from (it appears) when exim configuraiton was first getting applied
19:55:30 <clarkb> It is an hour later in the day relative to my local timezone
19:55:34 <frickler> ah, I read it the wrong way around
19:55:57 <clarkb> frickler: sorry I wasn't very clear on that. For pacific time meeting was at 11am today will be Noon next week
19:56:01 <fungi> crazy folk like me who set their clocks to utc don't need to worry ;)
19:56:35 <fungi> instead i get to deal with fixing the offset on all the non-community meetings in my remind file
19:56:37 <corvus> my irc client's clock is utc
19:57:00 <mordred> my irc client's clock is utc - and I put the infra meeting into my calendar using the iceland timezone so it just works :)
19:57:30 <fungi> i probably should tag the recurring meetings in my remind file which *don't* follow utc and then just script their time changes accordingly
19:58:01 <frickler> the EU might be stopping switching time zone soon ... maybe
19:58:10 <clarkb> https://review.openstack.org/#/c/639454/ is an example of a chagne that does post launch launch tasks
19:58:23 <clarkb> if anyone wants to do the cleanup exim paniclog change ^ is a good place to start probably
19:58:38 <corvus> i also voted to recommend that my state legislature consider beginning a process to investigate putting together a proposal to petition the us commerce department to allow us to stop switching between pst/pdt
19:58:43 <fungi> clarkb: thanks, we can probably just tack on a `rm -rf /var/log/exim4/paniclog` and call it a day
19:58:43 <corvus> so i'm *very* proactive on this issue.
19:58:51 <clarkb> corvus: nice
19:59:02 <clarkb> oregon has a bill that gets rejected every year to ditch dst (or go to dst fulltime)
19:59:16 <clarkb> And we are just about at time. Thank you everyone
19:59:19 <clarkb> #endmeeting