19:01:05 <clarkb> #startmeeting infra 19:01:06 <openstack> Meeting started Tue Mar 5 19:01:05 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:10 <openstack> The meeting name has been set to 'infra' 19:01:18 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-March/006291.html 19:01:40 <clarkb> #topic Announcements 19:02:20 <clarkb> The OpenStack TC election is happening right now. Those of you that have contributed to the infra project in the last year or so should have received a personal ballot in your email inbox. You have until 23:45UTC today to vote if you haven't yet 19:03:10 <clarkb> #link https://docs.openstack.org/infra/system-config/signing.html#attestation sign the Train release key 19:03:33 <fungi> #link https://sks-keyservers.net/pks/lookup?op=vindex&search=0xcdc08088c3cb45a9be08332b2354069e5b504663&fingerprint=on OpenStack Infra (Train Cycle) artifact signing key 19:03:33 <clarkb> fungi has set up a train release gpg key (thank you!) and those of us with root access should follow the steps in the link above to sign it 19:04:05 <clarkb> That is on my todo list for post meeting so hopefully I'll get that done soon. Something to do while waiting for afs to replicate a not small volume :) 19:04:15 <clarkb> #link https://www.openstack.org/summit/denver-2019/call-for-presentations Submit Forum session ideas now 19:04:28 <diablo_rojo> o/ 19:04:37 <clarkb> It is also that time of year where we plan to get together and talk about designing software/services 19:04:57 <clarkb> If you've got topics for that feel free to submit them 19:06:24 <clarkb> #topic Actions from last meeting 19:06:31 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-02-26-19.01.txt minutes from last meeting 19:06:50 <clarkb> I set an everyone action to sign up for gitea story tasks. I think that has gone reasonably well and we are making progress on the gitea front 19:07:06 <corvus> yep, all tasks accounted for 19:07:07 <clarkb> Lets bring that up during opendev discussion though 19:07:13 <clarkb> corvus: yay 19:07:30 <clarkb> #topic Specs Approval 19:07:51 <clarkb> We also merged the letsencrypt spec after reviews last week. THank you everyone for that 19:08:05 <clarkb> Mostly just mentioning it here in case you missed it. 19:08:19 <clarkb> #topic Priority Efforts 19:08:52 <clarkb> #topic Storyboard 19:09:25 <clarkb> I ran through running storyboard tests locally and pushed an update to the storyboard docs that appear to have merged 19:09:30 <diablo_rojo> Been slowly getting SotK's attachment patches through 19:09:56 <diablo_rojo> Also still being innundated with prospective outreachy interns 19:09:59 <clarkb> I did that in support of making it easier for outreachy applicants. Hopefully they find it useful 19:10:07 <fungi> yeah, mordred added an alternative solution relying on swift tempurls which will hopefully solve one of the security risks we missed with the previous client auth idea 19:10:25 <clarkb> cool sounds like good progress on the attachments implementation then 19:10:54 <mordred> fungi: oh - crappit - I need to finish that / follow up on those patches 19:11:04 <mordred> fungi: thanks for the reminder 19:11:04 <clarkb> for outreachy are there specific tasks/deadlines we should be aware of as a "hosting project" (I don't know what the actual term is) 19:11:24 <fungi> the outreachy hopefuls are also burning through our new-contributor-friendly stories at am amazing pace 19:11:28 <mordred> (fwiw, I've been saying "tempurl" when I mean "formsubmit" - but yes) 19:11:35 <diablo_rojo> I think the application proposals for interns closes soon. 19:11:41 <fungi> oh, thanks for the clarification mordred 19:11:45 <diablo_rojo> Not sure what day we need to pick an intern by though 19:12:02 <diablo_rojo> fungi, yeah that has been amazing 19:12:12 <fungi> oh, related to interns but not outreachy, hogepodge also has some capstone interns from ndsu he's sticking on some storyboard todo items as well 19:12:13 <diablo_rojo> I guess I will need to go triage some more soon 19:12:21 <diablo_rojo> Oh yeah 19:12:35 <diablo_rojo> specifically amending the migration script to handle bps as well 19:12:43 <diablo_rojo> optionally 19:12:56 <fungi> right, we've had a few teams request lp blueprints be migrated to stories 19:12:59 <diablo_rojo> So we could migrate bps and bugs, or just one or the other 19:13:08 <clarkb> diablo_rojo: we probably want to do a review pass of those changes as part of the application process? (and when I say we I really mean people that know storyboard) 19:13:11 <mordred> diablo_rojo: maybe after meeting let's sync on a plan for the formsubmit swift stuff 19:14:27 <diablo_rojo> mordred, sounds good 19:14:38 <diablo_rojo> SotK, should probably be around too for that 19:14:56 <clarkb> Are there other topics we should cover that aren't outreachy and the attachments implementation? 19:14:57 <diablo_rojo> clarkb, yeah I have reviewed a few so far, but there are more to do if anyone can help out. 19:15:11 <diablo_rojo> Placement will be using sb? Yay! 19:15:13 <clarkb> diablo_rojo: maybe we can tag them all with an outreachy topic? 19:15:26 <clarkb> diablo_rojo: that might help reviewers? not sure how difficult it is to sort them out from the other work 19:15:26 <diablo_rojo> clarkb, oh yeah thats a good call. I can go and do that 19:15:30 <clarkb> great thanks 19:15:45 <fungi> yeah, the last placement meeting talked about sb some, but ended in cdent's e-mail to the openstack-discuss ml 19:15:58 <fungi> so probably doesn't need separate summarizing 19:16:52 <clarkb> ok lets move on then 19:17:00 <clarkb> #topic Update Config Management 19:17:37 <clarkb> We are ready to start deploying puppet 4 in places. https://review.openstack.org/#/c/630391/ Does that for our dev servers but was updated to fix an issue frickler caught before I could approve it 19:17:47 <clarkb> rereviews very welcome and I'm happy to approve that and watch it 19:17:58 <cmurphy> yay 19:18:29 <clarkb> We continue to refine the docker-compose to deploy docker images with ansible process/tooling 19:18:41 <clarkb> we are running 4 services with that now? 19:18:47 <clarkb> corvus: ^ you probably know off the top of your head 19:19:19 <corvus> gitea, haproxy, registry, preview 19:19:24 <clarkb> Also my changes to add a zuulcd user to bridge are merged so I think we can start with Zuul triggering CD jobs again 19:19:50 <clarkb> I guess so far just one chagne and we need to have a discussion about auto updating key data 19:20:08 <clarkb> but that is sufficient for now if we want to start splitting out config management runs from the always run every 15 minutes to run when necessary 19:20:21 <corvus> i think we might be able to pull the git playbook out of base and run it on changes to projects.yaml 19:20:44 <clarkb> zuul-preview deployments likely another good candidate for that (run when zuul-preview updates) 19:20:57 <fungi> i also added some suggestions on the authorized_keys retrieval change 19:20:59 <corvus> yep, we can chain it to follow the promote job 19:21:12 <clarkb> fungi: thanks I'll take a look today 19:21:44 <fungi> as a side note there, is there a reason we can't have it retrieve keys from zuul.opendev.org instead of zuul.openstack.org? i think that solves several of my concerns 19:22:00 <corvus> fungi: we absolutely can. it was just written before it existed 19:22:04 <clarkb> yup that 19:22:12 <fungi> cool, i thought (hoped) that might be the case 19:22:18 <clarkb> if that will alleviate concern I'll go ahead and make that update 19:22:34 <fungi> i was having trouble putting together a working api url to do that yesterday but that was probably just my fault 19:22:57 <fungi> clarkb: the other remaining concern will be easier to take care of later with letsencrypt 19:23:13 <fungi> and is something we might want to consider doing for all our opendev le certs 19:23:51 <fungi> though i'm not 100% sure the python requests module supports it (yet) 19:24:21 <clarkb> ok I'll take a look today 19:24:40 <clarkb> Are there other config management related changes/items that we want to go over before we zoom in on opendev/gitea? 19:25:07 <fungi> none for which i'm aware 19:25:39 <clarkb> #topic OpenDev 19:25:47 <corvus> #link https://storyboard.openstack.org/#!/story/2004627 opendev tasks 19:26:13 <corvus> since last week, we're basically halfway done with the tasks needed before we are ready to actually move 19:26:38 <clarkb> We now have 8 gitea servers running behind haproxy hosting https://opendev.org 19:26:57 <corvus> i think the most urgent thing is actually a new task not on there, which is to fix the default branch setting for the projects we're replicating now as well as new projects 19:27:14 <corvus> mordred is working on that, with, presumably, clarkb and i helping 19:27:36 <clarkb> yup I'm happy to help on that 19:27:41 <corvus> once replication is done and looks correct, i plan on starting on the last task on the list, which is the announcement email 19:28:10 <corvus> i want to get that out as early as possible so folks know the plan and openstack and starlingx can make any preparations for repo moves 19:28:18 <clarkb> ++ 19:28:23 <fungi> sounds great 19:28:31 <corvus> but i also want to wait until we have a fully-functioning site to demo in the email 19:28:59 <fungi> i've just updated the story to include previously missed airship domain name 19:28:59 <corvus> meanwhile, we can work on the rest of the preparatory tasks 19:29:20 <corvus> fungi: thx :) 19:29:22 <clarkb> is anyone stuck or needing help on any of their tasks? 19:30:17 <fungi> i just need to carve out time to knock mine out. shouldn't take long 19:30:48 <ianw> I'll go through devstack plugins looking for git://'s 19:31:04 <frickler> the issues and release tabs don't look correct for some projects, do we have a plan for that? 19:31:13 <frickler> issues always redirect to sb 19:31:32 <ianw> corvus: is there a plan for port 9418 in the interim, to maybe forward to a proxy or something? 19:31:42 <clarkb> frickler: the issue there being we should link to launchpad in some cases? 19:31:43 <corvus> ianw: no, it's not possible to support 19:32:07 <clarkb> frickler: we should be able to fix that by looking up if a project uses storyboard or not in the project.yaml yaml file I expect. Then run the fixup playbook across all the projects once we haev that logic in place 19:32:13 <clarkb> frickler: probably worth a new task to track that though 19:32:26 <frickler> and releases look incomplete, compare https://opendev.org/openstack/designate/releases to https://github.com/openstack/designate/releases 19:32:29 <mordred> yeah. 19:33:01 <mordred> well - the releases tab is a thing we probably need to think about - maybe we should remove it from the ui for now until we have a good plan for it 19:33:17 <clarkb> frickler: looking at that almost appears to be a gitea bug? 19:33:30 <clarkb> the tags are available on the code screen at least 19:33:31 <corvus> can someone tell me what the bug is? 19:33:41 <clarkb> corvus: only the rc releases seem to be listed 19:34:01 <corvus> ah 19:34:02 <clarkb> well rc and beta. None of the actual releases are listed 19:34:03 <mordred> well - the bug for ME is that I don't want us hosting links to things people shouldn't ever download 19:34:08 <corvus> mordred: yeah, i think you were... that. yes. 19:34:12 <frickler> its not consistent it seems, but e.g. for designate 7.0.0 seems to be missing 19:34:17 <corvus> so let's just drop the releases page 19:34:20 <mordred> yeah 19:34:35 <mordred> I think it could be a nice thing to do something with at some point to have releases point to our actual releases 19:34:35 <corvus> i'll make new tasks for all of these things 19:34:36 <clarkb> mordred: the concern being the tarball there isn't a valid python sdist? I agree that is a proble mand removing the tab would be good 19:34:37 <fungi> makes sense to me 19:34:37 <mordred> but that's more work 19:34:43 <mordred> clarkb: yes. that is the problem 19:34:58 <mordred> same bug as github's ui has - a git export isn't always a valid source tarball 19:35:45 <clarkb> this is good feedback and godo for us to catch these things before we announce it more widely :) 19:36:00 <corvus> yep :) 19:36:47 <clarkb> Anything else opendev or gitea related? If not we'll move on to our general topics list 19:36:58 <corvus> story updated with 3 new tasks, 2 of which are unassigned 19:37:45 <corvus> clarkb: none from me 19:38:13 <clarkb> #topic General Topics 19:38:28 <clarkb> #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup Trusty Server Upgrades 19:38:37 <clarkb> This is mostly a status update and thank you to those helping with this 19:38:51 <clarkb> I've written down an afs fileserver upgrade plan on that etherpad (and am currently executing step 1) 19:39:07 <clarkb> if you have a chance to review that the input I've received so far has been helpful 19:39:19 <clarkb> And once again if you are able to upgrade a server or two please volunteer. 19:39:37 <clarkb> As for sprinting on this I haven't heard any responses to my email about that 19:39:53 <clarkb> For my schedule March 11 and 12 work best (next monday and tuesday) 19:40:13 <clarkb> so I'm probably going to go ahead and scribble that in as the sprint days and I'll see you there if you can make it? 19:40:57 <clarkb> Next up is ianw's backups topic 19:41:14 <clarkb> ianw seems we have some cleanup to do on the backup erver? 19:41:34 <ianw> the main problem is : /dev/mapper/main-backups 3.0T 2.8T 579M 100% /opt/backups 19:42:48 <ianw> now we have 3tb in "old" volumes, which we decided to clean up "in a few months" @ http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-10-31-19.01.log.html#l-189 19:43:16 <clarkb> ianw: I would be on board with cleaning that up by deleting the contents of those volumes, but then repurposing the voluming as the target for new backups 19:43:25 <clarkb> then we can do similar cleanup with the current volumes in the future? 19:43:31 <ianw> i actually had a thought that we document our process here as periodically keeping the old volume, and swapping 19:43:42 <clarkb> ianw: ++ 19:43:50 <ianw> clarkb: cool, i think we said the same thing :) 19:43:55 <clarkb> since bup is append only having a formal rotation process would be good 19:43:55 <corvus> i like it 19:44:15 <clarkb> ianw: is this something you'll be able to drive? or do we need volunteer(s)? 19:44:25 <ianw> clarkb: i can take an action item to do that 19:44:37 <clarkb> #action ianw rotate backup volumes on backup server 19:44:39 <clarkb> ianw: thanks 19:45:07 <ianw> yeah, i didn't know if we wanted to investigate some sort of automated pruning 19:45:26 <clarkb> Not to completely derail, but at times I wonder if borgbackup would be helpful here. I've been using it for personal backups and my understanding is it can do append only backups with pruning 19:45:35 <fungi> i wonder if we could abuse logrotate to do it 19:45:39 <ianw> personally, i feel like some custom script has a non-trivial chance of wiping out much more than it intends 19:46:11 <clarkb> I don't think you can prune bup backups 19:46:27 <clarkb> at least not in bup itself 19:46:45 <ianw> no, but we could do some sort of automated repo moving type thing 19:46:45 <fungi> yeah, i guess rotating with retention means losing deduplication of the rotated copy 19:47:02 <clarkb> ianw: ya we could move the dest repo out of the way and replace it with a new empty target 19:47:08 <clarkb> then after X time delete the moved repo 19:47:09 <ianw> it's probably spec worthy, maybe the whole backups process could do with a re-look in the opendev world 19:47:17 <clarkb> ++ 19:47:31 <corvus> well, pruning with bup is experimental 19:47:35 <clarkb> corvus: ah 19:47:44 <ianw> anyway, i won't sign up to take on a full overhaul :) but just flagging it for now 19:47:59 <clarkb> ianw: ya the earlier fix should give us breathing room to think it through 19:48:24 <ianw> the other quick thing i noticed was we're not backing up stats on graphite.opendev.org (previously openstack.org) 19:48:30 <clarkb> fwiw I'm likely to suggest borgbackup if we intend to do a big overhaul as I've really enjoyed working with it. You can fuse mount a backup to copy files and it encrypts at the source so you don't have to trust your remote :) 19:48:33 <ianw> not sure if that is a bug or feature 19:49:01 <corvus> i don't think we need to back up stats 19:49:02 <clarkb> ianw: I'm not sure how important it is to backup that data 19:49:13 <clarkb> its nice to have data that we can live without 19:49:32 <ianw> ok, if it's a feature, that's cool :) 19:49:37 <corvus> i agree with every one of the possible ways of reading clarkb's sentence :) 19:50:13 <corvus> (it's nice to have (data that we can live without)) and (it's (nice to have) data that (we can live without)) 19:50:34 <clarkb> :) 19:50:54 <clarkb> Anything else backup related? 19:51:13 <ianw> no, that's it for me 19:51:17 <clarkb> #topic Open Discussion 19:52:07 <clarkb> I'm going to be in Seattle on the 19th but I should arrive early enough and be settled enough to run our meeting (especially with DST change this weekend). However the week after that (26th) we are doing a mini road trip for spring break back to the seattle area and I will likely be afk 19:52:20 <clarkb> if anyone really wants to run the meeting in 3 weeks let me know 19:53:04 <ianw> i have no other plans, so can do that if nobody else wants to 19:53:37 <clarkb> Also DST change in the USA (and maybe canada?) is this weekend apparently 19:53:55 <frickler> one other thing regarding gitea: can we clean up the exim paniclogs to reduce the amount of daily mails? 19:54:05 <clarkb> This means that our meeting will happen an hour later in the day that it happened today for those of you in the usa 19:54:49 <fungi> frickler: i think it's a more general bootstrapping problem. we might want to wipe the exim paniclog if present at the end of launch-node.py 19:55:13 <frickler> clarkb: so the meeting won't stay at 19:00 UTC? 19:55:22 <clarkb> frickler: oh sorry it stays at 1900UTC 19:55:25 <fungi> pretty much every new server we've launched has a paniclog from (it appears) when exim configuraiton was first getting applied 19:55:30 <clarkb> It is an hour later in the day relative to my local timezone 19:55:34 <frickler> ah, I read it the wrong way around 19:55:57 <clarkb> frickler: sorry I wasn't very clear on that. For pacific time meeting was at 11am today will be Noon next week 19:56:01 <fungi> crazy folk like me who set their clocks to utc don't need to worry ;) 19:56:35 <fungi> instead i get to deal with fixing the offset on all the non-community meetings in my remind file 19:56:37 <corvus> my irc client's clock is utc 19:57:00 <mordred> my irc client's clock is utc - and I put the infra meeting into my calendar using the iceland timezone so it just works :) 19:57:30 <fungi> i probably should tag the recurring meetings in my remind file which *don't* follow utc and then just script their time changes accordingly 19:58:01 <frickler> the EU might be stopping switching time zone soon ... maybe 19:58:10 <clarkb> https://review.openstack.org/#/c/639454/ is an example of a chagne that does post launch launch tasks 19:58:23 <clarkb> if anyone wants to do the cleanup exim paniclog change ^ is a good place to start probably 19:58:38 <corvus> i also voted to recommend that my state legislature consider beginning a process to investigate putting together a proposal to petition the us commerce department to allow us to stop switching between pst/pdt 19:58:43 <fungi> clarkb: thanks, we can probably just tack on a `rm -rf /var/log/exim4/paniclog` and call it a day 19:58:43 <corvus> so i'm *very* proactive on this issue. 19:58:51 <clarkb> corvus: nice 19:59:02 <clarkb> oregon has a bill that gets rejected every year to ditch dst (or go to dst fulltime) 19:59:16 <clarkb> And we are just about at time. Thank you everyone 19:59:19 <clarkb> #endmeeting