19:01:09 #startmeeting infra 19:01:09 Meeting started Tue Mar 26 19:01:09 2019 UTC and is due to finish in 60 minutes. The chair is ianw. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:10 \o 19:01:11 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:13 The meeting name has been set to 'infra' 19:01:20 o/ 19:01:33 #link http://lists.openstack.org/pipermail/openstack-infra/2019-March/006298.html 19:01:50 #topic Announcements 19:02:21 clarkb remains AFK, hence me # tagging this one 19:02:29 o/ 19:02:44 no other announcements 19:02:56 #topic Actions from last meeting 19:03:06 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-03-19-19.01.html 19:03:23 clarkb was to ask about infra ptg days being thursday friday 19:03:33 anyone know if that happened? 19:03:54 i think i remember an irc message suggesting that it did 19:04:03 I seem to remember that that was confirmed, yes 19:04:08 i believe an updated ptg schedule has been published for feedback which has our time moved 19:04:12 * fungi looks 19:04:44 the published published one doesn't appear to have been updated: https://www.openstack.org/ptg/#tab_schedule 19:05:00 #link http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190321/3a3989bb/attachment-0001.png 19:05:22 from 19:05:28 #link http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004122.html Strawman Schedule 19:05:47 that seems to show infra/qa sharing a room thursday and friday 19:06:34 ok, i think let's leave an action item to just make sure everything published comes into sync? 19:06:39 diablo_rojo_phon probably has to let jamesmcarthur know when it's settled so the official one can be updated 19:07:12 #action clarkb double check infra ptg days published as thu/fri 19:07:14 correct :) 19:08:08 ok, all action items accounted for 19:08:22 #topic Specs Approval 19:08:34 I don't think there are any specs for approval at this point? 19:09:22 #topic Priority Efforts 19:09:54 cmurphy: i saw a few more puppet4 changes going out, just a matter of review at this point? 19:10:14 ianw: pretty much 19:10:31 clarkb caught up with me already so just proposed a few more 19:10:38 there was some fiddling over the weekend with the pip provider to work out getting gear upgraded on logstash01 19:10:51 but cmurphy got that sorted out 19:10:56 yay 19:11:18 oh again, more things related to the change in output of pip? 19:11:32 the same issue as before 19:11:36 we just hadn't merged that fix 19:11:41 till last weekend 19:12:14 oh right ... heh plenty of time for pip to have changed itself again in between :) 19:12:21 heh 19:12:32 #topic Opendev 19:12:53 i did send out the git -> https changes as discussed in last few meetings. no real issues 19:13:24 a few weird things with some repos not having .gitreview, and some missing zuul branch configs, both of which i didn't code for, but otherwise ok 19:14:00 jroll: want to update on where the renaming thread is at? 19:14:17 (for openstack) 19:14:27 ianw: it seems we have consensus on the plan, I need to spin a new version of the governance patch 19:14:48 it should be straightforward to make a list of the renames from there 19:15:10 what's that review # for a link? 19:15:15 (spoiler: we're going with option #2, all unofficial projects in one namespace, all official projects in the openstack namespace) 19:15:23 #link https://review.openstack.org/#/c/645601/ 19:16:17 sounds good (though unofficial projects can still self-select their own org at https://ethercalc.openstack.org/opendev-transition ) 19:16:31 well, all unofficial projects who don't request their own namespace from opendev before the maintenance in one namespace anyway 19:16:38 er, or what corvus said 19:16:41 sorry, yes, that 19:16:45 we have some takers there now (3 people!) 19:16:49 we'll make an attempt to tell them about it :) 19:16:59 jroll: much obliged! :) 19:17:10 #link https://storyboard.openstack.org/#!/story/2004627 19:17:24 but yeah, happy to stick anyone who doesn't respond into "unknown/" 19:17:33 or whatever (or whatever/) :) 19:17:59 so it's fungi signed up to do the modrewrite & gitreview update implementation of the renames once this is finalised? 19:18:41 yes, an initial stab at incorporating corvus's test redirects is already in place on files.o.o and covers git.openstack.org urls 19:18:52 i'm still working on the variant for the other git sites 19:19:16 but all the vhosts are configured there now with their ssl certs/keys installed 19:19:45 cool, well that all seems like it's moving along 19:19:49 i'm planning on testing whether zuul itself follows those redirects for the git driver later today/tomorrow 19:19:57 for the git-review and zuul configuration patching, i had a quick question... 19:20:03 ianw: re task 29707 -- i asked fungi for a cert for docs.opendev.org earlier; he went ahead and got review.opendev.org too. if the LE stuff is ready, great, and if it's not, we can use the comodo. 19:20:13 particularly on the zuul configuration itself 19:20:55 corvus: it's very close to reviewable; i just finished yesterday all the implementation bits i can think of. will beat into shape with docs etc and request reviews in next day or so 19:21:04 is it sufficient to just patch the contents of {.,}zuul{,.d/*}.yaml or do we also need to try to update any references to checked-out paths in playbooks and roles? 19:21:06 ianw: w00t 19:22:16 fungi: well, there are things like this https://opendev.org/openstack/openstack-chef/src/branch/master/playbooks/integration.yaml#L6 19:22:17 fungi: that's a good question ... if we don't update the playbooks/roles, i feel certain that some things will break; however it's understandable we may want to reduce the complexity there and leave that for humans... 19:22:44 can we make zuul create symlinks for the old path? 19:22:59 similar to the redirects? 19:23:27 fungi: you mean things like "upper_constraints: "{{ ansible_user_dir }}/{{ zuul.projects['git.openstack.org/openstack/requirements'].src_dir }}/upper-constraints.txt" 19:23:29 frickler: well, there's also the more 'zuulish' way of doing it which is using {{ zuul.projects['git.openstack.org/openstack/openstack-chef'].src_dir }} 19:23:41 ianw: frickler: yeah, those examples 19:24:03 corvus: will that work better when we s/git.o.o/opendev.org/ ? 19:24:21 frickler: no, but it won't be affected by symlinks, so jobs that use that would need a different workaround 19:24:46 corvus: ah, right 19:24:51 fungi: i feel like that will have to be updated, it will just break otherwise? if that's zuul.projects keeping an alias or a direct code change ... 19:25:02 so i have a feeling there are enough of these cases that my script should also recurse top-level playbooks and roles directories 19:26:11 i know zuul itself just expects its configs to have .yaml extensions, but ansible is likely to be more flexible in that regard. does also looking for *.yml in (any child of) those directories cover it? 19:26:12 fungi: that's probably not a bad idea. though playbooks can be anywhere (zuul.yaml will tell you where they are, but of course they could include_tasks in other directories) 19:26:24 fungi: generally yes. 19:26:39 oh, interesting, so the zuul configuration can set explicit paths to playbooks and roles? 19:26:48 fungi: playbooks, yes, roles no 19:27:03 roles are either at the top level of the repo or in roles/ 19:27:23 oh, so they can also be in the top level directory. okay 19:27:44 i bet if you just did roles/* and each of the directories holding a playbook, you'd get 99.99% 19:28:25 what's the configuration directive i should look for which sets the path to a playbook tree? 19:29:01 codesearch for "zuul.projects\['git.openstack.org" actually showed up way less than i thought it would be ... only around 60 matches 19:29:04 fungi: the values under 'pre-run', 'run', and 'post-run' are (lists of) paths to playbook files relative to repo root 19:29:41 thanks, that ought to be manageable 19:29:44 fungi: eg "pre-run: playbooks/my-playbook.yaml" is typical, but could be "pre-run: some/weird/dir/whatever.yaml" 19:30:08 and those are relative to the top-level directory of the repository? 19:30:11 yep 19:30:17 perfect 19:30:34 and they have to end in .yaml or .yml? 19:30:41 fungi: feel free to ping me with questions or even requests to help write code chunks 19:30:45 fungi: .yaml for zuul 19:30:53 er 19:30:54 1 sec 19:31:03 ansible will let them be anything though, i guess 19:31:13 i think zuul.yaml has to be .yaml i think playbooks (even those run by zuul) can be anything 19:31:22 yeah, didn't want to monopolize the meeting. i've got enough for a first stab at it 19:31:27 thanks! 19:32:07 any other updates? i think dtroyer was all good with your bits last week? 19:32:26 yup 19:32:57 #topic Storyboard 19:33:22 any updates? 19:33:44 the missing support for swift form-post signatures in openstacksdk has been proposed 19:34:14 #link https://review.openstack.org/639760 (openstack/openstacksdk) Add support for generating form-post signatures 19:34:30 this was a blocker for secure implementation of the story attachments feature 19:35:45 other than that, lots of pending changes to review 19:36:19 yesterday was the proposal deadline for outreachy candidates, so there was quite the flurry of last-minute activity 19:36:32 (or maybe the deadline is today) 19:37:19 cool, do the reviews require a lot of web-architecture pre-knowledge? or is it pretty standalone? 19:38:05 there's a bit of a split. the storyboard repo holds the api implementation in python 19:38:39 the storyboard-webclient repo holds the web front-end and is mostly javascript, using an oldish angular framework 19:39:28 luckily we also have draft builds of the webclient rendered in ci jobs, so you can see whether or not they work and/or if they introduce unexpected behavior 19:40:33 i don't know whether diablo_rojo_phon or SotK are around and have anything to add 19:41:01 Nothing from me I don't think. 19:41:09 Oh. 19:41:19 Outreachy application deadline was today 19:41:32 though we also have a storyboard irc meeting most weeks this time on wednesdays, if anyone wants to continue more in-depth discussions about such things (as well as any time in #storyboard) 19:41:34 So the flow of potential new contributors will be ebbing now 19:42:10 yeah, the channel was full of folks trying to figure out how to install storyboard and run local tests for the past month or more 19:42:32 but a big plus is that helped to fine-tune some of our developer docs and workflow 19:43:24 anyway, that's all i've got for now 19:43:52 #topic General Topics 19:44:12 #topic PTG planning 19:44:34 just wanted to stamp the link in again 19:44:38 #link https://etherpad.openstack.org/2019-denver-ptg-infra-planning 19:45:13 i'm sure ideas, questions and comments are welcome 19:45:58 we seem to be free of other general topics, which must mean that everything is perfect and we can all just go to the beach 19:46:20 we had nb nodes run full again 19:46:31 possibly related to suse breaking their repos 19:46:31 #topic Open Discussion 19:47:05 frickler: oh right, is the short version of the unbound issues that it was a super old package without the latest dnssec keys due to the mirror going away? 19:47:42 ianw: yes, except that it is still broken, mirror updates hopefully happening soon 19:48:32 also it wasn't mirror going down, but repo actively being published in an empty state iiuc 19:49:17 cool ... that explains why it looked so similar to the actual brokenness we had due to dnssec being stripped by opendns ... red herrings everywhere! 19:49:38 * fungi wants a pickled herring sandwich now 19:49:53 yes, the effect was the same, unbound failing to validate any dnssec response it receives 19:50:03 returning servfail to the clients 19:50:08 i think that a cron job on the nb* to clear out +x day files on /opt/dib_tmp might help slightly 19:50:38 but also, i had to clear out a few builds that were on disk, but not in "nodepool image-list" 19:50:55 how much quota do we have for volumes? maybe adding like 1tb to each would also help 19:51:06 so we don't fill it up within a few days 19:51:29 nb01 and nb02 have a 1tb volume each 19:51:52 nb03 is a harder one to solve since there's no cinder there, and instead we need to ask for a flavor with a larger rootfs 19:52:20 though cleanup got utilization on those volumes down to something like 50% anyway 19:52:29 not sure how much headroom we really want to provide 19:53:25 I think if we had space for like 1-2 weeks of builds, that would allow us to easier react on issues 19:53:25 aiui Shrews is looking into nodepool leaks 19:54:49 frickler: i think that a background cleanup job might provide a little more breathing space, for a small error 19:54:49 yeah, there was something about making sure nodepool removes the on-disk files corresponding to failed builds? 19:55:13 rather than just leaving them lying around 19:56:45 ok, we're just about at time, as usual discussion can continue in #openstack-infra 19:56:49 does anyone have a simple recipe to monitor /opt space in cacti? otherwise I can look into that 19:57:33 frickler: as in something to alert us? 19:58:07 ianw: as in being able to track usage. currently we only have / unless I looked wrong 19:58:10 as in right now /opt isn't tracked in cacti for the nodepool builders 19:58:35 oh right, yeah that would be a start 19:58:39 i thought the script did that. but if it's not working, it's just some clicking 19:59:25 i think it will add any filesystems which are present when the server is first added 19:59:44 but if new filesystems are added after the config has already been generated, that doesn't get updated 19:59:45 i guess we stopped running the script all the time? 20:00:02 or, rather, we have it ignore hosts that are already there? 20:00:16 sounds likely, but i haven't looked deeply 20:00:20 what's "the script"? 20:00:35 maybe there's an option to run it once for that host manually 20:01:01 i think that tracking it explicitly would be great, and we can then correlate when it starts to climb over the long term 20:01:14 frickler, want an action item to look into this? 20:01:16 i have no idea where the script ended up :) 20:01:30 ianw: yeah 20:01:49 https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/cacti.pp#n64 20:01:51 #link https://opendev.org/openstack-infra/system-config/src/branch/master/modules/openstack_project/manifests/cacti.pp#L72-L76 Exec[cacti_import_xml] 20:02:04 #action frickler look into disk usage tracking, particularly of /opt on nodepool builders 20:02:12 it's the create_graphs.sh script 20:02:34 oh, nevermind, yeah it's what corvus linked 20:02:38 https://opendev.org/openstack-infra/system-config/src/branch/master/modules/openstack_project/files/cacti/create_graphs.sh 20:02:40 there it is 20:02:46 #link https://opendev.org/openstack-infra/system-config/src/branch/master/modules/openstack_project/files/cacti/create_graphs.sh 20:03:18 and the cron at line 87 in the manifest runs it 20:03:28 o.k., I'll take a closer look tomorrow 20:03:37 hrm. that sort of looks like it should be running and idempotent 20:03:41 so there maybe a bug there 20:03:53 apparently, there's a log file :) 20:04:00 ok, i gotta dash. i'll end meeting here and we can pick this up later 20:04:02 #endmeeting