19:01:19 #startmeeting infra 19:01:19 Meeting started Tue Apr 17 19:01:19 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:19 morning! 19:01:20 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:22 The meeting name has been set to 'infra' 19:01:35 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:04 #topic Announcements 19:02:35 I don't have any 19:02:41 that's one 19:03:10 #topic Actions from last meeting 19:03:22 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-04-10-19.01.txt minutes from last meeting 19:03:46 There were no officially recorded #actions but mordred and pabelanger were palnning to update some specs 19:04:06 I know mordred didn't get to it I think combination of pip and meetings and dib updates meant that many of us were distracted by the fire of the day 19:04:14 but we can talk about that more during priority efforts 19:04:53 i was going to write a third-party spec, but yeah, have been pip-ing and dib-ing 19:04:59 #topic Specs approval 19:05:20 Similarly I havne't even looked at the specs repo for other specs recently 19:05:22 omg, pip and dib are typographic vertical mirror images 19:05:30 [sorry] 19:05:33 everyone's been pipping the dibs and dibbing the pips so far this week 19:05:42 anyone have a spec they want to call out for review before we move on? 19:06:54 #topic Priority Efforts 19:07:09 Good news is I hear fungi fixed storyboard bug updates via gerrit its plugin magic 19:07:23 #topic Storybaord 19:07:33 I'm leaving the typio 19:07:51 yeah, seems like the implicit configuration piggybacking on commentlinks stopped working for some reason 19:08:05 docs say it was deprecated but it just flat out stopped being considered i think 19:08:29 once i was able to translate the docs into human, i figured out the explicit config for it 19:08:53 anyway, as of the production gerrit restart on saturday it's working agani 19:09:00 fungi: was that the only known issue between gerrit and storyboard? the other behavior is working as expected? 19:09:14 yep 19:09:22 oh, so it's not tied to commentlinks now, it's an explicit option? 19:09:28 right 19:09:35 cool. that sounds like an improvement :) 19:09:39 and now that i see how to interpret the plugin documentation, there are some other behaviors we could turn on if we want 19:10:27 #link https://gerrit.googlesource.com/plugins/its-storyboard/ its-storyboard configuration docs 19:11:10 #undo 19:12:10 #link https://gerrit.googlesource.com/plugins/its-storyboard/+/stable-2.13/src/main/resources/Documentation/quick-install-guide.md#its_connection_its_connectionconnection-configuration its-storyboard configuration docs 19:13:06 anyway, that's all there is to say about that, i think 19:13:13 any other storyboard updates? looks like migrations are proceeding and that was a blocker for some projects iirc so yay 19:13:31 right, openstackclient and openstacksdk migrated last week 19:13:39 as did the tripleo validations subteam 19:14:14 tripleo is struggling to figure out how their task tracking workflows (which previously involved considering tripleo one giant fluid project on lp_ maps to per-repository task tracking in storyboard 19:15:06 i was going to start tinkering with search engine indexability next, but mordred pointed out that the angularjs upgrade he's working on comes with a feature to do that automagically 19:15:27 fungi: interesting the js gives bots what they need to index properly? 19:15:48 server-side javascript client that prerenders page content 19:16:23 huh 19:16:39 something called "universal pages" 19:17:14 #link https://universal.angular.io// Server-side rendered javascript 19:17:33 we may need to look at performance some time, https://storyboard.openstack.org/#!/story/list?status=active&tags=blocking-storyboard-migration just took like 40 seconds to load for me 19:17:58 yeah, a few people have noted that storytag-based queries are starting to take forever 19:18:12 i have a feeling there's an inefficient db operation there somewhere 19:19:15 ok anything else before we move on? 19:19:24 profilnig that is probably a little beyond my skillset though 19:19:28 er, profiling 19:19:32 nope, nothing from me at least 19:19:37 #topic Modernizing Config Management 19:20:04 * cmurphy lurks 19:20:07 I'm sort of operating under the assumption that this will be the next priority effort. Whether that is puppet4/5 or ansible or mordred's to be written use containers spec 19:20:25 given the rate of fires showing up over the last week I don't think we've really had any time to write down the thoughts on this in order to compare them directly 19:20:46 I'd like to shoot for next week but depending on how pip 10 related stuff goes I can see that being difficult too :/ 19:20:59 maybe we can switch to pip-based configuration 19:21:16 but if you are involved with one of the proposals updating the existing specs would be helpful so that we can compare them directly with up to date info across the board 19:21:53 i've started finishing up the last bits of http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet_4_prelim_testing.html 19:22:07 I don't really want to have a dig in conversation on this stuff until we have that information available (and hopefully everyone can read through it at least once before we do the major considering) 19:22:27 cmurphy: cool, do we need a next step spec for an actual migration plan that we can compare to the others? 19:22:40 "this is what a puppet4 migration looks like in infra" type of doc 19:23:04 clarkb: yeah i started working on that a while ago https://review.openstack.org/#/c/449933/ 19:23:04 patch 449933 - openstack-infra/infra-specs - Rename and expand Puppet 4 Preliminary Testing 19:23:14 and then decided to aim lower first 19:23:22 and then got very very distracted 19:23:59 cmurphy: cool, probably want to make sure that is up to date if anything has changed. Let me know if you need someone else to do that 19:24:25 clarkb: sure, will prioritize it this week 19:24:50 pabelanger: mordred: like I said not surprised that fires and other obligations kept people away from updating specs. Let me know if you need help or otherwise ENOTIME 19:25:42 I'd like to keep moving so that we have time to talk about some of those fires and just do a quick sanity check on where we are at on them 19:25:47 #topic General Topics 19:26:01 we'll start with frickler's question about advertising irc highlights for infra folks 19:26:30 one suggestion is to put it in the topic which isn't a bad idea since its close to where peolpe would be expected to use the highlights 19:26:49 I just wanted some feedback on whether folks want that or rather want to stay more hidden 19:26:50 I just worry the topic is quite full already 19:27:37 19:27 -!- Topic for #openstack-infra: Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source 19:27:38 https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/ 19:27:39 how to place it into the topic would be a seperate question 19:27:40 for reference ^ 19:27:52 or maybe where else to document it 19:28:26 do we really miss that many questions coming along in #openstack-infra? infra-root, etc certainly calls attention for people who know it, but not sure people are not getting help without it 19:28:57 maybe a mention in the top yellow box on https://docs.openstack.org/infra/manual/ would be enough, too 19:28:59 ianw: agreed 19:29:11 frickler: or https://docs.openstack.org/infra/ ? 19:29:17 or both :) 19:29:18 i don't always read scrollback, but when i do i try to follow up with people 19:29:23 ianw: not miss them, but certainly my responses may be delayed by an hour or more 19:29:30 that's the first link in the topic 19:30:16 ya updating the docs seems like a good first step 19:30:26 we could put basically the same yellow box in both docs 19:30:29 the docs are linked in the topic so if follwoing from topic to docs you hopefully find it 19:30:34 o.k., I can propose a patch for that 19:31:34 and with that I'd be o.k. to continue, when I added the topic, there wasn't much else listed 19:31:46 sounds like a plan then, thanks 19:31:48 but now we have other topics to handle I guess 19:31:52 people wait until the last minute to add topics 19:31:56 * fungi is part of the problem 19:32:02 also fires 19:32:12 The next item on the list was reorganizing the openstack-ci storyboard group to drop zuul specific projects and move them into a zuul specific group (from fungi) 19:32:21 this should be real fast 19:32:47 i asked in #zuul and there seemed to be some interest in putting the zuul family of repositories in a zuul project group for ease of overview 19:33:04 i'm +1 on creating/adding them to zuul group. i'm +/-0 on removing them from openstack-ci 19:33:15 just trying to figure out whether i should drop any or possibly all of them from the openstack-ci project group (we may also want to rename that at some point?) 19:33:19 maybe +0 19:33:43 for example, zone_zuul-ci.org maybe ought to appear in both project groups 19:33:46 I think I'm leaning towards the change if only to make it a little more clear there is a distinction now 19:33:55 and there are different groups responsible for each (even if membership overlaps) 19:34:07 i honestly don't know how many people rely on sb project grouping anyway 19:34:31 fungi: maybe not in sb directly but the projects.yaml file I think gets viewed in weird ways in places 19:34:41 but I also don't think it is a major issue 19:34:46 keeping zuul-jobs in both project groups may also make sense, but e.g. zuul-base-jobs (which openstack's deployment doesn't use) probably is fine being only in the zuul project group 19:37:05 doesn't seem to be any strong opinions either way 19:37:10 okay, i'll take a stab at this under the impression that we can mostly move zuul repos to a new zuul project group but that there are a handful of repos we might want to stick in both for convenience 19:37:33 and with that i have nothing else on this topic 19:37:40 onward! 19:37:45 Pip 10 released over the weekend 19:38:17 as expected this created problems for a non zero number of jobs. In some cases pip has been pinned to version 9 to get around this in others we've worked to accomodate the new version behaviors 19:38:49 Bring this up because i think we need to be careful where we pin pip 19:38:57 in some cases pip has also inadvertently been pinned as a side effect of image generation problems and pauses 19:39:17 its ok if openstack specifically wants to pin pip but places that we produce code or artifacts that are consumed outside of openstack likely shouldn't rely on that 19:39:40 but yes, capping pip should in general be viewed as a sad compromise we need to unwind as early as we can 19:39:45 zuul-jobs and dib in particular are places were I think we need to be careful to roll forward and continue to function rather than pinning pip 19:39:56 from devstack pov, i think the main job is pip10 safe, but i need to look into some sadness in the "platform" (!xenial) jobs -> https://review.openstack.org/#/c/561597/ 19:41:07 I've not gone looking too hard yet but on the infra side of things we should probably dig through puppet logs and see if we have any failures related to this as well 19:41:11 ianw: do we not need a venv fix for devstack? 19:41:47 curious how we're dealing with, e.g., the python-psutil ubuntu package not being uninstallable by pip 19:41:56 in devstack context i mean 19:42:00 right now aiui we are relying on pip pins 19:42:17 corvus: i don't think we *need* it ... that's not to say it is not a good idea 19:42:33 fungi: so in that package's case, we don't actually need that package installed 19:42:47 fyi I tried to list all the qa + infra issues here 19:42:51 #link https://etherpad.openstack.org/p/pip10-mitigation 19:42:58 ianw: great news ;) 19:43:03 i thought clarkb showed us that pip10 requires a venv. but ianw jut linked to a change which lifts the pin and the only failing jobs are "platform" jobs 19:43:16 so i'm missing something :) 19:43:35 corvus: ianw in the general case we can't control what distros package with (distutils or not) or what python system packages other packages we need will depend on 19:43:44 corvus: _if_ we can avoid preinstalling any python libraries via distro packages that have been built with distutils (some are built with setuptools instead) then pip 10 is able to "upgrade" from those fine 19:43:50 corvus: note that this currently checked only the plain devstack job. things may still be failing for specific projects like glance 19:44:13 so there may be specific scenarios where we can set things up such that they work 19:44:16 but others they don't 19:44:32 clarkb: ++ yes, it's whack-a-mole fix each issue 19:44:34 one particular concern of mine is that our images use glean and not cloud init so avoid a large amount of system python that many other users will have 19:44:34 i'm still missing what changed between clarkb's pip10 failure and ianw's pip10 success 19:44:49 so we can test on our images and say "it works. not a bug" but no one else will be able to run it sort of situation 19:44:55 corvus: so it's mainly an intersection of specific distro packages built on distutils and specific python packages we want to pip upgrade (primarily due to constraints version specifications) 19:45:10 corvus: I'm guessing they removed python-psutil from the installation list since ianw says it isn't required 19:45:30 https://review.openstack.org/#/c/561426/ 19:45:37 corvus: ianw is saying the workaround for the python-psutil failure was to stop preinstalling the distro-packaged python-psutil (if i understood correctly) 19:45:56 fungi / corvus: yes -> https://review.openstack.org/#/c/561426/ 19:45:56 frickler: ah, and that's a dep of 561597 19:46:01 got it 19:46:17 so that fixes the current problem, clarkb's venv solution still may be better long-term because of the points he just raised 19:46:22 corvus: right 19:46:41 however switching to venvs is likely also to be whack-a-mole of fixing random assumptions made in plugins and so on 19:46:50 cool... can we pretend i'm not devstack-core for a second and tell me what a 'devstack-platform' job is? 19:46:50 right, however we can manage to stop mixing distro package managers and pip is going to be more future-proof, i imagine 19:46:51 I've already had to address some of that in ironic with grenade 19:47:15 corvus: "platform" is just what i've called !xenial-based jobs 19:47:25 centos/suse/fedora 19:47:44 ianw: so they may have other random python packages installed (a la what clarkb was saying earlier) 19:47:57 corvus: that was the sentiment rougly expressed in http://lists.openstack.org/pipermail/openstack-dev/2018-April/129384.html too 19:48:04 corvus: yup and because the packagers are different they may use distutils on cetnos where ubuntu uses setuptools 19:48:07 and vice versa 19:48:22 okay, i think i grok 19:48:35 corvus: yes, and so as a first approach small hacks, such as possibly manually removing a .egg-info file so pip continues to do what it was doing before, might work best 19:49:22 I actually looked at the debian psutil package to see what sort of effort would be required to start untangling this at a distro level and I think it would not be easy 19:49:36 it is very package specific 19:50:05 clarkb: yeah, in some cases i've had packagers add the flag that writes the manifest file out and it's all good ... let me find that bug 19:50:47 I do think we are making reasonable progress. Its just a lot of situation specific cases of determining what the best way forward is 19:50:47 some distros have switched most of their packages to use something akin to 'pip install .' 19:50:56 and trying to avoid getting cornered into needing pip9 long term 19:51:05 as opposed to the problematic 'setup.py install' 19:51:53 clarkb: https://bugzilla.redhat.com/show_bug.cgi?id=1255206 for reference 19:51:54 bugzilla.redhat.com bug 1255206 in python-cffi "Consider adding installed-files.txt during build" [Unspecified,Closed: errata] - Assigned to ppicka 19:52:05 Briefly before we run out of time I also wanted to follow up on the xenial initramfs situation since it is related 19:52:43 Our xenial images are continuing to use global pip 9 on boot because we aren't able to build working xenial images that have pip 10 due to initramfs changes that prevent xenial from booting on rax and I guess other clouds 19:53:03 So be prepared for potential new set of failures as soon as that gets fixed and we upload new xenial iamges 19:53:27 ianw: pabelanger: sounds like you have it largely under control as far as debugging goes? 19:54:06 clarkb: that might be a little strong :) 19:54:08 though worth noting, we got tox-based jobs in general working with pip 10 according to my ad hoc tests upgrading pip before invoking the rest of the job 19:54:21 ianw: well at least we've identified the probable source of the problem now ot identify why it broke it :) 19:54:29 clarkb fixed the one known issue there (where tox-siblings relied on pip internals) 19:54:38 ianw: lsinitramfs was where I ended up before needing to do meeting things (and comparing the output across builds) 19:54:39 but i think we are narrowing down ... 19:55:07 #topic Open Discussion 19:55:20 Before we run out of time anything else? 19:55:29 clarkb: cool, yeah i thought i compared the initramfs's yesterday, but now i'm not sure if was comparing apples and oranges with the versions i grabbed. confirmation on that will be good 19:55:43 I'm visiting the dentist tomorrow at 1900UTC so will be out and about for that 19:56:14 i'm out all thursday 19:56:30 just mentioning that patchbot is an effort by notmyname. noticed it in #swift earlier and said I liked it 19:56:57 admittedly might need further discussion where to run it, thought 19:57:36 I think there was concern about its chattyness at one time? I tend to just smartfilter things out in weechat if I need to 19:58:03 yeah, it keeps finding its way back onto the meeting channels. project-specific channels who have some consensus in favor are of course free to have whatever bots they want join their channels 19:58:17 i think bots that run in official channels should be driven through the community maintained infrastructure 19:59:01 ya its nice to not lose features if someone can't keep running a bot for example 19:59:24 that i agree with 20:00:05 and we are at time 20:00:08 #endmeeting