| clarkb | also we should probably reaudit our uids and gids once this is solved | 00:00 |
|---|---|---|
| clarkb | https://docs.puppetlabs.com/puppet/latest/reference/lang_run_stages.html#limitations-and-known-issues covers the gotchas with stages | 00:01 |
| * SpamapS points at LDAP standing in the corner looking ignored and sad like most senior citizens. | 00:01 | |
| *** bhunter71 has quit IRC | 00:02 | |
| clarkb | jeblair: https://review.openstack.org/#/c/158177/4 is ready to be approved now. do you want to rereview? my rebase of it removed your earlier +2 | 00:02 |
| clarkb | SpamapS: I think ldap wold be massive overkill here but considering we only need to separate two sets of uid/gid groups | 00:03 |
| clarkb | s/but// | 00:03 |
| *** ryanpetrello has quit IRC | 00:03 | |
| nibalizer | also ldap+kerb is notoriously sadface | 00:03 |
| jeblair | clarkb: aprvd | 00:03 |
| nibalizer | clarkb: did you get it figured out? | 00:03 |
| clarkb | nibalizer: well I know why its happening but don't know fix | 00:03 |
| jeblair | SpamapS: i suspect the gertty bug has to do with the list of affected files being larger than the screen size. | 00:04 |
| clarkb | nibalizer: we need https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/users.pp#n11 something like that for User and Group globally | 00:04 |
| *** melwitt has quit IRC | 00:04 | |
| *** otter768 has joined #openstack-infra | 00:04 | |
| clarkb | SpamapS: also its only ~10 users | 00:04 |
| *** timcline has joined #openstack-infra | 00:05 | |
| *** camunoz has quit IRC | 00:05 | |
| clarkb | and last but not least I would probably have a sad if I needed ldap to login to things when clouds break | 00:05 |
| clarkb | nibalizer: were you there for the dns depends on ldap, ldap depends on dns explosion? | 00:06 |
| *** camunoz has joined #openstack-infra | 00:06 | |
| nibalizer | clarkb: no, before my time | 00:06 |
| crinkle | clarkb: https://docs.puppetlabs.com/puppet/latest/reference/lang_scope.html is the reference re: resource default scope, tldr is it depends where you put it | 00:06 |
| clarkb | crinkle: thank you | 00:07 |
| pleia2 | clarkb: good to know, thanks | 00:08 |
| nibalizer | clarkb: so given that code if you include openstack_project::users then realize () some users it should work | 00:08 |
| clarkb | nibalizer: yes that works fine | 00:08 |
| clarkb | nibalizer: problem is we also have system users | 00:08 |
| *** otter768 has quit IRC | 00:09 | |
| ianw | dstufft: sorry, looks like a garden variety dependency ordering issue. pip is trying to install cryptography before it has installed the version of six that it needs | 00:09 |
| openstackgerrit | Syed Ismail Faizan Barmawer proposed openstack-infra/project-config: Add ironic-lib to openstack https://review.openstack.org/157636 | 00:10 |
| nibalizer | clarkb: so the problem is you have conflicts or? | 00:10 |
| clarkb | nibalizer: the problem is puppet is mostly unordered so if you create a system user or regular users before realizing any of the admin users the gids/uids can conflict | 00:11 |
| *** zz_dimtruck is now known as dimtruck | 00:11 | |
| openstackgerrit | Merged openstack-infra/puppet-ansible: Remove embedded ansible role https://review.openstack.org/158177 | 00:11 |
| *** ryanpetrello has joined #openstack-infra | 00:12 | |
| *** thedodd has quit IRC | 00:12 | |
| *** rkukura has joined #openstack-infra | 00:14 | |
| nibalizer | clarkb: hrm okay | 00:14 |
| nibalizer | so login.defs needs to go before any packages get installed that set up their own users, that kind of thing? | 00:14 |
| clarkb | yes | 00:15 |
| nibalizer | login.defs seems to start the uids at 3000, so the 20xx uids of the sysadmins shouldn't conflict, as long as it goes in the right order | 00:15 |
| clarkb | or before puppet does anything | 00:15 |
| clarkb | right | 00:15 |
| nibalizer | so the default login.defs seems ot start uids at 1000, at least on ubuntu | 00:16 |
| clarkb | yes | 00:16 |
| nibalizer | so either login.defs hasn't been laid down yet, and users get created in the 1xxx range, or login.defs has been laid down, and users get created in the 3xxx range | 00:17 |
| nibalizer | i don't see how there is room to conflict on the 2xxx range | 00:17 |
| clarkb | nibalizer: well its doing both, for example with gerrit2 on gerrit server the uid:gid was 1001:2001 | 00:17 |
| clarkb | nibalizer: I think it may be a race | 00:17 |
| clarkb | but not sure where exactly | 00:17 |
| nibalizer | weird | 00:18 |
| clarkb | ensuring that login.defs happens well before anything else should hoepfully avoid such racing | 00:18 |
| *** patrickeast_ has joined #openstack-infra | 00:19 | |
| clarkb | I think what may have happened there was the group for mordred with gid 2000 ran before the login.defs applied | 00:19 |
| clarkb | because there is no require there | 00:19 |
| clarkb | then gerrit2 got a user at max_gid+1 == 2001 | 00:20 |
| clarkb | then gerrit2's uid was set to 1001 | 00:20 |
| *** patrickeast__ has joined #openstack-infra | 00:20 | |
| *** patri728 has joined #openstack-infra | 00:20 | |
| *** patri586 has joined #openstack-infra | 00:20 | |
| nibalizer | well you can stage => 'setup' on the login.defs, but that is notoriously hated | 00:20 |
| nibalizer | but might be a good use here | 00:20 |
| clarkb | then login.defs is applied | 00:20 |
| clarkb | then mordreds user with uid 2000 is create | 00:20 |
| nibalizer | ya that makes sense | 00:20 |
| *** MarkAtwood has joined #openstack-infra | 00:20 | |
| clarkb | we could just make an ansible play to apply the login.defs before puppet runs >_> | 00:22 |
| dstufft | ianw: what version of setuptools? | 00:22 |
| clarkb | anyways this is the sort of problem that deserves some though | 00:23 |
| clarkb | so I won't go trying to fix it right this second | 00:23 |
| *** markvoelker has quit IRC | 00:26 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 00:26 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/infra-specs: Update gerrit topic to downstream-puppet https://review.openstack.org/161038 | 00:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Add "downstream-puppet" gerrit topic https://review.openstack.org/161039 | 00:27 |
| jeblair | asselin: ha! :) | 00:28 |
| asselin | I got in first :) | 00:28 |
| jeblair | asselin: you win! i will abandon | 00:28 |
| nibalizer | only one can win | 00:28 |
| jeblair | asselin: even our commit messages were almost identical! | 00:28 |
| asselin | :) | 00:29 |
| clarkb | jeblair: any additional input on the user creation stuff? | 00:29 |
| clarkb | I am reapproving 158175 and child now | 00:30 |
| jeblair | clarkb: ah, yeah. i was also thinking that using ansible for this might not be terrible, but i think since the values are specified in puppet, we should try to keep doing it there to localize concerns | 00:30 |
| jeblair | clarkb: so i lean toward trying stage | 00:30 |
| *** ZZelle_ has quit IRC | 00:31 | |
| nibalizer | we could also make everything in openstack_project::server depend require the login.defs file | 00:31 |
| clarkb | ok I will draft that change up shortly | 00:31 |
| clarkb | nibalizer: I am not sure I trust all other things to require server explicitly to set up the graph right | 00:31 |
| *** yamamoto has quit IRC | 00:31 | |
| nibalizer | or put a Package { require => File['login.defs'] } | 00:32 |
| nibalizer | since really its packages that cause this issue | 00:32 |
| nibalizer | i suppose a user here and there | 00:32 |
| jeblair | nibalizer: oh that's an interesting solution too | 00:32 |
| clarkb | my concern with that is it requires us to actively do it or the entire world is broken | 00:32 |
| clarkb | which isn't a good way to fail | 00:33 |
| *** carl_baldwin has quit IRC | 00:33 | |
| jeblair | clarkb: i understood nibalizer's suggestion to be adding that one line to one place which would cause it to be applied universally? | 00:34 |
| clarkb | jeblair: no, it would't be universal, crinkle's link above explains it | 00:34 |
| *** bdpayne has quit IRC | 00:34 | |
| jeblair | i have not read that yet :( | 00:34 |
| clarkb | the only way to make it universal would be to put it in site.pp globally after deining the file in site.pp globally | 00:34 |
| clarkb | at least that wsa my reading of the scoping of resource defaults | 00:35 |
| ianw | dstufft: setuptools 0.9.8 i'm pretty sure. it seems "pip install "six>1.9" cryptography does not the get version of six installed before cryptography needs it | 00:35 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Require login.defs in resource defaults https://review.openstack.org/161041 | 00:35 |
| dstufft | ianw: you need a newer setuptools | 00:35 |
| dstufft | 7.0+ | 00:35 |
| nibalizer | jeblair: clarkb that ^^ is one approach | 00:35 |
| dstufft | ianw: luckily, I have a COPR just for you! | 00:35 |
| clarkb | nibalizer: don't you also need to define the file resource at a global level? | 00:35 |
| clarkb | nibalizer: I don't think that iwll work | 00:35 |
| dstufft | ianw: https://copr.fedoraproject.org/coprs/pypa/pypa/ | 00:35 |
| clarkb | but apply test should confirm or deny that | 00:36 |
| dstufft | (the answer about why this happens is finnicky and requires abunch of terrible background knowledge to fully grok) | 00:36 |
| nibalizer | clarkb: welllllll apply test uses csplit right... so it may cut those global definitions out | 00:36 |
| clarkb | nibalizer: oh that too | 00:36 |
| nibalizer | jeblair: the stage proposal is one line in one place, and should work | 00:37 |
| dstufft | but if you add that copr and then yup update to get a newer python-setuptools | 00:37 |
| dstufft | magic should shappen | 00:37 |
| nibalizer | but everyone hates stages in puppet (not that we're afraid of going against the grain) | 00:37 |
| nibalizer | most of the hatred of stages is that usually stages are tried when we need to bootstrap some component of puppet, and puppet fails hard at that | 00:37 |
| ianw | dstufft: this is the unholy world of devsatck ... see the last line of https://git.openstack.org/cgit/openstack-dev/devstack/tree/tools/install_pip.sh where we pip install -U setuptools | 00:37 |
| nibalizer | using stages to lay down login.defs so that all package and user resoruces behave correctly is probably a decent use | 00:37 |
| nibalizer | crinkle: Hunner may have thoughts on that | 00:38 |
| nibalizer | ill send up the stage proposal for review as well | 00:38 |
| dstufft | ianw: if you're doing pip install -U setuptools you should be using setuptools 12.3 or so | 00:38 |
| clarkb | nibalizer: cool that would be good to compare | 00:38 |
| clarkb | nibalizer: thank you | 00:38 |
| ianw | dstufft: yes, i think we are | 00:39 |
| clarkb | nibalizer: also don't forget to cleanup the current resource default as it won't be necessary and may not work given the require across stages | 00:39 |
| ianw | [root@devstack-ps centos]# pip list | grep setuptools | 00:39 |
| ianw | setuptools (12.3) | 00:39 |
| nibalizer | https://docs.puppetlabs.com/references/latest/metaparameter.html#stage | 00:39 |
| dstufft | ianw: hmm, that error looks like the one that setuptools 7 fixed, but if you're actually using setuptools 12.3 unles sit regressed it shouldn't happen there | 00:40 |
| dstufft | unless it's a different error.. | 00:40 |
| ianw | dstufft: but yeah, it doesn't get it right : http://paste.openstack.org/show/187009/ | 00:40 |
| clarkb | ugh only on classes too | 00:40 |
| ianw | dstufft: "pip install "six>=1.9" cryptography" ... leads to "Installing collected packages: cryptography, six" , wonder if it got flipped around somewhere | 00:41 |
| dstufft | ianw: the problem before was that it installed it, but the sys.path ended up in the wrong order | 00:42 |
| dstufft | I don't have time to dig into it more right now though :( | 00:43 |
| openstackgerrit | Merged openstack-infra/system-config: Install standalone ansible roles https://review.openstack.org/158175 | 00:43 |
| *** rkukura has quit IRC | 00:43 | |
| openstackgerrit | Merged openstack-infra/system-config: Rename roles.yml to roles.yaml https://review.openstack.org/159661 | 00:44 |
| ianw | dstufft: ok, that's cool, from http://paste.openstack.org/show/187009/ i can see that it hasn't made an attempt to install six first -- you think this is more a pip issue? i don't imagine the cryptography pkg dependencies are wrong | 00:44 |
| *** dmellado has quit IRC | 00:44 | |
| dstufft | ianw: it was a combination of pip/setuptools last time, and it appears it's probably the same again | 00:44 |
| *** jamespage has quit IRC | 00:45 | |
| ianw | dstufft: ok, i will file a github issue as a first step here | 00:45 |
| *** jamespage has joined #openstack-infra | 00:45 | |
| *** jamespage has quit IRC | 00:45 | |
| *** jamespage has joined #openstack-infra | 00:45 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Set the openstack_project::users class to run in 'setup' stage https://review.openstack.org/161048 | 00:45 |
| nibalizer | okay thats a draft, i'm not super confident that the syntax there is all right but we might need to iterate on it a bit | 00:46 |
| anteaya | clarkb: you had a comment on patchset 4 that they needed to add -21 to the job name to get an f21 node, is this correct? https://review.openstack.org/#/c/158726/4..6/jenkins/jobs/projects.yaml | 00:46 |
| openstackgerrit | Merged openstack-infra/project-config: Add #openstack-rally to accessbot https://review.openstack.org/160831 | 00:47 |
| clarkb | anteaya: ya | 00:47 |
| clarkb | nibalizer: thanks | 00:47 |
| anteaya | clarkb: thanks | 00:47 |
| clarkb | nibalizer: you need an explicit include stdlin? | 00:48 |
| anteaya | tried watching a movie, at the 6 minute mark the two main characters who had never met before were in bed | 00:48 |
| clarkb | eg stgaes aren't part of puppet proper? | 00:48 |
| *** dmellado has joined #openstack-infra | 00:48 | |
| anteaya | decided to come back and do code review as there was more reality in gerrit | 00:48 |
| clarkb | nibalizer: oh is stage setup part of stdlib? why not include stdlib closer to the stage use then? | 00:49 |
| nibalizer | clarkb: not really https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/manifests/stages.pp | 00:49 |
| nibalizer | probably we should stage { 'setup': before => Stage['main'] } in site.pp actually | 00:49 |
| clarkb | ya looks like that named stage is from stdlib | 00:49 |
| nibalizer | as opposed to using stdlibs stages | 00:49 |
| nibalizer | also i included stdlib wrong, its stdlib::stages | 00:50 |
| clarkb | nibalizer: I would like to avoid using site.pp if possible | 00:50 |
| nibalizer | server.pp works for me :P | 00:50 |
| *** mtanino has quit IRC | 00:50 | |
| clarkb | its better to keep things local to wher they are used imo | 00:50 |
| nibalizer | ya me adding stuf to site.pp braeks our csplitting | 00:50 |
| nibalizer | okay i have to change locations, can we pick this up later on this evening, or tomorrow? | 00:50 |
| clarkb | ya no rush, I am not making any new nodes today | 00:51 |
| *** gyee has quit IRC | 00:51 | |
| *** david-lyle has quit IRC | 00:52 | |
| ianw | dstufft: yeah, so if you switch the command-line ordering to "cryptography, six>=1.9" it works and six is installed first. https://github.com/pypa/pip/issues/2473 | 00:52 |
| *** rhsu has joined #openstack-infra | 00:52 | |
| dstufft | ianw: that makes sense, the order you specify them in matters for what order pip will install them in | 00:53 |
| dstufft | because pip doesn't have a dep solver | 00:53 |
| dstufft | so it tries to fake one | 00:53 |
| dstufft | :/ | 00:53 |
| anteaya | StevenK: so https://review.openstack.org/#/c/157626/3 help me understand why descriptions should be mandatory? | 00:53 |
| ianw | dstufft: well not really, if i say "six>=1.9, cryptography", i'd expect in that order, not the other way around? | 00:53 |
| StevenK | anteaya: Because browsing git.openstack.org/cgit should be self-explanatory, rather than having to know the code name or the repository name | 00:55 |
| dstufft | ianw: we reverse the order, because we add things in the order we find them, so we find "cryprography" first, and we do CRAP_TO_INSTALL.append("cryptgoraphy"), then we donwload that, and inspect it and we find it depends on six, so we do CRAP_TO_INSTALL.append("six") | 00:56 |
| dstufft | once we have a big list of things to install, we reverse it | 00:56 |
| anteaya | StevenK: well the heat ones aren't very useful | 00:56 |
| dstufft | to try and install dependencies first | 00:57 |
| dstufft | it's not very smart though | 00:57 |
| anteaya | as they just reiterate the repo name | 00:57 |
| anteaya | a lot of the python clients to the same | 00:57 |
| *** radez is now known as radez_g0n3 | 00:58 | |
| StevenK | anteaya: "OpenStack Orchestration (Heat) Templates" is not useful? | 00:58 |
| anteaya | I have no objection to you adding descriptions I'm just not sure testing for it is the right approach | 00:58 |
| ianw | dstufft: so you're saying "cryptography" should be *above* "six" in http://git.openstack.org/cgit/openstack/keystone/tree/requirements.txt ? maybe i've been reading this in the wrong order :( | 00:58 |
| anteaya | openstack/heat-templatesheat-templates | 00:58 |
| *** claudiub has joined #openstack-infra | 00:59 | |
| anteaya | that is what I see when I browse git.o.o/cgit | 00:59 |
| openstackgerrit | Merged openstack-infra/project-config: Functional test job for Kolla https://review.openstack.org/158726 | 00:59 |
| *** timcline has quit IRC | 00:59 | |
| StevenK | anteaya: That's the default description | 00:59 |
| dstufft | ianw: um | 00:59 |
| StevenK | anteaya: Which is used when no description is set | 00:59 |
| *** wenlock has quit IRC | 00:59 | |
| dstufft | ianw: that might fix it | 00:59 |
| dstufft | "fix" | 00:59 |
| dstufft | where fix means do a terrible hack that might mask the problem | 00:59 |
| openstackgerrit | Merged openstack-infra/project-config: Run smoke test for stackforge/networking-odl jobs https://review.openstack.org/160148 | 01:00 |
| anteaya | like I said I have no objection to you adding descriptions, I just don't know what you are going to get if you test for the presence of them | 01:00 |
| anteaya | if you want to review for a good description that makes more sense to me | 01:00 |
| *** SumitNaiksatam has quit IRC | 01:01 | |
| StevenK | anteaya: That was AJaeger's suggestion -- testing for a description works, since all of the repos under openstack/ and so on now have one set. Which also means that anyone adding a new repo will be required to have a description | 01:01 |
| clarkb | nibalizer: fwiw we do use stdlib so I think its ok to use its run stages if it helps readability with those familar with stsdlib | 01:01 |
| *** rkukura has joined #openstack-infra | 01:02 | |
| ianw | dstufft: ok, so i'm a bit confused if this is a bug in pip/setuptools, or expected behavior of the limited dep solver. understand if you don't have time, maybe you could comment on that github issue when you do | 01:02 |
| dstufft | ianw: it's a bug in that it shouldn't happen | 01:02 |
| dstufft | it might be a bug that occurs due to the lack of a dep solver | 01:02 |
| anteaya | we don't test that an acl file has a -release group for pushSignedTag but we sure do review for it | 01:03 |
| anteaya | can you split the patch and have the test be a child patch | 01:03 |
| anteaya | then we can merge your descriptions | 01:03 |
| anteaya | as I agree with having them | 01:03 |
| *** MarkAtwood has quit IRC | 01:04 | |
| *** gyee has joined #openstack-infra | 01:04 | |
| *** bhunter71 has joined #openstack-infra | 01:04 | |
| *** mmedvede_ has quit IRC | 01:04 | |
| *** bhunter71 has quit IRC | 01:05 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 01:06 | |
| *** kiseok7 has quit IRC | 01:07 | |
| anteaya | jhesketh: did you want to monitor this one as it goes in? https://review.openstack.org/#/c/156252/ | 01:08 |
| *** annegentle has quit IRC | 01:08 | |
| anteaya | jhesketh: I didn't approve it, just reviewed it | 01:08 |
| *** markvoelker has joined #openstack-infra | 01:09 | |
| jhesketh | anteaya: I don't know if nodepool has been restarted with the change yet | 01:09 |
| clarkb | anteaya: thinking about that one a bit more I wonder if we should add a trailing || true to have it not fail jobs if nodepool ins't writing the uuids yet | 01:09 |
| anteaya | jhesketh: very good | 01:09 |
| clarkb | might be a nice thing for any downstream contributors | 01:09 |
| clarkb | jhesketh: ya ^ | 01:09 |
| anteaya | clarkb: having it not fail is good | 01:09 |
| clarkb | but I can check if our nodepool is uuid'ing yet | 01:10 |
| jhesketh | clarkb: ++ | 01:10 |
| *** kiseok7 has joined #openstack-infra | 01:10 | |
| anteaya | clarkb: did you want to add that comment? | 01:10 |
| clarkb | does not look like we are | 01:11 |
| clarkb | ya I can leave a comment | 01:11 |
| * anteaya pictures clarkb with green ears | 01:11 | |
| anteaya | thanks | 01:11 |
| *** crc32 has quit IRC | 01:11 | |
| *** jaypipes has quit IRC | 01:11 | |
| *** annegentle has joined #openstack-infra | 01:12 | |
| *** dimtruck is now known as zz_dimtruck | 01:13 | |
| clarkb | hrm the nested cat may effectively make it work that way | 01:13 |
| clarkb | oh wait its checking th efile exists derp | 01:13 |
| *** bhunter71 has joined #openstack-infra | 01:13 | |
| clarkb | ya I think we can put this in as is for now then restart nodepool to get the info | 01:13 |
| clarkb | its late in the day is my excuse | 01:13 |
| *** markvoelker has quit IRC | 01:14 | |
| *** yamamoto has joined #openstack-infra | 01:14 | |
| anteaya | the motivation was sound | 01:14 |
| *** rhsu has quit IRC | 01:15 | |
| clarkb | I can approve it once the ansible thins are done (almost, waiting for next ru nto confirm everythingstill works) | 01:15 |
| *** ryanpetrello has quit IRC | 01:15 | |
| *** kmartin has quit IRC | 01:15 | |
| *** dannywilson has quit IRC | 01:16 | |
| *** ryanpetrello has joined #openstack-infra | 01:16 | |
| clarkb | hrm looks like it may have failed | 01:16 |
| clarkb | I am going to run it manually | 01:17 |
| anteaya | approve when you are ready | 01:17 |
| *** dims_ has joined #openstack-infra | 01:17 | |
| *** sputnik13 has quit IRC | 01:17 | |
| *** annegentle has quit IRC | 01:19 | |
| *** dims has quit IRC | 01:19 | |
| *** salv-orlando has quit IRC | 01:20 | |
| *** yamamoto has quit IRC | 01:20 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Need to force ansible role installs https://review.openstack.org/161059 | 01:21 |
| clarkb | mordred: jeblair ^ | 01:21 |
| clarkb | jhesketh: pleia2 ^ as well | 01:21 |
| jhesketh | clarkb: +2 | 01:22 |
| clarkb | thank you | 01:22 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Add backup before running db migrations https://review.openstack.org/160514 | 01:22 |
| mtreinish | man I suck at writing puppet, already 7 revs for something stupid simple | 01:23 |
| *** hdd has quit IRC | 01:25 | |
| openstackgerrit | Merged openstack-infra/project-config: channel configuration for #openstack-ko https://review.openstack.org/161020 | 01:26 |
| *** tiswanso has joined #openstack-infra | 01:30 | |
| *** jerryz_ has joined #openstack-infra | 01:32 | |
| openstackgerrit | Randy Bertram proposed openstack/requirements: Update Magic Search version https://review.openstack.org/160661 | 01:32 |
| clarkb | I am running 161059 by hand to confirm a full run with it is happy if so I will self approve | 01:34 |
| clarkb | now I get error: /etc/ansible/roles/puppet doesn't appear to contain a role. | 01:35 |
| *** achanda has quit IRC | 01:35 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Preload Story and Creator in state https://review.openstack.org/161063 | 01:35 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Preload tasks in state resolution. https://review.openstack.org/161064 | 01:35 |
| openstackgerrit | greghaynes proposed openstack-infra/zuul: Add base class for sources https://review.openstack.org/119531 | 01:36 |
| openstackgerrit | greghaynes proposed openstack-infra/zuul: Add base class for reporters https://review.openstack.org/119530 | 01:36 |
| greghaynes | jhesketh: ^ | 01:36 |
| greghaynes | jhesketh: let me know what you think | 01:36 |
| clarkb | ok I manually moved the old puppet ansible role aside and that got the forced ansible galaxy install to work | 01:37 |
| clarkb | mordred: ^ you may want to look in on that later but I am going to move ahead if this runs puppet | 01:37 |
| *** cebruns has joined #openstack-infra | 01:38 | |
| *** otter768 has joined #openstack-infra | 01:39 | |
| pleia2 | mtreinish: me too, and this is supposed to be my job, don't tell anyone :) | 01:41 |
| clarkb | pleia2: can haz review on 161059? I can self approve it just looking for extra eyes | 01:41 |
| *** achanda has joined #openstack-infra | 01:41 | |
| *** tiswanso has quit IRC | 01:41 | |
| *** zz_dimtruck is now known as dimtruck | 01:42 | |
| pleia2 | clarkb: approved, doesn't seem to be related to later errors you get | 01:42 |
| clarkb | ya I have worked around those now by manually moving aside the old puppet role | 01:42 |
| clarkb | I think ansible-galaxy was expecting a meta dir with content to know about the role package stuff | 01:42 |
| *** baoli has quit IRC | 01:43 | |
| clarkb | all the old data is preserved so mordred can look at it later | 01:43 |
| *** claudiub has quit IRC | 01:43 | |
| timrc | mordred, Hi... out of curiosity where are nodepool TODOs items kept. I'm specifically looking for the one where someone made mention of rebuild as being a replacement for delete/create. | 01:45 |
| timrc | TODO* | 01:45 |
| pleia2 | timrc: some are in specs, or proposed as specs | 01:46 |
| timrc | pleia2, Okay. I'll look for proposed. I don't see anything obvious in the current nodepool specs. | 01:46 |
| pleia2 | so http://specs.openstack.org/openstack-infra/infra-specs/ and https://review.openstack.org/#/q/status:open+project:openstack-infra/infra-specs,n,z | 01:46 |
| * pleia2 nods | 01:46 | |
| pleia2 | I don't personally know about that particular one | 01:47 |
| *** ryanpetrello has quit IRC | 01:47 | |
| *** vilobhmm has quit IRC | 01:48 | |
| *** ashleighfarnham has quit IRC | 01:49 | |
| *** Sukhdev has quit IRC | 01:49 | |
| *** ddieterly has joined #openstack-infra | 01:49 | |
| clarkb | we should also probably ocnsider expanding our parallelism. I think the puppetmaster can handle >10 nodes at a time | 01:50 |
| *** sarob has quit IRC | 01:51 | |
| *** ddieterl_ has joined #openstack-infra | 01:51 | |
| openstackgerrit | Merged openstack-infra/system-config: Need to force ansible role installs https://review.openstack.org/161059 | 01:51 |
| *** camunoz has quit IRC | 01:51 | |
| *** ddieterly has quit IRC | 01:51 | |
| *** yamamoto has joined #openstack-infra | 01:52 | |
| openstackgerrit | Timothy R. Chavez proposed openstack-infra/shade: Add 'rebuild' to shade https://review.openstack.org/161067 | 01:52 |
| clarkb | timrc: any initial data on whether or not rebuids are faster? | 01:52 |
| openstackgerrit | Timothy R. Chavez proposed openstack-infra/shade: Add 'rebuild' to shade https://review.openstack.org/161067 | 01:53 |
| timrc | clarkb, I haven't done that quite yet :) I futzed around with adding tests for my change :P | 01:53 |
| timrc | unit tests* | 01:54 |
| timrc | I noticed they are... lacking... for shade. | 01:54 |
| *** tiswanso has joined #openstack-infra | 01:58 | |
| *** jamielennox is now known as jamielennox|away | 01:59 | |
| *** tiswanso has quit IRC | 01:59 | |
| *** reed has quit IRC | 02:00 | |
| *** tiswanso has joined #openstack-infra | 02:00 | |
| openstackgerrit | greghaynes proposed openstack-infra/zuul: Add base class for triggers https://review.openstack.org/119532 | 02:03 |
| openstackgerrit | greghaynes proposed openstack-infra/zuul: Add base class for sources https://review.openstack.org/119531 | 02:03 |
| *** camunoz has joined #openstack-infra | 02:03 | |
| *** MarkAtwood has joined #openstack-infra | 02:03 | |
| *** MarkAtwood has quit IRC | 02:03 | |
| clarkb | timrc: its a mordred project :) | 02:04 |
| *** ryanpetrello has joined #openstack-infra | 02:04 | |
| *** ChuckC has quit IRC | 02:04 | |
| *** mfink_ has quit IRC | 02:06 | |
| *** dmsimard_away is now known as dmsimard | 02:08 | |
| mtreinish | pleia2: heh, well I guess it's good to know it's not just me | 02:09 |
| bhunter71 | do you think the following indicates 0 changes in total, or just 0 changes needing re-indexing? | 02:10 |
| bhunter71 | root@review:~# sudo su - gerrit2 -c -- "java -jar review_site/bin/gerrit.war reindex -d /home/gerrit2/review_site" | 02:10 |
| bhunter71 | [2015-03-04 01:57:31,668] INFO com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 444m | 02:10 |
| bhunter71 | [2015-03-04 01:57:32,107] INFO com.google.gerrit.server.cache.h2.H2CacheFactory : Enabling disk cache /home/gerrit2/review_site/cache | 02:10 |
| bhunter71 | Collecting projects: 9 | 02:10 |
| *** markvoelker has joined #openstack-infra | 02:10 | |
| bhunter71 | Reindexing changes: projects: 100% (9/9), done | 02:10 |
| *** kiseok7 has quit IRC | 02:10 | |
| bhunter71 | Reindexed 0 changes in 0.5s (0.0/s) | 02:10 |
| bhunter71 | it doesn't look 'good' to me | 02:10 |
| *** dimtruck is now known as zz_dimtruck | 02:11 | |
| *** ryanpetrello has quit IRC | 02:11 | |
| *** ryanpetrello has joined #openstack-infra | 02:14 | |
| *** markvoelker has quit IRC | 02:15 | |
| *** patrickeast has quit IRC | 02:19 | |
| *** patrickeast__ has quit IRC | 02:20 | |
| *** patri728 has quit IRC | 02:20 | |
| *** patri586 has quit IRC | 02:20 | |
| *** patrickeast_ has quit IRC | 02:20 | |
| *** annegentle has joined #openstack-infra | 02:20 | |
| *** salv-orlando has joined #openstack-infra | 02:20 | |
| *** koesik has joined #openstack-infra | 02:22 | |
| *** ryanpetrello has quit IRC | 02:23 | |
| *** ryanpetrello has joined #openstack-infra | 02:24 | |
| *** rkukura has quit IRC | 02:24 | |
| *** ryanpetrello has quit IRC | 02:24 | |
| *** ryanpetrello has joined #openstack-infra | 02:25 | |
| *** achanda has quit IRC | 02:25 | |
| *** annegentle has quit IRC | 02:25 | |
| openstackgerrit | Timothy R. Chavez proposed openstack-infra/shade: Add 'rebuild' to shade https://review.openstack.org/161067 | 02:26 |
| *** ayoung has joined #openstack-infra | 02:27 | |
| *** markvoelker has joined #openstack-infra | 02:27 | |
| *** unicell has joined #openstack-infra | 02:30 | |
| *** stevemar has joined #openstack-infra | 02:30 | |
| *** markvoelker has quit IRC | 02:33 | |
| *** zz_dimtruck is now known as dimtruck | 02:33 | |
| *** achanda has joined #openstack-infra | 02:33 | |
| *** stevemar has quit IRC | 02:35 | |
| *** stevemar has joined #openstack-infra | 02:36 | |
| *** jamielennox|away is now known as jamielennox | 02:37 | |
| *** baoli has joined #openstack-infra | 02:38 | |
| *** weshay has quit IRC | 02:39 | |
| *** erikwilson has joined #openstack-infra | 02:39 | |
| *** erikmwilson has quit IRC | 02:39 | |
| *** erikwilson is now known as erikmwilson | 02:39 | |
| *** erikwilson has joined #openstack-infra | 02:39 | |
| *** erikmwilson has quit IRC | 02:39 | |
| *** erikwilson is now known as erikmwilson_ | 02:40 | |
| *** Longgeek has joined #openstack-infra | 02:47 | |
| *** baoli has quit IRC | 02:48 | |
| *** marun has quit IRC | 02:49 | |
| *** ChuckC has joined #openstack-infra | 02:49 | |
| *** ryanpetrello has quit IRC | 02:51 | |
| *** jamielennox is now known as jamielennox|away | 02:52 | |
| *** ChuckC has quit IRC | 02:54 | |
| *** ChuckC has joined #openstack-infra | 02:54 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: skip rerunning requirements changes in devstack https://review.openstack.org/156214 | 02:54 |
| *** jamielennox|away is now known as jamielennox | 02:55 | |
| *** jamielennox is now known as jamielennox|away | 02:56 | |
| *** jamielennox|away is now known as jamielennox | 02:57 | |
| *** hdd_ has joined #openstack-infra | 03:00 | |
| *** achanda has quit IRC | 03:05 | |
| *** annegentle has joined #openstack-infra | 03:05 | |
| *** ChuckC has quit IRC | 03:06 | |
| *** ChuckC has joined #openstack-infra | 03:06 | |
| *** annegentle has quit IRC | 03:07 | |
| *** BharatK has joined #openstack-infra | 03:09 | |
| *** achanda has joined #openstack-infra | 03:10 | |
| *** baoli has joined #openstack-infra | 03:11 | |
| *** dimtruck is now known as zz_dimtruck | 03:13 | |
| *** asettle is now known as asettle-fooz | 03:14 | |
| *** asettle-fooz is now known as asettle-foodz | 03:14 | |
| *** ivar-lazzaro has quit IRC | 03:14 | |
| *** baoli has quit IRC | 03:14 | |
| *** erikmwilson_ is now known as erikmwilson | 03:15 | |
| *** timcline has joined #openstack-infra | 03:16 | |
| *** timcline has quit IRC | 03:17 | |
| *** dims_ has quit IRC | 03:17 | |
| *** Ryan_Lane has quit IRC | 03:17 | |
| *** timcline has joined #openstack-infra | 03:17 | |
| *** harlowja_ is now known as harlowja_away | 03:21 | |
| *** salv-orlando has quit IRC | 03:21 | |
| *** markvoelker has joined #openstack-infra | 03:28 | |
| openstackgerrit | Diego proposed openstack-infra/infra-manual: Add details on using git-review over https https://review.openstack.org/161078 | 03:28 |
| *** gyee has quit IRC | 03:29 | |
| *** yamahata has quit IRC | 03:29 | |
| *** achanda has quit IRC | 03:30 | |
| *** tsg has joined #openstack-infra | 03:30 | |
| *** camunoz has quit IRC | 03:30 | |
| *** patrickeast has joined #openstack-infra | 03:31 | |
| *** patrickeast has quit IRC | 03:33 | |
| *** markvoelker has quit IRC | 03:33 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:33 | |
| openstackgerrit | Zhi Yan Liu proposed openstack/requirements: Bump glance_store to 0.1.11 https://review.openstack.org/161079 | 03:34 |
| *** asettle-foodz is now known as asettle | 03:38 | |
| *** unicell has quit IRC | 03:40 | |
| *** jyuso1 has joined #openstack-infra | 03:43 | |
| *** ryanpetrello has joined #openstack-infra | 03:44 | |
| *** jpeeler has joined #openstack-infra | 03:45 | |
| *** camunoz has joined #openstack-infra | 03:47 | |
| *** changbl has joined #openstack-infra | 03:48 | |
| *** ryanpetrello has quit IRC | 03:49 | |
| *** ryanpetrello has joined #openstack-infra | 03:49 | |
| *** ddieterl_ has quit IRC | 03:51 | |
| *** zz_dimtruck is now known as dimtruck | 03:52 | |
| *** timcline has quit IRC | 03:54 | |
| *** achanda has joined #openstack-infra | 03:55 | |
| *** vilobhmm has joined #openstack-infra | 03:57 | |
| *** bhunter71 has quit IRC | 04:00 | |
| *** Clint has quit IRC | 04:01 | |
| *** nijaba has quit IRC | 04:01 | |
| *** ryanpetrello has quit IRC | 04:02 | |
| *** spzala has quit IRC | 04:03 | |
| *** nijaba has joined #openstack-infra | 04:06 | |
| *** ryanpetrello has joined #openstack-infra | 04:06 | |
| *** dhritishikhar has joined #openstack-infra | 04:09 | |
| *** tiswanso has quit IRC | 04:10 | |
| *** ryanpetrello has quit IRC | 04:12 | |
| *** ryanpetrello has joined #openstack-infra | 04:13 | |
| *** dannywilson has joined #openstack-infra | 04:13 | |
| greghaynes | clarkb: zuul race? http://logs.openstack.org/31/119531/16/check/gate-zuul-python26/cb67b71/console.html | 04:13 |
| greghaynes | Curious if your patch to fix race issues is for that one | 04:14 |
| clarkb | no but its likely a similar issue if you want to compare to my fix | 04:15 |
| greghaynes | fun | 04:15 |
| clarkb | I should just comb through all the tests for that issue | 04:16 |
| BharatK | clarkb: Hi, please review this https://review.openstack.org/160664 when you have some time | 04:16 |
| *** dannywilson has quit IRC | 04:17 | |
| *** Clint has joined #openstack-infra | 04:18 | |
| *** Clint has joined #openstack-infra | 04:18 | |
| *** ryanpetrello has quit IRC | 04:18 | |
| *** adam_g has quit IRC | 04:20 | |
| *** unicell has joined #openstack-infra | 04:20 | |
| *** stevebaker has quit IRC | 04:21 | |
| openstackgerrit | Diego proposed openstack-infra/git-review: Automatically use port 443/80 for http(s) reviews https://review.openstack.org/161090 | 04:22 |
| *** hdd_ has quit IRC | 04:23 | |
| *** hughsaunders has quit IRC | 04:24 | |
| openstackgerrit | Diego proposed openstack-infra/git-review: Automatically use port 443/80 for http(s) reviews https://review.openstack.org/161090 | 04:26 |
| *** adam_g has joined #openstack-infra | 04:28 | |
| *** adam_g has quit IRC | 04:28 | |
| *** adam_g has joined #openstack-infra | 04:28 | |
| *** Alexandra_ has joined #openstack-infra | 04:28 | |
| *** markvoelker has joined #openstack-infra | 04:29 | |
| *** dmsimard is now known as dmsimard_away | 04:29 | |
| *** alaski has quit IRC | 04:30 | |
| *** mgagne has quit IRC | 04:31 | |
| *** stevebaker has joined #openstack-infra | 04:31 | |
| *** otter768 has quit IRC | 04:31 | |
| *** otter768 has joined #openstack-infra | 04:33 | |
| *** markvoelker has quit IRC | 04:34 | |
| *** hughsaunders has joined #openstack-infra | 04:35 | |
| *** otter768 has quit IRC | 04:42 | |
| *** vilobhmm has quit IRC | 04:44 | |
| *** dimtruck is now known as zz_dimtruck | 04:44 | |
| *** mgagne has joined #openstack-infra | 04:45 | |
| *** mgagne is now known as Guest75711 | 04:45 | |
| *** openstackgerrit has quit IRC | 04:46 | |
| *** dhritishikhar has quit IRC | 04:49 | |
| *** ddieterly has joined #openstack-infra | 04:52 | |
| *** openstackgerrit has joined #openstack-infra | 04:52 | |
| *** rkukura has joined #openstack-infra | 04:54 | |
| *** ddieterly has quit IRC | 04:56 | |
| *** hdd_ has joined #openstack-infra | 04:58 | |
| *** comstud has quit IRC | 05:00 | |
| *** trinaths has joined #openstack-infra | 05:02 | |
| clarkb | if anyone notices that zuul is cranky you can blame reboots | 05:02 |
| clarkb | I'm not going to try and fix tonight as I don't know how many to expect or when they are complete | 05:03 |
| clarkb | easiest to just pick up the peices in the morning | 05:03 |
| *** Alexandra_ has quit IRC | 05:04 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/infra-manual: Add details on using git-review over https https://review.openstack.org/161078 | 05:06 |
| *** rkukura has quit IRC | 05:09 | |
| clarkb | I have got things moving for now at least but no promises it will survive overnight | 05:09 |
| *** koolhead17 has joined #openstack-infra | 05:09 | |
| *** koolhead17 has joined #openstack-infra | 05:09 | |
| *** garyh has quit IRC | 05:09 | |
| *** vilobhmm has joined #openstack-infra | 05:11 | |
| *** vilobhmm1 has joined #openstack-infra | 05:16 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Add reference to IRC services document https://review.openstack.org/159740 | 05:17 |
| *** vilobhmm has quit IRC | 05:19 | |
| *** yamamoto_ has joined #openstack-infra | 05:20 | |
| *** yamamoto has quit IRC | 05:20 | |
| *** ddieterly has joined #openstack-infra | 05:22 | |
| openstackgerrit | Merged openstack-infra/system-config: Add openstack-ko to eavesdrop https://review.openstack.org/160631 | 05:24 |
| *** ddieterly has quit IRC | 05:27 | |
| *** Alexandra_ has joined #openstack-infra | 05:27 | |
| *** achanda has quit IRC | 05:28 | |
| *** Alexandra_ has quit IRC | 05:28 | |
| *** Longgeek has quit IRC | 05:28 | |
| *** BharatK has quit IRC | 05:28 | |
| *** Alexandra_ has joined #openstack-infra | 05:29 | |
| *** woodster_ has quit IRC | 05:30 | |
| *** markvoelker has joined #openstack-infra | 05:31 | |
| *** ujuc has quit IRC | 05:31 | |
| *** jamielennox is now known as jamielennox|away | 05:32 | |
| *** ryanpetrello has joined #openstack-infra | 05:32 | |
| *** ujuc has joined #openstack-infra | 05:33 | |
| *** unicell1 has joined #openstack-infra | 05:33 | |
| *** unicell has quit IRC | 05:33 | |
| *** dhritishikhar has joined #openstack-infra | 05:34 | |
| *** openstack has joined #openstack-infra | 05:36 | |
| *** salv-orlando has joined #openstack-infra | 05:38 | |
| *** esker has joined #openstack-infra | 05:40 | |
| *** esker has quit IRC | 05:41 | |
| *** amotoki has joined #openstack-infra | 05:43 | |
| *** Alexandra_ has quit IRC | 05:43 | |
| *** BharatK has joined #openstack-infra | 05:45 | |
| *** bhunter71 has joined #openstack-infra | 05:47 | |
| *** sputnik13 has joined #openstack-infra | 05:56 | |
| *** ryanpetrello has joined #openstack-infra | 05:57 | |
| *** jbryce has quit IRC | 05:59 | |
| *** ujuc has quit IRC | 06:01 | |
| *** ryanpetrello has quit IRC | 06:02 | |
| *** camunoz has quit IRC | 06:02 | |
| *** vilobhmm1 has quit IRC | 06:07 | |
| *** Longgeek has joined #openstack-infra | 06:08 | |
| *** lbragstad has quit IRC | 06:08 | |
| *** roaet has quit IRC | 06:09 | |
| *** phschwartz has quit IRC | 06:09 | |
| openstackgerrit | Gauvain Pocentek proposed openstack-infra/system-config: Add #cloudkitty to the list of eavesdrop channels https://review.openstack.org/161106 | 06:09 |
| *** eharney has quit IRC | 06:10 | |
| *** lbragstad has joined #openstack-infra | 06:10 | |
| *** eharney has joined #openstack-infra | 06:10 | |
| *** gondoi has quit IRC | 06:10 | |
| *** roaet has joined #openstack-infra | 06:10 | |
| *** woodster_ has joined #openstack-infra | 06:10 | |
| *** zz_gondoi has joined #openstack-infra | 06:11 | |
| *** zz_gondoi is now known as gondoi | 06:11 | |
| *** phschwartz has joined #openstack-infra | 06:11 | |
| *** camunoz has joined #openstack-infra | 06:15 | |
| *** achanda has joined #openstack-infra | 06:16 | |
| *** hdd_ has quit IRC | 06:17 | |
| *** JayF has quit IRC | 06:17 | |
| *** pcrews has quit IRC | 06:19 | |
| *** yfried has quit IRC | 06:19 | |
| *** lbragstad has quit IRC | 06:19 | |
| *** mrmartin has joined #openstack-infra | 06:21 | |
| *** lbragstad has joined #openstack-infra | 06:22 | |
| *** ddieterly has joined #openstack-infra | 06:23 | |
| *** penguinRaider has quit IRC | 06:24 | |
| *** penguinRaider has joined #openstack-infra | 06:25 | |
| *** ryanpetrello has joined #openstack-infra | 06:26 | |
| *** tnovacik has joined #openstack-infra | 06:26 | |
| *** gondoi has quit IRC | 06:27 | |
| *** ddieterly has quit IRC | 06:27 | |
| *** phschwartz has quit IRC | 06:27 | |
| *** sputnik13 has quit IRC | 06:29 | |
| *** salv-orlando has quit IRC | 06:29 | |
| *** markvoelker has joined #openstack-infra | 06:32 | |
| *** cnesa has joined #openstack-infra | 06:36 | |
| *** punith has joined #openstack-infra | 06:36 | |
| *** markvoelker has quit IRC | 06:38 | |
| *** otter768 has joined #openstack-infra | 06:43 | |
| *** camunoz has quit IRC | 06:45 | |
| *** cnesa has quit IRC | 06:47 | |
| *** ryanpetrello has quit IRC | 06:47 | |
| *** otter768 has quit IRC | 06:48 | |
| *** asadoughi has quit IRC | 06:51 | |
| *** jogo has quit IRC | 06:53 | |
| *** bhunter71 has quit IRC | 06:53 | |
| *** yfried has joined #openstack-infra | 06:55 | |
| BharatK | https://review.openstack.org/#/c/160664/ | 07:00 |
| BharatK | I got Code-Review+2 Workflow+1 to this patch | 07:00 |
| BharatK | Waiting for Jenkins approval | 07:01 |
| BharatK | But I waiting since 3 hours, and I am not able to find this integration job in http://status.openstack.org/zuul/ | 07:01 |
| *** HeOS has quit IRC | 07:01 | |
| AJaeger | BharatK, that is strange, let me recheck... | 07:03 |
| BharatK | AJaeger: yeah... thank you | 07:04 |
| notmyname | looks like the test jobs are queued for that patch | 07:04 |
| *** ryanpetrello has joined #openstack-infra | 07:04 | |
| *** deepakcs has joined #openstack-infra | 07:05 | |
| BharatK | AJaeger: now it is running | 07:05 |
| BharatK | AJaeger: thank you | 07:05 |
| *** dhritishikhar has quit IRC | 07:06 | |
| BharatK | notmyname: I think we can see queued jobs also in http://status.openstack.org/zuul/ | 07:07 |
| *** scheuran has joined #openstack-infra | 07:08 | |
| *** lbragstad has quit IRC | 07:08 | |
| *** ryanpetrello has quit IRC | 07:08 | |
| *** pcrews has joined #openstack-infra | 07:09 | |
| BharatK | AJaeger: After giving recheck job created in check pipeline | 07:12 |
| AJaeger | yeah | 07:12 |
| BharatK | AJaeger: I think after workflow approval it should be in integrated right? | 07:12 |
| AJaeger | BharatK, correct | 07:13 |
| AJaeger | BharatK, clarkb mentioned that Zuul is cranky, so this is a result. Sorry for that | 07:13 |
| BharatK | AJaeger: Not a problem. I think giving one more +1 to work will run gate job I guess? | 07:15 |
| BharatK | AJaeger: * +1 to workflow | 07:15 |
| AJaeger | BharatK, it will go into the gate automatically. | 07:15 |
| BharatK | AJaeger: okay | 07:15 |
| AJaeger | But will take it's time since our CI is really busy right now | 07:15 |
| timrc | clarkb, jeblair, mordred, whomever else is interested: Whipped up a little program to time server recreates and rebuilds using same and different images. Results can be found here: http://paste.openstack.org/show/187334/, benchmark code can be found here: http://paste.openstack.org/show/187333/, and the shade review with the rebuild change can be found here: https://review.openstack.org/#/c/161067/ | 07:16 |
| BharatK | AJaeger: okay | 07:17 |
| timrc | We'd probably want bigger sample sizes then erm 1, but it does show a pretty clear difference in time between recreates and rebuilds. | 07:17 |
| *** roaet has quit IRC | 07:17 | |
| timrc | yolanda, ^^ | 07:17 |
| *** ildikov has quit IRC | 07:18 | |
| yolanda | nice, timrc, that's useful | 07:18 |
| yolanda | i'll go with reviews in a while | 07:18 |
| *** Longgeek has quit IRC | 07:21 | |
| *** yfried is now known as yfried|afk | 07:23 | |
| *** ddieterly has joined #openstack-infra | 07:24 | |
| *** yfried|afk is now known as yfried | 07:26 | |
| *** roaet has joined #openstack-infra | 07:26 | |
| *** dhritishikhar has joined #openstack-infra | 07:27 | |
| *** ghostpl_ has joined #openstack-infra | 07:28 | |
| *** ddieterly has quit IRC | 07:28 | |
| *** rkukura has joined #openstack-infra | 07:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Add ironic-lib to openstack https://review.openstack.org/157636 | 07:30 |
| *** dhritishikhar has quit IRC | 07:32 | |
| *** david-lyle_afk has joined #openstack-infra | 07:33 | |
| *** Longgeek has joined #openstack-infra | 07:34 | |
| *** roaet has quit IRC | 07:34 | |
| *** markvoelker has joined #openstack-infra | 07:34 | |
| *** vigneshvar has joined #openstack-infra | 07:34 | |
| *** roaet has joined #openstack-infra | 07:35 | |
| *** mpaolino has joined #openstack-infra | 07:38 | |
| *** markvoelker has quit IRC | 07:39 | |
| *** jcoufal has joined #openstack-infra | 07:41 | |
| *** devvesa has joined #openstack-infra | 07:43 | |
| *** mpaolino has quit IRC | 07:46 | |
| *** mpaolino has joined #openstack-infra | 07:47 | |
| *** afazekas has quit IRC | 07:48 | |
| *** doude has joined #openstack-infra | 07:49 | |
| *** openstackgerrit has quit IRC | 07:49 | |
| *** openstackgerrit has joined #openstack-infra | 07:49 | |
| *** rkukura has quit IRC | 07:52 | |
| *** mpavone has joined #openstack-infra | 07:54 | |
| *** mrunge has joined #openstack-infra | 07:56 | |
| *** amotoki has quit IRC | 07:57 | |
| *** amotoki has joined #openstack-infra | 07:57 | |
| AJaeger | clarkb, the check queue is not moving forward according to the status page | 07:59 |
| AJaeger | BharatK, hope your job is through tomorrow, please have some patience | 07:59 |
| *** mfink_ has joined #openstack-infra | 08:02 | |
| *** markus_z has joined #openstack-infra | 08:03 | |
| *** pabelanger has quit IRC | 08:06 | |
| *** psedlak has quit IRC | 08:06 | |
| *** mfink_ has quit IRC | 08:07 | |
| *** pabelanger has joined #openstack-infra | 08:08 | |
| *** shardy has joined #openstack-infra | 08:12 | |
| *** achanda has quit IRC | 08:12 | |
| *** sergsh has joined #openstack-infra | 08:12 | |
| *** devvesa has quit IRC | 08:14 | |
| *** mpaolino has quit IRC | 08:15 | |
| ttx | anteaya: FWIW I agree with jeblair's interpretation: ATCs (=voters) are the contributors which also happen to be Foundation members. | 08:15 |
| *** salv-orlando has joined #openstack-infra | 08:15 | |
| *** mpaolino has joined #openstack-infra | 08:15 | |
| ttx | The challenge always has been to check if contributors were actually members. | 08:16 |
| ttx | since they need to be both in order to be ATC (and vote) | 08:16 |
| AJaeger | ttx, FYI: We have a problem with reboot and Zuul is having problems, the check queue seems stuck right now ;( | 08:16 |
| *** sergsh has quit IRC | 08:17 | |
| AJaeger | with reboot: I understood clarkb to mean reboots of external cloud infrastructure | 08:17 |
| ttx | AJaeger: ack -- I think RAX had planned maintenance windows this week | 08:17 |
| AJaeger | That would explain it... | 08:18 |
| * AJaeger is going to send an email as service announcement... | 08:18 | |
| *** mpaolino has quit IRC | 08:18 | |
| ttx | anteaya: fully agree that the manual process won't work in a 20-30-large tent. We could use tooling to handle self-nomination / checking / listing of candidates instead of the ML | 08:19 |
| *** mpaolino has joined #openstack-infra | 08:19 | |
| ttx | AJaeger: let me quickcheck my facts | 08:19 |
| AJaeger | ttx, please do | 08:19 |
| *** chlong has quit IRC | 08:20 | |
| *** woodster_ has quit IRC | 08:20 | |
| *** e0ne has joined #openstack-infra | 08:20 | |
| *** mpaolino has quit IRC | 08:21 | |
| ttx | https://status.rackspace.com/ says there will be a sliding maintenance window but doesn't really say precisely when | 08:21 |
| *** mpaolino has joined #openstack-infra | 08:21 | |
| ttx | But I bet that is it | 08:21 |
| ttx | DFW: Tuesday, March 3rd 22:00 - Wednesday, March 4th 06:00 CST | 08:22 |
| *** devvesa has joined #openstack-infra | 08:22 | |
| ttx | AJaeger: ^ | 08:23 |
| *** mpaolino has quit IRC | 08:23 | |
| ttx | #link https://community.rackspace.com/general/f/53/p/4978/11947#11947 | 08:23 |
| *** mpaolino has joined #openstack-infra | 08:23 | |
| *** ddieterly has joined #openstack-infra | 08:25 | |
| AJaeger | tttx, thanks | 08:25 |
| AJaeger | ttx, what do you think of this email: http://paste.openstack.org/show/187392/ | 08:25 |
| ttx | sounds good :) | 08:26 |
| * ttx could try sticking a statusbot warning | 08:26 | |
| AJaeger | Ok, I'll send it out... | 08:26 |
| AJaeger | ttx, if you have the power to do so, it would be great! | 08:26 |
| *** yamahata has joined #openstack-infra | 08:26 | |
| ttx | #status alert Zuul check queue stuck due to reboot maintenance window at one of our cloud providers - no need to recheck changes at the moment, they won't move forward. | 08:28 |
| *** mpaolino has quit IRC | 08:28 | |
| openstackstatus | ttx: sending alert | 08:28 |
| *** mpaolino has joined #openstack-infra | 08:29 | |
| *** ddieterly has quit IRC | 08:29 | |
| *** Guest11742 has quit IRC | 08:29 | |
| AJaeger | is the bot affected by the reboot as well? ;( | 08:29 |
| ttx | heh | 08:29 |
| AJaeger | or just slow? | 08:29 |
| ttx | slow | 08:30 |
| AJaeger | slow - just got it on #openstack-doc | 08:30 |
| AJaeger | thanks, ttx | 08:30 |
| -openstackstatus- NOTICE: Zuul check queue stuck due to reboot maintenance window at one of our cloud providers - no need to recheck changes at the moment, they won't move forward. | 08:30 | |
| *** ChanServ changes topic to "Zuul check queue stuck due to reboot maintenance window at one of our cloud providers - no need to recheck changes at the moment, they won't move forward." | 08:30 | |
| *** vigneshvar has quit IRC | 08:30 | |
| *** trinaths has left #openstack-infra | 08:30 | |
| ttx | AJaeger: wed just need to remember to "#status ok" when things are back to normal | 08:31 |
| *** sergsh has joined #openstack-infra | 08:31 | |
| *** mpaolino has quit IRC | 08:32 | |
| *** mpaolino has joined #openstack-infra | 08:32 | |
| openstackstatus | ttx: finished sending alert | 08:32 |
| AJaeger | ttx, put a stick it up ;) | 08:32 |
| AJaeger | post-it I mean | 08:32 |
| *** psedlak has joined #openstack-infra | 08:34 | |
| *** arxcruz has joined #openstack-infra | 08:34 | |
| *** markvoelker has joined #openstack-infra | 08:35 | |
| *** yfried is now known as yfried|afk | 08:36 | |
| *** yfried|afk is now known as yfried | 08:36 | |
| GheRivero | morning all | 08:36 |
| *** macjack has quit IRC | 08:36 | |
| *** yamamoto_ has quit IRC | 08:38 | |
| *** markvoelker has quit IRC | 08:39 | |
| *** Hal has joined #openstack-infra | 08:41 | |
| *** e0ne has quit IRC | 08:42 | |
| *** _afezekas|pub has joined #openstack-infra | 08:42 | |
| *** Hal is now known as Guest45753 | 08:42 | |
| *** HeOS has joined #openstack-infra | 08:43 | |
| *** otter768 has joined #openstack-infra | 08:44 | |
| *** vigneshvar has joined #openstack-infra | 08:45 | |
| *** salv-orlando has quit IRC | 08:45 | |
| *** salv-orlando has joined #openstack-infra | 08:45 | |
| *** ujuc has joined #openstack-infra | 08:47 | |
| *** otter768 has quit IRC | 08:49 | |
| *** mpaolino has quit IRC | 08:50 | |
| *** jistr has joined #openstack-infra | 08:50 | |
| *** mpaolino has joined #openstack-infra | 08:50 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-cgit: Add configurable Timeout settings on git vhost https://review.openstack.org/160871 | 08:52 |
| *** achanda has joined #openstack-infra | 08:52 | |
| *** jedimike has joined #openstack-infra | 08:52 | |
| *** hichihara has quit IRC | 08:56 | |
| *** Guest45753 has quit IRC | 08:58 | |
| *** mpaolino has quit IRC | 09:00 | |
| *** mpaolino has joined #openstack-infra | 09:01 | |
| *** yfried is now known as yfried|afk | 09:01 | |
| *** macjack has joined #openstack-infra | 09:02 | |
| *** nibalizer has quit IRC | 09:02 | |
| *** yfried|afk is now known as yfried | 09:03 | |
| *** rainya has quit IRC | 09:03 | |
| *** mpaolino has quit IRC | 09:05 | |
| *** mpaolino has joined #openstack-infra | 09:06 | |
| DuncanT | Looks like wiki is out. I assume because of the rackspace maintenance work, but just a heads up | 09:06 |
| *** openstack has joined #openstack-infra | 15:28 | |
| jeblair | openstack: welcome back | 15:29 |
| *** hdd_ has joined #openstack-infra | 15:29 | |
| * mordred hands openstack a fluffy llama | 15:29 | |
| dims | yep, i can see it. thanks | 15:29 |
| dims | haha | 15:29 |
| anteaya | 6 and a half hours missing of channel log | 15:30 |
| sigmavirus24 | mordred: http://i.imgur.com/WVF70ya.gif | 15:30 |
| anteaya | it runs like a deer | 15:30 |
| anteaya | I had picutred it like a horse | 15:31 |
| mordred | who knew? | 15:31 |
| * mordred has a new knowledge now | 15:31 | |
| *** unicell has joined #openstack-infra | 15:31 | |
| anteaya | me too | 15:31 |
| anteaya | also I had deer on my lawn not long ago, but I didn't mention it before | 15:32 |
| *** unicell1 has quit IRC | 15:32 | |
| anteaya | y'know busy | 15:32 |
| anteaya | so elastic-recheck, how is it? | 15:32 |
| *** armax has joined #openstack-infra | 15:33 | |
| *** zz_dimtruck is now known as dimtruck | 15:34 | |
| anteaya | I also have no idea what kdc01 is but it looks like it is offline | 15:36 |
| jeblair | anteaya: i will check on the logstash system | 15:37 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Add openstack/developer-reference repository https://review.openstack.org/161015 | 15:38 |
| anteaya | jeblair: k | 15:39 |
| BobBall | Any sensible way to inject an API key into a node when running the jobs? I want to upload to swift directly from the job, but need to upload one-at-a-time to get the content-encoding right, so can't use tempurls and the formpost middleware which the current swift uploader uses... | 15:39 |
| jeblair | jenkins-log-client on logstash.o.o was not running | 15:39 |
| *** vhoward has joined #openstack-infra | 15:40 | |
| anteaya | I see if I can clone from all the git backends | 15:40 |
| jeblair | BobBall: are you using os-log-analyze to serve the logs from swift? | 15:41 |
| *** yfried has joined #openstack-infra | 15:41 | |
| BobBall | No - I tried to use zuul-swift-upload-logs-with-console | 15:42 |
| BobBall | But that (quite understandably) uses tempurl with formpost middleware to bulk-upload | 15:42 |
| BobBall | but formpost middleware doesn't support setting the content-encoding so file.txt.gz is a gzip binary rather than a txt file with gzip encoding so you have to download it | 15:43 |
| *** wenlock has joined #openstack-infra | 15:43 | |
| *** asadoughi has joined #openstack-infra | 15:43 | |
| mordred | awesome | 15:43 |
| *** mattoliverau has quit IRC | 15:43 | |
| *** mrmartin has quit IRC | 15:44 | |
| BobBall | I do have serving from swift working nicely, but using pyrax to upload (https://github.com/citrix-openstack/openstack-citrix-ci/blob/master/osci/swift_upload.py) but that needs the swift API key | 15:44 |
| anteaya | I ran this command `git clone git://git05.openstack.org:29418/openstack-infra/gerritlib` for git 01-05 and checked the git log on all clones, they worked and matched | 15:44 |
| BobBall | so I was trying to find a safe way to inject that into the nodes | 15:45 |
| jeblair | BobBall: we intentionally don't do that because we don't trust the jobs with real credentials | 15:45 |
| *** Sukhdev has joined #openstack-infra | 15:46 | |
| BobBall | safe(r) of course | 15:46 |
| BobBall | Understandable. Can I pull them back from the node then run the upload from Jenkins itself? I wouldn't know how to begin to do that since everything seems to be node-centered... | 15:47 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: Import release notes script from oslo-incubator https://review.openstack.org/161265 | 15:47 |
| *** michchap has quit IRC | 15:47 | |
| *** tnovacik has quit IRC | 15:47 | |
| *** devvesa has joined #openstack-infra | 15:48 | |
| anteaya | looks like this server doesn't exist anymore: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=11 | 15:48 |
| cmluciano | can anyone point me to the correct RDO kilo mirror? | 15:48 |
| jeblair | BobBall: can we look into why tempurl isn't setting the content type correctly? | 15:48 |
| *** nelsnelson has joined #openstack-infra | 15:48 | |
| anteaya | that was ci-puppetmaster.o.o | 15:49 |
| mordred | cmluciano: we don't really have anything to do with RDO in here, sorry | 15:49 |
| BobBall | encoding type, not content type. It just doesn't pass any additional headers through AFAIK | 15:49 |
| *** michchap has joined #openstack-infra | 15:49 | |
| cmluciano | mordred: Ah ok, thanks | 15:49 |
| jeblair | BobBall: that seems like a bug though, yeah? | 15:49 |
| mordred | notmyname: I hear you know things about swift ^^ is there any good way to deal with content type via formpost middleware? | 15:49 |
| BobBall | Arguably a bug, arguably a new feature :) | 15:50 |
| mordred | BobBall: :) | 15:50 |
| *** dustins_ has quit IRC | 15:50 | |
| jeblair | encoding, i'm told :) | 15:50 |
| jeblair | clarkb, mordred: remember how many log worker processes we ran on each log worker node? they don't seem to be set to start on boot... | 15:51 |
| anteaya | can someone check git05 server? the eth1 traffic is non-existant, is this expected? http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=125 | 15:51 |
| anteaya | it is different from git01-04 | 15:51 |
| *** dkranz has quit IRC | 15:51 | |
| mordred | jeblair: I do not | 15:52 |
| jeblair | mordred: ^ can you check that while i finish up logs? | 15:52 |
| mordred | yup | 15:52 |
| jeblair | clarkb, mordred: i'm starting 3 for no | 15:52 |
| jeblair | now | 15:52 |
| *** rkukura has joined #openstack-infra | 15:53 | |
| mordred | anteaya: it seems to be serving things | 15:53 |
| anteaya | mordred: okey dokey | 15:53 |
| mordred | anteaya: otoh - why do _any_ of the nodes have eth1 traffic | 15:53 |
| anteaya | I could clone from it, but the graph looked odd to me | 15:54 |
| mordred | that's the private interface - are we doing somethign with that? | 15:54 |
| clarkb | morning | 15:54 |
| jeblair | mordred: does the lb communicate with them on that? | 15:54 |
| mordred | it's a clarkb! | 15:54 |
| anteaya | morning clarkb | 15:54 |
| clarkb | jeblair 3 is the safe number | 15:54 |
| anteaya | clarkb: grab your kevlar | 15:54 |
| jeblair | clarkb: yay i guessed right! | 15:54 |
| *** ChuckC_ has joined #openstack-infra | 15:54 | |
| clarkb | jeblair not set to start on boot because the safe number wasnt really knowm when we added them. we should change that now. I can write the changes | 15:55 |
| jeblair | cool | 15:55 |
| *** rainya has joined #openstack-infra | 15:55 | |
| *** esker has quit IRC | 15:55 | |
| mordred | jeblair: nope | 15:55 |
| mordred | jeblair: they're configured to use public ips | 15:55 |
| jeblair | clarkb: i haven't check the es cluster health yet; want to check on that? | 15:55 |
| *** deepakcs has quit IRC | 15:56 | |
| *** lbragstad has joined #openstack-infra | 15:56 | |
| anteaya | clarkb: fwiw the zuul status graph says it has no life | 15:56 |
| clarkb | jeblair: sure | 15:56 |
| *** bswartz has joined #openstack-infra | 15:57 | |
| clarkb | "status" : "green", and "number_of_data_nodes" : 6, which looks good | 15:57 |
| clarkb | info from `curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'` on any of the nodes | 15:57 |
| *** dustins has joined #openstack-infra | 15:57 | |
| *** Ryan_Lane has joined #openstack-infra | 15:57 | |
| clarkb | er any of the esXX nodes | 15:57 |
| clarkb | anteaya: and yes this was fully expected to be a fun morning when I saw the rax announcement | 15:58 |
| AJaeger | jeblair, once everything is working again, please send a status ok to the statusbot on IRC and reply to my email and state that everything is fine again. I'm leaving now... | 15:58 |
| *** kmartin has joined #openstack-infra | 15:58 | |
| anteaya | ah life on the graph, it was probably flatlined as it had nothing to index | 15:58 |
| anteaya | clarkb: ah yes, you came prepared | 15:58 |
| *** phschwartz has joined #openstack-infra | 15:58 | |
| *** ihrachyshka has quit IRC | 15:58 | |
| anteaya | AJaeger: thanks for the email and help with the channel status | 15:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add note about checking ES cluster health https://review.openstack.org/161284 | 16:00 |
| jeblair | logstash workers should be running now | 16:00 |
| *** Ryan_Lane has quit IRC | 16:01 | |
| jeblair | anteaya: actually i think there was no line because the log-client, which runs the gearman server that is being graphed there, was not running | 16:02 |
| anteaya | ah | 16:02 |
| clarkb | jeblair: you said the gearman log client wasn't running? that does appear to be an enabled service in puppet so should start on boot | 16:02 |
| anteaya | that would be the jenkins-log-client you restarted | 16:02 |
| clarkb | same with the log workers I need to go read puppet docs apparently | 16:03 |
| *** Bsony has quit IRC | 16:03 | |
| anteaya | is the jenkins-log-client and the gearman log client two names for the same thing? | 16:03 |
| jeblair | clarkb: hrm, the symlink looks correct | 16:04 |
| *** dustins has quit IRC | 16:04 | |
| clarkb | anteaya: yes | 16:05 |
| *** dkranz has joined #openstack-infra | 16:05 | |
| *** mattoliverau has joined #openstack-infra | 16:05 | |
| anteaya | clarkb: thanks | 16:05 |
| *** dhritishikhar has quit IRC | 16:05 | |
| *** imcsk8 has quit IRC | 16:06 | |
| *** imcsk8 has joined #openstack-infra | 16:06 | |
| *** jlanoux has joined #openstack-infra | 16:06 | |
| *** garyh has quit IRC | 16:07 | |
| anteaya | I found this: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash/jenkins-log-client.yaml | 16:08 |
| notmyname | mordred: BobBall: good morning. how can I help (logs in swift, form post, content type stuff, I gather?) | 16:08 |
| *** _nadya_ has joined #openstack-infra | 16:08 | |
| *** david-lyle has joined #openstack-infra | 16:08 | |
| *** MaxV has quit IRC | 16:08 | |
| clarkb | jeblair: ya /etc/rc5.d/S20jenkins-log-worker* | 16:09 |
| *** Daisy_ has joined #openstack-infra | 16:09 | |
| clarkb | jeblair: is it possible that is happening before upstart has networking up? so connections are unhappy? | 16:09 |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config: Add networking-bagpipe-l2 project https://review.openstack.org/161290 | 16:10 |
| *** garyh has joined #openstack-infra | 16:11 | |
| anteaya | ah this looks better: http://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/jenkins-log-client.init | 16:11 |
| *** jerryz_ has quit IRC | 16:11 | |
| clarkb | jeblair: 'gaierror: [Errno -3] Temporary failure in name resolution' from the logs | 16:11 |
| jeblair | clarkb: i think we're at runlevel 2, but the links are there too | 16:11 |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config: new-project https://review.openstack.org/161290 | 16:12 |
| clarkb | jeblair: I don't think upstart does runlevels :( | 16:12 |
| *** tsg has quit IRC | 16:12 | |
| clarkb | so rc5 == rc3 == rc2 | 16:12 |
| clarkb | but that error is from when things went down not when it tried coming back up | 16:12 |
| jeblair | clarkb: well, it has to pick one of them :) | 16:13 |
| *** ihrachyshka has joined #openstack-infra | 16:13 | |
| jeblair | and "runlevel" reports 2 | 16:13 |
| clarkb | reading the logs from when it started back it appears to have only started when you manually restarted it | 16:13 |
| clarkb | so I don't think it tried to start on boot at all | 16:13 |
| clarkb | oh wait no I see | 16:14 |
| mtreinish | jeblair: heh, isn't that multiuser no networking? :) | 16:14 |
| *** jedimike has quit IRC | 16:14 | |
| jeblair | clarkb: unbound and log-client are both at S20 | 16:14 |
| clarkb | jeblair: http://paste.openstack.org/show/187856/ | 16:14 |
| *** MaxV has joined #openstack-infra | 16:14 | |
| sdague | mtreinish: not on ubuntu | 16:14 |
| clarkb | jeblair: yup I think we need to push back when the log pusher starts | 16:14 |
| clarkb | jeblair: probably to 95 to be around when es is coming up | 16:14 |
| clarkb | sdague: mtreinish or any other upstarted distro aiui | 16:15 |
| jeblair | clarkb: sounds good | 16:15 |
| *** ihrachyshka has quit IRC | 16:15 | |
| *** yjiang5 has quit IRC | 16:15 | |
| jeblair | oh mysql-proxy is also not running | 16:15 |
| *** david-lyle has quit IRC | 16:15 | |
| jeblair | started | 16:15 |
| *** _nadya_ has quit IRC | 16:15 | |
| clarkb | probably for similar reasons | 16:15 |
| *** ihrachyshka has joined #openstack-infra | 16:15 | |
| clarkb | so I think our use of unbound as a puppet managed service defaults to S20 which everythign else gets and may cause trouble | 16:15 |
| jeblair | oh, right, we use simpleproxy now | 16:16 |
| clarkb | and if name resolution causes you to fail (which is not unreasonable) things get cranky | 16:16 |
| *** nibalizer has joined #openstack-infra | 16:16 | |
| openstackgerrit | Gorka Eguileor proposed openstack/requirements: Bump rtslib version to have config persistence https://review.openstack.org/161291 | 16:17 |
| jeblair | clarkb: should probably move it to S15 | 16:17 |
| *** ujuc has quit IRC | 16:17 | |
| clarkb | ya I am trying to sort out how to change those values in puppet now | 16:17 |
| *** mfink_ has joined #openstack-infra | 16:18 | |
| *** Ryan_Lane has joined #openstack-infra | 16:19 | |
| *** scheuran_ has joined #openstack-infra | 16:20 | |
| *** Ryan_Lane has quit IRC | 16:21 | |
| clarkb | https://tickets.puppetlabs.com/browse/PUP-2664 | 16:21 |
| *** annegentle has quit IRC | 16:21 | |
| *** annegentle has joined #openstack-infra | 16:22 | |
| *** jedimike has joined #openstack-infra | 16:22 | |
| mordred | clarkb: so that says to me "you don't" | 16:23 |
| mordred | clarkb: actually - wouldn't you just make sure that there is an LSB block at the top of the script? | 16:23 |
| *** scheuran has quit IRC | 16:23 | |
| clarkb | mordred: so there is one but I have no idea if puppet respects that | 16:24 |
| mordred | clarkb: well it SHOULD just be running "update-rc.d foo defaults" | 16:24 |
| mordred | and if it's not, it's crazy | 16:24 |
| *** Daisy_ has quit IRC | 16:24 | |
| greghaynes | jhesketh: pong | 16:24 |
| *** mfink__ has joined #openstack-infra | 16:25 | |
| clarkb | I will rwite a change to use that method which is easy to test | 16:25 |
| *** asselin__ has joined #openstack-infra | 16:25 | |
| *** jaypipes has joined #openstack-infra | 16:26 | |
| clarkb | oh looks like I can tell it to start after $all | 16:26 |
| *** ParsectiX has joined #openstack-infra | 16:28 | |
| *** mfink_ has quit IRC | 16:29 | |
| *** asselin_ has quit IRC | 16:29 | |
| clarkb | ok, after 5 minutes of digging the unbound init script in ubuntu does not provide $named so we can't just depend on that | 16:30 |
| clarkb | which would've been a simplish fix | 16:30 |
| clarkb | I will use a require on $all instead | 16:30 |
| *** david-lyle has joined #openstack-infra | 16:30 | |
| clarkb | hrm elasticsearch depends on $named, maybe there is some magic here that I am not following | 16:31 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Disable gate-oslo-incubator-python34 for Icehouse https://review.openstack.org/161303 | 16:32 |
| *** skolekonov has quit IRC | 16:35 | |
| *** tnovacik has joined #openstack-infra | 16:36 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-log_processor: Start processes after network and named https://review.openstack.org/161304 | 16:36 |
| clarkb | now I need to figure out how to apply that to my puppet test env... | 16:36 |
| *** mpavone has quit IRC | 16:40 | |
| *** Ryan_Lane has joined #openstack-infra | 16:41 | |
| *** rwsu-afk is now known as rwsu | 16:42 | |
| *** mrodden has quit IRC | 16:42 | |
| *** mrda has quit IRC | 16:43 | |
| *** julim has quit IRC | 16:44 | |
| *** mrodden has joined #openstack-infra | 16:44 | |
| *** ildikov has joined #openstack-infra | 16:44 | |
| *** arxcruz has quit IRC | 16:45 | |
| *** Ryan_Lane has quit IRC | 16:45 | |
| *** julim has joined #openstack-infra | 16:45 | |
| *** sdake_ has joined #openstack-infra | 16:45 | |
| *** thedodd has joined #openstack-infra | 16:46 | |
| *** dtantsur is now known as dtantsur|afk | 16:51 | |
| openstackgerrit | Merged openstack-infra/project-config: Re-enabling the glusterfs job for Cinder https://review.openstack.org/160664 | 16:53 |
| *** HeOS has quit IRC | 16:53 | |
| *** vivcheri has joined #openstack-infra | 16:56 | |
| *** asselin__ has quit IRC | 16:56 | |
| vivcheri | How do I join the OpenStack Infra Team ? | 16:56 |
| anteaya | vivcheri: that is a good start | 16:57 |
| anteaya | welcome | 16:57 |
| *** gamepl has quit IRC | 16:57 | |
| vivcheri | Thanks | 16:57 |
| anteaya | what would you like to accomplish? | 16:57 |
| anteaya | as part of the team | 16:57 |
| anteaya | or do you know yet? | 16:57 |
| vivcheri | I am Vivek Varghese Cherian, an OpenStack Enthusiast and the Co-Author of the Open Stack Beginner's Guide Essex release. | 16:58 |
| anteaya | wonderful | 16:58 |
| *** Ryan_Lane has joined #openstack-infra | 16:58 | |
| vivcheri | I use OpenStack daily at work. | 16:58 |
| anteaya | awesome | 16:58 |
| anteaya | have you ever contributed a patch before? | 16:58 |
| anteaya | it would be good to start there if you have yet to do that | 16:59 |
| *** ihrachyshka has quit IRC | 16:59 | |
| vivcheri | Currently I am integrating OpenStack with Ceph as well as making OpenStack work with various hypervisors and creating a VM farm powered by OpenStack. | 16:59 |
| *** patrickeast has quit IRC | 16:59 | |
| anteaya | great | 17:00 |
| anteaya | so lots of ops experience | 17:00 |
| anteaya | we also do quite a bit of dev work | 17:00 |
| anteaya | and reviews | 17:00 |
| anteaya | lots and lots of reviews | 17:00 |
| vivcheri | Ok. | 17:00 |
| vivcheri | Ryan_Lane, Hi, this is Vivek Cherian. | 17:00 |
| anteaya | have you interacted with our gerrit before? it is at review.openstack.org | 17:00 |
| vivcheri | Nice to meet you after a while. | 17:01 |
| Ryan_Lane | vivcheri: hi | 17:01 |
| anteaya | Ryan_Lane: hey there | 17:01 |
| Ryan_Lane | howdy | 17:01 |
| clarkb | mordred: fwiw I cannot get update-rc.d to respect the lsb headers. defaults sets the defaults which are not what we want | 17:01 |
| vivcheri | Ryan_Lane, trying to help out here. | 17:01 |
| anteaya | nice to see you, how's things | 17:01 |
| vivcheri | Ryan_Lane, Do you still volunteer for the Wikimedia foundation, I know you moved jobs. Just wanted to know if you still volunteer there ? | 17:02 |
| vivcheri | I am fine. | 17:02 |
| Ryan_Lane | anteaya: great. at the salt conference right now | 17:02 |
| openstackgerrit | greghaynes proposed openstack-infra/zuul: Add base class for reporters https://review.openstack.org/119530 | 17:02 |
| anteaya | Ryan_Lane: awesome! say hi to everyone there for me | 17:02 |
| Ryan_Lane | vivcheri: I do volunteer there a bit still. not a lot, overall, though | 17:02 |
| Ryan_Lane | anteaya: will do :) | 17:02 |
| anteaya | :) | 17:02 |
| openstackgerrit | Michael Johnson proposed openstack-infra/git-review: Choose correct branch for rebase when submitting https://review.openstack.org/158877 | 17:02 |
| *** mikal has quit IRC | 17:03 | |
| *** Loquacities has quit IRC | 17:03 | |
| clarkb | mordred: unless and this is probably what is happening, $network and $named are both available at 20 | 17:03 |
| vivcheri | How did Fosdem go, we had the talk by Thierry Carrez right ? | 17:03 |
| anteaya | Thierry is ttx | 17:03 |
| *** dustins has joined #openstack-infra | 17:03 | |
| vivcheri | Ok. | 17:03 |
| anteaya | you can ask him :) | 17:04 |
| ttx | ohai | 17:04 |
| *** dustins has quit IRC | 17:04 | |
| *** unicell has quit IRC | 17:04 | |
| vivcheri | ok :), I wanted the video and slides :) | 17:04 |
| clarkb | mordred: nope changing to $all doesn't change the order | 17:04 |
| anteaya | ttx meet vivcheri, (s)he knows your work | 17:04 |
| ttx | vivcheri: let me see if that was uploaded yet | 17:04 |
| *** amotoki has quit IRC | 17:04 | |
| *** ghostpl_ has quit IRC | 17:05 | |
| ttx | Slides at https://fosdem.org/2015/schedule/event/openstack_infra_tools_to_borrow/ | 17:05 |
| *** jlanoux has quit IRC | 17:05 | |
| *** ddieterl_ has joined #openstack-infra | 17:05 | |
| *** rkukura has quit IRC | 17:05 | |
| ttx | No video yet: https://fosdem.org/~agk/video_status.txt | 17:05 |
| *** ddieterly has quit IRC | 17:05 | |
| *** bswartz has quit IRC | 17:05 | |
| anteaya | yay slides | 17:05 |
| vivcheri | anteaya, Thanks for the intro to ttx. | 17:06 |
| vivcheri | anteaya, and I am a male. | 17:06 |
| *** mrodden has quit IRC | 17:06 | |
| vivcheri | s/a male/male | 17:06 |
| *** mrodden has joined #openstack-infra | 17:06 | |
| clarkb | aha! we have a /etc/init.d/.legacy-bootordering that explains it | 17:07 |
| clarkb | I give up, init is hard | 17:07 |
| anteaya | vivcheri: cool thanks | 17:07 |
| mkj_wk | clarkb: thanks for the ssh -T suggestion, it worked :) | 17:07 |
| ttx | clarkb, jeblair: did anyone clear the statusbot alert I added this morning ? | 17:08 |
| mkj_wk | clarkb: it let me ditch the whole annoying patch | 17:08 |
| anteaya | vivcheri: so what next? how can I help you get started? | 17:08 |
| vivcheri | ttx, Thanks for the slides. | 17:08 |
| anteaya | ttx nope not yet | 17:08 |
| clarkb | mkj_wk: cool thats good news | 17:08 |
| clarkb | ttx: doesn't look like it no | 17:08 |
| mkj_wk | argh, and now after submitting it I see another review with more feedback to look at... behind on email d/t meetings | 17:09 |
| vivcheri | anteaya, Setting up, automating openstack deployments can be a good sarting point. Do we do that here ? | 17:09 |
| anteaya | well we do use devstack to set up testing environments | 17:10 |
| anteaya | we mostly run testing infrastructure in infra | 17:10 |
| ttx | clarkb: shall I clear now ? I think we are past the issue ? | 17:10 |
| *** devvesa has quit IRC | 17:10 | |
| anteaya | vivcheri: have you seen: http://status.openstack.org/zuul/ | 17:10 |
| anteaya | we do automated testing on all patches submitted to gerrit | 17:11 |
| *** Ala has quit IRC | 17:11 | |
| vivcheri | anteaya, I am talking a look. | 17:11 |
| vivcheri | I see, ok. | 17:11 |
| anteaya | so we have tools to read the gerrit event stream and trigger tests | 17:11 |
| *** Hefeweizen has quit IRC | 17:11 | |
| ttx | something like #status ok Issue solved, gate slowly digesting accumulated changes | 17:11 |
| anteaya | but the tests need to run on something, so we have nodes ready with a set testing environment for the jobs to run in | 17:12 |
| anteaya | ttx that works for me | 17:12 |
| *** tsg has joined #openstack-infra | 17:12 | |
| *** e0ne is now known as e0ne_ | 17:12 | |
| *** e0ne_ is now known as e0ne | 17:12 | |
| anteaya | vivcheri: we set up devstack thousands of times a day, we don't deploy openstack as an operator would though | 17:13 |
| anteaya | vivcheri: do you see the difference? | 17:13 |
| clarkb | ttx: yes, I think functioanlly we are past the issue | 17:13 |
| ttx | #status ok Issue solved, gate slowly digesting accumulated changes | 17:13 |
| openstackstatus | ttx: sending ok | 17:13 |
| *** jistr has quit IRC | 17:13 | |
| vivcheri | anteaya, yes. | 17:14 |
| anteaya | vivcheri: great | 17:14 |
| anteaya | so we create tools to help devs develop | 17:14 |
| anteaya | and the first step to understanding that workflow is from the perspective of a developer | 17:14 |
| vivcheri | anteaya, But devstack uses the trunk right ? | 17:14 |
| anteaya | yes | 17:14 |
| vivcheri | Ok. | 17:14 |
| anteaya | it has to, in order for the tests to be effective | 17:14 |
| anteaya | the tip of master everytime | 17:15 |
| anteaya | so if you have yet to submit a patch and interact with our gerrit at review.openstack.org | 17:15 |
| -openstackstatus- NOTICE: Issue solved, gate slowly digesting accumulated changes | 17:15 | |
| anteaya | I do think that would be a first step to understanding the big picture of what infra does | 17:15 |
| vivcheri | anteaya, Ok. | 17:16 |
| anteaya | vivcheri: so take a look at http://docs.openstack.org/infra/manual/developers.html | 17:17 |
| vivcheri | anteaya, Ok. | 17:17 |
| anteaya | and work your way through | 17:17 |
| vivcheri | sure | 17:17 |
| anteaya | it is designed to be a manual so someone can get started | 17:17 |
| anteaya | let us know if you have questions | 17:17 |
| openstackstatus | ttx: finished sending ok | 17:17 |
| anteaya | vivcheri: is that fair? | 17:17 |
| *** sedwards has joined #openstack-infra | 17:18 | |
| vivcheri | anteaya, Yes it's Kosher. | 17:18 |
| anteaya | great thanks | 17:18 |
| vivcheri | yw | 17:18 |
| anteaya | if you have a question, just ask you don't need to wait for me or ask to ask | 17:18 |
| anteaya | many people can help you | 17:18 |
| vivcheri | I got it. | 17:19 |
| anteaya | great | 17:19 |
| anteaya | welcome aboard | 17:19 |
| vivcheri | anteaya, Thanks. | 17:19 |
| *** ddieterl_ has quit IRC | 17:20 | |
| *** Sukhdev has quit IRC | 17:20 | |
| clarkb | ok our rax images seem to come with legacy bootordering built in | 17:21 |
| *** dhritishikhar has joined #openstack-infra | 17:21 | |
| clarkb | a dpkg-reconfigure sysv-rc appears happy so I am not sure that that is actually broken | 17:21 |
| *** e0ne is now known as e0ne_ | 17:23 | |
| zaro | morning | 17:24 |
| *** mrmartin has joined #openstack-infra | 17:24 | |
| *** ddieterly has joined #openstack-infra | 17:25 | |
| *** dhritishikhar has quit IRC | 17:25 | |
| *** Hefeweizen has joined #openstack-infra | 17:25 | |
| openstackgerrit | Merged openstack-infra/python-storyboardclient: Added milestones controller https://review.openstack.org/160334 | 17:25 |
| anteaya | morning zaro | 17:26 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Added new OAuth Parameters https://review.openstack.org/157464 | 17:26 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/system-config: Add logging to #openstack-api https://review.openstack.org/161330 | 17:27 |
| tsg | zigo: ping | 17:27 |
| openstackgerrit | greghaynes proposed openstack-infra/zuul: Add base class for sources https://review.openstack.org/119531 | 17:27 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Added new OAuth Parameters https://review.openstack.org/157464 | 17:27 |
| *** _shaps_ has quit IRC | 17:27 | |
| *** ryanpetrello has joined #openstack-infra | 17:28 | |
| *** Ryan_Lane has quit IRC | 17:28 | |
| *** sdake_ has quit IRC | 17:30 | |
| clarkb | mordred: https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/1024658/comments/7 | 17:31 |
| openstack | Launchpad bug 1024658 in sysvinit (Ubuntu) "problems during update: Can't exec "insserv": No such file or directory at /usr/sbin/update-rc.d line 402." [Undecided,Invalid] | 17:31 |
| *** aarefiev has joined #openstack-infra | 17:32 | |
| *** cdent has quit IRC | 17:33 | |
| *** nelsnelson has quit IRC | 17:33 | |
| *** ryanpetrello has quit IRC | 17:33 | |
| *** claudiub has quit IRC | 17:33 | |
| clarkb | mordred: jeblair so I think the answer here is packages | 17:35 |
| *** ryanpetrello has joined #openstack-infra | 17:35 | |
| clarkb | because ubuntu's init system is compeltely broken | 17:35 |
| tsg | clarkb: a question for you on our discussion from a few days ago on getting a new package (liberasurecode) integrated into jenkins slaves - we have a debian package available - does the package have to be pulled into an official ubuntu release before we submit a request to add it to infra? | 17:35 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config: Add gerrit notifications for #openstack-api https://review.openstack.org/161337 | 17:36 |
| clarkb | tsg: if you want to gate on it the typical answer is yes | 17:36 |
| tsg | clarkb: zigo has packaged the deb for Debian .. do you have guidance on how to get the package into Ubuntu? | 17:36 |
| clarkb | mordred: jeblair or we can upstartify these init scripts | 17:37 |
| clarkb | tsg: I do not, not sure what that process is | 17:37 |
| *** samueldmq has joined #openstack-infra | 17:39 | |
| *** ryanpetrello has quit IRC | 17:39 | |
| tsg | clarkb: ok - thanks. zigo said he was the openstack debian package maintainer, but said he had no control over when ubuntu pulls a package, so I thought may be you'd know. I will research this further | 17:39 |
| *** sergsh has quit IRC | 17:40 | |
| *** scheuran_ has quit IRC | 17:40 | |
| *** e0ne has joined #openstack-infra | 17:41 | |
| *** ryanpetrello has joined #openstack-infra | 17:41 | |
| *** e0ne has quit IRC | 17:41 | |
| *** unicell has joined #openstack-infra | 17:41 | |
| *** che-arne has quit IRC | 17:41 | |
| *** unicell has quit IRC | 17:42 | |
| *** unicell has joined #openstack-infra | 17:42 | |
| *** Ryan_Lane has joined #openstack-infra | 17:43 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:44 | |
| *** afazekas has quit IRC | 17:44 | |
| *** mrodden has quit IRC | 17:45 | |
| *** ryanpetrello has quit IRC | 17:45 | |
| *** mrodden has joined #openstack-infra | 17:45 | |
| clarkb | mordred: jeblair also I bet this works fine on debian... | 17:46 |
| *** dizquierdo has quit IRC | 17:46 | |
| vivcheri | anteaya, Hi | 17:46 |
| *** ryanpetrello has joined #openstack-infra | 17:46 | |
| tsg | clarkb: are the jenkins slaves based on Ubuntu 14.04.2? | 17:47 |
| *** patrickeast has joined #openstack-infra | 17:47 | |
| vivcheri | anteaya, Still there ?, If you guys use puppet, then I can help out in that area. | 17:47 |
| clarkb | tsg: 14.04 updated. I think that gets 14.04.latestpoint+updates | 17:47 |
| *** gyee has joined #openstack-infra | 17:47 | |
| *** jogo has joined #openstack-infra | 17:47 | |
| *** jcoufal_ has quit IRC | 17:48 | |
| tsg | clarkb: thank you. And do we have main, universe and multiverse (?) in apt config? | 17:48 |
| tsg | clarkb: also is there a way to add a new software source, or standard sources only? | 17:49 |
| *** mwagner_lap has quit IRC | 17:49 | |
| clarkb | yes it appears all three are enabled. I am semi surprised about multiverse thats probably just part of the base image | 17:49 |
| *** ryanpetrello has quit IRC | 17:51 | |
| *** dkranz has quit IRC | 17:51 | |
| *** ryanpetrello has joined #openstack-infra | 17:51 | |
| openstackgerrit | Ian Cordasco proposed openstack-infra/project-config: Add gerrit notifications for #openstack-api https://review.openstack.org/161337 | 17:51 |
| *** achanda has joined #openstack-infra | 17:54 | |
| anteaya | vivcheri: great | 17:55 |
| anteaya | vivcheri: I'm not a guy | 17:55 |
| anteaya | vivcheri: we use a lot of puppet | 17:55 |
| vivcheri | anteaya, ok, nice to see a woman contribiter. We need more women in computing :) | 17:55 |
| anteaya | lots in openstack | 17:56 |
| vivcheri | s/contribiter/contributer | 17:56 |
| anteaya | so everything that is openstack-infra/puppet* http://git.openstack.org/cgit | 17:56 |
| vivcheri | Yea, that is a healthy trend. | 17:56 |
| anteaya | plus openstack-infra/system-config | 17:56 |
| anteaya | yes | 17:56 |
| vivcheri | Ok. | 17:56 |
| anteaya | all puppet | 17:56 |
| anteaya | to contribute you will need to know how to use our gerrit | 17:56 |
| anteaya | so the developer document I pointed you to earlier | 17:57 |
| vivcheri | Ok. | 17:57 |
| vivcheri | I am read through it once. | 17:57 |
| sedwards | exit | 17:57 |
| *** sedwards has left #openstack-infra | 17:57 | |
| *** afazekas has joined #openstack-infra | 17:57 | |
| vivcheri | s/am/'ve | 17:57 |
| *** patrickeast_ has joined #openstack-infra | 17:58 | |
| *** patrickeast__ has joined #openstack-infra | 17:58 | |
| *** patri251 has joined #openstack-infra | 17:58 | |
| *** patri498 has joined #openstack-infra | 17:58 | |
| *** ssam2 has quit IRC | 17:58 | |
| *** penguinRaider has quit IRC | 17:59 | |
| anteaya | vivcheri: great now you have to follow the instructions | 18:00 |
| anteaya | and have a patch up on the sandbox repo | 18:00 |
| *** dkranz has joined #openstack-infra | 18:00 | |
| anteaya | http://git.openstack.org/cgit/openstack-dev/sandbox/ | 18:00 |
| vivcheri | Ok. | 18:00 |
| *** erlon has joined #openstack-infra | 18:00 | |
| anteaya | if you can get a patch up on sandbox you know you have completed the minium steps to contribute | 18:01 |
| *** patri461 has joined #openstack-infra | 18:01 | |
| tsg | clarkb: thank you. And was there a way to add a new source source (PPA)? | 18:01 |
| *** patri582 has joined #openstack-infra | 18:01 | |
| clarkb | tsg: there is but not for gating | 18:02 |
| *** ashleighfarnham has joined #openstack-infra | 18:02 | |
| tsg | clarkb: ok. so the package has to be part of standard repos for gate | 18:02 |
| clarkb | if you want to test a stackforge project with a PPA that would be fine but we have pretty much disallowed PPAs in the integrated gate because its hard to keep tabs on them and when they fail and so on | 18:02 |
| *** unicell has quit IRC | 18:02 | |
| clarkb | tsg: that is the current expectation | 18:02 |
| *** unicell has joined #openstack-infra | 18:02 | |
| clarkb | there has also been talk of having our own package repos for things but I don't think anyone is actively working on that | 18:03 |
| *** markus_z has quit IRC | 18:03 | |
| tsg | clarkb: ok. we'll try to work within | 18:03 |
| tsg | clarkb: I will file an Ubuntu bug to get liberasurecode synced from Debian and see how that goes | 18:04 |
| *** ryanpetrello has quit IRC | 18:05 | |
| notmyname | I saw a notice from the bot saying that the zuul change is done. will the channel topic messages be restored, or is that up to each channel to fix? | 18:05 |
| *** sputnik13 has joined #openstack-infra | 18:06 | |
| *** marun has joined #openstack-infra | 18:06 | |
| clarkb | hrm the bot should've reset the topics | 18:06 |
| vivcheri | anteaya, ok | 18:06 |
| * clarkb goes to read bot stuff to figure out why it didn't | 18:06 | |
| *** mayurig has joined #openstack-infra | 18:06 | |
| *** ryanpetrello has joined #openstack-infra | 18:07 | |
| notmyname | clarkb: thanks | 18:07 |
| *** bdpayne has joined #openstack-infra | 18:08 | |
| *** katyafervent is now known as katyafervent_awa | 18:09 | |
| *** MaxV has quit IRC | 18:09 | |
| *** SumitNaiksatam has quit IRC | 18:10 | |
| clarkb | looks like the bot was restarted between commands (likely due to the node reboots) and I think it lost the old topic state :/ | 18:11 |
| clarkb | jeblair: ^ that sound right to you? | 18:11 |
| *** _nadya_ has joined #openstack-infra | 18:11 | |
| *** _nadya_ has quit IRC | 18:11 | |
| openstackgerrit | greghaynes proposed openstack-infra/zuul: Add base class for triggers https://review.openstack.org/119532 | 18:12 |
| *** rkukura has joined #openstack-infra | 18:14 | |
| *** BobBall is now known as BobBall_AWOL | 18:15 | |
| *** MaxV has joined #openstack-infra | 18:16 | |
| *** bswartz has joined #openstack-infra | 18:19 | |
| *** luqas__ has quit IRC | 18:19 | |
| *** harlowja_away is now known as harlowja_ | 18:19 | |
| *** teran has joined #openstack-infra | 18:19 | |
| *** katyafervent_awa is now known as katyafervent | 18:22 | |
| *** e0ne has joined #openstack-infra | 18:22 | |
| *** teran has quit IRC | 18:23 | |
| *** vilobhmm has joined #openstack-infra | 18:24 | |
| *** MaxV has quit IRC | 18:24 | |
| *** vilobhmm1 has joined #openstack-infra | 18:25 | |
| *** Ryan_Lane has quit IRC | 18:25 | |
| *** jp_at_hp has quit IRC | 18:25 | |
| *** ddieterl_ has joined #openstack-infra | 18:26 | |
| *** ddieterl_ has quit IRC | 18:27 | |
| *** ddieterl_ has joined #openstack-infra | 18:27 | |
| *** stevelle has joined #openstack-infra | 18:28 | |
| *** vilobhmm has quit IRC | 18:29 | |
| *** ddieterly has quit IRC | 18:29 | |
| *** MaxV has joined #openstack-infra | 18:30 | |
| *** afazekas has quit IRC | 18:31 | |
| *** vilobhmm1 has left #openstack-infra | 18:32 | |
| sdague | clarkb: so... is the lack of in use nodes a function of hp cloud issues? and if so, are they reported, Kiall mentioned there was no issue listed when he looked this morning (and I didn't have context except for a comment you made yesterday) | 18:32 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Add openstack/developer-reference repository https://review.openstack.org/161351 | 18:32 |
| dhellmann | AJaeger, anteaya: I had to resubmit ^^ using a new change-id for the depends-on linking to work properly. | 18:33 |
| clarkb | sdague: I haven't checked this morning post rax reboot | 18:33 |
| clarkb | sdague: but I guess I can give up on fixing init on ubuntu for now and check nodepool | 18:33 |
| anteaya | dhellmann: isn't that interesting | 18:34 |
| clarkb | yes hpcloud is at almost 100% failure rate right now, let me see if logs tell us why | 18:34 |
| dhellmann | anteaya: yeah, I had both changes using the same change-id so they would be linked together, but that doesn't allow the depends-on ordering thing you wanted | 18:34 |
| *** MaxV has quit IRC | 18:34 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:35 | |
| anteaya | oh I see | 18:35 |
| anteaya | yes that would make sense | 18:35 |
| clarkb | sdague: ClientException: Unknown Error (HTTP 500) from novaclient | 18:36 |
| mordred | clarkb: that means there was an error | 18:37 |
| clarkb | mordred: yes I agree | 18:38 |
| mordred | clarkb: anything I can do to help investigate? | 18:38 |
| *** ivar-lazzaro has joined #openstack-infra | 18:38 | |
| clarkb | mordred: you know many things of novaclient at this point, any better way to get info? I suppose I can nova show out of band to see if I can catch the state transition with info? | 18:39 |
| mordred | clarkb: yeah - I Think that's what you probably need | 18:39 |
| mordred | clarkb: there may also be an error field in that show | 18:39 |
| *** ivar-lazzaro has quit IRC | 18:40 | |
| clarkb | hrm randomly selected building node says its active but without a floating ip | 18:40 |
| clarkb | let me check the floating ips | 18:40 |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: Bump oslo.config to latest release https://review.openstack.org/161353 | 18:40 |
| stevemar | dhellmann, ^ | 18:40 |
| *** Ryan_Lane has joined #openstack-infra | 18:40 | |
| clarkb | there are a large number of floating ips unassociated | 18:40 |
| *** ivar-lazzaro has joined #openstack-infra | 18:41 | |
| mordred | awesome. I love the floating ip leak | 18:41 |
| clarkb | 604 | 18:41 |
| mordred | we should delete those I think | 18:41 |
| pelix | zaro: tempted to approve https://review.openstack.org/#/c/133774/, https://review.openstack.org/146544 & https://review.openstack.org/#/c/68895/ - generally loath to be the second core reviewer on my stuff but these have been sitting waiting for a month or two with no negative feedback | 18:41 |
| clarkb | agreed let me dig up a thing to do that | 18:41 |
| clarkb | mordred: its better to use neutron client ya? | 18:41 |
| *** david-lyle has quit IRC | 18:42 | |
| dhellmann | stevemar: +2 | 18:42 |
| mordred | clarkb: yeah - I thnik it's one less API hop | 18:42 |
| pelix | course if anyone else with core reviewer on JJB would like to take a look over those ^^^ | 18:42 |
| *** david-lyle has joined #openstack-infra | 18:42 | |
| clarkb | mordred: also my favorite thing in the world is that neutron and nova clients use different commands for floating ips | 18:43 |
| stevemar | ty | 18:43 |
| *** MaxV has joined #openstack-infra | 18:45 | |
| mordred | clarkb: yup | 18:45 |
| *** mrunge_bbl has quit IRC | 18:46 | |
| *** ChuckC_ has quit IRC | 18:46 | |
| *** jhesketh has quit IRC | 18:47 | |
| clarkb | mordred: `venv/bin/neutron floatingip-list | grep -v '10\.0\.' | sed -e '1,3d' -e '$d' | cut -d'|' -f 2 | xargs -n 1 -P 10 venv/bin/neutron floatingip-delete` how does that look? | 18:47 |
| * clarkb gives it a whirl | 18:47 | |
| *** spzala has joined #openstack-infra | 18:47 | |
| *** otter768 has joined #openstack-infra | 18:49 | |
| vivcheri | anteaya, I am off to bed, talk to you later. Thanks folks for the warm welcome. Bye. | 18:49 |
| mordred | clarkb: looks good! | 18:49 |
| clarkb | lol I got ratelimited then neutronclient blows up on __init__() got an unexpected keyword argument 'retry_after' | 18:49 |
| *** gyee has quit IRC | 18:49 | |
| clarkb | so this may require a couple passes | 18:49 |
| anteaya | vivcheri: bye | 18:49 |
| mordred | wow | 18:49 |
| *** vivcheri has quit IRC | 18:49 | |
| *** nelsnelson has joined #openstack-infra | 18:49 | |
| clarkb | also I realize that the grep and sed and cut can all be a single sed but for whatever reason grep -v and cut make sense in my brain for those operations | 18:50 |
| clarkb | I am reducing to parallelism of 1 and trying again | 18:50 |
| *** tonytan4ever has quit IRC | 18:51 | |
| jeblair | clarkb: i didn't follow your thing about init scripts | 18:51 |
| clarkb | jeblair: let me collate everything in one cohesive thought here | 18:51 |
| jeblair | k | 18:51 |
| openstackgerrit | Andrey Pavlov proposed openstack/requirements: Add botocore to requirements https://review.openstack.org/154386 | 18:52 |
| *** tonytan4ever has joined #openstack-infra | 18:52 | |
| clarkb | jeblair: Ubuntu does not support insserv. insserv is the tool that update-rc.d uses to map out dependencies and set sequences based on LSB header info. So when you run update-rc.d $service defaults you get the legacy defaults which is start sequence 20 and stop sequence 80. You do not get any lsb header derived values | 18:52 |
| *** jhesketh has joined #openstack-infra | 18:53 | |
| *** MaxV has quit IRC | 18:53 | |
| clarkb | jeblair: ubuntu implements this by placing a /etc/init.d/.legacy-bootorder file in /etc/init.d/ and update-rc.d uses its presence as a flag to mean don't use LSB header info | 18:53 |
| clarkb | jeblair: if you remove .legacy-bootorder from /etc/init.d update-rc.d breaks because inisserv is not in its path because the ubuntu insserv package is broken | 18:53 |
| *** otter768 has quit IRC | 18:54 | |
| clarkb | I have a hunch that ubuntu has done this because they made a decision to not fix all sysv init scripts to include lsb header info and instead focus on upstart scripts | 18:54 |
| clarkb | the problem with that is it make sysv on ubuntu not super useful | 18:54 |
| jeblair | clarkb: can we specify the priority in puppet? | 18:54 |
| clarkb | jeblair: no, there is a puppet bug that essentially says that its the distros problem (via mechanisms like update-rc.d and chkconfig) | 18:55 |
| clarkb | the most aggravating thing about htis is ubuntu does nothing to update the update-rc.d and insserv man pages on ubuntu so that I can figure this out in a sane way | 18:56 |
| *** tonytan4ever has quit IRC | 18:57 | |
| jeblair | wow. sysv init worked for 32 years, and now this. | 18:57 |
| *** dustins_ has joined #openstack-infra | 18:57 | |
| jeblair | i feel like converting to upstart is not a good idea because our next operating system is certain not to use it (one way or another) | 18:58 |
| clarkb | jeblair: agreed | 18:58 |
| *** dustins_ is now known as dustins | 18:58 | |
| jeblair | i'm pretty sure that puppet bug is wrong. sometimes admins need to reorder things. | 18:59 |
| mordred | well ... | 18:59 |
| mordred | we could skip update-rc.d and the puppet thing | 18:59 |
| jeblair | perhaps we should just manage the symlink ourselves in puppet. | 18:59 |
| mordred | and just make the sym links | 18:59 |
| mordred | jeblair: jinx | 18:59 |
| *** mwagner_lap has joined #openstack-infra | 18:59 | |
| clarkb | so the tricky thing with that and I would need to test this. is if you set service enable => false puppet will actively remove the symlinks for you | 18:59 |
| clarkb | and if you set service enable => true it will make the symlinks | 19:00 |
| clarkb | so we basically need a new service resource that we can use :/ | 19:00 |
| *** alexpilotti_ has joined #openstack-infra | 19:01 | |
| clarkb | floating ips are mostly deleted at this point | 19:01 |
| clarkb | we should see hpcloud pick up | 19:01 |
| sdague | clarkb: cool, that would be nice | 19:02 |
| sdague | help chew through the backlog | 19:02 |
| sdague | clarkb: is there a way to get ahead of an issue like that one? | 19:02 |
| *** mrodden has quit IRC | 19:02 | |
| clarkb | sdague: fix neutron/nova | 19:03 |
| openstackgerrit | Darragh Bailey proposed openstack-dev/pbr: Allow overwriting sphinx builder from command line https://review.openstack.org/117478 | 19:03 |
| *** alexpilotti has quit IRC | 19:03 | |
| *** alexpilotti_ is now known as alexpilotti | 19:03 | |
| sdague | is there a bug for this? so I can figure out what part of fixing is needed? | 19:04 |
| clarkb | there is not a reliably unracy way to delete a floating ip then delete a node atomicly | 19:04 |
| clarkb | I think so, would have to dig through some git logs in nodepool | 19:04 |
| mordred | or a reliable way to create a server and a floating ip such that you could re-use an existing floating IP that isn't racy | 19:05 |
| *** bswartz has quit IRC | 19:05 | |
| *** e0ne is now known as e0ne_ | 19:05 | |
| *** e0ne_ has quit IRC | 19:05 | |
| *** MaxV has joined #openstack-infra | 19:05 | |
| clarkb | sdague: https://review.openstack.org/#/c/113754/ is the supposed fix for nova | 19:06 |
| *** sputnik13 has quit IRC | 19:06 | |
| clarkb | its merged so maybe hpcloud hasn't deployed it? | 19:06 |
| *** yfried has quit IRC | 19:06 | |
| sdague | I have no idea what's in hpcloud | 19:06 |
| sdague | but that would be in juno | 19:06 |
| clarkb | oh right you have to disassociate then delete | 19:07 |
| *** dannywilson has joined #openstack-infra | 19:07 | |
| *** mpavone has joined #openstack-infra | 19:07 | |
| zaro | waynr: i will most likely not attend jenkins conference. i think i have a schedule conflict. | 19:07 |
| mordred | tteggel, gilliard: ^^ any chance that that patch has been deployed in HP Public Cloud? | 19:07 |
| *** stevemar has quit IRC | 19:07 | |
| clarkb | except that completely breaks the world if it isn't atomic because your lookup table is nova/neutron | 19:07 |
| *** stevemar has joined #openstack-infra | 19:08 | |
| zaro | waynr: i coincidentally am going to give a JJB talk in seattle next month. my slides for it will be available soon. | 19:09 |
| clarkb | I wonder if novaclient needs to do something different too | 19:09 |
| clarkb | sdague: ^ | 19:09 |
| *** bswartz has joined #openstack-infra | 19:09 | |
| clarkb | sdague: https://review.openstack.org/#/c/113659/2 is a patch that was supposed to set us up for using novaclient properly | 19:09 |
| zaro | waynr: i do highly recommend for you to submit a talk @ juc though. | 19:09 |
| *** sputnik13 has joined #openstack-infra | 19:09 | |
| *** mrodden has joined #openstack-infra | 19:09 | |
| *** sdake_ has joined #openstack-infra | 19:09 | |
| *** dannywilson has quit IRC | 19:09 | |
| clarkb | sdague: because the delete call should disassociate and delete atomicly via the nova change above, but maybe that requires novaclient to do things differently | 19:09 |
| *** MaxV has quit IRC | 19:10 | |
| *** Sukhdev has joined #openstack-infra | 19:10 | |
| *** dannywilson has joined #openstack-infra | 19:10 | |
| *** dannywilson has quit IRC | 19:10 | |
| *** dannywilson has joined #openstack-infra | 19:11 | |
| clarkb | jeblair: thoughts on fixing the channel topics? statusbox was restarted between changing all of the topics and the ok | 19:11 |
| jeblair | clarkb: only way to do that is to dig through the logs and find the old topics | 19:11 |
| clarkb | jeblair: do we just need to go through each of them, find old topic in scrollback/logs and reset? | 19:12 |
| clarkb | kk | 19:12 |
| jeblair | clarkb: i'll look | 19:12 |
| clarkb | actually I wonder if statusbot logged that when it changed them | 19:12 |
| clarkb | which may be a good source of info | 19:12 |
| anteaya | I think it does | 19:12 |
| sdague | clarkb: ok, I need to dive into a few other things first. It would be nice if we had a consolidated bug on this. It would be interesting if there was a good reproduce in a devstack env for it so we could get it nailed down. | 19:12 |
| anteaya | topic changed to blah from foo | 19:13 |
| anteaya | sdague: have you had a chance to look at the nova-net to neutron migration spec yet? | 19:13 |
| clarkb | sdague: I have no idea what causes nova/neutron to disassociate then fail to delete | 19:13 |
| sdague | anteaya: that's my "few other things bit" | 19:13 |
| clarkb | sdague: so not sure where to start | 19:13 |
| anteaya | sdague: great thank you | 19:14 |
| sdague | clarkb: my guess.... load | 19:14 |
| clarkb | sdague: and there is a bug https://bugs.launchpad.net/nova/+bug/1356157 | 19:14 |
| openstack | Launchpad bug 1356157 in OpenStack Compute (nova) icehouse "make nova floating-ip-delete atomic with neutron" [High,Fix released] - Assigned to Aaron Rosen (arosen) | 19:14 |
| sdague | so the test case is probably a stress test | 19:14 |
| sdague | mordred: can we get a confirm that that code is in hpcloud, so we know whether to reopen? | 19:15 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/manual/ http://ci.openstack.org/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/" | 19:16 | |
| *** e0ne has joined #openstack-infra | 19:16 | |
| *** funzo has quit IRC | 19:16 | |
| *** e0ne is now known as e0ne_ | 19:16 | |
| clarkb | I'm going to step out for early lunch curious on further init hackarounds | 19:16 |
| clarkb | its possible that we could fix the insserv package and remove the .legacy-bootloader file and have it work properly | 19:17 |
| clarkb | I did not test that | 19:17 |
| *** ParsectiX has quit IRC | 19:17 | |
| clarkb | that would not prevent us from breaking if ubuntu psuehd packages with bad init scripts though | 19:17 |
| clarkb | so not sure I care for that option | 19:17 |
| *** ParsectiX has joined #openstack-infra | 19:17 | |
| pelix | zaro: once https://review.openstack.org/117478 lands for pbr and is released it should be possible to add a separate check for the documentation external links in JJB so that the standard 'tox -edocs' isn't dependent on external network access nor are we spamming the jenkins wiki from user desktops as well | 19:18 |
| *** gyee has joined #openstack-infra | 19:19 | |
| *** e0ne_ is now known as e0ne | 19:20 | |
| *** MaxV has joined #openstack-infra | 19:20 | |
| *** teran has joined #openstack-infra | 19:21 | |
| *** HeOS has joined #openstack-infra | 19:21 | |
| *** Bsony has joined #openstack-infra | 19:24 | |
| *** sputnik13 has quit IRC | 19:24 | |
| *** BharatK has quit IRC | 19:24 | |
| *** sputnik13 has joined #openstack-infra | 19:25 | |
| e0ne | hi, i've got a question about grenade job | 19:26 |
| e0ne | how should/can we merge patch fo cinder if it requires changes in a grenade job to modify config? | 19:27 |
| anteaya | e0ne: can you share some details | 19:27 |
| pelix | jhesket clarkb fungi jeblair: wonder if any of you might have a chance to look at https://review.openstack.org/#/c/133774/, https://review.openstack.org/146544 & https://review.openstack.org/#/c/68895/ and see if they are ready to be +2CR'ed? Probably a few others belonging to zaro that are blocked waiting on another core-reviewer to approve as well | 19:27 |
| anteaya | e0ne: have you a patch up? | 19:27 |
| e0ne | anteaya: we've got patch only for cinder now (https://review.openstack.org/#/c/134759/) | 19:28 |
| *** MaxV has quit IRC | 19:28 | |
| *** sputnik13 has quit IRC | 19:29 | |
| e0ne | anteaya: and we need something like https://review.openstack.org/#/c/139051 for cinder | 19:30 |
| pelix | jhesketh clarkb fungi jeblair: https://review.openstack.org/#/q/project:openstack-infra/jenkins-job-builder+status:open+label:Code-Review%252B2+-label:Code-Review-1+label:Verified%252B1+ownerin:jenkins-job-builder-core,n,z full list where we are currently trying to avoid self approving but essentially blocked from landing at the moment. | 19:30 |
| e0ne | but we don't know right way to get it merged - CI always will put -1 for it | 19:31 |
| pelix | would appreciate some help with those | 19:31 |
| jeblair | e0ne: i can't help with specifics, but i think that generally you should probably create a change to devstck or grenade that supports the new configuration, and then set your cinder change to "Depends-On" it | 19:31 |
| *** vilobhmm1 has joined #openstack-infra | 19:31 | |
| jeblair | pelix: ack | 19:31 |
| pelix | jeblair: thanks | 19:31 |
| anteaya | e0ne: when you say CI will always put a -1 for it, do you mean that it fails Jenkins tests? | 19:32 |
| e0ne | anteaya: yes | 19:32 |
| e0ne | jeblair: oh! is it cross-project dependency for patches? | 19:32 |
| anteaya | great following jeblair's instructions should get you to a place when the tests pass for jenkins | 19:32 |
| anteaya | e0ne: yes | 19:32 |
| jeblair | e0ne: http://docs.openstack.org/infra/manual/developers.html#cross-project-dependencies | 19:33 |
| e0ne | jeblair, anteaya: great! it's very easy. thanks | 19:33 |
| *** ddieterl_ has quit IRC | 19:33 | |
| *** sputnik13 has joined #openstack-infra | 19:33 | |
| clarkb | also note that grenade partially exusts to prevent you from doing that | 19:34 |
| clarkb | that is a backward incompatible change | 19:34 |
| anteaya | I wonder how other projects have started using oslo.middleware and have it work with grenade | 19:34 |
| anteaya | dhellmann: have you any thoughts? ^^ | 19:35 |
| jeblair | pelix, zaro: i will try to look into those this week | 19:35 |
| pelix | jeblair: cheers, appreciated | 19:35 |
| *** sputnik13 has quit IRC | 19:35 | |
| e0ne | anteaya: it's a good question | 19:36 |
| *** MaxV has joined #openstack-infra | 19:36 | |
| *** ghostpl_ has joined #openstack-infra | 19:37 | |
| anteaya | e0ne: so do the cross repo dependency and have a chat with dhellmann, hopefully that helps you make progress | 19:37 |
| *** ghostpl_ has quit IRC | 19:37 | |
| *** ghostpl_ has joined #openstack-infra | 19:37 | |
| *** sputnik13 has joined #openstack-infra | 19:37 | |
| *** mpavone has quit IRC | 19:38 | |
| *** dhritishikhar has joined #openstack-infra | 19:38 | |
| e0ne | anteaya: thank you. we try to to it tomorrow. i could update you with our status but it should be ok | 19:38 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Removed Process Pool execution. https://review.openstack.org/161374 | 19:40 |
| sdague | jeblair: so... given that the world was broken, this might be an invalid issue | 19:41 |
| anteaya | e0ne: let us know if you need more assistance | 19:41 |
| *** Ryan_Lane has quit IRC | 19:41 | |
| sdague | however - https://review.openstack.org/#/c/159824/ has another one of those Jenkins -1ing a thing for a merge conflict part way through a patch series | 19:41 |
| e0ne | anteaya: thanks once more. i'll do | 19:41 |
| anteaya | k | 19:41 |
| sdague | sorry, not for a merge conflict | 19:42 |
| sdague | "This change depends on a change that failed to merge." | 19:42 |
| *** sputnik13 has quit IRC | 19:43 | |
| *** MaxV has quit IRC | 19:43 | |
| *** sputnik13 has joined #openstack-infra | 19:44 | |
| *** _nadya_ has joined #openstack-infra | 19:44 | |
| openstackgerrit | Merged openstack/requirements: Pin sqlalchemy-migrate >= 0.9.5 https://review.openstack.org/159242 | 19:45 |
| *** mikal has joined #openstack-infra | 19:46 | |
| *** r-daneel has joined #openstack-infra | 19:47 | |
| *** mrmartin has quit IRC | 19:48 | |
| *** Bsony has quit IRC | 19:48 | |
| *** dprince has quit IRC | 19:49 | |
| *** MaxV has joined #openstack-infra | 19:51 | |
| *** zz_jgrimm is now known as jgrimm | 19:51 | |
| *** tonytan4ever has joined #openstack-infra | 19:51 | |
| *** Rockyg has joined #openstack-infra | 19:51 | |
| jogo | clarkb: want to +W the bottom of this series https://review.openstack.org/#/c/135768 | 19:53 |
| *** dkranz has quit IRC | 19:54 | |
| *** MaxV has quit IRC | 19:55 | |
| jeblair | sdague: cool, let me see if i can find something tractable in the logs, or if it's too entangled in the brokeness | 19:55 |
| *** sputnik13 has quit IRC | 19:55 | |
| *** e0ne is now known as e0ne_ | 19:55 | |
| *** nelsnelson has quit IRC | 19:56 | |
| *** sputnik13 has joined #openstack-infra | 19:56 | |
| *** nelsnelson has joined #openstack-infra | 19:56 | |
| *** EmilienM is now known as EmilienM|afk | 19:56 | |
| *** reed has joined #openstack-infra | 19:56 | |
| *** harlowja_ has quit IRC | 19:57 | |
| *** sdake_ has quit IRC | 19:57 | |
| *** devlaps has joined #openstack-infra | 19:58 | |
| *** sdake_ has joined #openstack-infra | 19:58 | |
| clarkb | jogo: I thought we were going to fix the cpumodel | 20:00 |
| clarkb | jogo: ideally we wouldn't test aiopcpu with i386 images unless there was no alternative | 20:00 |
| jogo | clarkb: I was going to save that for the next pass at things. | 20:01 |
| *** harlowja has joined #openstack-infra | 20:01 | |
| *** e0ne_ has quit IRC | 20:01 | |
| *** jaypipes has quit IRC | 20:01 | |
| jogo | clarkb: being this works and I want to get live migrationion testing landed | 20:01 |
| jogo | clarkb: then we can iterate | 20:01 |
| clarkb | jogo: except its still all experimental and you can iterate fine with depends on | 20:01 |
| jogo | clarkb: true depends on does help, didn't think of that | 20:02 |
| jogo | I do want to move this out of experimental sooner rather then later though | 20:02 |
| *** jaypipes has joined #openstack-infra | 20:02 | |
| jogo | just still unclear on how we do that | 20:02 |
| clarkb | I don't think we should do that with i386 test nodes | 20:02 |
| clarkb | or we should switch devstack proper to i386 | 20:02 |
| *** MaxV has joined #openstack-infra | 20:03 | |
| jogo | clarkb: whats the concern there? | 20:03 |
| jogo | clarkb: btw off topic, can you take a look at https://review.openstack.org/#/c/160933/ and its dependency | 20:04 |
| jogo | that should shave about 5 minutes off of all devstack jobs | 20:04 |
| jogo | (devstack has been taking 20 minutes recently) | 20:04 |
| clarkb | jogo: consistency, I don't want to have one or the other break and us spend days deugging because we forget that the architectures are different | 20:04 |
| clarkb | jogo: didn't jeblair -1 and say use the normal caching stuff? | 20:05 |
| jogo | clarkb: consistency of the cirros image being used you mean? | 20:05 |
| jogo | clarkb: yeah, mtreinish addressed that point I think | 20:05 |
| *** e0ne has joined #openstack-infra | 20:05 | |
| clarkb | jogo: of the image and the qemu emulation | 20:05 |
| *** dhritishikhar has quit IRC | 20:05 | |
| clarkb | jogo: but yours is doing a copy still? oh I see because it goes in files/ | 20:06 |
| clarkb | jogo: you should use mv | 20:06 |
| *** rkukura has quit IRC | 20:06 | |
| jogo | clarkb: mtreinish thought a cp was better because we may need something for the grenade case | 20:06 |
| jogo | clarkb: but I can make that a mv again. either way | 20:06 |
| clarkb | jogo: hrm devstack/grenade should use the same wheelhouse imo | 20:06 |
| *** dkranz has joined #openstack-infra | 20:06 | |
| jogo | clarkb: yeah, it should, but AFAIK doesn't right now | 20:07 |
| clarkb | regardless of new vs old | 20:07 |
| jogo | sdague: ^ | 20:07 |
| jogo | well old doesn't use a wheelhouse at all ... yet | 20:07 |
| mtreinish | clarkb: devstack uses $DEST/.wheelhouse as the default path for the wheelhouse | 20:07 |
| *** achanda has quit IRC | 20:07 | |
| clarkb | mtreinish: maybe it should use a different path then? | 20:07 |
| mtreinish | jogo: I guess we could just add a WHEELHOUSE=$path to the localrc | 20:07 |
| jogo | maybe we should first cahnge it to $HOME/.wheelhouse? | 20:07 |
| *** jedimike has quit IRC | 20:07 | |
| clarkb | I am trying to think if two would be required for some reason | 20:10 |
| *** tsg has quit IRC | 20:10 | |
| jogo | clarkb: I can't think of any reason | 20:10 |
| clarkb | but pip should do resolution properly in one as long as that single wheelhouse has the deps for both | 20:10 |
| *** tsg has joined #openstack-infra | 20:10 | |
| jogo | dtroyer: ping ^ | 20:10 |
| mtreinish | clarkb: it should based on how the caching script is written | 20:10 |
| *** MaxV has quit IRC | 20:10 | |
| mtreinish | it should make one big wheelhouse with all of the requirements in all the branches | 20:11 |
| clarkb | I wonder if it would just be simpler to mirror wheels | 20:11 |
| *** yamahata has quit IRC | 20:11 | |
| * jogo wonders that as well | 20:11 | |
| mtreinish | clarkb: well, devstack is running pip wheel at the beginning of it's run. Would that work with mirroring them? | 20:12 |
| *** funzo has joined #openstack-infra | 20:12 | |
| waynr | zaro: cool, with your blessing I will begin working on adapting my internal puppetlabs presentation for the upcoming JUC | 20:13 |
| *** andreykurilin_ has joined #openstack-infra | 20:13 | |
| *** dustins has quit IRC | 20:13 | |
| *** claudiub has joined #openstack-infra | 20:13 | |
| sdague | jeblair: thanks | 20:14 |
| jogo | clarkb: well does bandersnatch support wheel mirroring? | 20:15 |
| jogo | for our own wheels | 20:15 |
| clarkb | jogo: no, we would have to resurrect pypi-mirror or similar | 20:15 |
| sdague | clarkb: where would it mirror from? | 20:15 |
| jogo | clarkb: in that case, my hunch is that is more work then this | 20:15 |
| greghaynes | :) | 20:15 |
| *** tsg has quit IRC | 20:16 | |
| greghaynes | my pypi-mirror is still running strong | 20:16 |
| clarkb | sdague: our existing pypi mirrors | 20:16 |
| jogo | sdague: did you see the backscroll, wondering about devstack's default wheel dir | 20:16 |
| sdague | clarkb: we have wheels for all the things? | 20:16 |
| sdague | jogo: honestly, ask dtroyer | 20:16 |
| jogo | sdague: we *can* have wheels for all the things | 20:16 |
| jogo | sdague: guh, devstack takes 20 minutes to run now | 20:17 |
| jogo | and caching wheels shaves 5 minutes off | 20:17 |
| sdague | ok, anyway, I can't look at this one today | 20:17 |
| jogo | clarkb: how about this. I'll change my patch to do a mv | 20:17 |
| jogo | and we can sort out the devstack default dir when dtroyer is around | 20:18 |
| mtreinish | anteaya: how much is this really happening?: http://www.businessinsider.com/bank-of-canada-stop-drawing-leonard-nimoy-spock-on-money-2015-3 | 20:18 |
| * dtroyer catches up | 20:18 | |
| jogo | oh hai dtroyer | 20:18 |
| anteaya | mtreinish: ha ha ha | 20:18 |
| anteaya | usually with some frequency | 20:18 |
| *** MaxV has joined #openstack-infra | 20:18 | |
| anteaya | right now probably quite a bit | 20:18 |
| dtroyer | so I think a mv is safe, setting WHEELHOUSE might be just as easy though | 20:18 |
| anteaya | except the new money is plastic to doesn't take the ink | 20:19 |
| jogo | dtroyer: what about the default wheelhouse location | 20:19 |
| dtroyer | grenade itself isn't going to care, once $OLD does wheels, it just needs to have the right versions available, doesn't care where it comes from | 20:19 |
| anteaya | that is sir wilfred laurier under the ink | 20:19 |
| anteaya | the usual presider over the $5 | 20:19 |
| dtroyer | the location is arbitrary. I set the default so it was sane for developers | 20:19 |
| jogo | dtroyer: but by default it looks for them in $OLD/.wheelhouse | 20:20 |
| dtroyer | how? we don't have anything in a stable devstack doing this yet | 20:20 |
| mtreinish | jogo: that's what I was saying just put WHEELHOUSE in the localrcs | 20:20 |
| dtroyer | also, a wheel mirror should work just fine if you can come up with one ;) | 20:21 |
| mtreinish | anteaya: heh, ok, it seemed pretty cool to me. But, I was just wondering if it was something that got blown out of proportion by the press | 20:21 |
| dtroyer | the entire point of doing this is to only build c-dependant packages one time, pays off with N venvs down the road | 20:21 |
| mtreinish | so the bank of canada had to issue a statement | 20:21 |
| anteaya | nope probably not | 20:21 |
| anteaya | we have been doing it for years | 20:22 |
| anteaya | if you look back in the tc meeting logs I mentioned it when trying to build support for laurier | 20:22 |
| anteaya | as an L name | 20:22 |
| anteaya | if the bank of canada decided to issue a statement they aren't doing it for kicks | 20:22 |
| anteaya | the spock five | 20:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Zuul: log gerrit at debug level https://review.openstack.org/161396 | 20:23 |
| jeblair | sdague: ^ and also https://storyboard.openstack.org/#!/story/2000189 | 20:23 |
| dtroyer | sdague: right now wheels are built for a short list of annoying packages in files/venv-requirements.txt, it didn't seem worthwhile to do everything, at least for typical dev use | 20:23 |
| greghaynes | clarkb: what was the motivation for switching off pypi-mirror? | 20:23 |
| greghaynes | was it adding deps? | 20:23 |
| anteaya | bank of canada doesn't do anything it doesn't have to in terms of press releases | 20:24 |
| *** MaxV has quit IRC | 20:27 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: If wheelhouse is pre-generated, set WHEELHOUSE in localrc https://review.openstack.org/160933 | 20:27 |
| *** doude_ has joined #openstack-infra | 20:27 | |
| jogo | mtreinish dtroyer: like this | 20:27 |
| jogo | dtroyer clarkb: also mtreinish's patch to actually generate the wheelhouse needs a review or two https://review.openstack.org/#/c/160548/ | 20:27 |
| anteaya | mtreinish: the bank of canada takes care of printing the money, they aren't a retail bank | 20:28 |
| *** doude has quit IRC | 20:28 | |
| anteaya | mtreinish: they set interest rates though and the other big 5 national banks usually follow suit pretty closely | 20:28 |
| mtreinish | jogo: yeah I think that should work | 20:28 |
| openstackgerrit | Maru Newby proposed openstack-infra/devstack-gate: Revert "Add features.yaml entry for 'no services'" https://review.openstack.org/161402 | 20:28 |
| jogo | mtreinish: cool, and that solves the grenade case as well | 20:29 |
| *** dims has quit IRC | 20:29 | |
| mtreinish | anteaya: yeah I realized that. They're closer to the fed in the us. (except they print money too) | 20:29 |
| *** dims has joined #openstack-infra | 20:29 | |
| *** kmartin has quit IRC | 20:29 | |
| clarkb | greghaynes: it was so we didn't have to manually build the entire thing and manage that list | 20:29 |
| anteaya | yup | 20:29 |
| *** chlong has joined #openstack-infra | 20:30 | |
| anteaya | mtreinish: glad you knew that :) | 20:30 |
| *** andreykurilin_ has quit IRC | 20:30 | |
| greghaynes | clarkb: ah, it wasnt just in a cron? | 20:30 |
| clarkb | greghaynes: no, you have to have a list of packages to mirror | 20:30 |
| *** MaxV has joined #openstack-infra | 20:30 | |
| clarkb | greghaynes: and you have to build it on the distros and releases you support (at least for wheels) | 20:30 |
| greghaynes | ok, so what im wondering is why not hit the common case (global-reqs) and then also have bandersnatch as an index | 20:31 |
| greghaynes | so itll fall back | 20:31 |
| anteaya | sdague: to be clear the only part I am hoping you convey to operators is http://specs.openstack.org/openstack/neutron-specs/specs/kilo/migration-from-nova-net.html | 20:31 |
| anteaya | sdague: the merged part | 20:31 |
| anteaya | sdague: the part you just rechecked is in negotiation | 20:31 |
| anteaya | and the lastest is that the current direction is not supported by nova | 20:32 |
| anteaya | so we are in deep flux right now | 20:32 |
| anteaya | hoping to work that out this week and next | 20:32 |
| anteaya | but only the published part is what is agreed upon, the specs.o.o material | 20:32 |
| clarkb | greghaynes: because then you have to do what I describe | 20:32 |
| clarkb | we dont want to do that | 20:32 |
| *** rkukura has joined #openstack-infra | 20:34 | |
| sdague | anteaya: yeh, so the first part seems mostly background, I was just reading it and the unmerged bi | 20:35 |
| sdague | bit | 20:35 |
| *** MaxV has quit IRC | 20:35 | |
| anteaya | sdague: sure | 20:35 |
| anteaya | just know that is based on the idea of a proxy in nova | 20:35 |
| sdague | right | 20:35 |
| anteaya | which is deeply unsupported by dansmith | 20:35 |
| anteaya | so once we figure out the new direction, I expect that spec to completely change | 20:35 |
| anteaya | in the how portion, not so much the what | 20:36 |
| sdague | VM impact scope and the sideways matrix (from what to what is going to be supported) seems like the meaty parts | 20:36 |
| dansmith | anteaya: I'm -1 on the what too :) | 20:36 |
| sdague | those I feel comfortable with | 20:36 |
| sdague | so... we're deep into liberty then I assume for doing this, if the spec hasn't gotten agreement yet | 20:36 |
| dansmith | anteaya: the notion of a proxy in/from/near nova isn't so much the problem with the "how" | 20:36 |
| dansmith | anteaya: I talked to mikal last night and we seemed on the same page regarding all that stuff, I think he was going to sync back up with you/others | 20:37 |
| anteaya | dansmith: awesome | 20:37 |
| anteaya | he hasn't yet | 20:37 |
| anteaya | do go ahead | 20:37 |
| anteaya | what is the lastest | 20:37 |
| dansmith | I meant, I think he's on the same page as I am, so "the latest" hasn't changed, AFAIK | 20:38 |
| anteaya | sdague: oh yes, we have redirected to L now | 20:38 |
| anteaya | great | 20:38 |
| anteaya | so what page are you on | 20:38 |
| dansmith | ? | 20:38 |
| anteaya | pretend I'm an idoit | 20:38 |
| anteaya | idiot | 20:39 |
| dansmith | you keep telling people what page I'm on, so... | 20:39 |
| dansmith | mkay | 20:39 |
| anteaya | support | 20:39 |
| anteaya | I thought I knew | 20:39 |
| dansmith | I don't think that this is something we should be doing/spending time on at all | 20:39 |
| dansmith | that's the first thing | 20:39 |
| dansmith | but if we're going to, | 20:39 |
| anteaya | let's start with the first thing | 20:39 |
| anteaya | what is your vision of the world if we don't spend time on the migration | 20:40 |
| sdague | anteaya: yeh, so I think I have enough background on what's written, no glaring questions come up | 20:40 |
| *** ZZelle_ has joined #openstack-infra | 20:40 | |
| anteaya | sdague: thanks | 20:40 |
| *** achanda has joined #openstack-infra | 20:41 | |
| sdague | so, we should probably take this to either neutron or nova, and not filly up infra with it if we're going to keep going | 20:41 |
| anteaya | dansmith: tell me where you take it, I'll join you | 20:42 |
| dansmith | I didn't really expect to get into this conversation here | 20:42 |
| dansmith | I just keep getting pings of anteaya telling people I hate everything, so ... :) | 20:42 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-log_processor: Retry on EAI_AGAIN name resolution failures https://review.openstack.org/161409 | 20:42 |
| clarkb | jeblair: mordred ^ I am not proud of that but it may be the simplest fix for now | 20:43 |
| *** baoli_ has quit IRC | 20:43 | |
| *** MaxV has joined #openstack-infra | 20:43 | |
| mtreinish | SpamapS: the last time I did a cloud deploy I named it stratocumuliform, mostly because I feel clouds should be named after real clouds | 20:43 |
| anteaya | dansmith: I said you didn't support it | 20:43 |
| dansmith | anteaya: hyperbole added for effect :) | 20:43 |
| anteaya | well you and I don't ahve a good history of getting along, I am really am trying to improve that | 20:44 |
| clarkb | jeblair: mordred: also I need to figure out how to test that locally without completely hosing my machine | 20:44 |
| mordred | clarkb: docker? | 20:44 |
| clarkb | mordred: except docker breaks resolv.conf | 20:44 |
| *** esker has joined #openstack-infra | 20:44 | |
| anteaya | so I am trying here, but if you don't even support the direction of a migration, I would like to hear your thoughts | 20:44 |
| clarkb | mordred: so I don't think I can do that, but will give it a try | 20:44 |
| mordred | clarkb: sorry - I was total snark | 20:45 |
| mordred | ignore me | 20:45 |
| mordred | clarkb: change +2 | 20:45 |
| greghaynes | yeaaa how can docker even work for that, its a syscall | 20:45 |
| clarkb | greghaynes: well it will still refer to the /etc/resolv.conf in the container | 20:45 |
| clarkb | greghaynes: problem is docker bind mounts that in read only and sets it to google dns iirc | 20:46 |
| greghaynes | oh wow, I always assumed because the namespacing magic didnt exist for that | 20:47 |
| notmyname | clarkb: FYI, based on your conversation with tsg earlier around the pyeclib and dependencies, this is something that's critical for having EC in Swift for Kilo. The Mar 19 dependency freeze is what I'm looking at. it's a risk to ec in kilo | 20:48 |
| notmyname | clarkb: so I'm working with tsg on it and helping any way I can | 20:48 |
| mordred | should I read scrollback on that? | 20:48 |
| clarkb | mordred: probably | 20:49 |
| openstackgerrit | Ian Cordasco proposed openstack-infra/system-config: Add logging to #openstack-ansible https://review.openstack.org/161412 | 20:49 |
| *** otter768 has joined #openstack-infra | 20:50 | |
| openstackgerrit | Max Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Remove nonexistent servername and cleanup build publisher https://review.openstack.org/156091 | 20:50 |
| openstackgerrit | Maru Newby proposed openstack-infra/project-config: Clean up run-tox.sh to make it easier to reuse https://review.openstack.org/161414 | 20:50 |
| SpamapS | mtreinish: I love that .. http://en.wikipedia.org/wiki/Pileus_(meteorology) would be my favorite. ;) | 20:50 |
| clarkb | mordred: there is a new C lib that they wrote for it that isn't in ubuntu | 20:50 |
| SpamapS | "A small cap-like cloud over a cumulonimbus" | 20:50 |
| clarkb | mordred: we don't use PPAs in the gate | 20:51 |
| clarkb | mordred: how test | 20:51 |
| *** MaxV has quit IRC | 20:51 | |
| SpamapS | "Nothing to see here, just hanging out over a terrifying thunderstorm" | 20:51 |
| openstackgerrit | Sumit Naiksatam proposed openstack-infra/project-config: Enable functional test job in CI for neutron-fwaas https://review.openstack.org/161415 | 20:51 |
| mordred | clarkb: ah | 20:51 |
| mordred | clarkb: so - the first real-world example of "the project would like to express a c-depend that is not in the distros yet" | 20:51 |
| mtreinish | SpamapS: heh nice. You should put it on the surveymonkey, it'd get my vote | 20:51 |
| clarkb | mtreinish: yes | 20:51 |
| clarkb | er mordred ^ | 20:51 |
| mordred | notmyname, clarkb: link to the lib? | 20:52 |
| notmyname | mordred: getting it. know that it's not simple though :-) | 20:52 |
| notmyname | mordred: https://pypi.python.org/pypi/PyECLib/0.9.8 is on pypi (and repo links are there too) | 20:53 |
| mordred | notmyname: thanks | 20:53 |
| notmyname | mordred: but 0.9.8 is bad, not the least because it pulls stuff from the internet on install :-( | 20:53 |
| *** dmsimard is now known as dmsimard_away | 20:53 | |
| notmyname | mordred: however the 1.0 upstream repo is better. and has split some stuff into a c library | 20:53 |
| notmyname | mordred: liberasure (the C part) is in debian, but didn't get picked up by ubuntu in the last minor patch rev | 20:54 |
| notmyname | mordred: and so getting that stuff straightened out is what's currently going on | 20:54 |
| SpamapS | mtreinish: I'm kind of hoping we just name it infra cloud. :) | 20:54 |
| SpamapS | mtreinish: even though Pileus is more fun. :) | 20:54 |
| clarkb | SpamapS: I have a strong vote for vanilla | 20:54 |
| *** Bsony has joined #openstack-infra | 20:55 | |
| clarkb | I mean vanilla ice cream, vanilla coke, vanilla cloud: what do they all have in common? awesomeness | 20:55 |
| *** otter768 has quit IRC | 20:55 | |
| *** kmartin has joined #openstack-infra | 20:55 | |
| notmyname | mordred: current CI tests are based on 0.9.8 and are on the jenkins slaves, so we're so-so ok for now. but not for a release | 20:55 |
| mordred | vanilla coke is awesome | 20:55 |
| clarkb | SpamapS: also it bugs me that surveymonkey does not tell you what the numbers mean | 20:55 |
| mordred | notmyname: yeah- ok, so it's in debian at least | 20:55 |
| SpamapS | vanilla is also often misunderstood | 20:56 |
| SpamapS | Seeing as it is super super rare in it's natural form, it's weird that we use it to mean "the same as everything else" | 20:56 |
| notmyname | mordred: right. so I'm tracking it right now as part of the EC work to get done by kilo. since it's a dependency, I'm looking at the dep freeze date as my deadline | 20:56 |
| SpamapS | or "without specialness" | 20:56 |
| SpamapS | because vanilla is really hard to cultivate and find. | 20:56 |
| SpamapS | Fortunately, vanilin is really easy to synthesize. | 20:57 |
| notmyname | mordred: and since it's basically the part that actually does the erasure code math, it's pretty important to the feature in swift ;-) | 20:57 |
| mordred | notmyname: hehe | 20:57 |
| jeblair | SpamapS, clarkb: we have the name of the effort, but we also get to name the two installs of it as well. | 20:57 |
| notmyname | mordred: anyway, I know tsg and clarkb have been working on it. so my comments are more of an "infra, FYI, kinda important think you might see me yell about" thing | 20:58 |
| *** ddieterly has joined #openstack-infra | 20:58 | |
| jeblair | SpamapS, clarkb: so if we don't name the effort vanilla, we can name one of the installations vanilla | 20:58 |
| clarkb | mordred: I think I will just have to use a VM to test this EAI_AGAIN behavior | 20:58 |
| clarkb | mordred: I can't figure a sane way to docker it | 20:58 |
| notmyname | jeblair: isnt' vanilla the name of the openstack V release? | 20:58 |
| clarkb | jeblair: +1 :) | 20:58 |
| openstackgerrit | Maru Newby proposed openstack-infra/project-config: Move neutron api job definition to neutron.yaml https://review.openstack.org/161418 | 20:58 |
| jeblair | notmyname: nice... :) | 20:58 |
| clarkb | jeblair: before I go testing https://review.openstack.org/#/c/161409/1 is that something you would be ok with as an intermediate step? | 20:58 |
| clarkb | jeblair: I don't want to go spend time constructing a largish system test for this thing if you will -2 :) | 20:59 |
| *** dustins has joined #openstack-infra | 20:59 | |
| *** Bsony has quit IRC | 20:59 | |
| *** dizquierdo has joined #openstack-infra | 20:59 | |
| SpamapS | 6 responses so far | 21:00 |
| *** sdake_ has quit IRC | 21:00 | |
| *** kmartin has quit IRC | 21:00 | |
| jeblair | clarkb: that's okay. i feel like actually being able to specify boot order is not an insane thing for us to work on. | 21:01 |
| marun | jeblair: when you get a chance, can you please confirm that adding the neutron functional job -nv to the d-g check pipeline is ok as we discussed? https://review.openstack.org/#/c/159672/ | 21:01 |
| jeblair | clarkb: but +2 if that's what you want | 21:01 |
| clarkb | jeblair: ya specifying a boot order is something else we should work on, but I expect it to take more time as we figure out how to get puppet, update-rc.d, and whatever we come up with to work together | 21:01 |
| jeblair | marun: will do | 21:02 |
| *** ParsectiX has quit IRC | 21:02 | |
| marun | jeblair: thank you! | 21:02 |
| *** ParsectiX has joined #openstack-infra | 21:02 | |
| *** _nadya_ has quit IRC | 21:03 | |
| *** rkukura has quit IRC | 21:05 | |
| *** anthonyper has quit IRC | 21:05 | |
| clarkb | ugh hpcloud is still a mess | 21:05 |
| clarkb | OverLimit: Maximum number of ports exceeded (HTTP 413) (Request-ID: req-377aafe6-0585-43c0-819a-67158fab6968) | 21:05 |
| clarkb | theory time, something else is causing resources to leak | 21:05 |
| * clarkb goes to figure out how to unleak ports | 21:05 | |
| jeblair | we don't explicitly allocate those, do we? | 21:05 |
| clarkb | jeblair: we do not | 21:05 |
| openstackgerrit | Sumit Naiksatam proposed openstack-infra/project-config: Create experimental job for neutron-fwaas https://review.openstack.org/161415 | 21:06 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Drop ironic tempest regex, stop running all of Tempest https://review.openstack.org/161420 | 21:06 |
| clarkb | also Conflict: Instance 717da90b-516f-45b8-a2e3-dfc1462f277f is locked (HTTP 409) but not sure if that is on an hpcloud node yet | 21:06 |
| openstackgerrit | Maru Newby proposed openstack-infra/project-config: Remove use of DEVSTACK_GATE_TESTR_ARTIFACT_TARGET https://review.openstack.org/161421 | 21:06 |
| clarkb | yup ^ is hpcloud too | 21:06 |
| clarkb | port-list confirms we are basically at peak port | 21:07 |
| clarkb | but port-list doesn't seem to show any association to nodes | 21:07 |
| openstackgerrit | Diego proposed openstack-infra/infra-manual: Add details on using git-review over https https://review.openstack.org/161078 | 21:07 |
| clarkb | so not sure how you are supposed to figure out which ports are in use and which aren't, possibly by using the ip addrs as a key? | 21:07 |
| greghaynes | jhesketh: Hey, id like to pop https://review.openstack.org/#/c/119533/ out of the patch stack | 21:08 |
| greghaynes | er, sorry | 21:08 |
| greghaynes | oh, wait, no, thats right :) | 21:08 |
| *** rkukura has joined #openstack-infra | 21:09 | |
| clarkb | does anyone know of a better way to map ports to hosts so I can delete ports that are not assocaited with nodes? | 21:09 |
| *** Sukhdev has quit IRC | 21:10 | |
| *** Longgeek has quit IRC | 21:10 | |
| clarkb | ok nodepool and hpcloud are mostly in agreement over the number of nodes ~offby20 | 21:13 |
| *** anthonyper has joined #openstack-infra | 21:13 | |
| openstackgerrit | Maru Newby proposed openstack-infra/devstack-gate: Remove configurable testr artifact processing https://review.openstack.org/161422 | 21:13 |
| clarkb | which means we have leaked somewhere around 100 ports | 21:13 |
| *** e0ne has quit IRC | 21:14 | |
| clarkb | there doesn't seem to be an easy way to map fix ip to ndoe either | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| *** aysyd has quit IRC | 21:16 | |
| *** dkranz has quit IRC | 21:17 | |
| *** alexpilotti has quit IRC | 21:17 | |
| clarkb | I can list the ports that belong to routers... | 21:17 |
| sdague | marun: https://review.openstack.org/#/c/156214/ - now that your other fixes are in, do you think that's still -1 | 21:17 |
| *** rkukura has quit IRC | 21:17 | |
| marun | sdague: it should be good to go | 21:18 |
| *** tsg has joined #openstack-infra | 21:18 | |
| *** achanda has quit IRC | 21:19 | |
| *** mbacchi has quit IRC | 21:20 | |
| *** tsg_ has joined #openstack-infra | 21:20 | |
| *** rfolco has quit IRC | 21:20 | |
| *** sandywalsh has quit IRC | 21:23 | |
| *** tsg has quit IRC | 21:23 | |
| sdague | jeblair: oh, in case you needed a data point - https://review.openstack.org/#/c/160407/ is a review where author was reset in git, but stays the same in gerrit. | 21:24 |
| *** sandywalsh has joined #openstack-infra | 21:24 | |
| *** doude_ has quit IRC | 21:24 | |
| *** kmartin has joined #openstack-infra | 21:26 | |
| *** Rockyg has quit IRC | 21:26 | |
| *** patrickeast has quit IRC | 21:29 | |
| clarkb | ok neutronclient port-list by default has heavily editorialized what it shows you | 21:29 |
| clarkb | you can hwoever pass in an explicit list of columns via port-list -c | 21:29 |
| clarkb | the list seems to match up with what we have at http://developer.openstack.org/api-ref-networking-v2.html#ports | 21:30 |
| clarkb | that would be all fine and dandy if the client actually docuemnted this behavior :( anyways onward, new info indicates we may not actually be leaking ports | 21:31 |
| clarkb | we seem to have a device id for each port | 21:31 |
| *** patri582 has quit IRC | 21:31 | |
| *** ChuckC has joined #openstack-infra | 21:32 | |
| *** patrickeast__ has quit IRC | 21:32 | |
| *** dizquierdo has quit IRC | 21:32 | |
| *** patrickeast_ has quit IRC | 21:32 | |
| *** patri251 has quit IRC | 21:32 | |
| *** patri498 has quit IRC | 21:32 | |
| *** patri461 has quit IRC | 21:33 | |
| clarkb | I do see a lot of hpcloud timeouts | 21:34 |
| *** patrickeast has joined #openstack-infra | 21:34 | |
| clarkb | and if that nova/neutron floating ip leak is related to load perhaps we are seeing that happen more because thigns are slow? I am digging through more logs to see if I see anything | 21:35 |
| *** patrickeast_ has joined #openstack-infra | 21:35 | |
| *** patrickeast__ has joined #openstack-infra | 21:35 | |
| *** patri255 has joined #openstack-infra | 21:35 | |
| *** patri910 has joined #openstack-infra | 21:35 | |
| *** patri169 has joined #openstack-infra | 21:35 | |
| *** patri733 has joined #openstack-infra | 21:35 | |
| mriedem | fungi: py33 here wouldn't work right? http://git.openstack.org/cgit/openstack/glance_store/tree/tox.ini#n3 - i mean, we don't have py33 jobs anymore i thought | 21:38 |
| *** tiswanso has quit IRC | 21:39 | |
| clarkb | mriedem: its fine to have, we just won't exercise it | 21:40 |
| *** bswartz has quit IRC | 21:41 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Story events may now be filtered by event_type. https://review.openstack.org/160462 | 21:41 |
| *** EmilienM|afk is now known as EmilienM | 21:43 | |
| krtaylor | clarkb, do you know of any way to manually push a meeting log to eavesdrop.openstack.org ? | 21:44 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: New job for in-tree lbaasv2 tempest tests https://review.openstack.org/161432 | 21:44 |
| krtaylor | the openstack bot wasn't running during our meeting this morning | 21:44 |
| clarkb | krtaylor: other than an infra root copy pastaing logs from soewhere no | 21:45 |
| *** melwitt has joined #openstack-infra | 21:45 | |
| krtaylor | clarkb, I have the log, and it is still in channel | 21:45 |
| clarkb | krtaylor: it is still in channel? | 21:46 |
| *** bswartz has joined #openstack-infra | 21:46 | |
| krtaylor | clarkb, yes | 21:46 |
| clarkb | I don't understand what that means | 21:46 |
| krtaylor | #openstack-meeting-4 scrollback | 21:46 |
| krtaylor | at least my scrollback log anyway | 21:47 |
| clarkb | ya scrollback is client independent | 21:47 |
| clarkb | krtaylor: if you post the logs somewhere I can get them I can see what I can do | 21:47 |
| krtaylor | I guess I could just paste it in a little bit at a time | 21:47 |
| krtaylor | as long as no meeting is running | 21:47 |
| *** dustins_ has joined #openstack-infra | 21:47 | |
| clarkb | that would be one way I guess, sort of a meta log but probably will mostly work | 21:48 |
| *** dustins has quit IRC | 21:48 | |
| krtaylor | clarkb, if that would be easier for you, I'll give it a try | 21:49 |
| *** dustins_ has quit IRC | 21:49 | |
| dolphm | to whoever finally implemented Depends-On in commit messages: <3 | 21:49 |
| clarkb | krtaylor: well the benefit of replaying like that is you can get the bot to build the summary for you and all that | 21:49 |
| clarkb | krtaylor: if I was given a text log I would just stick that on the server without the summarization | 21:49 |
| *** Ryan_Lane has joined #openstack-infra | 21:49 | |
| krtaylor | clarkb, good point, will do | 21:50 |
| anteaya | dolphm: well that would be a whole lot of jeblair and a good helping of clarkb and fungi | 21:50 |
| anteaya | dolphm: and glad you like it | 21:50 |
| dolphm | jeblair: clarkb: fungi: thank you! https://twitter.com/dolphm/status/573238525203648512 | 21:50 |
| clarkb | krtaylor: I am happy to stick the log file in though if that is easiest :) either way works for me | 21:51 |
| lifeless | can depends-on be used within a single repo ? | 21:51 |
| *** MaxV has joined #openstack-infra | 21:51 | |
| clarkb | lifeless: yes, but you should likely use git deps instead | 21:52 |
| jeblair | lifeless: (and don't make a cycle) | 21:52 |
| lifeless | clarkb: why? | 21:52 |
| *** dhellmann has quit IRC | 21:52 | |
| lifeless | jeblair: obviously :) | 21:52 |
| clarkb | lifeless: so that your local testing does what you expect and for the additional checks provided by gerrit | 21:52 |
| dolphm | clarkb: can you use multiple Depends-On in a single commit? | 21:53 |
| lifeless | clarkb: git deps cause a lot of friction in deep stacks that are not always textually dependent | 21:53 |
| clarkb | dolphm: you can | 21:53 |
| lifeless | clarkb: your answer isn't a particularly satisfying one to me | 21:53 |
| clarkb | dolphm: just list multiple depends-on on multiple lines | 21:53 |
| clarkb | lifeless: ok | 21:53 |
| dolphm | clarkb: =D | 21:53 |
| lifeless | clarkb: what additional checks do we give up when we use depends-on ? | 21:53 |
| *** koolhead_ has quit IRC | 21:53 | |
| *** rkukura has joined #openstack-infra | 21:54 | |
| clarkb | lifeless: the gerrit sanity checks when pushing a stack is one | 21:54 |
| *** baoli has joined #openstack-infra | 21:54 | |
| clarkb | the did anything change checks | 21:54 |
| clarkb | lifeless: you also get merge checks though those aren't well exposed in current gerrit but will be when we upgrade | 21:55 |
| jeblair | lifeless: also, it might be nice to reflect that dependency in the git graph itself | 21:55 |
| *** dhellmann has joined #openstack-infra | 21:55 | |
| jeblair | lifeless: but we do support it so if you find a good reason to, you can :) | 21:55 |
| clarkb | the more I look at nodepool logs the more I think floating ip leak is a symptom not a cause of the problems | 21:56 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add microsecond columns to the test_runs table https://review.openstack.org/160057 | 21:56 |
| clarkb | numerous timeouts for many operations appears to be the larger issue and that did not go away with the floating ip cleanup | 21:56 |
| lifeless | jeblair: sure; I'm thinking of a couple of scenarios; one is when two people have divide-and-conquered a problem, e.g. A makes an API and B uses it, where the continual rebase-churn placed on B is quite a burden | 21:56 |
| *** dhellmann has quit IRC | 21:56 | |
| *** dhellmann has joined #openstack-infra | 21:57 | |
| lifeless | the second one is the reviewer impact on stacks of 5 or 10 deep patches where the bottom one has maybe one meaningful change per 10 or 20 updates | 21:57 |
| *** penguinRaider has joined #openstack-infra | 21:57 | |
| *** pelix has quit IRC | 21:57 | |
| jeblair | lifeless: *nod* let me know how it goes? :) | 21:57 |
| *** bswartz has quit IRC | 21:58 | |
| *** kgiusti has left #openstack-infra | 21:58 | |
| *** harlowja has quit IRC | 21:58 | |
| lifeless | jeblair: if and when we try, yes | 21:58 |
| jogo | when is the next nodepool image built? | 21:58 |
| *** sarob has joined #openstack-infra | 21:58 | |
| *** belmoreira has joined #openstack-infra | 21:58 | |
| clarkb | jogo: 1400UTC is when they start | 21:58 |
| jogo | so its too late to land https://review.openstack.org/#/c/160548/ until the next one | 21:59 |
| jogo | clarkb: I would really like to get https://review.openstack.org/#/c/160548/ and https://review.openstack.org/#/c/160933/ landed | 21:59 |
| jogo | as devstack is taking way too long right now | 21:59 |
| jogo | and this will shave off a lot of time from our jobs | 21:59 |
| clarkb | ok will review | 21:59 |
| *** stevemar2 has joined #openstack-infra | 21:59 | |
| *** stevemar has quit IRC | 22:00 | |
| jogo | jeblair: ^ | 22:00 |
| clarkb | so I think there is a major bug in 160548, that doesn't run in the chroot | 22:00 |
| jogo | clarkb: very possibly | 22:00 |
| jogo | mtreinish: ^ | 22:00 |
| * mkj_wk adds unit tests... | 22:00 | |
| *** crc32 has joined #openstack-infra | 22:01 | |
| clarkb | greghaynes: can you confirm my -1 on https://review.openstack.org/#/c/160548/ ? | 22:02 |
| *** Ryan_Lane has quit IRC | 22:02 | |
| jogo | clarkb: not sure where the chroot would go, don't see anything around the download command etc. | 22:02 |
| stevebaker | I'd really like to try this out, any chance for some reviews? https://review.openstack.org/#/c/159686/ | 22:02 |
| *** tsg_ has quit IRC | 22:02 | |
| clarkb | jogo: dib manages the chroot, we can get away with it when just downloading things because we aren't building packages. but wheels are essentially release specific packages so should go in the chroot'd steps of dib | 22:03 |
| *** tsg_ has joined #openstack-infra | 22:03 | |
| *** ivar-lazzaro has quit IRC | 22:03 | |
| clarkb | stevebaker: does build-heat-test-image.sh install all of its dependencies? | 22:03 |
| clarkb | stevebaker: the bare-trusty nodes are pretty bare | 22:03 |
| stevebaker | clarkb: right now it is brazenly assuming diskimage-builder is installed | 22:04 |
| stevebaker | clarkb: but I could always fix that after ^ lands? | 22:04 |
| *** ddieterly has quit IRC | 22:04 | |
| clarkb | stevebaker: sure, I have a few other comments though so either way | 22:05 |
| anteaya | it is a post job | 22:05 |
| stevebaker | clarkb: ok. does bare-trusty have pip? | 22:05 |
| anteaya | so what happens to a merged patch if a job fails in post? | 22:05 |
| clarkb | stevebaker: I don't even think it has that | 22:05 |
| anteaya | any repurcussions? | 22:06 |
| clarkb | anteaya: it merges then the job fails | 22:06 |
| jeblair | dolphm: :) | 22:06 |
| anteaya | okay so low risk | 22:06 |
| *** patrickeast has quit IRC | 22:06 | |
| *** ddieterly has joined #openstack-infra | 22:07 | |
| clarkb | stevebaker: actually no, bare-trusty does haev pip | 22:07 |
| *** pc_m has quit IRC | 22:07 | |
| stevebaker | oh, good | 22:08 |
| clarkb | stevebaker: so thats probably fine if you have your script install that | 22:08 |
| *** tsg_ has quit IRC | 22:08 | |
| clarkb | er s/that/your deps/ | 22:08 |
| stevebaker | yep | 22:08 |
| dolphm | jeblair: \o/ | 22:08 |
| clarkb | we install tox via pip, not sure why I was thinking no pip on that image | 22:08 |
| *** esker has quit IRC | 22:09 | |
| greghaynes | clarkb: yep, thats correct | 22:09 |
| anteaya | clarkb: what file do I check to see what bare-trusty has on it? | 22:09 |
| stevebaker | clarkb: do I need to use sudo for pip installd? | 22:09 |
| stevebaker | installs | 22:09 |
| jeblair | anteaya, marun: devstack-gate job changes lgtm, aprvd, thanks much! | 22:09 |
| anteaya | openstack_functions.py does't seem to b it | 22:09 |
| *** patrickeast_ has quit IRC | 22:09 | |
| *** patrickeast__ has quit IRC | 22:09 | |
| anteaya | jeblair: thank you! | 22:09 |
| greghaynes | jogo: basically that needs to happen in one of the install.d phases | 22:10 |
| *** patri169 has quit IRC | 22:10 | |
| *** patri910 has quit IRC | 22:10 | |
| *** patri255 has quit IRC | 22:10 | |
| greghaynes | jogo: rather than extra-data.d | 22:10 |
| *** annegentle has quit IRC | 22:10 | |
| *** patri733 has quit IRC | 22:10 | |
| clarkb | stevebaker: yes, and you should have sudo there | 22:10 |
| clarkb | stevebaker: also left one comment for unrelated thing | 22:10 |
| clarkb | anteaya: the nodepool/scripts/ prepare scripts | 22:10 |
| anteaya | clarkb: thanks | 22:11 |
| *** patrickeast has joined #openstack-infra | 22:11 | |
| *** annegentle has joined #openstack-infra | 22:11 | |
| sdague | jeblair: does Depends-On work inside a single repo? | 22:11 |
| greghaynes | clarkb: did we determine that shipping wheelhouse with images is the way to go for now? | 22:11 |
| clarkb | greghaynes: the work is mostly done for it so I am fine with it for now | 22:11 |
| *** loquacities has joined #openstack-infra | 22:11 | |
| sdague | or would I be doing terribleness if I used it that way | 22:11 |
| clarkb | greghaynes: especially sine we aren't making several wheelhouses for grenade | 22:11 |
| *** ChuckC has quit IRC | 22:11 | |
| clarkb | sdague: lifeless just asked about that :) its fine | 22:11 |
| sdague | ok | 22:12 |
| clarkb | sdague: the other rules still apply though, no cycles etc | 22:12 |
| sdague | yep | 22:12 |
| *** patrickeast has quit IRC | 22:12 | |
| *** macjack has joined #openstack-infra | 22:12 | |
| *** patrickeast has joined #openstack-infra | 22:12 | |
| *** chlong has quit IRC | 22:12 | |
| *** patrickeast_ has joined #openstack-infra | 22:13 | |
| *** patrickeast__ has joined #openstack-infra | 22:13 | |
| *** patri582 has joined #openstack-infra | 22:13 | |
| *** patri455 has joined #openstack-infra | 22:13 | |
| *** patri696 has joined #openstack-infra | 22:13 | |
| jeblair | lifeless, sdague: oh, right another tradeoff is that you miss the dependency links in the gerrit interface (i mean, you can still click depends on in the commit message, but you can't go the other direction, and fewer automated tools understand it) | 22:13 |
| sdague | it's just going to help in stacking these micro versions changes in nova and making sure we merge them in the right order without pushing everyone into 1 branch | 22:14 |
| *** peristeri has quit IRC | 22:14 | |
| jeblair | lifeless, sdague: still may be worth the tradeoff it some cases, just worth being aware | 22:14 |
| sdague | jeblair: yeh | 22:14 |
| *** jamielennox|away is now known as jamielennox | 22:14 | |
| sdague | so I think what it actually does really neat here is allow for multiple parents on a patch | 22:14 |
| jeblair | yep | 22:14 |
| jeblair | (without being a merge commit) | 22:15 |
| sdague | because we have a patch series for each of these, and the last one actually lands the microversion. Then we can chain those in the right order | 22:15 |
| sdague | yep | 22:15 |
| *** radez is now known as radez_g0n3 | 22:15 | |
| jhesketh | Morning | 22:15 |
| greghaynes | jhesketh: hello there | 22:16 |
| *** ivar-lazzaro has joined #openstack-infra | 22:16 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: import aeromancer to stackforge https://review.openstack.org/161447 | 22:16 |
| jhesketh | greghaynes: hey, thanks for the abc work. Did you see my comment on one of them? | 22:17 |
| *** MaxV_ has joined #openstack-infra | 22:17 | |
| *** ivar-laz_ has joined #openstack-infra | 22:17 | |
| *** ivar-lazzaro has quit IRC | 22:17 | |
| greghaynes | jhesketh: yes, and I replied | 22:17 |
| greghaynes | I also left a new one | 22:17 |
| * greghaynes finds | 22:17 | |
| mtreinish | jogo: yeah I saw clarkb's comment I'll move the wheelhouse building into a script in install.d | 22:18 |
| jhesketh | greghaynes: awesome, I'll take a look :-) | 22:18 |
| anteaya | morning jhesketh | 22:18 |
| greghaynes | jhesketh: https://review.openstack.org/#/c/119530/16 is the question you asked | 22:18 |
| greghaynes | jhesketh: basically I dont think thats something we want to assert, in general | 22:19 |
| *** Ryan_Lane has joined #openstack-infra | 22:19 | |
| greghaynes | jhesketh: Id also like to remove https://review.openstack.org/#/c/119533/ from the patch stack because it shouldnt be needed if were using abc correctly | 22:19 |
| *** achanda has joined #openstack-infra | 22:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Make devstack-gate stable jobs non-voting https://review.openstack.org/159900 | 22:21 |
| *** MaxV has quit IRC | 22:21 | |
| jhesketh | greghaynes: okay I'll have a play with that later | 22:22 |
| *** e0ne has joined #openstack-infra | 22:22 | |
| openstackgerrit | Merged openstack-infra/project-config: Add the neutron func job (nv) to devstack-gate https://review.openstack.org/159672 | 22:22 |
| *** jgrimm is now known as zz_jgrimm | 22:23 | |
| greghaynes | ok, let me know because the parent of https://review.openstack.org/#/c/119533/ needs to reparented / updated | 22:23 |
| dhellmann | AJaeger, anteaya, jeblair : can someone please explain to me what I'm supposed to do to make https://review.openstack.org/#/c/161351/ land? Do I have to resubmit the governance patch to get a new change id? | 22:23 |
| clarkb | dhellmann: that is probably the biggest gotcha with depends-on right now, long story short either remove the depends on or change it | 22:24 |
| clarkb | dhellmann: OR unabandon the other change and merge it | 22:24 |
| *** hashar has quit IRC | 22:24 | |
| clarkb | but I am guessing it was abandoned for reasons | 22:24 |
| dhellmann | clarkb: I have 2 patches with the same change-id because that used to be the way to link related things together | 22:25 |
| openstackgerrit | Merged openstack-infra/nodepool: Better image checking in update_image command https://review.openstack.org/151749 | 22:25 |
| clarkb | dhellmann: it still is | 22:25 |
| dhellmann | I abandoned one, because I couldn't have Depends-On point to a change id that was the same as the current patch | 22:25 |
| anteaya | I had asked for a depends-on | 22:25 |
| dhellmann | so I resubmitted the original project-config patch with a new id | 22:25 |
| clarkb | dhellmann: depends on doesn't distinguish between branches though | 22:25 |
| anteaya | which is what we decided would be the workflow for additions to the openstack namespace | 22:25 |
| clarkb | oh hrm is there ever a reason to have a depends on self? | 22:25 |
| *** achanda has quit IRC | 22:26 | |
| * dhellmann grumbles | 22:26 | |
| clarkb | maybe I need to understand that. (its implied fwiw) | 22:26 |
| dhellmann | I'll go resubmit the governance patch with a new distinct id and use that one | 22:26 |
| anteaya | dhellmann: yeah this line: Depends-on: I95dc9a36e2e2635dfbd5902c620ce525b981a19b | 22:26 |
| *** dimtruck is now known as zz_dimtruck | 22:26 | |
| dhellmann | I guess I was too clever? I don't know. | 22:26 |
| clarkb | why would you have a depends-on: self though? /me tries to understand the use case | 22:26 |
| dhellmann | clarkb: it was not depending on itself. It was depending on another patch in a different repo with the same change-id | 22:27 |
| clarkb | dhellmann: oh I see, ya that wouldn't work | 22:27 |
| clarkb | jeblair: ^ | 22:27 |
| *** yfried has joined #openstack-infra | 22:27 | |
| dhellmann | if I had known that I needed to use depends-on, I wouldn't have set up the original patches that way | 22:27 |
| dhellmann | I was trying to avoid having the TC have to revote on that patch, but I'll just submit a new one | 22:27 |
| openstackgerrit | Merged openstack-infra/system-config: Add Fedora 21 image build to nodepool https://review.openstack.org/140911 | 22:28 |
| jeblair | ianw: ^ | 22:28 |
| ianw | jeblair: thanks, i'll check in on the image build in a bit | 22:28 |
| *** harlowja has joined #openstack-infra | 22:28 | |
| clarkb | note that hpcloud is unhappy at the moment so we may not see that take practical effect for a bit | 22:28 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Add openstack/developer-reference repository https://review.openstack.org/161351 | 22:29 |
| ianw | jeblair: i'm guessing my counter-argument in https://review.openstack.org/#/c/153904/ (Add generate-log-config command) has not been persuasive? | 22:29 |
| jeblair | dhellmann: setting aside some other rough edges with depends-on here, i do think that may be a slightly overzealous use of a shared change-id; i usually think of it meaning the _same change_, eg, a backport from a different branch, or updating the pbr version in every project... | 22:29 |
| dhellmann | jeblair: that's fair | 22:30 |
| *** e0ne has quit IRC | 22:30 | |
| jeblair | dhellmann: i think you have found a grey area :) | 22:30 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Consolidate CFP settings into a common function https://review.openstack.org/158989 | 22:30 |
| dhellmann | jeblair: usually I use a blueprint or topic to link the changes, but there's no blueprint for this | 22:30 |
| dhellmann | jeblair: and now that i know, I won't make the mistake again | 22:30 |
| *** mriedem is now known as mriedem-away | 22:30 | |
| jogo | clarkb mtreinish: ahh good catch | 22:31 |
| dhellmann | it was just annoying to find another procedural change | 22:31 |
| dhellmann | anteaya: is the creator's guide up to date with this? | 22:31 |
| *** patrickeast___ has joined #openstack-infra | 22:31 | |
| jeblair | dhellmann, clarkb: that said, depends-on self could probably be excluded if it isn't already; and i'd be open to interpreting 'depends-on abandoned change' as treating the abandoned change as nonexistent. that's probably the right thing to do. | 22:31 |
| anteaya | I'll check | 22:31 |
| dhellmann | ah, it is | 22:31 |
| asselin | hi dib experts. What should I set RELEASE to? https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos#L34 | 22:33 |
| *** Ryan_Lane has quit IRC | 22:34 | |
| *** bcourt has quit IRC | 22:34 | |
| greghaynes | asselin: context? | 22:35 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/infra-manual: Mention Depends-On where the commit is made https://review.openstack.org/161454 | 22:35 |
| dhellmann | anteaya: ^^ | 22:35 |
| asselin | i'm just starting to try using disk image builder. I'm getting the error on line 34 posted ^^ | 22:35 |
| asselin | but it's not clear what it's supposed to be? ubunut? master? | 22:36 |
| lifeless | so thats nodepool specific code; I'd guess you need to look at the nodepool invocation of dib to see how its being called | 22:36 |
| greghaynes | asselin: the os element tends to set DIB_RELEASE IIRC, like ubuntu for example | 22:37 |
| lifeless | dib itself tries to guess | 22:37 |
| greghaynes | the ubuntu element sets it to trusty if its not set looks like | 22:37 |
| greghaynes | asselin: what elements are you building with? | 22:37 |
| asselin | oh I see..maybe I should try this on a trusty node. I'm using precise...probably not a good idea. ok thanks! | 22:37 |
| greghaynes | It should work fine on precise | 22:38 |
| openstackgerrit | Maru Newby proposed openstack-infra/project-config: Enhance run-tox.sh to support functional testing https://review.openstack.org/161456 | 22:38 |
| clarkb | greghaynes: mordred fwiw I just tried setting my resolv.conf to point at non existant resolver and got -2 error not -3 (EAI_AGAIN) I think what you need is actually a resolver that can't resolve rather than a resolver that ins't there? | 22:38 |
| asselin | greghaynes, ok, then I'll try setting it to ubuntu while on that os. | 22:38 |
| *** mattfarina has quit IRC | 22:38 | |
| clarkb | I think I may just put my changes on a node and reboot it >_> simplest way to test | 22:38 |
| jeblair | clarkb: ++ we do have 20 of them | 22:38 |
| lifeless | asselin: what exact command line are you running dib with ? | 22:39 |
| greghaynes | asselin: well my concern is that youre somehow not buliding with one of the elements that sets it, which is wierd | 22:39 |
| *** mayurig has quit IRC | 22:39 | |
| jeblair | clarkb: what's the latest on hpcloud-badness? | 22:39 |
| asselin | I did this: DISTRO="ubuntu" bash tools/build-image.sh https://github.com/openstack-infra/project-config/tree/master/nodepool/elements | 22:39 |
| clarkb | jeblair: lots of timeouts, I think this is the underlying issue leading to floating ip leaks | 22:39 |
| clarkb | jeblair: we do not appear to actually be leaking ports | 22:40 |
| jogo | I am having trouble booting my own VM in hp cloud | 22:40 |
| clarkb | jeblair: Exception: Timeout waiting for ssh access and URLError: <urlopen error [Errno 110] Connection timed out> are common | 22:40 |
| jogo | http://paste.openstack.org/show/188196 | 22:40 |
| jeblair | clarkb: the urlerror is jenkins04 | 22:40 |
| jogo | cloud-init-nonet[15.40]: waiting 120 seconds for network device | 22:40 |
| pleia2 | hanging out with cinerama doing zanata things \o/ | 22:40 |
| jeblair | pleia2, cinerama: rock on! | 22:41 |
| *** erlon has quit IRC | 22:41 | |
| clarkb | jeblair: I thought I saw it happen with nova clietn too | 22:41 |
| *** jogo has quit IRC | 22:41 | |
| *** jogo has joined #openstack-infra | 22:42 | |
| jeblair | we have 12 nodes from hpcloud in use :( | 22:42 |
| anteaya | dhellmann: thank you | 22:42 |
| greghaynes | asselin: looks like that should be running with the ubuntu element... | 22:42 |
| clarkb | also 04 looks down down, I am guessing it didn't come back up after reboots | 22:42 |
| cinerama | we are outside doing some OG street hacking | 22:42 |
| asselin | greghaynes, lifeless ok, I added the extra environment and it got past that error. Not sure what broke afterwards. full log is here: http://paste.openstack.org/show/188198/ | 22:43 |
| lifeless | asselin: that variable is wrong | 22:43 |
| rhe00 | if anyone has time for CI assistence I could use some help. my node is trying to load devstack as part of the dsvm test but fails with: "fatal: repository 'http://10.20.216.78/p//openstack-dev/devstack/' not found" | 22:44 |
| rhe00 | paste.openstack.org/show/188197/ | 22:44 |
| asselin | rhe00, that might be ok | 22:44 |
| greghaynes | yes, it should be trusty, im just still very confused why you need to set it | 22:44 |
| * asselin trying trusty | 22:44 | |
| lifeless | asselin: http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/ubuntu/README.rst | 22:45 |
| rhe00 | it stops loading devstack shortly after that without any error messages. sort of frozen | 22:45 |
| rhe00 | Jenkins still thinks the test is running | 22:45 |
| lifeless | asselin: thats the docs; it should not need to be set. if you're havnig to set it there is a bug somewhere. | 22:45 |
| *** mayurig has joined #openstack-infra | 22:45 | |
| greghaynes | hey, theres super fancy web docs now ;) http://docs.openstack.org/developer/diskimage-builder/elements/ubuntu/README.html | 22:45 |
| clarkb | jeblair: grep tells me I am wrong, no url errors from novaclient. So thats a good thing | 22:46 |
| lifeless | asselin: in fact yes, I think the nodepool element is buggy | 22:46 |
| greghaynes | Yes, thats my suspicion - I bet nodepool explicitly sets it | 22:46 |
| lifeless | asselin: DIB_RELEASE in the code is always guarded | 22:46 |
| greghaynes | which is why it hasnt come up | 22:46 |
| jogo | clarkb: you may be seeing something similar to what I am seeing on my HP acccount http://paste.openstack.org/show/188196 | 22:46 |
| lifeless | that said | 22:46 |
| *** dboik has quit IRC | 22:46 | |
| lifeless | elements/ubuntu-minimal/environment.d/10-ubuntu-distro-name.bash:export DIB_RELEASE=${DIB_RELEASE:-trusty} | 22:46 |
| clarkb | jogo: we don't cloud init though | 22:47 |
| *** nelsnelson has quit IRC | 22:47 | |
| clarkb | jogo: but maybe its related? I can try getting some console logs for our nodes | 22:47 |
| lifeless | ubuntu-minimal is not a dep of ubuntu | 22:47 |
| marun | jeblair: great! | 22:48 |
| lifeless | ah, its a totally different codepath | 22:48 |
| asselin | lifeliess, wehre do you see ubuntu-minimal? | 22:48 |
| jogo | clarkb: right, but network related? | 22:48 |
| lifeless | asselin: yep, confirmed - that nodepool code is broken | 22:48 |
| mtreinish | clarkb: inside the dib chroot can I get to /home/jenkins/cache/files? Or is that outside the chroot? | 22:48 |
| lifeless | asselin: DIB_RELEASE is an optional variable, the normal state is for it not to be set | 22:48 |
| asselin | btw, DIB_RELEASE="trusty" works. Fails with the usual "git" protocol which is blocked at my site | 22:48 |
| openstackgerrit | Michael Johnson proposed openstack-infra/git-review: Choose correct branch for rebase when submitting https://review.openstack.org/158877 | 22:48 |
| marun | clarkb: any suggestions for how to best test modifications to run-tox.sh? Is my only option recreating the gate env locally? | 22:49 |
| greghaynes | ah right, its not global | 22:49 |
| lifeless | clarkb: ^ see above | 22:49 |
| marun | clarkb: re: https://review.openstack.org/#/c/161456/ | 22:49 |
| clarkb | mtreinish: that is in the chroot, the extra-data scripst actually write into the FS that will be chrooted into so its availab | 22:49 |
| clarkb | mtreinish: the issue is to get wheel to build the right thing it needs to run in the chroot and not just copy into it | 22:49 |
| clarkb | marun: you should be able to run that script and have it run your tests | 22:50 |
| *** mayurig has quit IRC | 22:50 | |
| mkj_wk | clarkb: 12 unit tests added in this last review request :) | 22:50 |
| clarkb | mkj_wk: nice and that ssh flag is still working well? | 22:50 |
| *** tsg has joined #openstack-infra | 22:50 | |
| clarkb | jogo: ok I have infos | 22:50 |
| *** bknudson has quit IRC | 22:50 | |
| *** otter768 has joined #openstack-infra | 22:50 | |
| mkj_wk | clarkb: yup, never failed | 22:51 |
| mkj_wk | clarkb: 16 patch sets is rather many, though some of them were just to get jenkins to run to figure out why it failed on py26 (git version differences) | 22:51 |
| mtreinish | clarkb: ok sure, I just don't have a complete env picture in my head for the dib side. So I was writing it with a blind assumption that path would work | 22:51 |
| clarkb | jogo: jeblair http://paste.openstack.org/show/188214/ | 22:51 |
| mtreinish | figured I should check first | 22:51 |
| *** camunoz has joined #openstack-infra | 22:52 | |
| clarkb | mtreinish: yup no problem and you can write straight to it you don't need the prefix stuff like in the extra-data.d script | 22:52 |
| *** asselin_ has joined #openstack-infra | 22:52 | |
| clarkb | jogo: jeblair that is for a devstack-trusty node, going to see if I can find one that isn't dib image built | 22:52 |
| jogo | clarkb: that looks bad | 22:52 |
| clarkb | jogo: jeblair I want to rule out our image being bad | 22:52 |
| clarkb | jogo: yes | 22:52 |
| jeblair | clarkb: that would indeed explain a lack of ssh connectivity :) | 22:53 |
| * mkj_wk wanders home | 22:53 | |
| jogo | clarkb: related to the mtreinish's patch and using the chroot, doesn't that mean the apt-get cache stuff fails too | 22:53 |
| *** timcline has quit IRC | 22:54 | |
| clarkb | jogo: its possible | 22:54 |
| jogo | clarkb: we try sudo', 'apt-get', '-y', '-d', 'install | 22:54 |
| *** rlandy has quit IRC | 22:54 | |
| clarkb | jogo: which will just grab things for outside the chroot | 22:54 |
| jogo | but we are still downloading a lot of packages during each install | 22:54 |
| clarkb | jeblair: jogo ok bare-trusty looks better and it looks like we have those ready and used | 22:54 |
| clarkb | jogo: jeblair so it is entirely possible something in our images is causing this to explode | 22:55 |
| jogo | clarkb: right, and won't get copied into the images | 22:55 |
| clarkb | jogo: right we would still download them, just like we would still build wheels. They are probably not the correct data in either case | 22:55 |
| *** crc32 has quit IRC | 22:55 | |
| jeblair | clarkb: want to delete current and fall back to previous? | 22:55 |
| asselin | lifeless, greghaynes ok seems to be running now....caching repos....thanks | 22:55 |
| clarkb | jeblair: its worth a shot, will start with hpcloud-b1 | 22:56 |
| jogo | clarkb: right, but that command needs to be run inside the chroot | 22:56 |
| *** otter768 has quit IRC | 22:56 | |
| *** mayurig has joined #openstack-infra | 22:56 | |
| clarkb | jogo: yes, I agree | 22:56 |
| *** asselin_ has quit IRC | 22:57 | |
| *** ParsectiX has quit IRC | 22:59 | |
| *** Rockyg has joined #openstack-infra | 23:00 | |
| *** eharney has quit IRC | 23:00 | |
| *** bhunter71 has quit IRC | 23:00 | |
| *** ParsectiX has joined #openstack-infra | 23:00 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Build wheel house during nodepool devstack caching https://review.openstack.org/160548 | 23:01 |
| mtreinish | clarkb: ^^^ I think that should do the trick | 23:01 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Fix nodepool.yaml whitespace https://review.openstack.org/161462 | 23:01 |
| mtreinish | jogo: ^^^ you seem driven on this too :) | 23:01 |
| *** bhunter71 has joined #openstack-infra | 23:02 | |
| *** ghostpl_ has quit IRC | 23:02 | |
| jeblair | clarkb: aprvd | 23:03 |
| clarkb | jeblair: thanks | 23:03 |
| *** mrodden has quit IRC | 23:03 | |
| jeblair | course that will take a long time to land | 23:04 |
| *** annegentle has quit IRC | 23:04 | |
| clarkb | there are a set of nodes building in hpcloud-b1 with the old image now | 23:04 |
| clarkb | will pay attention to them to see if they fail | 23:05 |
| *** amitgandhinz has quit IRC | 23:05 | |
| clarkb | also, I wonder if it would be possible for qemu/kvm to say "hey booting failed let me ERROR" when running in a cloud | 23:05 |
| clarkb | because chances are good that busybox shell does me no good | 23:05 |
| *** mrodden has joined #openstack-infra | 23:05 | |
| *** alexpilotti has joined #openstack-infra | 23:06 | |
| jeblair | clarkb: hrm, i suppose you could build an initrd that halted or something... i wonder if it could issue an instruction that the hypervisor could treat as an error condition. | 23:06 |
| *** jaypipes has quit IRC | 23:09 | |
| *** gordc has quit IRC | 23:09 | |
| clarkb | I have disabled puppet on nodepool.o.o to get that whitespace change in | 23:10 |
| clarkb | command line nodepool commands break if they can't read the config | 23:10 |
| jeblair | yeah, we need a syntax gate check on that | 23:11 |
| *** marun has quit IRC | 23:12 | |
| jeblair | clarkb: also, i think nodepool stopped building nodes | 23:12 |
| clarkb | jeblair: I think it did that when I updated the config because the managers got reconfigured | 23:13 |
| clarkb | it should start again | 23:13 |
| jeblair | ya | 23:13 |
| jesusaurus | clarkb: can you give https://review.openstack.org/#/c/76057 another review? | 23:13 |
| clarkb | jesusaurus: not right now :) but will try to swing around when things are happy again | 23:13 |
| jesusaurus | clarkb: cool | 23:14 |
| jeblair | Exception: No image available for that upload. Please build one first. | 23:15 |
| jeblair | Exception: DIB failed creating /opt/nodepool_dib/devstack-fedora21-dib-1425510899.qcow2 | 23:15 |
| jeblair | AttributeError: 'NoneType' object has no attribute 'image_name' | 23:15 |
| jeblair | wow. | 23:15 |
| *** david-lyle has quit IRC | 23:16 | |
| clarkb | jeblair: ugh | 23:16 |
| *** david-lyle has joined #openstack-infra | 23:16 | |
| ianw | argh ... | 23:17 |
| *** tsg_ has joined #openstack-infra | 23:17 | |
| clarkb | jeblair: would that prevent new nodes from building? | 23:17 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Fix pkgs-to-install path https://review.openstack.org/161472 | 23:17 |
| clarkb | jeblair: I am still not seeing demand calculations or building nodesu | 23:17 |
| jogo | clarkb: I think I found the apt-get install -y issue ^ | 23:18 |
| jeblair | clarkb: i have no idea. i'm starting at the end and working backwards. :( | 23:18 |
| *** tsg has quit IRC | 23:19 | |
| clarkb | jeblair: yup confirmed, checkformissing images short circuits the run loop :( | 23:20 |
| clarkb | jeblair: I am going to manually back out the f21 dib change on nodepool.o.o unless you want more data | 23:20 |
| ianw | i'm looking for why it's trying to grab /pub/fedora/linux/releases/21/Cloud/Images/x86_64/Fedora-Cloud-Base-20141203-21.x86_64.qcow2/Fedora-Cloud-Base-20141203-21.x86_64.qcow2 | 23:20 |
| jeblair | clarkb: nope sounds good | 23:20 |
| ianw | clarkb: you going to look at the loop-stopping issue? | 23:21 |
| *** dkranz has joined #openstack-infra | 23:21 | |
| clarkb | ianw: sure, once I confirm nodepool is going again and get some images building in hpcloud I will look at the run loop | 23:22 |
| bookwar | https://review.openstack.org/#/c/143973/ can you help me with gating here? so it asks for rebase, but then it sets Verified +2 label | 23:22 |
| clarkb | http://paste.openstack.org/show/188251/ is the trace back fwiw | 23:22 |
| bookwar | do i need to wait for merge or still rebase is required | 23:22 |
| jogo | mtreinish: inline comments on your patch | 23:23 |
| clarkb | bookwar: gerrit decided there was a merge conflict there that zuul did not detect | 23:23 |
| ianw | clarkb: great, thanks ... | 23:23 |
| jogo | mtreinish: also figured out why we keep pulling apt packages from the network https://review.openstack.org/161472 | 23:23 |
| bookwar | clarkb: so it is better to rebase | 23:23 |
| clarkb | bookwar: yes you will need to rebase | 23:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Fix exception in dib image cleanup https://review.openstack.org/161473 | 23:24 |
| bookwar | thanks | 23:25 |
| *** annegentle has joined #openstack-infra | 23:25 | |
| *** alexpilotti has quit IRC | 23:26 | |
| *** ddieterly has quit IRC | 23:27 | |
| clarkb | still no building nodes after cleaning up nodepool.yaml | 23:27 |
| clarkb | ok I see why | 23:28 |
| clarkb | http://paste.openstack.org/show/188253/ | 23:28 |
| *** tonytan4ever has quit IRC | 23:28 | |
| clarkb | jeblair: ^ thoughts on just rebooting? | 23:28 |
| clarkb | er | 23:28 |
| greghaynes | jogo: so the interesting thing about that is http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/dpkg/root.d/99-shared_apt_cache | 23:28 |
| clarkb | restarting the service? | 23:28 |
| SpamapS | woo, looks like I'll be talking about infra on 3/26 at OpenStack LA | 23:28 |
| SpamapS | I should probably, you know, learn about infra. :) | 23:29 |
| clarkb | oh wait | 23:29 |
| greghaynes | jogo: and also http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/dpkg/finalise.d/99-clean-up-cache | 23:29 |
| *** david-lyle has quit IRC | 23:29 | |
| clarkb | its because I didn't remove that image completely from the config? | 23:29 |
| * clarkb tries again | 23:29 | |
| *** MaxV_ has quit IRC | 23:29 | |
| greghaynes | jogo: so im not sure how that apt-get -d install will be useful | 23:30 |
| lifeless | oh joy | 23:30 |
| jogo | greghaynes: look at the output of a real run | 23:30 |
| lifeless | https://freakattack.com/ | 23:30 |
| *** ghostpl_ has joined #openstack-infra | 23:30 | |
| jogo | we download a while slew of things | 23:30 |
| *** unicell has quit IRC | 23:30 | |
| jeblair | frankly, i'm not sure i want to approve any more nodepool config changes until there's some actual testing around this. i can't believe how many bugs we're finding right now. | 23:30 |
| greghaynes | jogo: yes. My point is that at the end of a dib bulid were going to clear the cache | 23:30 |
| *** unicell has joined #openstack-infra | 23:30 | |
| greghaynes | jogo: so even if you download stuff into it, itll get deleted | 23:30 |
| clarkb | ok I have more properly fixed the config, lets see if that shakes some nodes loose | 23:31 |
| jogo | greghaynes: http://logs.openstack.org/72/159772/7/check/check-tempest-dsvm-neutron-pg-full/63ed229//logs/devstacklog.txt.gz#_2015-03-04_22_40_38_057 | 23:31 |
| *** gokrokve has joined #openstack-infra | 23:31 | |
| jogo | greghaynes: hrrm | 23:31 |
| *** unicell has quit IRC | 23:31 | |
| greghaynes | I think we need to make a disable cleanup flag | 23:32 |
| *** unicell has joined #openstack-infra | 23:32 | |
| clarkb | jeblair: +1, I have started some of that work in a stack and ianw increased testing to catch some other issues. obviously much more we can do | 23:32 |
| *** annegentle has quit IRC | 23:32 | |
| *** gokrokve has quit IRC | 23:32 | |
| *** unicell has quit IRC | 23:33 | |
| *** unicell1 has joined #openstack-infra | 23:33 | |
| *** crc32 has joined #openstack-infra | 23:33 | |
| jogo | greghaynes: or just disable the sharing of apt-cache mirrors? | 23:33 |
| *** annegentle has joined #openstack-infra | 23:33 | |
| jogo | greghaynes: or just disable the sharing of apt caches* | 23:33 |
| *** david-lyle has joined #openstack-infra | 23:34 | |
| greghaynes | jogo: They are both issues - the sharing means youll get stuff in the cache you might not care about but it should still work, the cleanup means it doesnt matter for this use case | 23:34 |
| * greghaynes makes a couple patches | 23:34 | |
| clarkb | I don't think updateConfig is able to handle removals of images | 23:34 |
| jogo | greghaynes: thanks, I am trying to make devstack build faster (its close to 20 minutes right now) | 23:35 |
| clarkb | jeblair: I am ready to restart nodepool if you are | 23:35 |
| *** signed8bit has joined #openstack-infra | 23:35 | |
| jogo | greghaynes: which is the context for this | 23:35 |
| mtreinish | jogo: yep, I forgot to check the script exists first | 23:35 |
| jeblair | clarkb: go for it | 23:35 |
| *** sarob has quit IRC | 23:36 | |
| clarkb | done | 23:36 |
| greghaynes | jogo: yep, this is the same kind of stuff we battled with in tripleo CI for a while :) | 23:36 |
| ianw | clarkb / jeblair : as a start this indenting of the yaml file is rather insidious ... it's too easy to make copy-paste errors when you're moving from a local copy into the system-config copy ... and it's really long so it's not like it immediately stands out to your eye | 23:36 |
| clarkb | greghaynes: di deletes the cache in the image? | 23:36 |
| clarkb | greghaynes: *dib | 23:36 |
| clarkb | greghaynes: that is subpar | 23:36 |
| *** ghostpl_ has quit IRC | 23:37 | |
| greghaynes | clarkb: yes, because typically you prefer the image to be small and dont want the cache | 23:37 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Build wheel house during nodepool devstack caching https://review.openstack.org/160548 | 23:37 |
| clarkb | greghaynes: ugh | 23:37 |
| mtreinish | jogo: ^^^ added the check | 23:37 |
| jeblair | ianw: zuul has a config file syntax test, nodepool should too | 23:37 |
| greghaynes | clarkb: it also bind mounts the apt cache in from the build host | 23:37 |
| clarkb | now I think we are waiting for nodepool to timeout on jenkins04 | 23:38 |
| clarkb | maybe I will take a timeout and go reboot that node more forcefully | 23:38 |
| *** dkranz has quit IRC | 23:38 | |
| *** stevemar2 is now known as stevemar | 23:38 | |
| *** bswartz has joined #openstack-infra | 23:39 | |
| *** hichihara has joined #openstack-infra | 23:39 | |
| greghaynes | jogo: https://review.openstack.org/#/c/161477/ | 23:40 |
| *** achanda has joined #openstack-infra | 23:40 | |
| clarkb | hrm rax doesn't seem to support nova console-log | 23:41 |
| *** ddieterly has joined #openstack-infra | 23:41 | |
| clarkb | woot finally connection timed out | 23:41 |
| clarkb | I am going to give the web ui console thing a try | 23:42 |
| clarkb | before I reboot | 23:42 |
| *** chlong has joined #openstack-infra | 23:42 | |
| *** carl_baldwin has joined #openstack-infra | 23:43 | |
| ianw | clarkb / jeblair: alright, i'll copy the zuul method (using voluptuous) and add a command to validate config. sound ok? | 23:43 |
| *** annegentle has quit IRC | 23:43 | |
| greghaynes | jogo: also check out DIB_APT_LOCAL_CACHE var in http://docs.openstack.org/developer/diskimage-builder/elements/dpkg/README.html | 23:43 |
| clarkb | ianw: yup I think thats the route we should use for that | 23:43 |
| clarkb | ianw: it will cover whitespace things as well as more of the datastructure itself | 23:43 |
| *** annegentle has joined #openstack-infra | 23:43 | |
| *** signed8bit has quit IRC | 23:43 | |
| clarkb | well console failed with my browser not doing flash | 23:44 |
| clarkb | I will pull up the flash brwoser and try again | 23:44 |
| *** belmoreira has quit IRC | 23:44 | |
| clarkb | ok images are building again | 23:45 |
| ianw | part of my issue is that whatever emacs yaml mode does by default isn't what openstack yaml looks like. i'm sure there's something i can tweak just like the c indent modes | 23:45 |
| *** jlvillal has left #openstack-infra | 23:45 | |
| clarkb | I am ging to fix jenkins04, then go back to debugging this image and why it won't boot in hpcloud | 23:45 |
| openstackgerrit | Steve Baker proposed openstack-infra/project-config: Create and upload heat test images to tarballs https://review.openstack.org/159686 | 23:46 |
| *** chlong has quit IRC | 23:46 | |
| *** chlong has joined #openstack-infra | 23:46 | |
| clarkb | so jenkins04 does have a console no crazy errors on it, but it does unping and ssh doesn't connect | 23:47 |
| clarkb | so I think given that there isn't much more info to go off of I am going to try a reboot of it | 23:48 |
| *** annegentle has quit IRC | 23:48 | |
| clarkb | ok that unwedged 04 | 23:49 |
| *** Ryan_Lane has joined #openstack-infra | 23:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Do not try to upload building disk images https://review.openstack.org/161482 | 23:50 |
| jeblair | clarkb, ianw: that's the second of two patches i have written based on the exceptions we've just seen ^ | 23:50 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Add expiremental devstack plugin based ironic functional job https://review.openstack.org/161483 | 23:51 |
| jeblair | clarkb, ianw: (i have not tested those) | 23:51 |
| jeblair | ianw: that sounds awesome. also, i use emacs to edit yaml... | 23:51 |
| jeblair | ianw: that == using voluptuous | 23:51 |
| *** bknudson has joined #openstack-infra | 23:52 | |
| jogo | greghaynes: thanks, sounds like a few more patches to make the deb caching stuff work | 23:53 |
| clarkb | jeblair: I have removed the newer devstack-trusty image from all hpcloud-b* providers. I have also copied the qcow2 that was uploaded there to my homedir so that we can investigate why it doesn't boot without worrying about nodepool deleting it later | 23:53 |
| ianw | jeblair: have you tweaked it at all? it always indents too far for me | 23:53 |
| jeblair | ianw: not that i'm aware of, i'll try to pay attention next time i'm editing and see if it works, or if i'm just fixing it | 23:53 |
| clarkb | jeblair: should I manually upload it to hpcloud under a name not used by nodepool too? | 23:54 |
| clarkb | jeblair: better question, do you want to interact with this image in hpcloud? if so I can upload it under a non nodepool name | 23:54 |
| jeblair | clarkb: not at the moment | 23:54 |
| clarkb | ok I will just download it locally to dig into it here | 23:55 |
| jeblair | clarkb: regarding http://paste.openstack.org/show/188253/ did that keep happening, or was that a one-time thing? | 23:55 |
| *** jlvillal_ has joined #openstack-infra | 23:55 | |
| greghaynes | jogo: yes. actually, why arent you just installing the packages during the dib run rather than doanloading? | 23:56 |
| *** chuckC_ has joined #openstack-infra | 23:56 | |
| *** EmilienM is now known as EmilienM|afk | 23:56 | |
| *** annegentle has joined #openstack-infra | 23:56 | |
| *** jlvillal_ has left #openstack-infra | 23:56 | |
| jeblair | clarkb: nm i can answer that :) | 23:56 |
| jeblair | clarkb: twice | 23:56 |
| clarkb | jeblair: it kept happening | 23:56 |
| clarkb | jeblair: I think that the configUpdate method doesn't handle image removals gracefully | 23:56 |
| jogo | greghaynes: AFAIK its we don't want to actually install all packages | 23:56 |
| jogo | for example postgres vs mysql db | 23:57 |
| clarkb | jeblair: so it kept trying to make the devstack fedora21 dib image even after I removed it. This is why I restarted the service | 23:57 |
| *** annegentle has quit IRC | 23:57 | |
| *** yamamoto has joined #openstack-infra | 23:57 | |
| jeblair | clarkb: it seems like it wasn't in the diskimages section, but it was in the providers... | 23:57 |
| *** annegentle has joined #openstack-infra | 23:57 | |
| *** yamahata has joined #openstack-infra | 23:57 | |
| clarkb | jeblair: ya that was first issue, then I removed it completely (well commented it out) | 23:58 |
| clarkb | jeblair: and the exceptions in the main run loop continued | 23:58 |
| *** annegentle has quit IRC | 23:58 | |
| *** dmsimard_away is now known as dmsimard | 23:58 | |
| greghaynes | jogo: ah, that makes some sense. Although just installing all of them isnt bad, and then only using the ones needed on the test? | 23:58 |
| *** dmsimard is now known as dmsimard_away | 23:58 | |
| clarkb | qcow2 download is going to take some time, going to step out for a few | 23:59 |
| jogo | greghaynes: not sure, that is a really good question | 23:59 |
| clarkb | greghaynes: jogo it is intentional to not install anything | 23:59 |
| clarkb | greghaynes: jogo devstack is supposed to be self testing | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
