| mordred | StevenK: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/proposal_slave.pp | 00:00 |
|---|---|---|
| harlowja_ | clarkb let me know when u think i can recheck https://review.openstack.org/#/c/164836/ and it might work | 00:01 |
| *** alexpilotti has quit IRC | 00:01 | |
| pleia2 | I can confirm with AJaeger that script-wise that's the only place it's needed | 00:01 |
| clarkb | harlowja_: I have no idea, I was hoping mordred would have some great insight for us | 00:01 |
| pleia2 | we'll need to know where to put this puppet change anyway | 00:01 |
| harlowja_ | k | 00:01 |
| harlowja_ | mordred great insight required! | 00:02 |
| mordred | clarkb: why would i have insight? | 00:02 |
| harlowja_ | i don't think any py26 supporting project can update its requirements with those '~=' style until that gets addressed | 00:02 |
| *** claudiub has joined #openstack-infra | 00:02 | |
| *** alexpilotti has joined #openstack-infra | 00:02 | |
| clarkb | mordred: because you wrote all the recent changes to that pin | 00:02 |
| mordred | clarkb: oh - that was the pin thing | 00:03 |
| clarkb | harlowja_: maybe we shouldn't use ~= then | 00:03 |
| clarkb | harlowja_: because we can't garuntee everyone using this sfotware has pip 6.0 | 00:03 |
| mordred | clarkb: is that pinning to older or newer? | 00:03 |
| clarkb | mordred: older | 00:03 |
| harlowja_ | clarkb u won't get disagreement from me :-P | 00:03 |
| mordred | clarkb: yes, we can. we do not make any guarantees about non-most-recent pip | 00:03 |
| *** Ryan_Lane has quit IRC | 00:03 | |
| mordred | clarkb: in fact, we explicitly don't work on it, i believe | 00:03 |
| harlowja_ | https://github.com/openstack/requirements/commit/25ef10 put a bunch in though, lol | 00:03 |
| *** alexpilotti has quit IRC | 00:03 | |
| clarkb | mordred: we do we know this because centos6 | 00:04 |
| *** yamamoto has joined #openstack-infra | 00:04 | |
| pleia2 | I need to run off to an evening event, thanks jeblair, StevenK and mordred | 00:04 |
| mordred | clarkb: no - if you're installing from pip, you need to install latest pip | 00:04 |
| *** jp_at_hp has joined #openstack-infra | 00:04 | |
| mordred | we do not support the pip in centos6 | 00:04 |
| clarkb | mordred: that is not how we test | 00:04 |
| mordred | clarkb: then we're testing wrong | 00:04 |
| clarkb | mordred: thats fine we can fix that | 00:04 |
| mordred | clarkb: woot | 00:04 |
| clarkb | mordred: just pointing out that your assertion is false as of right now | 00:04 |
| *** sdake_ has joined #openstack-infra | 00:04 | |
| *** alexpilotti has joined #openstack-infra | 00:04 | |
| mordred | clarkb: cool. well - I refuse to support old pip except for transitionally | 00:05 |
| mordred | because it makes absolutely no sense | 00:05 |
| clarkb | the reason I suggest not using ~= is because >= and < should also work fine | 00:05 |
| clarkb | so there is little benefit in supporting fancy new symbol that isn't available everwhere | 00:05 |
| mordred | clarkb: right - but openstack is adopting ~= across the board pretty aggressively | 00:05 |
| clarkb | mordred: yes but there is no reason to | 00:05 |
| clarkb | mordred: why would we do that? | 00:05 |
| mordred | clarkb: there is no reason not to | 00:05 |
| mordred | clarkb: supporting out of date pip has no point | 00:05 |
| clarkb | mordred: there is | 00:05 |
| mordred | there is literally no benefit | 00:06 |
| clarkb | mordred: code change churn | 00:06 |
| clarkb | there is no benefit to change to ~= so lets just leave it be | 00:06 |
| mordred | tehn don't pip install from master | 00:06 |
| *** alexpilotti has quit IRC | 00:06 | |
| mordred | clarkb: not our call here | 00:06 |
| clarkb | mordred: this has nothing to do with pip | 00:06 |
| * harlowja_ has to go fix anvil to convert ~= -> rpm versions, lol, (code churn there, lol) | 00:06 | |
| mordred | sure it does | 00:06 |
| clarkb | mordred: and everything to do with we don't need hundreds of requirements updates to support ~= unnecessarily | 00:06 |
| clarkb | mordred: it just happens that old pip also fails at this | 00:06 |
| mordred | I'm not going to police that | 00:06 |
| mordred | if the project wants to adopt it, great | 00:06 |
| clarkb | sure, I haven't -1'd anything | 00:07 |
| mordred | it fits in with my understanding of our pip support | 00:07 |
| clarkb | just pointing out that there are valid reasons to not do this | 00:07 |
| mordred | which is "anything other than latest pip is insane" | 00:07 |
| mordred | sure | 00:07 |
| jeblair | clarkb, mordred: 155072 includes a formal adoption of it | 00:07 |
| mordred | I'm just saying that if we're pinning to an out of date venv - sorry, that's my bad | 00:07 |
| mordred | and we should stop doing it | 00:08 |
| *** sdake has quit IRC | 00:08 | |
| mordred | because we test with latest pip in devstack | 00:08 |
| *** mtanino has quit IRC | 00:08 | |
| mordred | and testing with older pip elsewhere because I suck is a bad policy :) | 00:08 |
| harlowja_ | mordred please suck less, thx | 00:08 |
| mordred | harlowja_: listen man - there is only so much suck I can remove on a given day | 00:08 |
| jroll | whoa, pip has ~= now? what does that do? | 00:08 |
| * jroll not sure if he wants to know | 00:09 | |
| clarkb | also Istill don't know why trusty works | 00:09 |
| clarkb | mordred: ^ which was mostly I think what I wanted clarification on before we changed anything | 00:09 |
| harlowja_ | jroll https://www.python.org/dev/peps/pep-0440/#compatible-release | 00:09 |
| clarkb | it appears the only place we install virtualenv is in the base.pp manifest | 00:09 |
| clarkb | and yet trusty nodes don't seem to get downgraded | 00:09 |
| *** jamielennox is now known as jamielennox|away | 00:09 | |
| *** liusheng has quit IRC | 00:10 | |
| jroll | huh | 00:10 |
| mordred | clarkb: hrm | 00:10 |
| harlowja_ | lol, jroll same response, idk either :-P | 00:10 |
| *** liusheng has joined #openstack-infra | 00:10 | |
| jroll | 00:06:11 clarkb | there is no benefit to change to ~= so lets just leave it be <- I like this argument | 00:10 |
| *** SumitNaiksatam has quit IRC | 00:11 | |
| *** jeffgdl has joined #openstack-infra | 00:11 | |
| jroll | but then again I would rather == would work | 00:11 |
| *** garyh has quit IRC | 00:12 | |
| *** ddieterly has joined #openstack-infra | 00:12 | |
| harlowja_ | i start to feel that someone was taking a little to many drugs when making up ~= as a version identifier | 00:12 |
| harlowja_ | lol | 00:12 |
| *** mrmartin has quit IRC | 00:12 | |
| jroll | I've only ever seen that in npm things | 00:12 |
| jroll | so.... you may be right | 00:12 |
| harlowja_ | maybe they share(d) drugs then | 00:12 |
| harlowja_ | ha | 00:12 |
| *** otter768 has joined #openstack-infra | 00:13 | |
| *** dims__ is now known as dims | 00:13 | |
| harlowja_ | i | 00:13 |
| *** sdake has joined #openstack-infra | 00:13 | |
| harlowja_ | *i'd rather have the versions be simple, have a nice solid dependency resolver, and then maybe start adding these edge-cases | 00:13 |
| ianw | fungi: is anyone actively looking at https://review.openstack.org/#/c/165620/ (Temporarily stop adding devstack-f20 nodes in rax)? | 00:14 |
| jroll | harlowja_: yeah | 00:14 |
| dims | fungi: clarkb: some oslo.messaging -docs, -tarball are stuck in the -post queue for a while. do i need to re-jiggle them some how? | 00:14 |
| *** achanda has quit IRC | 00:15 | |
| clarkb | dims: we need more nodes, I am currently cleaning up leaked nodes in hpcloud to hopefully free up capacity | 00:15 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-zuul: Modernize template syntax https://review.openstack.org/157136 | 00:16 |
| dims | clarkb: thanks (i saw the "zuul jobs" and "gerrit events" charts go south and thought i may have to do something) | 00:17 |
| *** otter768 has quit IRC | 00:17 | |
| anteaya | dims: thanks for keeping an eye on that | 00:17 |
| *** sdake_ has quit IRC | 00:17 | |
| dims | anteaya: :) | 00:18 |
| anteaya | dims: when the gate shifts and zuul needs to calculate the merge-check queue it can spike for a few minutes | 00:18 |
| anteaya | I've seen event go up over 8000 | 00:18 |
| greghaynes | asselin: Hey, im working on some fakes testing and im hitting the bug you patch https://review.openstack.org/#/c/162313 fixes, id like to base a test off it but I need to rebase your change off master if thats ok? | 00:18 |
| dims | gotcha | 00:18 |
| openstackgerrit | Nan Liu proposed openstack-infra/puppet-elasticsearch: Use archive to fetch/checksum instead of wget. https://review.openstack.org/164908 | 00:18 |
| anteaya | but they don't last long | 00:18 |
| anteaya | dims: but you are looking in the right spots | 00:18 |
| asselin | greghaynes, sure go ahead | 00:19 |
| dims | thanks anteaya | 00:20 |
| anteaya | :) | 00:21 |
| *** rfolco has quit IRC | 00:21 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add docs jobs for openstack-infra/openstackid repo https://review.openstack.org/165661 | 00:21 |
| fungi | ianw: i have no idea. it was agreed as a good thing to try, but got no reviews once uploaded | 00:22 |
| *** jamielennox|away is now known as jamielennox | 00:23 | |
| ianw | fungi: i mean the failure to boot; just trying to grab some logs on this now | 00:23 |
| clarkb | fungi: maybe just apply by hand and see if it makes any difference? | 00:24 |
| fungi | ianw: we get no logs. we never get an instance returned by nova when we request it to boot | 00:24 |
| fungi | clarkb: i was tempted to do that, but then worried i might accidentally leave puppet disabled on nodepool so thought better of it | 00:25 |
| fungi | clarkb: so if i was going to apply it by hand, better to just self-approve that change i guess (i proposed a wip revert along with it for everyone's convenience) | 00:25 |
| fungi | which i guess i can do | 00:26 |
| *** tkelsey has joined #openstack-infra | 00:26 | |
| *** gyee has quit IRC | 00:30 | |
| *** tkelsey has quit IRC | 00:31 | |
| *** __TheDodd__ has quit IRC | 00:32 | |
| *** MarkAtwood has quit IRC | 00:32 | |
| ianw | fungi: alright, well if i can help i will, obviously console logs, if it gets that far, would be good | 00:32 |
| *** jeffgdl has quit IRC | 00:33 | |
| *** ZZelle_ has quit IRC | 00:34 | |
| *** tonytan4ever has quit IRC | 00:34 | |
| *** Sukhdev has quit IRC | 00:34 | |
| *** markvoelker has quit IRC | 00:37 | |
| fungi | ianw: i connected to the console of one at random. i'm getting "boot: No DEFAULT or UI configuration directive found!" looping over and over there | 00:37 |
| fungi | but i'm about to knock off for the night. getting lateish here | 00:37 |
| *** e0ne has quit IRC | 00:38 | |
| *** tonytan4ever has joined #openstack-infra | 00:38 | |
| fungi | ianw: i'm guessing it's something not quite right with the bootloader installed to the image | 00:39 |
| patrickeast | hi, i have a nodepool question… is there a way to setup nodes with multiple floating ip’s each coming from a different pool? | 00:41 |
| patrickeast | i can make nodes with multiple networks, and a single floating ip from the pool specified in the providers ‘pool’ setting | 00:41 |
| patrickeast | but didn’t see anything for other nics that are configured | 00:42 |
| anteaya | patrickeast: did you get a private network for nodepool? | 00:42 |
| ianw | fungi: alright, i'll try to replicate a little | 00:42 |
| patrickeast | anteaya: heh still waiting on that, in the mean time i am playing with setting up iscsi multipathing to our flash array on test nodes | 00:43 |
| patrickeast | anteaya: should be soon, waiting on some new cables or something they needed to put it into the dmz | 00:43 |
| fungi | ianw: thanks! | 00:43 |
| *** baoli has quit IRC | 00:43 | |
| anteaya | patrickeast: okay | 00:43 |
| anteaya | patrickeast: and I have no idea how to answer your original question | 00:44 |
| *** marun has quit IRC | 00:45 | |
| *** asettle is now known as asettle-lunch | 00:45 | |
| clarkb | alien nodes are cleaned up in hpcloud now | 00:45 |
| clarkb | patrickeast: there is a way to specify multiple networks I don't think you can specify multiple floating ips though | 00:46 |
| patrickeast | clarkb: yea that lines up with what i’m seeing in the docs | 00:46 |
| patrickeast | figured I would check before hacking away at the code | 00:47 |
| anteaya | clarkb: yay | 00:47 |
| clarkb | patrickeast: what are you trying to accomplish with multiple floating ips? | 00:47 |
| patrickeast | we need to route traffic through a handful of 10Gb nic’s on the compute nodes to the test nodes so they can reach the iscsi ports on our flash array to test out multipathing | 00:48 |
| patrickeast | so each node would have 3 networks, management and 2x data | 00:48 |
| *** cdent has quit IRC | 00:48 | |
| openstackgerrit | Merged openstack-infra/system-config: Temporarily stop adding devstack-f20 nodes in rax https://review.openstack.org/165620 | 00:48 |
| patrickeast | all 3 of which are on different “external” networks as far as openstack is concerned | 00:48 |
| *** mfink_ has joined #openstack-infra | 00:49 | |
| clarkb | patrickeast: couldn't you do that all without floating ips? | 00:49 |
| clarkb | patrickeast: just put them on the appropriate network | 00:49 |
| *** claudiub has quit IRC | 00:50 | |
| *** ibiris is now known as ibiris_away | 00:50 | |
| patrickeast | clarkb: mm that would be easier, can I put instances directly on a neutron network/subnet that is external w/ a bridge? | 00:51 |
| ianw | fungi: yeah, just upgrading kernel & rebooting f20 on rax it's not happy with that error, so that's kind of good in that it's not specific ... | 00:52 |
| * patrickeast goes back to reading neutron docs | 00:52 | |
| fungi | ianw: awesome--glad it's not hard to reproduce. hopefully shortly we'll know whether hpcloud does any better in that regard | 00:53 |
| clarkb | patrickeast: I am not sure how neutron does it but if neutron can set up NAT rules in iptables for internal:external mappings then it should be able to just use IPs on those networks direclty | 00:53 |
| *** nanliu has quit IRC | 00:53 | |
| clarkb | I certainly hope that neutron allows that :/ | 00:53 |
| *** yamamoto has quit IRC | 00:53 | |
| fungi | ianw: is the depends-on in https://review.openstack.org/164901 a typo? | 00:53 |
| patrickeast | clarkb: hehe yea i would think so too | 00:54 |
| *** bhuvan has quit IRC | 00:54 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support to configure boolean parameter in PBT https://review.openstack.org/146665 | 00:55 |
| ianw | fungi: oh, that got abandoned ... should be the previous one; just a tick | 00:55 |
| fungi | otherwise lgtm | 00:55 |
| *** ddieterl_ has joined #openstack-infra | 00:55 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Add nodepool config-validation environment https://review.openstack.org/164901 | 00:55 |
| ianw | fungi: ^ updated, thanks | 00:56 |
| *** Rockyg has quit IRC | 00:56 | |
| *** ddieterly has quit IRC | 00:56 | |
| fungi | i see a devstack-f20 node building in hpcloud-b5 for 0.8 hours, so i think it's just unhappy everywheres | 00:57 |
| fungi | everwheres | 00:57 |
| *** isq has quit IRC | 00:57 | |
| fungi | i'll go ahead and approve the revert | 00:58 |
| *** jp_at_hp has quit IRC | 00:58 | |
| mordred | ianw: so - you enjoy diskimage-builder, right? | 00:58 |
| mordred | :) | 00:58 |
| mordred | ianw: I made this a while back: http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/ubuntu-minimal ... and a similar one http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/centos-minimal | 00:59 |
| ianw | mordred: well ... sometimes. my last change took about 6 months to merge | 00:59 |
| mordred | ianw: :) | 00:59 |
| mordred | ianw: the ubuntu one does what Id expect - which is make a super minimal base image that's similar in size and scope to the ubuntu docker image | 01:00 |
| mordred | ianw: the centos one is ... larger - and I think I did it wrong ... and I don't have a fedora one at all | 01:00 |
| *** bknudson has joined #openstack-infra | 01:00 | |
| *** Sukhdev has joined #openstack-infra | 01:00 | |
| mordred | ianw: if you get bored at some point and wanted to make a better one for centos/fedora that wound up with something tiny like a docker image ... I will send a freshly dried alpaca to your house | 01:00 |
| ianw | mordred: so one thing that stands out immediately is http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/centos-minimal/pkg-map <- redhat-lsb brings in a whole bunch of stuff, X and cups etc | 01:01 |
| ianw | that should probably be redhat-lsb-core | 01:01 |
| mordred | ianw: yeah - it's the only way I could figure out how to get the lsb_release command ... AH | 01:01 |
| mordred | see - this is you knowing more things than me | 01:01 |
| *** tonytan4ever has quit IRC | 01:01 | |
| mordred | ianw: I was also told that perhaps rinse is not actually a great way to accomplish this ... and perhaps something about just using rpm with an alternate root *me waves hands wildly* | 01:02 |
| fungi | mordred: also `rpm -qf lsb_release` | 01:02 |
| fungi | that is assuming you already have the packages installed, that will tell you which one is actually providing it | 01:03 |
| fungi | i'm sure yum has something akin to apt-file for searching filenames of packages available on the package mirror but that you haven't installed, i just don't know the cli | 01:04 |
| ianw | mordred: so don't we have the whole atomic thing? is there overlap here? | 01:04 |
| mordred | ianw: it's not quite the same thing ... I don't actually want docker | 01:05 |
| mordred | I want actual base images - just small without the things that my distro providers think I want in a "cloud image" | 01:05 |
| mordred | because I have this whole puppet infrastructure where I declare the dependencies I actually want :) | 01:05 |
| patrickeast | clarkb: hah nice, thanks for the suggestion… I was making that much more painful than it had to be, although now we need to setup the /etc/network/interface.d/ script for the node with its static ip (the floating ips let us use the neutron dhcp), do you know if the /etc/nodepool/node_private file has *all* of the private ip addresses for a node so we can set it up with a ready script? | 01:05 |
| mordred | patrickeast: fyi - I'm in the middle of replacing the openstack calls in nodepool with the shade library - so if you do find that there are some places where you need enhancements to the openstack interactions - like the floating-ips thing or richer handling of multiple nics - it's probably functionality we want to add to shade too ... | 01:06 |
| *** bhunter71 has joined #openstack-infra | 01:07 | |
| patrickeast | mordred: oo nice, is there a launchpad or something for it where I could document any feature requests? | 01:08 |
| ianw | mordred: you don't want a bootable image though? | 01:09 |
| mordred | patrickeast: https://storyboard.openstack.org/#!/project/760 | 01:09 |
| *** Longgeek has joined #openstack-infra | 01:09 | |
| mordred | ianw: no, bootable is good - but there are already other elements to handle things like grub and whatnot | 01:09 |
| mordred | ianw: the 'vm' element handles that | 01:10 |
| ianw | mordred: right, ok, so rinse gives you even less than downloading the upstream .qcow | 01:11 |
| *** isq has joined #openstack-infra | 01:11 | |
| ianw | upstream .qcow cloud images | 01:11 |
| *** garyh has joined #openstack-infra | 01:12 | |
| *** pelix has quit IRC | 01:14 | |
| *** YorikSar has quit IRC | 01:15 | |
| *** achanda has joined #openstack-infra | 01:15 | |
| *** markvoelker has joined #openstack-infra | 01:17 | |
| fungi | oh! not approving the revert yet. we've got a used devstack-f20 node in hpcloud now | 01:17 |
| fungi | so it does seem we can still boot them there | 01:17 |
| ianw | fungi: so it's saying "boot: No DEFAULT or UI configuration directive found!" because there's no default configuration directive. who'd have thunk ... | 01:18 |
| ianw | fungi: this looks like rackspace fiddling with extlinux.conf again ... similar thing happened with f21 | 01:18 |
| fungi | ianw: yeah, i've also had problems with them replacing grub2 with pv-grub in the mbr of my personal instances as well | 01:19 |
| mordred | ianw, fungi: is this file-injection feeling things? | 01:20 |
| fungi | not sure, but it would be nice if they would just not feel obligated to mess with bootloaders | 01:20 |
| mordred | ianw, fungi: I think Ng got glance turned on on his rax account - so hopefully we shoudl have the non-nova-agent based element soon | 01:20 |
| ianw | mordred: no, i think they edit /boot/extlinux.conf for some reason, and then the scripts in the rpm files that update it don't work properly | 01:20 |
| *** achanda has quit IRC | 01:21 | |
| mordred | oh - you mean in the images they make for us | 01:21 |
| ianw | yep | 01:21 |
| mordred | yeah - I'm very much looking forward to getting our cloud providers out of the pipeline for making images for us altogether | 01:21 |
| * mordred promises he's working on that | 01:22 | |
| *** markvoelker has quit IRC | 01:22 | |
| ianw | fungi: i'll communicate to jroll the issue, i can put a work-around into the prepare scripts | 01:22 |
| *** mwagner_lap has joined #openstack-infra | 01:22 | |
| *** garyh has quit IRC | 01:23 | |
| jroll | ianw: ? | 01:23 |
| fungi | that was fast ;) | 01:24 |
| jroll | >.> | 01:24 |
| * jroll runs away | 01:24 | |
| ianw | jroll: the rax f20 image is broken in a similar way to the f21 where extlinux.conf gets mangled by kernel updates. i'll get exact details and email | 01:24 |
| jroll | ianw: shoot me an email or else I'll forget | 01:24 |
| jroll | yeah | 01:24 |
| jroll | thanks! | 01:24 |
| *** mwagner_lap has quit IRC | 01:27 | |
| *** bhuvan has joined #openstack-infra | 01:28 | |
| *** garyh has joined #openstack-infra | 01:28 | |
| fungi | thanks as always jroll! | 01:30 |
| jroll | :) | 01:34 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool: Move nodepool creation in tests to common method https://review.openstack.org/165581 | 01:35 |
| openstackgerrit | Sukhdev Kapur proposed openstack-infra/project-config: Adding create permissions for networking-arista https://review.openstack.org/165673 | 01:36 |
| *** garyh has quit IRC | 01:38 | |
| *** mwagner_lap has joined #openstack-infra | 01:40 | |
| *** asettle-lunch is now known as asettle | 01:43 | |
| *** ivar-laz_ has joined #openstack-infra | 01:43 | |
| Sukhdev | fungi: Hi there - I just pushed the patch based upon your suggestion. Can you please give it a review - https://review.openstack.org/#/c/165673/ | 01:44 |
| *** spzala has quit IRC | 01:46 | |
| *** ivar-lazzaro has quit IRC | 01:46 | |
| *** ivar-laz_ has quit IRC | 01:47 | |
| *** isq has quit IRC | 01:48 | |
| *** ChuckC has joined #openstack-infra | 01:49 | |
| *** unicell has quit IRC | 01:51 | |
| rlucio | hey guys, i linked my ci to the sandbox repo to test, and i checked back and both of my slaves are offline saying "offline due to gearman request"... anyone know what that means? | 01:52 |
| fungi | rlucio: it means you've copied our configuration and are using an OFFLINE_NODE_WHEN_COMPLETE parameter in your jobs. we do this because we don't reuse workers, but delete them after a job completes and use a fresh one for each new job | 01:53 |
| *** unicell has joined #openstack-infra | 01:54 | |
| *** unicell has quit IRC | 01:54 | |
| *** unicell has joined #openstack-infra | 01:54 | |
| rlucio | fungi: ah ha, cool. yea i have static workers so i'll have to change that | 01:55 |
| Sukhdev | anteaya: Ping | 01:55 |
| rlucio | fungi: thanks! | 01:55 |
| fungi | rlucio: we have some static workers too, and for those we have a "reusable_node" parameter function which unsets that parameter for us | 01:55 |
| *** isq has joined #openstack-infra | 01:55 | |
| rlucio | fungi: yea i see that defined in zuul/openstack_functions.py | 01:56 |
| *** marun has joined #openstack-infra | 01:56 | |
| fungi | rlucio: if you look at our zuul/layout.yaml you'll see where we set it for jobs we're running on reusable nodes | 01:57 |
| *** bhunter71 has quit IRC | 01:57 | |
| *** mwagner_lap has quit IRC | 01:58 | |
| Sukhdev | Hi Infra cores - I need one more blessing on this patch, please - https://review.openstack.org/#/c/165673/ | 01:58 |
| fungi | rlucio: i take that back. we just apply it based on pattern matches of the job names these days. we used to set it from layout.yaml but i guess not any longer | 01:59 |
| rlucio | fungi: right, just looking at that now, in the jobs: section of layout.yaml | 01:59 |
| *** SumitNaiksatam has joined #openstack-infra | 02:00 | |
| rlucio | fungi: simple enough to fix :) | 02:00 |
| openstackgerrit | Merged openstack-infra/nodepool: Fix Nodepool DIB Uploads to Glance w/o region_name https://review.openstack.org/162313 | 02:00 |
| *** marun has quit IRC | 02:01 | |
| fungi | rlucio: yep. or just don't bother with the parameter function script we use. depending on your situation it's like a lot of unnecessary complication | 02:01 |
| fungi | s/like/likely/ | 02:01 |
| *** ajmiller has joined #openstack-infra | 02:01 | |
| *** unicell has quit IRC | 02:02 | |
| *** ddieterly has joined #openstack-infra | 02:06 | |
| *** ddieterl_ has quit IRC | 02:06 | |
| *** mwagner_lap has joined #openstack-infra | 02:11 | |
| *** Sukhdev has quit IRC | 02:11 | |
| *** harlowja_ is now known as harlowja_away | 02:14 | |
| *** otter768 has joined #openstack-infra | 02:14 | |
| *** Alexandra_ has joined #openstack-infra | 02:16 | |
| *** asettle has quit IRC | 02:16 | |
| *** markvoelker has joined #openstack-infra | 02:18 | |
| *** otter768 has quit IRC | 02:18 | |
| *** reed has quit IRC | 02:20 | |
| *** macjack1 has joined #openstack-infra | 02:22 | |
| *** markvoelker has quit IRC | 02:23 | |
| *** macjack1 has quit IRC | 02:23 | |
| *** yamahata has quit IRC | 02:24 | |
| *** achanda has joined #openstack-infra | 02:27 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 02:27 | |
| *** tkelsey has joined #openstack-infra | 02:28 | |
| *** baoli has joined #openstack-infra | 02:28 | |
| *** dims has quit IRC | 02:28 | |
| greghaynes | clarkb: we need a good novaclient fake for this to work | 02:29 |
| greghaynes | :( | 02:29 |
| *** tkelsey has quit IRC | 02:32 | |
| *** kaisers has joined #openstack-infra | 02:33 | |
| *** kaisers1 has quit IRC | 02:33 | |
| *** zz_dimtruck is now known as dimtruck | 02:35 | |
| *** dmorita_ has quit IRC | 02:36 | |
| *** otter768 has joined #openstack-infra | 02:38 | |
| *** unicell has joined #openstack-infra | 02:40 | |
| *** macjack1 has joined #openstack-infra | 02:42 | |
| *** unicell has quit IRC | 02:44 | |
| *** unicell has joined #openstack-infra | 02:45 | |
| *** macjack1 has quit IRC | 02:48 | |
| ianw | jroll: looks like it's a fedora bug | 02:49 |
| jroll | ianw: cool, I don't feel so bad now! | 02:50 |
| * jroll deflects to redhat employees | 02:50 | |
| *** erlon has quit IRC | 02:51 | |
| * ianw curses anyone who thinks "i've got a need to open and close a bunch of files and mangle a lot of strings ... i think i'll write this in C" | 02:52 | |
| jroll | s/open and close a bunch of files and mangle a lot of strings/do things with computers/ | 02:52 |
| jroll | (mostly joking) | 02:52 |
| *** mrmartin has joined #openstack-infra | 02:55 | |
| *** harlowja_at_home has joined #openstack-infra | 02:56 | |
| *** patrickeast has quit IRC | 02:58 | |
| *** spzala has joined #openstack-infra | 03:01 | |
| *** coolsvap|afk is now known as coolsvap | 03:04 | |
| *** stevemar has joined #openstack-infra | 03:08 | |
| *** harlowja_at_home has quit IRC | 03:11 | |
| *** spzala has quit IRC | 03:11 | |
| *** otter768 has quit IRC | 03:13 | |
| dougwig | hi infra. is there an easy way to see a list of a particular job? i tried here, but it's empty: https://jenkins.openstack.org/job/check-neutron-lbaas-dsvm-api/ i'm wanting to monitor it for stability. | 03:14 |
| *** mmedvede has quit IRC | 03:17 | |
| *** Somay has joined #openstack-infra | 03:19 | |
| *** sdake_ has joined #openstack-infra | 03:22 | |
| ianw | dougwig: i think you probably want to consult http://graphite.openstack.org/ | 03:25 |
| *** sdake has quit IRC | 03:26 | |
| dougwig | ianw: ty | 03:28 |
| *** camunoz_gone is now known as camunoz | 03:29 | |
| *** coolsvap is now known as coolsvap|afk | 03:32 | |
| *** camunoz has joined #openstack-infra | 03:33 | |
| *** changbl has joined #openstack-infra | 03:34 | |
| *** sputnik13 has joined #openstack-infra | 03:36 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Work-around grubby boot failure on Fedora 20 https://review.openstack.org/165681 | 03:38 |
| *** camunoz has quit IRC | 03:38 | |
| *** dimtruck is now known as zz_dimtruck | 03:39 | |
| *** coolsvap|afk is now known as coolsvap | 03:39 | |
| ianw | fungi: ^ when you look at scrollback | 03:40 |
| *** yamamoto has joined #openstack-infra | 03:40 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Revert "Temporarily stop adding devstack-f20 nodes in rax" https://review.openstack.org/165621 | 03:41 |
| *** achanda has quit IRC | 03:42 | |
| *** greghaynes has quit IRC | 03:43 | |
| *** achanda has joined #openstack-infra | 03:44 | |
| *** macjack has joined #openstack-infra | 03:46 | |
| *** mrmartin has quit IRC | 03:47 | |
| *** ddieterly has quit IRC | 03:47 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Move nodepool creation in tests to common method https://review.openstack.org/165581 | 03:49 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Monkeypatch Fake Clients for tests https://review.openstack.org/165682 | 03:49 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Add node deletion tests https://review.openstack.org/165582 | 03:50 |
| *** woodster_ has quit IRC | 03:50 | |
| *** camunoz has joined #openstack-infra | 03:51 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Move nodepool creation in tests to common method https://review.openstack.org/165581 | 03:54 |
| *** garyh has joined #openstack-infra | 03:54 | |
| *** tqtran has quit IRC | 03:55 | |
| *** sdake has joined #openstack-infra | 03:55 | |
| *** macjack has quit IRC | 03:56 | |
| *** tqtran has joined #openstack-infra | 03:56 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Monkeypatch Fake Clients for tests https://review.openstack.org/165682 | 03:56 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Add node deletion tests https://review.openstack.org/165582 | 03:56 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 03:57 | |
| *** sdake_ has quit IRC | 03:59 | |
| *** garyh has quit IRC | 04:00 | |
| *** sdake has quit IRC | 04:04 | |
| *** achanda has quit IRC | 04:05 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Move nodepool creation in tests to common method https://review.openstack.org/165581 | 04:06 |
| *** jamielennox is now known as jamielennox|away | 04:06 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Monkeypatch Fake Clients for tests https://review.openstack.org/165682 | 04:06 |
| *** VijayTripathi has joined #openstack-infra | 04:09 | |
| *** baoli has quit IRC | 04:10 | |
| *** mmedvede has joined #openstack-infra | 04:10 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Add node deletion tests https://review.openstack.org/165582 | 04:11 |
| *** Alexandra_ has quit IRC | 04:15 | |
| *** dims has joined #openstack-infra | 04:19 | |
| *** markvoelker has joined #openstack-infra | 04:19 | |
| *** koolhead17 has joined #openstack-infra | 04:22 | |
| *** markvoelker has quit IRC | 04:24 | |
| *** carl_baldwin has joined #openstack-infra | 04:26 | |
| *** dims has quit IRC | 04:27 | |
| *** Sukhdev has joined #openstack-infra | 04:31 | |
| *** markvan has quit IRC | 04:33 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Fix jquery setup on Gerrit server. https://review.openstack.org/165145 | 04:33 |
| *** mmedvede has quit IRC | 04:34 | |
| *** markvan has joined #openstack-infra | 04:35 | |
| *** achanda has joined #openstack-infra | 04:37 | |
| *** sdake has joined #openstack-infra | 04:38 | |
| *** dboik_ has quit IRC | 04:39 | |
| *** amotoki has joined #openstack-infra | 04:40 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Monkeypatch Fake Clients for tests https://review.openstack.org/165682 | 04:40 |
| *** achanda has quit IRC | 04:41 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Move nodepool creation in tests to common method https://review.openstack.org/165581 | 04:45 |
| *** dboik has joined #openstack-infra | 04:46 | |
| *** harlowja_at_home has joined #openstack-infra | 04:47 | |
| *** ddieterly has joined #openstack-infra | 04:48 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Monkeypatch Fake Clients for tests https://review.openstack.org/165682 | 04:51 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Add node deletion tests https://review.openstack.org/165582 | 04:51 |
| *** ddieterly has quit IRC | 04:53 | |
| *** garyh has joined #openstack-infra | 05:00 | |
| *** __afazekas is now known as afazekas | 05:03 | |
| *** yamamoto has quit IRC | 05:07 | |
| *** achanda has joined #openstack-infra | 05:08 | |
| *** garyh has quit IRC | 05:09 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding https://review.openstack.org/164264 | 05:11 |
| *** sdake has quit IRC | 05:12 | |
| *** otter768 has joined #openstack-infra | 05:14 | |
| *** otter768 has quit IRC | 05:18 | |
| *** markvoelker has joined #openstack-infra | 05:20 | |
| *** jyuso1 has joined #openstack-infra | 05:25 | |
| *** markvoelker has quit IRC | 05:25 | |
| *** liusheng has quit IRC | 05:26 | |
| *** liusheng has joined #openstack-infra | 05:26 | |
| *** armax has quit IRC | 05:27 | |
| *** mmedvede has joined #openstack-infra | 05:28 | |
| *** yfried|afk is now known as yfried | 05:31 | |
| *** yamamoto has joined #openstack-infra | 05:32 | |
| *** dannywilson has quit IRC | 05:34 | |
| *** greghaynes has joined #openstack-infra | 05:39 | |
| *** carl_baldwin has quit IRC | 05:39 | |
| *** harlowja_at_home has quit IRC | 05:40 | |
| *** coolsvap is now known as coolsvap|afk | 05:41 | |
| *** ChuckC has quit IRC | 05:45 | |
| *** MarkAtwood has joined #openstack-infra | 05:45 | |
| *** coolsvap|afk is now known as coolsvap | 05:48 | |
| *** ddieterly has joined #openstack-infra | 05:49 | |
| *** ddieterly has quit IRC | 05:53 | |
| *** MarkAtwood has quit IRC | 05:55 | |
| *** ibiris_away is now known as ibiris | 05:58 | |
| *** yfried is now known as yfried|afk | 05:59 | |
| *** stevemar has quit IRC | 06:00 | |
| *** sarob has quit IRC | 06:05 | |
| *** subscope_ has joined #openstack-infra | 06:09 | |
| *** garyh has joined #openstack-infra | 06:10 | |
| *** BharatK has joined #openstack-infra | 06:10 | |
| *** yfried|afk has quit IRC | 06:12 | |
| *** dims has joined #openstack-infra | 06:13 | |
| *** dims has quit IRC | 06:19 | |
| *** garyh has quit IRC | 06:20 | |
| *** markvoelker has joined #openstack-infra | 06:21 | |
| *** bswartz has quit IRC | 06:24 | |
| *** markvoelker has quit IRC | 06:26 | |
| *** achanda has quit IRC | 06:28 | |
| *** tkelsey has joined #openstack-infra | 06:29 | |
| *** Sukhdev has quit IRC | 06:31 | |
| *** bswartz has joined #openstack-infra | 06:32 | |
| *** pcaruana has quit IRC | 06:33 | |
| *** tkelsey has quit IRC | 06:33 | |
| *** mfink_ has quit IRC | 06:35 | |
| *** mfink_ has joined #openstack-infra | 06:37 | |
| *** ddieterly has joined #openstack-infra | 06:50 | |
| *** kiseok7_ has joined #openstack-infra | 06:50 | |
| *** achanda has joined #openstack-infra | 06:52 | |
| *** jcoufal has joined #openstack-infra | 06:53 | |
| *** ddieterly has quit IRC | 06:55 | |
| *** achanda has quit IRC | 07:00 | |
| *** ujuc has joined #openstack-infra | 07:04 | |
| *** achanda has joined #openstack-infra | 07:04 | |
| *** sarob has joined #openstack-infra | 07:05 | |
| *** subscope_ has quit IRC | 07:07 | |
| *** jyuso1 has quit IRC | 07:08 | |
| *** pblaho has joined #openstack-infra | 07:09 | |
| *** sarob has quit IRC | 07:11 | |
| *** kiseok7_ has quit IRC | 07:11 | |
| openstackgerrit | Fatih Degirmenci proposed openstack-infra/jenkins-job-builder: Pass through git revision to triggered jobs https://review.openstack.org/163149 | 07:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix failure detection in irc access check https://review.openstack.org/165508 | 07:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix failure detection in irc access check https://review.openstack.org/165508 | 07:13 |
| *** tnovacik has joined #openstack-infra | 07:13 | |
| *** otter768 has joined #openstack-infra | 07:15 | |
| *** bhuvan has quit IRC | 07:18 | |
| *** otter768 has quit IRC | 07:19 | |
| *** garyh has joined #openstack-infra | 07:20 | |
| *** markvoelker has joined #openstack-infra | 07:22 | |
| *** yfried|afk has joined #openstack-infra | 07:25 | |
| *** markvoelker has quit IRC | 07:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Move barbican client experimental job to voting gate job https://review.openstack.org/165534 | 07:27 |
| openstackgerrit | Merged openstack-infra/project-config: Adding create permissions for networking-arista https://review.openstack.org/165673 | 07:28 |
| *** yfried has joined #openstack-infra | 07:30 | |
| *** yfried|afk has quit IRC | 07:30 | |
| *** garyh has quit IRC | 07:31 | |
| *** scheuran has joined #openstack-infra | 07:33 | |
| *** VijayTripathi has quit IRC | 07:35 | |
| *** achanda has quit IRC | 07:37 | |
| *** HeOS has quit IRC | 07:38 | |
| *** ildikov has quit IRC | 07:38 | |
| *** mpaolino has joined #openstack-infra | 07:42 | |
| *** Bsony has joined #openstack-infra | 07:45 | |
| *** chlong has quit IRC | 07:47 | |
| *** mpavone has joined #openstack-infra | 07:50 | |
| *** ddieterly has joined #openstack-infra | 07:51 | |
| *** SumitNaiksatam has quit IRC | 07:51 | |
| *** belmoreira has joined #openstack-infra | 07:53 | |
| *** ominakov has joined #openstack-infra | 07:54 | |
| *** yfried is now known as yfried|afk | 07:54 | |
| *** ominakov has quit IRC | 07:54 | |
| *** ominakov_ has joined #openstack-infra | 07:54 | |
| *** ddieterly has quit IRC | 07:55 | |
| *** teran_ has joined #openstack-infra | 07:56 | |
| *** flaper87 has quit IRC | 07:56 | |
| *** flaper87 has joined #openstack-infra | 07:56 | |
| *** teran has quit IRC | 07:57 | |
| *** scheuran has quit IRC | 07:58 | |
| *** scheuran has joined #openstack-infra | 07:58 | |
| *** mrda is now known as mrda-away | 07:58 | |
| *** yfried|afk is now known as yfried | 08:01 | |
| *** mpaolino has quit IRC | 08:02 | |
| *** mpaolino has joined #openstack-infra | 08:03 | |
| *** dims has joined #openstack-infra | 08:04 | |
| *** markus_z has joined #openstack-infra | 08:07 | |
| *** dims has quit IRC | 08:10 | |
| *** hashar has joined #openstack-infra | 08:11 | |
| *** e0ne has joined #openstack-infra | 08:14 | |
| *** ildikov has joined #openstack-infra | 08:14 | |
| *** koolhead17 has quit IRC | 08:15 | |
| *** e0ne has quit IRC | 08:15 | |
| GheRivero | mornig all | 08:16 |
| *** koolhead17 has joined #openstack-infra | 08:16 | |
| *** jp_at_hp has joined #openstack-infra | 08:17 | |
| *** arxcruz has joined #openstack-infra | 08:17 | |
| *** jp_at_hp has quit IRC | 08:18 | |
| *** yfried is now known as yfried|afk | 08:19 | |
| *** koolhead17 has quit IRC | 08:21 | |
| *** tkelsey has joined #openstack-infra | 08:28 | |
| *** deepakcs has joined #openstack-infra | 08:29 | |
| *** dboik has quit IRC | 08:31 | |
| *** garyh has joined #openstack-infra | 08:31 | |
| *** mpaolino has quit IRC | 08:36 | |
| *** jistr has joined #openstack-infra | 08:37 | |
| *** mpaolino has joined #openstack-infra | 08:38 | |
| *** spredzy|afk is now known as spredzy | 08:39 | |
| *** yfried|afk is now known as yfried | 08:39 | |
| *** teran_ has quit IRC | 08:40 | |
| *** garyh has quit IRC | 08:41 | |
| *** fabo has quit IRC | 08:43 | |
| *** mfink_ has quit IRC | 08:49 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-cgit: Add configurable setting for MaxClients https://review.openstack.org/160885 | 08:53 |
| *** shardy_z is now known as shardy | 08:55 | |
| *** jlanoux has joined #openstack-infra | 08:56 | |
| *** flip214 has joined #openstack-infra | 08:57 | |
| *** andreykurilin_ has joined #openstack-infra | 08:58 | |
| *** nilasae has joined #openstack-infra | 08:59 | |
| flip214 | clarkb: ping | 09:00 |
| *** ujuc has quit IRC | 09:02 | |
| *** koolhead17 has joined #openstack-infra | 09:04 | |
| *** YorikSar has joined #openstack-infra | 09:05 | |
| *** fabo has joined #openstack-infra | 09:05 | |
| *** tqtran has quit IRC | 09:09 | |
| *** mfink_ has joined #openstack-infra | 09:11 | |
| *** koolhead17 has quit IRC | 09:13 | |
| *** skolekonov has joined #openstack-infra | 09:14 | |
| *** otter768 has joined #openstack-infra | 09:16 | |
| openstackgerrit | Anthony Michon proposed openstack-infra/project-config: Add cerberus-dashboard, python-cerberusclient, sticks, sticks-dashboard, python-sticksclient https://review.openstack.org/163876 | 09:19 |
| *** otter768 has quit IRC | 09:20 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:20 | |
| *** dims has joined #openstack-infra | 09:22 | |
| *** markvoelker has joined #openstack-infra | 09:23 | |
| *** HeOS has joined #openstack-infra | 09:23 | |
| *** dims has quit IRC | 09:26 | |
| *** Longgeek has quit IRC | 09:27 | |
| *** andreykurilin_ has quit IRC | 09:27 | |
| *** andreykurilin_ has joined #openstack-infra | 09:28 | |
| *** markvoelker has quit IRC | 09:28 | |
| *** freerunner has quit IRC | 09:28 | |
| *** Longgeek has joined #openstack-infra | 09:29 | |
| *** freerunner has joined #openstack-infra | 09:29 | |
| *** freerunner has quit IRC | 09:29 | |
| *** freerunner has joined #openstack-infra | 09:31 | |
| *** Longgeek has quit IRC | 09:31 | |
| *** freerunner has quit IRC | 09:31 | |
| *** freerunner has joined #openstack-infra | 09:33 | |
| *** mfink_ has quit IRC | 09:33 | |
| *** freerunner has quit IRC | 09:33 | |
| *** Somay has quit IRC | 09:35 | |
| *** ihrachyshka has joined #openstack-infra | 09:35 | |
| *** freerunner has joined #openstack-infra | 09:37 | |
| *** derekh has joined #openstack-infra | 09:37 | |
| *** freerunner has quit IRC | 09:37 | |
| *** Longgeek has joined #openstack-infra | 09:37 | |
| *** freerunner has joined #openstack-infra | 09:38 | |
| *** freerunner has quit IRC | 09:38 | |
| *** freerunner has joined #openstack-infra | 09:39 | |
| *** freerunner has quit IRC | 09:39 | |
| *** teran has joined #openstack-infra | 09:40 | |
| *** freerunner has joined #openstack-infra | 09:40 | |
| *** freerunner has quit IRC | 09:40 | |
| *** liusheng has quit IRC | 09:41 | |
| *** garyh has joined #openstack-infra | 09:42 | |
| *** liusheng has joined #openstack-infra | 09:42 | |
| *** teran_ has joined #openstack-infra | 09:43 | |
| *** ZZelle has quit IRC | 09:43 | |
| *** mfink_ has joined #openstack-infra | 09:43 | |
| *** ZZelle has joined #openstack-infra | 09:43 | |
| *** ihrachyshka has quit IRC | 09:44 | |
| *** freerunner has joined #openstack-infra | 09:44 | |
| *** teran has quit IRC | 09:45 | |
| *** freerunner has quit IRC | 09:45 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-graphite: Add ability to configure retention policy https://review.openstack.org/161679 | 09:45 |
| *** ihrachyshka has joined #openstack-infra | 09:45 | |
| *** dims has joined #openstack-infra | 09:45 | |
| *** freerunner has joined #openstack-infra | 09:46 | |
| *** freerunner has quit IRC | 09:46 | |
| *** rlucio has quit IRC | 09:47 | |
| *** andreykurilin_ has quit IRC | 09:47 | |
| *** freerunner has joined #openstack-infra | 09:47 | |
| *** freerunner has quit IRC | 09:48 | |
| *** amotoki has quit IRC | 09:48 | |
| *** freerunner has joined #openstack-infra | 09:50 | |
| *** Trozz has quit IRC | 09:50 | |
| *** mpaolino has quit IRC | 09:52 | |
| *** ddieterly has joined #openstack-infra | 09:52 | |
| *** garyh has quit IRC | 09:52 | |
| *** Trozz has joined #openstack-infra | 09:53 | |
| *** mpavone has quit IRC | 09:53 | |
| *** dtantsur|afk is now known as dtantsur | 09:54 | |
| *** yamamoto has quit IRC | 09:54 | |
| *** Hal1 has joined #openstack-infra | 09:54 | |
| *** ujuc has joined #openstack-infra | 09:54 | |
| *** Trozz is now known as Trozz_ | 09:55 | |
| *** Trozz_ is now known as Trozz | 09:55 | |
| *** ssam2 has joined #openstack-infra | 09:56 | |
| *** ddieterly has quit IRC | 09:57 | |
| *** pelix has joined #openstack-infra | 10:02 | |
| *** ghostpl_ has joined #openstack-infra | 10:04 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jenkins: Add extension settings in jjb https://review.openstack.org/162152 | 10:04 |
| *** zzz_pczesno has joined #openstack-infra | 10:10 | |
| *** wznoinsk_ has joined #openstack-infra | 10:10 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-user: Ensure home and home/.ssh folders are created https://review.openstack.org/165029 | 10:10 |
| *** hichihara has quit IRC | 10:11 | |
| *** alex_xu has quit IRC | 10:12 | |
| *** pczesno has quit IRC | 10:12 | |
| *** Longgeek has quit IRC | 10:13 | |
| *** Longgeek has joined #openstack-infra | 10:13 | |
| *** wznoinsk has quit IRC | 10:13 | |
| *** alex_xu has joined #openstack-infra | 10:14 | |
| *** ihrachyshka has quit IRC | 10:17 | |
| *** mmedvede has quit IRC | 10:18 | |
| *** yamamoto has joined #openstack-infra | 10:18 | |
| *** claudiub has joined #openstack-infra | 10:18 | |
| *** pc_m has joined #openstack-infra | 10:19 | |
| *** e0ne has joined #openstack-infra | 10:23 | |
| *** markvoelker has joined #openstack-infra | 10:24 | |
| *** Hal1 has quit IRC | 10:25 | |
| *** Hal1 has joined #openstack-infra | 10:26 | |
| *** markvoelker has quit IRC | 10:29 | |
| *** rlandy has joined #openstack-infra | 10:29 | |
| *** ihrachyshka has joined #openstack-infra | 10:29 | |
| *** rlucio has joined #openstack-infra | 10:33 | |
| *** yfried is now known as yfried|afk | 10:34 | |
| *** mpavone has joined #openstack-infra | 10:39 | |
| *** ujuc_ has joined #openstack-infra | 10:40 | |
| *** ujuc has quit IRC | 10:40 | |
| *** coolsvap is now known as coolsvap|afk | 10:41 | |
| rlucio | hey guys, are there any other options for hosting ci logs besides rolling your own server? like can you use dropbox or ftp or something? just wondering | 10:46 |
| *** dalgaaf has joined #openstack-infra | 10:49 | |
| *** mmedvede has joined #openstack-infra | 10:51 | |
| *** garyh has joined #openstack-infra | 10:53 | |
| *** ddieterly has joined #openstack-infra | 10:53 | |
| *** yfried|afk is now known as yfried | 10:54 | |
| *** mpaolino has joined #openstack-infra | 10:57 | |
| *** ddieterly has quit IRC | 10:58 | |
| *** mmedvede has quit IRC | 10:58 | |
| *** Hal1 has quit IRC | 10:59 | |
| *** Hal2 has joined #openstack-infra | 10:59 | |
| *** amotoki has joined #openstack-infra | 11:00 | |
| *** mfink_ has quit IRC | 11:00 | |
| *** mpaolino has quit IRC | 11:02 | |
| *** mpaolino has joined #openstack-infra | 11:03 | |
| *** jaypipes has joined #openstack-infra | 11:04 | |
| *** garyh has quit IRC | 11:04 | |
| *** pcaruana has joined #openstack-infra | 11:05 | |
| *** ominakov_ has quit IRC | 11:07 | |
| *** ominakov_ has joined #openstack-infra | 11:07 | |
| *** sambetts has quit IRC | 11:09 | |
| *** mpaolino has quit IRC | 11:11 | |
| *** sambetts has joined #openstack-infra | 11:11 | |
| *** mattfari_ has joined #openstack-infra | 11:11 | |
| *** mattfari_ has quit IRC | 11:12 | |
| *** macjack has joined #openstack-infra | 11:13 | |
| *** otter768 has joined #openstack-infra | 11:16 | |
| *** Somay has joined #openstack-infra | 11:17 | |
| *** otter768 has quit IRC | 11:21 | |
| *** mwagner_lap has quit IRC | 11:21 | |
| wznoinsk_ | rlucio: I haven't heard about any restrictions on where you want to host the artifacts, as long as it follows the requirements from http://ci.openstack.org/third_party.html | 11:24 |
| *** wznoinsk_ is now known as wznoinsk | 11:24 | |
| *** yamamoto has quit IRC | 11:25 | |
| *** markvoelker has joined #openstack-infra | 11:25 | |
| *** Somay has quit IRC | 11:28 | |
| *** markvoelker has quit IRC | 11:30 | |
| *** Somay has joined #openstack-infra | 11:30 | |
| *** deepakcs has quit IRC | 11:30 | |
| *** Hal2 has quit IRC | 11:30 | |
| *** Hal1 has joined #openstack-infra | 11:30 | |
| mordred | morning GheRivero | 11:33 |
| *** coolsvap|afk is now known as coolsvap | 11:36 | |
| openstackgerrit | Adrian Moisey proposed openstack-infra/jenkins-job-builder: Update the nofifications plugin to support 1.9 https://review.openstack.org/165785 | 11:36 |
| sdague | jhesketh: you still up and about? | 11:37 |
| *** e0ne is now known as e0ne_ | 11:37 | |
| sdague | I can no longer run os-loganalyze unit tests locally due to the swift code not failing gracefully | 11:38 |
| openstackgerrit | Adrian Moisey proposed openstack-infra/jenkins-job-builder: Add missing ' https://review.openstack.org/165786 | 11:38 |
| *** e0ne_ has quit IRC | 11:42 | |
| *** mmedvede has joined #openstack-infra | 11:47 | |
| *** macjack has quit IRC | 11:47 | |
| *** e0ne has joined #openstack-infra | 11:48 | |
| *** spredzy is now known as spredzy|afk | 11:49 | |
| *** yamamoto has joined #openstack-infra | 11:50 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/jenkins-job-builder: Stop mkdir output on each job write https://review.openstack.org/165789 | 11:50 |
| sdague | mordred: any idea why the # of usable test nodes seems so low? | 11:52 |
| sdague | looks like we've only got about 150 in flight | 11:52 |
| mordred | sdague: we've been having cloud issues with HP - we may have leaked zombies again ... lemme look | 11:53 |
| *** koolhead17 has joined #openstack-infra | 11:53 | |
| mordred | oh. my | 11:53 |
| mordred | that's ... interesting | 11:53 |
| mordred | novaclient.exceptions.ClientException: An unexpected error prevented the server from fulfilling your request: [Errno 12] Cannot allocate memory (Disable debug mode to suppress these details.) (HTTP 500) | 11:53 |
| *** weshay has joined #openstack-infra | 11:54 | |
| *** markvoelker has joined #openstack-infra | 11:54 | |
| mordred | sdague: is that actually passed through from the server? ^^ ?? | 11:54 |
| *** ddieterly has joined #openstack-infra | 11:54 | |
| sdague | apparently if debug is enabled? | 11:55 |
| sdague | that's interesting | 11:55 |
| *** mmedvede has quit IRC | 11:55 | |
| mordred | nevermind - that's not hpcloud | 11:57 |
| mordred | that's hp1.tripleo | 11:57 |
| *** kgiusti has joined #openstack-infra | 11:58 | |
| *** ddieterly has quit IRC | 11:59 | |
| *** cdent has joined #openstack-infra | 11:59 | |
| sdague | heh | 11:59 |
| *** lucasagomes has joined #openstack-infra | 11:59 | |
| *** aysyd has joined #openstack-infra | 12:00 | |
| *** woodster_ has joined #openstack-infra | 12:00 | |
| lucasagomes | hi, if you have some time please take a look at https://review.openstack.org/#/c/163344/ it's adding a gate check for openstack/ironic-python-agent (non-voting) to test PXE drivers using the IPA ramdisk | 12:00 |
| *** Hal1 has quit IRC | 12:01 | |
| lucasagomes | we want IPA to be our default deploy ramdisk, and it's already being tested on the Ironic gate. So it would be good to also check on the ironic-python-agent gate so we can capture failures quicker | 12:01 |
| *** Hal2 has joined #openstack-infra | 12:01 | |
| *** garyh has joined #openstack-infra | 12:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Don't die while doing alien list https://review.openstack.org/165792 | 12:04 |
| *** claudiub_ has joined #openstack-infra | 12:05 | |
| *** Somay has quit IRC | 12:07 | |
| *** mwagner_lap has joined #openstack-infra | 12:07 | |
| *** claudiub has quit IRC | 12:07 | |
| *** rfolco has joined #openstack-infra | 12:10 | |
| *** ibiris is now known as ibiris_away | 12:13 | |
| *** dprince has joined #openstack-infra | 12:13 | |
| *** chlong has joined #openstack-infra | 12:14 | |
| *** garyh has quit IRC | 12:14 | |
| *** adalbas has joined #openstack-infra | 12:15 | |
| alex_xu | ianw: good morning! I updated this patch https://review.openstack.org/160068 hope you can take a look at, just left few hours for freeze~ And the patch depend on this passed the ci tests https://review.openstack.org/150267 Thanks in advance! | 12:19 |
| *** Longgeek has quit IRC | 12:20 | |
| *** Longgeek has joined #openstack-infra | 12:20 | |
| mordred | sdague: ok - I'm deleting a bunch of alien nodes on hpcloud | 12:20 |
| sdague | mordred: thanks | 12:20 |
| mordred | clarkb, fungi, jeblair: I'm a bit concerned about these aliens - I do not see the pattern clarkb was talking about yesterday where they'd been deleted and then came back | 12:21 |
| mordred | I see no delete calls for them in teh logs at all, in fact - but they sure are not things nodepool groks | 12:21 |
| mordred | sdague: btw - I just learned about block_device_mapping_v2 | 12:22 |
| openstackgerrit | Jaroslav Henner proposed openstack-infra/jenkins-job-builder: Add checking of multijob phase condition. https://review.openstack.org/164669 | 12:23 |
| mordred | sdague: a) dude what a terrible name ... but b) exposes the type of capability I really want in things, so awesome | 12:23 |
| *** ajmiller has quit IRC | 12:23 | |
| *** sdake has joined #openstack-infra | 12:24 | |
| *** bknudson has quit IRC | 12:25 | |
| mordred | sdague: honestly, if there was a similar parameter for floating_ips, like "floating_ip_mapping = [{ delete_on_terminate: True, id = <floating_ip_uuid>, pool = <pool name>}] and if id was left out it would just make one for you - I think I'd have everything I need | 12:25 |
| mordred | sdague: well, except for the ssh hostkey thing - but I'll write that in the next cycle | 12:25 |
| *** sdake_ has joined #openstack-infra | 12:25 | |
| mordred | sdague: do I need to be writing a spec for that for liberty already? or is it still too early (never written a nova spec before) | 12:25 |
| *** chlong has quit IRC | 12:26 | |
| sdague | spec is fine at this point, I think they are effectively already open for liberty | 12:26 |
| *** spredzy|afk is now known as spredzy | 12:29 | |
| openstackgerrit | Sean Dague proposed openstack-infra/os-loganalyze: fix supports_sev matching https://review.openstack.org/165542 | 12:29 |
| openstackgerrit | Sean Dague proposed openstack-infra/os-loganalyze: let tests be run from test file location https://review.openstack.org/165799 | 12:29 |
| *** sdake has quit IRC | 12:29 | |
| sdague | clarkb: I respun your fix with some test cases just to convince myself that \b would work in this case | 12:30 |
| *** BobBall_AWOL is now known as BobBall | 12:31 | |
| openstackgerrit | Adrian Moisey proposed openstack-infra/jenkins-job-builder: Update the nofifications plugin to support 1.9 https://review.openstack.org/165785 | 12:32 |
| *** Hal2 has quit IRC | 12:32 | |
| *** Hal1 has joined #openstack-infra | 12:32 | |
| *** dkranz has joined #openstack-infra | 12:32 | |
| *** e0ne is now known as e0ne_ | 12:33 | |
| *** gordc has joined #openstack-infra | 12:34 | |
| *** ghostpl_ has quit IRC | 12:35 | |
| *** gordc has joined #openstack-infra | 12:37 | |
| *** Longgeek_ has joined #openstack-infra | 12:37 | |
| openstackgerrit | Adrian Moisey proposed openstack-infra/jenkins-job-builder: Update the nofifications plugin to support 1.9 https://review.openstack.org/165785 | 12:39 |
| *** Longgeek has quit IRC | 12:41 | |
| *** mrmartin has joined #openstack-infra | 12:41 | |
| *** achanda has joined #openstack-infra | 12:42 | |
| *** chlong has joined #openstack-infra | 12:43 | |
| *** bswartz has quit IRC | 12:43 | |
| *** unicell1 has joined #openstack-infra | 12:43 | |
| *** e0ne_ has quit IRC | 12:43 | |
| *** unicell has quit IRC | 12:44 | |
| *** bknudson has joined #openstack-infra | 12:45 | |
| *** achanda has quit IRC | 12:47 | |
| *** ujuc_ has quit IRC | 12:47 | |
| *** eharney has quit IRC | 12:53 | |
| *** mfink_ has joined #openstack-infra | 12:53 | |
| *** ddieterly has joined #openstack-infra | 12:55 | |
| *** ddieterly has quit IRC | 12:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Remove HP TripleO region https://review.openstack.org/165814 | 13:00 |
| *** dims has quit IRC | 13:01 | |
| *** sdake_ has quit IRC | 13:01 | |
| *** dims has joined #openstack-infra | 13:02 | |
| *** dboik has joined #openstack-infra | 13:03 | |
| mordred | Ng: any luck with rax glance? | 13:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add 'rebuild' to shade https://review.openstack.org/161067 | 13:05 |
| *** mjturek1 has joined #openstack-infra | 13:06 | |
| *** jistr has quit IRC | 13:08 | |
| *** ldnunes has joined #openstack-infra | 13:08 | |
| *** jistr has joined #openstack-infra | 13:09 | |
| *** sdake has joined #openstack-infra | 13:09 | |
| *** sdake_ has joined #openstack-infra | 13:12 | |
| *** dustins has joined #openstack-infra | 13:12 | |
| *** bswartz has joined #openstack-infra | 13:14 | |
| *** garyh has joined #openstack-infra | 13:15 | |
| *** sdake has quit IRC | 13:16 | |
| *** mrmartin has quit IRC | 13:17 | |
| *** marcusvrn1 has joined #openstack-infra | 13:17 | |
| *** ddieterly has joined #openstack-infra | 13:17 | |
| *** otter768 has joined #openstack-infra | 13:17 | |
| *** marcusvrn has quit IRC | 13:17 | |
| sdague | mordred: any luck in freeing up servers? | 13:18 |
| *** dboik has quit IRC | 13:18 | |
| sdague | looks like changes are waiting about an hour to get nodes in check right now, would be nice to help with that | 13:19 |
| sdague | especially as very few folks are awake atm | 13:19 |
| *** ddieterly has quit IRC | 13:19 | |
| pabelanger | afazekas, rumor has it you've done some work with multi-node jobs with nodepool? I was curious if you didn't mind pointing to some code for that | 13:19 |
| *** ibiris_away is now known as ibiris | 13:19 | |
| *** chlong has quit IRC | 13:21 | |
| *** mrmartin has joined #openstack-infra | 13:21 | |
| *** otter768 has quit IRC | 13:22 | |
| ttx | sdague: some check jobs have been in queue for 13 hours now | 13:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Remove tripleo-test-cloud-hp1 from nodepool config https://review.openstack.org/156615 | 13:23 |
| ttx | and jenkins04 fails to respond | 13:23 |
| sdague | oh, that's a different issue then | 13:23 |
| ttx | we might have a stuck jenkins instance | 13:24 |
| mordred | sdague: I'm not convinced the thing I was looking at was the thing | 13:24 |
| ttx | all stuck jobs seem to be on jenkins04 | 13:24 |
| mordred | yah | 13:24 |
| mordred | jenkins04 is hanging when I try to ssh to it | 13:24 |
| *** sdake has joined #openstack-infra | 13:25 | |
| ttx | yeah, confirmed all jobs >12hours wait are all on jenkins04 | 13:25 |
| ttx | yeah, confirmed all jobs >12hours wait are all on jenkins04 | 13:25 |
| *** garyh has quit IRC | 13:25 | |
| *** ilyashakhat has quit IRC | 13:25 | |
| *** peristeri has joined #openstack-infra | 13:26 | |
| anteaya | stuck ttx? | 13:26 |
| sdague | jenkins crash probably | 13:27 |
| *** e0ne has joined #openstack-infra | 13:27 | |
| * mordred poking it with a stick | 13:27 | |
| ttx | anteaya: some check jobs have been waiting for test results for 14 hours now | 13:28 |
| *** sdake_ has quit IRC | 13:28 | |
| anteaya | ttx: you posted the same thing twice | 13:29 |
| *** ghostpl_ has joined #openstack-infra | 13:29 | |
| anteaya | ttx: my lame attempt at making a funny | 13:29 |
| Ng | mordred: nope. starting to get quite annoyed at it. AFAICS I have the latest python-glanceclient installed fine in a venv, I'm exporting all the environment stuff I can, and still just 401. If you have a working environment, can we figure out what's different about mine from yours? like redact your password from whatever environment stuff you source, and a | 13:29 |
| Ng | pip freeze? | 13:29 |
| anteaya | ttx: and yes looking at the status page I confirm your findings | 13:29 |
| mordred | Ng: yup - did you get my paste yesterday? | 13:29 |
| *** garyh has joined #openstack-infra | 13:29 | |
| devananda | I think https://jenkins04.openstack.org/job/check-tempest-dsvm-ironic-pxe_ipa-nv/40/ is stuck ? | 13:29 |
| Ng | mordred: hrm, I must have missed that | 13:30 |
| mordred | devananda: yes. everything on jenkins04 went south | 13:30 |
| devananda | is there a way that it can be ignored at this point, as a non-voting job | 13:30 |
| devananda | tha tis now blocking a patch | 13:30 |
| mordred | devananda: not really - sorry - it's going to have to get rekicked | 13:30 |
| devananda | mordred: ah. lovely | 13:30 |
| * mordred just hard-rebooted jenkins04 | 13:30 | |
| devananda | gotcha | 13:30 |
| mordred | there will be a bit of fallout from that | 13:30 |
| mordred | Ng: you're probably missing export OS_IMAGE_API_VERSION=2 | 13:30 |
| Ng | mordred: I definitely have that exported | 13:31 |
| devananda | mordred: so, how can I initiate a recheck, when zuul hasn't finished the last job run? | 13:31 |
| mordred | ok. let me give you redacted | 13:31 |
| Ng | ta | 13:31 |
| *** baoli has joined #openstack-infra | 13:32 | |
| devananda | or will zuul figure that out in a minute on its own? | 13:32 |
| Ng | -(cmsj@stonker)-(~)- glance --version | 13:32 |
| Ng | 0.16.1 | 13:32 |
| Ng | -(cmsj@stonker)-(~)- echo $OS_IMAGE_API_VERSION | 13:32 |
| Ng | 2 | 13:32 |
| mordred | devananda: well, you could do something silly like change a punctuation in the commit message ttw | 13:32 |
| devananda | gah. right. | 13:32 |
| mordred | Ng: OS_TENANT_NAME ? | 13:32 |
| * devananda forgets these silly tricks sometimes | 13:32 | |
| Ng | mordred: I have that set to my account number | 13:33 |
| mordred | great. like 6 digits or whatever? | 13:33 |
| Ng | yeah | 13:33 |
| mordred | Ng: http://paste.openstack.org/show/193563/ | 13:34 |
| *** Longgeek_ has quit IRC | 13:35 | |
| mordred | Ng: and 'glance image-list' works fine for me with those on glanceclient 0.16.1 | 13:36 |
| mordred | Ng: if you do 'keystone catalog' - do you get an entry for Service: image ? | 13:36 |
| mordred | ttx: we're definitely feature freeze week, aren't we? :) | 13:37 |
| ttx | mordred: even FF day. The day I need that queue thing to work :) | 13:38 |
| Ng | mordred: nup, keystone is 401ing all over town too. Comparing to my env, I'm on a different region (their UK one). I'm wondering if they're not actually all the same | 13:38 |
| Ng | so far the only api client I've managed to get working with it, is their hacked up version of novaclient | 13:39 |
| Ng | (rackspace-novaclient in pypi) | 13:39 |
| *** dboik has joined #openstack-infra | 13:40 | |
| mordred | oh gross. do you have access to the non-UK regions? like, can you try doing against DFW? | 13:40 |
| mordred | ttx: the day you need it to work is the day it never will | 13:40 |
| *** nelsnelson has joined #openstack-infra | 13:42 | |
| Ng | mordred: same. I wonder if the reverse would work - if you set OS_REGION_NAME=LON and OS_AUTH_URL=https://lon.identity.api.rackspacecloud.com/v2.0/ | 13:43 |
| *** sarob has joined #openstack-infra | 13:43 | |
| mordred | let me try | 13:43 |
| *** mrmartin has quit IRC | 13:44 | |
| Ng | the web console seems to suggest taht accounts are region-locked, in that I can't do stuff in DFW via that, it tells me to create a new account | 13:44 |
| mordred | Ng: HEADDESK | 13:44 |
| Ng | I'm gonna go back to the support people and say I know for a fact this works against DFW, and if this somehow isn't possible against LON, can I have my account switched over/recreated | 13:45 |
| mordred | public endpoint for image service in LON region not found | 13:45 |
| mordred | is what I get when I try london | 13:45 |
| *** tnovacik has quit IRC | 13:46 | |
| mordred | jroll, JayF: you both have magical rackspace powers, right? pvo isn't ever on IRC anymore, so I can't poke him ... | 13:46 |
| *** ddieterly has joined #openstack-infra | 13:46 | |
| *** mtanino has joined #openstack-infra | 13:47 | |
| *** dkliban_afk is now known as dkliban | 13:47 | |
| anteaya | mordred: what is the status on jenkins04? | 13:48 |
| *** sarob has quit IRC | 13:48 | |
| anteaya | since you rebooted | 13:48 |
| anteaya | and it looks to me like we might need to clean up more alien nodes like clarkb did last night | 13:49 |
| ttx | zuul still shows them as stuck | 13:49 |
| anteaya | mordred: do you know how to do that? | 13:49 |
| * mordred is cleaning up alien nodes | 13:49 | |
| anteaya | thank you, that will help | 13:50 |
| mordred | however, there may be additional cleanup tasks that jeblair and clarkb suggest | 13:50 |
| anteaya | doing what you can is appreciated | 13:50 |
| anteaya | it being feature freeze day and all | 13:51 |
| mordred | yah | 13:51 |
| anteaya | thanks | 13:51 |
| *** amitgandhinz has joined #openstack-infra | 13:52 | |
| *** eharney has joined #openstack-infra | 13:53 | |
| *** larsks has joined #openstack-infra | 13:53 | |
| *** mfink__ has joined #openstack-infra | 13:53 | |
| anteaya | the number of zuul workers looks funny to me | 13:54 |
| anteaya | in that it looks like there aren't many | 13:54 |
| *** mfink_ has quit IRC | 13:54 | |
| fungi | skimmed scrollback. is jenkins04 still in need of help? | 13:55 |
| anteaya | or perhaps those are available workers, so the number is low if they aren't available | 13:55 |
| anteaya | fungi: I'm uncertain | 13:55 |
| fungi | ahh, looks like it got restarted but its slave list was not evacuated | 13:55 |
| *** ddieterly has quit IRC | 13:56 | |
| fungi | so it has many hundreds of offline slaves according to its webui | 13:56 |
| anteaya | awesome | 13:56 |
| jroll | mordred: what's up | 13:56 |
| fungi | okay, maybe not many hundreds, but a few hundred | 13:56 |
| *** mriedem_away is now known as mriedem | 13:57 | |
| *** mfink_ has joined #openstack-infra | 13:57 | |
| jroll | mordred: the only special rackspace power I have is internal irc, but if you have a paste or whatever with some details I can link it to people | 13:57 |
| *** ddieterly has joined #openstack-infra | 13:58 | |
| *** aysyd has quit IRC | 13:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record interesting info into nova metadata https://review.openstack.org/126621 | 13:58 |
| *** wuhg has joined #openstack-infra | 13:59 | |
| jroll | mordred: reading again this looks like an incoming rant about uk vs us accounts... I can't help with that but it is being worked on(tm) | 13:59 |
| mordred | fungi: yes - jenkins04 was completely dead | 13:59 |
| *** aysyd has joined #openstack-infra | 13:59 | |
| mordred | jroll: awesome - so, is the best bet just for Ng to poke support and get his account move - or to get a new account created? | 13:59 |
| *** koolhead17 has quit IRC | 13:59 | |
| *** spzala has joined #openstack-infra | 13:59 | |
| anteaya | mordred: so fungi appears to be suggesting that evacuating a jenkins slave list prior to a reboot might be something to consider for next time | 14:00 |
| mordred | jroll: it seems that the UK region has different operational semantics too - as in python-glanceclient does not work there | 14:00 |
| mordred | anteaya: sure - but the server was completely hung and it was not possible to connect to it | 14:00 |
| fungi | any more i've taken to killing the jenkins process, deleting all the lines inside <slaves></slaves> from the config.xml, marking everything in the nodepool database assigned to that jenkins as delete, then starting the jenkins process again | 14:00 |
| fungi | annoying but effective | 14:00 |
| mordred | fungi: it was not possible to ssh in to the box | 14:00 |
| *** mfink__ has quit IRC | 14:00 | |
| fungi | oh marvellous | 14:00 |
| mordred | yeah. | 14:00 |
| mordred | this is why I hard rebooted it | 14:00 |
| jroll | mordred: yeah, it's a huge problem. I hate it. | 14:00 |
| mordred | it was stuck very badly | 14:00 |
| fungi | anything interesting on the virtual console, or did you have a chance to check? | 14:00 |
| mordred | fungi: the virtual console also did not work :) | 14:00 |
| jroll | mordred: Ng: I'm not sure if poke support or make a new account is the best route, support can tell you :P | 14:01 |
| fungi | wow awesome | 14:01 |
| mordred | fungi: yah. it was pleasing | 14:01 |
| Ng | jroll: thanks. I shall see what my options are with support :) | 14:01 |
| *** asselin_ has joined #openstack-infra | 14:01 | |
| Ng | hopefully all of my options involve fanaticism of some kind | 14:01 |
| *** zz_dimtruck is now known as dimtruck | 14:01 | |
| *** yfried is now known as yfried|afk | 14:01 | |
| fungi | i'm going what i can to get as much cleaned up for nodepool as possible, and giving the other jenkins masters a health check | 14:02 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:02 | |
| anteaya | fungi: cool, thanks | 14:02 |
| jroll | Ng: IMO this separate account thin is strictly non-fanatical :) | 14:02 |
| fungi | jroll: Ng: fantastical! | 14:02 |
| *** dkranz has quit IRC | 14:03 | |
| jroll | heh | 14:03 |
| fungi | jenkins01 is pretty annoyed too | 14:04 |
| anteaya | :( | 14:04 |
| fungi | but not dead yet | 14:04 |
| anteaya | yay for life | 14:04 |
| fungi | i'll get it into a graceful restart cycle as soon as possible | 14:04 |
| fungi | and clean it up while it's restarted for good measure | 14:05 |
| mordred | fungi: I've got a loop going deleting things from alien-list btw | 14:05 |
| * anteaya nods | 14:05 | |
| fungi | mordred: awesome. i hadn't gotten that far yet, cleaning up stale tracked nodes in nodepool first | 14:05 |
| *** mfink__ has joined #openstack-infra | 14:05 | |
| openstackgerrit | Merged openstack-infra/system-config: Work-around grubby boot failure on Fedora 20 https://review.openstack.org/165681 | 14:06 |
| fungi | since there seemed to be rather a few in ready for far longer than is likely, and no sign of them on the jenkins masters to which they claimed to belong | 14:06 |
| fungi | that has a tendency to throw off the demand calculations pretty badly | 14:06 |
| fungi | oh yeah jenkins01 was not much longer for this life. took several minutes just to get logged into the webui | 14:06 |
| pabelanger | Was there a mid-cycle infra meeting this time around? If so, where could I look to locate the discussions? I don't see much on the infra wiki page | 14:07 |
| *** achuprin has joined #openstack-infra | 14:07 | |
| anteaya | fungi: glad you caught it | 14:07 |
| fungi | pabelanger: we don't usually have mid-cycle meetups | 14:07 |
| fungi | pabelanger: we prefer to do virtual sprints and similar sorts of activities where more people can participate without needing to travel | 14:07 |
| pabelanger | fungi, Oh, perhaps I'm thinking of informal meetings. Didn't something happen in NYC and Australia? | 14:08 |
| anteaya | nyc was in 2013 in june, we had an onboarding | 14:09 |
| *** yfried|afk is now known as yfried | 14:09 | |
| anteaya | australia was lca and we saw each other there | 14:09 |
| pabelanger | Makes sense | 14:09 |
| mordred | pabelanger: yeah - no mid-cycle this time | 14:09 |
| anteaya | the only thing that took place there that had an etherpad was I locked markmcclain and gus in a room to come up with a direction for nova-net to neutron migration | 14:09 |
| fungi | yeah, it's more that we have opportunistic gatherings when enough people already expect to be around | 14:09 |
| *** radez_g0n3 is now known as radez | 14:09 | |
| anteaya | yes that | 14:09 |
| fungi | the other jenkins masters look like they're doing okay for now. once jenkins01 is back to normal i'll gracefully restart 04 so that i can delete its pile of cruft slaves | 14:11 |
| *** mattfarina has joined #openstack-infra | 14:12 | |
| *** claudiub has joined #openstack-infra | 14:12 | |
| *** mfink__ has quit IRC | 14:12 | |
| *** mfink_ has quit IRC | 14:12 | |
| *** mfink_ has joined #openstack-infra | 14:13 | |
| *** claudiub_ has quit IRC | 14:14 | |
| *** ajmiller has joined #openstack-infra | 14:14 | |
| *** stevemar has joined #openstack-infra | 14:14 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config: Adds compute-hyperv in StackForge https://review.openstack.org/165611 | 14:14 |
| fungi | mordred: wow. we have 555 unassociated floating ips in hpcloud. cleaning those up now | 14:15 |
| fungi | though these may be from churn... some of them are already coming up not found when deleting | 14:17 |
| devananda | is jenkins06 having trouble now? | 14:17 |
| fungi | devananda: i don't think so--seems to be running jobs | 14:18 |
| fungi | devananda: what makes you ask? | 14:18 |
| * devananda is watching https://jenkins06.openstack.org/job/check-heat-dsvm-functional-mysql/662/ | 14:19 | |
| devananda | perhaps just slow ... | 14:19 |
| *** dhellmann has quit IRC | 14:19 | |
| *** dhellmann has joined #openstack-infra | 14:19 | |
| devananda | ah, yep. it's alive still | 14:20 |
| fungi | mostly dead is still partly alive! | 14:20 |
| devananda | indeed! | 14:20 |
| devananda | (and it passed. yay!) | 14:20 |
| *** timcline has joined #openstack-infra | 14:21 | |
| fungi | i'm rerunning my fip cruft check with a 5-minute separation instead of 1 | 14:21 |
| fungi | hopefully this will be marginally more accurate | 14:21 |
| *** e0ne is now known as e0ne_ | 14:22 | |
| *** ayoung has quit IRC | 14:22 | |
| *** e0ne_ is now known as e0ne | 14:23 | |
| *** _nadya_ has joined #openstack-infra | 14:24 | |
| *** mrmartin has joined #openstack-infra | 14:25 | |
| *** erlon has joined #openstack-infra | 14:26 | |
| *** yamamoto has quit IRC | 14:27 | |
| openstackgerrit | Merged openstack-infra/system-config: Remove tripleo-test-cloud-hp1 from nodepool config https://review.openstack.org/156615 | 14:28 |
| *** sushilkm has joined #openstack-infra | 14:28 | |
| fungi | mordred: something else is wrong with floating ips in hpcloud | 14:29 |
| *** jcoufal_ has joined #openstack-infra | 14:29 | |
| sushilkm | The gating jobs for trove seem to stuck | 14:29 |
| sushilkm | https://jenkins04.openstack.org/job/check-grenade-dsvm/28125/console | 14:29 |
| sushilkm | I think it needs to be restarted | 14:29 |
| sushilkm | Or some fix | 14:29 |
| mordred | fungi: sweet! | 14:29 |
| fungi | mordred: `nova floating-ip-delete 01178e82-74c6-4da4-8a60-3315a3a4be7c` returns "ERROR (CommandError): Floating IP 01178e82-74c6-4da4-8a60-3315a3a4be7c not found." but `nova floating-ip-list` says it exists | 14:30 |
| openstackgerrit | Sean Dague proposed openstack-infra/os-loganalyze: extract static methods https://review.openstack.org/165850 | 14:30 |
| openstackgerrit | Sean Dague proposed openstack-infra/os-loganalyze: unwind test class multiple inheritance https://review.openstack.org/165851 | 14:30 |
| fungi | mordred: nevermind. looks like nova can't delete floating ips by uuid any longer, only by ip address | 14:30 |
| anteaya | sushilkm: jenkins04 was stuck and has been rebooted | 14:32 |
| mordred | fungi: neutron can delete them by uuid | 14:32 |
| *** sdake_ has joined #openstack-infra | 14:32 | |
| *** sdake_ has quit IRC | 14:32 | |
| mordred | fungi: and it's more efficient to delete them with neutron, fwiw | 14:32 |
| *** jcoufal has quit IRC | 14:32 | |
| anteaya | sushilkm: when jenkins comments on your patch you may need to recheck to send it through again | 14:32 |
| fungi | sushilkm: yes, i think we're probably going to have to restart zuul to clear those up. it doesn't seem to know that the jobs which were running on jenkins04 before it died are no longer running | 14:32 |
| *** sdake__ has joined #openstack-infra | 14:33 | |
| sushilkm | Yes its stuck since 13 hrs | 14:33 |
| *** yamamoto has joined #openstack-infra | 14:33 | |
| *** sdake has quit IRC | 14:34 | |
| sdague | fungi: or push a new patch | 14:34 |
| sdague | sushilkm: edit the commit message to kick it back into the pool | 14:35 |
| fungi | sdague: yeah, i was just considering that. there's only 13 in that state | 14:35 |
| sdague | because they'll need to rerun for new tests anyway | 14:35 |
| fungi | so maybe better than restarting the whole world | 14:35 |
| sdague | yeh | 14:35 |
| *** achanda has joined #openstack-infra | 14:35 | |
| anteaya | yes | 14:35 |
| anteaya | fungi: do you have a list of the 13? I can edit and get them back in | 14:35 |
| sushilkm | Wud the recheck not work | 14:35 |
| fungi | anteaya: i can see them in the top-left corner of the zuul status page | 14:36 |
| sdague | it will not | 14:36 |
| fungi | anteaya: the 13 in the check pipeline with times of 13+ hours in queue | 14:36 |
| anteaya | fungi: ah okay I will get them | 14:36 |
| fungi | i mean, they're not doing any harm being there | 14:37 |
| anteaya | sushilkm: recheck only works if jenkins is finished and has reported back | 14:37 |
| anteaya | sushilkm: it won't work if they are stuck in the queue | 14:37 |
| fungi | so if people who own those 13 changes care they're also welcome to just push a new patchset, tweak the commit message in the webui, whatever | 14:37 |
| anteaya | oh okay | 14:37 |
| fungi | it's an independent pipeline, so it's not tying up any other changes | 14:37 |
| anteaya | was just going to be neighbourly | 14:37 |
| anteaya | but I don't have to | 14:37 |
| fungi | i'm currently more concerned about getting our worker capacity back to sanity | 14:38 |
| sdague | it might be good to wait on flushing them all until we have more ready nodes | 14:38 |
| sdague | fungi: ++ | 14:38 |
| mordred | anteaya: when I want to be neighborly, I tend to give people fluffy large animals | 14:38 |
| *** wenlock has joined #openstack-infra | 14:38 | |
| sdague | we're at < 100 in use nodes | 14:38 |
| mordred | anteaya: although I've been considering adding sea mammals to the mix recently | 14:38 |
| anteaya | mordred: I see that | 14:38 |
| sdague | which is quite bad for throughput | 14:38 |
| anteaya | mordred: oh like what? | 14:39 |
| fungi | you're not kidding | 14:39 |
| mordred | anteaya: your approach is likely more productive than mine | 14:39 |
| mordred | anteaya: I mean, dolphins are a good start - but manatees could also be fun | 14:39 |
| anteaya | mordred: yours makes more friends though | 14:39 |
| anteaya | mordred: my approach doesn' tend to make friends | 14:39 |
| *** tonytan4ever has joined #openstack-infra | 14:40 | |
| mordred | anteaya: maybe you should try distributing mooses? | 14:40 |
| anteaya | not sure that would achieve the goal | 14:40 |
| mordred | anteaya: probably not | 14:40 |
| anteaya | meese tend to take up quite a bit of space | 14:41 |
| anteaya | got chased by a moose once, that was exciting | 14:41 |
| mordred | anteaya: but they also tend to keep home invaders at bay | 14:41 |
| *** sushilkm has left #openstack-infra | 14:41 | |
| sdague | ah, the openssl vuln is just a DoS attack, not an exploit - https://www.openssl.org/news/vulnerabilities.html | 14:41 |
| anteaya | they consider anyone not them to be a home invader | 14:42 |
| sdague | hopefully that means no need to take down services for patching | 14:42 |
| *** baoli has quit IRC | 14:42 | |
| mtreinish | sdague: oh good, I wasn't really looking forward to having to do that again | 14:42 |
| sdague | oh, wait, there are a ton of CVEs in that list today | 14:42 |
| *** spzala has quit IRC | 14:43 | |
| *** baoli has joined #openstack-infra | 14:43 | |
| *** mfink_ has quit IRC | 14:43 | |
| *** achanda has quit IRC | 14:43 | |
| *** mfink_ has joined #openstack-infra | 14:43 | |
| sdague | there are 12 CVEs today actually | 14:43 |
| anteaya | :( | 14:43 |
| sdague | it looks like they are all DoS or segfault | 14:44 |
| *** sarob has joined #openstack-infra | 14:44 | |
| fungi | yeah, it's a fun batch | 14:45 |
| fungi | i'm still mulling it over | 14:45 |
| mordred | anteaya: yes - I mean, they also keep you away from your home - but your home will certainly be safe from other people if you're worried about that sort of thing | 14:45 |
| anteaya | right | 14:45 |
| *** baoli has quit IRC | 14:45 | |
| anteaya | if my home happens to be a muskeg swamp | 14:46 |
| anteaya | it will be nice and safe | 14:46 |
| mordred | anteaya: I don't want to judge people's home choices | 14:46 |
| anteaya | I did live close for a period of time | 14:46 |
| anteaya | was nice | 14:46 |
| anteaya | hence the experience being chased | 14:46 |
| *** baoli has joined #openstack-infra | 14:47 | |
| *** dkranz has joined #openstack-infra | 14:47 | |
| *** yfried is now known as yfried|afk | 14:48 | |
| *** timcline has quit IRC | 14:48 | |
| jeblair | fungi: only 5 leaked ports | 14:48 |
| anteaya | morning jeblair | 14:48 |
| *** mrmartin has quit IRC | 14:48 | |
| *** timcline has joined #openstack-infra | 14:49 | |
| *** sarob has quit IRC | 14:49 | |
| jeblair | anteaya: morning | 14:49 |
| jeblair | fungi: are you working on jenkins01? | 14:49 |
| *** prad has joined #openstack-infra | 14:49 | |
| *** flip214 has left #openstack-infra | 14:50 | |
| jeblair | ah yeh, it's down to 2 jobs | 14:50 |
| *** MarkAtwood has joined #openstack-infra | 14:50 | |
| *** bhunter71 has joined #openstack-infra | 14:51 | |
| *** jlanoux has quit IRC | 14:51 | |
| *** garyh has quit IRC | 14:51 | |
| *** mrunge has quit IRC | 14:52 | |
| *** yfried|afk is now known as yfried | 14:52 | |
| *** carl_baldwin has joined #openstack-infra | 14:54 | |
| fungi | jeblair: yeah waiting for jobs to wrap up there | 14:54 |
| jeblair | fungi: most of the errors i'm seeing are timeouts on server deletion; i'm not actually seeing any build errors | 14:55 |
| fungi | two more running, almost complete | 14:55 |
| jeblair | i've double checked the ssh timeouts i see, and those are actually just nodes that were in the process of being deleted but got picked up by the periodic check | 14:55 |
| fungi | interestingly, we have a crazy high number of nodes in a building state proportional to everything else | 14:56 |
| *** marcusvrn has joined #openstack-infra | 14:56 | |
| *** sarob has joined #openstack-infra | 14:56 | |
| openstackgerrit | Anthony Michon proposed openstack-infra/project-config: Add cerberus related projects https://review.openstack.org/163876 | 14:57 |
| jeblair | we also have quite a few ready nodes on jenkins01 and 04. i suspect those are errors. i will delete them | 14:57 |
| *** timcline has quit IRC | 14:57 | |
| *** timcline has joined #openstack-infra | 14:57 | |
| fungi | sounds like a good idea. i'm also planning to blow away everything assigned to jenkins01 here momentarily, as soon as it's down | 14:58 |
| fungi | and then circle back around to do the same for jenkins04 | 14:58 |
| fungi | we have 82 nodes in a building state for more than an hour, all in hpcloud | 14:59 |
| *** armax has joined #openstack-infra | 14:59 | |
| fungi | 320 for more than 30 minutes, also all in hpcloud | 15:00 |
| anteaya | :( | 15:00 |
| fungi | something's gumming up the works there | 15:00 |
| anteaya | same region? | 15:00 |
| fungi | we only use one region, but i'll correlate to az | 15:00 |
| anteaya | oh | 15:00 |
| anteaya | sorry | 15:01 |
| *** dboik_ has joined #openstack-infra | 15:01 | |
| mordred | anteaya: no need to be sorry - it was the right question | 15:01 |
| *** mrunge has joined #openstack-infra | 15:01 | |
| anteaya | k | 15:01 |
| fungi | 78% are in az3 | 15:01 |
| anteaya | wow | 15:01 |
| fungi | mordred: jeblair: ^ | 15:01 |
| mordred | fungi: excellent | 15:02 |
| jeblair | fungi: we target 63% of our nodes to be in az3 | 15:02 |
| jeblair | fungi: so that may not be off-mark | 15:02 |
| *** yfried is now known as yfried|afk | 15:02 | |
| fungi | yeah, so maybe slightly inflated ratio but not crazy then | 15:03 |
| fungi | so may just be roughly evenly distributed by assigned quota | 15:03 |
| *** garyh has joined #openstack-infra | 15:03 | |
| *** dboik has quit IRC | 15:04 | |
| *** pblaho_ has joined #openstack-infra | 15:04 | |
| *** asselin_ has quit IRC | 15:05 | |
| *** pblaho has quit IRC | 15:05 | |
| *** dkranz has quit IRC | 15:05 | |
| *** mrunge has quit IRC | 15:06 | |
| *** sputnik13 has quit IRC | 15:07 | |
| *** sdake has joined #openstack-infra | 15:07 | |
| *** timcline has quit IRC | 15:08 | |
| *** sputnik13 has joined #openstack-infra | 15:09 | |
| *** timcline has joined #openstack-infra | 15:09 | |
| *** sputnik13 has quit IRC | 15:09 | |
| *** wayneeseguin has quit IRC | 15:10 | |
| sdague | anteaya: what's the policy on things like this random 'noop' comment - https://review.openstack.org/#/c/165863/1 | 15:11 |
| sdague | looks like a poorly setup test system | 15:11 |
| *** spzala has joined #openstack-infra | 15:11 | |
| *** wayneeseguin has joined #openstack-infra | 15:11 | |
| *** sdake__ has quit IRC | 15:11 | |
| anteaya | sdague: policy is they get disabled | 15:12 |
| fungi | okay, jenkins01 has finally quiesced so i'm stopping it and cleaning up all its workers now | 15:12 |
| sdague | anteaya: could you do that please? thanks in advance | 15:12 |
| anteaya | sdague: dims wanted a chance to reach out to them, unless you would like them disabled now | 15:12 |
| anteaya | sdague: sure | 15:12 |
| anteaya | can someone please disable gerrit account id 13431: http://paste.openstack.org/show/193589/ https://review.openstack.org/#/c/165863/1 personal account posting ci comments | 15:15 |
| *** mika has quit IRC | 15:15 | |
| jeblair | anteaya: done | 15:16 |
| anteaya | jeblair: thank you | 15:16 |
| dims | anteaya: sdague: just emailed them | 15:16 |
| anteaya | I'll tell them they are disabled | 15:16 |
| dims | anteaya: ack thanks | 15:16 |
| *** dkranz has joined #openstack-infra | 15:17 | |
| *** otter768 has joined #openstack-infra | 15:18 | |
| jd__ | can I haz an ETA on https://review.openstack.org/#/c/162146/ because it's kinda "blocking" us to release? | 15:18 |
| anteaya | dims: keep an eye on them for me will you? we might have to clear cache if they have an open connection to prevent them from continuing to post comments | 15:19 |
| *** hashar has quit IRC | 15:19 | |
| anteaya | jd__: we are dealing with multiple challenges today | 15:20 |
| anteaya | we can't give ETA's | 15:20 |
| *** masayukig_ has joined #openstack-infra | 15:20 | |
| fungi | jenkins01 is on its way back up now. all nodepool nodes for it have been marked delete and all slave entries have been deleted from its config | 15:20 |
| anteaya | we are however doing everything we can to help developers and release mangers achieve their goals | 15:20 |
| anteaya | we have a shortage of workers to run tests and are doing what we can to make more available | 15:21 |
| anteaya | fungi: yay | 15:21 |
| *** dannywilson has joined #openstack-infra | 15:21 | |
| clarkb | renaming a project requires a scheduled gerriy downtime | 15:21 |
| clarkb | jd__ why is that blocking a release? cant you release gnocchi from the stackforge repo? | 15:22 |
| *** Hal1 has quit IRC | 15:22 | |
| *** mika has joined #openstack-infra | 15:22 | |
| *** otter768 has quit IRC | 15:23 | |
| *** yfried|afk is now known as yfried | 15:23 | |
| *** masayukig_ has quit IRC | 15:23 | |
| sdague | clarkb: https://review.openstack.org/#/c/165542/ - I redid your fix of my fix with a test to convince myself that your fix was the right one | 15:23 |
| fungi | jd__: best guess on the project renames is that we'll batch them up a week from saturday, since we don't want to add potential confusion to this saturday's gerrit os upgrade/ip address change | 15:24 |
| clarkb | sdague thanks | 15:24 |
| sdague | there is also a stack of test refactorings above that which are not urgent, but while I was in the code I did it anyway. | 15:24 |
| fungi | jd__: but we'll probably discuss the plan in greater detail in tuesday's meeting | 15:24 |
| *** ChuckC has joined #openstack-infra | 15:24 | |
| *** dannywil_ has joined #openstack-infra | 15:24 | |
| mordred | fungi: my loop of cleaning up all of the aliens has finished, btw | 15:25 |
| fungi | mordred: awesome. stale floating ip deletion is still going, though jenkins01 is starting to get workers again | 15:25 |
| jeblair | fungi, mordred: i'm seeing a lot of 2015-03-19 15:24:18,583 ERROR nodepool.NodeLauncher: EOFError launching node id: 1365036 in provider: hpcloud-b3 error: | 15:25 |
| jeblair | corvus@fuligin:~$ ssh 15.126.138.93 | 15:25 |
| jeblair | Connection closed by 15.126.138.93 | 15:25 |
| fungi | here in a moment i'll set jenkins04 to prepare for shutdown in preparation to clean it up | 15:25 |
| *** dannywilson has quit IRC | 15:26 | |
| jd__ | clarkb: I can release but I wanted to test things like PyPI etc at the same time as it's our first release; so it's not "blocking" (note the "") it's just less fun ;) | 15:26 |
| mordred | so - I've got a bunch of things in alien list again | 15:26 |
| jd__ | fungi: ok, thanks :) | 15:26 |
| mordred | I think something is going south there | 15:26 |
| clarkb | 04 appears to have leaked threads according to melody so the ssh-slaves plugin update does not correct that behavior :/ | 15:26 |
| fungi | jeblair: great. that could be anything from broken filesystem to sshd child processes getting killed to network issues | 15:26 |
| jeblair | mordred: make sure you take 2 snapshots and compare | 15:27 |
| jeblair | mordred: new nodes briefly show up in alien lists during builds | 15:27 |
| mordred | gotcha | 15:27 |
| mordred | jeblair: Conflict: Instance 2a478e27-ab9e-48c7-8d14-14190a95b9ed is locked (HTTP 409) (Request-ID: req-e0f296ad-4306-4e6c-a3ba-7df1dd87cc80) | 15:27 |
| mordred | what does that mean? | 15:27 |
| clarkb | https://jenkins04.openstack.org/monitoring?part=graph&graph=threadCount | 15:27 |
| jeblair | mordred: no idea; last time i asked the noc about it i didn't get a response | 15:27 |
| clarkb | mordred its nova locked | 15:28 |
| *** masayukig_ has joined #openstack-infra | 15:28 | |
| clarkb | but nova unlock doesnt work on it | 15:28 |
| anteaya | clarkb: :/ for threads | 15:28 |
| jeblair | what does "nova locked" mean? | 15:28 |
| jeblair | and how does something become "nova locked"? | 15:28 |
| *** AJaeger has joined #openstack-infra | 15:28 | |
| clarkb | jeblair `nova lock` is the command | 15:28 |
| mordred | clarkb: but we don't run that anywhere, do we? | 15:28 |
| AJaeger | fungi, could you review jeblair's irrc check access patch again, please? I rebased it... https://review.openstack.org/#/c/165508/ | 15:29 |
| clarkb | the help text is minimal and I think it says "lock an instance" so the actual semantics are fuzzy there | 15:29 |
| clarkb | mordred no | 15:29 |
| *** masayukig has quit IRC | 15:29 | |
| *** jlanoux has joined #openstack-infra | 15:29 | |
| jeblair | yeah, so i'm no closer to knowing what that means | 15:29 |
| mordred | clarkb, jeblair: I'm asking the noc about 2a478e27-ab9e-48c7-8d14-14190a95b9ed | 15:29 |
| mordred | which our logs show us as having requested deletion for but is in active state | 15:30 |
| *** sdake_ has joined #openstack-infra | 15:30 | |
| *** Hal1 has joined #openstack-infra | 15:30 | |
| AJaeger | fungi, thanks! | 15:31 |
| *** masayukig_ has quit IRC | 15:31 | |
| *** pblaho_ has quit IRC | 15:33 | |
| *** pblaho_ has joined #openstack-infra | 15:33 | |
| dims | anteaya: ack | 15:34 |
| *** sdake has quit IRC | 15:34 | |
| *** masayukig has joined #openstack-infra | 15:34 | |
| jeblair | fungi: i have found a console log difference between a host with EOFError and one without | 15:34 |
| fungi | oh! i've not had tons of luck with nova console yet | 15:35 |
| jeblair | fungi: http://paste.openstack.org/show/193593/ | 15:35 |
| jeblair | fungi: working node on top, failing node on bottom | 15:36 |
| fungi | jeblair: nice! theory is lack of entropy at boot, no hostkey generated | 15:36 |
| *** ayoung has joined #openstack-infra | 15:36 | |
| fungi | also why is /etc/sysconfig/network-scripts/ifcfg-* missing? | 15:37 |
| fungi | oh, it does that on both | 15:37 |
| *** gyee has joined #openstack-infra | 15:37 | |
| fungi | right, not red hat | 15:37 |
| *** reed has joined #openstack-infra | 15:37 | |
| AJaeger | jeblair, fungi, anteaya: My understanding of a discussion here on IRC some days ago (don't remember when exactly) is that we prefer if projects have a single lint target. In that spirit, I merged for project-config the two targets bashate and pep8 in patch https://review.openstack.org/164531 - and confused anteaya with it. Could you confirm whether this is the right direction, please? | 15:37 |
| jeblair | AJaeger: i'll have to look into that later | 15:38 |
| mordred | jeblair: you know how to open hp support tickets, yeah? | 15:38 |
| jeblair | mordred: through the web ui? | 15:38 |
| mordred | k | 15:39 |
| AJaeger | jeblair, ok, thanks. There's no urgency - just like to get direction | 15:39 |
| jeblair | mordred: and then go check infra-root's mailbox because they send the response there and forgot to copy-paste it into the support ticket tracker? | 15:39 |
| *** Bsony has quit IRC | 15:39 | |
| anteaya | AJaeger: yeah we are trying to get more workers available for jobs right now | 15:39 |
| clarkb | I prefered to email for that reason | 15:40 |
| jeblair | at least, that's the most recent procedure | 15:40 |
| clarkb | support@hpcloud.com iirc | 15:40 |
| openstackgerrit | Merged openstack-infra/project-config: Fix failure detection in irc access check https://review.openstack.org/165508 | 15:40 |
| anteaya | AJaeger: perhaps we can discuss it tomorrow and thanks for bring it up | 15:40 |
| AJaeger | anteaya, feel free to discuss when you have time and give me a pointer if I'm not around ;) I can read the logs on eavesdrop then | 15:40 |
| anteaya | you can bring it up, it is your issue, just another time is better | 15:41 |
| *** unicell1 has quit IRC | 15:41 | |
| *** unicell has joined #openstack-infra | 15:42 | |
| jeblair | fungi: i've confirmed the eoferror and working nodes i looked at were built from the same image in the same region, and within less than 1 second of each other | 15:43 |
| fungi | jeblair: was there any console output earlier during boot about host key generation breaking? | 15:43 |
| *** Trozz has quit IRC | 15:46 | |
| *** bhuvan has joined #openstack-infra | 15:46 | |
| *** Trozz has joined #openstack-infra | 15:47 | |
| jeblair | fungi: it's missing more things like Your identification has been saved in /etc/ssh/ssh_host_rsa_key. | 15:47 |
| jeblair | Generating public/private rsa key pair. | 15:47 |
| jeblair | fungi: check ~corvus/okaylog and ~corvus/eoflog on puppetmaster | 15:47 |
| *** dangers_away is now known as dangers | 15:47 | |
| fungi | jeblair: awesome, thanks | 15:47 |
| fungi | i wonder if it's a filesystem problem | 15:47 |
| jeblair | fungi: 2015-03-19 14:57:14,832 - util.py[WARNING]: Failed fetching metadata from url http://169.254.169.254/2009-04-04/meta-data | 15:48 |
| jeblair | fungi: that seems to be where it starts to diverge | 15:49 |
| fungi | we rely on the metadata server after all? | 15:49 |
| *** marun has joined #openstack-infra | 15:49 | |
| *** bdpayne has joined #openstack-infra | 15:49 | |
| fungi | for some reason i thought we'd managed to not need it | 15:49 |
| jeblair | me too, but it seems somehow tied into cloud-init's setting up of ssh keys? | 15:49 |
| jeblair | mordred, SpamapS: ^ can you shed light on this? | 15:50 |
| mordred | this is hp or rax? | 15:50 |
| fungi | hp | 15:50 |
| mordred | yeah - that's how keypair extension works | 15:50 |
| mordred | this is why I want to not install cloud-init | 15:50 |
| mordred | and just bake in keys | 15:50 |
| mordred | I am very sorry that is taking so long | 15:51 |
| fungi | well, it seems to be the underlying reason for a majority of boot failures in hpcloud | 15:51 |
| *** ashleighfarnham has joined #openstack-infra | 15:51 | |
| *** tjones1 has joined #openstack-infra | 15:51 | |
| jeblair | 397 in the past few hours | 15:51 |
| clarkb | can we config drive insyead? | 15:51 |
| mordred | as a stop-gap we could switch to config-drive and we could change our base image cloud-init config to ignore ec2 metadata service | 15:51 |
| fungi | i wonder if cloud-init couldn't just be coaxed into retrying that for longer as a stopgap? | 15:52 |
| fungi | or that | 15:52 |
| mordred | or - since it's hp - we _could_ just uninstall cloud-init in the base image and bake a nodepool key ni to it | 15:52 |
| clarkb | fungi metadata is likely notnroutable | 15:52 |
| clarkb | fungi so timeouts wont help | 15:52 |
| anteaya | what changed, something changed about 24 hours ago to result in these errors | 15:52 |
| mordred | we don't have to fix the problem for real yet - uninstalling cloud-init will not break rax | 15:52 |
| fungi | clarkb: it's linklocal. that's never routable. you mean it's not responding? | 15:52 |
| mordred | you want I should make a patch to do that real quick? | 15:53 |
| clarkb | fungi no the way the metadata service works is it gets nated amd must be routable | 15:53 |
| clarkb | fungi my experience is this is fragile and breaks a lot | 15:53 |
| clarkb | mordred does nodepool support that? | 15:53 |
| mordred | clarkb: it doesn't need to | 15:54 |
| fungi | clarkb: where do those nat rules live? something has to arp reply for that linklocal address on the local broadcast segment obviously | 15:54 |
| clarkb | mordred initial boot relies on cloud init iirc | 15:54 |
| *** dhritishikhar has joined #openstack-infra | 15:54 | |
| mordred | clarkb: no | 15:54 |
| clarkb | fungi nova/neutron set them up | 15:54 |
| mordred | I mean, image build step will wonky | 15:54 |
| mordred | but that doesn't affect node throughput | 15:54 |
| clarkb | fungi it lives on the router device iirc | 15:54 |
| mordred | lemme make a patch - it'll be 5 minutes | 15:54 |
| fungi | clarkb: so nova/neutron is failing to present that interface or a nat rule attached to it into the bridge to which the instance gets attached? | 15:55 |
| clarkb | mordred if image build doesnt work then you get no images with your fix... | 15:55 |
| rhe00 | I am experiencing an apparent race condition between nodepool and jenkins. Job #91 finishes up and nodepool receives the onCompleted and onFinalized notification, before nodepool has removed the node from the jenkins pool, the node gets picked up by Jenkins for a new job (#92). #92 quickly fails because the test can't grab the FC devices (the same node is still holding on to them). This all repeats for job #93. | 15:55 |
| jeblair | clarkb, mordred: i don't think zmq from jenkins04 is working | 15:55 |
| mordred | jeblair: awesome | 15:55 |
| mordred | clarkb: we can boot images _some_ times | 15:55 |
| clarkb | fungi my experience was that it was presented ok but packet return path is easily borked because they only DNAT | 15:55 |
| mordred | clarkb: all we need to do boot is one node during image build | 15:55 |
| mordred | for snapshot images | 15:55 |
| mordred | once we do that, all the rest of the nodes booted from taht one will be fine | 15:56 |
| fungi | oh lovely, and now freenode is falling over on me | 15:56 |
| mordred | I think it's worth a try | 15:56 |
| clarkb | mordred ok | 15:56 |
| anteaya | fungi: :( | 15:56 |
| *** gampel has quit IRC | 15:56 | |
| rhe00 | does anyone have any ideas what might be going on? I am about to dig into nodepool and see if I can close the race window. | 15:56 |
| anteaya | rhe00: at the moment we are all focused on getting working nodes for our zuul queues | 15:57 |
| openstackgerrit | Adrian Moisey proposed openstack-infra/jenkins-job-builder: Update the Nofifications plugin to support 1.9 https://review.openstack.org/165785 | 15:57 |
| anteaya | rhe00: we might have more time to help in a few hours | 15:57 |
| fungi | rhe00: we set our jobs to pass an OFFLINE_NODE_WHEN_COMPLETE parameter for the job with a value of 1 so that the gearman plugin offlines the node in jenkins before it can pick up a new job | 15:58 |
| rhe00 | anteaya: ok, I can wait. I will dig some more on my own. thanks! | 15:58 |
| anteaya | rhe00: thanks for understanding | 15:58 |
| rhe00 | fungi: ok, I will look into that. | 15:58 |
| fungi | i've got jenkins04 in prepare for shutdown, and will clean up its slave cruft once i can safely stop it | 15:59 |
| zaro | morning | 15:59 |
| openstackgerrit | Jerry Zhao proposed openstack-infra/nodepool: add option to use ipv6 for image update and node launching https://review.openstack.org/156178 | 15:59 |
| fungi | rhe00: we use a zuul parameter function to add that to our jobs by default | 15:59 |
| anteaya | morning zaro | 15:59 |
| clarkb | jeblair its likely the sockets werent closed on the other ends so they never reconnected | 15:59 |
| jeblair | clarkb: i think so | 15:59 |
| clarkb | we will probably need to restart nodepool and logstash client | 16:00 |
| jeblair | fungi, clarkb, mordred: i suspect nodepool wil need to be restarted to correct the zmq error with 04 | 16:00 |
| jeblair | ya | 16:00 |
| JayF | mordred: I don't have magical powers, but I do know an array of people to prod if something is wrong | 16:00 |
| jeblair | 04 is in shutdown mode now, it should prevent the problem from getting worse. i will delete more 04 nodes | 16:00 |
| *** esker has joined #openstack-infra | 16:00 | |
| fungi | jeblair: i can restart nodepoold once i have jenkins04 stopped | 16:01 |
| JayF | mordred: nevermind, I see jroll "resolved" your issue | 16:01 |
| *** rhsu has joined #openstack-infra | 16:01 | |
| *** baoli has quit IRC | 16:02 | |
| clarkb | we can try killing the tcp connection on the clients but iirc that never works well | 16:03 |
| *** baoli has joined #openstack-infra | 16:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Get rid of cloud-init on single-use-slaves https://review.openstack.org/165889 | 16:03 |
| mordred | clarkb, fungi, jeblair: ^^ | 16:03 |
| mordred | that's what I'm thinking to deal with the metadata service instability | 16:03 |
| *** koolhead17 has joined #openstack-infra | 16:03 | |
| devananda | gertty has started dumping urllib3 warnings this morning. making it somewhat non functional ... | 16:04 |
| clarkb | mordred that will break because the distros want you to use not root | 16:04 |
| *** LinuxJed_ has joined #openstack-infra | 16:04 | |
| clarkb | mordred I thibk we ssh as jenkins anyways so maybe unnecessary | 16:04 |
| mordred | clarkb: oh right. so I need to add it to each of the distro users | 16:04 |
| mordred | clarkb: when we boot, don't we do an ssh in to test things? | 16:05 |
| mordred | as the nodepool user | 16:05 |
| clarkb | mordred or maybe we dont need it at all if nodepool sshs in as jenkins | 16:05 |
| mordred | to the distro user | 16:05 |
| mordred | it does not | 16:05 |
| mordred | it ssh's in as the nodepool user to the appropriate distro user | 16:05 |
| mordred | modified patch coming | 16:05 |
| clarkb | then ya it needs to be the distro user | 16:05 |
| jeblair | i'm deleting all nodes on 04 which are not running a job | 16:05 |
| fungi | thanks | 16:06 |
| fungi | that should help | 16:06 |
| *** ashleighfarnham has quit IRC | 16:06 | |
| *** tnovacik has joined #openstack-infra | 16:07 | |
| *** Somay has joined #openstack-infra | 16:07 | |
| *** jcoufal_ has quit IRC | 16:08 | |
| *** LinuxJedi has quit IRC | 16:08 | |
| mordred | taht's ubuntu on ubuntu, cloud-user on centos and fedora on fedora? | 16:08 |
| clarkb | sounds right, there is a list in nodepool itself | 16:09 |
| mordred | it's not mapped - we just try each of them :) | 16:09 |
| mordred | oh - centos is root | 16:09 |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding https://review.openstack.org/164264 | 16:10 |
| zaro | fungi: question about this ^ | 16:10 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack-infra/project-config: Add "heat-coe-templates" project https://review.openstack.org/164806 | 16:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Get rid of cloud-init on single-use-slaves https://review.openstack.org/165889 | 16:11 |
| *** _nadya_ has quit IRC | 16:11 | |
| zaro | fungi: do you think the same code to manage my.cnf should also be duplicated in gerrit::mysql ?? | 16:11 |
| mordred | clarkb: ^^ that should fix it | 16:11 |
| *** belmoreira has quit IRC | 16:12 | |
| fungi | zaro: well, i'm not sure that ! defined(Class['gerrit::mysql']) will actually ensure that gerrit::mysql isn't used, just that it hasn't been defined before that gets tested. but someone with better puppet mojo should confirm | 16:12 |
| clarkb | mordred the linter may not like the extra whitespace but other than that it looks good | 16:12 |
| SpamapS | jeblair: just now sat down. Reading backscroll. | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Get rid of cloud-init on single-use-slaves https://review.openstack.org/165889 | 16:12 |
| mordred | fungi, jeblair, clarkb: ^^ whitespace fixed | 16:13 |
| mordred | I mean, given the state of the queue, we may need ninja-merge that | 16:13 |
| zaro | nibalizer: I've adding refreshonly and subscribe to this https://review.openstack.org/#/c/165145/6/modules/openstack_project/manifests/gerrit.pp | 16:13 |
| clarkb | mordred ya | 16:13 |
| mordred | so I'd love to get eyes on it | 16:13 |
| fungi | zaro: as far as doing something similar in the gerrit module's mysql.pp, if you want to do that you probably need to implement it via the mysqldb::config class (i think that's what it's called) | 16:13 |
| zaro | nibalizer: was wondering if i should remove some of the requires? | 16:13 |
| SpamapS | FYI I believe the reason HP's ec2 metadata service fails has something to do with conntrack filling up on compute hosts or neutron-metadata-agent nodes. I forget which. | 16:14 |
| clarkb | mordred the package is the same name on centos and fedora too? | 16:14 |
| mordred | clarkb: checking | 16:14 |
| anteaya | mordred: you don't need commas after 'fedora' 'ubuntu' and 'root'? | 16:14 |
| mordred | anteaya: nope - it's a case statement, not a parameter list ... but good eye | 16:14 |
| anteaya | okay | 16:14 |
| mordred | clarkb: it's the same on centos - I'm not sure I've got a fedora lying aroudn ... | 16:15 |
| mordred | oh - I can check yum in a docker image | 16:15 |
| *** scheuran has quit IRC | 16:15 | |
| SpamapS | Kiall: ^^ would you concur, that the reason HP Cloud's ec2 metadata stops responding is conntrack related? | 16:15 |
| fungi | devananda: known issue. you can either roll back to an earlier urllib3, install requests[security] or apply the proposed gertty change to silence that warning | 16:15 |
| zaro | fungi: hmm, is that an existing class? i can't seem to find a config.pp | 16:15 |
| *** ildikov has quit IRC | 16:15 | |
| fungi | zaro: it's not our module | 16:15 |
| *** ildikov has joined #openstack-infra | 16:15 | |
| Kiall | SpamapS: I think there might be another reason behind that .. | 16:15 |
| devananda | fungi: great. thanks | 16:16 |
| *** unicell has quit IRC | 16:16 | |
| mordred | SpamapS: enjoy my most recent patch :) | 16:16 |
| mordred | clarkb: yes - also cloud-init on fedora | 16:16 |
| fungi | zaro: aha, it's called mysql::server::config but probably the parameter you want is exposed through mysql::server https://forge.puppetlabs.com/puppetlabs/mysql#customizing-server-options | 16:16 |
| *** mpavone has quit IRC | 16:18 | |
| SpamapS | mordred: to which thing? | 16:18 |
| mordred | clarkb: I got the public key value via: | 16:18 |
| mordred | nodepool@nodepool:~$ ssh-keygen -y -f ~/.ssh/id_rsa | 16:18 |
| mordred | clarkb: fwiw | 16:18 |
| mordred | SpamapS: https://review.openstack.org/#/c/165889/ | 16:19 |
| SpamapS | mordred: oh, so why aren't we just switching to config drive? | 16:19 |
| openstackgerrit | Max Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add support for 'raw' to inject direct xml to job configs. https://review.openstack.org/160158 | 16:19 |
| mordred | SpamapS: because that would involve adding features to nodepool, and we'd STILL have to make a system-cofig change to configure cloud-init to skip ec2 metadata | 16:20 |
| SpamapS | mordred: skipping ec2 metadata would just speed things up a bit | 16:20 |
| mordred | SpamapS: whereas, the only thing we use it for on these hosts is getting a known ssh key on them | 16:20 |
| SpamapS | mordred: it would work .. just would have a 2 minute boot delay | 16:20 |
| mordred | SpamapS: so, really, that patch above solves everything :) | 16:20 |
| mordred | and removes one more piece of fragility | 16:20 |
| *** rhsu has quit IRC | 16:20 | |
| SpamapS | mordred: sure, I think it's fine. It does mean our images are now _OUR_ images. But that was always true really, nobody's using them stock right? | 16:21 |
| zaro | fungi: from what i've gathered defined('gerrit::mysql') check that the class has been defined. while defined(Class['gerrit::mysql']) checks whether it has been declared. | 16:21 |
| zaro | https://docs.puppetlabs.com/references/latest/function.html#defined | 16:21 |
| *** dboik_ has quit IRC | 16:21 | |
| SpamapS | Building in SSH keys is now things worked on EC2 for a lot of people too before cloud-init. | 16:21 |
| mordred | SpamapS: that's right | 16:21 |
| SpamapS | Because fetching bits from ec2 metadata was poorly understood. | 16:21 |
| *** pblaho__ has joined #openstack-infra | 16:22 | |
| *** amotoki has quit IRC | 16:22 | |
| *** dboik has joined #openstack-infra | 16:22 | |
| *** ddieterly has quit IRC | 16:22 | |
| mordred | yup | 16:22 |
| clarkb | SpamapS I use them stock | 16:22 |
| clarkb | but I have the benefit of having my key baked in :) | 16:22 |
| mordred | clarkb: jeblair is busy, but says he's ++ on 86ing cloud-init ... fungi - opinion? | 16:22 |
| mordred | clarkb: :) | 16:22 |
| *** ddieterly has joined #openstack-infra | 16:22 | |
| mordred | clarkb: honestly, right now, I care more about us and less about people who might reuse this - we can do better later | 16:22 |
| clarkb | mordred ya | 16:22 |
| *** ihrachyshka has quit IRC | 16:22 | |
| jeblair | ya, sorry :( | 16:23 |
| fungi | yep, wfm | 16:23 |
| mordred | great. I'm going to ninja approve it | 16:23 |
| larsks | Hello all. I'd like to add a project to stackforge that will run a commit check using tox, but *not* pep8 (because it's not python). What's the correct way to set this up? | 16:23 |
| fungi | mordred: what's with the require => File['/root/.ssh'] there? | 16:23 |
| SpamapS | larsks: IIRC pep8 is in python-jobs | 16:24 |
| mordred | fungi: THANK YOU | 16:24 |
| SpamapS | larsks: so don't list python-jobs | 16:24 |
| larsks | SpamapS: Right. But I'm not entirely clear how to add a *new* job. | 16:24 |
| *** radez is now known as radez_g0n3 | 16:25 | |
| *** pblaho_ has quit IRC | 16:25 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Get rid of cloud-init on single-use-slaves https://review.openstack.org/165889 | 16:26 |
| mordred | fungi, clarkb: ^^ | 16:26 |
| *** emagana has joined #openstack-infra | 16:26 | |
| *** teran_ has quit IRC | 16:26 | |
| SpamapS | larsks: maybe look at how devstack.yaml is done in project-config/jenkins/jobs | 16:27 |
| larsks | SpamapS: I'll take a look...thanks. | 16:27 |
| clarkb | mordred lgtm | 16:27 |
| openstackgerrit | Merged openstack-infra/system-config: Get rid of cloud-init on single-use-slaves https://review.openstack.org/165889 | 16:27 |
| mordred | ok. now to kick off a new image ... I'm going to start with one in hp | 16:28 |
| mordred | just to see | 16:28 |
| SpamapS | do you have to put the private key in the secret place for infra root only? | 16:28 |
| mordred | SpamapS: that is the public key for the already-existing private key for the nodepool user | 16:28 |
| SpamapS | I assume there's puppet code that just references variables that aren't set in the normal hiera? | 16:28 |
| * SpamapS is using words he's not entirely comfortable with | 16:28 | |
| SpamapS | mordred: OH | 16:28 |
| mordred | SpamapS: all of the puppet code that references things in the secret hiera is in the hiera() calls | 16:29 |
| SpamapS | but that is still true yes? | 16:29 |
| mordred | yes | 16:29 |
| SpamapS | ok | 16:29 |
| * SpamapS stops sweating | 16:29 | |
| *** ghostpl_ has quit IRC | 16:29 | |
| mordred | SpamapS: if you look in manifest/site.pp, you'll see: | 16:29 |
| mordred | nodepool_ssh_private_key => hiera('jenkins_ssh_private_key_contents', 'XXX'), | 16:29 |
| mordred | clarkb: heh. ^^ you were half-right before | 16:30 |
| mordred | it is the jenkins KEY that gets used | 16:30 |
| mordred | :) | 16:30 |
| ttx | fungi: is there a way to remove the stuck jobs from zuul check queue display ? | 16:30 |
| anteaya | ttx by uploading a new commit message | 16:30 |
| anteaya | we discussed that | 16:30 |
| fungi | ttx: yeah, new patchset to them. are they targeted for the freeze? | 16:30 |
| fungi | or just unsightly? | 16:30 |
| anteaya | but figured gettin gmore workers was a higher priority than kicking them back in and having them use resources | 16:30 |
| mordred | clarkb: root@nodepool:~# nodepool image-update hpcloud-b5 nodepool-trusty | 16:31 |
| mordred | is what I'm doing right now | 16:31 |
| ttx | fungi: not necessarily, it's just that people tend to scream "OMG please tag K3 Monday check queue is BORK" | 16:31 |
| ttx | even if it isn't | 16:31 |
| anteaya | it is very slow | 16:31 |
| anteaya | which we are working on but getting new workers | 16:31 |
| anteaya | but k-3 isn't a surprise | 16:31 |
| mordred | clarkb: I lied: root@nodepool:~# nodepool image-update hpcloud-b5 devstack-trusty | 16:32 |
| anteaya | and some ptls were willing to merge stuff on the weekend to avoid problems | 16:32 |
| anteaya | mikal mentioned it at last week's nova meeting | 16:32 |
| *** esker has quit IRC | 16:32 | |
| *** dmorita has joined #openstack-infra | 16:32 | |
| *** Bsony has joined #openstack-infra | 16:32 | |
| clarkb | mordred ok that should do dib build then upload, if it works dont image-update that image again, instead just image-upload it to the other providers | 16:32 |
| mordred | clarkb: ++ | 16:33 |
| mordred | clarkb: nice thing - since this is the dib nodes, we don't even have to wait on a template node to be able to spin up against the metadata borking | 16:33 |
| *** tsg_ has joined #openstack-infra | 16:33 | |
| *** tqtran has joined #openstack-infra | 16:34 | |
| *** sarob has quit IRC | 16:34 | |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack-infra/project-config: Add "heat-coe-templates" project https://review.openstack.org/164806 | 16:34 |
| *** ghostpl_ has joined #openstack-infra | 16:34 | |
| anteaya | I'm going to be afk for a bit | 16:35 |
| anteaya | back later | 16:36 |
| pleia2 | good morning | 16:36 |
| *** _nadya_ has joined #openstack-infra | 16:37 | |
| jeblair | mordred: "nodepool-trusty" or "devstack-trusty" ? | 16:37 |
| mordred | jeblair: the second | 16:37 |
| *** yfried has quit IRC | 16:37 | |
| jeblair | cool, that'll be better | 16:37 |
| mordred | jeblair: turns out, we don't have any images called nodepool-trusty | 16:37 |
| mordred | jeblair: luckily - nodepool refused to operate :) | 16:38 |
| nibalizer | zaro: shrug | 16:39 |
| greghaynes | clarkb: did you see I got https://review.openstack.org/#/c/165682/ worked out | 16:39 |
| mordred | clarkb: STAB STAB STAB | 16:39 |
| nibalizer | requires never hurt anybody | 16:39 |
| greghaynes | clarkb: for doing monkeypatching of fakes | 16:39 |
| greghaynes | ended up working nicely | 16:39 |
| *** ayoung has quit IRC | 16:39 | |
| jeblair | deleting another round of jenkins04 nodes | 16:39 |
| mordred | clarkb: in our dib images, we don't have an ubuntu user, apparently | 16:39 |
| zaro | nibalizer: cool, i'll take that :) | 16:40 |
| mordred | SpamapS: is that expected in images built from dib on the ubuntu element? | 16:40 |
| clarkb | mordred lol | 16:40 |
| jeblair | mordred: i thought that was a desired feature of dib images? | 16:40 |
| *** sandywalsh has joined #openstack-infra | 16:40 | |
| mordred | jeblair: yes, I believe so - it just makes this puppet mildly harder | 16:40 |
| clarkb | greghaynes I noted you pushed a lot of code while I watched day9 play grim fandango then I fell asleep | 16:41 |
| fungi | mordred: well, only until we're all-dib there ;) | 16:41 |
| SpamapS | mordred: yes | 16:41 |
| greghaynes | mordred: wierd, I thought we got that for free with ubuntu cloudimg | 16:41 |
| fungi | then it makes the puppet much, much easier | 16:41 |
| SpamapS | mordred: cloud-init makes the user | 16:41 |
| mordred | fungi: yes. :) | 16:41 |
| greghaynes | oh! | 16:41 |
| SpamapS | mordred: should be true on all OS's | 16:41 |
| greghaynes | lolol | 16:41 |
| mordred | SpamapS: hahahahaahahah | 16:41 |
| mordred | SpamapS: orly | 16:41 |
| mordred | ? | 16:41 |
| clarkb | SpamapS wait | 16:41 |
| SpamapS | since cloud-init 0.7 | 16:41 |
| mordred | SpamapS: so - does the "you can't log in with the root user" come from cloud-init too? | 16:41 |
| fungi | WAT | 16:41 |
| clarkb | this isnt a distro sillyness? | 16:41 |
| clarkb | its cloud init? | 16:41 |
| clarkb | what are they smoking | 16:41 |
| mordred | SpamapS: like, if I remove cloud-init - can I just put the key in /root/ ? | 16:42 |
| mordred | and have it work? | 16:42 |
| SpamapS | mordred: No I think they just smash that into sshd's config | 16:42 |
| fungi | mordred: i believe so. i patch that out in my cloudinit config | 16:42 |
| SpamapS | during image build | 16:42 |
| clarkb | wow | 16:42 |
| fungi | SpamapS: it has an option for it at least | 16:42 |
| clarkb | I dont | 16:42 |
| clarkb | even | 16:42 |
| mordred | ok. follow up patch coming | 16:42 |
| SpamapS | or did they add that too? | 16:42 |
| clarkb | wat | 16:42 |
| mordred | clarkb: see - this is why we're getting out of the cloud-init business | 16:42 |
| SpamapS | Wouldn't surprise me if they put all cloud image intelligence into cloudinit | 16:42 |
| mordred | STOP TRYING TO BE CLEVER | 16:42 |
| greghaynes | Why dont youall bake in a user you know the username of? | 16:43 |
| mordred | we're installing sshd_config files with puppet already aren't we? | 16:43 |
| greghaynes | I love suggesting this | 16:43 |
| SpamapS | mordred: I believe this makes the cloud images more consumable in different environments. | 16:43 |
| mordred | greghaynes: it's called "root" | 16:43 |
| greghaynes | but everyone hates doing that for some reason | 16:43 |
| jeblair | greghaynes: we could call it... root | 16:43 |
| mordred | it's amazing | 16:43 |
| mordred | it's a user that has root access | 16:43 |
| fungi | SpamapS: rackspace installs a /etc/cloud/cloud.cfg.d/10_rackspace.cfg which turns off passwordauthentication and permitrootlogin for example | 16:43 |
| *** jordanP has joined #openstack-infra | 16:43 | |
| greghaynes | hah, fine | 16:43 |
| SpamapS | mordred: don't hate on something people do actually like. You're just made because it doesn't work well on hpcloud. ;) | 16:43 |
| *** unicell has joined #openstack-infra | 16:43 | |
| greghaynes | that works too :) | 16:43 |
| SpamapS | or rax | 16:43 |
| SpamapS | ;) | 16:43 |
| mordred | SpamapS: or, well, for any of our use cases | 16:43 |
| mordred | SpamapS: but I heard you | 16:43 |
| mordred | hear | 16:43 |
| SpamapS | EC2 is the 800lb. gorilla, and most of what cloud-init does is at the request of EC2. The rest is at the request of harlowja_away | 16:44 |
| *** BobBall is now known as BobBall_AWOL | 16:44 | |
| jeblair | why would someone who isn't ec2... nevermind, i'm falling into the same trap | 16:44 |
| openstackgerrit | Douglas Mendizábal proposed openstack-infra/project-config: Add Python 2.6 jobs to Castellan https://review.openstack.org/165913 | 16:44 |
| clarkb | yes its amazing that every install comes with a user preinstalled to do exactly this | 16:44 |
| jordanP | hi guys. I am in charge of a third party CI. I am using jenkins + gerrit trigger (not zuuul). I have a formating issue with how to report back the results. Can someone help me ? Do you know if someone is using successfully gerrit trigger ? | 16:44 |
| greghaynes | clarkb: hah about day9, I think I was actually netsplit from openstackbot when I was pushing that and assume no one was seeing me sneak in all those patches | 16:45 |
| clarkb | then for some reason everyone has decided to not use that user | 16:45 |
| SpamapS | I'll end my cloud-init sermon with this: The dream is that one root FS can be used on any cloud, for any user, with userdata being the only thing that customizes the image. | 16:46 |
| fungi | clarkb: because being able to log in as root is bad. because it's a predictable name which eases brute-forcing. so replace it with an equally-guessable name | 16:46 |
| SpamapS | The reality is nobody cares about that except distro vendors. :) | 16:46 |
| cinerama | anyone here have some historical knowledge about openstackid? was messing with it the other day to get logins going on zanata | 16:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Just install the key into the root user https://review.openstack.org/165914 | 16:46 |
| clarkb | SpamapS: using root as the default user does nothing to impede that goal | 16:46 |
| mordred | fungi, SpamapS, clarkb, jeblair: ^^ | 16:46 |
| fungi | SpamapS: actually i care a lot about that. the problem is that the methods for getting the userdata to the instances is, well, not straightforward | 16:47 |
| SpamapS | clarkb: using root as the default user removes all chance at audit logging via sudo. | 16:47 |
| SpamapS | It's a sane general default. | 16:47 |
| clarkb | mordred: sorry that my initial comment sent us down this path | 16:47 |
| mordred | clarkb: it's ok | 16:47 |
| mordred | SpamapS: I hear that - but in my experience so far - userdata via cloud init is WAY less useful than userdata via ssh post boot | 16:47 |
| clarkb | SpamapS: enabling sudo removes all chance at audit logging via sudo | 16:47 |
| mordred | all one needs is one very simple method to get an ssh key onto the host | 16:47 |
| lucasagomes | hi, if you have some time please take a look at https://review.openstack.org/#/c/163344/ it's adding a gate check for openstack/ironic-python-agent (non-voting) to test PXE drivers using the IPA ramdisk | 16:47 |
| SpamapS | mordred: becuase your userdata delivery methods suck. :) | 16:47 |
| jeblair | mordred, clarkb: so if this runs on a snapshot image, won't we still have an ubuntu user? | 16:47 |
| SpamapS | clarkb: eh? | 16:48 |
| clarkb | jeblair: yes, but hrm gah | 16:48 |
| clarkb | SpamapS: once you have root via sudo its game over regardless | 16:48 |
| mordred | jeblair: but if we install the key onto the root user | 16:48 |
| mordred | jeblair: then the nodepool user cycle should still find it | 16:48 |
| mordred | regardless of whetehr the ubuntu user exists | 16:48 |
| SpamapS | clarkb: the Ubuntu security team's thinking has always been that the system should be setup to ship auth/security logs elsewhere. So you should, in theory, at least get a 'sudo rm /etc/syslog/....' before it goes dark | 16:48 |
| clarkb | mordred: except that the initial cloud init would have broken our ability to ssh as root | 16:48 |
| jeblair | mordred: right, because we try root first | 16:48 |
| mordred | clarkb: but we install an sshd_config | 16:49 |
| mordred | with puppet | 16:49 |
| mordred | so we will ahve fixed it | 16:49 |
| clarkb | mordred: gotcha | 16:49 |
| *** hdd has joined #openstack-infra | 16:49 | |
| mordred | clarkb: let me verify that we do install an sshd_config | 16:49 |
| jeblair | we better :) | 16:49 |
| SpamapS | clarkb: so that's not exactly "game over" , that is "first battle goes to the person who gained access to the passwordless sudo user".. but you at least have a breadcrumb to that moment. | 16:49 |
| mordred | I know we do on our long-lived hosts | 16:50 |
| *** thingee has joined #openstack-infra | 16:50 | |
| SpamapS | For the purposes of single use nodes, yes.. just root in baby. | 16:50 |
| clarkb | mordred: ya I think you are correct, the sshd config is applied universally | 16:50 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-zuul: Add fetch_url and poll parameters to zuul.conf https://review.openstack.org/162591 | 16:50 |
| jeblair | SpamapS: (as an aside, if you want the breadcrumb with root logins, just have ssh log the key id of incoming users) | 16:50 |
| mordred | clarkb: I cannot find where we do it | 16:51 |
| *** sandywalsh has quit IRC | 16:51 | |
| clarkb | mordred: let me find it | 16:51 |
| SpamapS | jeblair: the theory is that you want regular interactive users to be trained to never shell as root, so they're used to sudo'ing. | 16:51 |
| SpamapS | jeblair: It's all theory. | 16:52 |
| mordred | SpamapS: sure. and I appluad that theory | 16:52 |
| SpamapS | I don't want it to turn into a security theory debate, just trying to explain why I think it's fine to force people through the sudo funnel. | 16:52 |
| mordred | I just think that a second shared full-root user like the ubuntu user is actually a bad idea | 16:52 |
| *** dustins_ has joined #openstack-infra | 16:52 | |
| clarkb | mordred: include ssh in template.pp I think | 16:52 |
| clarkb | mordred: openstack_project::template | 16:52 |
| mordred | awesome | 16:52 |
| *** yamahata has joined #openstack-infra | 16:52 | |
| SpamapS | mordred: well in theory 90% of what you want to do as a sysadmin is fine as a regular user. top, ps, ls, netstat, etc. | 16:53 |
| clarkb | SpamapS: maybe cloud init should have created a cloud-init user if that was the goal? | 16:53 |
| mordred | clarkb: oh - actually - we're going to need ... | 16:53 |
| mordred | blast | 16:53 |
| clarkb | SpamapS: using distro specific users provides zero benefit and a lot of work | 16:53 |
| SpamapS | clarkb: it did, but each distro decided it should be a marketing opportunity. :) | 16:53 |
| jeblair | mordred: i say "we better" install an sshd_config, because that was literally change 1 in gerrit ;) | 16:53 |
| SpamapS | I agree 100% that having the user be the distro name was short sighted. | 16:53 |
| jeblair | https://review.openstack.org/1 | 16:53 |
| fungi | nice! | 16:54 |
| SpamapS | even worse was Fedora making it 'ec2-user' | 16:54 |
| *** baoli has quit IRC | 16:54 | |
| mordred | jeblair: we do have that with PermitRootLogin off | 16:54 |
| mordred | and | 16:54 |
| mordred | Match host puppetmaster.openstack.org | 16:54 |
| mordred | PermitRootLogin without-password | 16:54 |
| clarkb | mordred: ha! | 16:54 |
| mordred | :) | 16:54 |
| mordred | one sec ... | 16:54 |
| fungi | so, yeah, we'll need some tweaking | 16:54 |
| *** baoli has joined #openstack-infra | 16:55 | |
| mordred | does anybody know if a per-key ssh option will override that? | 16:55 |
| clarkb | how hard would it be to just config drive instead? | 16:55 |
| mordred | harder | 16:55 |
| clarkb | mordred: we can't just set the image metadata in nodepool config? | 16:55 |
| *** dustins has quit IRC | 16:55 | |
| *** ildikov has quit IRC | 16:55 | |
| clarkb | mordred: iirc that was all we needed to do right (and one reason we added image metadata) | 16:55 |
| mordred | clarkb: it will be at LEAST as much work as this | 16:55 |
| mordred | because we'll ahve to configure the nodes to mount config-drive | 16:56 |
| clarkb | mordred: yes but thats a one line update to nodepool.yaml | 16:56 |
| mordred | I'm going to keep working on this | 16:56 |
| mordred | because cloud-init can suck it | 16:56 |
| clarkb | then cloud init does the rest? | 16:56 |
| mordred | no it doesn't | 16:56 |
| *** Ryan_Lane has joined #openstack-infra | 16:56 | |
| mordred | cloud-init does not mount config-drive | 16:56 |
| mordred | because BONGHITS | 16:56 |
| *** skolekonov has quit IRC | 16:56 | |
| *** gyee has quit IRC | 16:56 | |
| clarkb | I thought it did | 16:57 |
| SpamapS | but Ubuntu does? | 16:57 |
| mordred | yeah - me too | 16:57 |
| mordred | it dose not | 16:57 |
| SpamapS | hrm | 16:57 |
| jeblair | mordred: silly question time -- are you focused on dib or snapshot images? | 16:57 |
| mordred | I promise | 16:57 |
| mordred | jeblair: both - I'm just doing this in puppet so it should hit both of them | 16:57 |
| jeblair | mordred: so nodepool does not ssh into anything other than jenkins on the dib path | 16:57 |
| pleia2 | AJaeger: if you're still about, could use some confirmation about where we need zanata client installed, I suspect it's just our translations slave but I want to be sure | 16:57 |
| mordred | jeblair: oh - ok - so I can make this simpler then | 16:58 |
| pleia2 | AJaeger: translations/proposals - it auto-submits changes to gerrit, like 75%+ translations pulled from transifex | 16:58 |
| *** hyakuhei has joined #openstack-infra | 16:59 | |
| *** ihrachyshka has joined #openstack-infra | 17:00 | |
| fungi | okay, jenkins04 just achieved quiescence, so stopping the jenkins service now and cleaning it up, then i'll restart nodepoold before i start jenkins again | 17:00 |
| jeblair | mordred: and on the snapshot path -- it only does that when creating a snapshot, before puppet is run (that ssh connection is so that it can run puppet) | 17:00 |
| clarkb | fungi: let me know when you do that and I will restart the logstash client too | 17:00 |
| fungi | clarkb: you bet | 17:01 |
| *** Swami has joined #openstack-infra | 17:01 | |
| mordred | jeblair: really? ok, then I'm confused as to why the metadata server is giving us grief | 17:01 |
| *** arxcruz has quit IRC | 17:01 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding https://review.openstack.org/164264 | 17:02 |
| clarkb | mordred: because cloud init is still enabled | 17:02 |
| jeblair | mordred: well, i mean cloud-init is run | 17:02 |
| clarkb | how does disabling cloud init affect ssh host key generation? | 17:02 |
| fungi | mordred: i believe it's because cloud init blanks the ssh host key and wants to recreate it so that it's unique, but goes sideways when it can't talk to the metadata server | 17:02 |
| clarkb | will nodepool stop working because too many hosts with the same key? | 17:02 |
| *** derekh has quit IRC | 17:02 | |
| fungi | would be good to verify though | 17:02 |
| jeblair | fungi: that seems reasonable | 17:03 |
| mordred | so, actually then, all we need to do is remove cloud-init | 17:03 |
| mordred | because we already have the key baked in that we care about | 17:03 |
| jeblair | mordred: correct | 17:03 |
| mordred | and it literally does nothing | 17:03 |
| mordred | awesome | 17:03 |
| mordred | that's the easiest patch ever | 17:03 |
| fungi | that sounds likely anyway. likely enough to test | 17:03 |
| *** unicell has quit IRC | 17:03 | |
| *** unicell1 has joined #openstack-infra | 17:03 | |
| *** lucasagomes has quit IRC | 17:03 | |
| SpamapS | Honestly, if you're not going to replace the key that is in the fetchable image .. telnet ftw? ;) | 17:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Just remove cloud-init https://review.openstack.org/165914 | 17:04 |
| mordred | SpamapS: what? | 17:04 |
| mordred | SpamapS: that makes no sense at all | 17:04 |
| mordred | SpamapS: the key is a public key | 17:04 |
| SpamapS | the key that is not being replaced is the private host key | 17:04 |
| mordred | oh - the one that provides no security at all because of how clouds work? | 17:05 |
| mordred | yeah | 17:05 |
| mordred | important to replace that | 17:05 |
| fungi | SpamapS: don't most sshd initscripts create the key at start if it's missing? | 17:05 |
| SpamapS | Exactly> :) | 17:05 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Add stackforge/puppet-nscld https://review.openstack.org/165922 | 17:05 |
| mordred | since I have no secure path to verify it | 17:05 |
| jeblair | SpamapS is right -- we will have identical private keys (which can be retrieved by a job that issues 'cat /etc/ssh/id_rsa...') | 17:05 |
| SpamapS | fungi: yes | 17:05 |
| fungi | so we could just delete the host keys in our images | 17:05 |
| jeblair | i am not sure what someone would do with that, but still... | 17:05 |
| *** pcaruana has quit IRC | 17:05 | |
| fungi | as part of the image build process | 17:05 |
| jeblair | fungi: that sounds like it would work. | 17:05 |
| SpamapS | with that one can mitm and decrypt passively sniffed packets with a full ssh session. | 17:06 |
| mordred | fungi: want me to put that into the puppet change above? | 17:06 |
| *** sputnik13 has joined #openstack-infra | 17:06 | |
| SpamapS | ergo: telnet ftw :) | 17:06 |
| SpamapS | I'm half serious. | 17:06 |
| fungi | mordred: yeah, that can't hurt. at least we'll find out where it works and where we need to add a boot-time shim to call ssh-keygen to do it | 17:06 |
| jeblair | yeah, this is probably not important for most of our single-use nodes today, but eventually, we will want to run important things on single-use nodes, so, we should probably fix it | 17:07 |
| SpamapS | mordred: there's a secure way to verify host keys, but it involves passing one in that you generate, and then throwing away that one and generating a new one. | 17:07 |
| *** jistr has quit IRC | 17:07 | |
| mordred | jeblair: you want to fix it with this patch or as a follow on | 17:07 |
| SpamapS | and that assumes nobody has rooted the box before you get to it to replace the one you passed in. | 17:07 |
| jeblair | mordred: this one | 17:07 |
| mordred | jeblair: k on it | 17:07 |
| greghaynes | SpamapS: I think the issue is still anyone can make a job to cat it right? | 17:07 |
| *** amitgandhinz has quit IRC | 17:08 | |
| SpamapS | greghaynes: they'd be catting their own host's private key. It would be useless after that node was destroyed. | 17:08 |
| clarkb | SpamapS: and mostly useless in the interim since they already have access to that node | 17:08 |
| greghaynes | SpamapS: No, because you could still use it to mitm since itll verify fine | 17:08 |
| fungi | clarkb: i'm restarting nodepoold now | 17:08 |
| SpamapS | greghaynes: you generate a small (1024bit) RSA key, pass it in via userdata, and let the box install that. Once you SSH in with that one, you rm that key, and regenerate the private host key, noting the public key. | 17:09 |
| greghaynes | SpamapS: you have to also revoke it and then its complicated | 17:09 |
| *** harlowja_away is now known as harlowja_ | 17:09 | |
| SpamapS | clarkb: right | 17:09 |
| SpamapS | greghaynes: You wouldn't put it in your normal known_hosts file. :) | 17:09 |
| fungi | okay, nodepoold is running again | 17:09 |
| harlowja_ | SpamapS lol (just saw cloud-init comment) | 17:09 |
| fungi | and i've marked all the jenkins04 nodes delete | 17:09 |
| SpamapS | greghaynes: that initial SSH would be in known_hosts_for_this_one_time_only .. | 17:09 |
| fungi | and deleted them from the jenkins config | 17:10 |
| greghaynes | ah, yea, that works | 17:10 |
| SpamapS | so the window for compromise relies on somebody intercepting that private key between the time you fed it to nova and the time you SSH in. | 17:10 |
| fungi | jenkins service is coming back online on jenkins04 now | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Remove ssh host keys during image build https://review.openstack.org/165924 | 17:10 |
| mordred | jeblair, fungi: ^^ | 17:10 |
| clarkb | fungi: ok restarting the gearman client now | 17:10 |
| SpamapS | this prevents mitm w/ arbitrary key. it does not prevent #allthethings. | 17:10 |
| mordred | clarkb, SpamapS: ^^ you too | 17:11 |
| jeblair | mordred: both lgtm | 17:11 |
| greghaynes | nice | 17:11 |
| SpamapS | mordred: +1'd | 17:11 |
| *** sandywalsh has joined #openstack-infra | 17:12 | |
| clarkb | mordred: see comment | 17:12 |
| clarkb | mordred: I don't think the dib element needs sudo but pretty sure snapshot does | 17:12 |
| *** timcline has quit IRC | 17:12 | |
| mordred | ++ | 17:12 |
| *** nilasae has quit IRC | 17:13 | |
| cinerama | pleia2: i meant to ask you if we should try to move to the later wildfly | 17:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Remove ssh host keys during image build https://review.openstack.org/165924 | 17:13 |
| mordred | clarkb: nice catch | 17:13 |
| clarkb | mordred: I have hit that enough times myself :) | 17:13 |
| *** Hal2 has joined #openstack-infra | 17:14 | |
| clarkb | mordred: also reading cloud init docs and config drive docs you are correct | 17:14 |
| *** Hal1 has quit IRC | 17:14 | |
| clarkb | mordred: it appears you must mount the config drive yourself | 17:14 |
| clarkb | which makes me wonder why cloud init even exists | 17:14 |
| mordred | yes | 17:14 |
| pleia2 | cinerama: we'll need to do some testing, not all of the wildfly version work just yet | 17:14 |
| mordred | clarkb: every. single. use. of cloud init is broken in some little tiny way | 17:14 |
| mordred | clarkb: drives me batty | 17:14 |
| openstackgerrit | Merged openstack-infra/system-config: Just remove cloud-init https://review.openstack.org/165914 | 17:14 |
| openstackgerrit | Merged openstack-infra/project-config: Remove ssh host keys during image build https://review.openstack.org/165924 | 17:15 |
| harlowja_ | no mount yourself, lol | 17:15 |
| SpamapS | mordred: harlowja_ suggests that configdrive should be mounted by clout-init if Ubuntu's cloud image has it enabled. | 17:15 |
| harlowja_ | liesss | 17:15 |
| pleia2 | cinerama: we had been stuck on a particular one because of compatibility problems, but with the latest zanata release now is a good time to check | 17:15 |
| harlowja_ | ^ ya, not a lie what SpamapS says, lol, lie that u need to mount it yourself :-P | 17:15 |
| cinerama | pleia2: i mean as long as we haven't rolled out yet it might be worth a shot | 17:15 |
| clarkb | I have checked jenkins thread counts on our masters, no master appears to currently link threads | 17:15 |
| mordred | harlowja_: sure. except that it does not work | 17:15 |
| clarkb | s/link/leak/ | 17:15 |
| mordred | harlowja_: empirical testing has shown me that it does not | 17:15 |
| pleia2 | cinerama: also fyi - like gerrit, we'll have a zanata-dev server as well once we go into production, so we'll have a space to work on upgrade testing in the future :) | 17:15 |
| harlowja_ | mordred hmmm, works on yahoo cloud for a long time | 17:15 |
| mordred | so, I don't really know or care about the theory - in teh existing images on the public clouds I touch, it does not work | 17:16 |
| cinerama | pleia2: party time in upgrade city | 17:16 |
| mordred | I'm sure it coudl be configured to | 17:16 |
| clarkb | but 04 did have latest ssh-slaves plugin and still leaked so that may not be the fix, but 02, 05, and 07 all have that plugin version to and haven't leaked since so it is possible that this helps somewhat | 17:16 |
| harlowja_ | maybe public clouds disabled it | 17:16 |
| mordred | but why it would ever be configured NOT to is insane | 17:16 |
| pleia2 | cinerama: (as much as I *love* our hacky vagrant/cloud testing right now ;)) | 17:16 |
| *** tonytan4ever has quit IRC | 17:16 | |
| *** dimtruck is now known as zz_dimtruck | 17:16 | |
| mordred | and once again, a good idea being made useless by vendors | 17:16 |
| *** timcline has joined #openstack-infra | 17:16 | |
| SpamapS | ./cloud.cfg.d/90_dpkg.cfg:datasource_list: [ NoCloud, ConfigDrive, OpenNebula, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, Ec2, CloudStack, None ] | 17:16 |
| harlowja_ | mordred do u need a hug | 17:16 |
| SpamapS | mordred: ^^ | 17:16 |
| clarkb | harlowja_: I would accept working cloud | 17:17 |
| harlowja_ | lol | 17:17 |
| *** claudiub has quit IRC | 17:17 | |
| harlowja_ | just a hug, jeez | 17:17 |
| mordred | I'm rebuilding devstack-trusty on hcloud-b5 again | 17:17 |
| SpamapS | mordred: thats from my instance on HPCloud East btw. | 17:17 |
| harlowja_ | stop being so greedy, lol | 17:17 |
| *** Bsony has quit IRC | 17:17 | |
| jeblair | i do not expect working cloud so will cut my losses and accept a hug | 17:17 |
| mordred | SpamapS: have you tried booting an instance in hpcloud with config drive enabled and then logging in to see if it's mounted? | 17:17 |
| mordred | jeblair: ++ | 17:17 |
| harlowja_ | so then u may need to look @ /var//log/cloud-init.log | 17:17 |
| SpamapS | mordred: I ran the 'nova boot' command for that about 30s ago. ;) | 17:18 |
| mordred | harlowja_: that's hard to do if cloud-init has an error and you can't log in :) | 17:18 |
| mordred | SpamapS: and config drive is mounted? | 17:18 |
| SpamapS | mordred: it's not booted yet | 17:18 |
| harlowja_ | mordred agreed | 17:18 |
| mordred | oh | 17:18 |
| *** Hal2 has quit IRC | 17:18 | |
| mordred | SpamapS: if it works for you, you and I are going to sit in a dark room and figure out why | 17:18 |
| mordred | because it's never worked for me :) | 17:18 |
| harlowja_ | always worked for me, in fact all yahoo vms on openstack are using it, lol | 17:19 |
| harlowja_ | but i can help trouble shoot it (since i'm one of the creators, lol) | 17:19 |
| *** otter768 has joined #openstack-infra | 17:19 | |
| mordred | heh | 17:19 |
| *** Bsony has joined #openstack-infra | 17:19 | |
| harlowja_ | http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceConfigDrive.py#L73 (the magical mounting code) | 17:19 |
| clarkb | harlowja_: what version is that first in? | 17:20 |
| fungi | harlowja_: i wonder if ubuntu 12.04 cloud-init is too old | 17:20 |
| harlowja_ | hmmm | 17:20 |
| harlowja_ | there should be a version of it in that | 17:20 |
| fungi | 0.6.3 | 17:20 |
| * SpamapS drums fingers while instance schedules | 17:20 | |
| mordred | SpamapS: good luck with that | 17:20 |
| harlowja_ | let me download 0.6.3; forgot what was in it | 17:20 |
| SpamapS | at this point nova boot hasn't even come back | 17:21 |
| SpamapS | ahh, only 90s to respond | 17:21 |
| *** ashleighfarnham has joined #openstack-infra | 17:21 | |
| *** patrickeast has joined #openstack-infra | 17:21 | |
| *** xyang1 has joined #openstack-infra | 17:21 | |
| fungi | SpamapS: welcome to hpcloud! 90s is a good response time from what i've seen | 17:22 |
| harlowja_ | so 0.6.3 is pretty damn old | 17:22 |
| harlowja_ | it may/may not work correctly, i don't think it has support for what nova is writing into it (but i'm not 100% sure) | 17:23 |
| harlowja_ | it has the old config-drive v1 code | 17:23 |
| fungi | harlowja_: ubuntu 14.04 (the other ubuntu we run) has 0.7.5 at least | 17:23 |
| harlowja_ | yup | 17:23 |
| *** otter768 has quit IRC | 17:23 | |
| SpamapS | Cloud-init v. 0.7.5 finished at Thu, 19 Mar 2015 17:22:21 +0000. Datasource DataSourceConfigDriveNet [net,ver=2][source=/dev/sr0]. Up 50.63 seconds | 17:24 |
| SpamapS | mordred: ^ | 17:24 |
| harlowja_ | 0.6.3 was before my huge refactor | 17:24 |
| harlowja_ | 0.7.0 include josh super sauce | 17:24 |
| harlowja_ | 0.7.0+ | 17:24 |
| harlowja_ | (which is why there is no 0.6.4, lol) | 17:24 |
| * SpamapS watches harlowja_ flex | 17:24 | |
| clarkb | harlowja_: ok so its likely will not work for us | 17:24 |
| *** bhuvan has quit IRC | 17:24 | |
| harlowja_ | SpamapS oh ya | 17:25 |
| harlowja_ | lol | 17:25 |
| harlowja_ | i've seen people ask for it to be backported; but i don't think thats how canonical works | 17:25 |
| SpamapS | mordred: 2015-03-19 17:21:40,798 - util.py[DEBUG]: Running command ['mount', '-o', 'ro,sync', '/dev/sr0', '/tmp/tmpVPaVgX'] with allowed return codes [0] (shell=False, capture=True) | 17:25 |
| harlowja_ | amazing :-P | 17:25 |
| fungi | JayF: and centos 6 has cloud-init 0.7.5 as well (or 0.7.4 in epel) | 17:25 |
| fungi | er, harlowja_ ^ | 17:26 |
| fungi | sorry JayF | 17:26 |
| harlowja_ | ya | 17:26 |
| JayF | It's fine, It's good for me to know too | 17:26 |
| JayF | probably means I should bump our downstream patched versino of cloud-init as well, heh | 17:26 |
| SpamapS | mordred: so what you may not like, is that it just mounts it _for its purposes_ | 17:26 |
| SpamapS | mordred: but it leaves everything it found in /var/lib/cloud/instance | 17:26 |
| mordred | SpamapS: HEADDESK | 17:27 |
| mordred | SpamapS: STAB STAB STAB | 17:27 |
| harlowja_ | lol | 17:27 |
| SpamapS | so you have the entirety of the data there. | 17:27 |
| *** mwagner_lap has quit IRC | 17:27 | |
| harlowja_ | http://cloudinit.readthedocs.org/en/latest/topics/dir_layout.html | 17:27 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add tempest + postgres job for manila https://review.openstack.org/165935 | 17:27 |
| harlowja_ | and more.... | 17:27 |
| *** achanda has joined #openstack-infra | 17:28 | |
| SpamapS | mordred: pretty sure you just want to hate. Perhaps you should go to Belgium and help them hate the Flemmish instead? | 17:28 |
| fungi | okay, so cloud-init mounts it, copies the data to a known stable location in the root filesystem and then unmounts it again. that's not so bad | 17:28 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add tempest + postgres job for manila https://review.openstack.org/165935 | 17:28 |
| mordred | SpamapS: but I don't have it where config-drive | 17:28 |
| mordred | puts it | 17:28 |
| mordred | anyway | 17:28 |
| SpamapS | you do | 17:28 |
| harlowja_ | mordred hug, lol | 17:28 |
| SpamapS | its still there on /dev/sr0 | 17:28 |
| clarkb | SpamapS: that was a trusty node right? | 17:28 |
| *** achanda has quit IRC | 17:28 | |
| SpamapS | clarkb: aye | 17:28 |
| clarkb | SpamapS: we will need to check precise as well | 17:28 |
| *** ominakov_ has quit IRC | 17:28 | |
| SpamapS | Description:Ubuntu 14.04.1 LTS | 17:28 |
| *** xyang1 has quit IRC | 17:28 | |
| harlowja_ | http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceConfigDrive.py#L228 (not so magical find rountine) | 17:28 |
| fungi | maybe once we're able to stop supporting precise we can revisit | 17:28 |
| clarkb | since precise is older cloud init and we need precise | 17:28 |
| *** achanda has joined #openstack-infra | 17:28 | |
| SpamapS | nova boot --config-drive 1 --key-name default --image 9d25fe2d-cf31-4b05-8c58-f238ec78e633 test-cdrive --flavor 100 | 17:28 |
| clarkb | fungi: ya | 17:29 |
| SpamapS | in useast | 17:29 |
| *** xyang1 has joined #openstack-infra | 17:29 | |
| clarkb | SpamapS: how did you get that to work without network info? | 17:29 |
| SpamapS | clarkb: magic? | 17:29 |
| *** wuhg has quit IRC | 17:29 | |
| clarkb | SpamapS: unrelated its just my biggest annoyance booting nodes in hpcloud | 17:29 |
| SpamapS | I have a network already setup of course. | 17:29 |
| SpamapS | which is like o_O when you have to set it up | 17:30 |
| *** e0ne has quit IRC | 17:30 | |
| *** dtantsur is now known as dtantsur|afk | 17:30 | |
| clarkb | SpamapS: yes but even then you still have to provide a network to boot on | 17:30 |
| SpamapS | but once I made it, I only have one, so hpcloud attaches it | 17:30 |
| clarkb | SpamapS: then you have to attach floating ip to get into the node | 17:30 |
| *** ominakov has joined #openstack-infra | 17:30 | |
| SpamapS | clarkb: No it will attach all the networks you have access to | 17:30 |
| clarkb | oh ou have only one | 17:30 |
| SpamapS | or did they screw that up now? | 17:30 |
| clarkb | SpamapS: it errors for us an our many networks if you don't supply a specific network | 17:30 |
| SpamapS | is there a reason you have many networks? | 17:30 |
| *** mestery_ has joined #openstack-infra | 17:31 | |
| clarkb | SpamapS: yes, the router/networks can only handle about 100 nodes a piece | 17:31 |
| SpamapS | lurvely | 17:31 |
| fungi | SpamapS: neutron router instances don't scale well to our nova instance count | 17:31 |
| fungi | also even if they did, they're provisioned as a /24 | 17:31 |
| SpamapS | DVR should solve that | 17:31 |
| *** andreykurilin_ has joined #openstack-infra | 17:31 | |
| SpamapS | DVR won't solve that | 17:31 |
| fungi | so we couldn't put more than ~253 nodes on one | 17:31 |
| clarkb | fungi: ya we can make bigger networks it just wasn't required since we were told ~100 nodes per router | 17:31 |
| SpamapS | but you can make it a /8 if you want | 17:31 |
| fungi | oh, if we have control to push that lower than /24 then cool | 17:32 |
| SpamapS | I don't know if they rolled out DVR in the public cloud tho | 17:32 |
| clarkb | SpamapS: why would DVR solve the scaling issue? | 17:32 |
| fungi | thought they'd imposed a hard limit there but i must have misremembered | 17:32 |
| clarkb | SpamapS: aiui DVR is more similar to nova net multihost | 17:32 |
| *** _nadya_ has quit IRC | 17:32 | |
| clarkb | SpamapS: which is more about redundancy than performance | 17:32 |
| clarkb | (though you get some performance improvment because you are partitioned) | 17:32 |
| fungi | jenkins04 is now looking good and happy | 17:33 |
| *** mestery has quit IRC | 17:33 | |
| *** _nadya_ has joined #openstack-infra | 17:33 | |
| *** coolsvap is now known as coolsvap|afk | 17:33 | |
| SpamapS | clarkb: IIRC, east/west gets faster, north/south still has to go through l3 agent. | 17:33 |
| clarkb | ugh people need to stop using those terms | 17:33 |
| SpamapS | because you don't like them or because they're wrong? | 17:34 |
| clarkb | SpamapS: because the are non descriptive and tell me little | 17:34 |
| clarkb | networking has nothing to do with compass direction | 17:34 |
| SpamapS | clarkb: it has to do with the most common network layout diagram | 17:34 |
| fungi | east is east of west right? though i guess east is also west of west. damn you spherical coordinate system | 17:34 |
| SpamapS | north == out, south == in, east/west == amongst peers | 17:35 |
| *** packet has joined #openstack-infra | 17:35 | |
| clarkb | SpamapS: and for our use case that won't help anyting since we re prodominently pushing packets off network | 17:35 |
| fungi | that's such a marvellous example of cargo culting terminology. almost as great as the origin of "cloud" | 17:35 |
| clarkb | jenkins <-> slave <-> mirrors | 17:35 |
| SpamapS | fungi: In this particular diagram east of east thar be dragons. | 17:36 |
| jroll | I need a gif of 1s and 0s flying in random directions | 17:36 |
| fungi | SpamapS: ooh! hplcoud is deployed on a flat earth! | 17:36 |
| SpamapS | clarkb: yeah, agree. | 17:36 |
| *** ildikov has joined #openstack-infra | 17:36 | |
| *** markus_z has quit IRC | 17:36 | |
| SpamapS | fungi: explains a lot | 17:36 |
| fungi | now i have proof for my spherical fallacy conspiracy | 17:36 |
| *** ominakov_ has joined #openstack-infra | 17:36 | |
| *** ominakov has quit IRC | 17:36 | |
| *** claudiub has joined #openstack-infra | 17:37 | |
| *** amitgandhinz has joined #openstack-infra | 17:37 | |
| *** andreykurilin__ has joined #openstack-infra | 17:37 | |
| *** andreykurilin_ has quit IRC | 17:37 | |
| *** xyang1 has quit IRC | 17:37 | |
| greghaynes | The real question is how the east/west map into toroid networks | 17:37 |
| SpamapS | greghaynes: mostly with a hammer to your head and some shrooms | 17:38 |
| clarkb | SpamapS: also why not describe it as l2 vs l3 communication? | 17:38 |
| greghaynes | clarkb: its not | 17:39 |
| clarkb | in any case I should go back to poking at jenkinses | 17:39 |
| greghaynes | its about moving up the tree | 17:39 |
| SpamapS | clarkb: because east/west can describe L3 segment hops too. | 17:39 |
| *** garyh has quit IRC | 17:39 | |
| *** marcusvrn has quit IRC | 17:39 | |
| SpamapS | clarkb: in the DVR case, if you're just sending to another 10.x range DVR has that route and doesn't need l3 agent, it just sends. | 17:39 |
| clarkb | SpamapS: what about the term "east/west" describes any of that? | 17:39 |
| *** mestery_ is now known as mestery | 17:39 | |
| clarkb | anyways I am just grumping, I find the terms to be silly as they do not describe what is going on in a meaningful way | 17:40 |
| *** ominakov_ has quit IRC | 17:40 | |
| clarkb | fungi: jenkins01 was restarted too right? | 17:40 |
| *** bhuvan has joined #openstack-infra | 17:41 | |
| fungi | clarkb: yep | 17:41 |
| SpamapS | clarkb: east/west is generally used to mean "doesn't go through the thing that connects north/south" .. In a big network, you'll have L2 segments behind L3 routers linked together as peers and then north of those the uplinks to the internet/WAN. | 17:41 |
| SpamapS | clarkb: I don't know a better short hand for this, but I'm open to suggestions. | 17:42 |
| clarkb | fungi: https://jenkins01.openstack.org/monitoring?part=graph&graph=threadCount it also leaked threads but isn't running the latest ssh-slaves plugin | 17:42 |
| *** garyh has joined #openstack-infra | 17:42 | |
| *** amitgandhinz has quit IRC | 17:42 | |
| *** amitgandhinz has joined #openstack-infra | 17:43 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:43 | |
| clarkb | fungi: it did not leak as badly as 04 which is probably why you were able to more gracefully restart it | 17:43 |
| *** yamamoto has quit IRC | 17:44 | |
| fungi | clarkb: well, i caught 01 by manually looking for misbehaving servers. in the wake of 04 which mordred caught once it had fallen over to the point where it wasn't even responding to ssh any longer | 17:44 |
| fungi | so i have a feeling 01 would have gotten there itself in due time | 17:45 |
| clarkb | so, we can update ssh-slaves plugin everywhere simply because it appears to help but not fix the problem (05, 02 and 07 haven't leaked yet) but not sure what causes the leaks so that may just be wishful thinking | 17:45 |
| *** ivar-lazzaro has quit IRC | 17:45 | |
| clarkb | fungi: ya, the leak does not seem to be cleaned up | 17:45 |
| clarkb | fungi: I tested that on 05 by leaving it in shutdown mode for a day a few weeks back and the thread count did not fall | 17:46 |
| fungi | fun | 17:46 |
| clarkb | mordred: how is that image build going? | 17:46 |
| *** andreykurilin__ has quit IRC | 17:46 | |
| clarkb | zaro: ^ any other ideas on the thread leak? | 17:46 |
| clarkb | zaro: that patch to the ssh-slaves plugin looked so promising | 17:46 |
| *** tonytan4ever has joined #openstack-infra | 17:47 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:47 | |
| *** crc32 has joined #openstack-infra | 17:47 | |
| *** hyakuhei has quit IRC | 17:49 | |
| mordred | clarkb: reading logs | 17:50 |
| *** ayoung has joined #openstack-infra | 17:50 | |
| mordred | Exception: DIB failed creating /opt/nodepool_dib/devstack-trusty-1426785436.qcow2 | 17:51 |
| *** tonytan4ever has quit IRC | 17:51 | |
| *** dustins_ has quit IRC | 17:52 | |
| *** zz_dimtruck is now known as dimtruck | 17:52 | |
| *** Somay has quit IRC | 17:53 | |
| *** ghostpl_ has quit IRC | 17:53 | |
| *** Somay has joined #openstack-infra | 17:53 | |
| mordred | clarkb: except it seems to have succeeded | 17:54 |
| clarkb | mordred: that doesn't end up in the image.log for me to read right? because it was a command not the daemon | 17:55 |
| *** SumitNaiksatam has joined #openstack-infra | 17:55 | |
| *** HeOS has quit IRC | 17:56 | |
| mordred | yah | 17:56 |
| mordred | 2015-03-19 17:41:15,623 INFO nodepool.DiskImageUpdater: Image devstack-trusty-1426786672.template.openstack.org in hpcloud-b5 is ready | 17:56 |
| *** tonytan4ever has joined #openstack-infra | 17:57 | |
| mordred | clarkb: http://paste.openstack.org/show/193643/ | 17:58 |
| sdague | fyi, the ubuntu ssl package updates seem to have just surfaced in the mirrors | 17:58 |
| clarkb | mordred: so it uploaded the previous image I think | 17:59 |
| clarkb | mordred: I would do image-build not image-update now | 17:59 |
| clarkb | mordred: but look further back for where dib failed | 17:59 |
| *** reed has quit IRC | 17:59 | |
| mordred | clarkb: look at the top of that | 18:00 |
| greghaynes | mordred: that looks like dib cleanup at the top | 18:00 |
| mordred | clarkb: oh - wait - no, not the rm -rf | 18:00 |
| mordred | 2015-03-19 17:37:51,310 INFO nodepool.image.build.devstack-trusty: qemu-img: error while compressing sector 13184128: No space left on device | 18:00 |
| mordred | there it is | 18:00 |
| clarkb | mordred: you need to go back, this is one problem with dib, something will fail about 1000 log lines earlier before you get the exit one | 18:00 |
| clarkb | mordred: :( | 18:00 |
| greghaynes | thar ya go | 18:00 |
| *** jlanoux has quit IRC | 18:00 | |
| mordred | /dev/xvde2 72G 65G 3.2G 96% /opt | 18:01 |
| mordred | greghaynes: I havea bunch of devstack-fedora21-dib-1426774571.d directories | 18:02 |
| mordred | those are all leftovers from failed buidls, right? | 18:02 |
| clarkb | mordred: those are fallout from f21 image builds failing Ithink | 18:02 |
| clarkb | dib doesn't cleanup those dirs after failure | 18:02 |
| *** bhunter71 has quit IRC | 18:02 | |
| *** boris-42 has quit IRC | 18:02 | |
| greghaynes | in what locatoin? | 18:02 |
| clarkb | but they are small | 18:02 |
| clarkb | greghaynes: in the image output dir | 18:02 |
| clarkb | mordred: I think adding the f21 image pushed our little disk over the limit | 18:03 |
| fungi | sdague: thanks for the heads up. i poured over the list of vulnerabilities today and it's at worst remote denial of service for some ssl-enabled services. i'm okay with letting them get applied through our usual automatic channels | 18:03 |
| greghaynes | ah, youre making unique image names | 18:03 |
| *** ZZelle_ has joined #openstack-infra | 18:03 | |
| clarkb | greghaynes: yes | 18:03 |
| greghaynes | yes, those arent cleaned | 18:03 |
| SpamapS | python needs a mode where it ignores any .pyc's that don't have their supporting .py | 18:03 |
| *** _nadya_ has quit IRC | 18:03 | |
| *** enikanorov has joined #openstack-infra | 18:03 | |
| greghaynes | I dont see it because I tend to reuse the same image name | 18:03 |
| greghaynes | I think its just manifests | 18:03 |
| clarkb | greghaynes: yup | 18:03 |
| SpamapS | The amount of time I've spent this week chasing ghosts then running 'find . -name '*.pyc' -print0 | xargs -0 rm' is stupid | 18:03 |
| clarkb | its not a significant amount of disk space | 18:04 |
| clarkb | mordred: we can probably get rid of the precise-dib and trusty-dib images | 18:04 |
| greghaynes | huh, looks like for some reason the dpkg element depends on manifests | 18:05 |
| mordred | clarkb: kk | 18:05 |
| greghaynes | that should probably just be removed | 18:05 |
| clarkb | mordred: I will nodepool image-delete the two older ones of that image | 18:05 |
| clarkb | mordred: that should give you enough to get a new image for now | 18:05 |
| clarkb | mordred: also you didn't sudo -H -u nodepool so you will need to fix the dib cache and all that | 18:06 |
| *** enikanorov__ has quit IRC | 18:06 | |
| SpamapS | greghaynes: probably just a bug that got missed. | 18:06 |
| mordred | I will now sudo -H -u nodepool | 18:06 |
| clarkb | mordred: first you need to fix it | 18:06 |
| *** gyee has joined #openstack-infra | 18:06 | |
| clarkb | mordred: I need to remember what the fix is | 18:06 |
| mordred | clarkb: its chown -R nodepool:nodepool /opt/dib_cache now? | 18:06 |
| mordred | no? | 18:06 |
| clarkb | mordred: ya that looks right | 18:07 |
| mordred | doing that | 18:07 |
| clarkb | mordred: also chown the -new or maybe rm that image in /opt/nodepool_dib | 18:07 |
| clarkb | mordred: I think that is your failed image | 18:07 |
| SpamapS | greghaynes: 4c467964 .. probably just need to drop a file in environment.d for manifests, and then make dpkg inspect whether it is set or not. | 18:08 |
| greghaynes | SpamapS: yep | 18:08 |
| SpamapS | looks like source-repositories also deps on it but could use that method too | 18:08 |
| * SpamapS was never really fully onboard with the manifests plan but wanted to get out of their way because it seemed important to the people working on it. | 18:09 | |
| clarkb | mordred: you have 16GB of disk, should be enough to build that image | 18:09 |
| greghaynes | Yea, its hard to tell a user with a clear use case no | 18:09 |
| clarkb | mordred: I ran sudo -H -u nodepool nodepool dib-image-delete devstack-*-dib essentially | 18:09 |
| *** sdake_ has quit IRC | 18:09 | |
| *** sdake has joined #openstack-infra | 18:11 | |
| *** coolsvap|afk has quit IRC | 18:11 | |
| *** crc32 has quit IRC | 18:11 | |
| fungi | we've had a steady (and steep) climb in building nodes corresponding to clearing up tons of delete nodes. not sure if this is something we've triggered or something improving in hpcloud | 18:12 |
| clarkb | fungi: I don't think we have any of the fixes in yet other than having unstuck some jenkins | 18:13 |
| fungi | hrm, or things getting worse in hpcloud? we have 125 nodes building for >1 hour | 18:14 |
| fungi | looks like maybe this is still the replenishment from the nodepool restart | 18:14 |
| jeblair | clarkb, fungi: unsticking jenkins actually helped a bit -- i believe when jenkins is very slow to respond, it slows down the state conversions of servers enough to have an effect | 18:15 |
| mordred | clarkb: rebuilding | 18:15 |
| fungi | jeblair: ahh, yep that would make sense | 18:15 |
| jeblair | clarkb, fungi: also, if it's slow enough to error out, that can actually kill a build that otherwise worked | 18:15 |
| *** dustins has joined #openstack-infra | 18:15 | |
| jeblair | clarkb, fungi: so, for example, once jenkins01 was shut down, there was an immediate uptick of about 50 ready nodes | 18:15 |
| clarkb | jeblair: gotcha | 18:16 |
| fungi | it does look like the current building hill is peaking, so maybe we'll see lots more used nodes here shortly | 18:16 |
| *** hodos has joined #openstack-infra | 18:16 | |
| dims | anteaya: still see comments from that id we banned this AM - https://review.openstack.org/#/c/165947/ | 18:17 |
| *** bhunter71 has joined #openstack-infra | 18:17 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Remove check-tempest-dsvm-f20 https://review.openstack.org/165532 | 18:17 |
| clarkb | dims: its cached, you have to wait for that to get invalidated | 18:17 |
| dims | clarkb: ack thanks | 18:17 |
| *** ddieterly has quit IRC | 18:17 | |
| fungi | i don't recall if anyone got around to disabling that account yet | 18:17 |
| clarkb | fungi: I think jeblair did it | 18:18 |
| fungi | k | 18:18 |
| clarkb | but the cache affects disabled accounts too | 18:18 |
| fungi | yep | 18:18 |
| *** ddieterly has joined #openstack-infra | 18:19 | |
| *** cdent has quit IRC | 18:19 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 18:19 | |
| asselin | anyone know how to get the apikey forthe jenkin's user credintials-id? | 18:20 |
| clarkb | asselin: login as that user, go to settings page click button to get api key | 18:21 |
| clarkb | asselin: I think you have to do that at least once but from that point on the key is available on disk in $JENKINSHOME/users/username/config.xml or something like that | 18:21 |
| asselin | clarkb, yeah...in that file it's ecrypted...but then there's no credentails-id | 18:22 |
| mordred | 2015-03-19 18:21:08,826 INFO nodepool.image.build.devstack-trusty: Downloading pandas-0.15.2.tar.gz (4.6MB) | 18:22 |
| *** jordanP has quit IRC | 18:22 | |
| mordred | I just think that's a funny line | 18:22 |
| clarkb | asselin: not sure then | 18:23 |
| clarkb | asselin: but I think you need to request credentials before they have n id | 18:23 |
| clarkb | asselin: so try getting the api key if you haven't yet | 18:23 |
| *** Somay has quit IRC | 18:23 | |
| asselin | clarkb, yeah I crated a new user, got the api key, but the're no credentials-id....seems they moved stuff around in the latest version of jenkins....I will poke a bit more | 18:24 |
| anteaya | dims: thank you | 18:24 |
| anteaya | clarkb: did we want to clear that cache? | 18:25 |
| SpamapS | mordred: It's been years since I downloaded pandas | 18:25 |
| clarkb | anteaya: it clears on its own | 18:25 |
| anteaya | how long does it take? | 18:25 |
| sdague | anteaya: have you banned "Evgeny Antyshev" yet? | 18:25 |
| clarkb | anteaya: I am unsure | 18:25 |
| sdague | https://review.openstack.org/#/c/165953/ - they are commenting on *every* nova patch atm | 18:25 |
| anteaya | sdague: yes, but with open connections we have to either clear the cache or wait for it to expire | 18:25 |
| clarkb | ldap groups has a cache of 1 hour, web sessions has a cache of 12 hours | 18:26 |
| sdague | any idea who they work for? | 18:26 |
| clarkb | I don't think this will be in ldap groups | 18:26 |
| anteaya | we disalbed them more than an hour ago | 18:26 |
| anteaya | sdague: parallels.com? | 18:27 |
| *** ghostpl_ has joined #openstack-infra | 18:27 | |
| clarkb | I don't think its a web session cache either since comments should come via ssh? | 18:27 |
| anteaya | we disabled them 3 hours ago | 18:27 |
| *** AJaeger_ has joined #openstack-infra | 18:28 | |
| *** reed has joined #openstack-infra | 18:29 | |
| anteaya | I'm not seeing that they created a new account | 18:29 |
| AJaeger_ | pleia2: We need the zanata client installed on systems that run the post translation jobs and the translation proposal | 18:30 |
| *** spzala has quit IRC | 18:30 | |
| asselin | clarkb, I got it thanks | 18:30 |
| anteaya | and nodes in building state just went up over 450 | 18:30 |
| clarkb | show-caches makes me think it is the accounts cache but I can't find docs on the timeout for that | 18:30 |
| AJaeger_ | fungi: You just answered my question on https://review.openstack.org/165661 - but I'm still confused | 18:30 |
| *** emagana has quit IRC | 18:31 | |
| anteaya | clarkb: can we clear the cache? | 18:31 |
| AJaeger_ | Wouldn't a normal merge have already updated the content and the release job do it again? | 18:31 |
| clarkb | anteaya: we can, is this super urgent and letting the cache invalidate itself isn't cutting it? | 18:31 |
| clarkb | (I don't know how often this user is commenting) | 18:31 |
| anteaya | sdague said it is every nova patch | 18:32 |
| pleia2 | AJaeger_: AJaeger_ so translations proposal is the one that queries transifex/zanata to look for 75%+ translations and proposes them to gerrit, what are the post translations jobs? | 18:32 |
| fungi | AJaeger_: updating the _version_ if embedded in the document | 18:32 |
| *** emagana has joined #openstack-infra | 18:32 | |
| fungi | AJaeger_: as in release version number of the repo | 18:32 |
| sdague | clarkb: it's just really misleading as we are now getting "This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset." | 18:32 |
| AJaeger_ | fungi: Ah, so we would have an update after the merge, than one with new version after a tag - and another one with the next change. | 18:33 |
| sdague | from their poorly configured zuul | 18:33 |
| AJaeger_ | So, if merges are rare, it's worth doing it both times - or what is the reason here? | 18:33 |
| clarkb | the flush-caches command is running now | 18:34 |
| anteaya | clarkb: thank you | 18:34 |
| AJaeger_ | pleia2: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n385 and http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n392 - the upstream jobs | 18:34 |
| fungi | AJaeger_: it totally depends on the merge and release cadence of a given project. since i'm not intimately familiar with this one i simply proposed it do both. the extra run when the repo is tagged is not a lot of overhead | 18:34 |
| AJaeger_ | pleia2: so, this is the push to transifex/zanata | 18:35 |
| AJaeger_ | pleia2: while the proposal is the other way round | 18:35 |
| AJaeger_ | fungi: Ok, understood. Thanks for the explanation | 18:35 |
| clarkb | mordred: can the following files in /opt/nodepool_dib be deleted: devstack-fedora21-dib-1426774571.qcow2-new devstack-trusty-1426785436.qcow2-new devstack-trusty-dib-1426774571.qcow2-new ? | 18:35 |
| pleia2 | AJaeger_: ah, gotcha, I'll dig into which slaves these are | 18:35 |
| mordred | /opt/nodepool_dib/devstack-trusty-1426788873.qcow2-new is the current in-flight one | 18:36 |
| mordred | clarkb: ^^ | 18:36 |
| clarkb | sdague: anteaya dims caches are flushed, if you see comments after now then I would worry | 18:36 |
| anteaya | clarkb: thank you | 18:36 |
| sdague | clarkb: thanks | 18:36 |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Temporarily stop adding devstack-f20 nodes in rax" https://review.openstack.org/165621 | 18:37 |
| thingee | clarkb: hi, Phillip who you have helped in the past http://lists.openstack.org/pipermail/openstack-dev/2015-February/057585.html is claiming that his CI is waiting on some changes from you. I'm just verifying if you have any idea what he's talking about so I can take into consideration with Cinder's CI deadline for drivers. | 18:37 |
| jeblair | fungi: oh, did rax fix the problem ^ ? | 18:37 |
| clarkb | thingee: uh | 18:37 |
| fungi | i'll keep an eye on ^ and make sure the new images there are working with ianw's patch in place | 18:38 |
| fungi | jeblair: in theory, yes | 18:38 |
| clarkb | thingee: if there is a change up to propose the job to run on one node then I guess they would be waiting on us | 18:38 |
| clarkb | thingee: otherwise no I don't think there is anything preventing ^ | 18:38 |
| thingee | clarkb: ok, is that up for review? Sorry would check myself but context switching a bit at the moment. | 18:38 |
| fungi | jeblair: patch to install_puppet.sh at https://review.openstack.org/165681 | 18:38 |
| clarkb | thingee: my suggestion was to work on making the simple thing go before worrying about multi node and all that | 18:38 |
| clarkb | thingee: I don't think the simple thing has been proposed but I haven't checked recently | 18:39 |
| thingee | clarkb: ok, thank you | 18:39 |
| fungi | jeblair: hopefully we can clean that back up as soon as we stop using f20 | 18:39 |
| clarkb | thingee: I would ask them | 18:39 |
| jeblair | fungi: oh ok | 18:39 |
| *** ominakov has joined #openstack-infra | 18:39 | |
| fungi | which i assume is not too far away given the testing going on with f21 currently | 18:40 |
| clarkb | thingee: see also http://lists.openstack.org/pipermail/openstack-dev/2015-February/057588.html | 18:41 |
| mordred | clarkb: I believe it worked | 18:41 |
| mordred | 2015-03-19 18:40:28,630 INFO nodepool.DiskImageUpdater: Image devstack-trusty-1426790224.template.openstack.org in hpcloud-b5 is ready | 18:41 |
| clarkb | thingee: jeblair has a note on the simplest and quickest way to move forward | 18:41 |
| mordred | 2015-03-19 18:37:04,903 INFO nodepool.DiskImageUpdater: Uploading dib image id: 11617 from /opt/nodepool_dib/devstack-trusty-1426788873.qcow2 for devstack-trusty-1426790224.template.openstack.org in hpcloud-b5 | 18:41 |
| clarkb | thingee: that is what I think has not happened | 18:41 |
| *** hashar has joined #openstack-infra | 18:41 | |
| clarkb | mordred: yay now we wait for one of those to boot successfully | 18:42 |
| *** hdd has quit IRC | 18:42 | |
| mordred | so - hpcloud-b5 is region to watch | 18:42 |
| clarkb | mordred: and if so you want to do `sudo -H -u nodepool nodepool image-upload hpcloud-bX devstack-trusty` | 18:42 |
| *** dimtruck is now known as zz_dimtruck | 18:42 | |
| jeblair | thingee: i don't see a pending project config change from someone named Philip | 18:42 |
| mordred | yup | 18:43 |
| dims | clarkb: thanks! | 18:43 |
| jeblair | mordred, clarkb, fungi: fyi up to 625 instances of eoferror, so it's still been happening (so hopefully the new cloudinitless image will still help current situation) | 18:44 |
| *** yamamoto has joined #openstack-infra | 18:44 | |
| fungi | fingers crossed | 18:44 |
| mordred | jeblair: it's good that it's still happening, so that we can see if the patch helped | 18:44 |
| clarkb | mordred: I have cleaned up those -new images that are not the one you just built | 18:45 |
| clarkb | mordred: that gives us a total of 22GB free disk after I also removed the older precise-dib, trusty-dib and centos7-dib images | 18:45 |
| clarkb | mordred: but basically we need more disk | 18:45 |
| clarkb | mordred: or fewer images | 18:46 |
| * fungi is trying to make traction on the fewer images idea | 18:46 | |
| mordred | clarkb: fewer images | 18:46 |
| *** claudiub has quit IRC | 18:46 | |
| clarkb | or smaller images | 18:46 |
| clarkb | or all of the above :) | 18:46 |
| fungi | speaking of fewer images, nova py27 job on devstack-trusty node using my experimental job which prepped the db and installed the distro packages needed http://logs.openstack.org/44/164444/1/experimental/experimental-nova-dsvm-python27/f4d9841/console.html | 18:47 |
| fungi | log confirms migration tests ran | 18:47 |
| clarkb | nice | 18:47 |
| clarkb | unfortunately since we use snapshots for all bare-* nodes ths doesn't quite help us :) | 18:48 |
| fungi | some eyeballs on that for obvious issues would be appreciated once things cool down | 18:48 |
| clarkb | at least not with the ENODISK problem | 18:48 |
| *** tkelsey has quit IRC | 18:48 | |
| *** yfried has joined #openstack-infra | 18:48 | |
| *** yamamoto has quit IRC | 18:49 | |
| fungi | well, if we switched the tox builder to always run the database prep builders, we could drop mysql and postgres setup from our image build scripts and switch bare-.* to dib sooner | 18:49 |
| clarkb | the back button doesn't work on jenkins' jira | 18:49 |
| clarkb | fungi: yup | 18:50 |
| *** SotK has quit IRC | 18:50 | |
| fungi | which is still a reasonable stepping-stone to where i want to get with bindep used for defining the additional packages to cache on the devstack images and install at the start of jobs | 18:50 |
| clarkb | "If this is a CloudBees customer we have two hotfixes that seem to work around the deadlock with the side-effect of degrading UI performance" | 18:50 |
| fungi | clarkb: open core! | 18:51 |
| * greghaynes is curious if anyone is up for explaining how bindep gets us a single image vs many (that is the goal, right?) | 18:51 | |
| clarkb | I am not going to login to respond to that but figured I should point out that this goes on on their tracker | 18:51 |
| clarkb | greghaynes: single image per distro because every job can describe its system deps | 18:51 |
| greghaynes | oh! so the bindep will be at runtime | 18:51 |
| clarkb | yes | 18:52 |
| greghaynes | and then we can remove those deps out from buildtime | 18:52 |
| greghaynes | make sense | 18:52 |
| *** spzala has joined #openstack-infra | 18:52 | |
| fungi | greghaynes: it _eventually_ enables us to start putting lists of required distro packages in individual repos, allowing us to self-test whether they're sufficient too | 18:52 |
| clarkb | with lots of caching in the image build still | 18:52 |
| greghaynes | yea, that makes a ton of sense all of a sudden ;) | 18:52 |
| fungi | right, we'd iterate over all branches of all repos in gerrit and pre-cache the union of all their bindep files | 18:52 |
| *** radez_g0n3 is now known as radez | 18:52 | |
| fungi | conveniently, we already clone those repos onto the images anyway | 18:53 |
| greghaynes | Is there any next to dos on that? | 18:53 |
| fungi | greghaynes: you can start by reviewing my changes to bindep and testing it out if you haven't already | 18:53 |
| reed | fungi, isn't there a backup-as-a-service for the machines managed by infra? (re: the question on the community list about askbot) | 18:54 |
| greghaynes | sounds good | 18:54 |
| *** yolanda has quit IRC | 18:54 | |
| fungi | reed: there is, but what evgeny's talking about is dumping copies of critical files and database contents as a snapshot onto the local filesystem | 18:54 |
| fungi | reed: then we can back that up to a remote server like we do for other critical systems, but the local dump is convenient for other reasons | 18:55 |
| clarkb | I am not finding any bugs on the jenkins tracker that matcht the behavior we see | 18:56 |
| fungi | greghaynes: i also have a bindep list mostly assembled that effectively turns our current devstack-.* workers into bare-.* workers https://review.openstack.org/164447 though that also needs some rigorous testing | 18:56 |
| clarkb | I will try to file one, pretty sure I have an account I can use | 18:56 |
| openstackgerrit | Adam Gandelman proposed openstack/requirements: Relax pysendfile requirement https://review.openstack.org/165969 | 18:56 |
| fungi | greghaynes: if nothing else, it's a good example of a complex-ish list | 18:56 |
| greghaynes | awesome, ill try it out | 18:57 |
| reed | fungi, ok | 18:57 |
| *** SotK has joined #openstack-infra | 18:58 | |
| clarkb | hrm doesn't look like the heap stack error dumps that I collected the first time around have stack traces | 18:58 |
| reed | fungi, I think Marton overlooked those | 18:58 |
| clarkb | well I will file what I have and give it a go | 18:58 |
| fungi | reed: they were non-obvious, but thankfully also something which can be implemented in parallel with testing the rest since they're non-service-impacting to add | 18:58 |
| mordred | clarkb: we still don't have a GREAT way of answering "what image did this instance boot from" do we? | 18:59 |
| clarkb | mordred: we do not | 18:59 |
| clarkb | mordred: what you can do is nova show the instance | 19:00 |
| clarkb | mordred: then take the image id from there and cross check against nodepoo | 19:00 |
| mordred | nod | 19:00 |
| clarkb | mordred: so its doable, but you have to query nova not just nodepool | 19:00 |
| jeblair | clarkb, mordred: i did an image-list, looked at the age of the most recent image, then found instances in build whose ages were less than that | 19:00 |
| jeblair | clarkb, mordred: that got me, among a few others, 1370425 | 19:00 |
| jeblair | which i'm watching | 19:01 |
| jeblair | and it's in error state. | 19:01 |
| fungi | mordred: or if you can ssh into it, you can see the image logged in /etc/nodepool but if the instance is erroring then not so much | 19:01 |
| jeblair | | fault | {"message": "No valid host was found. ", "code": 500, "created": "2015-03-19T18:56:46Z"} | | 19:01 |
| *** timcline has quit IRC | 19:01 | |
| jeblair | next is 1370426 | 19:01 |
| jeblair | | image | devstack-trusty-1426790224.template.openstack.org (026f353e-0365-4769-a5a8-910e57d454a8) | | 19:02 |
| jeblair | mordred: is that right ^ ? | 19:02 |
| mordred | yes. that's the one we want | 19:02 |
| jeblair | no floatig ip yet | 19:02 |
| jeblair | b5 has a task queue 85 deep, could be a few mins | 19:03 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add design and skeleton for an object interface https://review.openstack.org/164506 | 19:06 |
| jeblair | hrm. we should invert what the provider manager does -- so that the calls happen in the worker threads, but the provider manager serializes and paces them. | 19:07 |
| jeblair | that way when a deletion call takes 30 seconds, everything isn't waiting on it, but you still don't issue more than the target rate... | 19:07 |
| *** pelix has left #openstack-infra | 19:07 | |
| *** Hal1 has joined #openstack-infra | 19:08 | |
| jeblair | i wonder if openstack calculates rate based on when the request is received or completed? | 19:08 |
| *** claudiub has joined #openstack-infra | 19:08 | |
| *** Hal1 has quit IRC | 19:08 | |
| fungi | i wonder where that rate limiting is implemented | 19:08 |
| *** sushilkm has joined #openstack-infra | 19:08 | |
| fungi | nova? keystone? | 19:08 |
| *** Hal1 has joined #openstack-infra | 19:09 | |
| *** marun has quit IRC | 19:09 | |
| sushilkm | is there a way to rerun the tests in between ... because out of multiple jenkins job the smalles job has failed .... and bigger jobs would continue running and come back later saying for a recheck | 19:10 |
| anteaya | sushilkm: why have the jobs failed? | 19:10 |
| anteaya | sushilkm: there may be a problem with the patch | 19:10 |
| anteaya | in which case the jobs are working as expected | 19:10 |
| sushilkm | no problem is not with patch while verifying it worked, now there are same tests running for merge | 19:11 |
| sushilkm | and smallest test has failed | 19:11 |
| sushilkm | its a known bug | 19:11 |
| anteaya | which patch? | 19:11 |
| sushilkm | which comes up sometimes | 19:11 |
| sushilkm | its regarding a patch in trove | 19:11 |
| anteaya | well if your patch is triggering a known bug your patch may be making the bug worse | 19:11 |
| sushilkm | https://jenkins02.openstack.org/job/gate-trove-python27/2302/console | 19:12 |
| *** Hal2 has joined #openstack-infra | 19:12 | |
| *** Hal1 has quit IRC | 19:13 | |
| *** zz_dimtruck is now known as dimtruck | 19:13 | |
| anteaya | yeah patchset 2 failed 5 times, patchset 5 failed 3 times | 19:14 |
| anteaya | I think you need to consider the possiblity your patch may be increasing the tendency of a bug to appear | 19:14 |
| clarkb | https://issues.jenkins-ci.org/browse/JENKINS-27514 | 19:15 |
| fungi | clarkb: now you're famous | 19:15 |
| sushilkm | ok anteaya .. i wud have a look | 19:16 |
| sushilkm | thanks for the suggestion | 19:16 |
| clarkb | fungi: filing a bug is all it took to be famous? I should've done this ealier (I really thought that change I linked in the bug was the fix :( ) | 19:16 |
| fungi | you're in the who's who of jenkins jira bugs | 19:17 |
| pleia2 | hehe | 19:17 |
| fungi | maybe cloudbees will offer you an open-core hotfix | 19:18 |
| anteaya | ha ha ha | 19:18 |
| *** marun has joined #openstack-infra | 19:19 | |
| *** otter768 has joined #openstack-infra | 19:20 | |
| openstackgerrit | Khai Do proposed openstack-infra/gerritlib: Update project docs and setup for tests https://review.openstack.org/165979 | 19:20 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: DO NOT MERGE COPY ALL OF .TESTREPOSITORY to log server https://review.openstack.org/149719 | 19:20 |
| *** HeOS has joined #openstack-infra | 19:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Start using neutron for periodic stress job https://review.openstack.org/164705 | 19:22 |
| clarkb | mordred: any idea if your new image is working yet? | 19:24 |
| *** bdpayne has quit IRC | 19:24 | |
| *** garyh has quit IRC | 19:24 | |
| *** otter768 has quit IRC | 19:25 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Run tasks in worker threads https://review.openstack.org/165982 | 19:25 |
| cinerama | trying the new warfile in the puppet module | 19:25 |
| jeblair | clarkb, mordred: the test node i looked at is still in the nodepool queue | 19:25 |
| fungi | SlickNik: there are a couple changes in the gate for trove consistently failing python 2.7 unit tests. known issue? | 19:25 |
| jeblair | clarkb, mordred, fungi: which prompted me to write https://review.openstack.org/165982 | 19:25 |
| *** xyang1 has joined #openstack-infra | 19:26 | |
| pleia2 | cinerama: fingers crossed :) | 19:26 |
| cinerama | "we've secretly replaced the coffee here with folger's crystals. let's watch" | 19:26 |
| jeblair | clarkb, mordred, fungi: which is either going to be a _massive_ improvement to our throughput and efficiency, radically reducing the amount of servers we have sitting in build and delete | 19:26 |
| *** hdd has joined #openstack-infra | 19:26 | |
| clarkb | jeblair: looking at it now | 19:26 |
| jeblair | clarkb, mordred, fungi: or it could make everything blow up all at once | 19:26 |
| clarkb | jeblair: thankfully we have some tests now | 19:26 |
| jeblair | clarkb, mordred, fungi: frankly, a little hard to tell from where i'm sitting | 19:26 |
| SlickNik | fungi: funny you should mention that — that's what I'm looking at right now. | 19:26 |
| jeblair | clarkb: actually it passes tests :) | 19:26 |
| cinerama | oh pants, i forgot to update the template | 19:27 |
| fungi | SlickNik: okay, cool. just making sure it was on your radar | 19:27 |
| clarkb | jeblair: thats a start then :) | 19:27 |
| *** asettle has joined #openstack-infra | 19:27 | |
| pleia2 | cinerama: I haven't done a diff on the new config file, but I expect it to be unpleasant | 19:27 |
| jeblair | clarkb: i think the failure mode is likely to be that we find that "start of api request" is not how rate limits are calculated | 19:27 |
| jeblair | clarkb: i think the next most probably failure is "nodepool runs closer to specified api rate limit which is wrong and needs to be adjusted to slow it down a bit" | 19:28 |
| jeblair | that's pretty tractable | 19:28 |
| clarkb | jeblair: ya we should be able to handle that easily | 19:28 |
| *** bhunter71 has quit IRC | 19:28 | |
| fungi | there's a knob for that | 19:28 |
| *** asettle has quit IRC | 19:28 | |
| fungi | i'm cool with that risk | 19:28 |
| jeblair | i need to grab food; if people like that change, i can hand-apply it on nodepool and restart when i get back | 19:29 |
| *** bhunter71 has joined #openstack-infra | 19:29 | |
| anteaya | fix the world or blow it up, what's not to like | 19:29 |
| clarkb | jeblair: can you explain the changes to error handling? | 19:30 |
| fungi | the world's partway to blown up already, so not sure anyone will notice | 19:30 |
| cinerama | pleia2: it looks like we only need the infinispan bit (though the version is different). the other bit in the release notes refers to a one-off thing needed for migration from an older version so if we start on the new version we should be good | 19:30 |
| jeblair | clarkb: old error handling was to capture the exception in provider manager thread, and transfer it to calling thread | 19:30 |
| cinerama | now for some spaghetti testing = throw it against the wall & see if it sticks | 19:30 |
| clarkb | jeblair: I see | 19:30 |
| jeblair | clarkb: (i thought that was a neat trick ;) | 19:30 |
| clarkb | jeblair: but now with run happening in the calling thread it can bubble straight up | 19:31 |
| jeblair | clarkb: new change runs actual work in calling thread, so is simpler -- task.main() will just raise the exception in the right context | 19:31 |
| jeblair | yep | 19:31 |
| *** ddieterly has quit IRC | 19:31 | |
| *** ddieterl_ has joined #openstack-infra | 19:31 | |
| anteaya | fungi: good point | 19:32 |
| greghaynes | clarkb: it could before I think | 19:32 |
| *** pc_m has quit IRC | 19:32 | |
| clarkb | greghaynes: ya they bubbled up before via the transfer thing jeblair described | 19:32 |
| clarkb | but thats no longer necessary so we can rip it out | 19:32 |
| greghaynes | ah, sorry, I thought you were pointing that out as an issue not a statement of fact ;) | 19:33 |
| clarkb | greghaynes: ah, mostly just wrapping my head around the change | 19:33 |
| clarkb | jeblair: it lgtm I say go for it when you are done with lunch | 19:33 |
| greghaynes | I wonder if it passes my monkeypatch tests | 19:34 |
| greghaynes | I had to poke around a lot in providermanager for that | 19:34 |
| fungi | clarkb: jeblair: looks good to me too. wanting to hold off approving it until it's proven in production? | 19:36 |
| greghaynes | test in prod! | 19:36 |
| *** tjones1 has quit IRC | 19:37 | |
| fungi | greghaynes: well, nodepool _is_ already tested | 19:37 |
| *** SumitNaiksatam has quit IRC | 19:39 | |
| clarkb | it is afternoon already | 19:39 |
| fungi | yep, it's about time to rename the board ml | 19:40 |
| zaro | clarkb: sorry, haven't been paying attention. scp plugin is still leaking memory? | 19:40 |
| clarkb | zaro: no, the ssh-slaves plugin is leaking threads see https://issues.jenkins-ci.org/browse/JENKINS-27514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel | 19:40 |
| clarkb | zaro: this is the same issue from a few weeks ago, updating the ssh-slaves plugin does not fix it | 19:40 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack-infra/project-config: Add "heat-coe-templates" project https://review.openstack.org/164806 | 19:41 |
| *** sushilkm has left #openstack-infra | 19:41 | |
| *** garyh has joined #openstack-infra | 19:41 | |
| *** _nadya_ has joined #openstack-infra | 19:42 | |
| *** _nadya_ has quit IRC | 19:43 | |
| clarkb | SlickNik: looks like trove is still using probuscis | 19:43 |
| clarkb | SlickNik: looks like something database related though? | 19:45 |
| SlickNik | clarkb: yes for its legacy fake tests. We've moved all of the new unit tests to run under testr. | 19:45 |
| SlickNik | It's one of the new testr unit tests that is causing the issue. | 19:45 |
| clarkb | SlickNik: its failing in the old tests too | 19:45 |
| SlickNik | Specifically: FAIL - mgmt.test_models.TestMgmtInstanceDeleted.test_show_deleted_mgmt_instances | 19:46 |
| clarkb | oh wait those tracebacks from run_tests are normal? | 19:46 |
| clarkb | I should just stop looking | 19:46 |
| *** dprince has quit IRC | 19:47 | |
| SlickNik | clarkb: Yeah it sucks that the old tests puke stacktraces for the negative tests that are supposed to result in exceptions. There's a bug to fix that in the works. | 19:47 |
| *** sdake_ has joined #openstack-infra | 19:48 | |
| clarkb | SlickNik: you should be able to just wrap in expected exception context managers or asserts | 19:48 |
| Shrews | fungi: clarkb: When are the infra meetings held? | 19:50 |
| anteaya | tuesdays 1900 | 19:50 |
| Shrews | thx | 19:50 |
| anteaya | in #openstack-meeting | 19:50 |
| zaro | clarkb: so is it happening only on some servers and not others? | 19:50 |
| anteaya | thank you for asking, be nice to see you there | 19:50 |
| SlickNik | clarkb: yes we do. There was a change to the way that we patch eventlet for some of the fake tests that caused this to happen. I haven't dug deep enough into it yet to figure out why that was the case. | 19:50 |
| Shrews | anteaya: do you have a posted agenda? | 19:50 |
| anteaya | yes | 19:50 |
| * anteaya fetches | 19:50 | |
| clarkb | zaro: so far I think it has hit 01, 07, 02, 05, 04 | 19:50 |
| EmilienM | can I get a chance to get a +A on https://review.openstack.org/#/c/164451/ ? - it's about "zuul: add check-grenade-dsvm-neutron to tempest checks" | 19:51 |
| anteaya | Shrews: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting | 19:51 |
| Shrews | anteaya: thanks again | 19:51 |
| *** sdake has quit IRC | 19:51 | |
| anteaya | welcome | 19:51 |
| anteaya | EmilienM: not from me today, anything that isn't required for ff gets put off to tomorrow for me | 19:52 |
| anteaya | EmilienM: we are having difficulty getting resources for patches in check | 19:52 |
| zaro | clarkb: ooh, 5 out of 9? i thought it might be something different between the machines, like the underlying ssh library? | 19:52 |
| anteaya | don't want to add to that | 19:52 |
| fungi | Shrews: the agenda is usually in a mild state of limbo between things which were discussed in the previous meeting which need to be removed and things which have been added to discuss in the next meeting, but add whatever you want to talk about | 19:52 |
| clarkb | zaro: 5 out of 7 | 19:52 |
| EmilienM | anteaya: ack | 19:52 |
| anteaya | EmilienM: thanks | 19:53 |
| *** sdake has joined #openstack-infra | 19:53 | |
| *** [HeOS] has joined #openstack-infra | 19:53 | |
| *** sdake_ has quit IRC | 19:55 | |
| *** kgiusti has left #openstack-infra | 19:56 | |
| mestery | Hi infra folks! Looks like this patch (https://review.openstack.org/155470) has been stuck in the gate for 20 hours or so, any ideas what may be happening? /cc otherwiseguy | 19:56 |
| *** HeOS has quit IRC | 19:56 | |
| zaro | clarkb: gearman only offlines the node correct? not actually disconnects them. | 19:57 |
| mestery | the check-tempest-neutron-dsvm-dvr job appears to be just hung there | 19:57 |
| clarkb | zaro: correct, then nodepool deletes them | 19:57 |
| clarkb | mestery: that job was running on a jenkins master that fell over, you can push a new patchset to unstick it | 19:57 |
| mestery | armax: The stuck patch is yours, FYI | 19:57 |
| mestery | clarkb: Ack, armax can you do that? | 19:57 |
| mestery | clarkb: Thanks! | 19:57 |
| clarkb | a small number of changes were affected so rather than restart the world we decided that patch authors that are affected could address their specific changes | 19:58 |
| mestery | clarkb: Makes sense to me | 19:58 |
| *** eharney has quit IRC | 20:00 | |
| anteaya | mestery armax changing the commit message should be enough for zuul to consider that an event | 20:00 |
| *** teran has joined #openstack-infra | 20:01 | |
| mestery | thanks anteaya | 20:02 |
| *** timcline has joined #openstack-infra | 20:02 | |
| anteaya | mestery: you're welcome | 20:03 |
| clarkb | mordred: http://paste.openstack.org/show/193676/ still shows up in the console log so I don't think removing that package completely cleans out cloud init | 20:03 |
| clarkb | mordred: but it doesn't appear to look for metadata so at least that bit seems to work | 20:04 |
| anteaya | clarkb: can it find the keys? | 20:04 |
| *** mmedvede has joined #openstack-infra | 20:04 | |
| clarkb | anteaya: nodepool hasn't tried to do that yet | 20:04 |
| *** kgiusti has joined #openstack-infra | 20:04 | |
| anteaya | ah | 20:04 |
| clarkb | which is what jeblair's change from before should address | 20:04 |
| mordred | clarkb: does it have an ip address yet? | 20:05 |
| *** amuller has joined #openstack-infra | 20:06 | |
| anteaya | ah having the requests performed in the calling thread | 20:06 |
| clarkb | mordred: only on the 10 net | 20:06 |
| *** sambetts has quit IRC | 20:06 | |
| mordred | sigh | 20:06 |
| *** timcline has quit IRC | 20:06 | |
| mordred | clarkb: I really want to know why it's trying to do something with cloud-init | 20:06 |
| clarkb | mordred: I think thats an init script | 20:06 |
| *** sambetts has joined #openstack-infra | 20:06 | |
| clarkb | mordred: its lilely that init script isn't removed by the ensure => absent | 20:07 |
| zaro | clarkb: looks like there's a known issue with memory leak if too many slaves. https://issues.jenkins-ci.org/browse/JENKINS-20620 | 20:07 |
| *** shardy has quit IRC | 20:07 | |
| *** AJaeger_ has quit IRC | 20:07 | |
| zaro | how many slaves do we have setup for max? | 20:07 |
| mordred | clarkb: probably should have done => purged - does puppet support that? | 20:07 |
| clarkb | zaro: its about 200 per max | 20:07 |
| clarkb | mordred: I do not know | 20:07 |
| clarkb | zaro: but its not really a memory leak. The threads leak | 20:08 |
| clarkb | zaro: the threads are relatively light on memory | 20:08 |
| clarkb | zaro: but we only have like 100MB for threds to live in | 20:08 |
| mordred | clarkb: yp - that's in an init script | 20:08 |
| clarkb | so once we hit that we lose | 20:08 |
| *** e0ne has joined #openstack-infra | 20:09 | |
| *** amuller has left #openstack-infra | 20:10 | |
| *** dprince has joined #openstack-infra | 20:10 | |
| clarkb | mordred: 8b8f8f1b-e1bf-48f4-aae6-3b0e91d13978 is the node I looked at fwiw | 20:10 |
| clarkb | mordred: yes you can use purged | 20:11 |
| clarkb | mordred: https://docs.puppetlabs.com/references/latest/type.html#package-attribute-ensure | 20:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Purge cloud-init https://review.openstack.org/165996 | 20:11 |
| mordred | clarkb: I agree ^^ :) | 20:11 |
| *** kgiusti has left #openstack-infra | 20:12 | |
| clarkb | console log shows it started ssh too but without mention of generating new host keys | 20:13 |
| clarkb | lunch now, back in a bit | 20:13 |
| anteaya | enjoy food | 20:13 |
| pleia2 | eek lunch, it's late! | 20:14 |
| *** e0ne has quit IRC | 20:14 | |
| anteaya | happy food to you too | 20:14 |
| pleia2 | thanks :) | 20:15 |
| fungi | mordred: also http://admintweets.com/regenerate-ssh-host-keys-on-boot-on-ubuntu/ (looking now to confirm) | 20:15 |
| mordred | fungi: sigh | 20:16 |
| mordred | fungi: so I guess I probably want to do that then | 20:16 |
| fungi | it can't hurt, but it's got me curious because i thought most platforms took care of that | 20:16 |
| *** mrda-away is now known as mrda | 20:17 | |
| fungi | i'm testing with a held bare-trusty worker now | 20:18 |
| *** Hal2 has quit IRC | 20:19 | |
| *** Hal1 has joined #openstack-infra | 20:19 | |
| *** emagana has quit IRC | 20:20 | |
| fungi | mordred: yeah, at least for the bare-trusty worker i tested with in rax, it comes back up with sshd running but erroring on connect | 20:20 |
| mordred | fungi: k. I've got a patch coming | 20:20 |
| fungi | SpamapS: theory disproven :/ | 20:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Regenerate ssh host key on boot https://review.openstack.org/166000 | 20:21 |
| mordred | fungi: ^^ we can do that better - with a proper upstart script that goes after openssh server rather than in rc.local | 20:22 |
| mordred | fungi: but we have an rc.local we're installing anyway | 20:22 |
| anteaya | mordred: nice round number on that patch | 20:22 |
| mordred | woohoo! | 20:22 |
| anteaya | mordred: extra line in https://review.openstack.org/#/c/166000/1/nodepool/scripts/prepare_node.sh | 20:23 |
| * greghaynes points at his patch 100000 | 20:23 | |
| anteaya | greghaynes: I was just thinking of that | 20:23 |
| greghaynes | :) | 20:23 |
| anteaya | I remember the race for that number | 20:24 |
| anteaya | I had forgotten it was you that won the lottery | 20:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Regenerate ssh host key on boot https://review.openstack.org/166000 | 20:24 |
| greghaynes | Also, seems like 99-unbound is becoming the 'do everything at the end' script rather than 99-unbound | 20:24 |
| mordred | yeah | 20:24 |
| mordred | that whole stack needs some serious refactoring | 20:24 |
| *** _nadya_ has joined #openstack-infra | 20:24 | |
| *** _nadya_ has quit IRC | 20:24 | |
| fungi | mordred: testing that rc.local patch with another held trusty machine | 20:24 |
| mordred | fungi: woo | 20:25 |
| fungi | should know in a second when it finishes booting | 20:25 |
| *** emagana has joined #openstack-infra | 20:25 | |
| fungi | no dice. i'll try to step through what's wrong there | 20:25 |
| *** Swami has quit IRC | 20:26 | |
| *** salv-orlando has quit IRC | 20:27 | |
| *** tkelsey has joined #openstack-infra | 20:27 | |
| *** tjones1 has joined #openstack-infra | 20:27 | |
| fungi | mordred: i think your conditional is wrong | 20:27 |
| zaro | clarkb: have you read this? https://wiki.jenkins-ci.org/display/JENKINS/Remoting+issue | 20:27 |
| fungi | `type dpkg-reconfigure >/dev/null 2>&1 ! test -f /etc/ssh/ssh_host_rsa_keg` (checking for a nonexistent file) returns 1 for me there | 20:27 |
| jeblair | fungi, mordred: just to confirm -- you found a node with mordred's change, but it didn't work? | 20:28 |
| fungi | jeblair: no, i manually tried adding mordred's rc.local change to regenerate host keys to a held trusty node and then deleted its host keys and rebooted it | 20:28 |
| fungi | "functional test" | 20:28 |
| *** pritesh has joined #openstack-infra | 20:29 | |
| jeblair | fungi: oh ok. so you're pretty sure it wouldn't work. the node i was looking at failed the ssh test | 20:29 |
| jeblair | 2015-03-19 19:40:47,007 ERROR nodepool.NodeLauncher: EOFError launching node id: 1370426 in provider: hpcloud-b5 error: | 20:29 |
| jeblair | fungi: which i think may coroborrate your experiment | 20:29 |
| mordred | yah | 20:29 |
| jeblair | okay, so while mordred iterates, i'm going to apply nodepool patch and restart | 20:29 |
| mordred | wow | 20:29 |
| greghaynes | mordred: what is the type dpkg-reconfigure trying to do? just see if dpkg-reconfigure exists? | 20:29 |
| mordred | jeblair: fungi did I raelly do keg? | 20:30 |
| jeblair | oh noes! my patch failed pep8! | 20:30 |
| greghaynes | mordred: why not just go for it and || true, if it fails then it doesnt exist ;) | 20:30 |
| clarkb | zaro: I had not but the problem isn't connection issues | 20:30 |
| anteaya | jeblair: no | 20:30 |
| mordred | greghaynes: could do that too | 20:30 |
| *** marcusvrn1 has quit IRC | 20:30 | |
| clarkb | zaro: the problem is that jenkins leaks the threads when trying to disconnect nodes | 20:30 |
| fungi | mordred: no, that was an intentional typo on my part to simulate a missing host key | 20:30 |
| greghaynes | (I think the type line is wrong and thats easier than my thinking of how to fix it) | 20:30 |
| clarkb | zaro: but the nodes go away and all that | 20:30 |
| fungi | mordred: as in test that conditional yourself, i think it doesn't do what you think it does | 20:31 |
| *** tkelsey has quit IRC | 20:31 | |
| mordred | fungi: I did - but I left out the && | 20:31 |
| mordred | copypasta problem | 20:31 |
| fungi | aha, yep. thought that might be what you were going for | 20:31 |
| greghaynes | ah, that would explain my confusion for what youre trying to do ;) | 20:31 |
| *** wenlock has quit IRC | 20:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Run tasks in worker threads https://review.openstack.org/165982 | 20:32 |
| fungi | mordred: inserting && it definitely does what i think we want | 20:32 |
| Shrews | mordred: your shade refactor proposal makes me yearn for python class templates | 20:32 |
| mordred | Shrews: ++ | 20:32 |
| clarkb | Shrews: ABC kinda does that | 20:32 |
| fungi | i'll give that another round of functional testing | 20:32 |
| zaro | clarkb: go away because we don't see it the reference in the gerrit ui? | 20:32 |
| clarkb | zaro: ya | 20:33 |
| clarkb | zaro: we will only have a couple hundred nodes but maybe 8k threads | 20:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Regenerate ssh host key on boot https://review.openstack.org/166000 | 20:33 |
| mordred | fungi: this time with && | 20:33 |
| jeblair | clarkb, fungi, mordred: i'm going to restart nodepool with my change now | 20:33 |
| fungi | jeblair: awesome | 20:33 |
| mordred | clarkb: yeah - I used abc ... but it's still not exactly what we want | 20:33 |
| mordred | clarkb: because C++ class tempaltes are AWESOME | 20:33 |
| zaro | wondering if maybe the thread for the connection is still around, just not reported on jenkins ui? | 20:33 |
| clarkb | mordred: you don't need them in python... | 20:34 |
| *** dims has quit IRC | 20:34 | |
| mordred | clarkb: yeah you do - because you also need typing in python - because it's the biggest problem with python programming | 20:34 |
| jeblair | it is doing things very very quickly | 20:34 |
| mordred | jeblair: woot! | 20:34 |
| clarkb | jeblair: yay | 20:34 |
| fungi | mordred: one other question. why the duplicate check for test -f /etc/ssh/ssh_host_rsa_key | 20:34 |
| mordred | jeblair: is it doing good things very very quickly? | 20:34 |
| mordred | fungi: because my fingers move too fast | 20:35 |
| clarkb | mordred: 3.5 is adding that or is it 3.6? but I think many pythonistas are cranky about it | 20:35 |
| jeblair | mordred: it's getting 500 unknown errors from hpcloud faster than ever before | 20:35 |
| fungi | mordred: i'm going to test without the redundant inner one | 20:35 |
| dprince | Has there been a Zuul change that would effect Fedora 20 TripleO nodes today? | 20:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Regenerate ssh host key on boot https://review.openstack.org/166000 | 20:35 |
| jeblair | Timeout: HTTPSConnectionPool(host='region-b.geo-1.compute.hpcloudsvc.com', port=443): Read timed out. | 20:35 |
| dprince | I've not seen Fedora 20 CI jobs running for several hours... | 20:35 |
| anteaya | mordred: it isn't blowing up, the choices were doing things quickly or blowing up | 20:35 |
| jeblair | that is new | 20:35 |
| mordred | jeblair: ooh. that is new | 20:36 |
| asselin | does infra host a build copy of scp.hpi anywhere? | 20:36 |
| clarkb | asselin: on tarballs.openstack.org | 20:36 |
| mordred | dprince: well, we've been having one of those days | 20:37 |
| jeblair | mordred: f20 is different | 20:38 |
| dprince | mordred: sorry to hear that. I figured I'd ask because I'm perplexed by this. | 20:38 |
| clarkb | f20 broke because rax did something weird in the base image or something | 20:38 |
| clarkb | fwiw I am not sure that install_puppet.sh is where we should put the workaround for these rpobems | 20:38 |
| clarkb | should go in prepare_node.sh | 20:39 |
| greghaynes | Would that effect tripleo f20 though? | 20:39 |
| *** ssam2 has quit IRC | 20:39 | |
| mordred | which workaroudn is in install_puppet? | 20:39 |
| mordred | clarkb: ?? | 20:39 |
| clarkb | mordred: the f20 make grub work workaround | 20:39 |
| mordred | ah. gotcha | 20:39 |
| mordred | yeah | 20:39 |
| mordred | I agree | 20:39 |
| clarkb | greghaynes: oh probably not | 20:39 |
| *** dkranz has quit IRC | 20:39 | |
| greghaynes | Yea, that would be wierd if it did | 20:39 |
| clarkb | greghaynes: unless f20 is broken across the board and tripleo has same bug as rax? | 20:39 |
| mordred | clarkb: is it possible taht the f20 make grub work workaround broke the tripleo cloud? | 20:39 |
| dprince | FWIW, I see Fedora 20 nodes getting created in our cloud. But nothing is running actualy CI jobs for a few hours now | 20:39 |
| clarkb | mordred: maybe | 20:39 |
| jeblair | clarkb, fungi, mordred: i've lowered the rate limit to see if that affects the read time outs | 20:40 |
| mordred | jeblair: cool | 20:40 |
| fungi | jeblair: that's a good sign though, i think | 20:40 |
| clarkb | I wonder if those nodes got lost when 04 fell over | 20:40 |
| clarkb | I can look at nodepool to check | 20:40 |
| clarkb | dprince: all of the nodes are either building or deleting | 20:40 |
| clarkb | let me try manually deleting the deleting nodes | 20:41 |
| mordred | fungi: any luck with that latest patch? | 20:41 |
| fungi | mordred: looks like it worked | 20:41 |
| anteaya | yay | 20:41 |
| dprince | clarkb: okay, I saw some ACTIVE instances from a Nova prospective | 20:41 |
| fungi | mordred: yep, new host keys on boot now | 20:41 |
| *** yfried is now known as yfried|afk | 20:42 | |
| *** timcline has joined #openstack-infra | 20:42 | |
| mordred | fungi: woot. then I'm going to push in those two patches and try rebuilding b5 again | 20:42 |
| dprince | clarkb: perhaps they are stuck in one of the prepare scripts? | 20:42 |
| *** timcline has quit IRC | 20:42 | |
| fungi | mordred: doing one last boot without deleting the keys just to make sure it's also not regenerating them when they already exist | 20:42 |
| clarkb | dprince: maybe, hopefully this foreground delete will shed some light on what may be happening | 20:42 |
| mordred | fungi: excellent point | 20:42 |
| *** timcline has joined #openstack-infra | 20:42 | |
| fungi | mordred: also fine | 20:42 |
| openstackgerrit | Merged openstack-infra/system-config: Purge cloud-init https://review.openstack.org/165996 | 20:43 |
| fungi | mordred: good to go, at least for trusty | 20:43 |
| *** radez is now known as radez_g0n3 | 20:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Regenerate ssh host key on boot https://review.openstack.org/166000 | 20:43 |
| dprince | sudo rm -f /etc/ssh/ssh_host_* ? | 20:44 |
| dprince | Are we sure ^^^ that doesn't hose everything? | 20:44 |
| * dprince isn't sure what is deployed... just checking history | 20:44 | |
| mordred | dprince: well, we are fixing that right now ... | 20:44 |
| dprince | mordred: okay, I'll leave you to it and check back in a bit | 20:44 |
| clarkb | and it wasn't deployed to tripleo | 20:44 |
| mordred | dprince: although that's very new - so it's unlikely to be in your tripleo nodes | 20:45 |
| clarkb | and it should work just fine on fedora because supposedly fedora does the right thing unlike ubuntu | 20:45 |
| mordred | rebuilding b5 | 20:46 |
| jeblair | ClientException: Unknown Error (HTTP 502) | 20:46 |
| jeblair | also seeing a bunch of those | 20:46 |
| jeblair | the read timeouts have subsided though | 20:47 |
| mordred | jeblair: UnknownError are my favorite | 20:47 |
| cinerama | pleia2: okay i think we got it going with the new zanata | 20:47 |
| openstackgerrit | Khai Do proposed openstack-infra/gerritlib: Update project docs and setup for tests https://review.openstack.org/165979 | 20:47 |
| mordred | cinerama: woo | 20:47 |
| cinerama | pleia2: oh wait i need more coffee :) | 20:47 |
| openstackgerrit | Khai Do proposed openstack-infra/gerritlib: Update project docs https://review.openstack.org/165979 | 20:48 |
| cinerama | i reverted one of my manifests but it also reverted the version change :) | 20:50 |
| cinerama | but the new template doesn't fail? :) | 20:50 |
| *** jaypipes has quit IRC | 20:50 | |
| mordred | nibalizer: 2015-03-19 20:50:42,147 INFO nodepool.image.build.devstack-trusty: Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations | 20:50 |
| mordred | nibalizer: apparently puppet really wants us to know that | 20:51 |
| *** ibiris is now known as ibiris_away | 20:51 | |
| anteaya | ttx is waiting on https://review.openstack.org/#/c/150267/ to cut nova, just thought I would let you know | 20:51 |
| anteaya | it is in check and almost finished running tests | 20:52 |
| clarkb | Exception: Timeout waiting for server 22243675-f398-4779-9650-d3e7da24b782 deletion in tripleo-test-cloud-rh1 | 20:52 |
| mordred | clarkb: wow. they're having the delete server problem too? | 20:52 |
| clarkb | mordred: I guess? I am trying a different node to see if it is consistent | 20:52 |
| mordred | Shrews: btw - I figure it'll take us a few passes at one of the resources to get the pattern right | 20:53 |
| nibalizer | clarkb: wut | 20:53 |
| openstackgerrit | Khai Do proposed stackforge/python-jenkins: Add info on how to contribute. https://review.openstack.org/166006 | 20:53 |
| mordred | Shrews: I'm starting to think we actually don't want ResourceRequest after all | 20:53 |
| nibalizer | mordred: it DOES | 20:53 |
| nibalizer | clarkb: IT NEEDS YOU TO KNOW | 20:53 |
| nibalizer | er mordred | 20:53 |
| mordred | Shrews: since you coudl also "get" a resource and have that resource be still in a building state | 20:53 |
| nibalizer | clarkb: so what did I break? | 20:53 |
| clarkb | nibalizer: I do not know what you are referring to | 20:53 |
| mordred | Shrews: so I think I may have been wrong about the ResourceRequest class | 20:53 |
| *** emagana has quit IRC | 20:53 | |
| Shrews | mordred: I'm working through modeling a scenario now to see what this would actually look like | 20:54 |
| mordred | ++ | 20:54 |
| mordred | I started going down that path and then got distracted | 20:54 |
| cinerama | time for more wackypants java errors | 20:54 |
| clarkb | cinerama: o/ | 20:54 |
| *** emagana has joined #openstack-infra | 20:54 | |
| Shrews | mordred: i'll post something up that's depedent on your review, when i have it | 20:55 |
| cinerama | this has really been old home week for languages i haven't done in a while | 20:55 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Add standard openstack info on how to contribute. https://review.openstack.org/166008 | 20:55 |
| *** Hal1 has quit IRC | 20:56 | |
| cinerama | i'm going to have to do perl soon aren't i? | 20:56 |
| fungi | cinerama: algol | 20:56 |
| cinerama | fungi: i have an algol book somewhere around here | 20:57 |
| *** ominakov has quit IRC | 20:57 | |
| fungi | don't we all? | 20:57 |
| *** Sukhdev has joined #openstack-infra | 20:57 | |
| mordred | clarkb: blast - today has been so fun I forgot to turn on the day of sportsball | 20:58 |
| clarkb | mordred: any upsets yet? I haven't really followed it either | 20:58 |
| cinerama | i can see it from here actually. it's next to 'code complete' and 'common lisp' | 20:58 |
| Shrews | mordred: it has been exciting | 20:58 |
| Shrews | notre dame really tried to lose | 20:58 |
| clarkb | maybe I should go downstairs and turn on the tv | 20:58 |
| *** bdpayne has joined #openstack-infra | 20:59 | |
| Sukhdev | clarkb: need a quick favor - hope you can help | 20:59 |
| mordred | clarkb: yes - iowa state lost | 20:59 |
| clarkb | Sukhdev: maybe | 20:59 |
| anteaya | Sukhdev: just ask | 20:59 |
| fungi | cinerama: you need "fortran programming: a spiral approach" i can't recommend it enough | 20:59 |
| mordred | clarkb: to uab | 20:59 |
| mordred | clarkb: and Georgia State beat Baylor | 20:59 |
| Sukhdev | If I look at these instructions, http://docs.openstack.org/infra/manual/drivers.html#end-of-release | 21:00 |
| mordred | I had both of those in the 8 - so that's not great for the old bracket | 21:00 |
| mordred | but go GA State | 21:00 |
| Sukhdev | It says to delete a release select the release and delete it | 21:00 |
| clarkb | the eastern wa game is one I want to se | 21:00 |
| cinerama | fungi: sweet cover but i'm disappointed it's not spiral bound | 21:00 |
| Sukhdev | but, when I go here - https://review.openstack.org/#/admin/projects/stackforge/networking-arista,branches | 21:00 |
| Sukhdev | there is no way to select the release - | 21:01 |
| *** asselin_ has joined #openstack-infra | 21:01 | |
| Sukhdev | eg. I want to delete proposed/kilo | 21:01 |
| mordred | clarkb: I have georgetown there- but I'll be rooting for ewa | 21:01 |
| anteaya | Sukhdev: have you tagged your release? | 21:01 |
| Sukhdev | not yet | 21:01 |
| clarkb | mordred: at some point the big sky is going to win a tournament game | 21:01 |
| anteaya | you have to tag before you delete | 21:01 |
| clarkb | mordred: and this is a good shot for them to do it | 21:01 |
| anteaya | Sukhdev: This step should be performed by the OpenStack Release Manager after the release is tagged. | 21:01 |
| fungi | Sukhdev: see where that section starts with "This step should be performed by the OpenStack Release Manager..." | 21:02 |
| *** yfried|afk is now known as yfried | 21:02 | |
| jeblair | clarkb, fungi, mordred: i'm continuing to tweak tunables to see if i can get this to perform better | 21:02 |
| mordred | jeblair: cool | 21:02 |
| fungi | Sukhdev: you'll need to get a release manager or infra admin to delete branches for you. it's not a gerrit permission we can safely grant | 21:02 |
| mordred | clarkb: yeah - I don' tbelieve in georgetown at all | 21:02 |
| *** patrickeast_ has joined #openstack-infra | 21:02 | |
| clarkb | fungi: I think we can grant it to stackforge projecs? | 21:02 |
| fungi | clarkb: didn't we try? | 21:03 |
| *** tsg_ has quit IRC | 21:03 | |
| clarkb | fungi: I am trying to remember if that was possible, they have to be branch owners right? | 21:03 |
| Sukhdev | fungi: Ah ha - that is the part I was missing - thanks for clarification | 21:03 |
| anteaya | how do we prevent them from deleteing prior to tagging | 21:03 |
| anteaya | that will be a fun one to try to fix | 21:03 |
| fungi | clarkb: i think they had to be project owners. zaro: do you recall? | 21:03 |
| *** dannywil_ has quit IRC | 21:03 | |
| *** patrickeast has quit IRC | 21:03 | |
| *** patrickeast_ is now known as patrickeast | 21:03 | |
| clarkb | jeblair: anything we can do to help? | 21:03 |
| clarkb | fungi: ya thats what it was | 21:03 |
| clarkb | fungi: which imprated a bunch of other prems that werent great | 21:03 |
| fungi | anteaya: we don't prevent anything. we just load the gun and they can tell us if they want it aimed at their foot | 21:04 |
| clarkb | ok second node in rh1 failed to delete due to a timeout as well | 21:04 |
| clarkb | so all the clouds are broken | 21:04 |
| anteaya | fungi: yes | 21:04 |
| *** dannywilson has joined #openstack-infra | 21:04 | |
| anteaya | picturing infra become a waiting room with bleeding feet next release | 21:04 |
| *** dangers is now known as dangers_away | 21:04 | |
| *** ivar-laz_ has joined #openstack-infra | 21:04 | |
| mordred | clarkb: have I mentioned that I don't think deleting things should be hard? | 21:04 |
| Sukhdev | fungi anteaya: Actually, I first called it proposed/kilo, and then later decided, I should really call it kilo - so, end up creating two branches - will tag the kilo one, but, wanted to get rid of proposed/kilo - | 21:04 |
| anteaya | Sukhdev: glad you are thinking about your own feet | 21:05 |
| *** asselin_ has quit IRC | 21:05 | |
| *** rfolco has quit IRC | 21:05 | |
| *** ivar-laz_ has quit IRC | 21:05 | |
| fungi | Sukhdev: if you're ready for that branch to be deleted, i can do so. but having a branch named "kilo" instead of "stable/kilo" is going to cause integration testing problems | 21:06 |
| jeblair | ClientException: Unknown Error (HTTP 503) | 21:06 |
| jeblair | wow! | 21:06 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add missing ' https://review.openstack.org/165786 | 21:06 |
| jeblair | so many unknown errors! | 21:06 |
| clarkb | BadRequest: Error. Unable to associate floating ip (HTTP 400) (Request-ID: req-ae2cf0db-28bc-4081-b09b-79a60e5495f2) is the rh1 issue I think | 21:06 |
| *** ihrachyshka has quit IRC | 21:06 | |
| *** ivar-laz_ has joined #openstack-infra | 21:06 | |
| jeblair | 500, 502, 503... what, no 501? | 21:06 |
| mordred | clarkb: also - texas just lost - so that's 3 losses for the B12 | 21:06 |
| mordred | jeblair: I'm sure there's a 501 coming ... | 21:06 |
| anteaya | you are missing the full series | 21:06 |
| anteaya | 501 is the collector's item apparently | 21:06 |
| clarkb | dprince: so if you want ot look into something I think ^ is the thing | 21:06 |
| Sukhdev | fungi: Thanks for pointing this out - in that case, can you please do me a favor and delete them both - I will create stable/kilo and tag that one. | 21:06 |
| *** ivar-laz_ has quit IRC | 21:07 | |
| *** bhuvan has quit IRC | 21:07 | |
| *** ivar-laz_ has joined #openstack-infra | 21:07 | |
| Sukhdev | fungi: Thank you - that did it…appreciate it... | 21:08 |
| fungi | Sukhdev: done. i've deleted them both, they each had the same commit sha as the tip of master, 467aa2a8c0f5cc1f52eeef07e36fb6894f8641c3 | 21:08 |
| *** ivar-lazzaro has quit IRC | 21:08 | |
| *** masayukig has quit IRC | 21:08 | |
| *** masayukig has joined #openstack-infra | 21:08 | |
| dprince | clarkb: are you getting 501 errors from Nova API? | 21:09 |
| clarkb | dprince: no that is a 400 error | 21:09 |
| *** bswartz has quit IRC | 21:10 | |
| mordred | hpcloud-b5 image built - uploading | 21:10 |
| mordred | 2015-03-19 21:09:40,919 INFO nodepool.DiskImageBuilderThread: DIB image 118717 with file devstack-trusty is built | 21:10 |
| dprince | clarkb: I'll check the logs, thanks | 21:11 |
| *** andreykurilin_ has joined #openstack-infra | 21:12 | |
| *** yfried is now known as yfried|afk | 21:12 | |
| *** Somay has joined #openstack-infra | 21:12 | |
| clarkb | wow we upload logs for every instance booted by tempest? | 21:13 |
| mtreinish | clarkb: well the qemu logs, we added that a while ago | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| *** ivar-laz_ has quit IRC | 21:15 | |
| clarkb | dprince: fwiw I can keystone catalog but operations like nova list and neutron floatingip-list hang | 21:16 |
| *** ominakov has joined #openstack-infra | 21:16 | |
| *** yamamoto has joined #openstack-infra | 21:16 | |
| *** radez_g0n3 is now known as radez | 21:17 | |
| Sukhdev | fungi: Is there a way to delete the tags? | 21:17 |
| fungi | Sukhdev: not really, no. tag deletion propagates to push mirror but not via pull mechanisms like fetch all or remote update | 21:19 |
| fungi | Sukhdev: so lots of places will have the old tags cached after deletion | 21:19 |
| clarkb | dprince: ERROR (ConnectionRefused): Unable to establish connection to http://10.1.8.37:5000/v2.0/tokens the plot thickens | 21:19 |
| clarkb | dprince: seems like a broken proxy or something | 21:19 |
| jeblair | clarkb, fungi, mordred: i'm continuing to slow the rate and increase timeouts on hpcloud, but i'm also continuing to see problems | 21:19 |
| Sukhdev | fungi: cool - thanks | 21:19 |
| *** andreykurilin_ has quit IRC | 21:20 | |
| *** dboik_ has joined #openstack-infra | 21:20 | |
| mordred | jeblair, clarkb, fungi, pleia2: hpcloud seems to be having some systemic issues, fwiw | 21:20 |
| *** andreykurilin_ has joined #openstack-infra | 21:20 | |
| *** dboik_ has quit IRC | 21:20 | |
| *** otter768 has joined #openstack-infra | 21:21 | |
| *** yamamoto has quit IRC | 21:21 | |
| *** davideagnello has joined #openstack-infra | 21:21 | |
| *** dboik_ has joined #openstack-infra | 21:21 | |
| clarkb | mordred: yes, ec2 meta service is unwork :) | 21:21 |
| jeblair | i hope we didn't cause them. afaik, we never exceeded our rate limit on the calling side | 21:21 |
| mordred | clarkb: nope- bigger | 21:21 |
| mordred | clarkb: an incident has started | 21:22 |
| anteaya | no | 21:22 |
| anteaya | not an incident | 21:22 |
| mordred | clarkb: not just "metadata service is yuck" | 21:22 |
| clarkb | mordred: jeblair I would not be surprised if this has been part of the larger underlying issue we have had with hpcloud over the last week or two | 21:22 |
| mordred | clarkb: ++ | 21:22 |
| *** ominakov has quit IRC | 21:22 | |
| *** dboik has quit IRC | 21:23 | |
| jeblair | we're currently at around one api call every 0.8 seconds (across all 5 providers) | 21:23 |
| *** davideagnello has quit IRC | 21:24 | |
| jeblair | considering that my 'nova show' command i've been trying periodically for the past 10 minutes still is returning 502 every time i try, i'm starting to think something has fallen over | 21:24 |
| *** ivar-lazzaro has joined #openstack-infra | 21:24 | |
| *** dkliban is now known as dkliban_afk | 21:24 | |
| *** davideagnello has joined #openstack-infra | 21:25 | |
| *** xyang1 has quit IRC | 21:25 | |
| *** otter768 has quit IRC | 21:26 | |
| jeblair | and hrm, we've got basically nothing from hpcloud in use | 21:27 |
| *** salv-orlando has joined #openstack-infra | 21:27 | |
| *** achanda has quit IRC | 21:28 | |
| anteaya | oi | 21:28 |
| clarkb | maybe we should take this as an opportunity to patch apache against openssl problems >_> | 21:30 |
| *** andreykurilin_ has quit IRC | 21:30 | |
| *** andreykurilin_ has joined #openstack-infra | 21:30 | |
| jogo | I see the Evgeny Antyshev user is still spamming | 21:30 |
| clarkb | jogo: link? | 21:31 |
| jogo | clarkb: https://review.openstack.org/#/c/165767/ | 21:31 |
| jogo | hmm | 21:31 |
| jogo | one moment | 21:31 |
| jogo | clarkb: ahh yup https://review.openstack.org/#/c/165767/ | 21:32 |
| anteaya | that happened just a few minutes ago | 21:32 |
| anteaya | same email | 21:32 |
| *** dkranz has joined #openstack-infra | 21:33 | |
| anteaya | is there more that one user with the same full name and email address in the gerrit db? | 21:33 |
| clarkb | no, and I have confirmed that user is marked inactive | 21:33 |
| clarkb | and I did flush the caches | 21:33 |
| clarkb | so maybe if they have a connection open it stays open? | 21:34 |
| anteaya | I get one account id for that email | 21:34 |
| clarkb | anteaya: yes, 13431 | 21:34 |
| anteaya | any idea how we id which connection is theirs and kill it? | 21:34 |
| anteaya | clarkb: confirmed | 21:34 |
| clarkb | anteaya: no, not sure | 21:34 |
| clarkb | its possible we could find PTR records that belonged to parallels.com and shoot those tcp connections? | 21:35 |
| anteaya | I'm for that | 21:35 |
| anteaya | how can I help? | 21:35 |
| *** carl_baldwin has quit IRC | 21:35 | |
| *** andreykurilin_ has quit IRC | 21:35 | |
| clarkb | anteaya: well I am not really for it :) | 21:35 |
| anteaya | okay | 21:35 |
| anteaya | what are you for? | 21:35 |
| clarkb | its imprecise and even then the tools for it are not great | 21:35 |
| *** mattfarina has quit IRC | 21:35 | |
| jogo | anyone try contacting that email address? | 21:36 |
| clarkb | so first question, has anyone sent email to that address? | 21:36 |
| anteaya | yes | 21:36 |
| *** andreykurilin_ has joined #openstack-infra | 21:36 | |
| anteaya | jogo: dims did this morning and cc'd me, no reply | 21:36 |
| *** ChuckC has quit IRC | 21:36 | |
| *** achanda has joined #openstack-infra | 21:37 | |
| anteaya | 11:15am eastern time | 21:37 |
| clarkb | oh sweet gerrit show-connections will show you uids too | 21:38 |
| anteaya | awesome | 21:38 |
| clarkb | now to figure out if there is a gerrit command to kill a connection | 21:39 |
| *** alexpilotti has joined #openstack-infra | 21:39 | |
| *** dims has joined #openstack-infra | 21:39 | |
| clarkb | zaro: do you know if gerrit kill can be fed a session id? | 21:39 |
| jogo | good news is I only count about 40 comments from him today | 21:40 |
| jogo | so this is bad but not terribad | 21:40 |
| *** salv-orlando has quit IRC | 21:40 | |
| anteaya | jogo: well we needed to know if flushing caches solved the problem | 21:40 |
| jogo | anteaya: right | 21:40 |
| anteaya | and it didn't, so thanks for letting us know | 21:40 |
| clarkb | https://groups.google.com/forum/#!topic/repo-discuss/_xWxgvEryVg | 21:41 |
| anteaya | 2013 | 21:41 |
| jogo | anteaya: do you have a email for CloudByte CI ? | 21:41 |
| jeblair | so, last-ditch option is restart gerrit | 21:42 |
| jogo | hmm never mind may not be related | 21:42 |
| anteaya | openstack-dev@cloudbyte.com | 21:42 |
| *** salv-orlando has joined #openstack-infra | 21:42 | |
| *** garyh has quit IRC | 21:43 | |
| jogo | anteaya: it looks like he was testing open patches in the sandbox and there are a lot of cloudbyte ones | 21:43 |
| anteaya | jogo: ah | 21:43 |
| clarkb | jeblair: just looking at the graphs for nodes it doesn't appear that rax has fallen over due to your chnage | 21:43 |
| jogo | was hoping he was working on a CI system that had a contact ... :/ | 21:43 |
| clarkb | jeblair: seems likely that your change is a good one considering that | 21:44 |
| jeblair | clarkb: i agree | 21:44 |
| *** dboik_ has quit IRC | 21:44 | |
| jeblair | clarkb: well, i don't know if it has caused hpcloud to fall over :/ | 21:44 |
| anteaya | german company going by parallels.com | 21:44 |
| *** rlandy has quit IRC | 21:44 | |
| asselin | jogo, I have some e-mails from cloudbyte | 21:44 |
| clarkb | jeblair: ya I guess that it may still have caused that :) | 21:44 |
| *** dboik has joined #openstack-infra | 21:45 | |
| *** timcline has quit IRC | 21:45 | |
| asselin | jogo, let me konw if you want more names e-mail addresses | 21:45 |
| *** peristeri has quit IRC | 21:46 | |
| anteaya | I'm more concerned we don't have a way of closing an open connection for a disabled account | 21:46 |
| anteaya | I'm surprised this never came up before | 21:46 |
| *** dboik_ has joined #openstack-infra | 21:46 | |
| *** dhritishikhar has quit IRC | 21:48 | |
| *** eharney has joined #openstack-infra | 21:48 | |
| *** mtanino has quit IRC | 21:48 | |
| *** mriedem is now known as mriedem_away | 21:49 | |
| clarkb | anteaya: ya not sure what gerrit is doing internally there | 21:49 |
| *** tsg has joined #openstack-infra | 21:49 | |
| clarkb | anteaya: its possible that the session timeout of 12 hours is what will clean it up | 21:49 |
| bhunter71 | darn. I see a 'gerrit close-connection' command in gerrit 2.11. | 21:49 |
| mordred | there's a 2.11 already? | 21:50 |
| *** dboik has quit IRC | 21:50 | |
| *** eharney has quit IRC | 21:50 | |
| jogo | asselin: that was a red herring, thanks anyway | 21:50 |
| anteaya | yes I'm looking and so far your unanswered question from 2013 is the best lead | 21:50 |
| bhunter71 | modred: perhaps not, google just took me to 2.11-rc1 | 21:51 |
| bknudson | one thing kind of funny about depends-on is that I *could* use the same change-id in 2 projects... e.g., same changeid in keystone as in devstack. | 21:51 |
| *** dboik_ has quit IRC | 21:51 | |
| bknudson | and I would prefer to since it makes the other one easy to find. | 21:51 |
| cinerama | pleia2: this time for real :) | 21:52 |
| *** achanda has quit IRC | 21:53 | |
| clarkb | since things are sort of derpy right now anyone want to review https://review.openstack.org/#/c/165542/ to fix html rendering for console.html files? | 21:53 |
| pleia2 | cinerama: how's it going? :) | 21:53 |
| mordred | 2015-03-19 21:28:40,286 INFO nodepool.DiskImageUpdater: Image devstack-trusty-1426799380.template.openstack.org in hpcloud-b5 is ready | 21:54 |
| openstackgerrit | Stephanie Miller proposed openstack-infra/puppet-zanata: Initial commit of Zanata-specific sections https://review.openstack.org/147947 | 21:54 |
| cinerama | pleia2: actually works now. i've just pushed the template changes | 21:54 |
| pleia2 | cinerama: nice! I'll test | 21:54 |
| *** andreykurilin_ has quit IRC | 21:55 | |
| cinerama | pleia2: there were a few more changes in the standalone template that weren't advertised in the release notes | 21:55 |
| anteaya | clarkb: yes the kill command takes a task id, but I don't see any returned values from show-connections that could be construed as a task id | 21:55 |
| *** andreykurilin_ has joined #openstack-infra | 21:55 | |
| anteaya | clarkb: could we try username though or ip? | 21:55 |
| pleia2 | cinerama: I'm not surprised | 21:55 |
| zaro | clarkb: it looks like you kill with an ID. | 21:55 |
| zaro | from the docs anyways | 21:55 |
| clarkb | zaro: ya but session ids are not task ids from what I can tell | 21:55 |
| cinerama | pleia2: i added the infinispan stuff they mentioned, which curiously is not in the prebaked 3.6.0 standalone xml | 21:56 |
| anteaya | clarkb: can we try? | 21:56 |
| pleia2 | cinerama: yeah, I don't even know what that is | 21:56 |
| anteaya | clarkb: worst thing that can happen is it doesn't work, yeah? | 21:56 |
| clarkb | anteaya: no worst thing is we kill a real task that is doing something else with the same id number | 21:56 |
| anteaya | clarkb: oh | 21:56 |
| anteaya | clarkb: yeah okay let's not that | 21:57 |
| anteaya | sigh | 21:57 |
| zaro | clarkb: you could probably just do suexec to do anything you want | 21:57 |
| clarkb | zaro: well we need to disconnect an inactivated account, I don't think suexec will do that | 21:58 |
| *** dustins has quit IRC | 21:58 | |
| *** yfried|afk has quit IRC | 21:58 | |
| anteaya | zaro: account 13431 was disabled about 6 hours ago | 21:58 |
| anteaya | and we flushed the caches | 21:58 |
| *** yfried|afk has joined #openstack-infra | 21:58 | |
| anteaya | but it is still spamming patches with comments from a misconfigured ci | 21:59 |
| *** hdd has quit IRC | 21:59 | |
| anteaya | most recent was about 30 minutes ago | 21:59 |
| *** oomichi has joined #openstack-infra | 21:59 | |
| anteaya | not a big deal but we would like the ability to be able to stop disabled accounts from posting patches after they are disabled | 21:59 |
| jeblair | clarkb, mordred, fungi: hpcloud is effectively offline for us | 22:00 |
| anteaya | posting comments to patches, sorry | 22:00 |
| anteaya | jeblair: :( | 22:00 |
| jeblair | no new nodes in the past couple of hours | 22:00 |
| *** eharney has joined #openstack-infra | 22:00 | |
| clarkb | sdague: comment on one of your os-loganalyze test cleanups, I think we can make it even simpler | 22:00 |
| zaro | clarkb, anteaya : you then you probably can do with close-connection command? | 22:00 |
| mordred | jeblair: my node upload just wokred | 22:01 |
| clarkb | zaro: that doesn't seem to exist in our gerrit | 22:01 |
| anteaya | zaro: do tell us about the close-connection command, does it have some docs? | 22:01 |
| clarkb | mordred: image upload? probably beause nova != glance? | 22:01 |
| zaro | oops must be new, https://gerrit-review.googlesource.com/Documentation/cmd-close-connection.html | 22:01 |
| mordred | clarkb: it had been failing for the last $time | 22:01 |
| mordred | clarkb: becaue the glance apis are broken too | 22:01 |
| *** achanda has joined #openstack-infra | 22:01 | |
| jeblair | mordred: my 'nova show' still fails | 22:01 |
| clarkb | mordred: fun | 22:01 |
| anteaya | https://gerrit-review.googlesource.com/Documentation/cmd-close-connection.html | 22:01 |
| mordred | jeblair: sigh | 22:02 |
| zaro | clarkb, anteaya maybe i can cherry pick that into our gerit? | 22:02 |
| anteaya | zaro: 2.11-rc1 | 22:02 |
| anteaya | zaro: oh please | 22:02 |
| anteaya | yes please | 22:02 |
| jhesketh | Morning | 22:02 |
| clarkb | well that won't fix anything immediately | 22:02 |
| clarkb | jhesketh: good morning | 22:02 |
| anteaya | morning jhesketh | 22:02 |
| anteaya | clarkb: true | 22:02 |
| anteaya | clarkb: do we have any options for what will? | 22:02 |
| *** dannywilson has quit IRC | 22:03 | |
| *** virmitio has joined #openstack-infra | 22:03 | |
| clarkb | btw the heat functional test takes almost 2 hours | 22:03 |
| anteaya | wonderful | 22:04 |
| anteaya | good thing ttx went to bed | 22:04 |
| *** hashar has quit IRC | 22:04 | |
| jeblair | wow, so much for the 1 hour budget | 22:04 |
| mordred | jeblair: yah - nova command also no worky for me | 22:05 |
| clarkb | jhesketh: basically hpcloud and rh1 tripleo cloud are both borked | 22:06 |
| clarkb | jhesketh: so its been a fun day :) https://review.openstack.org/#/c/165542/ may interest you | 22:06 |
| *** adalbas has quit IRC | 22:07 | |
| zaro | anteaya, clarkb : looks promising, no conflicts. now just need to test it. | 22:07 |
| jhesketh | clarkb: ouch, sounds fun.. Let me know if I can help | 22:07 |
| jhesketh | Will take a look, thanks | 22:07 |
| *** dannywilson has joined #openstack-infra | 22:07 | |
| jogo | just got a few more spams from the parallels guy ... :( | 22:07 |
| anteaya | zaro: k, I see that command has a wait option, so if the wait option is not invoked it kills an open command immediately? | 22:08 |
| anteaya | zaro: that is the functionality I'm looking for | 22:08 |
| anteaya | jogo: dang, so far I don't know of a good way to kill that connection | 22:09 |
| anteaya | jogo: right now the best option I have heard is wait on the expiration of some other kind of cache | 22:09 |
| clarkb | at this point I say we ride it out to the 12 hour session invalidation | 22:09 |
| *** jamielennox|away is now known as jamielennox | 22:09 | |
| clarkb | if its still trouble then we restart gerrit | 22:10 |
| anteaya | zaro: open connection, not open command | 22:10 |
| anteaya | clarkb: if the only option is to restart gerrit then yes | 22:10 |
| *** dboik has joined #openstack-infra | 22:10 | |
| *** e0ne has joined #openstack-infra | 22:11 | |
| jogo | sounds like this is a feature we want for future gerrit versions | 22:11 |
| *** dboik_ has joined #openstack-infra | 22:11 | |
| anteaya | jogo: yes | 22:11 |
| anteaya | so a session is invalid 12 hours after it begins? | 22:12 |
| anteaya | clarkb: do we have any ability to search that account in open-connections and see when the session began? | 22:12 |
| clarkb | anteaya: 12 hours is when the cache should check if the account is still valid | 22:12 |
| anteaya | gerrit username: eantyshev | 22:12 |
| clarkb | anteaya: we do, but I am not sure its from when the session began | 22:12 |
| anteaya | hmmm | 22:12 |
| clarkb | its from whenver the cache was last updated for that entry most likely | 22:13 |
| clarkb | at worst it should be no later than 12 hours after we marked the account inactive | 22:13 |
| zaro | anteaya: yeah, this one right? https://gerrit-review.googlesource.com/#/c/63100/ | 22:13 |
| anteaya | from the docs it should show-connections should show a start time | 22:13 |
| anteaya | oh | 22:13 |
| anteaya | zaro: that looks like the one, let me look at the patch | 22:14 |
| *** dboik has quit IRC | 22:15 | |
| *** packet has quit IRC | 22:15 | |
| *** timcline has joined #openstack-infra | 22:16 | |
| anteaya | looks like you can feed it a list | 22:16 |
| anteaya | and looks like the wait option is if you want the command to wait for the connection to close before exiting | 22:16 |
| *** VijayTripathi has joined #openstack-infra | 22:16 | |
| anteaya | I had read that wrong | 22:16 |
| anteaya | you feed it a session id | 22:17 |
| *** yamamoto has joined #openstack-infra | 22:17 | |
| anteaya | clarkb: show-connections gives you session ids? | 22:17 |
| anteaya | zaro: yes please, I would like taht functionality | 22:18 |
| *** aysyd has quit IRC | 22:18 | |
| *** e0ne has quit IRC | 22:18 | |
| *** AJaeger has quit IRC | 22:18 | |
| *** dims has quit IRC | 22:19 | |
| YorikSar | Is there any place where Jenkins job run times are collected and stats for specific jobs can be viewed? | 22:20 |
| *** aysyd has joined #openstack-infra | 22:20 | |
| anteaya | http://status.openstack.org/zuul/ | 22:20 |
| *** timcline has quit IRC | 22:20 | |
| anteaya | stats? what kind of stats? | 22:21 |
| cinerama | vagrant upgrade, time to put the kettle on | 22:21 |
| anteaya | we have graphs at the bottom | 22:21 |
| *** ghostpl_ has quit IRC | 22:21 | |
| *** yamamoto has quit IRC | 22:21 | |
| fungi | YorikSar: graphite.openstack.org | 22:22 |
| YorikSar | anteaya: I want to see if some specific jobs became faster or changes pace in any way. | 22:22 |
| anteaya | ah yes as fungi says http://graphite.openstack.org/ | 22:22 |
| *** ashleighfarnham has quit IRC | 22:22 | |
| *** dprince has quit IRC | 22:23 | |
| fungi | YorikSar: also we have a new service tracking individual tests by interpreting the subunit data and extracting failure rates and timing for them, though it's currently only exposed as a read-only mysql query socket | 22:23 |
| YorikSar | fungi: Oh, cool. Will try to find it there... | 22:23 |
| fungi | but that may be more granular than you need | 22:23 |
| *** ChuckC has joined #openstack-infra | 22:24 | |
| openstackgerrit | Merged openstack-dev/pbr: Don't avoid egg-info quite so often https://review.openstack.org/165511 | 22:24 |
| YorikSar | fungi: Wow... MySQL socket exposed to Internet?.. | 22:24 |
| YorikSar | fungi: Sounds dangerous | 22:24 |
| fungi | YorikSar: yep. we're brave like that | 22:25 |
| *** xyang1 has joined #openstack-infra | 22:25 | |
| mtreinish | YorikSar: the docs for the db service are here: http://docs.openstack.org/developer/subunit2sql/ | 22:25 |
| fungi | trust me we talked through all the security implications therein ;) | 22:25 |
| YorikSar | fungi: OpenStack Infra - bunch of cool brave people :) | 22:26 |
| mtreinish | fungi: heh, didn't we just say something like meh we'll throw it away if there's an issue :) | 22:26 |
| mordred | YorikSar: it's ... aggregated test data - if you hack it - we will have lost aggregated test run timing data - it's probably ok :) | 22:26 |
| mordred | yah | 22:26 |
| fungi | mtreinish: yep! | 22:26 |
| *** bknudson has quit IRC | 22:27 | |
| YorikSar | fungi: Oh, I don't need to trust you. It's you who needs to trust you ;) | 22:27 |
| mordred | YorikSar: well, I stopped doing thatyears ago | 22:27 |
| fungi | i try not to trust me | 22:27 |
| YorikSar | mordred: I'd prefer to mine some *coins on your MySQL cluster, why kill it? ;) | 22:27 |
| *** xyang1 has quit IRC | 22:28 | |
| *** xyang1 has joined #openstack-infra | 22:28 | |
| fungi | YorikSar: just cut us in on the take | 22:28 |
| clarkb | anteaya: yes session ids are returned | 22:29 |
| *** mtanino has joined #openstack-infra | 22:29 | |
| *** amitgandhinz has quit IRC | 22:29 | |
| cinerama | we should just create infracoin | 22:29 |
| zaro | anteaya, clarkb : https://review.openstack.org/166025 | 22:29 |
| fungi | cinerama: for all you know, we already have | 22:29 |
| zaro | anteaya: you need to be an admin to close connections though. | 22:29 |
| mordred | cinerama: your patch cannot land until it has mined a coin | 22:30 |
| anteaya | clarkb: great | 22:30 |
| zaro | anteaya, clarkb : tested and worked like a charm. | 22:30 |
| clarkb | jhesketh: fwiw I think we should move forward are using swift logs only for not dsvm jobs, and also add swift logs to the dsvm jobs but still copy to disk | 22:30 |
| cinerama | fungi: nice digital currency you've got there...it'd be a real shame if someone busted it up | 22:30 |
| anteaya | zaro: understood, but not at least an admin can close a connection, right now they can't | 22:30 |
| clarkb | jhesketh: then the last task will be sorting out the help footer | 22:30 |
| anteaya | zaro: awesome thank you | 22:30 |
| *** AJaeger has joined #openstack-infra | 22:30 | |
| *** AJaeger has quit IRC | 22:30 | |
| *** AJaeger has joined #openstack-infra | 22:30 | |
| YorikSar | Oh, it looks (from the first Google page) that noone thought about mining coins inside MySQL cluster. | 22:30 |
| clarkb | jhesketh: but everything continues to work with swift so keep pushing on it imo | 22:30 |
| anteaya | zaro: what is your position on getting this into gerrit on saturday? | 22:31 |
| *** xyang1 has quit IRC | 22:31 | |
| fungi | we have this new digital currency whereby service providers donate virtual machine resources and in return they get tested software they can use | 22:31 |
| mtreinish | YorikSar: it would probably just be easier to push up a self replicating patch to do that inside of gate jobs... :) | 22:31 |
| jhesketh | clarkb: sounds good to me. I need to get onto the footer stuff, but will push some more jobs to flick over soon | 22:31 |
| zaro | anteaya: ohh, wait up that was for 2.9 wasn't it? let me give it a try for 2.8 and see | 22:32 |
| YorikSar | mtreinish: Naaah, too easy :) | 22:32 |
| anteaya | zaro: I'm fine with either but if we can have it for 2.8 on saturday I'd love to try | 22:32 |
| anteaya | zaro: do you still have a 2.8 up for testing? | 22:32 |
| clarkb | hrm is this saturday that saturday? | 22:33 |
| *** patrickeast has quit IRC | 22:33 | |
| * clarkb may have made a scheduling fubar | 22:33 | |
| zaro | yep this is that | 22:33 |
| mordred | this saturday is trusty | 22:33 |
| anteaya | clarkb: this saturday is upgrade gerrit os to trusty yes | 22:33 |
| *** mjturek1 has quit IRC | 22:34 | |
| anteaya | clarkb: starting at 1500 | 22:34 |
| clarkb | derp, I am supposed to go learn to not kill babies on saturday | 22:34 |
| anteaya | clarkb: you were on errands | 22:34 |
| anteaya | clarkb: well that is important too | 22:34 |
| openstackgerrit | Zane Bitter proposed openstack-infra/elastic-recheck: Add fingerprint for bug 1433555 https://review.openstack.org/166026 | 22:34 |
| openstack | bug 1433555 in heat "functional tests are taking too long and cause false failures" [Critical,In progress] https://launchpad.net/bugs/1433555 - Assigned to Angus Salkeld (asalkeld) | 22:34 |
| mordred | clarkb: here's what you need to know: "don't kill them" | 22:34 |
| anteaya | clarkb: considering you have impending babies, I'd go with that as the higher priority | 22:34 |
| anteaya | clarkb: we have almost everyone else I think | 22:34 |
| pleia2 | yeah, we've got a full house | 22:34 |
| mordred | I might be drunk | 22:34 |
| clarkb | mordred: did duke already lose? | 22:35 |
| clarkb | >_> | 22:35 |
| * mordred throws an agitated emu at clarkb | 22:35 | |
| anteaya | how can you tell an emu is agitated | 22:35 |
| anteaya | I've looked at one | 22:35 |
| anteaya | how do you know? | 22:35 |
| *** boris-42 has joined #openstack-infra | 22:36 | |
| mordred | anteaya: more rapid hollow thumping noises | 22:36 |
| anteaya | ah | 22:36 |
| anteaya | this one made zero thumping noises | 22:36 |
| mordred | I believe emu's win the prize for one of the weirdest noises | 22:36 |
| *** VijayTripathi has quit IRC | 22:36 | |
| zaro | clarkb: if it's that cpr thing, i fogot it all like the next day | 22:37 |
| cinerama | kookaburras sound like random jungle noises from every movie ever | 22:37 |
| clarkb | zaro: its like a full day of everything | 22:37 |
| zaro | yeah, forgot it all | 22:37 |
| anteaya | mordred: really | 22:37 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add FindBugs support to Publishers https://review.openstack.org/166028 | 22:37 |
| anteaya | mordred: for me that is won by partriges | 22:38 |
| anteaya | clarkb: the biggest thing you learn is who to call and when | 22:38 |
| anteaya | clarkb: so you don't have to think about it when you have to call | 22:38 |
| *** gordc has quit IRC | 22:38 | |
| cinerama | cpr...there's some mnemonic...Airway Be Closing? | 22:38 |
| fungi | airways always be closing | 22:39 |
| jogo | https://review.openstack.org/#/c/165629/3 | 22:39 |
| jogo | This change depends on a change that failed to merge. | 22:39 |
| jogo | not sure what is failing there | 22:39 |
| anteaya | https://review.openstack.org/#/c/135768/ this isn't approved | 22:40 |
| jogo | is it because it is on top of a patch that has a depends-on in it | 22:40 |
| jogo | anteaya: right, not trying to merge anything per se, just have the tests run | 22:40 |
| pleia2 | cinerama: when you're testing, is your local.pp still the one I pastebined a while back? | 22:40 |
| mordred | jhesketh: oh - do you have powers to do magical things to people's accounts? | 22:41 |
| anteaya | jogo: oh depends-on and dependencies shouldn't prevent check jobs running on patches that need check jobs to run | 22:41 |
| jeblair | jogo: oh, you got that in the check pipeline? | 22:41 |
| cinerama | pleia2: lemme get what i currently have for you...i split out the mysql on request | 22:41 |
| jogo | jeblair: yup | 22:41 |
| *** erlon has quit IRC | 22:41 | |
| pleia2 | cinerama: right, thank you | 22:41 |
| jeblair | jogo: cool, we've been waiting for that to appear again since we turned on extra debugging | 22:41 |
| mordred | jhesketh: because Ng made a rackspace account, but they put him in the LON region which apparently has nothing to do with the rest of rackspace | 22:41 |
| *** patrickeast has joined #openstack-infra | 22:41 | |
| jogo | jeblair: o_O so your saying it may not be me doing something silly | 22:41 |
| mordred | jeblair: I don't know if you caught that this morning before you crisis-moded - but apparently rackspace LON does not work like DFW, IAD and ORD | 22:42 |
| mordred | jeblair: and it is not possible to use python-glanceclient with it | 22:42 |
| jeblair | jogo: yeah, is a bug somewhere i think; you can recheck for now, and i'll go see if we have logs that tell us what happened | 22:42 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add FindBugs support to Publishers https://review.openstack.org/166028 | 22:42 |
| jeblair | mordred: i guess we shouldn't ask for quota there then? :) | 22:42 |
| fungi | mordred: or presumably syd which is where i split half of my personal systems for global redundancy purposes | 22:42 |
| *** VijayTripathi has joined #openstack-infra | 22:42 | |
| fungi | syd and iad seem to work very similarly from what i've seen anyway | 22:43 |
| jogo | jeblair: thanks | 22:43 |
| cinerama | pleia2: something like http://paste.openstack.org/show/193725/ | 22:43 |
| pleia2 | cinerama: thanks | 22:43 |
| pleia2 | simple, I like it :) | 22:44 |
| cinerama | pleia2: the mysql module does funky things at the moment if you specify a blank password for the zanata db user | 22:44 |
| YorikSar | Ok, I'm lost... I'm trying to find job run time somewhere in "stats.timers.zuul.pipeline.check.job.check-tempest-dsvm-neutron-full.SUCCESS.*", but I don't understand those meters. | 22:44 |
| *** radez is now known as radez_g0n3 | 22:44 | |
| pleia2 | cinerama: yeah, I saw your comment in the review a couple days back, nice find | 22:45 |
| fungi | i think i got the foundation-board{,-confidential} ml split done correctly. after the list admin gets a chance to confirm, the foundation-board ml goes public with an (initially) empty archive | 22:45 |
| cinerama | pleia2: i mention it again because i don't want it to eat your afternoon :) | 22:45 |
| pleia2 | cinerama: much appreciated :) | 22:45 |
| anteaya | fungi: splitting mailing lists, was it fun? | 22:45 |
| fungi | anteaya: as fun as splitting firewood, but minus the blisters | 22:46 |
| anteaya | oh I like splitting firewood | 22:46 |
| anteaya | and yay for no blisters | 22:46 |
| fungi | i'm okay with splitting firewood except when you get a really dense hardwood like locust where the blade just bounces right back at you | 22:47 |
| anteaya | ah yes | 22:47 |
| anteaya | never swung at locust | 22:47 |
| anteaya | but ironwood bounces | 22:47 |
| mordred | fungi: I find the best way to split wood is to go to your neighborhood artisinal log store | 22:48 |
| mordred | fungi: you can get some lovely heirloom organic logs | 22:48 |
| zaro | anteaya: not so easy to get close-connection onto gerrit 2.8 | 22:48 |
| anteaya | with your tight pants | 22:48 |
| anteaya | zaro: not to worry | 22:49 |
| jhesketh | mordred: sadly no sorry. I'd be talking to support, likely like you are | 22:49 |
| zaro | anteaya: build errors, will continue to look at it. | 22:49 |
| fungi | mordred: manhattan never ceases to amaze me | 22:49 |
| anteaya | zaro: 2.9 is not far away | 22:49 |
| jhesketh | And yeah, lon is a little special | 22:49 |
| mordred | jhesketh: yah - that's already in the works | 22:49 |
| anteaya | zaro: don't hurt yourself, we can get by for another month or so | 22:49 |
| mordred | jhesketh: I just figured I never know what you have passwords to | 22:49 |
| anteaya | zaro: glad 2.9 was an easy cherry pick | 22:49 |
| anteaya | zaro: thank you | 22:49 |
| zaro | anteaya: ok, i won't bother then. | 22:49 |
| mordred | fungi: :) | 22:49 |
| clarkb | mordred: what are you londoning? | 22:49 |
| anteaya | zaro: yeah, don't bother | 22:49 |
| clarkb | last I looked into it I needed a new account entirel | 22:50 |
| jhesketh | mordred: yep fair question. I'm also happy to help bat for things as needed too | 22:50 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support to CFP in builders sections https://review.openstack.org/164504 | 22:50 |
| mordred | jhesketh: thanks! I'll let you know if we run in to any roadblocks - as soon as we get the account working, Ng can finish hacking on the systemd stuff for the nodepool-dib things for rax | 22:51 |
| fungi | clarkb: i think the idea was not to london anything | 22:51 |
| fungi | clarkb: but Ng got inadvertently londoned and can't break out | 22:51 |
| *** Somay has quit IRC | 22:51 | |
| anteaya | I've heard that about that place | 22:52 |
| anteaya | something about a tower | 22:52 |
| clarkb | Ng can't use our fine artisinal MURICAN VMs? | 22:52 |
| fungi | not even the organic ones | 22:52 |
| jeblair | clarkb: lol! | 22:52 |
| *** asettle has joined #openstack-infra | 22:53 | |
| jhesketh | mordred: sounds good | 22:53 |
| *** sputnik13 has quit IRC | 22:54 | |
| mordred | clarkb: the MURICAN VMs have the ssh port on the opposite side | 22:54 |
| *** mmedvede has quit IRC | 22:54 | |
| *** gordc has joined #openstack-infra | 22:54 | |
| mordred | clarkb: the ohio state vcu game is pretty good | 22:54 |
| fungi | mordred: and you don't "boot" murican vms, you trunk them | 22:55 |
| clarkb | fungi: ++ | 22:55 |
| *** sputnik13 has joined #openstack-infra | 22:55 | |
| * jeblair groans | 22:55 | |
| mordred | ++ | 22:55 |
| clarkb | that was good I laughed | 22:55 |
| mordred | fungi wins today | 22:55 |
| jeblair | that was good i groaned | 22:55 |
| * fungi fires off a groaner | 22:55 | |
| openstackgerrit | Khai Do proposed stackforge/python-jenkins: Add info on how to contribute https://review.openstack.org/166006 | 22:56 |
| clarkb | looks like the current top of stack heat test is giong to cuase a reset | 22:56 |
| clarkb | I am tempted to just shoot that test in the head early | 22:56 |
| jeblair | so what happened to our one hour budget anyway? | 22:56 |
| clarkb | jeblair: I have no idea | 22:56 |
| anteaya | clarkb: no | 22:56 |
| anteaya | rargh | 22:56 |
| openstackgerrit | Min Pae proposed openstack-infra/system-config: adding openstack-cue to eavesdrop https://review.openstack.org/166033 | 22:56 |
| anteaya | ttx needs the nova patch next in line, which has completed successfully | 22:57 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Add standard openstack info on how to contribute https://review.openstack.org/166008 | 22:57 |
| jeblair | sdague: was there a conscious decision to abandon the 1-hour budget? | 22:57 |
| *** tiswanso has joined #openstack-infra | 22:57 | |
| anteaya | boo | 22:57 |
| clarkb | https://review.openstack.org/#/c/154325/ bumped it | 22:57 |
| *** Sukhdev has quit IRC | 22:57 | |
| clarkb | I reviewed it and didn't catchthe affect it would have | 22:58 |
| clarkb | merely pointed out how to make the timeout more better :/ | 22:58 |
| jeblair | well, i mean we just approved a 1.5 hour trove change yesterday too | 22:58 |
| *** dannywilson has quit IRC | 22:58 | |
| *** gordc has quit IRC | 22:58 | |
| *** arxcruz has joined #openstack-infra | 22:59 | |
| *** hodos has quit IRC | 22:59 | |
| clarkb | greghaynes: btw one good thing that did happen today is we deployed your nodepool fix | 22:59 |
| clarkb | I did not get around to landing the nodepool image build logging change though and would rather avoid that right now simply because hair on fire | 22:59 |
| *** bswartz has joined #openstack-infra | 23:00 | |
| openstackgerrit | Min Pae proposed openstack-infra/project-config: Add openstack-cue to gerritbot https://review.openstack.org/158011 | 23:00 |
| *** vipul is now known as vipuls | 23:00 | |
| clarkb | anteaya: to make that test failure even more awesome it failed due to an internal timeout | 23:01 |
| fungi | jeblair: yep, though that trove job was already at 1.5 hours and we bumped it up an extra 10 minutes or something | 23:01 |
| *** ddieterl_ has quit IRC | 23:01 | |
| clarkb | anteaya: took longer than 1200 seconds | 23:01 |
| anteaya | clarkb: wooooo | 23:01 |
| anteaya | love that | 23:01 |
| clarkb | which I think is what the bug above that zaneb's change points out is about | 23:01 |
| jeblair | stevebaker: hi, what would it take to get the heat functional job down to about an hour? | 23:02 |
| *** oomichi has quit IRC | 23:02 | |
| jeblair | stevebaker: is there something that could be more efficient, or more paralellized, or can we split it into two jobs? | 23:02 |
| fungi | looks like they're trying to address it via https://review.openstack.org/166030 | 23:03 |
| harlowja_ | refrain refrain! | 23:03 |
| harlowja_ | lol | 23:03 |
| jeblair | fungi, stevebaker: oh cool :) | 23:04 |
| mordred | clarkb: overtime | 23:04 |
| clarkb | mordred: what channel do I need to turn on? | 23:04 |
| *** bhunter71 has quit IRC | 23:04 | |
| clarkb | fungi: that change hasn't run that job yet but should tell us soonish I guess | 23:04 |
| mordred | clarkb: tnt | 23:04 |
| fungi | indeed | 23:04 |
| clarkb | hrm no more dprince | 23:04 |
| *** andreykurilin_ has quit IRC | 23:05 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: For neutron and neutron-lbaas, skip more wasted jobs https://review.openstack.org/166035 | 23:05 |
| *** pritesh has quit IRC | 23:05 | |
| clarkb | I would sggest just turning off the rh1 region but there is no fallback so meh | 23:05 |
| *** dimtruck is now known as zz_dimtruck | 23:05 | |
| ianw | clarkb: np; the config-validation jobs are also ready to go i think | 23:05 |
| fungi | harlowja_: I OK kook i rigor w assessment of ofiii | 23:05 |
| harlowja_ | lol | 23:05 |
| *** asettle has quit IRC | 23:05 | |
| harlowja_ | what u say about my mom | 23:05 |
| harlowja_ | how dare u | 23:05 |
| fungi | i tried to run it through rot-13 but to no avail | 23:05 |
| *** asettle has joined #openstack-infra | 23:06 | |
| clarkb | fungi: I prefer rot26 | 23:06 |
| clarkb | easier to read | 23:06 |
| fungi | clarkb: yeah, i think that e-mail was drunken keyboard mashing run through rot-26 | 23:06 |
| *** arxcruz has quit IRC | 23:06 | |
| fungi | clarkb: did my change to turn off tripleo-hp1 finally merge? | 23:07 |
| fungi | ahh, yep, earlier today looks like\ | 23:07 |
| mordred | fungi: yup | 23:07 |
| *** jamespage has quit IRC | 23:08 | |
| *** gnuoy has quit IRC | 23:08 | |
| mordred | fungi: turns out nodepool alien-list with it still in there was UNHAPPY | 23:08 |
| *** gnuoy has joined #openstack-infra | 23:08 | |
| *** jamespage has joined #openstack-infra | 23:08 | |
| fungi | mordred: yep, that was one of the reasons i submitted that change | 23:09 |
| mordred | fungi: I also submitted a nodepool patch to not be unhappy on that command if a cloud is unhappy | 23:09 |
| mordred | fungi: but it's been a day - so I have not followed up on it | 23:09 |
| fungi | oh, great idea | 23:09 |
| clarkb | add a test please :) | 23:10 |
| anteaya | yes test please | 23:10 |
| *** bookwar has left #openstack-infra | 23:10 | |
| clarkb | we actually have some decent testing going on now | 23:10 |
| anteaya | yay testing | 23:10 |
| clarkb | oh this game is at the rose garden | 23:11 |
| mordred | clarkb: yah. you have rounds there in town | 23:11 |
| fungi | i've been forcing myself to add tests for each of the new bindep features i committed too, since it unsurprisingly started out with a decent testing framework and i didn't want to regress it | 23:11 |
| anteaya | nice work | 23:12 |
| mordred | clarkb: so the real question - is harvard going to beat unc? | 23:12 |
| anteaya | that heat test is still running | 23:12 |
| jeblair | okay, so i've confirmed that a gerrit query for a change issued immediately after a patchset is created may lack depends-on information | 23:13 |
| anteaya | started at 21:08 | 23:13 |
| clarkb | jeblair huh | 23:13 |
| mordred | OH | 23:13 |
| anteaya | finished | 23:13 |
| mordred | jeblair: aroo? | 23:13 |
| jeblair | so the error that jogo (and sdague before him) reported about changes in check being ejected because they depend on a change that failed to merge is because later on someone comments on a change and zuul updates information for it, and the second time sees the dependency that was not originally there | 23:13 |
| anteaya | Build timed out (after 125 minutes). Marking the build as failed. | 23:14 |
| mordred | jeblair: that must have been fascinting to track | 23:14 |
| *** tiswanso_ has joined #openstack-infra | 23:15 | |
| *** tiswanso has quit IRC | 23:15 | |
| jeblair | mordred, clarkb: it may happen when a patch series is being updated simultaneously | 23:15 |
| clarkb | jeblair does it see the old patch of the dep or no dep at all? | 23:16 |
| fungi | jeblair: so you were able to diagnose that with the additional logging then? | 23:16 |
| jeblair | clarkb: the dependsOn section is entirely missing from the returned data | 23:16 |
| jeblair | fungi: yeah, logging all of the traffic to/from gerrit | 23:17 |
| *** aysyd has quit IRC | 23:17 | |
| fungi | huh, so gerrit's omitting that from the event sometimes? | 23:18 |
| jeblair | yeah, i suspect it either hasn't received the other commit, or processed it, or something. | 23:18 |
| fungi | that seems like it's going to be weird to work around | 23:18 |
| jeblair | i can think of a thorough-and-slow way and a good-enough-and-fast way... | 23:19 |
| jeblair | the thorough way would be to query for the parent commit sha, and if it is not a merged change, keep waiting for a depends on to show up. | 23:19 |
| jeblair | the good-enough way would be to sleep for a few seconds :) | 23:20 |
| anteaya | ha ha ha | 23:20 |
| clarkb | this is thr gerrit internal depends on ya? | 23:20 |
| fungi | that's how i interpreted it | 23:21 |
| fungi | not our commit message header | 23:21 |
| jeblair | correct, only talking about git-depends here | 23:21 |
| fungi | so, yeah, the sleep doesn't seem all that bad | 23:21 |
| *** ghostpl_ has joined #openstack-infra | 23:21 | |
| *** otter768 has joined #openstack-infra | 23:22 | |
| *** tjones1 has quit IRC | 23:22 | |
| fungi | it's already a rare enough occurrence | 23:22 |
| fungi | i've only seen it reported a few times in the past couple weeks | 23:22 |
| fungi | so if that gets rid of most of the incidents, then it'll be rare enough we just don't care | 23:23 |
| fungi | i wouldn't be surprised if it only happens when gerrit's really busy | 23:23 |
| *** stevemar has quit IRC | 23:23 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Promote lbaasv2 api tempest tests to voting for neutron and neutron-lbaas https://review.openstack.org/166038 | 23:24 |
| *** kutija has quit IRC | 23:24 | |
| *** _nadya_ has joined #openstack-infra | 23:25 | |
| *** ghostpl_ has quit IRC | 23:26 | |
| *** otter768 has quit IRC | 23:27 | |
| ianw | do we have issues with symlinks in repos? | 23:27 |
| *** chlong has joined #openstack-infra | 23:27 | |
| fungi | ianw: packaging them is problematic | 23:27 |
| ianw | particularly i'm wondering if symlinking top-level HACKING.rst into the docs directory is ok | 23:27 |
| fungi | ianw: you can already tell sphinx to include files from the top-level directory. we have at least some packages which do that for their readme | 23:29 |
| ianw | ahh, yeah, that's a better path to take | 23:29 |
| fungi | i don't know an example off the top of my head, but someone else might | 23:29 |
| *** bdpayne has quit IRC | 23:30 | |
| *** _nadya_ has quit IRC | 23:30 | |
| *** camunoz has quit IRC | 23:32 | |
| mordred | jeblair: ah - I totally thought you were talking about Depends-On footers | 23:33 |
| *** stevemar has joined #openstack-infra | 23:33 | |
| mordred | ianw: we have people doing symlinks | 23:33 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add 'Job Log Logger' wrapper https://review.openstack.org/164338 | 23:34 |
| mordred | openstack-infra/infra-specs:doc/source is a | 23:34 |
| mordred | great place to look | 23:34 |
| *** MarkAtwood has quit IRC | 23:34 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Remove nonexistent servername and cleanup build publisher https://review.openstack.org/156091 | 23:34 |
| *** garyh has joined #openstack-infra | 23:34 | |
| fungi | mordred: is it mostly symlinking directories and/or python files which gets hairy for packaging? | 23:35 |
| *** dannywilson has joined #openstack-infra | 23:35 | |
| mordred | fungi: I think it's mostly the python files | 23:36 |
| fungi | ianw: though poking around, you can apparently do: | 23:38 |
| fungi | .. include:: ../../HACKING.rst | 23:38 |
| *** tonytan4ever has quit IRC | 23:38 | |
| fungi | which will cause it to just slurp that content into your doc/source/hacking.rst or whatever | 23:39 |
| *** tjones1 has joined #openstack-infra | 23:39 | |
| *** stevemar has quit IRC | 23:40 | |
| mtreinish | fungi: yeah that's what I think everyone does. Some projects used to use symlinks but there was some reason we stopped | 23:40 |
| mtreinish | I can't remember exactly why though | 23:40 |
| fungi | which is apparently slightly more friendly to our comrades on filesystems which lack symlinks | 23:40 |
| mtreinish | fungi: bah, I think we should just insert arbitrary symlinks into all repos | 23:41 |
| clarkb | symlinks break distutils or setuptools or both | 23:41 |
| *** garyh has quit IRC | 23:42 | |
| mordred | fungi, jeblair, clarkb: we may be about to get a request to turn off HP for a bit to help them recover | 23:43 |
| *** tjones1 has quit IRC | 23:44 | |
| fungi | wow! | 23:44 |
| *** camunoz has joined #openstack-infra | 23:44 | |
| greghaynes | hah | 23:45 |
| clarkb | mordred it is effectively off | 23:45 |
| mordred | yup. they would like us to stop hitting their API for a bit | 23:45 |
| mordred | clarkb: no it's not | 23:45 |
| *** radez_g0n3 is now known as radez | 23:45 | |
| mordred | it's continually hitting their api asking for things | 23:45 |
| clarkb | well ya but its not actually doing anything | 23:46 |
| fungi | hpcloud is effectively off for us, we're not effectively off for hpcloud | 23:46 |
| mordred | right. what fungi said | 23:46 |
| ianw | clarkb: many things can disturb the distutils and setuptools gods. i bet someone didn't make their sacrifice and now they've smote hp cloud | 23:46 |
| mordred | they're having an API layer disaster at the moment | 23:46 |
| fungi | hpcloud may _also_ be effectively off because of us. who knows | 23:46 |
| *** ajmiller has quit IRC | 23:46 | |
| anteaya | ianw: I haven't seen a smoting in centuries | 23:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Turn off HP Public Cloud https://review.openstack.org/166043 | 23:48 |
| mordred | fungi, clarkb, jeblair, pleia2, jhesketh: ^^ | 23:48 |
| anteaya | reviewed | 23:49 |
| pleia2 | sad trombone | 23:49 |
| mordred | jeblair: do you have local changes applied? | 23:49 |
| *** derekh has joined #openstack-infra | 23:50 | |
| *** mfink_ has quit IRC | 23:51 | |
| openstackgerrit | Merged openstack-infra/system-config: Turn off HP Public Cloud https://review.openstack.org/166043 | 23:52 |
| clarkb | might also be worth pointing out that they don't need us to do that for them... | 23:52 |
| clarkb | but maybe firewalling off nodepool.o.o isn't a nice thing to do? | 23:52 |
| *** derekh has quit IRC | 23:52 | |
| SpamapS | well on the bright side if there's no more changes to make hpcloud work, that's more time we can spend making the new cloud ;) | 23:53 |
| anteaya | well when I deal with third party ci, I always consider it polite if they turn themselves off | 23:53 |
| anteaya | rather than forcing me to shut them down, as I am slow to turn them back on | 23:53 |
| clarkb | anteaya: sure, happy to do it (and have done it many times before) | 23:53 |
| anteaya | yup | 23:53 |
| clarkb | for all the clouds too | 23:53 |
| anteaya | so hoping they sort this out | 23:54 |
| mordred | jeblair: ok - puppet is disabled on nodepool I believe because you have local changes | 23:54 |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Bump tempest-lib min version https://review.openstack.org/166044 | 23:54 |
| *** mtanino has quit IRC | 23:54 | |
| clarkb | mordred: jeblair probably the change to nodepool to do the requests at the rate limit? | 23:54 |
| mordred | clarkb, jeblair: I applied the quota changes to nodepool.yaml by hand | 23:55 |
| mordred | clarkb: it's been a while - is there a better way to get nodepool to pick up nodepool.yaml changes? | 23:55 |
| clarkb | mordred: no, next time through the run loop it will get them | 23:56 |
| fungi | mordred: it automagically sees them | 23:56 |
| mordred | clarkb: awesome | 23:56 |
| mordred | thanks | 23:56 |
| clarkb | mordred: maybe we should be looking at spreading our across east and west? can we ask about using west again? | 23:58 |
| *** garyh has joined #openstack-infra | 23:59 | |
| mordred | clarkb: I will ask about that post-crisis | 23:59 |
| clarkb | s/our/out/ | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!