| *** alexpilotti has quit IRC | 00:03 | |
| *** cyeoh is now known as c_yoyo | 00:05 | |
| fungi | pabelanger: i think we're all scared to approve nodepool and zuul changes jeblair hasn't reviewed while he's still at pycon, lest we anger the computer gods | 00:09 |
|---|---|---|
| pabelanger | fungi: Ah, fair enough. Didn't see he was in MTL. | 00:10 |
| clarkb | http://paste.openstack.org/show/75513/ I present to an almost working simple makefile | 00:14 |
| clarkb | note the dependency resolution for venvs isn't quite right and it doesn't do everything like tox. It will fail fast rather than running all the things | 00:15 |
| fungi | clarkb: i tip my hat to you | 00:15 |
| clarkb | if someone can see why .venv/$(VNAME) deps seem to be met if and .venv/$(VNAME) exists I would be so happy but I have ot run and pick up new glasses now | 00:15 |
| fungi | clarkb: you can solve the rest with setuptools, i swear (and if you can't, it'll be amusing to watch anyway) | 00:15 |
| fungi | s/setuptools/autotools/ | 00:15 |
| * fungi inadvertently mixes his environments this time of night | 00:16 | |
| clarkb | I don't even think autotools is super necessary. But I may be wrong | 00:16 |
| clarkb | anyways that does work if I make clean between target changes | 00:16 |
| fungi | no, it's not. i was just trying to add to the absurd | 00:16 |
| clarkb | :P | 00:17 |
| fungi | that however is a remarkably succinct makefile | 00:17 |
| clarkb | ya what we do with tox isn't terribly complicated | 00:17 |
| clarkb | virtualenv ends up doing most of the tox like work for us | 00:18 |
| *** CaptTofu has joined #openstack-infra | 00:19 | |
| * fungi welcomes our new "make all" overlords | 00:19 | |
| clarkb | mordred: ^ please apply your make wisdom to that file | 00:19 |
| clarkb | note that isn't windows friendly | 00:20 |
| clarkb | ok really need to go get glasses /me & | 00:20 |
| fungi | clarkb: go | 00:21 |
| fungi | as for deps being considered met when (file|dir) exists, that's an optimization of make unless i'm misunderstanding you | 00:21 |
| fungi | e.g. don't recompile foo.c if foo.o exists | 00:22 |
| clarkb | yup but it should stat the subdir right? | 00:22 |
| clarkb | it seems that .venv satisfies .venv/py27 | 00:22 |
| fungi | oh... i'll ponder that a little longer | 00:22 |
| fungi | anyway, go obtain replacement bionic eyes | 00:23 |
| *** sabari has quit IRC | 00:23 | |
| *** sabari has joined #openstack-infra | 00:23 | |
| *** matsuhashi has joined #openstack-infra | 00:24 | |
| fungi | ugh... just booted a debian vm in my rax account and note that the sources.list on it also omits security.debian.org | 00:25 |
| * fungi sighs | 00:25 | |
| fungi | people were even complaining about it this week on the debian-cloud ml | 00:26 |
| *** jhesketh_ has joined #openstack-infra | 00:26 | |
| *** jhesketh_ has left #openstack-infra | 00:26 | |
| clarkb | ya I had to edit my sources on my irc box | 00:27 |
| *** krtaylor has joined #openstack-infra | 00:28 | |
| fungi | apparently amazon did the same on their images | 00:28 |
| *** thuc has quit IRC | 00:30 | |
| *** thuc has joined #openstack-infra | 00:30 | |
| *** khyati_ has quit IRC | 00:31 | |
| *** weshay has quit IRC | 00:32 | |
| *** thuc has quit IRC | 00:35 | |
| *** c_yoyo is now known as cyeoh | 00:36 | |
| *** cyeoh is now known as cyoyo | 00:37 | |
| *** sabari has quit IRC | 00:41 | |
| *** thomasem_ has joined #openstack-infra | 00:41 | |
| *** ianw has quit IRC | 00:44 | |
| *** praneshp has joined #openstack-infra | 00:44 | |
| *** ianw has joined #openstack-infra | 00:44 | |
| *** thomasem has quit IRC | 00:44 | |
| praneshp | Hi, is the gate-entropy-python33 check broken? | 00:45 |
| *** asettle has joined #openstack-infra | 00:45 | |
| praneshp | a couple of my jobs are stuck in that one check | 00:45 |
| *** thomasem_ has quit IRC | 00:45 | |
| *** SumitNaiksatam has joined #openstack-infra | 00:46 | |
| openstackgerrit | A change was merged to openstack-infra/config: Enable baremetal scenario test in Ironic's d-s-g https://review.openstack.org/85227 | 00:47 |
| *** asettle is now known as settledown | 00:47 | |
| *** marun has joined #openstack-infra | 00:48 | |
| *** settledown is now known as asettle | 00:48 | |
| *** asettle is now known as SettleDown | 00:48 | |
| fungi | praneshp: it looks like nodepoold still isn't trying to build any py3k-precise nodes at all. i'm checking its logs now to see if i can tell why | 00:49 |
| clarkb | praneshp: there may be node contention. we had to remove hp slaves | 00:49 |
| praneshp | clarkb: cool. It looked like an outage or something to me because its 5 hours now. | 00:49 |
| fungi | clarkb: have you gotten new eyes yet? | 00:49 |
| praneshp | thanks fungi | 00:49 |
| clarkb | or that /me gets out of fungi's way | 00:50 |
| fungi | yeah, we have absolutely no py3k-precise nodes in any state now that i've deleted the handful stuck in a building state in hpcloud regions a while ago | 00:50 |
| clarkb | fungi yes now dinner and eyes adjusting | 00:51 |
| *** tianst20 has joined #openstack-infra | 00:51 | |
| praneshp | thanks fungi and clarkb | 00:51 |
| fungi | i think maybe nodepoold does get confused when sum(min-ready)>max-servers | 00:52 |
| fungi | i'm going to stop puppet agent on nodepool.o.o for a few minutes to experiment | 00:53 |
| clarkb | k | 00:54 |
| *** Ryan_Lane1 has joined #openstack-infra | 00:54 | |
| fungi | just spotted the issue | 00:54 |
| *** tianst20 has quit IRC | 01:00 | |
| *** yamahata has joined #openstack-infra | 01:01 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Revert "Temporarily stop using rax for py3k-precise nodes" https://review.openstack.org/86752 | 01:01 |
| fungi | praneshp: clarkb: ^ that ;) | 01:01 |
| fungi | cramming through | 01:02 |
| praneshp | ha, first openstack-infra review for me :) | 01:02 |
| clarkb | fungi oh lol | 01:02 |
| fungi | (a revert in name only, given that the entire configuratio has been refactored out from under it since the introduction of nodepool labels) | 01:03 |
| fungi | i may need to force some image updates for those too... checking now | 01:03 |
| *** amcrn has quit IRC | 01:04 | |
| *** ilives has joined #openstack-infra | 01:04 | |
| fungi | nope. nodepoold has been dilligently building completely unused images there the entire time. hopefully they still work | 01:04 |
| *** cody-somerville has joined #openstack-infra | 01:04 | |
| *** cody-somerville has quit IRC | 01:04 | |
| *** cody-somerville has joined #openstack-infra | 01:04 | |
| *** dcramer_ has joined #openstack-infra | 01:05 | |
| *** ilives has quit IRC | 01:05 | |
| *** marun has quit IRC | 01:07 | |
| *** Ryan_Lane has quit IRC | 01:09 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Temporarily stop using rax for py3k-precise nodes" https://review.openstack.org/86752 | 01:10 |
| *** yamahata has quit IRC | 01:10 | |
| *** yamahata has joined #openstack-infra | 01:10 | |
| fungi | praneshp: clarkb: JayF: i have 10 py3k-precise nodes building in nodepool now | 01:12 |
| fungi | hopefully the congestion will be lifted in short order | 01:13 |
| *** UtahDave has quit IRC | 01:15 | |
| *** ilives has joined #openstack-infra | 01:15 | |
| *** Ryan_Lane has joined #openstack-infra | 01:15 | |
| *** mriedem has quit IRC | 01:18 | |
| *** yamahata has quit IRC | 01:19 | |
| praneshp | thanks fungi | 01:21 |
| fungi | yep, looks like they're running now | 01:24 |
| fungi | and up to 40 py3k-precise nodes in various states now | 01:25 |
| *** melwitt has quit IRC | 01:25 | |
| praneshp | is there a way to see job-wise queues? | 01:26 |
| praneshp | for example all queued jobs for py-33? | 01:26 |
| fungi | praneshp: you could probably digest http://zuul.openstack.org/status/json into whatever orientation you want with a little bit of python | 01:27 |
| praneshp | great, thanks | 01:28 |
| fungi | that's the data the status page at http://status.openstack.org/zuul/ is built from | 01:28 |
| *** SettleDown has quit IRC | 01:35 | |
| *** Ryan_Lane has quit IRC | 01:37 | |
| *** praneshp has quit IRC | 01:37 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Minor README.rst tweaks https://review.openstack.org/86464 | 01:39 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Fix capitalization of my gerrit username https://review.openstack.org/86142 | 01:39 |
| *** thuc has joined #openstack-infra | 01:41 | |
| *** thuc has quit IRC | 01:45 | |
| *** cody-somerville has quit IRC | 01:49 | |
| *** nosnos has joined #openstack-infra | 01:51 | |
| jogo | fungi: I am not sure either, but I figured I want version x or y and thats it | 01:52 |
| fungi | weeeeeird that it actually works | 01:54 |
| *** esker has joined #openstack-infra | 01:55 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Add monotonic time library for python 2.x https://review.openstack.org/86757 | 01:55 |
| jogo | fungi: I figure comma is like an or operator | 01:56 |
| fungi | except if "," is "or" then the "!" negation would be a no-op | 01:57 |
| fungi | not to mention other comparative operators like < and > and their inclusive endpoint variants | 01:58 |
| *** thomasem has joined #openstack-infra | 01:59 | |
| fungi | i mean reading ">1,<2" as ">1 or <2" is nonsense | 02:00 |
| *** saschpe has quit IRC | 02:00 | |
| fungi | given that every member of the universal set is greater than one or less than two | 02:01 |
| *** esker has quit IRC | 02:01 | |
| jogo | fungi: hmm you make a good point, I don't know what to say | 02:02 |
| *** esker has joined #openstack-infra | 02:02 | |
| *** saschpe has joined #openstack-infra | 02:02 | |
| fungi | i am merely a victim of having taken logic in multiple departments while at university | 02:04 |
| *** thomasem has quit IRC | 02:04 | |
| fungi | i found the overlap between mathematics and philosophy simultaneously disturbing and comforting | 02:05 |
| *** asettle has joined #openstack-infra | 02:05 | |
| *** asettle has quit IRC | 02:05 | |
| StevenK | fungi: Strange use of the word 'victim' :-P | 02:05 |
| *** alex-away is now known as asettle | 02:06 | |
| fungi | StevenK: rational though has been thrust upon me through no fault of my own | 02:06 |
| fungi | s/though/thought/ | 02:06 |
| StevenK | Haha | 02:06 |
| *** esker has quit IRC | 02:06 | |
| * fungi attempts to escape back into fantasy for a little while | 02:06 | |
| jogo | fungi: ^_^ | 02:09 |
| jogo | the requirements jobs are backed up on pypy http://status.openstack.org/zuul/ | 02:10 |
| *** asettle is now known as SettleDown | 02:11 | |
| *** mwagner_lap has joined #openstack-infra | 02:11 | |
| fungi | yeah, see scrollback just above... should clear up soonish | 02:12 |
| *** yamahata has joined #openstack-infra | 02:15 | |
| *** yaguang has joined #openstack-infra | 02:15 | |
| jogo | ahh | 02:15 |
| jogo | thanks | 02:15 |
| *** gyee has quit IRC | 02:17 | |
| StevenK | fungi: I'm guessing the tripleo image build finished sometime while you were unconscious? | 02:17 |
| fungi | StevenK: shortly after, gauging from the ages i saw on the newest images when i awoke | 02:17 |
| StevenK | fungi: Ah, excellent. | 02:18 |
| StevenK | Let's rebase and review and see if I'm loved now | 02:18 |
| *** coolsvap|afk is now known as coolsvap | 02:18 | |
| *** wendar_ is now known as wendar | 02:21 | |
| *** yamahata has quit IRC | 02:22 | |
| *** CaptTofu has quit IRC | 02:25 | |
| *** thomasem has joined #openstack-infra | 02:25 | |
| *** dims has quit IRC | 02:26 | |
| *** thomasem has quit IRC | 02:29 | |
| *** rlandy has quit IRC | 02:30 | |
| *** zehicle has joined #openstack-infra | 02:31 | |
| fungi | jogo: sorry... http://logs.openstack.org/96/85496/4/check/gate-requirements-python27/f891c66/console.html#_2014-04-10_23_47_52_855 | 02:34 |
| fungi | flake8==2.0 | 02:34 |
| fungi | pep8==1.4.5 | 02:34 |
| fungi | pyflakes==0.7.3 | 02:34 |
| *** zehicle_at_dell has quit IRC | 02:35 | |
| fungi | so it doesn't all asplodey, but it also doesn't test with the later versions | 02:35 |
| * StevenK grumbles at more failures | 02:36 | |
| jogo | fungi: so how is the right way to do it? | 02:36 |
| fungi | jogo: just increase the cap to allow the newer versions you want? | 02:37 |
| jogo | we want version a or b | 02:37 |
| fungi | i'm pretty sure you want something which isn't pip in that case | 02:37 |
| jogo | fungi: I think it actually is working | 02:40 |
| jogo | at least with the order reversed :) | 02:41 |
| fungi | jogo: well, yes, the change isn't self-testing that the latest specified version is installable however | 02:41 |
| *** unicell has quit IRC | 02:42 | |
| fungi | jogo: and the mirror is going to mirror the version that pip installs, which won't be those newer versions you're wanting | 02:42 |
| *** Loquacities is now known as lanabanana | 02:42 | |
| jogo | fungi: http://paste.openstack.org/show/75515/ | 02:43 |
| *** jgrimm has joined #openstack-infra | 02:44 | |
| *** david-lyle has joined #openstack-infra | 02:44 | |
| fungi | jogo: oh, it just dawned on me, that job was using the versions specified in the test-requirements.txt, not global-requirements.txt | 02:44 |
| jogo | hehe | 02:45 |
| jogo | which is hacking | 02:45 |
| fungi | jogo: but yeah, if we have flake8==2.0,==2.1 in global-reqs, pypi-mirror is going to download and cache 2.0, so we'll still not have 2.1 in the mirror | 02:45 |
| *** rfolco has joined #openstack-infra | 02:45 | |
| *** yjiang has joined #openstack-infra | 02:46 | |
| jogo | fungi: why won't pypi-mirror do that? | 02:46 |
| jogo | it works as a pip req | 02:46 |
| *** CaptTofu has joined #openstack-infra | 02:46 | |
| fungi | jogo: your paste demonstrates it | 02:46 |
| fungi | pip downloaded 2.0, not 2.1 | 02:46 |
| jogo | that was against our pypi mirror | 02:46 |
| fungi | oh, gotcha | 02:47 |
| jogo | FWIW I am just as confused as you | 02:47 |
| jogo | http://paste.openstack.org/show/75516/ | 02:47 |
| fungi | if you don't override PIP_INDEX_URL you get 2.1? | 02:47 |
| jogo | (venv)jogo@lappy:~/Develop/venv$ pip freeze | grep flake8 | 02:47 |
| jogo | flake8==2.1.0 | 02:47 |
| jogo | yup | 02:47 |
| fungi | indeed | 02:48 |
| jogo | strange how 'logic'works | 02:48 |
| *** signed8bit has quit IRC | 02:48 | |
| fungi | so it has been suggested that pip works in a last-match fashion most of the time | 02:48 |
| *** yamahata has joined #openstack-infra | 02:48 | |
| fungi | perhaps this is what we're seeing demonstrated in that case | 02:48 |
| jogo | flake8==2.0,==2.1 | 02:49 |
| jogo | is the rule I was testing | 02:49 |
| *** esker has joined #openstack-infra | 02:49 | |
| dstufft | I need to get around to fixing our version selector :[ | 02:50 |
| fungi | separate the components of the version spec at the commas... for each component if you have an existing install which doesn't meet the requirement uninstall it, then select the highest version matching that comparison. repeat for each additional element in the comma-separated list? | 02:50 |
| fungi | violates principle of least surprise, but then again if you're easily surprised, pip is not the tool for you ;) | 02:51 |
| jogo | fungi: :) | 02:51 |
| jogo | fungi: so now that i can prove my patch works, even though it shouldn't want to +2 it? | 02:52 |
| jogo | https://review.openstack.org/#/c/85496/ | 02:52 |
| fungi | jogo: shall i assume the next patch will be coming along right behind it to remove the first == components of those once the mirror updates? | 02:53 |
| fungi | and that this is a two-step update purely to work around the co-gating/mirror-update race? | 02:54 |
| jogo | so that would work actually, I was going to switch the versions over but clarkb didn't want a backwards incompatible change | 02:55 |
| jogo | fungi: let me post that patch next -- with a note saying not to approve right away | 02:55 |
| jogo | before landing this one | 02:55 |
| fungi | right. backwards-incompat change causes changes behind you in the gate (and in check once you merge) to fail until the mirror has a working package set meeting the new reqs | 02:55 |
| fungi | anyway, going ahead and approving this one, then taking a break for the evening | 02:56 |
| jogo | hmm I think I should have added pep8 1.45, 1.46 and 1.5 in this but didn' think it would take so long to land | 02:57 |
| jogo | going to add 1.5 in the next patch | 02:57 |
| fungi | s/==1.45,==1.46/==1.46,==1.5/ in the next change then? | 02:58 |
| jogo | yup | 02:58 |
| jogo | not sure what version of 1.5 to put in though | 02:58 |
| jogo | a new one was just posted today 1.5.5 | 02:58 |
| jogo | sigh pep8 is soo buggy | 02:58 |
| jogo | and afraid to add in 1.5 | 02:59 |
| fungi | understandable | 02:59 |
| *** yamahata has quit IRC | 03:04 | |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Remove old pep8, pyflakes and flake8 versions https://review.openstack.org/86759 | 03:04 |
| *** yamahata has joined #openstack-infra | 03:04 | |
| *** esker has quit IRC | 03:08 | |
| *** matsuhashi has quit IRC | 03:14 | |
| *** harlowja is now known as harlowja_away | 03:14 | |
| *** CaptTofu has quit IRC | 03:16 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Use pep8's noqa argument https://review.openstack.org/86199 | 03:19 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Speed up license checks https://review.openstack.org/86726 | 03:19 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Move hacking to pep8 1.5.4 https://review.openstack.org/86197 | 03:19 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Only compile regex for hacking_except_format once https://review.openstack.org/86695 | 03:19 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Move except checks into there own module https://review.openstack.org/86694 | 03:19 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Move comment checks into there own module https://review.openstack.org/86693 | 03:19 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Make test_doctest use entry points https://review.openstack.org/86692 | 03:19 |
| jogo | rebase ^ | 03:19 |
| *** nosnos has quit IRC | 03:20 | |
| *** nosnos has joined #openstack-infra | 03:20 | |
| *** zns has joined #openstack-infra | 03:22 | |
| *** zns has joined #openstack-infra | 03:24 | |
| *** nosnos has quit IRC | 03:25 | |
| *** matsuhashi has joined #openstack-infra | 03:28 | |
| *** msabramo has joined #openstack-infra | 03:37 | |
| openstackgerrit | A change was merged to openstack/requirements: Revert "add mirror requirements list" https://review.openstack.org/86707 | 03:40 |
| *** david-lyle has quit IRC | 03:43 | |
| *** coolsvap is now known as coolsvap|afk | 03:46 | |
| *** wchrisj has joined #openstack-infra | 03:47 | |
| *** wchrisj has quit IRC | 03:47 | |
| *** chandan_kumar has joined #openstack-infra | 03:48 | |
| *** esker has joined #openstack-infra | 03:48 | |
| *** unicell has joined #openstack-infra | 03:51 | |
| *** esker has quit IRC | 03:53 | |
| *** wchrisj has joined #openstack-infra | 03:55 | |
| *** rfolco has quit IRC | 03:59 | |
| *** coolsvap|afk is now known as coolsvap | 03:59 | |
| openstackgerrit | A change was merged to openstack/requirements: Bump pep8, pyflakes and flake8 versions https://review.openstack.org/85496 | 04:06 |
| openstackgerrit | Jenkins proposed a change to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/86770 | 04:07 |
| *** thuc has joined #openstack-infra | 04:09 | |
| *** thuc_ has joined #openstack-infra | 04:10 | |
| *** stillbert has quit IRC | 04:12 | |
| *** lanabanana has quit IRC | 04:12 | |
| *** mikal has joined #openstack-infra | 04:14 | |
| *** thuc has quit IRC | 04:14 | |
| *** nosnos has joined #openstack-infra | 04:14 | |
| *** mikal is now known as stillbert | 04:14 | |
| *** chandan_kumar has quit IRC | 04:22 | |
| *** om has joined #openstack-infra | 04:33 | |
| *** loquacities has joined #openstack-infra | 04:33 | |
| *** chandan_kumar has joined #openstack-infra | 04:35 | |
| *** loquacities is now known as loquacious | 04:36 | |
| *** loquacious is now known as Loquacities | 04:36 | |
| *** Loquacities is now known as lanabanana | 04:37 | |
| *** CaptTofu has joined #openstack-infra | 04:37 | |
| *** esker has joined #openstack-infra | 04:39 | |
| *** CaptTofu has quit IRC | 04:43 | |
| *** yfried__ has quit IRC | 04:49 | |
| *** skraynev_afk is now known as skraynev | 04:50 | |
| *** ildikov has quit IRC | 04:52 | |
| *** dstanek has quit IRC | 04:52 | |
| *** Sukhdev has joined #openstack-infra | 04:54 | |
| *** wenlock has joined #openstack-infra | 05:02 | |
| *** zns has quit IRC | 05:07 | |
| *** zns has joined #openstack-infra | 05:08 | |
| *** thuc has joined #openstack-infra | 05:09 | |
| *** thuc_ has quit IRC | 05:13 | |
| *** thuc has quit IRC | 05:13 | |
| *** basha_ has joined #openstack-infra | 05:15 | |
| *** saju_m has joined #openstack-infra | 05:16 | |
| *** mihgen has joined #openstack-infra | 05:20 | |
| *** chandan_kumar has quit IRC | 05:24 | |
| *** yolanda has joined #openstack-infra | 05:34 | |
| *** yolanda has quit IRC | 05:36 | |
| *** wchrisj has quit IRC | 05:36 | |
| *** chandan_kumar has joined #openstack-infra | 05:37 | |
| *** zhiyan_ is now known as zhiyan | 05:39 | |
| *** _nadya_ has joined #openstack-infra | 05:41 | |
| *** om has quit IRC | 05:55 | |
| *** _nadya_ has quit IRC | 05:58 | |
| *** aconrad has joined #openstack-infra | 06:00 | |
| *** basha_ has quit IRC | 06:01 | |
| *** praneshp has joined #openstack-infra | 06:09 | |
| *** amotoki has joined #openstack-infra | 06:10 | |
| *** Ryan_Lane1 has quit IRC | 06:14 | |
| *** pcrews has quit IRC | 06:15 | |
| *** _nadya_ has joined #openstack-infra | 06:15 | |
| *** thuc has joined #openstack-infra | 06:20 | |
| *** aconrad has quit IRC | 06:22 | |
| *** thuc has quit IRC | 06:25 | |
| *** ildikov has joined #openstack-infra | 06:25 | |
| *** CaptTofu has joined #openstack-infra | 06:25 | |
| *** mihgen has quit IRC | 06:25 | |
| *** mihgen has joined #openstack-infra | 06:26 | |
| *** CaptTofu has quit IRC | 06:31 | |
| *** wenlock has quit IRC | 06:33 | |
| *** afazekas has quit IRC | 06:36 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 06:40 |
| *** _nadya_ has quit IRC | 06:40 | |
| *** andreykurilin_ has joined #openstack-infra | 06:43 | |
| *** yfried__ has joined #openstack-infra | 06:44 | |
| *** Ryan_Lane has joined #openstack-infra | 06:44 | |
| *** cody-somerville has joined #openstack-infra | 06:45 | |
| *** michchap has quit IRC | 06:45 | |
| *** liam_ has joined #openstack-infra | 06:45 | |
| *** liam_ is now known as Guest89383 | 06:45 | |
| *** michchap has joined #openstack-infra | 06:45 | |
| *** ildikov_ has joined #openstack-infra | 06:46 | |
| *** jcoufal has joined #openstack-infra | 06:46 | |
| *** ildikov has quit IRC | 06:48 | |
| *** Guest89383 has quit IRC | 06:49 | |
| *** chandan_kumar has quit IRC | 06:49 | |
| *** Sukhdev has quit IRC | 06:49 | |
| tchaypo | dstufft: around? I'd like to talk about which things I should test for my pip change | 06:50 |
| tchaypo | Idle time of 78 minutes suggests not. | 06:50 |
| *** Ryan_Lane has quit IRC | 06:54 | |
| *** jamielennox is now known as jamielennox|away | 06:54 | |
| *** Ryan_Lane has joined #openstack-infra | 06:55 | |
| *** chandan_kumar has joined #openstack-infra | 06:56 | |
| *** cody-somerville has quit IRC | 06:56 | |
| *** Ryan_Lane has quit IRC | 06:59 | |
| *** mihgen has quit IRC | 07:01 | |
| *** afazekas has joined #openstack-infra | 07:01 | |
| *** manslaughter is now known as mrda_away | 07:01 | |
| *** pblaho has joined #openstack-infra | 07:01 | |
| *** nati_ueno has quit IRC | 07:02 | |
| *** jlibosva has joined #openstack-infra | 07:04 | |
| *** SettleDown is now known as asettlke | 07:04 | |
| *** asettlke is now known as asettle | 07:04 | |
| *** andreykurilin_ has quit IRC | 07:07 | |
| *** oomichi is now known as ken1ohmichi | 07:13 | |
| *** skolekonov has joined #openstack-infra | 07:16 | |
| *** tnurlygayanov has quit IRC | 07:16 | |
| *** tnurlygayanov has joined #openstack-infra | 07:17 | |
| *** skolekonov has quit IRC | 07:22 | |
| *** jgallard has joined #openstack-infra | 07:24 | |
| *** e0ne has joined #openstack-infra | 07:26 | |
| *** skolekonov has joined #openstack-infra | 07:26 | |
| *** jgallard has quit IRC | 07:28 | |
| *** jgallard has joined #openstack-infra | 07:28 | |
| *** mspreitz has joined #openstack-infra | 07:29 | |
| mspreitz | Is it just me, or is pypi wedged? | 07:29 |
| mspreitz | OK, unwedged now | 07:30 |
| *** nati_ueno has joined #openstack-infra | 07:30 | |
| *** mspreitz has quit IRC | 07:30 | |
| *** blamar has quit IRC | 07:31 | |
| *** matsuhashi has quit IRC | 07:35 | |
| *** matsuhashi has joined #openstack-infra | 07:35 | |
| *** e0ne has quit IRC | 07:36 | |
| *** e0ne has joined #openstack-infra | 07:37 | |
| *** saper has quit IRC | 07:38 | |
| *** pcrews has joined #openstack-infra | 07:39 | |
| *** e0ne has quit IRC | 07:39 | |
| *** flaper87|afk is now known as flaper87 | 07:40 | |
| *** markwash has joined #openstack-infra | 07:42 | |
| *** rallegre has joined #openstack-infra | 07:48 | |
| *** hashar has joined #openstack-infra | 07:48 | |
| *** ihrachyshka has joined #openstack-infra | 07:49 | |
| *** Ryan_Lane has joined #openstack-infra | 07:55 | |
| *** sdake_ has quit IRC | 07:58 | |
| *** saper has joined #openstack-infra | 07:59 | |
| *** Ryan_Lane has quit IRC | 08:00 | |
| *** rallegre has quit IRC | 08:01 | |
| *** mihgen has joined #openstack-infra | 08:04 | |
| *** derekh has joined #openstack-infra | 08:12 | |
| *** reed has joined #openstack-infra | 08:12 | |
| *** CaptTofu has joined #openstack-infra | 08:14 | |
| *** achuprin has joined #openstack-infra | 08:18 | |
| *** e0ne has joined #openstack-infra | 08:18 | |
| *** CaptTofu has quit IRC | 08:19 | |
| *** yjiang has quit IRC | 08:19 | |
| *** achuprin has quit IRC | 08:20 | |
| *** achuprin has joined #openstack-infra | 08:21 | |
| *** julienvey has joined #openstack-infra | 08:22 | |
| *** matsuhas_ has joined #openstack-infra | 08:25 | |
| *** matsuhashi has quit IRC | 08:25 | |
| *** saper has quit IRC | 08:28 | |
| *** hashar has quit IRC | 08:29 | |
| *** achuprin has quit IRC | 08:29 | |
| *** achuprin has joined #openstack-infra | 08:29 | |
| *** achuprin has quit IRC | 08:30 | |
| *** achuprin has joined #openstack-infra | 08:30 | |
| *** achuprin has joined #openstack-infra | 08:31 | |
| *** hashar has joined #openstack-infra | 08:31 | |
| *** rallegre has joined #openstack-infra | 08:33 | |
| *** e0ne has quit IRC | 08:34 | |
| *** ildikov has joined #openstack-infra | 08:35 | |
| *** ildikov_ has quit IRC | 08:37 | |
| *** praneshp has quit IRC | 08:38 | |
| *** jooools has joined #openstack-infra | 08:41 | |
| *** xchu_ has joined #openstack-infra | 08:41 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Make logger available during tests https://review.openstack.org/86825 | 08:43 |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Allow to specify multiple branches per project in Gerrit trigger https://review.openstack.org/85648 | 08:43 |
| *** dizquierdo has joined #openstack-infra | 08:45 | |
| *** saper has joined #openstack-infra | 08:45 | |
| *** mkerrin has quit IRC | 08:46 | |
| *** zns has quit IRC | 08:46 | |
| *** jp_at_hp has joined #openstack-infra | 08:48 | |
| *** ociuhandu has joined #openstack-infra | 08:48 | |
| *** mkerrin has joined #openstack-infra | 08:52 | |
| *** matrohon has quit IRC | 08:52 | |
| *** xchu_ has quit IRC | 08:55 | |
| *** Ryan_Lane has joined #openstack-infra | 08:56 | |
| *** morganfainberg is now known as morganfainberg_Z | 08:57 | |
| *** ken1ohmichi has quit IRC | 08:58 | |
| *** nati_ueno has quit IRC | 08:59 | |
| *** Ryan_Lane has quit IRC | 09:05 | |
| *** chandan_kumar has quit IRC | 09:10 | |
| *** ihrachyshka has quit IRC | 09:14 | |
| *** chandan_kumar has joined #openstack-infra | 09:15 | |
| *** nijaba has joined #openstack-infra | 09:21 | |
| *** nijaba has joined #openstack-infra | 09:21 | |
| *** lanabanana is now known as loquacity | 09:24 | |
| *** yolanda has joined #openstack-infra | 09:24 | |
| *** loquacity is now known as loquacities | 09:25 | |
| *** saschpe has quit IRC | 09:27 | |
| *** e0ne has joined #openstack-infra | 09:27 | |
| *** saschpe has joined #openstack-infra | 09:29 | |
| *** gmoro has quit IRC | 09:29 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Allow to specify multiple branches per project in Gerrit trigger https://review.openstack.org/85648 | 09:35 |
| *** hashar_ has joined #openstack-infra | 09:35 | |
| *** jaypipes has quit IRC | 09:35 | |
| *** hashar has quit IRC | 09:38 | |
| *** e0ne has quit IRC | 09:39 | |
| *** e0ne has joined #openstack-infra | 09:40 | |
| *** hashar_ is now known as H4|- | 09:40 | |
| *** gmoro has joined #openstack-infra | 09:41 | |
| *** H4|- is now known as hashar | 09:42 | |
| *** rallegre has quit IRC | 09:43 | |
| *** e0ne has quit IRC | 09:44 | |
| *** saju_m has quit IRC | 09:45 | |
| *** ociuhandu has quit IRC | 09:46 | |
| *** e0ne has joined #openstack-infra | 09:47 | |
| *** chandan_kumar has quit IRC | 09:47 | |
| *** jaypipes has joined #openstack-infra | 09:48 | |
| *** reed has quit IRC | 09:49 | |
| *** ominakov has joined #openstack-infra | 09:50 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Add fedora devstack instances to nodepool https://review.openstack.org/86842 | 09:52 |
| *** che-arne has joined #openstack-infra | 09:53 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Add support for multiple Gerrit servers in Gerrit trigger https://review.openstack.org/86843 | 09:54 |
| *** zhiyan is now known as zhiyan_ | 09:55 | |
| *** rallegre has joined #openstack-infra | 09:55 | |
| *** rallegre has quit IRC | 09:57 | |
| *** CaptTofu has joined #openstack-infra | 09:58 | |
| *** CaptTofu has quit IRC | 09:58 | |
| *** CaptTofu has joined #openstack-infra | 09:59 | |
| *** saju_m has joined #openstack-infra | 10:02 | |
| *** nosnos has quit IRC | 10:03 | |
| *** matsuhas_ has quit IRC | 10:04 | |
| *** hashar has quit IRC | 10:05 | |
| *** ihrachyshka has joined #openstack-infra | 10:06 | |
| *** markwash has quit IRC | 10:06 | |
| *** jgallard has quit IRC | 10:09 | |
| *** ociuhandu has joined #openstack-infra | 10:24 | |
| *** saju_m has quit IRC | 10:25 | |
| *** chandan_kumar has joined #openstack-infra | 10:29 | |
| *** yolanda has quit IRC | 10:32 | |
| *** markmc has joined #openstack-infra | 10:33 | |
| *** yolanda has joined #openstack-infra | 10:34 | |
| *** mbacchi has joined #openstack-infra | 10:43 | |
| *** fifieldt has quit IRC | 10:47 | |
| *** alexpilotti has joined #openstack-infra | 10:56 | |
| *** yamahata has quit IRC | 10:57 | |
| *** CaptTofu has quit IRC | 10:59 | |
| *** reed has joined #openstack-infra | 11:01 | |
| sdague | fungi / jeblair: looks like nodepool has starved out pypi and py33 jobs again | 11:04 |
| sdague | so everything is stalled | 11:04 |
| *** mcdonagm has joined #openstack-infra | 11:04 | |
| *** nprivalova has joined #openstack-infra | 11:07 | |
| *** mcdonagm has quit IRC | 11:08 | |
| *** mfmcdonagh has joined #openstack-infra | 11:09 | |
| *** thomasem has joined #openstack-infra | 11:11 | |
| *** hashar has joined #openstack-infra | 11:14 | |
| *** ociuhandu has quit IRC | 11:19 | |
| *** ociuhandu has joined #openstack-infra | 11:20 | |
| *** eharney has joined #openstack-infra | 11:26 | |
| *** ociuhandu has quit IRC | 11:29 | |
| *** chandan_kumar has quit IRC | 11:41 | |
| *** slagle has quit IRC | 11:41 | |
| *** slagle has joined #openstack-infra | 11:42 | |
| *** rallegre has joined #openstack-infra | 11:46 | |
| *** coolsvap is now known as coolsvap|afk | 11:47 | |
| *** dstanek has joined #openstack-infra | 11:49 | |
| *** mwagner_lap has quit IRC | 11:50 | |
| *** chandan_kumar has joined #openstack-infra | 11:54 | |
| *** yaguang has quit IRC | 11:56 | |
| *** rallegre has quit IRC | 12:00 | |
| *** dims has joined #openstack-infra | 12:01 | |
| *** rallegre has joined #openstack-infra | 12:02 | |
| *** dkranz has quit IRC | 12:02 | |
| *** rallegre has quit IRC | 12:06 | |
| *** rallegre has joined #openstack-infra | 12:13 | |
| *** msabramo has quit IRC | 12:13 | |
| *** pdmars has joined #openstack-infra | 12:13 | |
| *** rallegre has quit IRC | 12:14 | |
| *** reed has quit IRC | 12:14 | |
| *** pdmars has quit IRC | 12:14 | |
| *** pdmars has joined #openstack-infra | 12:15 | |
| *** dizquierdo has quit IRC | 12:16 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Navigate to story details after create complete https://review.openstack.org/86571 | 12:16 |
| *** jhesketh has quit IRC | 12:17 | |
| *** dkliban_afk is now known as dkliban | 12:18 | |
| *** rallegre has joined #openstack-infra | 12:18 | |
| *** pdmars has quit IRC | 12:20 | |
| *** rallegre has quit IRC | 12:21 | |
| *** locke1051 has joined #openstack-infra | 12:22 | |
| *** dkehn_ has joined #openstack-infra | 12:22 | |
| *** CaptTofu has joined #openstack-infra | 12:22 | |
| *** davidlenwell_ has joined #openstack-infra | 12:22 | |
| *** rallegre has joined #openstack-infra | 12:23 | |
| *** rahmu_ has joined #openstack-infra | 12:24 | |
| *** pdmars has joined #openstack-infra | 12:24 | |
| *** rahmu has quit IRC | 12:24 | |
| *** locke105 has quit IRC | 12:24 | |
| *** dkehnx has quit IRC | 12:24 | |
| *** davidlenwell has quit IRC | 12:24 | |
| *** phschwartz has quit IRC | 12:24 | |
| *** rahmu_ is now known as rahmu | 12:24 | |
| *** jgallard has joined #openstack-infra | 12:25 | |
| *** phschwartz has joined #openstack-infra | 12:25 | |
| *** rallegre has quit IRC | 12:27 | |
| *** rlandy has joined #openstack-infra | 12:29 | |
| *** aysyd has joined #openstack-infra | 12:30 | |
| *** signed8bit has joined #openstack-infra | 12:30 | |
| *** signed8bit has quit IRC | 12:30 | |
| *** dkranz has joined #openstack-infra | 12:31 | |
| *** blamar has joined #openstack-infra | 12:34 | |
| *** basha_ has joined #openstack-infra | 12:35 | |
| *** dcramer_ has quit IRC | 12:35 | |
| *** miqui has joined #openstack-infra | 12:36 | |
| *** rallegre has joined #openstack-infra | 12:36 | |
| *** basha_ has quit IRC | 12:37 | |
| *** rallegre has quit IRC | 12:39 | |
| *** pelix has joined #openstack-infra | 12:43 | |
| *** adalbas has joined #openstack-infra | 12:45 | |
| *** jeckersb_gone is now known as jeckersb | 12:47 | |
| pelix | Is there anything else required to get https://review.openstack.org/#/c/85177 approved? | 12:48 |
| pelix | I have a change that will be much simpler once this and a few others are merged first :) | 12:48 |
| *** thuc has joined #openstack-infra | 12:52 | |
| *** thuc_ has joined #openstack-infra | 12:52 | |
| *** dprince has joined #openstack-infra | 12:53 | |
| jeblair | sdague: ack | 12:53 |
| *** weshay has joined #openstack-infra | 12:53 | |
| *** zehicle has quit IRC | 12:54 | |
| jeblair | sdague, fungi: there are 8 nodes stuck in the delete state, 2 are building | 12:55 |
| sdague | jeblair: it also looks like the mix right be wrong. With things like refstack and rally using those kinds of nodes, the old limits might not be enough | 12:56 |
| *** thuc has quit IRC | 12:56 | |
| jeblair | sdague: *nod* | 12:57 |
| *** mwagner_lap has joined #openstack-infra | 12:58 | |
| jeblair | so the easiest way out of this is to add the hpcloud capacity back... let me see if we have a node i can check | 12:58 |
| sdague | cool | 12:58 |
| sdague | is there a way to make this adaptive? | 12:58 |
| jeblair | sdague: in what way? | 12:59 |
| sdague | or what would be the way to make it adaptive. Because I think node mix might vary over time | 12:59 |
| *** rfolco has joined #openstack-infra | 12:59 | |
| fungi | jeblair: oh, fun | 12:59 |
| fungi | taking a look now | 12:59 |
| jeblair | fungi: was the error that http connects to pypi.o.o always failed? | 12:59 |
| sdague | basically we're starved right now because we have the wrong node mix | 12:59 |
| sdague | right? | 12:59 |
| fungi | jeblair: it was intermittent. maybe 1 in 100, which leads to >half the run there jobs failing, which led to all teh changes failing | 13:00 |
| *** msabramo has joined #openstack-infra | 13:01 | |
| anteaya | I see by the clock on the wall (or the time stamp in my irc client) it is 13:01 utc | 13:01 |
| anteaya | do I see correctly? | 13:01 |
| fungi | i'm checking in a bit to see if either rs or hp listed any events from yesterday that they corrected before reverting the disable change | 13:01 |
| fungi | anteaya: yes, your clock matches mine | 13:01 |
| anteaya | wonderful thank you | 13:01 |
| anteaya | closing polls | 13:02 |
| jeblair | sdague: looking at the allocation reports, i think we may have an error in the allocator related to providers with min-ready=0 | 13:02 |
| sdague | ok | 13:02 |
| *** dkranz has quit IRC | 13:02 | |
| jeblair | i'll look into it later, but in the mean time, we may want to remove providers from labels instead of setting max-servers to 0 | 13:02 |
| sdague | so the answer is that it should be rebalancing, but there is a bug that just needs to be worked out | 13:02 |
| fungi | jeblair: but if we divide by that, we can have an infinite number of nodes | 13:02 |
| jeblair | sorry, i should have said max-servers=0 the first time | 13:03 |
| fungi | got it | 13:03 |
| sdague | fungi: would it be a countable or uncountable infinity :) | 13:03 |
| jeblair | sdague: yeah, that's the first thing to check; there may be something else we can do after that, but first i'd want to get the negative numbers out of the allocation report :) | 13:03 |
| *** dkranz has joined #openstack-infra | 13:03 | |
| jeblair | 2014-04-11 13:00:29,354 DEBUG nodepool.NodePool: <AllocationSubRequest for -317.0 (out of 317.0) of bare-precise from hpcloud-az2> | 13:03 |
| fungi | sdague: aleph-null | 13:03 |
| fungi | sdague: not aleph-prime | 13:03 |
| sdague | jeblair: nice :) | 13:03 |
| jeblair | the totals actually work out though, so it may just be weird, not terribly incorrect | 13:04 |
| *** david-lyle has joined #openstack-infra | 13:04 | |
| fungi | jeblair: okay, so i guess we're tending toward +/- infinity (in which case -317 has a way to go still) | 13:04 |
| *** adalbas has quit IRC | 13:04 | |
| *** ildikov_ has joined #openstack-infra | 13:04 | |
| *** rallegre has joined #openstack-infra | 13:05 | |
| jeblair | if the negative numbers aren't harmful, then probably the thing we want to look at is... | 13:05 |
| jeblair | under constraints, np uses the normal server mix to decide how to allocate its constrained resources | 13:05 |
| fungi | trying a delete --now on one of the oldest py3k-precise stuck in a delete state, looks like the nova call is going to time out on it | 13:05 |
| jeblair | sdague: so i think we'd want to factor the current demand ratio in to that more... | 13:06 |
| jeblair | sdague: or only look at that. | 13:07 |
| *** ildikov has quit IRC | 13:07 | |
| jeblair | sdague: can you file a story for that? | 13:07 |
| sdague | sure, is there active storyboard being used for this? | 13:07 |
| jeblair | fungi: my wifi is about to die, can you take over looking at hpcloud? | 13:07 |
| fungi | jeblair: yep, researching currently | 13:08 |
| jeblair | sdague: as of now | 13:08 |
| fungi | jeblair: https://status.rackspace.com/index/viewincidents?group=19&start=1397102400 looks like it *could* be related | 13:08 |
| sdague | url for starting point? | 13:08 |
| *** jeckersb is now known as jeckersb_gone | 13:08 | |
| jeblair | (i'm in the keynote which is about to start, so everyone is on laptops in one room) | 13:09 |
| jeblair | https://storyboard.openstack.org/#!/project/668/overview | 13:09 |
| jeblair | (nodepool doesn't have an existing bug tracker so it gets to go first) | 13:09 |
| *** msabramo has quit IRC | 13:09 | |
| *** prad_ has joined #openstack-infra | 13:10 | |
| jeblair | well, second after storyboard | 13:10 |
| *** mfer has joined #openstack-infra | 13:10 | |
| jeblair | ok networks dead, bbl | 13:10 |
| fungi | k | 13:10 |
| *** ameade has joined #openstack-infra | 13:11 | |
| sdague | https://storyboard.openstack.org/#!/story/63 | 13:15 |
| *** dkehn_ is now known as dkehnx | 13:15 | |
| *** adalbas has joined #openstack-infra | 13:17 | |
| *** mriedem has joined #openstack-infra | 13:18 | |
| *** prad_ has quit IRC | 13:18 | |
| fungi | i've got a couple of idle instances in hpcloud downloading the latest pbr tarball from pypi.o.o repeatedly in a loop. prognosis not good | 13:26 |
| fungi | when it fails, it looks like http://paste.openstack.org/show/75563/ | 13:28 |
| *** prad_ has joined #openstack-infra | 13:29 | |
| *** prad_ has quit IRC | 13:31 | |
| kiall | fungi: that's weird - why is it even trying IPv6? Pretty sure we don't have IPv6 for instances at HPCloud yet | 13:32 |
| fungi | kiall: that looks like a fallback behavior from wget | 13:32 |
| fungi | silly to try it when it doesn't even have a route to that prefix | 13:32 |
| kiall | Yea.. Just never seen that before :) | 13:32 |
| *** prad_ has joined #openstack-infra | 13:32 | |
| fungi | i did confirm, those instances definitely only have linklocal v6 addresses on their interfaces and default gateway | 13:33 |
| fungi | no default v6 gateway, that is | 13:33 |
| kiall | Anyway - What region/az is the instance in? | 13:33 |
| fungi | the two i'm testing are in az2, but we were also seeing this yesterday from az1, az2 and region b | 13:34 |
| *** reed has joined #openstack-infra | 13:35 | |
| fungi | i'm still building up an acceptable sample set, but it looks like about 1 in 200 connections times out | 13:35 |
| fungi | which is plenty enough to stop devstack dead in its tracks | 13:36 |
| sdague | yep, well we set -o errexit for a reason | 13:36 |
| kiall | fungi: humm .. So, your seeing it from region-a and region-b ? (Sounded like az1/az2 in A and also region-b) | 13:37 |
| *** thuc has joined #openstack-infra | 13:38 | |
| *** pblaho has quit IRC | 13:40 | |
| fungi | er, i meant yesterday we also saw it from az2 | 13:40 |
| fungi | gah | 13:40 |
| fungi | az3 | 13:40 |
| *** markmcclain has joined #openstack-infra | 13:40 | |
| fungi | my fingers clearly don't want to hit the right keys today | 13:40 |
| *** jeckersb_gone is now known as jeckersb | 13:40 | |
| *** thuc_ has quit IRC | 13:41 | |
| *** dkranz has quit IRC | 13:42 | |
| *** thuc has quit IRC | 13:43 | |
| fungi | no packet loss on icmp/echo for the same path though, seems to be impacting tcp sockets, so i suspect an overloaded nat or other state-management device mucking with layer 4 | 13:43 |
| fungi | i've set up a similar download loop from another region in rackspace and no failures at all | 13:44 |
| fungi | so at least between rackspace regions we're not seeing this, but from hp to rackspace we are | 13:44 |
| kiall | Checking with our NOC for any known issues... Maybe they've had other similar reports.. | 13:44 |
| fungi | which matches what we were seeing with test failures yesterday | 13:44 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Pass change and pipeline to the status_url https://review.openstack.org/86900 | 13:44 |
| kiall | Also - Can you paste bin your script? | 13:45 |
| fungi | going to try to collect some slightly better stats with hping3 | 13:45 |
| fungi | i can paste it right here--it's a one-liner after all ;) | 13:45 |
| fungi | count=0;while wget http://pypi.openstack.org/openstack/pbr/pbr-0.8.0.tar.gz>/dev/null;do rm pbr-0.8.0.tar.gz;count=$[$count+1];done;echo $count | 13:45 |
| *** alaski is now known as lascii | 13:45 | |
| kiall | ;) | 13:45 |
| dhellmann | fungi: do you have any gerrit downtime scheduled? we could go ahead and rename the oslo.test repository when it's convenient | 13:47 |
| fungi | dhellmann: i don't think we've scheduled anything yet. i believe moving barbican from stackforge to openstack and renaming climate (once foundation legal weighs in on the new name choice) are currently on the list, along with the attic moves | 13:48 |
| fungi | dhellmann: can you add that one to the agenda for teh infra meeting? | 13:48 |
| dhellmann | certainly | 13:48 |
| fungi | we've been keeping a tally of them in there so we can review status at the weekly meetings | 13:48 |
| dhellmann | there's no rush, so whenever you're going to do the others is fine | 13:48 |
| sdague | krotscheck: should I file new stories for bugs I find in storyboard? | 13:49 |
| *** dcramer_ has joined #openstack-infra | 13:49 | |
| dhellmann | fungi: added | 13:49 |
| fungi | dhellmann: thanks! | 13:49 |
| dhellmann | fungi: fridays are bad for landing big ci changes like https://review.openstack.org/#/c/85487/ but maybe early next week? | 13:52 |
| *** dizquierdo has joined #openstack-infra | 13:53 | |
| *** signed8bit has joined #openstack-infra | 13:53 | |
| fungi | dhellmann: yeah, i don't want to approve something like that without it getting a second core review, especially since i'm leaving for two weeks in the morning | 13:53 |
| dhellmann | fungi: oh, yeah, I wasn't asking for you to approve it on your own | 13:54 |
| fungi | don't want to create a mess and then leave others to clean it up ;) | 13:54 |
| dhellmann | and I don't want to create a mess right before the weekend :-) | 13:54 |
| dhellmann | I'll see how things are looking monday and ask for some attention then | 13:54 |
| dhellmann | I was going to ask this week on heartbleed day :-) | 13:54 |
| fungi | sounds like a good idea | 13:55 |
| *** skolekonov has quit IRC | 13:58 | |
| *** basha_ has joined #openstack-infra | 13:59 | |
| *** dkranz has joined #openstack-infra | 14:00 | |
| *** thuc has joined #openstack-infra | 14:01 | |
| *** thuc has quit IRC | 14:02 | |
| *** thuc has joined #openstack-infra | 14:02 | |
| *** jcoufal has quit IRC | 14:03 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Add a check for nova-docker https://review.openstack.org/86910 | 14:03 |
| *** ociuhandu has joined #openstack-infra | 14:03 | |
| *** pelix has quit IRC | 14:04 | |
| *** dansmith is now known as superdan | 14:05 | |
| *** yamahata has joined #openstack-infra | 14:06 | |
| *** russellb is now known as rustlebee | 14:07 | |
| *** eharney has quit IRC | 14:08 | |
| *** zhiyan_ is now known as zhiyan | 14:09 | |
| openstackgerrit | Brant Knudson proposed a change to openstack/requirements: Don't use flake8: noqa https://review.openstack.org/86911 | 14:09 |
| *** eharney has joined #openstack-infra | 14:10 | |
| fungi | hping3 from a couple instances in hpcloud got me 28 tcp connect timeouts out of 2000 (spaced in 1-second intervals) so that roughly correlates to what i was seeing with the wget loops | 14:16 |
| *** jaypipes is now known as leakypipes | 14:17 | |
| sdague | fungi: any idea if it is hp, rax, or something inbetween that's dying? | 14:18 |
| fungi | sdague: not sure yet. between rax azs i see no problem. i'm testing from hp to google now to get another baseline | 14:19 |
| sdague | cool | 14:19 |
| *** thuc has quit IRC | 14:19 | |
| *** thuc has joined #openstack-infra | 14:20 | |
| *** david-lyle has quit IRC | 14:20 | |
| *** reed has quit IRC | 14:22 | |
| *** aysyd has quit IRC | 14:24 | |
| *** thuc has quit IRC | 14:24 | |
| *** e0ne_ has joined #openstack-infra | 14:25 | |
| *** marun has joined #openstack-infra | 14:26 | |
| *** vponomaryov has quit IRC | 14:26 | |
| *** miarmak_ has joined #openstack-infra | 14:27 | |
| *** achuprin has quit IRC | 14:27 | |
| *** vponomaryov has joined #openstack-infra | 14:27 | |
| *** achuprin has joined #openstack-infra | 14:27 | |
| *** miarmak has quit IRC | 14:28 | |
| *** bogdando has quit IRC | 14:28 | |
| *** redixin has joined #openstack-infra | 14:28 | |
| *** wchrisj has joined #openstack-infra | 14:28 | |
| *** bogdando has joined #openstack-infra | 14:28 | |
| *** e0ne has quit IRC | 14:28 | |
| *** asettle has quit IRC | 14:29 | |
| sdague | jeblair: if you have a few minutes today, I'd like to discuss the feature selection in d-g relative to the needs of - https://review.openstack.org/#/c/86577/2/specs/branchless-tempest.rst | 14:30 |
| *** chandan_kumar has quit IRC | 14:31 | |
| *** redixin has left #openstack-infra | 14:34 | |
| *** imcsk8_lap has quit IRC | 14:35 | |
| *** imcsk8_lap has joined #openstack-infra | 14:35 | |
| *** mihgen has quit IRC | 14:36 | |
| *** wenlock has joined #openstack-infra | 14:38 | |
| *** wenlock has quit IRC | 14:39 | |
| *** wenlock has joined #openstack-infra | 14:39 | |
| *** thingee has quit IRC | 14:40 | |
| *** thingee has joined #openstack-infra | 14:41 | |
| *** thuc has joined #openstack-infra | 14:44 | |
| *** thuc_ has joined #openstack-infra | 14:45 | |
| *** lpetrut has joined #openstack-infra | 14:46 | |
| *** e0ne_ has quit IRC | 14:46 | |
| *** e0ne has joined #openstack-infra | 14:47 | |
| *** david-lyle has joined #openstack-infra | 14:47 | |
| *** thuc has quit IRC | 14:48 | |
| sdague | fungi: is there any way to manually rebalance nodepool | 14:49 |
| sdague | check queue is 177 right now | 14:49 |
| sdague | with an 8hr delay | 14:49 |
| fungi | sdague: i'm trying | 14:50 |
| sdague | ok | 14:50 |
| *** michchap has quit IRC | 14:50 | |
| *** michchap has joined #openstack-infra | 14:50 | |
| *** marun has quit IRC | 14:50 | |
| sdague | I also wonder if a good zuul future change would be to wait no more than 1 hr past the normal job completion for non voting results | 14:50 |
| fungi | i've ripped hpcloud out of the configuration entirely, restarted nodepool, reset min-ready way up for the less common node types, clearing away some cobwebs and seeing if i can get things back into shape for now | 14:50 |
| sdague | because in at least 1/3 of these cases, we're stalled waiting for non voting results | 14:51 |
| *** e0ne has quit IRC | 14:51 | |
| *** vhoward has joined #openstack-infra | 14:51 | |
| sdague | ok cool | 14:52 |
| *** lpetrut has left #openstack-infra | 14:52 | |
| *** thedodd has joined #openstack-infra | 14:52 | |
| *** bookwar has joined #openstack-infra | 14:52 | |
| *** marun has joined #openstack-infra | 14:53 | |
| *** e0ne has joined #openstack-infra | 14:54 | |
| fungi | also still trying to nail down enough details about the network flakiness between hp and rs to have something useful we can put in a trouble ticket | 14:54 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Drive puppet from the master over ssh https://review.openstack.org/80976 | 14:56 |
| *** adalbas has quit IRC | 14:58 | |
| *** basha_ has quit IRC | 15:02 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Zake 0.0.15 released https://review.openstack.org/86928 | 15:02 |
| sdague | ttx: when do the stable/icehouse branches get set? There is a ton of qa process that can't start until those exist. | 15:05 |
| sdague | especially with branchless tempest | 15:06 |
| fungi | sdague: release day | 15:06 |
| ttx | sdague: yeah | 15:06 |
| ttx | what fungi said | 15:06 |
| sdague | ttx: well that basically means we'll have a bunch of days where stable is breakable | 15:06 |
| ttx | why? | 15:07 |
| *** katyafervent is now known as katyafervent_awa | 15:08 | |
| sdague | because we can't start testing tempest master changes gating on stable/icehouse until after stable icehouse is created and we can get working jobs on it | 15:08 |
| *** briancline has joined #openstack-infra | 15:08 | |
| sdague | it also means we're currently fully exposed on upgrading from what will be stable/icehouse to master | 15:09 |
| mordred | sdague: could we test branchless tempest on the proposed branches? | 15:09 |
| sdague | mordred: possibly, but there is a bunch of logic that took a long time to be right in all the d-g grenade work | 15:09 |
| ttx | sdague: you mean, we aren't testing tempest changes against milestone-proposed at all | 15:09 |
| fungi | have the jobs look for stable/icehouse and then fall back to looking for milestone-proposed and treat them the same? | 15:10 |
| *** aysyd has joined #openstack-infra | 15:10 | |
| sdague | ttx: we are testing m-p against tempest | 15:10 |
| ttx | fungi: that would work | 15:10 |
| sdague | but not tempest master against m-p | 15:10 |
| ttx | right | 15:10 |
| *** adalbas has joined #openstack-infra | 15:10 | |
| sdague | so people could land a tempest change that wedges m-p | 15:10 |
| fungi | sdague: we do test tempest master against m-p right now, as long as tempest doesn't have an m-p branch itself | 15:11 |
| ttx | sdague: not if we do what fungo said | 15:11 |
| ttx | i* | 15:11 |
| fungi | oh, wait | 15:11 |
| fungi | we test milestone proposed against tempest master | 15:11 |
| fungi | i see what sdague is saying | 15:11 |
| sdague | ttx: sure, but this is the issue I brought up the other day | 15:11 |
| fungi | it's asymmetrical | 15:11 |
| sdague | fungi: correct | 15:11 |
| ttx | sdague: it's only an issue in the new, branchless tempest world, right ? | 15:12 |
| fungi | tempest changes are checked against master server branches, but m-p server changes are tested against tempest master | 15:12 |
| sdague | and more importantly, putting logic in for m-p which only exists real < 20% of our time means it's going to break | 15:12 |
| sdague | ttx: the grenade issue is an existing issue | 15:12 |
| sdague | and honestly a giant pain | 15:12 |
| fungi | ttx: no, it's how it's been even in previous releases | 15:12 |
| sdague | fungi: right, the tempest issue existed in previous releases as well | 15:12 |
| ttx | sdague: hmm, I wouldn't touch that one week before release, but we need to close that hole | 15:13 |
| sdague | basically the release process doesn't really support the current testing process very well | 15:13 |
| ttx | for the time being I would suggest not landing too many changes in tempest for the coming week :) | 15:13 |
| *** mrmartin has joined #openstack-infra | 15:13 | |
| fungi | ttx: right now someone could land a change to tempest's master branch which works with master branches of all the servers but not m-p branches or eventual stable/icehouse branches of the servers, effectively breaking milestone-proposed tests until fixed | 15:13 |
| fungi | if tempest cut m-p branches in previous cycles around rc for the servers, then maybe this wasn't an issue back then | 15:14 |
| sdague | ttx: but it has other substantial implications, like the fact that the nova team can't make certain changes until grenade is going stable/icehouse -> master | 15:14 |
| fungi | or was but only much more briefly | 15:14 |
| sdague | and we can't do that until stable/icehouse exists | 15:14 |
| sdague | and even then, it takes time | 15:14 |
| *** yolanda has quit IRC | 15:14 | |
| sdague | so the infra to do stable/icehouse -> master can't be ready on release day | 15:14 |
| mtreinish | fungi: we only did that for grizzly | 15:15 |
| ttx | sdague: right, but the tooling is very reliant on branch names at this point. The last thing I would change one week before release would be to rename branches | 15:15 |
| ttx | I agree it's a problem we need to solve | 15:15 |
| ttx | but we might need to live with it for the last week of this cycle and address it early in the next | 15:16 |
| mrmartin | re | 15:16 |
| mrmartin | guys, is everything allright with zuul? http://status.openstack.org/zuul/ | 15:16 |
| sdague | ttx: ok. As long as it doesn't get lost | 15:17 |
| ttx | sdague: so at this point i would freeze changes that might put us in trouble. One more week. | 15:17 |
| ttx | sdague, fungi: where would you want to discuss that ? QA track ? | 15:17 |
| sdague | ttx: honestly, it's hard to really determine those | 15:17 |
| *** vhoward has left #openstack-infra | 15:18 | |
| sdague | ttx: this is release issue honestly | 15:18 |
| fungi | mrmartin: trouble adding new workers to run jobs. i'm currently trying to sort it out | 15:18 |
| mrmartin | fungi: ok. | 15:18 |
| ttx | sdague: Ok, will file against RelMgt | 15:18 |
| sdague | cool | 15:18 |
| ttx | sdague: that means I lose by wildcard slot so we need to accept reed's cross-project session :) | 15:18 |
| *** yolanda has joined #openstack-infra | 15:20 | |
| *** thuc_ has quit IRC | 15:20 | |
| sdague | well you can fight mtreinish to see if he wants to dedicate a slot :) | 15:20 |
| *** thuc has joined #openstack-infra | 15:20 | |
| therve | Quick ping about https://review.openstack.org/#/c/83761/ for the heat-templates job | 15:21 |
| therve | fungi, jeblair and others if you have time | 15:21 |
| *** thuc has quit IRC | 15:25 | |
| *** zul has quit IRC | 15:26 | |
| pabelanger | for the people running zuul, has anybody had a problem getting statsd working properly? I have it setup properly (env variables, but seems zuul (specifically statsd lib, doesn't properly send the data over the wire). | 15:27 |
| pabelanger | you can see my success rate: http://goo.gl/94iAGk | 15:28 |
| *** eharney has quit IRC | 15:28 | |
| *** skraynev is now known as skraynev_afk | 15:28 | |
| *** SumitNaiksatam has quit IRC | 15:29 | |
| *** david-lyle has quit IRC | 15:29 | |
| *** markmcclain has quit IRC | 15:29 | |
| *** zul has joined #openstack-infra | 15:30 | |
| *** serverascode has joined #openstack-infra | 15:31 | |
| *** david-lyle has joined #openstack-infra | 15:36 | |
| ttx | sdague: http://summit.openstack.org/cfp/details/293 -- merged it with the requirements session | 15:38 |
| ttx | if requirements are discussed somewhere else, we can remove it from that session | 15:39 |
| sdague | works for me | 15:43 |
| *** davidhadas_ has joined #openstack-infra | 15:43 | |
| *** sabari2 has joined #openstack-infra | 15:45 | |
| *** davidhadas has quit IRC | 15:46 | |
| *** tjones has left #openstack-infra | 15:48 | |
| *** mrmartin has quit IRC | 15:49 | |
| *** ihrachyshka_ has joined #openstack-infra | 15:50 | |
| *** cody-somerville has joined #openstack-infra | 15:51 | |
| *** marun has quit IRC | 15:51 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Mark projects that use storyboard https://review.openstack.org/86943 | 15:52 |
| *** markmc has quit IRC | 15:52 | |
| *** ihrachyshka has quit IRC | 15:53 | |
| *** markmc has joined #openstack-infra | 15:54 | |
| *** thuc has joined #openstack-infra | 15:54 | |
| *** hashar has quit IRC | 15:55 | |
| *** thuc_ has joined #openstack-infra | 15:55 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Only create projects we ask for https://review.openstack.org/86946 | 15:56 |
| *** dkliban is now known as dkliban_afk | 15:57 | |
| *** thuc has quit IRC | 15:59 | |
| fungi | okay, part of my confusion after the nodepool restart was that i missed that nodepoold was returning control without exiting nonzero, but then quietly dying when it found an existing pidfile | 15:59 |
| fungi | i've got the stale nodes mostly cleared at this point and it's running and bringing up new ones | 16:00 |
| fungi | so with hp entirely out of the configuration rather than just set to max-servers=0 it hopefully should dtrt with ratios | 16:00 |
| *** nati_ueno has joined #openstack-infra | 16:01 | |
| fungi | yep, i see a pypy job running | 16:02 |
| fungi | and changes in the gate are getting nodes too | 16:02 |
| *** jlibosva has quit IRC | 16:03 | |
| *** wchrisj has quit IRC | 16:04 | |
| *** ihrachyshka_ is now known as ihrachyshka | 16:06 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for heat timeout bug 1306029 https://review.openstack.org/86636 | 16:07 |
| uvirtbot | Launchpad bug 1306029 in heat "tempest.api.orchestration.stacks.test_stacks times out during tearDownClass" [Undecided,New] https://launchpad.net/bugs/1306029 | 16:07 |
| *** gyee has joined #openstack-infra | 16:07 | |
| *** david-lyle has quit IRC | 16:12 | |
| *** zns has joined #openstack-infra | 16:13 | |
| *** zns has quit IRC | 16:14 | |
| *** zns has joined #openstack-infra | 16:14 | |
| clarkb | fungi: woot, so nodepool is happy again? | 16:15 |
| clarkb | fungi: did that also require editing the min values? | 16:15 |
| *** david-lyle has joined #openstack-infra | 16:16 | |
| fungi | clarkb: just doing some remaining cleanup but it seems to be back on track. i ended up just removing all traces of hpcloud from the config since it looked like the demand ratio was being miscalculated when only zeroing max-servers | 16:16 |
| clarkb | mordred: btw I wanted to mention I have been using wheels built for precise on my saucy laptop without any apparent problems | 16:16 |
| fungi | i ended up not changing any of the min-ready values | 16:16 |
| clarkb | fungi: interesting | 16:16 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Add nova-specs https://review.openstack.org/86949 | 16:17 |
| *** amotoki_ has joined #openstack-infra | 16:17 | |
| fungi | i've got puppet stopped on it for the moment, but will prop up a temporary change matching my config edits shortly so we can get puppet agent going on it again | 16:17 |
| *** jlibosva has joined #openstack-infra | 16:18 | |
| *** e0ne has quit IRC | 16:19 | |
| *** e0ne has joined #openstack-infra | 16:20 | |
| *** comstud is now known as bearhands | 16:22 | |
| *** Sukhdev has joined #openstack-infra | 16:22 | |
| zaro | morning | 16:23 |
| *** e0ne has quit IRC | 16:24 | |
| *** julienvey has quit IRC | 16:25 | |
| *** harlowja_away is now known as harlowja | 16:25 | |
| *** dkranz has quit IRC | 16:26 | |
| *** vponomaryov has left #openstack-infra | 16:27 | |
| *** markwash has joined #openstack-infra | 16:28 | |
| *** jooools has quit IRC | 16:28 | |
| *** cody-somerville has quit IRC | 16:28 | |
| *** ominakov has quit IRC | 16:29 | |
| *** nati_ueno has quit IRC | 16:31 | |
| *** markwash_ has joined #openstack-infra | 16:33 | |
| *** markwash has quit IRC | 16:33 | |
| *** markwash_ is now known as markwash | 16:33 | |
| *** dizquierdo has quit IRC | 16:35 | |
| *** rallegre has joined #openstack-infra | 16:37 | |
| *** dkranz has joined #openstack-infra | 16:38 | |
| *** markmc has quit IRC | 16:40 | |
| *** zhiyan is now known as zhiyan_ | 16:40 | |
| *** zhiyan_ is now known as zhiyan | 16:40 | |
| *** basha_ has joined #openstack-infra | 16:41 | |
| *** basha_ has quit IRC | 16:42 | |
| *** Sukhdev has quit IRC | 16:42 | |
| *** zhiyan is now known as zhiyan_ | 16:43 | |
| *** e0ne has joined #openstack-infra | 16:44 | |
| *** basha_ has joined #openstack-infra | 16:44 | |
| *** nati_ueno has joined #openstack-infra | 16:44 | |
| *** afazekas has quit IRC | 16:46 | |
| *** e0ne has quit IRC | 16:46 | |
| *** e0ne has joined #openstack-infra | 16:47 | |
| *** gnuoy has quit IRC | 16:47 | |
| *** gnuoy has joined #openstack-infra | 16:48 | |
| *** nati_ueno has quit IRC | 16:48 | |
| *** bnemec is now known as beekneemech | 16:50 | |
| *** gnuoy has quit IRC | 16:50 | |
| *** gnuoy has joined #openstack-infra | 16:50 | |
| *** zns has quit IRC | 16:51 | |
| *** e0ne has quit IRC | 16:51 | |
| *** aburaschi has joined #openstack-infra | 16:53 | |
| *** wchrisj has joined #openstack-infra | 16:56 | |
| *** david_lyle_ has joined #openstack-infra | 16:57 | |
| *** zns has joined #openstack-infra | 16:58 | |
| *** rallegre has quit IRC | 16:59 | |
| *** basha_ has quit IRC | 16:59 | |
| *** zns has quit IRC | 17:00 | |
| *** david-lyle has quit IRC | 17:00 | |
| *** david_lyle_ has quit IRC | 17:02 | |
| *** dkranz has quit IRC | 17:04 | |
| *** rallegre has joined #openstack-infra | 17:05 | |
| superdan | what's going on with the check queue? | 17:08 |
| *** rwsu has quit IRC | 17:09 | |
| fungi | superdan: it should be catching up now | 17:12 |
| superdan | fungi: okay, what happened? | 17:12 |
| fungi | but slowly since we're running only on rackspace for the moment | 17:12 |
| superdan | ah, I guess that answers the other quesiton? :) | 17:12 |
| fungi | the way we removed hpcloud yesterday apparently screwed with how nodepool does its demand ratio calculations | 17:12 |
| superdan | oh, why did we remove hp cloud? | 17:12 |
| *** ihrachyshka has quit IRC | 17:12 | |
| fungi | so things were wonky until i fiddled with it a little while ago | 17:12 |
| *** reed has joined #openstack-infra | 17:13 | |
| fungi | tcp connection timeouts to our servers in rackspace | 17:13 |
| fungi | around 1-2% but enough to make most jobs run there fail badly | 17:13 |
| superdan | huh, interesting | 17:13 |
| *** ildikov_ has quit IRC | 17:13 | |
| *** leakypipes has quit IRC | 17:15 | |
| *** dkranz has joined #openstack-infra | 17:16 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: More thoroughly temporarily pull hp from nodepool https://review.openstack.org/86960 | 17:16 |
| fungi | okay, that ^ should allow me to turn the puppet agent back on before i head to lunch | 17:16 |
| *** moted has quit IRC | 17:16 | |
| *** moted has joined #openstack-infra | 17:17 | |
| *** homeless has quit IRC | 17:18 | |
| clarkb | http://paste.openstack.org/show/75583/ I think that gets Makefiles out of my system for the day | 17:19 |
| clarkb | it is slightly repetetive but fixes yesterdays issues | 17:19 |
| *** amcrn has joined #openstack-infra | 17:19 | |
| clarkb | you can use it as an almost drop in tox replacement now | 17:19 |
| openstackgerrit | A change was merged to openstack-infra/config: More thoroughly temporarily pull hp from nodepool https://review.openstack.org/86960 | 17:22 |
| fungi | pulling that ^ onto the puppetmaster and then onto nodepool and starting puppet there so i can grab lunch | 17:22 |
| *** praneshp has joined #openstack-infra | 17:22 | |
| *** nati_ueno has joined #openstack-infra | 17:22 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:22 | |
| *** nati_ueno has quit IRC | 17:23 | |
| *** homeless has joined #openstack-infra | 17:23 | |
| *** jgallard has quit IRC | 17:23 | |
| *** rallegre has quit IRC | 17:23 | |
| *** homeless has quit IRC | 17:23 | |
| *** nati_ueno has joined #openstack-infra | 17:24 | |
| fungi | done. puppet agent is back running on nodepool.o.o | 17:24 |
| fungi | now disappearing for food | 17:24 |
| *** Ryan_Lane has joined #openstack-infra | 17:24 | |
| jeblair | fungi: nonono don't remove hp from the config entirely! | 17:25 |
| jeblair | that's not what i suggested at all | 17:25 |
| jeblair | just remove it from the node labels | 17:25 |
| jeblair | eg, tell it "don't build devstack-precise nodes on hp" | 17:25 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/reviewstats: Add qa-specs to tracked projects https://review.openstack.org/86963 | 17:25 |
| clarkb | jeblair: ooohhhh | 17:25 |
| jeblair | fungi: if you remove it entirely, the next time it cleans up images, it will delete the existing images in hp | 17:25 |
| clarkb | jeblair: yes that already happened | 17:26 |
| jeblair | fungi: and then we'll be spending two days trying to get | 17:26 |
| jeblair | sigh | 17:26 |
| clarkb | or is happening | 17:26 |
| jeblair | clarkb: can you please add hp back and start doing image builds in a loop there? | 17:27 |
| jeblair | so that when the problem is fixed we don't have to wait _days_ to get an image build that makes it through hp's metadata server? | 17:27 |
| clarkb | jeblair: sure | 17:27 |
| jeblair | clarkb: thanks | 17:28 |
| clarkb | I will propose two changes a revert of the removal and one on top of that to remove them from the node labels | 17:28 |
| jeblair | sdague: i think branchless tempest is going to have to wait until next week, sorry | 17:28 |
| jeblair | (rather, the questions you want to ask me about it) | 17:28 |
| *** morganfainberg_Z is now known as morganfainberg | 17:30 | |
| sdague | jeblair: ok. Though there are time constraints, because the minute we set the stable/icehouse branch we need to be about ready to go. But if we can do it monday, that's cool | 17:30 |
| clarkb | woo vim still segfaults on that file | 17:31 |
| * clarkb installs emacs | 17:31 | |
| mtreinish | clarkb: don't go over to the darkside | 17:34 |
| *** chandan_kumar has joined #openstack-infra | 17:35 | |
| clarkb | mtreinish: I wish I didn't need to but saucy's vim compile segfaults on nodepool.yaml.erb | 17:35 |
| clarkb | I could build vim from scratch and debug the problem but installing emacs gets this edit done quickly | 17:35 |
| *** thomasem has quit IRC | 17:36 | |
| *** thomasem has joined #openstack-infra | 17:36 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Revert "More thoroughly temporarily pull hp from nodepool" https://review.openstack.org/86968 | 17:37 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Don't use hpcloud as a label provider. https://review.openstack.org/86969 | 17:37 |
| clarkb | jeblair: fungi ^ | 17:37 |
| *** khyati has joined #openstack-infra | 17:37 | |
| jeblair | clarkb: vim mode | 17:38 |
| clarkb | ooh /me investigates this thing | 17:38 |
| clarkb | jeblair: if my changes look good to you you can approve and I can babysit | 17:38 |
| jeblair | clarkb: lgtm | 17:39 |
| *** e0ne has joined #openstack-infra | 17:39 | |
| jeblair | done | 17:39 |
| jeblair | clarkb: erm, though you might want to force-merge them | 17:39 |
| clarkb | oh right | 17:41 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "More thoroughly temporarily pull hp from nodepool" https://review.openstack.org/86968 | 17:42 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't use hpcloud as a label provider. https://review.openstack.org/86969 | 17:42 |
| clarkb | puppet is going to update shortly, I won't fight it | 17:43 |
| *** ociuhandu has quit IRC | 17:44 | |
| *** basha_ has joined #openstack-infra | 17:46 | |
| *** thomasem has quit IRC | 17:46 | |
| *** thomasem has joined #openstack-infra | 17:46 | |
| rcarrillocruz | hey guys, anyone with some spare time to review https://review.openstack.org/#/c/86382/ ? it adds LP-Gerrit integration for changes that are abandoned... | 17:50 |
| *** david-lyle has joined #openstack-infra | 17:54 | |
| *** marun has joined #openstack-infra | 17:55 | |
| *** e0ne has quit IRC | 17:55 | |
| *** e0ne has joined #openstack-infra | 17:56 | |
| *** basha_ has quit IRC | 17:57 | |
| *** ilives has quit IRC | 17:58 | |
| *** rallegre has joined #openstack-infra | 17:58 | |
| *** e0ne_ has joined #openstack-infra | 18:01 | |
| *** e0ne has quit IRC | 18:01 | |
| *** jp_at_hp has quit IRC | 18:01 | |
| *** esker has joined #openstack-infra | 18:03 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use threads to update jobs in jenkins https://review.openstack.org/45371 | 18:05 |
| *** wchrisj has quit IRC | 18:06 | |
| *** ominakov has joined #openstack-infra | 18:08 | |
| *** morganfainberg is now known as morganfainberg_Z | 18:10 | |
| *** ominakov has quit IRC | 18:10 | |
| *** morganfainberg_Z is now known as morganfainberg | 18:11 | |
| *** rallegre has quit IRC | 18:16 | |
| *** e0ne_ has quit IRC | 18:16 | |
| *** e0ne has joined #openstack-infra | 18:16 | |
| *** talluri has joined #openstack-infra | 18:17 | |
| *** ildikov has joined #openstack-infra | 18:18 | |
| clarkb | jeblair: hrm, looking at the jobs launched graph I wonder if what you suggest doesn't work as expected | 18:21 |
| clarkb | jeblair: there is clearly a spike where fungi did his thing then a fall off later | 18:21 |
| *** adalbas has quit IRC | 18:23 | |
| *** unbounced has joined #openstack-infra | 18:23 | |
| clarkb | fungi: ^ | 18:24 |
| clarkb | hrm a py3k test did just start so maybe not? | 18:25 |
| *** marun has quit IRC | 18:26 | |
| *** marun has joined #openstack-infra | 18:28 | |
| *** jasondotstar has joined #openstack-infra | 18:30 | |
| *** chandan_kumar has quit IRC | 18:30 | |
| *** david_lyle_ has joined #openstack-infra | 18:31 | |
| *** david-lyle has quit IRC | 18:34 | |
| *** sdake_ has joined #openstack-infra | 18:34 | |
| jeblair | clarkb: 1 building node, 1 used | 18:34 |
| *** sdake_ has quit IRC | 18:34 | |
| *** sdake_ has joined #openstack-infra | 18:34 | |
| *** liam__ has joined #openstack-infra | 18:35 | |
| jeblair | clarkb: since we now have single-use nodes with sudo, we could conditionally change the version of tox on the system based on the content of tox.ini | 18:35 |
| jeblair | clarkb: to handle a phased migration across projects | 18:35 |
| *** liam__ has quit IRC | 18:36 | |
| jeblair | if we don't convince hpk | 18:36 |
| clarkb | jeblair: ya | 18:36 |
| clarkb | jeblair: did you see my makefile? | 18:36 |
| *** david_lyle_ has quit IRC | 18:36 | |
| clarkb | I think it is not sucky >_> | 18:36 |
| jeblair | clarkb: no, and neat :) | 18:36 |
| *** adalbas has joined #openstack-infra | 18:36 | |
| *** Sukhdev has joined #openstack-infra | 18:36 | |
| clarkb | jeblair: the other thing we can do to be slightly mean is change tox versions forcing all projects that don't update before hand to update as first order of business | 18:37 |
| clarkb | jeblair: http://paste.openstack.org/show/75583/ tested with zuul | 18:37 |
| *** esker has quit IRC | 18:37 | |
| jeblair | things i learned about ansible: it fails fast, and stores a retry file so it can try failed hosts again | 18:37 |
| clarkb | make venv ARGS='command I want to run' | 18:37 |
| mgagne | jeblair: and I guess you stayed for the puppet one | 18:38 |
| jeblair | mgagne: actually i just jumped over to the angularjs talk | 18:38 |
| jeblair | but then i'm going back for the salt talk | 18:39 |
| *** dims has quit IRC | 18:39 | |
| sdague | jeblair: they lest angularjs in to pycon? :) | 18:39 |
| jeblair | sdague: "straigtening out angularjs with python" :) | 18:40 |
| sdague | nice | 18:40 |
| clarkb | jeblair: looking at 5 minute graph points from graphite for jobs launched over the last 3 hours it actually doesn't look too bad | 18:40 |
| *** _nadya_ has joined #openstack-infra | 18:40 | |
| *** david-lyle has joined #openstack-infra | 18:40 | |
| *** andreykurilin_ has joined #openstack-infra | 18:40 | |
| clarkb | I think ourchange may not have impacted things like I initiall thought | 18:40 |
| jeblair | storyboard is using it and horizon is looking so i figured i should drop in | 18:40 |
| jeblair | clarkb: cool | 18:40 |
| *** marun has quit IRC | 18:41 | |
| clarkb | I think deletes on rax are not as instantaneous as would be desired | 18:41 |
| krotscheck | wha? who? someone said angular? | 18:41 |
| clarkb | so deleted nodes hang around for a while preventing new nodes from being allocated | 18:41 |
| jeblair | ansible has lots of included modules, and lots of roles are available in a community-contributed 'galaxy' | 18:42 |
| jeblair | we should be able to easily do the separate password thing like we do with hiera | 18:42 |
| jeblair | and it has openstack api support | 18:42 |
| jeblair | and the primary author is in durham and went to ncsu | 18:42 |
| jeblair | mordred, fungi: ^ important angular stuff; especially the last :) | 18:43 |
| jeblair | err ansible | 18:43 |
| jeblair | someone in the room keeps saying angular | 18:43 |
| clarkb | jeblair: I wasn't a fan of the config files. They read very clunky | 18:43 |
| clarkb | but relying on ssh instead of $othertransport is a really neat feature to bootstrap nodes | 18:43 |
| jeblair | clarkb: the examples he showed looked like nice yaml; they're going for "infrastructure is data" rather than "infrastructure is code" | 18:44 |
| jeblair | clarkb: but i haven't seen many yet; i should look more | 18:44 |
| clarkb | jeblair: right, but then they treat that data as code :P. iirc its basically an imperative lisp like language with yaml as the representation | 18:44 |
| jeblair | clarkb: i'm sold! | 18:45 |
| clarkb | :P | 18:45 |
| jeblair | ok, paying attention to talk now, bbl | 18:46 |
| clarkb | I have evil working in emacs now btw | 18:46 |
| clarkb | for whenever vim decides to segfault | 18:46 |
| clarkb | I need to grab lunch shortly | 18:46 |
| fungi | jeblair: clarkb: the hpcloud images had already been marked for deletion, though i think that was because of the updates for the new jenkins ssh keys? | 18:47 |
| *** rlandy has quit IRC | 18:48 | |
| *** rlandy has joined #openstack-infra | 18:48 | |
| fungi | though if they're not deleted completely yet, i can un-mark them in the db | 18:49 |
| *** chandan_kumar has joined #openstack-infra | 18:53 | |
| *** SumitNaiksatam has quit IRC | 18:55 | |
| *** wchrisj has joined #openstack-infra | 18:56 | |
| fungi | also, my earlier tcp pings from hpcloud to pypi.o.o are now returning 0.0% loss, so whatever was going on seems to have subsided | 18:57 |
| *** miqui has quit IRC | 18:59 | |
| *** Ryan_Lane has quit IRC | 19:03 | |
| *** dprince has quit IRC | 19:03 | |
| *** Ryan_Lane has joined #openstack-infra | 19:03 | |
| *** Ryan_Lane has joined #openstack-infra | 19:03 | |
| *** khyati has quit IRC | 19:04 | |
| fungi | running a few more tests, but whatever the issue was seems to have finally subsided | 19:06 |
| *** pcrews has quit IRC | 19:06 | |
| clarkb | fungi: no, I hadn't marked anything for deletion because of keys, I think nodepool did that when it noticed the provider had gone awway | 19:07 |
| Shrews | jeblair: many awesome people are in durham | 19:08 |
| clarkb | fungi: that is great news. I had kicked off some hpcloud image builds for new keys and not delete state | 19:08 |
| clarkb | fungi: I will continue to kick those off so that we have vialbe images if we can use hpcloud again | 19:08 |
| fungi | both copies of old images in hpcloud-az{1,2,3} for all four image types have been switched back from delete to ready after confirming via nova image-list that they still all actually do exist | 19:08 |
| *** nati_ueno has quit IRC | 19:09 | |
| fungi | but yes, updated images in those azs would be good. you want my "loop until created" shell script i've been using? | 19:10 |
| clarkb | fungi: I may switch to that, first I want to be able to watch it a bit more so that I can understand where the trouble may exist | 19:10 |
| fungi | well, looping over them doesn't stop you from seeing where they fail, but in the past i've definitely seen them timing out consistently trying to get a response from the metadata server | 19:11 |
| clarkb | ok, I should try to follow up on that more forcefully | 19:11 |
| clarkb | I see that happen in my personal account too. dunno why tracking down this issue is so problematic | 19:12 |
| fungi | in region b they build the first try every time | 19:12 |
| fungi | but in the old azs they time out on that most of the time | 19:12 |
| clarkb | huh region b iswhere my personal account fials | 19:12 |
| fungi | or at least in region b they succeed quickly enough i never notice | 19:13 |
| fungi | okay, several more tests of 1000 tcp/syn packets each from a couple old nodes in az2 and i had only one timeout | 19:13 |
| clarkb | I'm here to help if it becomes a problem again but we are in a pretty sad state without those extra nodes | 19:14 |
| *** unbounced has left #openstack-infra | 19:16 | |
| *** e0ne has quit IRC | 19:17 | |
| *** talluri has quit IRC | 19:18 | |
| *** e0ne has joined #openstack-infra | 19:18 | |
| fungi | i also find it hard to believe, comparing the graphs vs the changes in flight, that we're running anywhere near as many jobs as is implied by the used node count | 19:18 |
| *** miqui has joined #openstack-infra | 19:20 | |
| *** Ryan_Lane1 has joined #openstack-infra | 19:22 | |
| *** rwsu has joined #openstack-infra | 19:24 | |
| *** thedodd has quit IRC | 19:29 | |
| *** thedodd has joined #openstack-infra | 19:31 | |
| mgagne | jeblair: not sure why each "properties" are prefixed by an hypen if they are unique like an hash/map key =) | 19:33 |
| jeblair | mgagne: is ordering useful at all? | 19:35 |
| *** SumitNaiksatam has joined #openstack-infra | 19:35 | |
| *** pcrews has joined #openstack-infra | 19:36 | |
| mgagne | jeblair: you mean the implicit top to bottom execution or the ordering implied by the YAML syntax used? | 19:36 |
| jeblair | mgagne: this is an implementation detail, but pyyaml will not raise an error if you duplicate a key, but you can check for that if you put yours keys in a list | 19:36 |
| *** _nadya_ has quit IRC | 19:37 | |
| jeblair | mgagne: the second; i'm brainstorming why they might have done that | 19:37 |
| jeblair | mgagne: but yeah, aside from those ideas, i don't know; it looks like a dict would work fine :) | 19:38 |
| mgagne | jeblair: I agree | 19:38 |
| mgagne | jeblair: or it could be that "templates" could be used/reused in a resource (user) definition | 19:39 |
| mgagne | jeblair: could this be done? https://gist.github.com/mgagne/10495489 | 19:40 |
| *** freyes has joined #openstack-infra | 19:41 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove devstack-check label and increase centos/py3k values https://review.openstack.org/87008 | 19:42 |
| fungi | jeblair: mgagne: i've noted we've used list types in a lot of our various yaml files where associative arrays would have made more sense | 19:42 |
| *** dcramer_ has quit IRC | 19:42 | |
| jeblair | clarkb, fungi: ^ | 19:42 |
| jeblair | worth considering; but i can't basysit right now; up to you ^ | 19:42 |
| mgagne | jeblair: I like the reactor thing. Could it be used to orchestrate/sync a multi-node deployment where an other node could depends on a particular state of an other node? | 19:42 |
| fungi | jeblair: mgagne: though agreed on the long-standing and oft-requested feature of getting pyyaml to raise exceptions on duplicate keys in an associative array | 19:43 |
| clarkb | it lgtm and I can babysit | 19:43 |
| clarkb | fungi: want to look at that change and force it through if you think it will help? | 19:43 |
| jeblair | mgagne: yes, anteaya and fungi were looking into that but didn't get all the details worked out | 19:43 |
| fungi | clarkb: yep | 19:43 |
| *** nati_ueno has joined #openstack-infra | 19:43 | |
| mgagne | jeblair: cool, will keep an eye on your work, I have an interest in salt | 19:43 |
| jeblair | mgagne: i'm not sure if we're going to finish that soon -- we will probably do something hacky near term to fix our manage-projects problem | 19:44 |
| jeblair | then do something nicer after that | 19:45 |
| *** khyati has joined #openstack-infra | 19:46 | |
| fungi | also, we could probably consider turning hpcloud back on soon... my repeated tests are continuing to come back clean (0.0% failure rates now) | 19:47 |
| jeblair | fungi: ++ | 19:47 |
| fungi | though not a couple hours ago it was around 1.5% failures | 19:47 |
| fungi | so depending on how long we want to see it stable before we jump back in... | 19:48 |
| jeblair | fungi: i think we've got the pattern for taking it in/out quickly now | 19:48 |
| fungi | agreed. simpler if we need to do it again, and we know what to be looking for this time | 19:48 |
| fungi | so we'll be able to diagnose the situation much more rapidly as well | 19:48 |
| jeblair | mgagne: i'm heading to the openstack booth now | 19:49 |
| mgagne | jeblair: cool | 19:49 |
| fungi | was there already a wip change up for readding hpcloud, or shall i go ahead and get it ready? | 19:49 |
| clarkb | fungi: with current node starvation levels you may have to force submit that change to get it in | 19:49 |
| clarkb | fungi: there is not a wip change yet | 19:49 |
| fungi | clarkb: force submit isn't needed | 19:50 |
| fungi | clarkb: i merely jumped the check with zuul enqueue | 19:50 |
| clarkb | ninja | 19:50 |
| fungi | since the gate has priority | 19:50 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove devstack-check label and increase centos/py3k values https://review.openstack.org/87008 | 19:50 |
| fungi | and there it is | 19:50 |
| *** CaptTofu has quit IRC | 19:52 | |
| *** wchrisj has quit IRC | 19:56 | |
| *** dcramer_ has joined #openstack-infra | 19:56 | |
| *** _nadya_ has joined #openstack-infra | 19:56 | |
| *** CaptTofu has joined #openstack-infra | 19:58 | |
| *** vipuls is now known as vipul | 19:58 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Reenable hpcloud in nodepool https://review.openstack.org/87014 | 20:00 |
| fungi | clarkb: jeblair: mordred: SergeyLukjanov: ^ | 20:00 |
| *** ociuhandu has joined #openstack-infra | 20:00 | |
| clarkb | fungi: lgtm, though if we can wait a little longer on it so that I can try to finish buildingthese hpcloud images that would be great | 20:01 |
| fungi | clarkb: agreed, having fresh images there would be convenient | 20:01 |
| *** wchrisj has joined #openstack-infra | 20:01 | |
| clarkb | we need 7 more to build | 20:02 |
| fungi | if i were a betting man, i'd say 4 of those are in az2 | 20:03 |
| clarkb | 2 are in az2, 4 are az1 and one is az3 | 20:03 |
| fungi | oh, wow, so you haven't had an image stick yet in az1? | 20:03 |
| clarkb | nope | 20:04 |
| fungi | maybe az1 is the new az2 | 20:04 |
| *** jasondotstar has quit IRC | 20:04 | |
| *** sdake_ has quit IRC | 20:05 | |
| *** julim has quit IRC | 20:05 | |
| clarkb | I am going to start trying concurrent builds shortly | 20:06 |
| *** david-lyle has quit IRC | 20:06 | |
| sdague | is paste.openstack.org out of commission? | 20:09 |
| *** _nadya_ has quit IRC | 20:10 | |
| *** chandan_kumar has quit IRC | 20:10 | |
| fungi | doesn't seem so. system load and memory utilization are relatively nominal | 20:12 |
| sdague | hmmm... ok. Well I'm connecting from public wifi, so maybe it's just silly | 20:13 |
| sdague | oh, it finally connected | 20:13 |
| clarkb | fungi: region-b and az3 are done now | 20:14 |
| fungi | awesome | 20:14 |
| *** roaet has joined #openstack-infra | 20:14 | |
| *** freyes has quit IRC | 20:14 | |
| *** reed has quit IRC | 20:15 | |
| dhellmann | are the issues you're working on related to the long queue waiting for pypy nodes? | 20:15 |
| fungi | dhellmann: indirectly, yes | 20:16 |
| dhellmann | fungi: ok, as long as that's not a new problem :-) | 20:17 |
| fungi | dhellmann: putting hpcloud back into service shortly after about 24 hours of connectivity issues between it and servers in rackspace | 20:17 |
| fungi | dhellmann: so we've been running below 50% of our usual capacity | 20:17 |
| dhellmann | so just a capacity thing, and not that some node types only run in one cloud? | 20:18 |
| *** aysyd has quit IRC | 20:18 | |
| fungi | and starvation is bringing out some corner cases of our node type demand calculations, so we're also doing some retuning there while we have the opportunity | 20:18 |
| dhellmann | ah | 20:18 |
| fungi | dhellmann: right | 20:18 |
| fungi | we're down ~500 nodes of capacity | 20:18 |
| fungi | dhellmann: https://review.openstack.org/#/c/87014/1/modules/openstack_project/templates/nodepool/nodepool.yaml.erb | 20:19 |
| dhellmann | ok, like I said, I knew there were some issues with hpcloud and wanted to make sure that the pypy thing wasn't different | 20:19 |
| fungi | dhellmann: yep, related | 20:19 |
| dhellmann | ah, neat, I haven't really looked at the nodepool stuff before | 20:20 |
| fungi | that's the change to turn hpcloud back on now that we're seeing the network issues have subsided, but we're also trying to get fresh images there while we're at it since we need those to be able to phase out some keys we're replacing as part of post-heartbleed due dilligence | 20:21 |
| *** nati_ueno has quit IRC | 20:22 | |
| dhellmann | might as well clean things up while the azs are offline anyway | 20:24 |
| *** maxbit has joined #openstack-infra | 20:24 | |
| *** david-lyle has joined #openstack-infra | 20:26 | |
| *** rustlebee is now known as russellb | 20:27 | |
| *** nati_ueno has joined #openstack-infra | 20:28 | |
| *** roaet has quit IRC | 20:28 | |
| *** weshay has quit IRC | 20:28 | |
| *** roaet_ has joined #openstack-infra | 20:28 | |
| *** dims has joined #openstack-infra | 20:28 | |
| *** thuc_ has quit IRC | 20:29 | |
| *** dims_ has joined #openstack-infra | 20:29 | |
| *** thuc has joined #openstack-infra | 20:29 | |
| clarkb | down to 2 centos6 images and one bare-preice and one devstack-precise | 20:32 |
| *** thuc_ has joined #openstack-infra | 20:33 | |
| *** aysyd has joined #openstack-infra | 20:33 | |
| *** dims has quit IRC | 20:33 | |
| *** thuc_ has quit IRC | 20:33 | |
| *** dprince has joined #openstack-infra | 20:34 | |
| *** thuc_ has joined #openstack-infra | 20:34 | |
| *** thuc has quit IRC | 20:34 | |
| *** Sukhdev has quit IRC | 20:36 | |
| *** aysyd has quit IRC | 20:37 | |
| *** Ryan_Lane1 has quit IRC | 20:37 | |
| *** ameade has quit IRC | 20:42 | |
| *** nati_ueno has quit IRC | 20:43 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Navigate to story details after create complete https://review.openstack.org/86571 | 20:45 |
| *** rwsu has quit IRC | 20:46 | |
| *** rwsu has joined #openstack-infra | 20:46 | |
| *** CaptTofu has quit IRC | 20:48 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make jenkins proposal jobs use dedicated user. https://review.openstack.org/87026 | 20:50 |
| clarkb | fungi: ^ how does that look? I can generate the new key and create the jenkins-proposal service account if it looks good to you | 20:50 |
| *** harlowja is now known as harlowja_away | 20:51 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Navigate to story details after create complete https://review.openstack.org/86571 | 20:51 |
| *** mfer has quit IRC | 20:53 | |
| clarkb | fungi: thoughts on putting hpcloud into use with the exception of the 4 images we are still waiting on? | 20:53 |
| *** UtahDave has joined #openstack-infra | 20:53 | |
| fungi | clarkb: well, we have images for them, just not as recent | 20:54 |
| clarkb | hpcloud az1 centos6, hpcloud az1 devstack-precise, hpcloud az2 centos6, and hpcloud az2 bare-precise | 20:54 |
| *** e0ne has quit IRC | 20:56 | |
| clarkb | fungi: I think jeblairs change may have affecting things oddly | 20:56 |
| clarkb | we are running fewer tests now... | 20:56 |
| *** e0ne has joined #openstack-infra | 20:56 | |
| clarkb | fungi: re images, I cannot swap out the jenkins key until after all slave images have the new key | 20:56 |
| clarkb | those 4 above don't have the new key yet | 20:56 |
| *** gyee has quit IRC | 20:57 | |
| fungi | clarkb: right, it's a question of whether you want to hold off using part of hpcloud and rotate the keys early, or keep trying to get those last few images | 20:58 |
| *** mwagner_lap has quit IRC | 20:58 | |
| fungi | clarkb: one option would be to just delete the handful of old images for those so that we won't get broken nodes and nodepool will keep retrying to make new images in the meantime | 20:58 |
| *** jaypipes has joined #openstack-infra | 20:58 | |
| *** msuriar has quit IRC | 20:59 | |
| clarkb | yeah I had considered that but that may leave that az out of commission for a very long time depending on how this goes | 20:59 |
| *** harlowja_away is now known as harlowja | 21:00 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Adds metadata to story detail page https://review.openstack.org/87030 | 21:00 |
| *** tjones has joined #openstack-infra | 21:00 | |
| *** e0ne has quit IRC | 21:00 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Adds metadata to story detail page https://review.openstack.org/87030 | 21:00 |
| *** jeckersb is now known as jeckersb_gone | 21:01 | |
| *** msuriar has joined #openstack-infra | 21:01 | |
| fungi | clarkb: what are the odds that nodepoold is gauging the demand for devstack-precise nodes as low because it's seeing all the dsvm jobs in the check pipeline as wanting the devstack-precise-check label? | 21:02 |
| clarkb | maybe it is doing that? I wouldn't expect it to since it doesn't know about the check label anymore right? | 21:03 |
| *** miqui has quit IRC | 21:03 | |
| tjones | im trying to ./run-tests.sh on a mac. it's always dying on EnvironmentError: mysql_config not found. do you think i need to install all of mysql to get this working? or is there a subset? | 21:04 |
| clarkb | tjones: I would use tox | 21:04 |
| tjones | clarkb: not make ;-) | 21:04 |
| tjones | ? | 21:04 |
| clarkb | tjones: right not yet. Tox is still the thing | 21:04 |
| clarkb | would require a lot of work to stop using tox | 21:04 |
| tjones | clarkb: ok i'll give that a shot. thanks | 21:05 |
| clarkb | tjones: and you should only need mysqlclient libs for unittests | 21:05 |
| fungi | clarkb: http://paste.openstack.org/show/75592/ | 21:05 |
| *** wchrisj has quit IRC | 21:05 | |
| *** wenlock has quit IRC | 21:05 | |
| tjones | clarkb: can you remind me which version of tox? | 21:06 |
| tjones | 1.6.1 or something? | 21:06 |
| fungi | clarkb: perhaps the main loop is doing nothing now because of that keyerror exception | 21:06 |
| clarkb | tjones: yes that is the version | 21:06 |
| clarkb | fungi: that would explain it, I guess we can't remove labels without a restart? | 21:07 |
| *** UtahDave has quit IRC | 21:07 | |
| *** nati_ueno has joined #openstack-infra | 21:07 | |
| fungi | clarkb: starting to look that way | 21:07 |
| fungi | time to add a story | 21:07 |
| sdague | tjones: you still need mysql_config today | 21:07 |
| clarkb | hpcloud az2 bare-precise is built now | 21:07 |
| *** UtahDave has joined #openstack-infra | 21:08 | |
| tjones | sdague: can you point me to which package contains mysql_config? | 21:08 |
| sdague | on a mac, no idea :) | 21:08 |
| sdague | it's whatever has the development headers | 21:09 |
| tjones | on ubuntu (may be the same) | 21:09 |
| tjones | ? | 21:09 |
| sdague | the mysql driver needs to be compiled | 21:09 |
| mordred | morning clarkb | 21:09 |
| clarkb | mordred: morning | 21:09 |
| tjones | sdague: i'll ask google ;-) | 21:09 |
| sdague | tjones: libmysqlclient-dev on ubuntu | 21:09 |
| NobodyCam | tjones: you prob have the file just looking in the wrong place | 21:10 |
| NobodyCam | see: http://stackoverflow.com/questions/15314791/django-error-vertualenv-environmenterror-mysql-config-not-found | 21:10 |
| *** pdmars has quit IRC | 21:10 | |
| clarkb | fungi: may as well add the key back for now? | 21:10 |
| tjones | NobodyCam: i was trying to avoid installing all of mysql | 21:10 |
| fungi | https://storyboard.openstack.org/#!/story/64 | 21:10 |
| * NobodyCam thinks thats how I fixed.. but I agree with clarkb I run tox | 21:10 | |
| tjones | NobodyCam: limited download from where i am | 21:10 |
| NobodyCam | ahh | 21:11 |
| krotscheck | fungi: I should add carriage return support, shouldn’t i.... | 21:11 |
| fungi | krotscheck: meh, i like the one huge paragraph look... ;) | 21:12 |
| fungi | clarkb: i was just going to restart nodepool again since it's doing nearly nothing now anyway | 21:12 |
| fungi | clarkb: objections? | 21:12 |
| clarkb | fungi: that works too | 21:12 |
| clarkb | go for it | 21:12 |
| *** dprince has quit IRC | 21:13 | |
| fungi | doing | 21:14 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make jenkins proposal jobs use dedicated user. https://review.openstack.org/87026 | 21:14 |
| clarkb | fungi: jeblair ^ that makes the new proposal bot account slightly better documented | 21:15 |
| clarkb | if you guys are happy with that name, email address, and username I will create that user and merge the change | 21:15 |
| fungi | clarkb: lgtm | 21:16 |
| mordred | clarkb: looks good - why review@o.o ? | 21:16 |
| mordred | (_much_ clearer to have it split out with its own name) | 21:17 |
| clarkb | mordred: because that address actually has an MX that cares about it | 21:17 |
| *** nati_ueno has quit IRC | 21:17 | |
| *** aburaschi has quit IRC | 21:18 | |
| clarkb | mordred: I can use the openstack-infra@lists.openstack.org address too | 21:18 |
| clarkb | mordred: open to ideas here | 21:18 |
| fungi | the ml doesn't seem like a bad choice there | 21:19 |
| mordred | I wish we ran o.o email (not really, but a little bit) so that we could add mail aliases in commits like this where we're adding system accoutn things | 21:20 |
| clarkb | fungi: mordred ok I will switch it to he list address | 21:20 |
| mordred | proposal-bot@o.o would be nice - but not sure it's worth the effort | 21:20 |
| *** dims_ has quit IRC | 21:20 | |
| mordred | clarkb: honestly, I think I'm fine with review@o.o - either way really | 21:20 |
| clarkb | the list is a bit more clear I think | 21:20 |
| fungi | if we really wanted we could puppet lists.o.o:/etc/aliases lines | 21:20 |
| fungi | and make infra.o.o a cname to it and add it as a local domain name in exim | 21:21 |
| *** nati_ueno has joined #openstack-infra | 21:21 | |
| fungi | then proposals@infra.openstack.org would just dump to openstack-infra@lists.openstack.org | 21:22 |
| fungi | or something along those lines | 21:22 |
| fungi | </brainstorm> | 21:22 |
| mordred | fungi: I was just thinking a similar thing - but we probably don't have to do that today :) | 21:22 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make jenkins proposal jobs use dedicated user. https://review.openstack.org/87026 | 21:23 |
| clarkb | there, I have added the hiera keys to hiera so you can approve at will | 21:23 |
| clarkb | I still need to create the account though | 21:23 |
| clarkb | but wanted consensus on the details before doing that | 21:23 |
| fungi | mordred: *i* am almost certainly not going to do that today. it's about time for me to stop working and start packing for the next two weeks of internetlessness | 21:23 |
| clarkb | fungi: how is nodepool doing? | 21:23 |
| fungi | clarkb: graceful restart is going to have to be aborted i think | 21:24 |
| clarkb | k | 21:24 |
| fungi | i'll make a quick list of remaining nodes between killing and starting, then delete those in an hour or so once i'm sure they're no longer busy | 21:24 |
| *** dcramer_ has quit IRC | 21:25 | |
| fungi | okay, nodepoold is running again now | 21:26 |
| *** e0ne has joined #openstack-infra | 21:27 | |
| clarkb | fungi: mordred: so current account details good? I will create the account if so | 21:27 |
| fungi | as for hpcloud, my two scratch instances there looped redownloading the pbr tarball a combined ~20,000 times in a row without a single failure | 21:28 |
| *** e0ne has quit IRC | 21:28 | |
| *** e0ne has joined #openstack-infra | 21:29 | |
| mordred | clarkb: ++ | 21:29 |
| fungi | clarkb: e-mail change lgtm | 21:29 |
| clarkb | az1 and az2 are missing centos6 images, otherwise all images are up to date | 21:29 |
| mordred | fungi: ooh. so hp is fixed? | 21:29 |
| clarkb | I have 2 concurrent builds going for each to hopefully pseed things along | 21:29 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Fixed display of user-entered whitespace. https://review.openstack.org/87033 | 21:29 |
| krotscheck | clarkb, fungi: ^^ | 21:29 |
| fungi | mordred: as of a couple hours ago when i returned from lunch, the communication issues have magically cleared up | 21:29 |
| mordred | excellent | 21:29 |
| clarkb | ok creating new account now | 21:30 |
| fungi | krotscheck: that was quick! | 21:30 |
| krotscheck | Easy repro, easy fix. | 21:30 |
| fungi | krotscheck: i even thought about entering a story for that, but hadn't gotten around to doing so yet ;) | 21:30 |
| mordred | krotscheck: w00t | 21:30 |
| krotscheck | I also happened to be digging into that part of the code at the time, so everything was set up :) | 21:30 |
| krotscheck | Though, well, the queue’s a bit wonky | 21:30 |
| clarkb | fungi: mordred: when I create propsal bot via the ssh command line which group should it go in? | 21:30 |
| fungi | white-space: pre-wrap | 21:31 |
| fungi | there's the magic beans | 21:31 |
| mordred | krotscheck: so, how hard would it be to also get it to display urls in comments as links? am I getting ahead of myself? | 21:31 |
| clarkb | Non-interactive users? | 21:31 |
| mordred | clarkb: yeah, I thin kso | 21:31 |
| clarkb | or continuous integration tools? | 21:31 |
| mordred | oh | 21:31 |
| krotscheck | mordred: That requires rather more sophisticated HTML parsing. | 21:31 |
| mordred | hrm | 21:31 |
| clarkb | I think we want non interactive | 21:31 |
| krotscheck | So, maybe 15 minutes? | 21:31 |
| mordred | hehe | 21:31 |
| krotscheck | :-P | 21:31 |
| clarkb | so that this doesn't get special privs that jenkins/zuul would get via group membership | 21:31 |
| clarkb | that sound right? | 21:31 |
| mordred | clarkb: yeah - non-interactive | 21:32 |
| krotscheck | mordred: That’s actually a bigger question, because right now the form fields are click-to-edit, which I don’t think is going to fly if we want different sections of the text be clickable. | 21:32 |
| mordred | krotscheck: that is an excellent point | 21:32 |
| fungi | clarkb: hrm... i think so. checking the all-projects acl now | 21:32 |
| krotscheck | I suspect that click-to-edit is less useful than having links in descriptions. | 21:32 |
| *** cody-somerville has joined #openstack-infra | 21:32 | |
| fungi | krotscheck: ctrl-meta-cokebottle-click-to-edit | 21:33 |
| *** SumitNaiksatam has quit IRC | 21:33 | |
| fungi | "ease of use!" | 21:33 |
| mordred | fungi: M-x edit-field | 21:33 |
| mordred | krotscheck: can we add an embedded emacs? | 21:33 |
| krotscheck | mordred: only if we can embed storyboard in emacs. | 21:34 |
| mordred | fungi: https://review.openstack.org/#/c/86943/ and https://review.openstack.org/#/c/86946/ will let me go clean up the db, btw | 21:34 |
| mordred | krotscheck: I _have_ written an emacs major mode before ... it may have been a few years though | 21:34 |
| krotscheck | It’s emacs, all the way down. | 21:35 |
| fungi | clarkb: yeah, "Non-Interactive Users" looks safe since we just need a registered user for this anyway | 21:35 |
| clarkb | yup creating user now | 21:35 |
| fungi | clarkb: it seems to have no special rights at all in all-projects | 21:35 |
| mordred | krotscheck: now if only emacs could deploy emacs on bare metal... | 21:36 |
| mordred | wait | 21:36 |
| * mordred shuts up | 21:36 | |
| clarkb | fungi: ya I think it is a collection point for service accounts that don't need special rights | 21:36 |
| * krotscheck lights the devananda signal! | 21:36 | |
| *** david-lyle has quit IRC | 21:38 | |
| clarkb | ok user is created | 21:38 |
| clarkb | mordred: I have evil working in emacs now as my emergency editor | 21:38 |
| clarkb | I am going to approve the puppet change | 21:38 |
| fungi | clarkb: so it's worse than that... jenkins node labels which nodepool doesn't know about cause that error, restart or not | 21:39 |
| clarkb | huh | 21:39 |
| clarkb | because they are in geard? | 21:39 |
| fungi | clarkb: in short, it's still throwing that error after the restart | 21:39 |
| clarkb | :( I guess we add the label back? | 21:39 |
| fungi | and continuing to do nothing | 21:39 |
| fungi | yep, reverting that part of the change | 21:39 |
| *** hogepodge_ has quit IRC | 21:41 | |
| clarkb | are you going to do it in place or propose a change? | 21:41 |
| fungi | clarkb: both | 21:41 |
| clarkb | if in place we can add in all but az1/2 centos6 to the label list | 21:41 |
| *** mrodden1 is now known as mrodden | 21:42 | |
| fungi | clarkb: in that case should i amend 87014 with those tweaks instead? | 21:42 |
| *** rlandy has quit IRC | 21:42 | |
| clarkb | fungi: sure | 21:42 |
| fungi | that way we don't have to stop puppet agent on nodepool and then have people forget it's stopped | 21:43 |
| clarkb | ++ | 21:43 |
| *** signed8bit has quit IRC | 21:43 | |
| *** stillbert is now known as mikal | 21:45 | |
| *** tjones has left #openstack-infra | 21:46 | |
| *** SumitNaiksatam has joined #openstack-infra | 21:47 | |
| *** nati_ueno has quit IRC | 21:47 | |
| *** nati_ueno has joined #openstack-infra | 21:49 | |
| *** jasondotstar has joined #openstack-infra | 21:49 | |
| *** aysyd has joined #openstack-infra | 21:49 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Reenable hpcloud in nodepool https://review.openstack.org/87014 | 21:49 |
| zaro | fungi: gerrit 2.8.4 was just released. what is your preference on picking this one? | 21:50 |
| zaro | fungi: i mean how should we update our current 2.8.3 fork with 2.8.4? A few of our changes have already merged in our gerrit-2.8.3 fork. | 21:51 |
| *** maxbit has quit IRC | 21:51 | |
| *** CaptTofu has joined #openstack-infra | 21:51 | |
| fungi | zaro: we should probably port forward to 2.8.4 before we upgrade our production systems, so as to reduce the delta of patches we're carrying | 21:53 |
| fungi | zaro: though clarkb/mordred/jeblair will have to help with that... i'm about to vanish in an unreachable way until the upgrade week | 21:54 |
| fungi | zaro: and if we do end up only upgrading to 2.8.3+morepatches because of lack of manpower/testing for 2.8.4, it's probably not the end of the world | 21:55 |
| fungi | grar stray whitespace in my new patchset. fixing | 21:56 |
| *** CaptTofu has quit IRC | 21:56 | |
| *** mriedem has quit IRC | 21:56 | |
| *** Ryan_Lane1 has joined #openstack-infra | 21:56 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Reenable hpcloud in nodepool https://review.openstack.org/87014 | 21:57 |
| zaro | fungi: ok. that's what i thought. thanks. | 21:57 |
| fungi | clarkb: if 87014 lgty i'll go ahead and pull it from a puppet dev env onto nodepool.o.o | 21:58 |
| clarkb | looking | 21:58 |
| fungi | oh, apparently needs a rebase | 21:59 |
| clarkb | fungi: +2 otherwise | 21:59 |
| fungi | or not--my dev env needed a reset ;) | 22:00 |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/config: Bump OS_TEST_TIMEOUT for ironic-virtual d-s-g https://review.openstack.org/87043 | 22:04 |
| clarkb | /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill': execution expired (Timeout::Error) is what is plaguing centos6 btw | 22:06 |
| *** CaptTofu has joined #openstack-infra | 22:07 | |
| clarkb | jeblair: fungi mordred fyi I am going to have my eyeball(s) looked at again on Monday morning PST. They will probably be dilated again and I won't be useful until the afternoon | 22:07 |
| fungi | clarkb: i think that is the facter trying to access the metadata server issue | 22:07 |
| clarkb | fungi: ah good to know | 22:07 |
| fungi | clarkb: though i could be wrong, i believe it manifested as a ruby execution timeout | 22:08 |
| clarkb | yeah because facter sees the ec2 stuff and tries to get it | 22:08 |
| fungi | in happier news, nodepool's debug logs look MUCH healthier | 22:08 |
| clarkb | \o/ | 22:08 |
| devananda | krotscheck: ?? | 22:09 |
| devananda | krotscheck: ooh. no. it should be vim. | 22:10 |
| *** david-lyle has joined #openstack-infra | 22:10 | |
| openstackgerrit | A change was merged to openstack-infra/config: Reenable hpcloud in nodepool https://review.openstack.org/87014 | 22:11 |
| fungi | and things ^ are being tested/merged once more! | 22:11 |
| fungi | devananda: agreed, we already have an operating system anyway | 22:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Mark projects that use storyboard https://review.openstack.org/86943 | 22:11 |
| krotscheck | fungi: I hope you’re talking about AmigaDOS. Because nothing else comes close. | 22:11 |
| fungi | krotscheck: vax/vms 4evar! | 22:12 |
| krotscheck | If my scotch is older than your operating system, I’m not interested. | 22:12 |
| *** davidlenwell_ is now known as davidlenwell | 22:13 | |
| fungi | krotscheck: Initial release October 25, 1977; 36 years ago | 22:14 |
| fungi | krotscheck: sounds like it would make some mighty good scotch! | 22:14 |
| krotscheck | Wow, it’s almost as old as I am! | 22:15 |
| fungi | youngster! | 22:15 |
| *** gyee has joined #openstack-infra | 22:16 | |
| fungi | amigaos (of which amigados was a part) is only 28 years old, but would still be pretty darn tasty | 22:16 |
| nibalizer | are we gonna get openstack running on a PDP11 next? | 22:17 |
| fungi | nibalizer: that's what hp has decided on for the new new new project moonshot, didn't you hear? | 22:17 |
| nibalizer | omg yess | 22:17 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Only create projects we ask for https://review.openstack.org/86946 | 22:18 |
| fungi | arm and atom are so last quarter/fiscal year | 22:18 |
| fungi | nibalizer: on the other hand, if they retarget it to pdp-10, then you could run it on a toad too | 22:18 |
| *** blamar has quit IRC | 22:20 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Hard delete for auth stuff https://review.openstack.org/86271 | 22:20 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Logging configuration improvement https://review.openstack.org/86558 | 22:20 |
| *** mbacchi has quit IRC | 22:20 | |
| *** aysyd has quit IRC | 22:21 | |
| *** jaypipes has quit IRC | 22:21 | |
| *** mwagner_lap has joined #openstack-infra | 22:26 | |
| fungi | puppet is running again on nodepool.o.o | 22:26 |
| mordred | woot | 22:27 |
| *** CaptTofu has quit IRC | 22:32 | |
| openstackgerrit | A change was merged to openstack-infra/config: make contactstore option overridable https://review.openstack.org/85753 | 22:34 |
| *** thedodd has quit IRC | 22:36 | |
| mordred | krotscheck, fungi, clarkb: double check me here: https://etherpad.openstack.org/p/ZC8Rzox4pS | 22:39 |
| fungi | mordred: i personally know of no others we're using it for yet | 22:39 |
| mordred | fungi: does the sql seem reasonable? | 22:40 |
| krotscheck | …seeems ok. | 22:40 |
| fungi | mordred: i'm not really up on the storyboard schema, but syntax looks reasonable | 22:40 |
| krotscheck | We may want to purge all stories that have no tasks? | 22:40 |
| clarkb | I don't know storyboard'd schema but that list of projects looks good | 22:40 |
| *** thuc_ has quit IRC | 22:41 | |
| mordred | krotscheck: hrm. those are possible? | 22:41 |
| *** salv-orlando_ has joined #openstack-infra | 22:43 | |
| *** salv-orlando has quit IRC | 22:44 | |
| *** salv-orlando_ is now known as salv-orlando | 22:44 | |
| krotscheck | mordred: Yup. As per the meeting on monday, we’re allowing hard deletion of tasks. That means we can theoretically have a story whose tasks have all been deleted. | 22:47 |
| mordred | krotscheck: so then we _shouldn't delete task-less stories | 22:48 |
| *** david-lyle has quit IRC | 22:48 | |
| mordred | I'm running the queries - I've taken a db backup in case I'm doing something terrible | 22:48 |
| mordred | krotscheck: ok. done | 22:50 |
| mordred | the db is now clean | 22:50 |
| mordred | and I've re-run load_projects by hand just to make sure we didn't grow them again | 22:51 |
| *** jasondotstar has quit IRC | 22:59 | |
| fungi | i guess the only new project patch proposed this week is jeblair's 86627 for stackforge/python-jenkins | 23:00 |
| *** david-lyle has joined #openstack-infra | 23:00 | |
| fungi | though we do still need to run manage-projects to catch acl changes which were made over the course of the week (i forget which ones they were) | 23:00 |
| krotscheck | mordred: Awesome. Looks good | 23:00 |
| fungi | anybody have opinions on whether we should merge 86627 and/or run manage-projects? | 23:01 |
| clarkb | fungi: might be good to save that for after mordred is done | 23:02 |
| fungi | clarkb: agreed | 23:02 |
| *** prad_ has quit IRC | 23:03 | |
| *** yamahata has quit IRC | 23:07 | |
| *** UtahDave has quit IRC | 23:09 | |
| *** markwash has quit IRC | 23:17 | |
| *** nati_ueno has quit IRC | 23:18 | |
| mordred | fungi, clarkb: I wouldn't mind using that as a test-case for some things ... | 23:25 |
| clarkb | yeah we can hold off on it for now | 23:25 |
| mordred | krotscheck: has there been any discussion aroudn the possibility of making urls operate on names too? | 23:26 |
| *** nati_ueno has joined #openstack-infra | 23:26 | |
| krotscheck | Oh, you mean having the URL be human readable rather than an ID? | 23:26 |
| mordred | krotscheck: like, it would be neat if: https://storyboard.openstack.org/#!/project/668/stories was also reachable by https://storyboard.openstack.org/#!/project/openstack-infra/nodepool/stories | 23:26 |
| mordred | yeah | 23:26 |
| mordred | so that I can type them from memory :) | 23:26 |
| mordred | or infer them | 23:26 |
| mordred | I have no idea how hard/involved that woudl be or what the ramifications are | 23:27 |
| mordred | https://storyboard.openstack.org/#!/story/63 seems fine - I dont' expect to ever want to express that any other way | 23:27 |
| krotscheck | We started there, actually - the issue was that we ended up with name conflicts. | 23:27 |
| mordred | on the projects urls? | 23:27 |
| NobodyCam | fungi: did I see that use just rebuilt the bm nodes? I'm really strange ssh errors https://jenkins03.openstack.org/job/check-tripleo-ironic-undercloud-precise/267/console | 23:28 |
| krotscheck | And, quite frankly, if you’re having to resort to direct URL modification to navigate storyboard, then the UI has failed. | 23:28 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/os-loganalyze: Enable log analysis for Ironic https://review.openstack.org/87056 | 23:28 |
| clarkb | NobodyCam: I rebuilt teh tripleo precise image today | 23:29 |
| NobodyCam | gah... s/I'm really strange/I'm SEEING really strange/ | 23:29 |
| clarkb | and we have added a second key to authorized_keys for the jenkins user | 23:29 |
| clarkb | what are the details on that particular ssh connection and why does it insist on trying over and over? | 23:30 |
| krotscheck | mordred: It’s basically the same as reaching for google maps because highway signage sucks. Better ways of navigating make secondary workarounds unecessary. | 23:30 |
| clarkb | NobodyCam: that looks like the seed node isn't starting ssh? | 23:30 |
| clarkb | and not actually something related to the host slave? | 23:30 |
| NobodyCam | ack just saw the errors, didn't recall see ehis error b4 ... thought I'd check | 23:31 |
| mordred | krotscheck: indeed to taht - I think my main usecase tends to be giving people references to lists of things in slides, irc or emails | 23:34 |
| mordred | like "look at the zuul bugs at http://bugs.launchpad.net/zuul" (as a for instance) | 23:34 |
| fungi | krotscheck: i'm not entirely convinced of that argument. for me the primary ui entrypoint is my web browser, not a specific web page where i click things to find the things i want. being able to see in my url tab completion which url was the one to the bugs for a given project is helpful | 23:34 |
| fungi | based on prior navigation history | 23:34 |
| krotscheck | I can buy that. | 23:35 |
| fungi | browser bookmarks are so 1995 ;) | 23:35 |
| mordred | wait - your browser supports bookmarks? | 23:35 |
| * mordred surfs the web with curl and vi | 23:35 | |
| fungi | and tab-completing the url to the thing i want saves me additional interaction | 23:35 |
| NobodyCam | lynx command line browser all the way | 23:36 |
| NobodyCam | curl and vi tooo much work :-p | 23:36 |
| fungi | anyway, just a use case anecdote. i don't feel so strongly that i think it should be prioritized in any way | 23:36 |
| mordred | ++ | 23:36 |
| *** rfolco has quit IRC | 23:37 | |
| *** mriedem has joined #openstack-infra | 23:37 | |
| mordred | gerrit certainly does not have a url scheme like that - I _do_ tend to bookmark searches there | 23:37 |
| *** flaper87 is now known as flaper87|afk | 23:38 | |
| fungi | the browser plug-in i use for mouseless interaction implements keyword-based autosearching through my url history, so with a couple keywords on the browser command line i actually quite easily get to prior gerrit searches i've performed | 23:38 |
| clarkb | mordred: any chance you can poke people about metadata servers not being reliable particularly on centos6 images? | 23:39 |
| mordred | clarkb: I can try? | 23:39 |
| clarkb | mordred: fwiw I tried before | 23:39 |
| fungi | for example, i found the list of new projects for this week not by typing a query into the gerrit search box, but just by typing new-project into the command line of the browser and it was the first result in my history | 23:40 |
| NobodyCam | clarkb: will thoses test time out? https://github.com/openstack/tripleo-incubator/blob/master/scripts/boot-seed-vm#L213-L216 the code does not look like it will. | 23:41 |
| clarkb | NobodyCam: eventually it will | 23:41 |
| NobodyCam | :) | 23:42 |
| clarkb | NobodyCam: your code really should give up after a couple hundred attempts :P | 23:42 |
| clarkb | maybe even after 1 attempt | 23:42 |
| clarkb | since permissions denied probably won't change | 23:42 |
| NobodyCam | ack will put a patch to time out | 23:43 |
| mordred | clarkb: permissions denied might change | 23:43 |
| clarkb | NobodyCam: actually, there may be a race between ssh keys being put in place by whatever injection mechanism you are usign and you starting ssh | 23:43 |
| mordred | yeah. that | 23:44 |
| clarkb | mordred: yeah I just realized that nodepool runs into that too | 23:44 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add nova-specs https://review.openstack.org/86949 | 23:44 |
| clarkb | NobodyCam: so a timer would be good | 23:44 |
| mordred | :) | 23:44 |
| clarkb | NobodyCam: iirc we have a 120second timer on nodepool | 23:44 |
| clarkb | but it is configurable pre cloud | 23:44 |
| NobodyCam | :) | 23:44 |
| *** timrc is now known as timrc-afk | 23:45 | |
| *** CaptTofu has joined #openstack-infra | 23:46 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Fixed display of user-entered whitespace. https://review.openstack.org/87033 | 23:48 |
| *** yamahata has joined #openstack-infra | 23:49 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add qa-specs to tracked projects https://review.openstack.org/86963 | 23:50 |
| *** ilives has joined #openstack-infra | 23:54 | |
| *** nati_ueno has quit IRC | 23:56 | |
| *** ilives has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!