| openstackgerrit | Merged openstack-infra/tripleo-ci: No longer split gerrit ref on " " https://review.openstack.org/259508 | 00:00 |
|---|---|---|
| jeblair | fungi: this change exists: https://review.openstack.org/259281 | 00:00 |
| jeblair | i do not know whether it would have an effect on the leak | 00:00 |
| jeblair | fungi: we could try merging it and see | 00:00 |
| *** mdenny has joined #openstack-infra | 00:00 | |
| fungi | jeblair: oh, right, i had already mostly reviewed that one | 00:01 |
| jeblair | other than that, i think our choices are wait for jhesketh to return and investigate it, or revert the connections branch. | 00:01 |
| fungi | i'll run back through it real quick and approve if cool | 00:01 |
| anteaya | is jhesketh on holidays? | 00:01 |
| jeblair | yes | 00:01 |
| * anteaya nods | 00:01 | |
| fungi | clarkb also seems to have looked over that some, but is probably not really around to rereview | 00:02 |
| *** peristeri has quit IRC | 00:02 | |
| *** bpokorny has joined #openstack-infra | 00:02 | |
| jeblair | i would offer to try to fix it myself, but even in zuulv3, i still haven't gotten to the point where i understand how connections can interact with the rest of the data model. | 00:02 |
| jeblair | so i'm very likely to just repeat my incomplete zuulv3 work on the v2 branch, and i don't think that's helpful. | 00:03 |
| anteaya | would a restart make any difference? | 00:03 |
| jeblair | anteaya: yes, for a while. | 00:03 |
| jeblair | i think the most sensible thing to do is to merge https://review.openstack.org/259281 and restart it and see if it makes a difference. | 00:04 |
| *** dimtruck is now known as zz_dimtruck | 00:04 | |
| fungi | 259281 lgtm, i can restart zuul once it merges, unless someone else is around to do that | 00:04 |
| * anteaya has no objections | 00:04 | |
| jeblair | by the time we can tell if it has, we can probably hand the problem off to jhesketh | 00:04 |
| anteaya | heh | 00:04 |
| clarkb | sorry not really here right this second, but if my comments appear to be addressed I had nothing else | 00:04 |
| fungi | yeah, just want to keep an eye on it for a few days (assuming this patch isn't like zomg crazy revert now) | 00:05 |
| *** Qiming has quit IRC | 00:06 | |
| jhesketh | Hey. So yes, I'm on leave until Tuesday but am trying to take a look. However I only have my phone with me at the moment | 00:06 |
| jeblair | jhesketh: go back on holiday! :) | 00:06 |
| jhesketh | So we don't know what it is exactly, but some kind of memory leak right | 00:06 |
| *** ayoung has quit IRC | 00:06 | |
| fungi | jhesketh: don't look | 00:06 |
| fungi | jhesketh: vacation is useless if you don't commit to it ;)\ | 00:07 |
| jhesketh | Heh | 00:07 |
| anteaya | jhesketh: see you next week | 00:07 |
| fungi | we're giving your change a whirl to see if it helps, and if not it'll be here waiting for you one way or another | 00:07 |
| jhesketh | Okay but if there's anything I can do to help or particular questions etc please ping | 00:07 |
| mgagne | *jhesketh has been kicked from #openstack-infra channel* | 00:08 |
| fungi | heh | 00:08 |
| anteaya | ha ha ha | 00:08 |
| jhesketh | (I won't answer if I'm busy ;-)) | 00:08 |
| jhesketh | Heh | 00:08 |
| *** sabeen3 has quit IRC | 00:09 | |
| jhesketh | Yep that seems wise. When was the last time zuul was restarted? Does anybody know how long we may have been on the connection changes | 00:09 |
| jhesketh | Ie I thought we had been running it at least for a bit without any obvious regressions | 00:09 |
| fungi | jhesketh: connection changes merged sometime between the gerrit upgrade and the last zuul restart prior to that. at the gerrit upgrade related restart we see memory usage climb to ~2x its previous steady state. zuul was restarted again on ~ december 23 for reasons only irc channel logging probably remembers | 00:10 |
| fungi | however, load on our system really didn't pick back up until this past week, when memory usage climbed to >10x that | 00:11 |
| krotscheck | Will using puppet to include mod_rewrite do all the fancy allowoverride things as well? | 00:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Validate grafyaml based on master https://review.openstack.org/265009 | 00:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: Update OVH graph with axis labels https://review.openstack.org/265010 | 00:11 |
| anteaya | fungi: the dec 23 log doesn't have the words zuul and restart in the same sentence anywhere I can find | 00:12 |
| jeblair | btw, 'last' is really useful with ansible. | 00:13 |
| *** SumitNaiksatam has joined #openstack-infra | 00:13 | |
| fungi | anteaya: i was approximating. looks like ~16:00 utc on december 22 | 00:13 |
| jhesketh | Hmm okay.. So it could be a number of things but connections possibly most suspicious | 00:13 |
| *** SumitNaiksatam has left #openstack-infra | 00:14 | |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=392&rra_id=3&view_type=&graph_start=1450759057&graph_end=1450845174&graph_height=120&graph_width=500&title_font_size=10 | 00:14 |
| fungi | jhesketh: well, also lots of things about gerrit potentially changed at that point, so could also be something tickled by that | 00:14 |
| jhesketh | I would prefer to avoid reverting if possible. I think it'd be difficult and very time consuming to find the leak and re review everything | 00:14 |
| jeblair | jhesketh: yeah, it's at the top of my list because it touches the Change objects and event handling. second on my list is fungi's theory of 'unhandled gerrit events cause leaks'. | 00:14 |
| anteaya | fungi: ah | 00:15 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Add wheel mirror to mirrors https://review.openstack.org/253521 | 00:16 |
| jeblair | jhesketh: well, if we revert, it'll be a single commit, and we can un-revert with a single commit followed by a small commit that fixes the problem. alternatively, we could pin to a prior sha. | 00:16 |
| anteaya | fungi: yep http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-12-22.log.html#t2015-12-22T16:02:13 | 00:16 |
| krotscheck | jeblair: Finally got around to including that rewrite rule ^^ | 00:16 |
| * krotscheck goes and fixes the wheel distribution job now. | 00:16 | |
| mgagne | was about to ask for y axis legend/unit =) | 00:16 |
| jhesketh | Sure, I just mean finding the cause before the revert might be easier, but of course that's not necessarily true | 00:16 |
| jeblair | jhesketh: yeah, if you want to run in a debugger, etc. | 00:17 |
| openstackgerrit | Merged openstack-infra/project-config: Adds an API migration and docs generation tool, fairy-slipper https://review.openstack.org/245352 | 00:19 |
| openstackgerrit | Merged openstack-infra/puppet-ansible: Stop installing the puppet inventory https://review.openstack.org/250926 | 00:19 |
| openstackgerrit | Merged openstack-infra/zuul: Fix regression in change tracking https://review.openstack.org/259281 | 00:19 |
| *** jaypipes has quit IRC | 00:19 | |
| krotscheck | Oh, drat. Made a mistake | 00:20 |
| *** Keedya has joined #openstack-infra | 00:20 | |
| *** tzn has quit IRC | 00:21 | |
| *** dims has joined #openstack-infra | 00:22 | |
| jeblair | fungi: are you doing the restart once that propagates or shall i? | 00:24 |
| *** Keedya has quit IRC | 00:24 | |
| *** dahoo has joined #openstack-infra | 00:24 | |
| *** dahoo is now known as jerryz | 00:24 | |
| fungi | jeblair: it's all yours if you want it. i can stick around or fade out to evening | 00:24 |
| jeblair | fungi: have an evening, i'll do it | 00:24 |
| fungi | jeblair: thanks! | 00:25 |
| *** salv-orlando has quit IRC | 00:25 | |
| fungi | now i need to stop polishing this atc listing script | 00:25 |
| *** Keedya has joined #openstack-infra | 00:25 | |
| *** salv-orlando has joined #openstack-infra | 00:25 | |
| anteaya | fungi: happy evening to you | 00:25 |
| anteaya | the cacti graphs have a blank at 23:40 | 00:26 |
| jeblair | 2016-01-08 00:26:11,853 DEBUG zuul.Scheduler: Project None not found | 00:26 |
| jeblair | the number of those lines is also disappointing. | 00:26 |
| jeblair | (and may be a clue) | 00:27 |
| jeblair | anteaya: that means the machine was too busy to respond to cacti | 00:27 |
| anteaya | ah makes sense | 00:28 |
| jeblair | anteaya: which makes sense since it was also too busy to respond to us :) | 00:28 |
| anteaya | right | 00:28 |
| fungi | anteaya: cacti polls snmp over udp by default, and udp doesn't get retransmits or retries or anything, so if the machine is busy... silence | 00:28 |
| anteaya | ah | 00:28 |
| jeblair | i have stopped zuul | 00:28 |
| anteaya | okey dokey | 00:28 |
| fungi | as opposed to tcp which is a little more resilient (if we used snmp v3 we could opt to switch to tcp instead of udp) | 00:29 |
| anteaya | I got what I needed, blank space == bad | 00:29 |
| anteaya | others may need something more | 00:29 |
| fungi | when you have to fill in the blanks with snmp, fill them in with all your worst fears | 00:29 |
| anteaya | my worst fears don't fit in that tiny space | 00:30 |
| anteaya | my passing fears perhaps | 00:30 |
| anteaya | whimsical fears | 00:30 |
| *** mriedem has quit IRC | 00:31 | |
| *** openstackgerrit has quit IRC | 00:32 | |
| *** openstackgerrit has joined #openstack-infra | 00:32 | |
| jeblair | re-enqueue complete | 00:34 |
| anteaya | yay | 00:35 |
| jeblair | hi | 00:35 |
| jeblair | can someone who decided to run bash8 on project-config tell me how to fix this: 2016-01-08 00:20:40.861 | [E] E011: Then keyword is not on same line as if or elif keyword: 'if [ ! -e /usr/zuul-env/bin/zuul-cloner ] &&' | 00:35 |
| jerryz | anteaya: is it possible to push stable/liberty code to newly created networking-* project? | 00:35 |
| fungi | jeblair: while not a fan of linters myself, i think it means do `if foo ; then` | 00:36 |
| anteaya | jerryz: why would you want that? | 00:36 |
| *** mriedem has joined #openstack-infra | 00:36 | |
| fungi | jeblair: rather than putting the "then" on a following line | 00:36 |
| jeblair | fungi: right, i'm kind of assuming, things being the way they are around here, that we have a severe allergy to lines longer than 80 chars as well. | 00:36 |
| anteaya | jerryz: stable/liberty has passed | 00:36 |
| jeblair | fungi: so i'm sort of trying to reconcile the two. | 00:36 |
| fungi | jeblair: dunno, antihistamines are working out well for me | 00:37 |
| jeblair | A set of rules that help keep things consistent in control blocks. These are ignored on long lines that have a continuation, because unrolling that is kind of "interesting" | 00:37 |
| *** otter768 has joined #openstack-infra | 00:37 | |
| jeblair | according to the README | 00:37 |
| jeblair | so | 00:37 |
| jeblair | erm, i guess i need to put a \ on the line to make bashate happy | 00:37 |
| jeblair | despite the fact that the syntax doesn't require it at all | 00:37 |
| fungi | i guess it doesn't like a natural line break in place of a ; for solving long lines | 00:38 |
| jeblair | and in fact, the thing it's railing against is the perfectly intentional aspect of the syntax that allows it to be on more than one line? | 00:38 |
| fungi | yeah | 00:38 |
| jerryz | anteaya: because we claim to support liberty but the project was just recently created upstream | 00:38 |
| *** oomichi has joined #openstack-infra | 00:38 | |
| fungi | jeblair: perhaps the best fix is to put E011 in the ignore list | 00:39 |
| jeblair | http://docs.openstack.org/developer/bashate is a 404 | 00:39 |
| mriedem | anyone else seeing this today? http://logs.openstack.org/80/256180/1/gate/gate-grenade-dsvm/d1909a9/logs/grenade.sh.txt.gz#_2016-01-07_19_40_08_188 | 00:39 |
| jeblair | fungi: that seems slightly better form than what i was about to do. any idea where we keep the ignore list? | 00:39 |
| *** Keedya has quit IRC | 00:40 | |
| fungi | i had hoped it was in tox.ini... no? | 00:40 |
| fungi | hrm, bashate is probably _not_ a flake8 plugin | 00:40 |
| jeblair | this is what i mean when i say this stuff is a waste of peoples' time | 00:41 |
| anteaya | 260966 and 260967 are the two bashate changes to project-config | 00:41 |
| *** thorst has joined #openstack-infra | 00:41 | |
| *** otter768 has quit IRC | 00:42 | |
| jeblair | 260967 apparently explains how to ignore things | 00:42 |
| anteaya | mriedem: I have not | 00:43 |
| mriedem | anteaya: bingo https://pypi.python.org/pypi/django_compressor/2.0 | 00:43 |
| mriedem | 2.0 release breaks things | 00:43 |
| anteaya | wooooo | 00:43 |
| * anteaya is so surprised | 00:43 | |
| fungi | jeblair: also http://git.openstack.org/cgit/openstack-dev/bashate/tree/doc/source/man/bashate.rst#n37 | 00:43 |
| fungi | -I apparently | 00:44 |
| jeblair | and, for some reason, comma separated, even though argparse allows repitition | 00:44 |
| jeblair | (which i mention because apparently, i guessed wrong) | 00:44 |
| anteaya | jerryz: well it seems to me you have two options, 1) come to grips with the fact we are in mitaka and re-evaluate your project claims or 2) try to work against our release workflow to satisfy your claims | 00:45 |
| anteaya | I'm supportive of 1) | 00:45 |
| fungi | old optparse habits die hard? | 00:46 |
| jeblair | erm | 00:47 |
| jeblair | i get different output when i run bashate locally | 00:47 |
| fungi | different bashate release? | 00:47 |
| jeblair | fungi: using tox | 00:47 |
| jeblair | and i assure you, i've never run that tox env before :) | 00:47 |
| fungi | hrm. interestinger | 00:47 |
| openstackgerrit | Jesse Keating proposed openstack/requirements: Cap django_compressor below 2.0 https://review.openstack.org/265025 | 00:47 |
| jeblair | Running bashate on /home/corvus/git/openstack-infra/project-config/tools/run-compare-xml.sh | 00:47 |
| jeblair | [W] E006: Line too long: '/usr/zuul-env/bin/zuul-cloner -m ../tools/run-compare-clonemap.yaml --cache-dir /opt/git git://git.openstack.org openstack-infra/jenkins-job-builder' | 00:47 |
| jeblair | oh wait | 00:48 |
| jeblair | maybe that's because of the argparse thing | 00:48 |
| *** piet has joined #openstack-infra | 00:48 | |
| fungi | -I E006,E011 | 00:48 |
| fungi | ;) | 00:48 |
| *** Swami_ has quit IRC | 00:49 | |
| timrc | pabelanger: Cool. I'll look this evening when I trick my kids to go to sleep | 00:49 |
| docaedo | Hi, I got an lint test failure with package update failures like "W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/trusty-updates/main/binary-amd64/Packages Hash Sum mismatch" (http://logs.openstack.org/98/260198/7/gate/gate-app-catalog-npm-run-lint/c578247/console.html) | 00:49 |
| *** zz_dimtruck is now known as dimtruck | 00:49 | |
| docaedo | I'm not sure what to do about it? | 00:49 |
| jeblair | docaedo: wait and try again | 00:49 |
| *** ybathia has joined #openstack-infra | 00:50 | |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Cap django_compressor<2.0 https://review.openstack.org/265026 | 00:50 |
| mriedem | any reqs stable peeps around? ^ | 00:50 |
| mriedem | sdague: dhellmann: dims: mtreinish: ^ | 00:50 |
| docaedo | jeblair: thanks - I'll make some dinner and then .. for "try again" do I add a recheck comment or something? This was after giving a workflow +1 | 00:50 |
| fungi | docaedo: it's a long-winded way of saying "the us.archive.ubuntu.com mirror your job is hitting is broken" | 00:51 |
| mtreinish | mriedem: +2 | 00:51 |
| dims | mriedem : +1 from me. i was just looking at pypi+django like 2 mins before i saw your message :) | 00:51 |
| fungi | docaedo: yeah, recheck comment in your review | 00:51 |
| mriedem | yeah, 2.0 drops support for django<1.8 | 00:51 |
| *** bpokorny has quit IRC | 00:51 | |
| mriedem | and stable/kilo g-r has django capped at 1.8 | 00:51 |
| docaedo | fungi: thanks, thats what I thought but wanted to make sure I wasn't doing something wrong | 00:52 |
| *** bpokorny has joined #openstack-infra | 00:52 | |
| docaedo | fungi jeblair - thank you, I'll give it a shot again in a bit :) | 00:52 |
| fungi | we're all doing something wrong. just enjoy your dinner ;) | 00:52 |
| docaedo | hahaha | 00:52 |
| *** bpokorny has quit IRC | 00:53 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Validate grafyaml based on master https://review.openstack.org/265009 | 00:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: Update OVH graph with axis labels https://review.openstack.org/265010 | 00:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Ignore E011 in bashate https://review.openstack.org/265027 | 00:53 |
| *** bpokorny has joined #openstack-infra | 00:53 | |
| jeblair | anteaya: you're aware that the project-config repo runs a job that performs grafyaml validation, right? | 00:55 |
| anteaya | yes | 00:56 |
| anteaya | my laptop is faster, plus I get to feel I am learning something | 00:56 |
| *** woodster_ has quit IRC | 00:56 | |
| jeblair | anteaya: okay; just be careful zuul doesn't complain to the union :) | 00:56 |
| anteaya | oh I'm a good union member | 00:57 |
| * anteaya waves her I.A.T.S.E card | 00:57 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for django_compressor 2.0 bug 1532048 https://review.openstack.org/265028 | 00:57 |
| openstack | bug 1532048 in OpenStack Dashboard (Horizon) "grenade fails setting up horizon with "ImportError: No module named utils" for compressor package" [Undecided,Confirmed] https://launchpad.net/bugs/1532048 | 00:57 |
| mriedem | mtreinish: ^ | 00:57 |
| mtreinish | mriedem: your just full of demands tonight | 00:57 |
| oomichi | mriedem: we are facing the problem on stable/liberty also. so do you have it fo liberty? | 00:57 |
| anteaya | mtreinish: that's different than any other night? | 00:57 |
| oomichi | mriedem: ah, I maybe misunderstand. it is not necessary | 00:58 |
| mriedem | oomichi: it's failing liberty b/c of grenade | 00:58 |
| mriedem | grenade in liberty fails to install on the kilo side | 00:58 |
| oomichi | mriedem: yeah, I see the point | 00:58 |
| mtreinish | anteaya: well, just because I'm paying attention tonight | 00:58 |
| anteaya | mtreinish: ah points for you | 00:58 |
| *** pvaneck has quit IRC | 00:58 | |
| oomichi | mriedem: because the problem happens on grenade only | 00:59 |
| *** mudassirlatif has quit IRC | 01:00 | |
| zaro | fungi: looks like there a gerrit bug. Currently the required permission to allow viewing emails is Modify Account but it should be View All Accounts. | 01:00 |
| zaro | and additional bug is that modify accounts permission should automatically allow users to view accounts. | 01:01 |
| zaro | will have a fix up shortly. | 01:01 |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Cap django_compressor below 2.0 https://review.openstack.org/265025 | 01:02 |
| mriedem | sdague: want to approve this one https://review.openstack.org/#/c/265025/ ? jlk got that up first | 01:02 |
| mtreinish | mriedem: done | 01:03 |
| *** Keedya has joined #openstack-infra | 01:03 | |
| mriedem | thanks | 01:03 |
| fungi | zaro: have a link to the issue url? i'd like to track it in a code comment for future cleanup | 01:04 |
| zaro | writing it up now :) | 01:05 |
| openstackgerrit | Tony Breeds proposed openstack/requirements: Cap django-compressor https://review.openstack.org/265029 | 01:05 |
| *** Keedya has quit IRC | 01:06 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add baremetal hpuswest node definition https://review.openstack.org/208751 | 01:07 |
| StevenK | crinkle: My task on that story can be marked invalid, we likely are not going to deploy using packages, and we don't have any infrastructure around building them anyway. | 01:07 |
| gmann | clarkb: ping | 01:09 |
| *** Qiming has joined #openstack-infra | 01:09 | |
| anteaya | tonyb: mriedem has 265025 up | 01:09 |
| tonyb | anteaya: Thanks I saw that too late. | 01:10 |
| gmann | clarkb: i remember we had issue with server host name in case of multinode where it gets appended with some extra number | 01:10 |
| anteaya | ah, cool | 01:10 |
| tonyb | anteaya: I've dropped mine | 01:10 |
| gmann | clarkb: we had issue in live migration tempest tests | 01:10 |
| tonyb | 3 people all looking at grenade failures for liberty. That's gotta be a record :) | 01:11 |
| gmann | clarkb: its hitting in new multinode tests too - http://logs.openstack.org/60/257660/13/experimental/experimental-tempest-dsvm-multinode-full-dibtest/153a3aa/console.html#_2016-01-07_07_55_57_815 | 01:11 |
| *** Swami has joined #openstack-infra | 01:11 | |
| anteaya | tonyb: well better than noone looking | 01:11 |
| * anteaya rings a bell | 01:11 | |
| crinkle | StevenK: okay thank you | 01:11 |
| jlk | mriedem: with this change you and I both made, does devstack complete okay? I honestly fired off the patch before my local test env could complete | 01:12 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for django_compressor 2.0 bug 1532048 https://review.openstack.org/265028 | 01:14 |
| openstack | bug 1532048 in OpenStack Dashboard (Horizon) "grenade fails setting up horizon with "ImportError: No module named utils" for compressor package" [Undecided,Confirmed] https://launchpad.net/bugs/1532048 | 01:14 |
| *** piet has quit IRC | 01:15 | |
| anteaya | seems the -meeting channel needs a topic refresh, not sure why | 01:15 |
| jeblair | fungi, clarkb: are you aware of tox env variable isolation? | 01:16 |
| fungi | jeblair: with the passenv option to list what you want whitelisted? | 01:17 |
| jeblair | yeah | 01:17 |
| fungi | then... yes | 01:17 |
| jeblair | apparently this is a thing as of like half a year ago | 01:17 |
| fungi | 1.7 i think | 01:17 |
| jeblair | 2.0 | 01:17 |
| openstackgerrit | hardik proposed openstack-infra/project-config: Fixed mistral experimental job https://review.openstack.org/265030 | 01:17 |
| jeblair | this will prevent zuul-cloner from working in a tox env | 01:17 |
| jeblair | which happens a bit and is happening more | 01:17 |
| fungi | without whitelisting all the necessary ZUUL_.* envvars anyway | 01:17 |
| *** mtanino has quit IRC | 01:18 | |
| jeblair | right | 01:18 |
| fungi | worth noodling on a cleaner solution to | 01:18 |
| jeblair | i'm saying i guess no one realized this was happening 6 months ago | 01:18 |
| jeblair | this could have a big impact on the neutron stuff | 01:18 |
| Thelo | 2016-01-08 08:59:38,191 INFO nodepool.image.build.dpc: + sudo chown -R jenkins:jenkins /opt/git/openstack/tempest/preseed-streams | 01:18 |
| Thelo | 2016-01-08 08:59:38,367 INFO nodepool.image.build.dpc: sudo: unable to resolve host hps10 | 01:18 |
| Thelo | 2016-01-08 08:59:38,370 INFO nodepool.image.build.dpc: chown: cannot access '/opt/git/openstack/tempest/preseed-streams': No such file or directory | 01:18 |
| Thelo | 2016-01-08 08:59:38,373 INFO nodepool.image.build.dpc: ++ check_break after-error run_in_target bash | 01:18 |
| oomichi | can anyone resubmit a new tempest-lib release? | 01:18 |
| oomichi | The release was failed during uploading to pypi.python.org like http://logs.openstack.org/f0/f0b44f3e4292353b186ee3b32d9b89f9bfe9d894/release/tempest-lib-pypi-both-upload/92ab2a5/console.html | 01:19 |
| Thelo | Does anyone know what is this project for: chown: cannot access '/opt/git/openstack/tempest/preseed-streams? | 01:19 |
| fungi | especially since they tend to want to have fallbacks on missing zuul envvars assuming that they're being invoked on developer machines and not the ci | 01:19 |
| fungi | oomichi: that's an unfortunate pypi failure | 01:20 |
| fungi | oomichi: requeuing that tag's jobs now | 01:21 |
| oomichi | fungi: that means it is ok that I just am waiting for the release? | 01:21 |
| fungi | oomichi: yeah, zuul's got the jobs queued for it now | 01:22 |
| openstackgerrit | Merged openstack/diskimage-builder: Make check for image-should-fail quiet https://review.openstack.org/258762 | 01:22 |
| openstackgerrit | Merged openstack/diskimage-builder: Fix discoverd bug when dmidecode reports GB https://review.openstack.org/258757 | 01:22 |
| mriedem | jlk: i tested mine, it was fine | 01:23 |
| fungi | oomichi: i mean, it required manual intervention. i basically told zuul to pretend it just saw the 0.13.0 tag again | 01:23 |
| mriedem | jlk: i'm watching yours in zuul | 01:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Validate grafyaml based on master https://review.openstack.org/265009 | 01:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: Update OVH graph with axis labels https://review.openstack.org/265010 | 01:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Pass ZUUL_* env vars through tox https://review.openstack.org/265031 | 01:23 |
| fungi | jeblair: oh! passenv will do glob matching? | 01:23 |
| openstackgerrit | Merged openstack/diskimage-builder: Properly account for pipefail during cleanup https://review.openstack.org/262827 | 01:23 |
| oomichi | fungi: thanks for doing this! I see it on zuul :-) | 01:23 |
| jeblair | fungi: yeah... but wait till you see my next idea :) | 01:24 |
| anteaya | fungi: zuul must like the milk and cookies you offered | 01:24 |
| fungi | oomichi: seems it was successful on retry http://logs.openstack.org/f0/f0b44f3e4292353b186ee3b32d9b89f9bfe9d894/release/tempest-lib-pypi-both-upload/c5bdd55/console.html | 01:25 |
| fungi | oomichi: lgtm https://pypi.python.org/pypi/tempest-lib | 01:25 |
| oomichi | fungi: thanks so much, we can see new version on https://pypi.python.org/pypi/tempest-lib now | 01:25 |
| fungi | awesome | 01:25 |
| oomichi | fungi: yeah :) | 01:25 |
| *** Keedya has joined #openstack-infra | 01:26 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/requirements: Bump minimum tempest-lib version https://review.openstack.org/263106 | 01:28 |
| *** tsg_ has quit IRC | 01:28 | |
| anteaya | topic in -meeting still shows open discussion for nova which ended some time ago <insert interpretive dance> | 01:30 |
| zaro | fungi: https://code.google.com/p/gerrit/issues/detail?id=3754 | 01:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Have Zuul force tox to obey Zuul https://review.openstack.org/265032 | 01:30 |
| jeblair | fungi: ^ :) | 01:30 |
| fungi | jeblair: 265009 lgtm but a couple ideas commented inline you could do in a new patchset or follow-on change if you're so inclined | 01:31 |
| zaro | fungi: modify account permission will work, but you probably don't want to set that. | 01:31 |
| anteaya | zaro: yeah, I don't want that | 01:31 |
| jeblair | fungi: good points, thanks | 01:31 |
| fungi | zaro: right, that's not so great. thanks for working on improving it! i'll track that issue in a code comment for now and we can revisit if and when it gets solved | 01:31 |
| *** Keedya_ has joined #openstack-infra | 01:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Ignore E011 in bashate https://review.openstack.org/265027 | 01:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/grafyaml: Add support for template query https://review.openstack.org/250893 | 01:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Revert "Pass ZUUL_* env vars through tox" https://review.openstack.org/265034 | 01:33 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Add a CONTRIBUTING details https://review.openstack.org/265035 | 01:34 |
| asselin | fungi, Thelo as promissed ^^ | 01:34 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Add a CONTRIBUTING.rst document https://review.openstack.org/265035 | 01:35 |
| *** apoorvad has joined #openstack-infra | 01:35 | |
| *** Keedya has quit IRC | 01:36 | |
| *** changbl has joined #openstack-infra | 01:36 | |
| openstackgerrit | Matthew Kassawara proposed openstack-infra/project-config: Add Sam-I-Am to IRC operator list https://review.openstack.org/265036 | 01:38 |
| *** sputnik13_ has quit IRC | 01:38 | |
| *** Keedya_ has quit IRC | 01:38 | |
| anteaya | obviously I should know what the ramifications of tox and zuul not behaving as zuul wants but I don't | 01:38 |
| fungi | jeblair: heh, TOX_TESTENV_PASSENV almost seems like it defeats the point ;) | 01:38 |
| fungi | anteaya: some projects are invoking zuul-cloner in scripts called under tox, and both those scripts and zuul-cloner itself will potentially have different behaviors if they don't see the ZUUL_* envvars which they expect when called by a zuul-based ci system | 01:40 |
| fungi | anteaya: because those developers also want to be able to run tox -e whatever on their systems and get ci-free behaviors | 01:41 |
| anteaya | ah ha | 01:41 |
| anteaya | thank you | 01:41 |
| fungi | anteaya: and they're looking at absence of those envvars as an indicator that they're not running in the ci when called | 01:41 |
| fungi | and tox's default behavior now is to filter any envvars unless you tell it to let specific ones through | 01:42 |
| anteaya | ah | 01:42 |
| anteaya | thank you, that helps | 01:42 |
| anteaya | well helps me understand that is | 01:42 |
| *** Keedya has joined #openstack-infra | 01:43 | |
| fungi | so for example, zuul might be starting a job which calls tox -e functional and then the functional job is cloning the master branch tip instead of a change zuul is suggesting via ZUUL_REF or whatever. end result is that the job isn't testing what people think it's testing and could let bugs slip through | 01:43 |
| anteaya | perfect | 01:44 |
| anteaya | in the sarcastic sense of the word | 01:44 |
| pabelanger | nibalizer: clarkb: Right, so like I said, the only reason we don't have never versions of Fedora ATM, is because each time we attempt a launch, we've exposed another issue / bug. A lot has to do with running DIBs, glean and so on, which sadly fedora's been the first to run into most of the issues. It be cool if we could not do a DIB for fedora 22 and upload that while ian works on Fedora 23 issues. | 01:44 |
| pabelanger | Either way, no -1 from me, just sad we are moving our last fedora now. | 01:44 |
| *** ybathia has quit IRC | 01:44 | |
| *** Keedya has quit IRC | 01:45 | |
| anteaya | so how do I test 265032? | 01:45 |
| fungi | anteaya: you could create a special "foo" tox env which echoes back the value of $ZUUL_FOO and then export ZUUL_FOO=bar in your environment before running tox -e foo | 01:47 |
| *** Swami has quit IRC | 01:47 | |
| anteaya | I'll see how far I get trying that | 01:47 |
| fungi | it won't directly test the parameter function script being modified in 265032 but it'll validate the tox feature that's depending on | 01:48 |
| fungi | what i suggested should fail, but then export TOX_TESTENV_PASSENV=ZUUL_FOO and try again | 01:49 |
| fungi | and hopefully you'll see it say "bar" | 01:49 |
| * anteaya hopes it will say "bar" | 01:50 | |
| *** MarkAtwood1 has quit IRC | 01:51 | |
| pabelanger | timrc: https://review.openstack.org/#/c/250893/ would be a good one to start on :) | 01:52 |
| *** Sukhdev has quit IRC | 01:52 | |
| anteaya | I'm waiting for local tox to finish with 265031 first | 01:52 |
| *** grassass has joined #openstack-infra | 01:54 | |
| *** Sukhdev has joined #openstack-infra | 01:55 | |
| anteaya | no likey: http://paste.openstack.org/show/483290/ | 01:56 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics https://review.openstack.org/263971 | 01:56 |
| *** apoorvad has quit IRC | 01:57 | |
| anteaya | I had just run `tox` no arguments | 01:57 |
| *** rajinir has quit IRC | 02:05 | |
| Sam-I-Am | anteaya: thx for the +2 | 02:05 |
| anteaya | Sam-I-Am: welcome :) happy to see you volunteer to take on the work | 02:06 |
| anteaya | thank you | 02:06 |
| Sam-I-Am | np | 02:06 |
| *** sridhar_ram has quit IRC | 02:06 | |
| fungi | anteaya: http://paste.openstack.org/show/483291/ | 02:07 |
| fungi | anteaya: note that the first tox invocation doesn't output bar, bur the second does | 02:07 |
| fungi | er, but the second does | 02:08 |
| anteaya | fungi: thank you I will try that | 02:09 |
| fungi | the bash -c is so that i could get a subprocess operating on that environment variable, and the deps = was just to make it go faster (no need to install requirements) | 02:09 |
| anteaya | ah ye requirements take a while | 02:10 |
| anteaya | my system didn't fancy 265031 | 02:10 |
| *** bpokorny has quit IRC | 02:12 | |
| *** Sukhdev has quit IRC | 02:13 | |
| anteaya | I don't seem to be able to get it to work: http://paste.openstack.org/show/483292/ | 02:15 |
| *** zhurong has joined #openstack-infra | 02:15 | |
| anteaya | I also have ZUUL_FOO=bar and TOX_TESTENV_PASSENV=ZUUL_FOO in my environment variables | 02:18 |
| anteaya | not that I care just saying they are there | 02:18 |
| *** bharathm has quit IRC | 02:19 | |
| fungi | anteaya: the \ gets interpreted out in my >> tox.ini | 02:20 |
| fungi | if you manually edited that into your tox.ini drop the \ | 02:20 |
| anteaya | yeah that was it | 02:21 |
| anteaya | though now since i have TOX_TESTENV_PASSENV as an env var it just prints bar both times | 02:21 |
| anteaya | but thanks okay that one works | 02:21 |
| anteaya | why is my system so unhappy with 031 I wonder? | 02:21 |
| fungi | yeah, if you were exporting them in your env you can `unset TOX_TESTENV_PASSENV ; export TOX_TESTENV_PASSENV` to get it good and gone again | 02:22 |
| anteaya | thanks | 02:22 |
| *** jerryz has quit IRC | 02:27 | |
| anteaya | I get this just before the xml tests finish: You have a macro ('tox') defined for 'builder' component type that is masking an inbuilt definition | 02:27 |
| *** apoorvad has joined #openstack-infra | 02:30 | |
| anteaya | okay my system is failing to build Pillow apparently | 02:30 |
| anteaya | http://paste.openstack.org/show/483294/ | 02:30 |
| *** otter768 has joined #openstack-infra | 02:31 | |
| *** thorst has quit IRC | 02:32 | |
| clarkb | anteaya: you need libjpeg-dev | 02:32 |
| anteaya | okay | 02:32 |
| anteaya | should that be downloaded for me, or this is expected I would have that? | 02:32 |
| * anteaya downloads libjpeg-dev | 02:33 | |
| *** thorst has joined #openstack-infra | 02:33 | |
| *** ayoung has joined #openstack-infra | 02:35 | |
| *** jaosorior has quit IRC | 02:35 | |
| *** jaosorior has joined #openstack-infra | 02:35 | |
| *** dims has quit IRC | 02:36 | |
| *** uberjay has quit IRC | 02:39 | |
| *** sdague has quit IRC | 02:39 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:39 | |
| *** uberjay has joined #openstack-infra | 02:40 | |
| *** sdague has joined #openstack-infra | 02:40 | |
| *** thorst has quit IRC | 02:41 | |
| anteaya | thanks clarkb that was it | 02:41 |
| *** piet has joined #openstack-infra | 02:42 | |
| anteaya | jeblair: I approved 265032, I hope that was okay to merge now | 02:45 |
| *** tqtran has quit IRC | 02:47 | |
| *** Qiming has quit IRC | 02:49 | |
| openstackgerrit | Merged openstack-infra/project-config: Have Zuul force tox to obey Zuul https://review.openstack.org/265032 | 02:50 |
| *** Keedya has joined #openstack-infra | 02:53 | |
| openstackgerrit | Merged openstack-infra/puppet-ansible: Remove an ensure absent https://review.openstack.org/250927 | 02:56 |
| *** piet has quit IRC | 02:59 | |
| anteaya | I think that will get picked up by the next successful image build run, yeah? | 03:01 |
| *** Keedya has quit IRC | 03:01 | |
| *** ybathia has joined #openstack-infra | 03:02 | |
| *** ybathia_ has joined #openstack-infra | 03:03 | |
| *** yamahata has quit IRC | 03:04 | |
| *** Qiming has joined #openstack-infra | 03:04 | |
| *** unicell has quit IRC | 03:04 | |
| *** gyee has quit IRC | 03:04 | |
| *** ybathia has quit IRC | 03:06 | |
| *** ybathia_ is now known as ybathia | 03:06 | |
| *** eil397 has joined #openstack-infra | 03:09 | |
| *** coolsvap|away is now known as coolsvap | 03:15 | |
| *** ybathia has quit IRC | 03:20 | |
| *** apoorvad has quit IRC | 03:23 | |
| *** yuanying has quit IRC | 03:26 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add explict start dates to meeting yaml https://review.openstack.org/265056 | 03:34 |
| *** piet has joined #openstack-infra | 03:35 | |
| *** yuanying has joined #openstack-infra | 03:37 | |
| *** gongysh has quit IRC | 03:39 | |
| *** thorst has joined #openstack-infra | 03:39 | |
| *** SumitNaiksatam has quit IRC | 03:46 | |
| *** thorst has quit IRC | 03:47 | |
| *** piet has quit IRC | 03:49 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: create GRE bridge interfaces only once https://review.openstack.org/239363 | 03:49 |
| *** piet has joined #openstack-infra | 03:50 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Initial add of gentoo support for diskimage-builder https://review.openstack.org/263349 | 03:56 |
| bkero | Hehehehe | 03:57 |
| *** rguillebert has quit IRC | 03:57 | |
| *** Keedya has joined #openstack-infra | 04:02 | |
| *** edmondsw has quit IRC | 04:05 | |
| *** yuanying has quit IRC | 04:05 | |
| *** Keedya has quit IRC | 04:05 | |
| *** yuanying has joined #openstack-infra | 04:07 | |
| *** salv-orl_ has joined #openstack-infra | 04:09 | |
| *** salv-orlando has quit IRC | 04:12 | |
| *** piet has quit IRC | 04:12 | |
| *** piet has joined #openstack-infra | 04:13 | |
| *** eil397 has quit IRC | 04:15 | |
| jeblair | anteaya: yep, and appears to be working with no ill effect: https://jenkins02.openstack.org/job/gate-keystone-pep8/1194/parameters/ | 04:18 |
| *** yamamoto_ has joined #openstack-infra | 04:19 | |
| *** mriedem has quit IRC | 04:20 | |
| *** gongysh has joined #openstack-infra | 04:21 | |
| *** Ramanjaneya has joined #openstack-infra | 04:23 | |
| *** links has joined #openstack-infra | 04:27 | |
| *** yamamot__ has joined #openstack-infra | 04:28 | |
| *** yamamoto_ has quit IRC | 04:29 | |
| *** salv-orl_ has quit IRC | 04:30 | |
| *** salv-orlando has joined #openstack-infra | 04:30 | |
| *** unicell has joined #openstack-infra | 04:37 | |
| *** markvoelker has quit IRC | 04:38 | |
| *** flwang has quit IRC | 04:42 | |
| *** treaki_ has joined #openstack-infra | 04:43 | |
| *** thorst has joined #openstack-infra | 04:44 | |
| *** flwang1 has joined #openstack-infra | 04:46 | |
| *** treaki__ has quit IRC | 04:47 | |
| *** thorst has quit IRC | 04:51 | |
| *** otter768 has quit IRC | 04:54 | |
| *** MarkAtwood1 has joined #openstack-infra | 04:57 | |
| *** MarkAtwood1 has quit IRC | 04:59 | |
| *** MarkAtwood has joined #openstack-infra | 05:00 | |
| *** MarkAtwood has quit IRC | 05:01 | |
| *** tlian2 has quit IRC | 05:04 | |
| *** Ramanjaneya has quit IRC | 05:05 | |
| *** yamamoto_ has joined #openstack-infra | 05:07 | |
| *** yamamot__ has quit IRC | 05:07 | |
| *** salv-orlando has quit IRC | 05:12 | |
| *** salv-orlando has joined #openstack-infra | 05:12 | |
| *** harlowja_at_home has joined #openstack-infra | 05:18 | |
| *** piet has quit IRC | 05:19 | |
| *** yamamoto_ has quit IRC | 05:19 | |
| *** bharathm has joined #openstack-infra | 05:20 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Initial add of gentoo support for diskimage-builder https://review.openstack.org/263349 | 05:22 |
| *** Sukhdev has joined #openstack-infra | 05:24 | |
| *** bharathm has quit IRC | 05:25 | |
| *** amotoki has joined #openstack-infra | 05:27 | |
| *** jamielennox is now known as jamielennox|away | 05:27 | |
| *** esker has joined #openstack-infra | 05:31 | |
| *** fawadkhaliq has joined #openstack-infra | 05:32 | |
| *** jaosorior has quit IRC | 05:47 | |
| *** thorst has joined #openstack-infra | 05:49 | |
| *** harlowja_at_home has quit IRC | 05:53 | |
| *** mrmartin has joined #openstack-infra | 05:55 | |
| *** thorst has quit IRC | 05:56 | |
| *** aeng has quit IRC | 05:58 | |
| *** yuanying_ has joined #openstack-infra | 06:00 | |
| *** yuanying has quit IRC | 06:01 | |
| *** yuanying_ has quit IRC | 06:01 | |
| *** yuanying has joined #openstack-infra | 06:01 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix cleanup of check-osc-plugins https://review.openstack.org/265085 | 06:02 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Add dib deletion test https://review.openstack.org/215275 | 06:03 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image delete using gearman https://review.openstack.org/198114 | 06:03 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image uploads using gearman https://review.openstack.org/197722 | 06:03 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Builders can be run on their own https://review.openstack.org/232251 | 06:03 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Decouple builders from nodepool instance https://review.openstack.org/232135 | 06:03 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Builders distinguish between failure and exception https://review.openstack.org/242131 | 06:03 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Make secure.conf optional and backwards-compat https://review.openstack.org/232800 | 06:03 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Move nodepool config loading into module https://review.openstack.org/231592 | 06:03 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image builds using gearman https://review.openstack.org/195175 | 06:03 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Decouple nodepool db and config from builders https://review.openstack.org/222398 | 06:03 |
| greghaynes | clarkb: ^ ok, hopefully that worked | 06:03 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix cleanup of check-osc-plugins https://review.openstack.org/265085 | 06:03 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 06:03 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 06:05 | |
| *** AJaeger has joined #openstack-infra | 06:07 | |
| *** mrmartin has quit IRC | 06:08 | |
| *** jbryce has quit IRC | 06:08 | |
| *** notmyname has quit IRC | 06:09 | |
| *** rainya_ has quit IRC | 06:09 | |
| *** mgagne has quit IRC | 06:09 | |
| *** andreaf has quit IRC | 06:09 | |
| *** notmyname has joined #openstack-infra | 06:10 | |
| openstackgerrit | Dimitry Ushakov proposed openstack-infra/project-config: Add solum-pythonclient to projects https://review.openstack.org/265088 | 06:11 |
| *** dtroyer has quit IRC | 06:11 | |
| *** esker has quit IRC | 06:13 | |
| *** vgridnev has joined #openstack-infra | 06:13 | |
| *** markvoelker has joined #openstack-infra | 06:13 | |
| *** gongysh has quit IRC | 06:14 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/265091 | 06:15 |
| *** mgagne has joined #openstack-infra | 06:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Do not run openstack-api-site https://review.openstack.org/265093 | 06:16 |
| *** mgagne is now known as Guest41804 | 06:16 | |
| *** rainya has joined #openstack-infra | 06:16 | |
| AJaeger | infra-root, could I get a review of 265093, please? We broke the index page of developer.openstack.org and that change is part of fixing it... | 06:16 |
| *** jbryce has joined #openstack-infra | 06:17 | |
| *** andreaf has joined #openstack-infra | 06:17 | |
| *** dtroyer has joined #openstack-infra | 06:17 | |
| openstackgerrit | Merged openstack/requirements: Bump upper constraint on python-novaclient to 3.2.0 https://review.openstack.org/264855 | 06:17 |
| greghaynes | I think nodepool is totally broken in ci - getting access denied to database. Locally I don't get this... | 06:18 |
| greghaynes | :( | 06:18 |
| greghaynes | so probably not doing builder deploy tomorrow | 06:18 |
| * greghaynes -> bed | 06:18 | |
| *** markvoelker has quit IRC | 06:18 | |
| *** yamamoto_ has joined #openstack-infra | 06:20 | |
| *** vgridnev has quit IRC | 06:20 | |
| *** jaosorior has joined #openstack-infra | 06:21 | |
| Thelo | Hi all, does anyone know what password nodepool is talking about ? (2016-01-08 13:55:21,221 INFO nodepool.utils: Password auth exception. Try number 222...) | 06:22 |
| greghaynes | Thelo: ssh'ing in to the node | 06:24 |
| fungi | AJaeger: that doesn't need infra-root. any project-config reviewers can also approve | 06:27 |
| AJaeger | fungi: thanks, yes - but I feared none of them was around ;/ | 06:28 |
| *** dimtruck is now known as zz_dimtruck | 06:28 | |
| AJaeger | and we don't have yet project-config-cores ;) | 06:28 |
| AJaeger | fungi: sorry, didn't want to wake you up | 06:28 |
| *** Swami has joined #openstack-infra | 06:29 | |
| *** Swami has quit IRC | 06:29 | |
| *** hdd has quit IRC | 06:31 | |
| *** yamamoto_ has quit IRC | 06:31 | |
| *** hdd has joined #openstack-infra | 06:33 | |
| *** jcoufal has joined #openstack-infra | 06:34 | |
| *** darvon has quit IRC | 06:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Do not run openstack-api-site https://review.openstack.org/265093 | 06:38 |
| *** darvon has joined #openstack-infra | 06:40 | |
| *** pahuang has quit IRC | 06:42 | |
| *** pahuang has joined #openstack-infra | 06:42 | |
| *** rraja_ has quit IRC | 06:48 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/263592 | 06:48 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: move osc-plugin-job from experimental to check https://review.openstack.org/265107 | 06:50 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/265091 | 06:50 |
| *** coolsvap is now known as coolsvap|away | 06:52 | |
| Thelo | greghaynes: thanks | 06:52 |
| *** Sukhdev has quit IRC | 06:54 | |
| *** thorst has joined #openstack-infra | 06:54 | |
| *** otter768 has joined #openstack-infra | 06:55 | |
| *** ybathia has joined #openstack-infra | 06:59 | |
| *** pahuang has quit IRC | 07:00 | |
| *** otter768 has quit IRC | 07:00 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Set BRANCH for periodic unittest bitrot jobs https://review.openstack.org/245525 | 07:00 |
| *** pahuang has joined #openstack-infra | 07:01 | |
| *** thorst has quit IRC | 07:02 | |
| *** abregman|afk has quit IRC | 07:08 | |
| *** coolsvap|away is now known as coolsvap | 07:13 | |
| *** camunoz has quit IRC | 07:15 | |
| *** ozamiatin has joined #openstack-infra | 07:15 | |
| *** _nadya_ has joined #openstack-infra | 07:18 | |
| openstackgerrit | Merged openstack/requirements: Updated gabbi to 1.11.0 https://review.openstack.org/255598 | 07:23 |
| *** unicell has quit IRC | 07:24 | |
| *** unicell has joined #openstack-infra | 07:24 | |
| *** flepied1 has quit IRC | 07:27 | |
| *** ParsectiX has joined #openstack-infra | 07:30 | |
| *** ybathia has quit IRC | 07:39 | |
| *** mpavone has joined #openstack-infra | 07:40 | |
| openstackgerrit | Merged openstack-infra/project-config: move osc-plugin-job from experimental to check https://review.openstack.org/265107 | 07:40 |
| *** mpavone has quit IRC | 07:40 | |
| *** _nadya_ has quit IRC | 07:43 | |
| *** sam_wan has joined #openstack-infra | 07:44 | |
| *** ihrachys has joined #openstack-infra | 07:45 | |
| *** yamamoto_ has joined #openstack-infra | 07:45 | |
| *** amotoki has quit IRC | 07:46 | |
| *** amotoki has joined #openstack-infra | 07:46 | |
| yolanda | morning | 07:47 |
| *** ihrachys has quit IRC | 07:48 | |
| *** ihrachys has joined #openstack-infra | 07:48 | |
| *** mrmartin has joined #openstack-infra | 07:52 | |
| *** hashar has joined #openstack-infra | 07:53 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Initial sphynx doc setup for publishing https://review.openstack.org/259245 | 07:53 |
| AJaeger | morning, yolanda ! | 07:54 |
| yolanda | back to life after a long holiday :) | 07:54 |
| AJaeger | yolanda: could you review https://review.openstack.org/#/c/261857/ , please? I'll babysit it ... | 07:54 |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add command deleting *.pyc before executing ostestr https://review.openstack.org/265125 | 07:54 |
| yolanda | sure | 07:54 |
| AJaeger | yolanda: great to have you back - and hope you really relaxed | 07:54 |
| AJaeger | yolanda: you were far too often here ;) | 07:55 |
| *** ozamiatin has quit IRC | 07:55 | |
| yolanda | too much time without a computer near, i miss it :) | 07:55 |
| openstackgerrit | Merged openstack-infra/puppet-storyboard: Add config template for the email plugin https://review.openstack.org/263304 | 07:55 |
| AJaeger | ;) | 07:56 |
| *** ihrachys_ has joined #openstack-infra | 07:56 | |
| *** oomichi has quit IRC | 07:56 | |
| AJaeger | yolanda: could you add Anne Gentle to fairy-slipper-core, please? see https://review.openstack.org/#/c/245352/ - https://review.openstack.org/#/admin/groups/1240,members and it's annegentle@justwriteclick.com | 07:57 |
| *** ihrachys has quit IRC | 07:57 | |
| yolanda | ok will do | 07:57 |
| AJaeger | thanks | 07:58 |
| yolanda | AJaeger, are you familiar with that translation scripts? i have a question | 07:59 |
| AJaeger | yolanda: yes, I'm familiar | 07:59 |
| AJaeger | but have a call now - might answer in 30 mins... | 07:59 |
| *** salv-orlando has quit IRC | 08:00 | |
| yolanda | will add the question to the review | 08:00 |
| *** thorst has joined #openstack-infra | 08:00 | |
| *** salv-orlando has joined #openstack-infra | 08:00 | |
| *** coolsvap is now known as coolsvap|away | 08:00 | |
| AJaeger | ok | 08:00 |
| *** otter768 has joined #openstack-infra | 08:00 | |
| yolanda | added, let's talk later | 08:00 |
| *** otter768 has quit IRC | 08:02 | |
| *** otter768_ has joined #openstack-infra | 08:02 | |
| *** otter768 has joined #openstack-infra | 08:04 | |
| *** fawadkhaliq has quit IRC | 08:04 | |
| *** coolsvap|away is now known as coolsvap | 08:05 | |
| *** otter76__ has joined #openstack-infra | 08:06 | |
| *** samuelBartel has joined #openstack-infra | 08:06 | |
| *** thorst has quit IRC | 08:06 | |
| *** otter768_ has quit IRC | 08:06 | |
| *** hdd has quit IRC | 08:07 | |
| *** otter768_ has joined #openstack-infra | 08:08 | |
| *** otter76__ has quit IRC | 08:08 | |
| *** ihrachys_ has quit IRC | 08:08 | |
| *** [patrol] has left #openstack-infra | 08:08 | |
| *** otter768 has quit IRC | 08:09 | |
| *** cschwede has joined #openstack-infra | 08:10 | |
| *** otter768 has joined #openstack-infra | 08:10 | |
| *** otter768_ has quit IRC | 08:10 | |
| *** otter768_ has joined #openstack-infra | 08:12 | |
| *** otter768 has quit IRC | 08:12 | |
| AJaeger | yolanda: I'm back again and commented. | 08:13 |
| AJaeger | If there's a better way to do this, I'm all ears... | 08:13 |
| AJaeger | Or should I just add a comment that this is set as side effect? | 08:13 |
| *** otter768 has joined #openstack-infra | 08:13 | |
| *** otter768_ has quit IRC | 08:14 | |
| *** bexelbie has quit IRC | 08:14 | |
| *** bexelbie has joined #openstack-infra | 08:14 | |
| yolanda | AJaeger, what's the effect of not having a CHANGE_ID= because in the else, we don't populate one | 08:14 |
| *** oomichi has joined #openstack-infra | 08:14 | |
| yolanda | is that ok to ignore if not change id? | 08:14 |
| *** markvoelker has joined #openstack-infra | 08:15 | |
| *** yamahata has joined #openstack-infra | 08:15 | |
| AJaeger | yolanda: we handle today already the case where it's not set, so this is fine | 08:15 |
| *** otter768 has quit IRC | 08:15 | |
| *** otter768_ has joined #openstack-infra | 08:15 | |
| *** yamamoto_ has quit IRC | 08:15 | |
| AJaeger | logic here is: Do something special if there exist already an old review | 08:16 |
| *** jaosorior has quit IRC | 08:16 | |
| *** jtomasek has joined #openstack-infra | 08:16 | |
| *** jaosorior has joined #openstack-infra | 08:16 | |
| yolanda | ok i think it's fine. As the code is now hidden on the function, it's not obvious to see where the value comes from, but don't want to -1 for that | 08:17 |
| yolanda | i'll approve | 08:17 |
| *** otter768 has joined #openstack-infra | 08:17 | |
| *** otter76__ has joined #openstack-infra | 08:19 | |
| *** markvoelker has quit IRC | 08:19 | |
| *** otter768_ has quit IRC | 08:19 | |
| AJaeger | thanks | 08:20 |
| AJaeger | let me update the followup change with a comment ;) | 08:21 |
| *** otter768_ has joined #openstack-infra | 08:21 | |
| *** otter76__ has quit IRC | 08:21 | |
| *** otter768 has quit IRC | 08:22 | |
| yolanda | it's worth noting that we can set CHANGE_ID and COMMIT_MSG on the nested function | 08:23 |
| *** otter768 has joined #openstack-infra | 08:23 | |
| *** otter768_ has quit IRC | 08:23 | |
| *** jistr has joined #openstack-infra | 08:24 | |
| *** otter768 has quit IRC | 08:25 | |
| *** otter768_ has joined #openstack-infra | 08:25 | |
| *** markus_z has joined #openstack-infra | 08:26 | |
| *** yamamoto has joined #openstack-infra | 08:27 | |
| *** otter768 has joined #openstack-infra | 08:27 | |
| *** otter768_ has quit IRC | 08:27 | |
| *** arxcruz has joined #openstack-infra | 08:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Unify usages of checking for existing changes (2/2) https://review.openstack.org/261858 | 08:29 |
| *** otter768_ has joined #openstack-infra | 08:29 | |
| AJaeger | yolanda: I've added comments everywhere, please review ^ | 08:29 |
| *** otter768_ has quit IRC | 08:30 | |
| yolanda | done | 08:30 |
| *** otter768_ has joined #openstack-infra | 08:31 | |
| *** otter768 has quit IRC | 08:32 | |
| *** yamamoto has quit IRC | 08:32 | |
| *** otter768_ has quit IRC | 08:32 | |
| *** otter768 has joined #openstack-infra | 08:33 | |
| *** otter768 has quit IRC | 08:34 | |
| *** otter768 has joined #openstack-infra | 08:34 | |
| *** otter768_ has joined #openstack-infra | 08:36 | |
| AJaeger | thanks, yolanda | 08:38 |
| *** salv-orlando has quit IRC | 08:38 | |
| *** otter76__ has joined #openstack-infra | 08:38 | |
| *** otter768_ has quit IRC | 08:39 | |
| *** yaume has joined #openstack-infra | 08:39 | |
| *** otter768 has quit IRC | 08:39 | |
| *** salv-orlando has joined #openstack-infra | 08:39 | |
| *** yaume is now known as yaume[away] | 08:39 | |
| *** otter76__ has quit IRC | 08:40 | |
| *** markmcd has quit IRC | 08:40 | |
| *** otter768 has joined #openstack-infra | 08:40 | |
| *** markmcd has joined #openstack-infra | 08:42 | |
| *** otter768_ has joined #openstack-infra | 08:42 | |
| *** otter768 has quit IRC | 08:42 | |
| *** yaume[away] is now known as yaume | 08:42 | |
| *** yaume is now known as yaume[away] | 08:43 | |
| *** otter768 has joined #openstack-infra | 08:44 | |
| *** otter768_ has quit IRC | 08:44 | |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Support GitHub PR webhooks https://review.openstack.org/163117 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Make the string representation of change transparent https://review.openstack.org/238948 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Lower the log level in tests https://review.openstack.org/263364 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Merge pull requests from github reporter https://review.openstack.org/243250 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Encapsulate determining the event purpose https://review.openstack.org/247487 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Fix job hierarchy bug. https://review.openstack.org/192457 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: support github pull reqeust labels https://review.openstack.org/247421 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Add 'push' and 'tag' github webhook events. https://review.openstack.org/191207 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Add 'pr-comment' github webhook event https://review.openstack.org/239203 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Support for dependent pipelines with github https://review.openstack.org/247500 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Configurable SSH access to GitHub https://review.openstack.org/239138 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Allow github trigger to match on branches/refs https://review.openstack.org/258448 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Allow using webapp from connections https://review.openstack.org/215642 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Allow list values in template parameters. https://review.openstack.org/191208 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Add basic Github Zuul Reporter. https://review.openstack.org/191312 | 08:44 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Support for github commit status https://review.openstack.org/239303 | 08:44 |
| *** matrohon has joined #openstack-infra | 08:45 | |
| *** otter768 has quit IRC | 08:46 | |
| Thelo | Hi all, I created an image with nodepool, uploaded it to a openstack cloud and now try to run jobs from jenkins however it looks like jenkins cannot ssh the jenkins slave (SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 38 ms) | 08:46 |
| *** otter768 has joined #openstack-infra | 08:46 | |
| Thelo | does anyone know what key nodepool use by default when we follow this procedure (http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md)? | 08:46 |
| *** yaume[away] is now known as yaume | 08:47 | |
| *** yaume is now known as yaume[away] | 08:48 | |
| *** otter768_ has joined #openstack-infra | 08:48 | |
| *** scheuran has joined #openstack-infra | 08:49 | |
| *** yaume[away] is now known as yaume | 08:49 | |
| *** yaume is now known as yaume[away] | 08:50 | |
| *** otter76__ has joined #openstack-infra | 08:50 | |
| *** otter768_ has quit IRC | 08:50 | |
| *** otter768 has quit IRC | 08:51 | |
| *** otter76__ has quit IRC | 08:51 | |
| *** otter768 has joined #openstack-infra | 08:52 | |
| *** otter768_ has joined #openstack-infra | 08:54 | |
| *** yaume[away] is now known as yaume | 08:55 | |
| *** otter76__ has joined #openstack-infra | 08:56 | |
| *** otter768_ has quit IRC | 08:56 | |
| openstackgerrit | Merged openstack-infra/project-config: Unify usages of checking for existing changes (1/2) https://review.openstack.org/261857 | 08:56 |
| *** otter768 has quit IRC | 08:57 | |
| mrmartin | morning | 08:57 |
| *** otter76__ has quit IRC | 08:57 | |
| *** otter768 has joined #openstack-infra | 08:57 | |
| *** shardy has joined #openstack-infra | 08:57 | |
| mrmartin | is it ok that we have a ton of dependency warnings after deploying a clean puppet environment: http://paste.openstack.org/show/483303/ | 08:58 |
| openstackgerrit | Merged openstack-infra/project-config: Test gate-grenade-dsvm-partial-ncpu on requirements repo https://review.openstack.org/264957 | 08:58 |
| *** otter768 has quit IRC | 08:59 | |
| *** jlanoux has joined #openstack-infra | 08:59 | |
| *** otter768 has joined #openstack-infra | 08:59 | |
| *** ifarkas has joined #openstack-infra | 09:00 | |
| *** otter768 has quit IRC | 09:01 | |
| *** otter768 has joined #openstack-infra | 09:02 | |
| *** savihou has joined #openstack-infra | 09:03 | |
| *** otter768 has quit IRC | 09:03 | |
| *** otter768_ has joined #openstack-infra | 09:03 | |
| *** thorst has joined #openstack-infra | 09:04 | |
| *** zeih has joined #openstack-infra | 09:04 | |
| *** otter768_ has quit IRC | 09:05 | |
| *** otter768 has joined #openstack-infra | 09:05 | |
| savihou | Hello Gents, sorry for the noob question; I'm in the process of setting a jenkinks based (with the help of zuul and gearman) CI system. I can get the gerrit upstream however, cannot post a review to ci-sandbox), any chance I'm missing something really stupid here ? I cannot see any trace of failures within zuul logs . your help will be much appreciated. | 09:07 |
| *** otter768_ has joined #openstack-infra | 09:07 | |
| *** otter768 has quit IRC | 09:07 | |
| *** otter768_ has quit IRC | 09:08 | |
| *** otter768 has joined #openstack-infra | 09:09 | |
| *** sfinucan has joined #openstack-infra | 09:09 | |
| Thelo | mrmartin: I had this yesterday too, I reinstall ubuntu 14.04.2 and tried again and that time got no warning | 09:12 |
| *** thorst has quit IRC | 09:12 | |
| *** otter768 has quit IRC | 09:14 | |
| mrmartin | we have those dependency warnings for example on ask-staging site, and I think on all other instances. I'm not sure that it makes any troubles, maybe just a result of not properly defined deps, or leftovers. | 09:14 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add explict start dates to meeting yaml https://review.openstack.org/265056 | 09:14 |
| *** dstanek has quit IRC | 09:16 | |
| *** dstanek has joined #openstack-infra | 09:18 | |
| *** dizquierdo has joined #openstack-infra | 09:18 | |
| *** rossella_s has joined #openstack-infra | 09:19 | |
| *** ihrachys has joined #openstack-infra | 09:20 | |
| *** ihrachys has quit IRC | 09:21 | |
| *** gchamoul is now known as gchamoul_Doc_afk | 09:22 | |
| *** ihrachys has joined #openstack-infra | 09:24 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add min ver to extra libs for graph https://review.openstack.org/265153 | 09:30 |
| *** yaume has quit IRC | 09:35 | |
| *** ihrachys has quit IRC | 09:35 | |
| *** kushal has joined #openstack-infra | 09:36 | |
| *** ihrachys has joined #openstack-infra | 09:38 | |
| *** mpaolino has joined #openstack-infra | 09:38 | |
| *** jordanP has joined #openstack-infra | 09:39 | |
| *** ihrachys_ has joined #openstack-infra | 09:40 | |
| *** dtantsur|afk is now known as dtantsur | 09:40 | |
| *** yamahata has quit IRC | 09:41 | |
| *** ihrachys has quit IRC | 09:43 | |
| *** yaume has joined #openstack-infra | 09:43 | |
| *** bauzas is now known as bauwser | 09:43 | |
| *** jtomasek has quit IRC | 09:45 | |
| *** ihrachys_ has quit IRC | 09:51 | |
| *** hashar has quit IRC | 09:52 | |
| *** hashar has joined #openstack-infra | 09:53 | |
| *** derekh has joined #openstack-infra | 09:53 | |
| *** rguillebert has joined #openstack-infra | 09:53 | |
| *** [1]Thelo has joined #openstack-infra | 09:54 | |
| *** ihrachys has joined #openstack-infra | 09:55 | |
| *** fawadkhaliq has joined #openstack-infra | 09:56 | |
| *** Thelo has quit IRC | 09:56 | |
| *** [1]Thelo is now known as Thelo | 09:56 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/nodepool: tools: shell script to gracefully stop and wait https://review.openstack.org/219442 | 10:02 |
| *** ociuhandu has quit IRC | 10:06 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/nodepool: node deletion delay is now configurable https://review.openstack.org/245220 | 10:07 |
| Thelo | Hi all, I created an image with nodepool, uploaded it to a openstack cloud and now try to run jobs from jenkins however it looks like jenkins cannot ssh the jenkins slave (SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 38 ms) | 10:08 |
| Thelo | does anyone know what key nodepool use by default when we follow this procedure (http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md)? | 10:08 |
| *** salv-orl_ has joined #openstack-infra | 10:09 | |
| *** thorst has joined #openstack-infra | 10:09 | |
| yolanda | hi Thelo, you can configure the key for the image in nodepool.yaml file | 10:09 |
| yolanda | for each image, you can define a private-key setting, that you can point to the ssh file you want to use | 10:10 |
| yolanda | look at http://docs.openstack.org/infra/nodepool/configuration.html | 10:10 |
| jklare | AJaeger hi and a happy new year | 10:11 |
| AJaeger | happy new year, jklare ! | 10:11 |
| Thelo | yolanda: Hi, I did that, so I guess the node might not have the public key | 10:12 |
| *** salv-orlando has quit IRC | 10:12 | |
| Thelo | yolanda: when is this one passed to the node | 10:12 |
| jklare | AJaeger i have a question regarding the process of creating eol tags before removing old branches | 10:12 |
| yolanda | Thelo, have you also tried manually? from nodepool node, try ssh using that key as identity, with jenkins user. So you can check if the key is correct | 10:12 |
| *** ihrachys has quit IRC | 10:13 | |
| AJaeger | jklare: best talk with fungi later on. | 10:13 |
| jklare | AJaeger i read the wiki about the release process and how to push tags during this one, do you know if its similar for eol tags? | 10:13 |
| jklare | AJaeger ah ok | 10:13 |
| AJaeger | jklare: fungi will create a eol tag as part of removing the branch, so nothing for you to do | 10:13 |
| jklare | AJaeger thats great, less work for me then :) | 10:14 |
| Thelo | yolanda: it works fine, so why jenkins cannot do it? | 10:14 |
| jklare | AJaeger thanks | 10:14 |
| *** tzn has joined #openstack-infra | 10:14 | |
| *** dizquierdo has quit IRC | 10:14 | |
| yolanda | Thelo, one step is nodepool connecting by ssh to the vm | 10:15 |
| *** markvoelker has joined #openstack-infra | 10:15 | |
| yolanda | then, nodepool runs a create node script, that may add the keys for your jenkins masters and configure properly | 10:15 |
| yolanda | so i guess you have some problem in that creation scirpt | 10:15 |
| *** thorst has quit IRC | 10:17 | |
| Thelo | yolanda: Shouldn't it be the opposite? If I can log in manually using this key, it could means that jenkins is not using the right key no? | 10:17 |
| yolanda | looks like that is the problem, yes | 10:18 |
| yolanda | you should check the scripts you use to prepare your image, and ensure that you are granting permissions to jenkins user with the right keys | 10:19 |
| *** markvoelker has quit IRC | 10:20 | |
| Thelo | yolanda: I ssh-ed manually using the user jenkins and the key so my guess is that the image should be working, do you know where I can see which user/key jenkins is using to ssh the node? | 10:20 |
| *** mpaolino has quit IRC | 10:23 | |
| yolanda | Thelo, if you use puppet-jenkins, it relies on jenkins::user class. This needs to be present on the slaves | 10:23 |
| yolanda | that key is normally managed by hiera | 10:24 |
| Thelo | yolanda: You are very right I check the key that jenkins used and that one is not correct, my bad | 10:24 |
| *** sam_wan has quit IRC | 10:25 | |
| yolanda | glad to help | 10:25 |
| Thelo | yolanda: You are very right I check the key that jenkins used and found that this "file" does not exits | 10:26 |
| Thelo | yolanda: however, puppet created the user jenkins but I cannot find any /home/jenkins directory | 10:28 |
| yolanda | Thelo, check in /var/lib/jenkins/ | 10:30 |
| *** zhurong has quit IRC | 10:30 | |
| *** zhurong has joined #openstack-infra | 10:31 | |
| Thelo | yolanda: you mean the .ssh folder? | 10:33 |
| Thelo | that one contains the right key apparently | 10:33 |
| Thelo | yolanda: however through jenkins ui I can seen that for this node it is not taking this key | 10:34 |
| yolanda | have you checked that step? http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md#n347 | 10:35 |
| openstackgerrit | Thierry Carrez proposed openstack/requirements: Add missing license information https://review.openstack.org/264754 | 10:35 |
| *** zhurong has quit IRC | 10:37 | |
| openstackgerrit | Thierry Carrez proposed openstack/requirements: Add missing license information https://review.openstack.org/264754 | 10:37 |
| *** jaosorior has quit IRC | 10:37 | |
| *** jaosorior has joined #openstack-infra | 10:38 | |
| Thelo | yolanda: No sorry, I skipped the optional | 10:38 |
| Thelo | I will do it | 10:38 |
| yolanda | that will help with your creds problem | 10:39 |
| *** jlanoux has quit IRC | 10:40 | |
| openstackgerrit | Christoph Albers proposed openstack-infra/project-config: Removed testjobs and corresponding chef-jobs https://review.openstack.org/264130 | 10:40 |
| *** electrofelix has joined #openstack-infra | 10:41 | |
| *** ihrachys has joined #openstack-infra | 10:42 | |
| openstackgerrit | Merged openstack/requirements: Cap django_compressor below 2.0 https://review.openstack.org/265025 | 10:50 |
| Thelo | yolanda: Thanks, however I still get JenkinsException: Error in request. Possibly authentication failed [401]: Invalid password/token for user: jenkins | 10:52 |
| Thelo | yolanda: and I just retry to log in using jenkins and this key manually with success | 10:52 |
| yolanda | Thelo, that's a different error than before right? | 10:53 |
| *** ihrachys has quit IRC | 10:54 | |
| Thelo | yes | 10:54 |
| yolanda | check jenkins global security and token, as listed on documentation | 10:55 |
| *** mrmartin has quit IRC | 10:55 | |
| *** Qiming has quit IRC | 10:55 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements https://review.openstack.org/246114 | 10:56 |
| *** gchamoul_Doc_afk is now known as gchamoul | 10:57 | |
| *** daemontool has joined #openstack-infra | 10:58 | |
| Thelo | yolanda : at http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md#n353 if I click on advanced I can see that the passphrase field is not empty, is it normal? | 10:59 |
| yolanda | cannot tell, it may be a jenkins feature to mask password | 11:00 |
| *** fawadkhaliq has quit IRC | 11:05 | |
| *** fawadkhaliq has joined #openstack-infra | 11:05 | |
| Thelo | yolanda: I still get this in nodepool's log (http://paste.openstack.org/show/483308/) | 11:05 |
| Thelo | I double check the configuration and everything seems fine | 11:05 |
| *** pahuang has quit IRC | 11:07 | |
| yolanda | Thelo, that looks as a problem with jenkins api key, check that the value you set for api key and credentials matches the hiera value, and is correctly set on nodepool, on /etc/nodepool/secure.conf | 11:07 |
| Thelo | buthe hiera value is the value define in common.yaml? | 11:08 |
| *** dims has joined #openstack-infra | 11:09 | |
| *** otter768 has joined #openstack-infra | 11:10 | |
| yolanda | yes | 11:10 |
| *** lucas-dinner is now known as lucasagomes | 11:12 | |
| Thelo | yolanda: I don't get it everything seems fine, should I restart nodepool? | 11:13 |
| yolanda | it may pick new config automatically, but is worth trying | 11:13 |
| *** otter768 has quit IRC | 11:14 | |
| *** thorst has joined #openstack-infra | 11:15 | |
| *** piet has joined #openstack-infra | 11:17 | |
| dims | sdague : let's switch on the grenade job for o.vo please? https://review.openstack.org/#/c/264992/ | 11:18 |
| Thelo | yolanda: it stucks at restarting, does it require to connect to jenkins to start? | 11:19 |
| *** jlanoux has joined #openstack-infra | 11:20 | |
| yolanda | better stop and start it | 11:20 |
| Thelo | indeed | 11:21 |
| odyssey4me | hmm, it seems that git archive doesn't work against git.o.o - is that correct? | 11:22 |
| *** thorst has quit IRC | 11:22 | |
| *** ociuhandu has joined #openstack-infra | 11:23 | |
| Thelo | yolanda: now the error log in nodepool didnt show up however I still get the [01/08/16 19:21:59] SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 76 ms in jenkins logs | 11:23 |
| *** fawadkhaliq has quit IRC | 11:25 | |
| *** fawadkhaliq has joined #openstack-infra | 11:25 | |
| *** jlanoux_ has joined #openstack-infra | 11:27 | |
| *** jlanoux has quit IRC | 11:29 | |
| yolanda | if the failure is by ssh, you have a mismatch on the jenkins ssh keys. And if that on jenkins pass, it means you have the wrong jenkins pass and api key | 11:30 |
| *** piet has quit IRC | 11:31 | |
| *** Roamer` has joined #openstack-infra | 11:34 | |
| Roamer` | hi, are people aware of the fact that the xio-ise-iscsi-ci account has gone ballistic on https://review.openstack.org/#/c/264056? | 11:37 |
| *** shardy has quit IRC | 11:37 | |
| *** ozamiatin has joined #openstack-infra | 11:38 | |
| *** dizquierdo has joined #openstack-infra | 11:40 | |
| *** ihrachys has joined #openstack-infra | 11:43 | |
| *** Qiming has joined #openstack-infra | 11:43 | |
| *** rfolco has joined #openstack-infra | 11:45 | |
| *** amotoki_ has joined #openstack-infra | 11:52 | |
| *** ociuhandu_ has joined #openstack-infra | 11:54 | |
| *** amotoki has quit IRC | 11:54 | |
| *** ociuhandu has quit IRC | 11:56 | |
| *** ociuhandu_ is now known as ociuhandu | 11:56 | |
| *** ihrachys_ has joined #openstack-infra | 11:57 | |
| *** ihrachys has quit IRC | 11:57 | |
| *** khappone_ has quit IRC | 12:00 | |
| *** markvoelker has joined #openstack-infra | 12:01 | |
| *** scheuran has quit IRC | 12:02 | |
| *** zeih has quit IRC | 12:05 | |
| *** zeih has joined #openstack-infra | 12:06 | |
| *** markvoelker has quit IRC | 12:06 | |
| *** matrohon has quit IRC | 12:10 | |
| *** ihrachys has joined #openstack-infra | 12:11 | |
| *** ihrachys_ has quit IRC | 12:12 | |
| sdague | dims: lgtm | 12:12 |
| dims | sdague : thanks! | 12:15 |
| *** ihrachys has quit IRC | 12:16 | |
| *** mrmartin has joined #openstack-infra | 12:16 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add openstack/python-ironicclient to PROJECTS https://review.openstack.org/265230 | 12:17 |
| *** sheeprine has quit IRC | 12:17 | |
| *** sheeprine has joined #openstack-infra | 12:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Add a grenade job for testing oslo.versionedobjects https://review.openstack.org/264992 | 12:19 |
| *** rhallisey has joined #openstack-infra | 12:19 | |
| *** thorst has joined #openstack-infra | 12:19 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/puppet-askbot: Fix broken Pillow pip dependency https://review.openstack.org/265231 | 12:22 |
| *** thorst has quit IRC | 12:27 | |
| *** ihrachys has joined #openstack-infra | 12:31 | |
| *** ihrachys has quit IRC | 12:31 | |
| *** coolsvap is now known as coolsvap|away | 12:34 | |
| *** khappone has joined #openstack-infra | 12:39 | |
| *** dims_ has joined #openstack-infra | 12:40 | |
| *** dims has quit IRC | 12:41 | |
| *** weshay_xchat has joined #openstack-infra | 12:41 | |
| *** weshay_xchat is now known as weshay | 12:42 | |
| *** markvoelker has joined #openstack-infra | 12:45 | |
| *** fawadkhaliq has quit IRC | 12:50 | |
| *** flepied has joined #openstack-infra | 12:53 | |
| *** fawadkhaliq has joined #openstack-infra | 12:55 | |
| *** alaski is now known as lascii | 12:59 | |
| *** thorst has joined #openstack-infra | 13:01 | |
| *** hashar has quit IRC | 13:02 | |
| *** dprince has joined #openstack-infra | 13:09 | |
| *** matrohon has joined #openstack-infra | 13:10 | |
| *** otter768 has joined #openstack-infra | 13:11 | |
| jordanP | clarkb, fungi I am looking for another +2 here https://review.openstack.org/#/c/239363/ It's a trivial patch. Could you help me please ? | 13:12 |
| *** esker has joined #openstack-infra | 13:13 | |
| *** tlian has joined #openstack-infra | 13:14 | |
| *** otter768 has quit IRC | 13:16 | |
| *** dims_ has quit IRC | 13:16 | |
| AJaeger | jordanP: that change merged 9 hours ago, didn't it? | 13:18 |
| jordanP | AJaeger, ... wrong link, sorry ! I meant https://review.openstack.org/#/c/250879/ | 13:18 |
| *** esker has quit IRC | 13:20 | |
| anteaya | yolanda fungi we need X-IO OpenStack CI disabled, should be gerrit id 14969 | 13:23 |
| *** matrohon has quit IRC | 13:23 | |
| anteaya | yolanda fungi they are repeatedly commenting on 264056 | 13:23 |
| anteaya | Roamer`: thank you, potentially activity like this can bring down our zuul, should you see it again in future, please make a louder fuss | 13:24 |
| anteaya | Roamer`: thanks for saying something | 13:24 |
| anteaya | infra-root: ^^ | 13:24 |
| *** matrohon has joined #openstack-infra | 13:25 | |
| *** links has quit IRC | 13:25 | |
| anteaya | yolanda fungi I believe the username on that account is xio-ise-iscsi-c | 13:26 |
| *** pradk has joined #openstack-infra | 13:26 | |
| *** pradk has quit IRC | 13:26 | |
| anteaya | xio-ise-iscsi-ci | 13:26 |
| *** dims has joined #openstack-infra | 13:26 | |
| anteaya | sorry it is our gerrit that shows the effects, not zuul | 13:27 |
| *** Guest76580 has quit IRC | 13:27 | |
| anteaya | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=34&rra_id=all | 13:27 |
| *** dingyichen has quit IRC | 13:29 | |
| *** edmondsw has joined #openstack-infra | 13:35 | |
| anteaya | I have to leave soon to run an errand but I think this is all the info any infra-root requires to disable 14969 to stop the commenting loop on 264056 | 13:39 |
| anteaya | I'll be back in a few hours | 13:40 |
| * anteaya tosses the message in a bottle into the ocean and hopes someone sees it soon | 13:40 | |
| *** claudiub has joined #openstack-infra | 13:40 | |
| Roamer` | anteaya, no problem, it actually brought down my (StorPool) Zuul, that's how I found out about it :) | 13:43 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename .log to .log.txt in run-tox https://review.openstack.org/203099 | 13:46 |
| *** sileht has quit IRC | 13:46 | |
| AJaeger | fungi, this is an old patch of you that I updated - could you review, please? ^ | 13:47 |
| *** rcarrillocruz has quit IRC | 13:47 | |
| *** sileht has joined #openstack-infra | 13:48 | |
| *** dkranz has joined #openstack-infra | 13:48 | |
| *** rcarrillocruz has joined #openstack-infra | 13:49 | |
| *** ujuc has joined #openstack-infra | 13:49 | |
| anteaya | Roamer`: :( | 13:49 |
| *** gordc has joined #openstack-infra | 13:50 | |
| anteaya | AJaeger: if you are online when yolanda or fungi appears can you point them at the request to disable 14969 please as a priority? | 13:51 |
| *** e0ne has joined #openstack-infra | 13:52 | |
| *** Hal has joined #openstack-infra | 13:52 | |
| *** Hal is now known as Guest28831 | 13:53 | |
| AJaeger | anteaya: yes, will do. | 13:53 |
| anteaya | AJaeger: thank you | 13:54 |
| anteaya | rhedlind is the operators irc nick but he isn't online right now | 13:55 |
| anteaya | operator's* | 13:55 |
| AJaeger | anteaya: thanks for gathering all information! | 13:55 |
| anteaya | AJaeger: thanks :) | 13:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Make gate-rally-dsvm-cinder job voting for Cinder patches https://review.openstack.org/203680 | 13:59 |
| openstackgerrit | Merged openstack-infra/release-tools: use send-announcements-to metadata for deliverables https://review.openstack.org/262281 | 14:00 |
| openstackgerrit | Merged openstack-infra/release-tools: use default from address for emails https://review.openstack.org/263912 | 14:00 |
| openstackgerrit | Merged openstack-infra/release-tools: add command to actually send email announcements https://review.openstack.org/263913 | 14:00 |
| *** bcourt has quit IRC | 14:02 | |
| *** jaosorior has quit IRC | 14:02 | |
| *** annegentle has quit IRC | 14:02 | |
| *** annegentle has joined #openstack-infra | 14:02 | |
| *** dtantsur is now known as dtantsur|brb | 14:02 | |
| anteaya | will check back when I return | 14:03 |
| *** baoli has joined #openstack-infra | 14:03 | |
| *** baoli has quit IRC | 14:05 | |
| *** baoli has joined #openstack-infra | 14:05 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Rudimentary Templates for Tasks https://review.openstack.org/264255 | 14:07 |
| *** baoli_ has joined #openstack-infra | 14:07 | |
| *** salv-orl_ has quit IRC | 14:08 | |
| *** salv-orlando has joined #openstack-infra | 14:08 | |
| *** rlandy has joined #openstack-infra | 14:09 | |
| *** baoli has quit IRC | 14:10 | |
| *** jordanP is now known as jordanP|away | 14:14 | |
| *** lucasagomes is now known as lucas-hungry | 14:14 | |
| *** esker has joined #openstack-infra | 14:17 | |
| *** bcourt has joined #openstack-infra | 14:17 | |
| *** hdd has joined #openstack-infra | 14:21 | |
| *** amitgandhinz has joined #openstack-infra | 14:21 | |
| *** esker has quit IRC | 14:22 | |
| *** rossella_s has quit IRC | 14:22 | |
| *** rossella_s has joined #openstack-infra | 14:22 | |
| *** esker has joined #openstack-infra | 14:22 | |
| *** peristeri has joined #openstack-infra | 14:24 | |
| AJaeger | dougwig, could you review https://review.openstack.org/#/c/258992/ , please? | 14:28 |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Add python-ironicclient to PROJECTS https://review.openstack.org/263335 | 14:29 |
| *** jsavak has joined #openstack-infra | 14:30 | |
| *** shardy has joined #openstack-infra | 14:31 | |
| *** ari has joined #openstack-infra | 14:31 | |
| openstackgerrit | Dirk Mueller proposed openstack/requirements: Use OpenStack Foundation for setup.py copyright https://review.openstack.org/264949 | 14:33 |
| *** dtantsur|brb is now known as dtantsur | 14:34 | |
| *** bcourt has quit IRC | 14:34 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config: Mark swarm test as non-voting in Magnum https://review.openstack.org/264998 | 14:34 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config: [WIP] Revert "Mark swarm test as non-voting in Magnum" https://review.openstack.org/265273 | 14:34 |
| *** kgiusti has joined #openstack-infra | 14:35 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config: Add prettytable into openstack https://review.openstack.org/265278 | 14:38 |
| *** dkehn has quit IRC | 14:40 | |
| *** dkehn has joined #openstack-infra | 14:41 | |
| *** mriedem has joined #openstack-infra | 14:42 | |
| *** ujuc has quit IRC | 14:43 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/grafyaml: Add support for template query https://review.openstack.org/250893 | 14:45 |
| mriedem | is the reqs proposal bot broken? | 14:45 |
| pabelanger | morning | 14:46 |
| AJaeger | mriedem: https://jenkins.openstack.org/job/propose-requirements-updates/ | 14:46 |
| AJaeger | stable/kilo failed, rest looks fine... | 14:46 |
| AJaeger | morning, pabelanger | 14:46 |
| *** bcourt has joined #openstack-infra | 14:46 | |
| AJaeger | mriedem: why do you ask? | 14:47 |
| mriedem | i don't see a sync to horizon for https://review.openstack.org/#/c/265025/ | 14:47 |
| *** nelsnelson has quit IRC | 14:47 | |
| *** esker has quit IRC | 14:48 | |
| *** jaypipes has joined #openstack-infra | 14:49 | |
| mriedem | and that's kilo | 14:49 |
| mriedem | so that's probably why | 14:49 |
| AJaeger | yep | 14:49 |
| mriedem | i'll sync horizon on kilo manually | 14:49 |
| *** ParsectiX has quit IRC | 14:49 | |
| pabelanger | yolanda: mind checking out 262017 and 262019? | 14:50 |
| *** pradk has joined #openstack-infra | 14:50 | |
| AJaeger | mriedem: https://review.openstack.org/#/c/265179/ | 14:50 |
| *** jordanP|away is now known as jordanP | 14:50 | |
| *** doug-fish has joined #openstack-infra | 14:50 | |
| AJaeger | mriedem: and the kilo job has run in general - just failed on at least one repository | 14:51 |
| mriedem | AJaeger: ah, didn't see it on the dashboard i have since jenkins was -1 | 14:51 |
| *** amotoki_ has quit IRC | 14:51 | |
| AJaeger | horizon fails with testrepository problem ;( | 14:51 |
| mriedem | should be able to just backport a fix i'd think, i'm looking | 14:52 |
| *** esker has joined #openstack-infra | 14:54 | |
| *** fawadkhaliq has quit IRC | 14:54 | |
| *** fawadkhaliq has joined #openstack-infra | 14:54 | |
| *** fawadkhaliq has quit IRC | 14:55 | |
| *** ari is now known as Ari | 14:55 | |
| *** Ari is now known as alivigni | 14:55 | |
| openstackgerrit | Vincent Françoise proposed openstack-infra/project-config: Enable Tempest tests for Watcher https://review.openstack.org/265291 | 14:57 |
| yolanda | anteaya, i was having lunch and a short break | 14:58 |
| *** esker has quit IRC | 14:58 | |
| yolanda | i never disabled an account so i need to read about it | 14:58 |
| Sam-I-Am | anteaya: you sleep less than i do :) | 14:59 |
| AJaeger | yolanda: anteaya is out now and I don't know what to do either ;( | 14:59 |
| *** ganso has quit IRC | 14:59 | |
| yolanda | i can check for the account in gerrit and check permissions but i'm not sure if that's the right approach | 15:00 |
| AJaeger | Is any other infra-root around to help here? | 15:00 |
| AJaeger | Just checked the situation still happens ;( | 15:00 |
| openstackgerrit | Vincent Françoise proposed openstack-infra/project-config: Enable Tempest tests for Watcher https://review.openstack.org/265291 | 15:00 |
| *** vgridnev has joined #openstack-infra | 15:01 | |
| *** esker has joined #openstack-infra | 15:01 | |
| yolanda | AJaeger, i can mark the account as inactive in gerrit, i hope that this helps | 15:02 |
| pabelanger | nibalizer: yolanda: crinkle: clarkb: What do you all think of 205596? Switching to puppetlabs-apache for nodepool using apache::vhost::custom | 15:02 |
| asselin_ | Thelo, mrmartin I think those puppet warnings are ok... | 15:02 |
| mrmartin | asselin_ hi | 15:03 |
| mrmartin | ok. | 15:03 |
| yolanda | account was marked as inactive now | 15:03 |
| yolanda | pabelanger, taking a look at reviews now | 15:04 |
| asselin_ | Thelo, yolanda setting up jenkins security is optional last time I tried. It should work without it. | 15:07 |
| AJaeger | thanks, yolanda | 15:07 |
| *** eharney has joined #openstack-infra | 15:08 | |
| *** esker has quit IRC | 15:08 | |
| TheJulia | crinkle: when you get on, I have a question w/r/t the mysql module upgrade and db grants on bare machines | 15:08 |
| yolanda | asselin, seems Thelo had some mismatch on the ssh key and creds, but haven't figured the problem | 15:08 |
| *** jaypipes is now known as leakypipes | 15:08 | |
| *** hdd has quit IRC | 15:09 | |
| *** zz_dimtruck is now known as dimtruck | 15:09 | |
| *** esker has joined #openstack-infra | 15:09 | |
| asselin_ | yolanda, Thelo I think only the jenkins ssh key needs to be correct in the file system /var/lib/jenkins/.ssh/id_rsa, which is done via puppet with value from heira | 15:10 |
| *** otter768 has joined #openstack-infra | 15:12 | |
| yolanda | anyway, infra-root, when you come online, i'd like to know if i did the right thing just setting the account to inactive to disable a third-party-ci | 15:12 |
| Roamer` | yolanda, thanks a lot for doing that! | 15:12 |
| Roamer` | yolanda, however, it doesn't seem that it's enough :( https://review.openstack.org/#/c/264056 still gets new votes from that account :( | 15:13 |
| *** zeih has quit IRC | 15:14 | |
| Roamer` | yolanda, maybe the CI in question is still logged in via SSH or something | 15:14 |
| asselin_ | yolanda, I remember fungi mentioning closing out some connections | 15:15 |
| *** esker has quit IRC | 15:15 | |
| *** zeih has joined #openstack-infra | 15:16 | |
| *** otter768 has quit IRC | 15:16 | |
| yolanda | ok let me check | 15:16 |
| pabelanger | yolanda: great | 15:19 |
| *** Qiming has quit IRC | 15:19 | |
| *** matrohon has quit IRC | 15:19 | |
| *** esker has joined #openstack-infra | 15:20 | |
| yolanda | asselin, i cannot see any ssh connections to kill | 15:20 |
| *** zeih has quit IRC | 15:20 | |
| *** lucas-hungry is now known as lucasagomes | 15:20 | |
| *** hashar has joined #openstack-infra | 15:21 | |
| openstackgerrit | Dimitry Ushakov proposed openstack-infra/project-config: Add solum-pythonclient to projects https://review.openstack.org/265088 | 15:22 |
| *** fawadkhaliq has joined #openstack-infra | 15:23 | |
| *** dansmith is now known as superdan | 15:23 | |
| openstackgerrit | Dimitry Ushakov proposed openstack-infra/project-config: Add solum-pythonclient to projects https://review.openstack.org/265088 | 15:24 |
| *** esker has quit IRC | 15:25 | |
| fungi | yolanda: anteaya: looking at the gerrit ssh logs, xio-ise-iscsi-ci continues to leave lots of comments even now | 15:26 |
| yolanda | fungi, what is the play for that? i just set that as inactive on gerrit | 15:26 |
| yolanda | and flushed caches just in case | 15:26 |
| fungi | ossh -p 29418 review gerrit show-connections -n|grep 14969 | 15:26 |
| fungi | ossh -p 29418 review gerrit close-connection d9a23b3d | 15:27 |
| fungi | et cetera | 15:27 |
| yolanda | ok, missed the -n | 15:27 |
| yolanda | i did a gerrit show-connections and didn't find anything | 15:27 |
| fungi | without the -n you have to look for the username, and it may be truncated to fit the column | 15:28 |
| fungi | unfortunate that this is the same account which went ballistic and looped not so long ago | 15:28 |
| *** nwkarsten has joined #openstack-infra | 15:29 | |
| *** esker has joined #openstack-infra | 15:30 | |
| yolanda | fungi, i closed them but new ones appear, even if the account is inactive | 15:30 |
| *** esker has quit IRC | 15:30 | |
| *** mtanino has joined #openstack-infra | 15:31 | |
| *** ganso has joined #openstack-infra | 15:32 | |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Allow ironic jobs to run with tempest plugin https://review.openstack.org/265311 | 15:34 |
| *** savihou has quit IRC | 15:34 | |
| *** andrearosa has joined #openstack-infra | 15:35 | |
| fungi | yolanda: the api claims that 14969 account to still be active | 15:36 |
| fungi | yolanda: how did you attempt to disable it? | 15:36 |
| yolanda | fungi, first in the database and flushing caches, and after that i also tried a gerrit set-account --inactive | 15:36 |
| fungi | the db does indeed have inactive=Y for 14969 in the accounts table | 15:38 |
| yolanda | yes, i did that update, but doesn't seem to be reflected | 15:39 |
| jordanP | http://developer.openstack.org/api-ref-compute-v2.1.html returns 404.... | 15:39 |
| AJaeger | fungi, clarkb: could you later check https://review.openstack.org/#/c/264978/ and tell me whether moving the script to the app-catalog repo that runs from the periodic job is fine, please? | 15:39 |
| AJaeger | jordanP: please report on #openstack-doc | 15:39 |
| *** e0ne has quit IRC | 15:42 | |
| fungi | gerrit flush-caches is taking several minutes to complete | 15:43 |
| *** hashar has quit IRC | 15:44 | |
| fungi | yolanda: i used the ssh api to toggle the account to active and back to inactive, then flushed all gerrit's caches, then looked for and killed all open connections for the account in question and the comments seem to have subsided now | 15:44 |
| *** yamahata has joined #openstack-infra | 15:44 | |
| yolanda | cool, so maybe doing that with database needs a restart? | 15:45 |
| fungi | directly modifying the backend database has a tendency to not get reflected in running memory, yeah | 15:45 |
| fungi | depends on what it is, but it seems that was probably where things went badly | 15:45 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config: [WIP] Revert "Mark swarm test as non-voting in Magnum" https://review.openstack.org/265273 | 15:45 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config: Mark swarm test as non-voting in Magnum https://review.openstack.org/264998 | 15:45 |
| *** zz_zz_ja has quit IRC | 15:46 | |
| yolanda | fungi, new lesson learnt, thanks. I'll add this to the runbook for third party, so it gets reflected for another time | 15:46 |
| *** zz_ja has joined #openstack-infra | 15:46 | |
| *** piet has joined #openstack-infra | 15:46 | |
| *** links has joined #openstack-infra | 15:47 | |
| *** sabeen has joined #openstack-infra | 15:47 | |
| *** abregman has joined #openstack-infra | 15:48 | |
| *** jlanoux_ has quit IRC | 15:49 | |
| jeblair | pabelanger: can you approve https://review.openstack.org/264985 and https://review.openstack.org/264988 ? | 15:49 |
| Roamer` | fungi, yolanda, anteaya, AJaeger, thanks a lot to all of you! Now let's see if my Zuul will be able to cope with the Gerrit events... | 15:49 |
| *** jlanoux has joined #openstack-infra | 15:49 | |
| *** bnemec has quit IRC | 15:50 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Remove Python 2.6 compatibility patch. https://review.openstack.org/265321 | 15:51 |
| fungi | yolanda: for future reference, you're also supposed to be able to make an account inactive through the rest api by sending a delete for /a/accounts/{account-id}/active | 15:51 |
| jeblair | AJaeger, fungi, clarkb, docaedo, nibalizer: i tend to think that since it runs on the proposal slave which has a private key we would like to keep private, it should remain in project-config | 15:51 |
| *** sabeen2 has joined #openstack-infra | 15:52 | |
| fungi | yolanda: and a get of that returns a 200 ok if the account is active, or a 204 no content if inactive | 15:52 |
| jeblair | AJaeger, fungi, clarkb, docaedo, nibalizer: it is at least only a periodic job, which means that someone has to merge a malicios change to app-catalog rather than project-config (so it's harder to exploit than simply uploading a change...) | 15:52 |
| *** mrmartin has quit IRC | 15:53 | |
| jeblair | has anyone removed the comments from https://review.openstack.org/#/c/264056 ? | 15:53 |
| *** markus_z has quit IRC | 15:53 | |
| mordred | wow. that's an exciting change | 15:54 |
| fungi | jeblair: not yet, no | 15:54 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Remove Python 2.6 compatibility patch. https://review.openstack.org/265321 | 15:54 |
| *** sabeen has quit IRC | 15:54 | |
| jeblair | i will do so | 15:54 |
| pabelanger | jeblair: done | 15:55 |
| jeblair | select count(*) from change_messages where author_id=14969; | 15:55 |
| jeblair | 16894 | 15:55 |
| jeblair | oh, forgot the change id :) | 15:56 |
| jeblair | select count(*) from change_messages where author_id=14969 and change_id=264056; | 15:56 |
| jeblair | 10198 | 15:56 |
| *** amuller has joined #openstack-infra | 15:56 | |
| jeblair | 6k otherwise seems like a lot though, we should look into that after this. | 15:56 |
| jeblair | fungi, mordred: ^ that look ready to turn into a delete? | 15:57 |
| electrofelix | there is a change queued up in JJB that affects the XML generated by adding a new tag to the output. it does not change the default behaviour of the jobs | 15:57 |
| jeblair | electrofelix: does it change all of our jobs? | 15:57 |
| electrofelix | just that it exposes the option for users to disable the default excludes when archiving | 15:57 |
| electrofelix | jeblair: yes | 15:57 |
| electrofelix | so I'm hesitant to approve until it's clear that it's ok | 15:58 |
| jeblair | electrofelix: we just need to run jjb manually on all our hosts then; if you want to approve it now, we can do that today | 15:58 |
| *** nwkarste_ has joined #openstack-infra | 15:58 | |
| electrofelix | essentially it exposes the option added by https://issues.jenkins-ci.org/browse/JENKINS-20086 | 15:58 |
| electrofelix | new protection added? | 15:59 |
| jeblair | huh, how about that | 15:59 |
| *** esker has joined #openstack-infra | 16:00 | |
| fungi | jeblair: lgtm, that looks like the only change number i see it flooding earlier | 16:00 |
| jeblair | electrofelix: no, that's always been the problem -- updating all our jobs takes hours, so puppet times out every time it tries and they never get updated again; we need to run jjb manually in screen to get them all to update, then puppet triggered runs will work again. | 16:00 |
| *** bnemec has joined #openstack-infra | 16:00 | |
| jeblair | fungi: complete | 16:00 |
| electrofelix | ah | 16:00 |
| *** esker has quit IRC | 16:00 | |
| jeblair | yolanda: ^ see sql above | 16:00 |
| *** piet has quit IRC | 16:00 | |
| mordred | jeblair: yes | 16:00 |
| *** fawadkhaliq has quit IRC | 16:01 | |
| mordred | jeblair: delete from change_messages where author_id=14969 and change_id=264056; yah | 16:01 |
| yolanda | jeblair, looks fine to me, to purge the messages from queue | 16:01 |
| rhe00 | I'm here now. Looking at the X-IO CI - again. | 16:01 |
| *** rhe00 is now known as rhedlind | 16:02 | |
| *** jsavak has quit IRC | 16:02 | |
| yolanda | jeblair, deleting like that doesn't have any effect on the change? new comments can be queued again? i am interested in that for downstream as well | 16:02 |
| *** nwkarsten has quit IRC | 16:03 | |
| fungi | rhedlind: did we get that zuul bug you found merged upstream? you were running with it applied on your system too, right? | 16:03 |
| *** tonytan4ever has joined #openstack-infra | 16:03 | |
| rhedlind | fungi: it was merged upstream. looking at my system now. | 16:04 |
| fungi | yolanda: we've done it in the past and it hasn't seemed to present an issue, though that was also before 2.11 i think so it's anybody's guess | 16:04 |
| *** jsavak has joined #openstack-infra | 16:05 | |
| rhedlind | Argh! the server in question is running zuul 2.1.1.dev109. The fix is in dev110. | 16:05 |
| fungi | er, actually we have done it since gerrit 2.11 so should be fine | 16:05 |
| rhedlind | I upgraded one of the servers, but not this one. | 16:05 |
| jeblair | fungi: it the comments don't appear to be gone from that change, even after a flush-caches --all | 16:06 |
| crinkle | TheJulia: what's up? | 16:06 |
| *** regXboi has joined #openstack-infra | 16:06 | |
| *** ujuc has joined #openstack-infra | 16:07 | |
| *** piet has joined #openstack-infra | 16:08 | |
| *** salv-orl_ has joined #openstack-infra | 16:08 | |
| jeblair | fungi: i wonder if we need a restart, or worse, a reindex? | 16:08 |
| *** dizquierdo has quit IRC | 16:09 | |
| *** zigo has quit IRC | 16:09 | |
| *** zigo has joined #openstack-infra | 16:10 | |
| *** nwkarste_ has quit IRC | 16:10 | |
| *** salv-orlando has quit IRC | 16:11 | |
| jeblair | anteaya: can we just merge the series at https://review.openstack.org/265031 since it's already tested and +2d? | 16:11 |
| *** abregman has quit IRC | 16:12 | |
| *** daemontool has quit IRC | 16:15 | |
| *** annegentle has quit IRC | 16:15 | |
| bknudson_ | gerrit seems slow today | 16:16 |
| *** ujuc has quit IRC | 16:17 | |
| anteaya | jeblair: works for me | 16:18 |
| jeblair | bknudson_: well, there was a runaway third-party ci earlier; not sure if that would have an effect (certainly on the one change with 10k comments, unsure whether it would have follow-on effects) | 16:19 |
| jeblair | bknudson_: clarkb has been monitoring java gc time; we might need to check in on that and see if it's increased | 16:19 |
| anteaya | jeblair: I had thought that the patch that merged last night made 265031 redundant | 16:19 |
| anteaya | jeblair: I guess I mis-understood, I have changed my vote now, if you want to self-approve | 16:19 |
| jeblair | anteaya: it does. i just don't want to waste any more time restacking it and waiting for reviews. | 16:20 |
| anteaya | no problem | 16:20 |
| *** _tzn has joined #openstack-infra | 16:20 | |
| TheJulia | crinkle: so on bare nodes, the openstack_citest user seems to be able to create users, and create databases, however grants seem to fail when they are attempted. Does this ring any bells? | 16:21 |
| fungi | jeblair: yeah, i can't recall whether we restarted after the last one | 16:21 |
| *** mriedem is now known as mriedem_meeting | 16:21 | |
| openstackgerrit | Merged openstack-infra/grafyaml: Add support for nullPointMode https://review.openstack.org/264985 | 16:21 |
| openstackgerrit | Merged openstack-infra/grafyaml: Add tooltip customization support https://review.openstack.org/264988 | 16:22 |
| fungi | TheJulia: it's possible we don't grant granting to openstack_citest. have you looked at the database setup script to see what it does? | 16:22 |
| crinkle | TheJulia: is this new in the last few days? | 16:22 |
| pabelanger | jeblair: template variables for grafyaml: https://review.openstack.org/#/c/250893/ when you have a moment | 16:23 |
| *** aopincar has joined #openstack-infra | 16:23 | |
| crinkle | TheJulia: openstack_citest should have privileges 'ALL' on 'openstack_baremetal_citest.*' | 16:23 |
| docaedo | jeblair: regarding 264978 and the script that runs from app-catalog repo | 16:23 |
| TheJulia | fungi: on bare machines, its explicit in prepare scripts | 16:23 |
| TheJulia | crinkle: new in the last few days | 16:23 |
| electrofelix | jeblair: I'll check that nothing obvious is broken with the change and jenkins 1.565, as an older version is being used by openstack, then I'll approve. | 16:23 |
| *** tzn has quit IRC | 16:23 | |
| TheJulia | pulling grants, it looks like openstack_citest has openstack_baremetal_citest.* and *.* defined | 16:24 |
| TheJulia | s/pulling/show/ | 16:24 |
| docaedo | jeblair: I mainly took this route so I could fix/adjust/iterate quicker. But I understand the concern that we could merge something malicious or scary | 16:24 |
| *** annegentle has joined #openstack-infra | 16:24 | |
| AJaeger | jeblair: thanks for commenting! | 16:25 |
| anteaya | rhedlind fungi jeblair yolanda: thanks for your help here, I just returned and have company so am sort of here ish, I feel an email to -announce is in order but am willing to wait until we have further details | 16:25 |
| *** dims has quit IRC | 16:25 | |
| Shrews | TheJulia: sounds like you need "with grant option" | 16:26 |
| fungi | TheJulia: on a related note, in designing our new non-bare jobs i've been assuming openstack_baremetal_citest.* is redundant now | 16:26 |
| *** dims has joined #openstack-infra | 16:26 | |
| TheJulia | Shrews: this is the ansible module failing :\ | 16:26 |
| crinkle | TheJulia: the mysql docs say ALL [PRIVILEGES]Grant all privileges at specified access level except GRANT OPTION | 16:26 |
| clarkb | fungi it is | 16:26 |
| crinkle | so maybe we need to have 'ALL', 'GRANT OPTION' instead of just 'ALL' | 16:26 |
| *** tonytan4ever has quit IRC | 16:27 | |
| * TheJulia looks at the ansible module code | 16:27 | |
| fungi | if we need grant option, i'd appreciate a patch to keep http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n580 in sync with what gets added | 16:27 |
| *** links has quit IRC | 16:27 | |
| clarkb | fungi it was a thing when we didnt allow db creation and nova and ironic used separate dbs | 16:29 |
| *** devkulkarni has joined #openstack-infra | 16:29 | |
| *** salv-orl_ has quit IRC | 16:29 | |
| AJaeger | jeblair: https://review.openstack.org/#/c/265009/6/tools/grafyaml-validate.sh has some comments by fungi - do you want to redo or should we merge as is? | 16:30 |
| TheJulia | fungi: It looks like the mysql puppet module is in the drivers seat w/r/t default settings for the user | 16:30 |
| *** salv-orlando has joined #openstack-infra | 16:30 | |
| *** jsavak has quit IRC | 16:31 | |
| *** bpokorny has joined #openstack-infra | 16:31 | |
| *** jsavak has joined #openstack-infra | 16:32 | |
| fungi | AJaeger: should be safe to approve and clean up later. it's not wrong, just could be simplified | 16:32 |
| *** aopincar has quit IRC | 16:33 | |
| fungi | TheJulia: oh, that might explain the recent change you're seeing in that casde | 16:33 |
| fungi | case | 16:33 |
| crinkle | I don't immediately see why this would be different between versions of the puppet module | 16:33 |
| *** jistr has quit IRC | 16:33 | |
| clarkb | crinkle: maybe old added grant to all? | 16:34 |
| TheJulia | basically I get errors saying openstack_citest@localhost is unable to open the database I want to grant access to. | 16:34 |
| *** EricGonczer_ has joined #openstack-infra | 16:34 | |
| jeblair | AJaeger: as is | 16:34 |
| AJaeger | fungi, jeblair: +2A | 16:35 |
| jeblair | clarkb: i think we need to try restarting gerrit to see if those extra comments go away; do you want to intervene for memory/gc issues? | 16:35 |
| clarkb | jeblair let me see what melody says really quick | 16:35 |
| fungi | yeah, now might be a good time to make sure any other tuning patches we want are in place | 16:36 |
| AJaeger | jeblair: I'll wait with approving https://review.openstack.org/#/c/265010 until the other changes are in... | 16:36 |
| *** samuelBartel has quit IRC | 16:36 | |
| *** MarkAtwood has joined #openstack-infra | 16:36 | |
| jeblair | AJaeger: i think the dependencies have merged | 16:36 |
| clarkb | GC is still at roughly 2% average | 16:36 |
| TheJulia | bingo, openstack_citest does not have grant privs | 16:37 |
| AJaeger | jeblair: they have - but the testing from master patch not. shouldn't we wait for that to be in? | 16:37 |
| * TheJulia had to hunt through logs | 16:37 | |
| AJaeger | Wait, that was tested in this case... | 16:37 |
| clarkb | the change zaro linked which schedules daily git gc of repos in gerrit is redundant with our git repack cron I think. and I do not expect that to affect java gc | 16:37 |
| jeblair | AJaeger: yep, depends-on :) | 16:37 |
| ttx | As I suspected, people misunderstood completely the stackforge retirement as meaning that their projects were part of the big tent without requiring a TC approval | 16:37 |
| clarkb | unfortunately garbage collecting is highly overloaded here | 16:37 |
| AJaeger | but 265010 needs another +2... | 16:38 |
| ttx | http://lists.openstack.org/pipermail/openstack-dev/2016-January/083589.html | 16:38 |
| clarkb | that said we can probably merge that as redundant should be safe and if it does help then woot | 16:38 |
| ttx | jeblair: ^ sometimes I hate to be right | 16:38 |
| *** nelsnelson has joined #openstack-infra | 16:38 | |
| *** EricGonc_ has joined #openstack-infra | 16:38 | |
| clarkb | jeblair fungi https://review.openstack.org/#/c/262075/ | 16:39 |
| ttx | Also I'm not sure we are seeing the expected decline in things-in-gerrit-but-not-in-big-tent | 16:39 |
| *** tonytan4ever has joined #openstack-infra | 16:40 | |
| jeblair | ttx: sure, that's because you don't want them in the big tent. | 16:40 |
| ttx | jeblair: or because they don't care enough to join | 16:40 |
| *** gyee has joined #openstack-infra | 16:40 | |
| *** EricGonczer_ has quit IRC | 16:41 | |
| jeblair | ttx: i hope you recall that those are two orthogonal issues -- stackforge retirement alone was not expected to change anything related to the number of official openstack projects, only lowering the standards for joining openstack officially was meant to do that. and i think we saw quite a few projects join officially since those standards were lowered. | 16:41 |
| openstackgerrit | Merged openstack-infra/project-config: Pass ZUUL_* env vars through tox https://review.openstack.org/265031 | 16:42 |
| fungi | ttx: people with a loose grasp of english misunderstood anyway | 16:42 |
| jeblair | ttx: anyway, people get confused. there were lots of projects that didn't understand our governance structure before, and there are still lots of projects that don't. | 16:42 |
| fungi | or who didn't read closely enough | 16:42 |
| AJaeger | smcginnis: are you around? Can we discusss on #openstack-doc? | 16:43 |
| ttx | jeblair: sure, I recall... it's just that the confusion I expected just appeared | 16:43 |
| *** sfinucan has quit IRC | 16:43 | |
| jeblair | ttx: stick around and review the project-config repo and you'll see some more. | 16:43 |
| ttx | anyway, that's precisely why I derailed that thread, to have the opportunity for education | 16:43 |
| ttx | :) | 16:44 |
| jeblair | ttx: you might even be able to educate people early there... | 16:44 |
| zaro | morning | 16:44 |
| jeblair | ttx: maybe you could come up with a standard education message for new projects that could be posted as a comment on project-config new repo creation patches. | 16:44 |
| ttx | once I'm done fixing things inside the tent (like giving all of them an accurate release model) I'll certainly move out | 16:44 |
| openstackgerrit | Merged openstack-infra/project-config: Validate grafyaml based on master https://review.openstack.org/265009 | 16:45 |
| jeblair | ttx: we could also more aggressively retire the _word_ 'stackforge' | 16:45 |
| fungi | (for example, fuel creating lots of unofficial repos for fuel plugins) | 16:45 |
| jeblair | ttx: at the request of the tc, we still use 'stackforge' to describe unofficial projects | 16:45 |
| jeblair | ttx: this costs us the opportunity to say "unofficial projects" in some cases | 16:45 |
| ttx | jeblair: to call them what... unofficial ? | 16:45 |
| ttx | right | 16:45 |
| jeblair | ttx: continuing to use 'stackforge' probably just means that people think the documentation is outdated | 16:46 |
| jeblair | ttx: see http://docs.openstack.org/infra/system-config/stackforge.html | 16:46 |
| jeblair | ttx: i've had to -1 several patches that remove stackforge from there | 16:47 |
| ttx | jeblair: I think we could (more aggressively retire "stackforge" in benefit of "unofficial") | 16:47 |
| fungi | i keep saying unofficial projects for clarity in any public communication, but our documentation still mentions stackforge (at the request of the tc) | 16:48 |
| *** fawadkhaliq has joined #openstack-infra | 16:48 | |
| ttx | now that everyone understands that projects are created in openstack/ | 16:48 |
| ttx | worth a try. It's all about guessing what will be the most confusing :) | 16:49 |
| zaro | clarkb: from reading the git documentation it doesn't seem like git gc and git repack do the same things. therefore i don't think it's redundant. | 16:50 |
| samueldmq | AJaeger: ping, about https://review.openstack.org/#/c/265148 | 16:50 |
| AJaeger | samueldmq: yes? | 16:50 |
| samueldmq | AJaeger: I thought only keeping 'deps = .[bandit]' would be enough | 16:51 |
| AJaeger | samueldmq: My understanding is that this is not enough. Did you try it? | 16:51 |
| samueldmq | AJaeger: what are the dependencies from the base env ('{[testenv]deps}') needed for ? | 16:51 |
| clarkb | zaro ah ok | 16:52 |
| ttx | fungi: lol at tc-approved release | 16:52 |
| clarkb | zaro but why would git gc affect java gc? | 16:52 |
| AJaeger | samueldmq: that's the test-requirements and requirements - to install hacking, flake8, bandit, etc | 16:52 |
| clarkb | zaro it is separate garbage collection | 16:52 |
| ttx | especially given my call in 8min with the man behind the name | 16:52 |
| samueldmq | AJaeger: no I didn't, I am just trying to understand it, I will try now | 16:52 |
| AJaeger | samueldmq: I'll try right now as well ;) | 16:52 |
| fungi | ttx: well, the wording in the question was dangerously close to confusing tc-approved release with official projects | 16:52 |
| samueldmq | AJaeger: ah, and is that the default if you don't specify deps ? | 16:53 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Allow openstack_citest 'grant option' privilege https://review.openstack.org/265348 | 16:53 |
| samueldmq | AJaeger: nice | 16:53 |
| crinkle | TheJulia: figured it out ^ | 16:53 |
| AJaeger | samueldmq: exactly, the deps line overrides the default | 16:53 |
| fungi | ttx: but also, yes, amusing cosmic timing | 16:53 |
| AJaeger | samueldmq: so, I have to get the default added back... | 16:53 |
| TheJulia | crinkle: awesome! | 16:53 |
| AJaeger | samueldmq: with my change: pep8 installdeps: .[bandit], -r/home/aj/Software/vcs/OpenStack/openstack/keystone/test-requirements.txt, .[ldap,memcache,mongodb] | 16:54 |
| AJaeger | samueldmq: your proposal: pep8 installdeps: .[bandit] | 16:54 |
| AJaeger | so that will be quite different environment | 16:54 |
| zaro | clarkb: it probably doesn't. i don't have any other tuning changes targeted specifically for java gc atm. | 16:55 |
| *** ddieterly has joined #openstack-infra | 16:55 | |
| samueldmq | AJaeger: I agree, because it will intall bandit, but not the other things potentially needed for running pep8 itself | 16:55 |
| samueldmq | install* | 16:55 |
| zaro | clarkb: the only thing that i can think of that might be related is that if not running git gc gerrit may use more memory to load git changes? just a guess. | 16:55 |
| jeblair | fungi, ttx: https://review.openstack.org/265352 | 16:56 |
| samueldmq | AJaeger: you're correct, without it: | 16:58 |
| samueldmq | WARNING:test command found but not installed in testenv | 16:58 |
| samueldmq | cmd: /usr/local/bin/flake8 | 16:58 |
| samueldmq | env: /home/samuel/workspace/keystone/.tox/pep8 | 16:58 |
| zaro | clarkb: the perf tuning cheat sheet says that if you don't run git gc regularly then it will have an affect on "smooth source code browsing experience". | 16:58 |
| samueldmq | AJaeger: thanks | 16:58 |
| clarkb | zaro it shouldnt since gc will only kill things that arent valid refs | 16:59 |
| AJaeger | samueldmq: thanks for asking! | 16:59 |
| jeblair | ttx: with that, we can update http://docs.openstack.org/infra/system-config/stackforge.html and then point all new projects requests at it to make sure they understand the implications. | 16:59 |
| clarkb | zaro so anything associated with a change should not get git gc'd | 16:59 |
| jeblair | pleia2: see fungi and my conversation with ttx above and also https://review.openstack.org/265352 | 16:59 |
| anteaya | rhedlind: I have to send something to the -announce mailing list regarding today's incident, what is your current system status? | 17:00 |
| *** derekh has quit IRC | 17:00 | |
| openstackgerrit | Merged openstack-infra/system-config: Schedule Gerrit's git garbage collection https://review.openstack.org/262075 | 17:01 |
| jeblair | wait what? ^ | 17:01 |
| jeblair | weren't we *just* talking about how that's not necessary? | 17:02 |
| zaro | clarkb: not that i can think of. | 17:02 |
| *** jsavak has quit IRC | 17:03 | |
| clarkb | jeblair zaro points out that it doesnt overlap with repack and the tuning guide says you should do it for "smooth code review" (though I didnt approve) | 17:03 |
| clarkb | still trying to sort through what is what | 17:03 |
| jeblair | clarkb: who says it doesn't overlap with repack? | 17:03 |
| *** ujuc has joined #openstack-infra | 17:03 | |
| clarkb | jeblair zaro just a few minutes ago | 17:03 |
| *** erlon has joined #openstack-infra | 17:03 | |
| jeblair | zaro: source? | 17:03 |
| rhedlind | anteaya: I have properly upgraded zuul to the latest version, upgraded jenkins to the latest and working on upgrading nodepool. it should be back up in a few minutes | 17:03 |
| anteaya | okay thank you | 17:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Revert "Schedule Gerrit's git garbage collection" https://review.openstack.org/265356 | 17:04 |
| *** hdd has joined #openstack-infra | 17:04 | |
| jeblair | I'm self-approving that change now ^ | 17:04 |
| zaro | jeblair, clarkb : gc.interval http://tinyurl.com/q2aw6ba | 17:04 |
| *** sputnik13 has joined #openstack-infra | 17:04 | |
| jeblair | zaro: where in that does it say that the effect of gc is not included in repack? | 17:05 |
| anteaya | request for comment, does anyone have any objection to turning rhedlind's ci system xio-ise-iscsi-ci back on once he is finished his system upgrades? | 17:05 |
| anteaya | if you have any hesitation, do state it please | 17:05 |
| zaro | jeblair: ohh, i inferred it from git documentation https://git-scm.com/docs/git-repack | 17:05 |
| *** jsavak has joined #openstack-infra | 17:06 | |
| *** sputnik13 has quit IRC | 17:06 | |
| *** devkulkarni has quit IRC | 17:06 | |
| *** mriedem_meeting is now known as mriedem | 17:07 | |
| *** devkulkarni has joined #openstack-infra | 17:07 | |
| AJaeger | anteaya: let's ask rhedlind for a 24x7 emergency phone number ;) | 17:08 |
| zaro | clarkb: one concern i have is that we are setting ssh.threads too high relative to the cores on review.o.o | 17:08 |
| anteaya | AJaeger: I wouldn't call it anyway | 17:08 |
| anteaya | :) | 17:08 |
| zaro | clarkb: and memory | 17:08 |
| rhedlind | AJaeger: :) | 17:09 |
| zaro | clarkb: documentation recommends 2*<#Cores> | 17:09 |
| jeblair | zaro, clarkb, fungi: can we get a +1 from me on any tuning changes please? | 17:09 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add instructions on how to disable a third party ci account https://review.openstack.org/265359 | 17:09 |
| clarkb | zaro we have it set to 4*? | 17:11 |
| openstackgerrit | Merged openstack-dev/pbr: Add dependencies for building wheels https://review.openstack.org/264010 | 17:11 |
| zaro | clarkb: i thought review.dev was only 8 core? | 17:11 |
| clarkb | that sounds right | 17:11 |
| clarkb | but I would have to double check | 17:12 |
| zaro | clarkb: and i believe sshd.threads is set to 100? | 17:12 |
| openstackgerrit | Merged openstack-dev/pbr: Make setup.py --help-commands work without testrepository https://review.openstack.org/263928 | 17:12 |
| *** otter768 has joined #openstack-infra | 17:13 | |
| jeblair | where are all the comments from when i tuned gerrit before? | 17:13 |
| *** ujuc has quit IRC | 17:13 | |
| jeblair | ah, found them. system-config/modules/openstack_project/manifests/review.pp | 17:14 |
| anteaya | yolanda: whitespace error in 265359 | 17:15 |
| jeblair | zaro: so that value was set because we were seeing things queue up in show-queue | 17:15 |
| *** AJaeger has quit IRC | 17:15 | |
| jeblair | zaro: if we decrease it, we will be de-tuning something we have already tuned | 17:15 |
| yolanda | anteaya, thanks, i'll correct | 17:15 |
| openstackgerrit | Merged openstack-infra/project-config: Grafana: Update OVH graph with axis labels https://review.openstack.org/265010 | 17:15 |
| jeblair | zaro: we may want to increase it if we see things queue up again | 17:15 |
| jeblair | zaro: but i don't think we've seen that. | 17:15 |
| electrofelix | jeblair: got distracted, confirmed that a jenkins instance of 1.565.2 doesn't have an issue with the extra tag added by https://review.openstack.org/259666, so should be ok to update the jobs manually after approval. | 17:16 |
| *** jlanoux has quit IRC | 17:16 | |
| zaro | jeblair: i'm assuming you are referring to sshd.threads? | 17:16 |
| jeblair | zaro: yes | 17:16 |
| jeblair | zaro: so i wonder why you want to decrease it? | 17:16 |
| *** esker has joined #openstack-infra | 17:16 | |
| zaro | jeblair: i'm not suggesting to decrease it. i just have a concern that it's too high. | 17:17 |
| zaro | for the resources available on review.o.o. | 17:17 |
| pleia2 | jeblair: thanks, it was indeed a confusing situation | 17:17 |
| *** otter768 has quit IRC | 17:17 | |
| jeblair | zaro: what resources are being impacted by it being too high? | 17:17 |
| clarkb | and fwiw it is 8vcpu | 17:17 |
| zaro | jeblair: heap issues | 17:18 |
| anteaya | yolanda: I also added a comment about location | 17:18 |
| yolanda | thanks | 17:18 |
| anteaya | yolanda: I think the information is valuable and should be documented | 17:18 |
| jeblair | zaro: what effect does the number of sshd threads have on the heap? | 17:18 |
| anteaya | yolanda: I would just like to see if we have a better, meant for infra root use kind of document that would work | 17:19 |
| yolanda | anteaya, can you review https://review.openstack.org/256389 and https://review.openstack.org/260068 as well? also documentation bits | 17:19 |
| zaro | jeblair: ok, that's a good question that i haven't done much research into yet. that's why i haven't proposed a change | 17:19 |
| * anteaya reviews | 17:20 | |
| jeblair | zaro: okay. are there any proposed tuning changes i should review? | 17:20 |
| zaro | jeblair: the collab net doc says "if you run into heap space issues, scale down this number" which is why i thought this might be an issue. | 17:20 |
| yolanda | anteaya, it can fit as well on gerrit page | 17:20 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add instructions on how to disable a third party ci account https://review.openstack.org/265359 | 17:21 |
| zaro | jeblair: all gerrit tuning changes have topic 'gerrit-tuning' | 17:21 |
| fungi | we can revert 262075, i was taking clarkb's comment at 16:39 to mean that we should go ahead and merge it before restarting gerrit | 17:21 |
| jeblair | zaro: right, but that will cause other problems like people being unable to push, fetch, query or just about anything else; i think increasing the heap size to accomodate the need of the sshd threads is better. | 17:22 |
| clarkb | fungi: oh sorry, I was just ointing out what had been proposed for tuning | 17:22 |
| fungi | ahh, whoops | 17:22 |
| zaro | jeblair: yes, i agree. i think we need to give review.o.o more resources. | 17:22 |
| fungi | also rereading that now, i worry that it was conflating git gc with hava gc | 17:22 |
| *** mriedem is now known as mriedem_lunch | 17:23 | |
| fungi | java gc | 17:23 |
| *** unicell has quit IRC | 17:23 | |
| jeblair | fungi: yes, see my revert at https://review.openstack.org/265356 | 17:23 |
| *** annegentle has quit IRC | 17:24 | |
| *** nelsnels_ has joined #openstack-infra | 17:24 | |
| *** esker has quit IRC | 17:25 | |
| anteaya | yolanda: great, I think the gerrit page would be a better location for how to disable a third party ci account, thank you | 17:25 |
| *** e0ne has joined #openstack-infra | 17:25 | |
| *** fawadkhaliq has quit IRC | 17:26 | |
| *** fawadkhaliq has joined #openstack-infra | 17:26 | |
| * anteaya doesn't know the difference between git gc/java gc/gerrit gc but acknowledges that they appear to be different things | 17:26 | |
| *** docaedo has quit IRC | 17:27 | |
| *** yolanda has quit IRC | 17:27 | |
| *** annegent_ has joined #openstack-infra | 17:27 | |
| *** nelsnelson has quit IRC | 17:27 | |
| *** pfallenop has quit IRC | 17:27 | |
| *** morgabra has quit IRC | 17:27 | |
| *** superdan has quit IRC | 17:27 | |
| *** unicell has joined #openstack-infra | 17:27 | |
| *** dansmith has joined #openstack-infra | 17:27 | |
| *** pfallenop has joined #openstack-infra | 17:27 | |
| *** docaedo25 has joined #openstack-infra | 17:27 | |
| fungi | java garbage collection is cleaning up unreferenced memory allocations inside the jvm, git garbage collection is removing unreferenced git objects in a repository | 17:27 |
| *** morgabra_ has joined #openstack-infra | 17:27 | |
| *** yolanda has joined #openstack-infra | 17:28 | |
| greghaynes | clarkb: have you seen the nodepool ci fail http://logs.openstack.org/75/195175/37/check/gate-nodepool-python27/a99adbf/console.html ? | 17:29 |
| *** dtantsur is now known as dtantsur|afk | 17:29 | |
| *** mudassirlatif has joined #openstack-infra | 17:29 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add instructions on how to disable a third party ci account https://review.openstack.org/265359 | 17:29 |
| greghaynes | clarkb: did the puppet mysql stuff upgrade mysql in images? | 17:29 |
| clarkb | greghaynes: I haven't | 17:29 |
| clarkb | greghaynes: it wasn't supposed to, but TheJulia found that grants aren't working | 17:29 |
| greghaynes | clarkb: that sounds like what this is | 17:29 |
| clarkb | greghaynes: I think crinkle is looking at it | 17:30 |
| jeblair | fungi, clarkb: we've gotten severely distracted, but i believe we still need to either restart or reindex gerrit in order to fix the issue with so many comments on https://review.openstack.org/#/c/264056 | 17:30 |
| greghaynes | clarkb: ah, ok | 17:30 |
| clarkb | jeblair: yes, a restart would make sense. I think the only thing worth delaying a restart for is the web session cache maxAge value if we want to update that really quickly | 17:30 |
| fungi | jeblair: have you tried reloading the page? i don't see the comments on it | 17:31 |
| jeblair | fungi: yes, i've shift-reloaded. should i quit browser reload? | 17:31 |
| fungi | you may still have them cached in javascript or something | 17:31 |
| *** dansmith is now known as superdan | 17:31 | |
| fungi | i had never browsed that page, so pulling it up now i only see maybe a couple dozen comments from various accounts and not the account in question | 17:31 |
| *** pblaho has quit IRC | 17:32 | |
| fungi | and that's with toggle ci expanded, for reference | 17:32 |
| jeblair | fungi: that has corrected the issue. that's disturbing. | 17:32 |
| jeblair | so nevermind about the gerrit restart. | 17:32 |
| fungi | back to the sidetrack! | 17:32 |
| *** dizquierdo has joined #openstack-infra | 17:33 | |
| anteaya | fungi: thank you | 17:33 |
| jeblair | zaro: i have looked at the other outstanding changes. a max file size seems fine. i don't think we should increase the diff timeout without serious considuration. | 17:33 |
| anteaya | jeblair: comments are gone for me | 17:33 |
| *** morgabra_ is now known as morgabra | 17:34 | |
| *** morgabra has quit IRC | 17:34 | |
| *** morgabra has joined #openstack-infra | 17:34 | |
| crinkle | clarkb: greghaynes i am looking and have https://review.openstack.org/#/c/265348/1 but it is not going to work and I am getting on a plane soon | 17:34 |
| *** lezbar has quit IRC | 17:34 | |
| anteaya | crinkle: safe flight | 17:34 |
| electrofelix | jeblair: are you ok to update the JJB jobs manually? | 17:35 |
| greghaynes | crinkle: ah | 17:35 |
| *** salv-orlando has quit IRC | 17:35 | |
| anteaya | crinkle: how are your infra-cloud storyboard clean-up efforts going? https://storyboard.openstack.org/#!/story/2000175 | 17:35 |
| clarkb | thats a fun behavior, throw out the list if all is used | 17:35 |
| jeblair | infra-root: ^ is someone available to run manual jjb updates after electrofelix merges an update-all-jobs jjb change? | 17:35 |
| zaro | jeblair: sounds good. | 17:35 |
| *** salv-orlando has joined #openstack-infra | 17:36 | |
| crinkle | clarkb: yeah >.< | 17:36 |
| *** ifarkas has quit IRC | 17:36 | |
| crinkle | anteaya: i have poked some people, can't really focus on it today | 17:36 |
| fungi | jeblair: i can take care of that, sure | 17:37 |
| jeblair | fungi: thanks; electrofelix ^ | 17:37 |
| clarkb | crinkle: will you be able to update with a list of all the options? | 17:37 |
| clarkb | crinkle: or should I start working on pushing that patchset | 17:37 |
| electrofelix | fungi: thanks, I'll pull the trigger on it now so | 17:37 |
| fungi | i'm getting logged in and screen sessions readied on all our masters now | 17:37 |
| crinkle | clarkb: i will have scattered time today so if you want to work on it now that would be awesome | 17:38 |
| *** jsavak has quit IRC | 17:38 | |
| electrofelix | jeblair: thanks | 17:38 |
| clarkb | crinkle: kk | 17:38 |
| *** unicell has quit IRC | 17:38 | |
| *** jsavak has joined #openstack-infra | 17:38 | |
| *** mrmartin has joined #openstack-infra | 17:38 | |
| greghaynes | clarkb: crinkle this is going to require a new image upload so it isnt really a rush - will be a day minimum before it is fixed | 17:39 |
| *** esker has joined #openstack-infra | 17:39 | |
| clarkb | ya, but better to fix now while I am thinking of it | 17:39 |
| *** MarkAtwood has quit IRC | 17:40 | |
| *** dprince has quit IRC | 17:40 | |
| fungi | electrofelix: for reference, all our masters currently report jenkins-job-builder 1.4.1.dev21 is installed | 17:40 |
| clarkb | actually I think we may be able to use two different mysql_grant calls | 17:40 |
| *** dprince has joined #openstack-infra | 17:41 | |
| *** dprince has quit IRC | 17:41 | |
| clarkb | one to grant all and one to grant the grant option. | 17:41 |
| clarkb | crinkle: ^ does that make more sense to you or do you think one call with complete grant list is best? | 17:41 |
| *** dprince has joined #openstack-infra | 17:41 | |
| *** jerryz has joined #openstack-infra | 17:41 | |
| *** docaedo25 is now known as docaedo | 17:41 | |
| fungi | electrofelix: once i see that rise to .dev22 i'll start manual config updates on all of them | 17:41 |
| docaedo | jeblair: did you have more thoughts/opinions about the script for the app-catalog periodic job living in app-catalog? | 17:42 |
| *** odlD2 has quit IRC | 17:42 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for default-excludes in archives https://review.openstack.org/259666 | 17:43 |
| jeblair | docaedo: i'd like other infra-root folks to weigh in on it | 17:43 |
| electrofelix | fungi: thanks, change has merged, so presume it shouldn't take too long | 17:43 |
| docaedo | jeblair: roger that :) I'm around for discussion if other infra-root peeps want to discuss | 17:43 |
| nibalizer | I am mostly traveling today | 17:43 |
| clarkb | jeblair: docaedo I remember saying it should be in project-config a little while back | 17:43 |
| *** esker has quit IRC | 17:44 | |
| *** Swami has joined #openstack-infra | 17:44 | |
| *** jordanP has quit IRC | 17:45 | |
| docaedo | clarkb: I can move it back there | 17:46 |
| fungi | jeblair: clarkb: docaedo: yes, if it's a script running on a sensitive worker with access to a gerrit push account, we need that script reviewed in project-config | 17:46 |
| docaedo | fungi clarkb jeblair - understood, I'm on it (though this foils my botnet plans :/ ) | 17:47 |
| crinkle | clarkb: that won't work because the grant uses the name/host/db in the title so you would have the duplicate definition problem | 17:47 |
| *** sabeen2 has quit IRC | 17:47 | |
| clarkb | ugh | 17:47 |
| anteaya | crinkle: understood, good work there | 17:47 |
| *** piet has quit IRC | 17:48 | |
| jeblair | crinkle: what's "name" in that triplet? | 17:48 |
| fungi | user? | 17:48 |
| crinkle | jeblair: username, excuse me | 17:48 |
| jeblair | crinkle: np, thx. | 17:48 |
| *** yaume has quit IRC | 17:49 | |
| jeblair | fungi: are index.old and cache.old directories in ~gerrit2/review_site created by gerrit or by us? | 17:49 |
| fungi | jeblair: by us i think. that is probably cruft from our 2.10 upgrade attempt in may judging from the last modified dates on them | 17:50 |
| openstackgerrit | Christopher Aedo proposed openstack-infra/project-config: Update app-catalog link check job https://review.openstack.org/264978 | 17:50 |
| jeblair | that's about 1.7G we could get back | 17:51 |
| fungi | jeblair: yep, sudo grep 'cache.*old' ~gerrit2/.bash_history | 17:52 |
| fungi | confirms it | 17:52 |
| jeblair | nice | 17:52 |
| jeblair | fungi: how about i delete those? | 17:52 |
| rhedlind | asselin: ping | 17:52 |
| fungi | jeblair: please do. they're way older than we could ever hope to make use of | 17:52 |
| *** sridhar_ram has joined #openstack-infra | 17:53 | |
| jeblair | done | 17:53 |
| anteaya | rhedlind: I'll post to -announce about the incident today and that your system is currently disabled, acknowledging you are aware and are addressing the issue at your end | 17:53 |
| asselin | rhedlind, just ask | 17:53 |
| fungi | jeblair: i'm also doing an apt-get autoremove which is freeing ~5gb | 17:53 |
| anteaya | rhedlind: we can follow up when you are ready | 17:53 |
| rhedlind | anteaya: sounds good | 17:54 |
| jeblair | fungi: ah nice | 17:54 |
| fungi | and then i'll do a clean and see how much that gets us back | 17:54 |
| *** odlD2 has joined #openstack-infra | 17:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Pass ZUUL_* env vars through tox" https://review.openstack.org/265034 | 17:54 |
| rhedlind | asselin: I upgraded to the latest nodepool and now nodepool cannot authenticate with the provider. from what I understand the provider user and password still goes in the nodepool.yaml file, right? | 17:54 |
| *** reed has quit IRC | 17:55 | |
| *** reed has joined #openstack-infra | 17:55 | |
| fungi | unfortunately most of what autoremove is doing is cleaning out old kernel packages we're no longer using, and each of those fires a rescan of the initrds for the remaining kernels... it's slow-ish | 17:55 |
| jeblair | i think the bulk of / is mysql backups; we probably don't need 31 days -- iff bup is working as expected. :) | 17:55 |
| *** EricGonc_ has quit IRC | 17:55 | |
| fungi | yeah, on another server (which one?) we backed down to 10 days retention i think | 17:55 |
| fungi | ask.o.o probably | 17:55 |
| asselin | rhedlind, I haven't checked the latest nodepool....afaik that functionaliy hasn't been removed, but you'd have to look at the commit history | 17:56 |
| jeblair | rhedlind: use clouds.yaml | 17:56 |
| asselin | rhedlind, the newer way is to use a secure.conf & clouds.yaml file. the puppet scripts have been updated to do that. | 17:56 |
| jeblair | right, that :) | 17:57 |
| fungi | electrofelix: 1.4.1.dev23 (not .dev22 because of the merge commit) is starting to show up on some of our masters now | 17:57 |
| fungi | electrofelix: yep, propagated to all of them now, so i'll go ahead with the config reprocessing on them | 17:58 |
| *** doug-fis_ has joined #openstack-infra | 17:58 | |
| *** doug-fis_ has quit IRC | 17:58 | |
| rhedlind | ah, so I did update to use the secure.conf. I will look into clouds.yaml | 17:59 |
| *** doug-fis_ has joined #openstack-infra | 17:59 | |
| rhedlind | asselin, jeblair: thanks! | 17:59 |
| *** AJaeger has joined #openstack-infra | 18:00 | |
| *** piet has joined #openstack-infra | 18:00 | |
| *** e0ne has quit IRC | 18:01 | |
| *** flepied has quit IRC | 18:01 | |
| *** yamahata_ has joined #openstack-infra | 18:01 | |
| AJaeger | sdague: We analyzed the translation import, it's a bug in babel 2.2.0 that is fixed now (from fix to merge in 24 hours) but not yet released. Since this is part of the scripting we do after downloading, it's fine to merge translations. Please go ahead. At one time you might get the translators back ;) | 18:01 |
| *** doug-fish has quit IRC | 18:01 | |
| fungi | infra-root: okay, manual jjb update is running on every one of our masters in screen sessions under my account (if you want to attach, `sudo su - fungi` and then `script /dev/null` to spawn a fresh tty before you `screen -x`) | 18:02 |
| *** dims_ has joined #openstack-infra | 18:02 | |
| *** bpokorny has quit IRC | 18:03 | |
| fungi | jeblair: we're down to 24gb used on review.o.o's rootfs now | 18:03 |
| AJaeger | sdague: https://review.openstack.org/#/c/252197/ is current import for nova if you could approve, please | 18:03 |
| *** doug-fis_ has quit IRC | 18:03 | |
| jeblair | fungi: oh good | 18:03 |
| fungi | jeblair: 17gb of which is /var/backups/mysql_backups | 18:03 |
| *** tristanC_ has joined #openstack-infra | 18:04 | |
| *** wfoster has quit IRC | 18:04 | |
| *** annegentle has joined #openstack-infra | 18:04 | |
| *** tristanC has quit IRC | 18:04 | |
| *** annegent_ has quit IRC | 18:04 | |
| *** yamahata has quit IRC | 18:04 | |
| *** dims has quit IRC | 18:05 | |
| *** wfoster has joined #openstack-infra | 18:05 | |
| *** amuller is now known as amuller_afk | 18:05 | |
| fungi | pointed out before, but jenkins, jenkins01 and jenkins02 are considerably slower than the other production masters... chances are they're on an older nova flavor and could stand a rebuild at some point in the future | 18:05 |
| *** yamahata_ has quit IRC | 18:06 | |
| *** rhedlind has quit IRC | 18:06 | |
| *** rhe00 has joined #openstack-infra | 18:06 | |
| fungi | it's less impactful to jenkins.o.o since its load is negligible but 01 and 02 crawl | 18:06 |
| *** shardy has quit IRC | 18:07 | |
| *** unicell has joined #openstack-infra | 18:07 | |
| jeblair | clarkb, fungi: http://grafana.openstack.org/dashboard/db/nodepool | 18:07 |
| jeblair | clarkb, fungi: jenkins06 is anamolous | 18:07 |
| *** dprince has quit IRC | 18:08 | |
| fungi | indeed. that usually means it's having problems | 18:08 |
| *** dprince has joined #openstack-infra | 18:08 | |
| fungi | piles of ready and deleting nodes indicate the api was unresponsive i think | 18:08 |
| fungi | it's probably got a runaway thread leak going again | 18:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/grafyaml: Add support for template query https://review.openstack.org/250893 | 18:09 |
| fungi | oh yeah, very slow to respond on the webui, possibly because i'm also hammering it with jjb right now | 18:09 |
| clarkb | melody confirms >7k threads | 18:09 |
| fungi | i can put it in prepare for shutdown for now so it doesn't get worse and then see if that gets us through the job updates | 18:10 |
| jeblair | fungi: ++ | 18:10 |
| fungi | done | 18:11 |
| *** doug-fish has joined #openstack-infra | 18:12 | |
| anteaya | clarkb: in terms of things you are juggling are you full or do you have a hand free for logstash 2.0 on worker 20 for a moment? | 18:12 |
| anteaya | clarkb: in terms of exchanging thoughts | 18:13 |
| clarkb | anteaya: sure, mostly just digigng up mysql docs to generate a list of grant permissions | 18:13 |
| anteaya | understood | 18:13 |
| clarkb | trying to determine if hte lists are different between versions | 18:13 |
| clarkb | mordred: ^ you may just know of the top of your head | 18:13 |
| anteaya | via cacit I see worker20's memory usage has increased in the past 3 days, it is now up to 2G | 18:13 |
| mordred | clarkb: aroo? | 18:14 |
| greghaynes | clarkb: generate a list of grant perms? | 18:14 |
| clarkb | greghaynes: ya | 18:14 |
| *** ujuc has joined #openstack-infra | 18:14 | |
| greghaynes | clarkb: oh, see what the grant perms are for a given user | 18:14 |
| mordred | clarkb: like, somethin gmore than "show grants" ? | 18:14 |
| mordred | do you want the grants a user has, or a list of possible things that can be granted? | 18:14 |
| clarkb | yes more than that, I basically need to see if what ALL aliases to has changed over releases | 18:15 |
| clarkb | mordred: the possible things that ALL is aliased to | 18:15 |
| clarkb | because Ineed to do ALL + GRANT OPTION in puppet that doesn't break and want to double check that if we run against different versions of mysql the list isn't different | 18:15 |
| *** dizquierdo has quit IRC | 18:15 | |
| clarkb | as an alternative we could just use an exec | 18:16 |
| mordred | clarkb: use mysql ; show create table user \G | 18:16 |
| clarkb | which is maybe the least bad way to do this | 18:16 |
| mordred | clarkb: then look at columns that end in _priv | 18:16 |
| *** doug-fish has quit IRC | 18:16 | |
| clarkb | anteaya: looks like a couple worker processes have soakced up memory. python is bad at releasing memory so that doesn't surprise me | 18:17 |
| mordred | clarkb: grant is just sugar around entries in that table; | 18:17 |
| anteaya | clarkb: okay | 18:17 |
| *** dimtruck is now known as zz_dimtruck | 18:18 | |
| *** ujuc has quit IRC | 18:18 | |
| clarkb | mordred: greghaynes ya I think I am just going to exec mysql and do a proper grant for the case where we give all perms out | 18:19 |
| * anteaya goes shopping for system-config core reviews on Logstash 2.0 compat ES output rule 247714 so that clarkb can upgrade the other 19 logstash workers to logstash 2.0 | 18:19 | |
| greghaynes | clarkb: SGTM | 18:19 |
| clarkb | that way I can just rely on the alias | 18:19 |
| mordred | ++ | 18:19 |
| greghaynes | ++ for good ol sql | 18:19 |
| *** thorst has quit IRC | 18:20 | |
| mrmartin | hi | 18:20 |
| AJaeger | jeblair: do you want to remove your -2 from the app-catalog change https://review.openstack.org/#/c/264978/3 ? | 18:20 |
| *** thorst has joined #openstack-infra | 18:20 | |
| *** MarkAtwood has joined #openstack-infra | 18:21 | |
| *** piet has quit IRC | 18:21 | |
| jeblair | AJaeger: +5 | 18:21 |
| anteaya | oh and if someone with irc ops feels like refreshing the topic in -meeting that would be great | 18:22 |
| crinkle | clarkb: ++ to exec | 18:23 |
| mrmartin | fungi: https://ask-staging.openstack.org is a bit broken, and returns an HTTP 500. I started to check what happened. | 18:23 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Publish openstackci docs https://review.openstack.org/259587 | 18:23 |
| fungi | anteaya: done | 18:24 |
| fungi | mrmartin: i can take a look too | 18:24 |
| *** thorst has quit IRC | 18:25 | |
| anteaya | fungi: thank you | 18:25 |
| mrmartin | fungi: yeap, I already found 7-8 things related to changes in askbot-devel repo. | 18:25 |
| anteaya | fungi jeblair openstackci docs published to http://docs.openstack.org/infra/openstackci/ any objection? | 18:25 |
| * anteaya prepares to +A asselin's patch | 18:25 | |
| asselin | very exciting! | 18:25 |
| fungi | mrmartin: that's... exciting | 18:25 |
| AJaeger | jeblair: 5? You're generous today ;) | 18:26 |
| fungi | AJaeger: the difference between -2 and +3 | 18:26 |
| *** sputnik13 has joined #openstack-infra | 18:26 | |
| anteaya | fungi jeblair patch I am looking at is here: https://review.openstack.org/#/c/259587/2 | 18:27 |
| mrmartin | fungi: I have a patch for most of the issues, but the setup we are using break at db sync and migrate due some change in the language handling. http://paste.openstack.org/show/483343/ | 18:27 |
| fungi | mrmartin: yeah, "IOError: sys.stdout access restricted by mod_wsgi" looks pretty bad | 18:27 |
| mrmartin | it is not the error, just a result of an error situation :) | 18:28 |
| AJaeger | fungi: Indeed... | 18:28 |
| fungi | and "Target WSGI script '/srv/askbot-site/config/django.wsgi' cannot be loaded as Python module." | 18:28 |
| AJaeger | armax, dougwig : Could you check https://review.openstack.org/#/c/258992/ , please? | 18:28 |
| fungi | mrmartin: did you have any ideas for how to manually work around the migration failure there? | 18:29 |
| mrmartin | fungi: I cannot even redeploy the production ask.o.o in a vm, because a non-explicit dependency definition broke due a missing ubuntu package. The solution lives here: https://review.openstack.org/#/c/265231/ | 18:30 |
| mrmartin | fungi: yeap, I'll take care of it and deliver some solution | 18:30 |
| *** AJaeger has quit IRC | 18:30 | |
| * armax looks | 18:30 | |
| fungi | mrmartin: oh, yep, we've seen pillow 3.x libjpeg integration breaking lots of other stuff too | 18:31 |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Schedule Gerrit's git garbage collection" https://review.openstack.org/265356 | 18:31 |
| *** yamahata_ has joined #openstack-infra | 18:31 | |
| fungi | mrmartin: thanks for the heads up, and let me know if there's anything else you need | 18:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Allow openstack_citest 'grant option' privilege https://review.openstack.org/265348 | 18:31 |
| clarkb | crinkle: greghaynes mordred ^ | 18:31 |
| fungi | infra-puppet-core reviewers, 265231 solves a problem with latest pillow in puppet-askbot | 18:32 |
| electrofelix | fungi: just happened to merge another change as well that has no impact | 18:32 |
| mordred | clarkb: lgtm | 18:32 |
| mrmartin | nothing actually, first of all I want to bring it back alive in a vm, than deliver the patches made there. | 18:32 |
| fungi | electrofelix: ahh, well anyway the updates are progressing well | 18:32 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Improve usability of the "Edit board" form https://review.openstack.org/263761 | 18:33 |
| *** kdas_ has joined #openstack-infra | 18:34 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on Asymetric key thumbprint calculation: https://review.openstack.org/265397 | 18:34 |
| *** Guest28831 has quit IRC | 18:35 | |
| *** bharathm has joined #openstack-infra | 18:35 | |
| *** kushal has quit IRC | 18:35 | |
| *** mriedem_lunch is now known as mriedem | 18:35 | |
| *** electrofelix has quit IRC | 18:38 | |
| *** reaperhulk has joined #openstack-infra | 18:38 | |
| *** zz_dimtruck is now known as dimtruck | 18:42 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add range search functionality https://review.openstack.org/264869 | 18:42 |
| *** _nadya_ has joined #openstack-infra | 18:42 | |
| anteaya | asselin: I didn't see any objections | 18:44 |
| pabelanger | jeblair: so, does this regex look right to you, to match for rax? (?<=rax-).* | 18:44 |
| asselin | anteaya, me neither | 18:44 |
| pabelanger | jeblair: tested at http://pythex.org/ but grafana doesn't like it. I wonder if it just lacks support for it | 18:45 |
| jeblair | pabelanger: what graphite version did groupbyregex (or whatever it's called) appear in? | 18:46 |
| pabelanger | jeblair: unsure | 18:47 |
| anteaya | fungi: this is an aging patch of yours regarding constraints updates, do you feel that keeping it alive has value? https://review.openstack.org/#/c/202210/ | 18:48 |
| jeblair | pabelanger: ah, aliasSub | 18:48 |
| jeblair | that's the one i was thinking of | 18:48 |
| jeblair | pabelanger: and it's in 0.9.12 which is what we're running | 18:49 |
| jeblair | &target=aliasSub(ip.*TCP*,"^.*TCP(\d+)","\1") | 18:49 |
| fungi | anteaya: given the raciness of it, probably needs to be brought back up from its base argument and possible solutions proposed again. i'll abandon that for now | 18:50 |
| thingee | fungi: there was some ping back in tuesday for something to note in this week's dev summary? | 18:50 |
| pabelanger | jeblair: perfect | 18:50 |
| thingee | fungi: I was reading the logs but couldn't make sense of what to note | 18:50 |
| fungi | thingee: i remember the when but not the what... looking again | 18:50 |
| jeblair | pabelanger: that does look correct -- i expect that to match "rax-iad" and return "iad" for the match. | 18:51 |
| anteaya | fungi: thank you | 18:51 |
| fungi | thingee: oh, a reminder that our worker quota may be dropping sharply around the end of january with hpcloud sunsetting on the 31st | 18:51 |
| openstackgerrit | Merged openstack-infra/project-config: Adds check-requirements job in networking-hyperv https://review.openstack.org/258992 | 18:52 |
| fungi | thingee: but that we're in the process of bringing new providers online to absorb the impact | 18:52 |
| fungi | thingee: so just noting that there may be slightly longer delays that usual to get job results at peak activity times | 18:53 |
| fungi | er, than usual | 18:53 |
| openstackgerrit | Merged openstack-infra/project-config: Publish openstackci docs https://review.openstack.org/259587 | 18:54 |
| *** amuller_afk is now known as amuller | 18:54 | |
| *** jcoufal has quit IRC | 18:54 | |
| *** sabeen has joined #openstack-infra | 18:55 | |
| *** Sukhdev has joined #openstack-infra | 18:55 | |
| asselin | anteaya, do you konw how to invoke the publish job? | 18:56 |
| thingee | fungi: got the previous mention of this on the ML to link for referene? | 18:57 |
| fungi | thingee: yeah, let me get you a link | 18:57 |
| anteaya | asselin: create a change to the document, after the job has been propogated out to all the jenkins' | 18:57 |
| anteaya | asselin: sorry merge a change to the document | 18:58 |
| clarkb | anteaya: watching worker20 I don't think this is doing anything terrible, logstash is using a sane amount of memroy and the workers are each using a bit more than they were before | 18:58 |
| clarkb | anteaya: there is sitll quite a bit of memroy available that is being used for caches | 18:58 |
| anteaya | clarkb: ack, thanks for taking a look | 18:58 |
| *** sabeen3 has joined #openstack-infra | 18:58 | |
| anteaya | wonderful | 18:58 |
| asselin | anteaya, you mean merge any change to puppet-openstackci? | 18:59 |
| anteaya | clarkb: I haven't been any help getting review attention on 247714, sorry | 18:59 |
| asselin | or it really needs to update the doc? | 18:59 |
| anteaya | asselin: I don't know | 18:59 |
| clarkb | anteaya: thats ok, though I amostly back at this point | 18:59 |
| clarkb | so will probably beat that drum next week | 18:59 |
| anteaya | asselin: I would aim for update the doc, but keep an eye on anychange | 18:59 |
| anteaya | clarkb: okey dokey | 18:59 |
| *** sabeen has quit IRC | 19:00 | |
| anteaya | clarkb: I need to start waving the debug the gate banner again | 19:00 |
| asselin | anteaya, just +A this simple change. see if it works: Phani Raju Yayavaram Narasimha | 19:00 |
| asselin | https://review.openstack.org/#/c/265035/ | 19:00 |
| anteaya | I've been holding off until logstash is all changed, so was going to deal with it then | 19:00 |
| anteaya | but can try to figure out something new to generate interest | 19:00 |
| *** Sukhdev has quit IRC | 19:01 | |
| anteaya | asselin: the job might not be out to the jenkins masters yet | 19:01 |
| anteaya | asselin: your patch might merge before the job is available | 19:01 |
| asselin | oh...right | 19:01 |
| anteaya | -2 will prevent it from merging | 19:01 |
| anteaya | adding to the commit message should get it out of the gate | 19:01 |
| asselin | i WIP -1 it | 19:02 |
| anteaya | asselin: that should prevent it from merging after the gate tests finish running | 19:02 |
| fungi | thingee: http://lists.openstack.org/pipermail/openstack-infra/2015-October/003309.html is probably the closest thing we have. not so much an announcement as a discussion back at the summit about it (since that was soon after they announced their timeline) | 19:02 |
| *** jsavak has quit IRC | 19:03 | |
| jeblair | not sure i'd link to that; it's a bit 'of the moment'. | 19:03 |
| fungi | right | 19:03 |
| anteaya | maybe link to the hp announcement: http://community.hpe.com/t5/Grounded-in-the-Cloud/A-new-model-to-deliver-public-cloud/ba-p/6804409#.VifwJJcnFz3 | 19:04 |
| fungi | thingee: i have it on the agenda to discuss again on tuesday at the meeting, so maybe better for next week's round-up | 19:04 |
| jeblair | and the docs about how to contribute http://docs.openstack.org/infra/system-config/contribute-cloud.html | 19:04 |
| anteaya | it says Therefore, we will sunset our HP Helion Public Cloud offering on January 31, 2016. in that post | 19:04 |
| *** tonytan4ever has quit IRC | 19:05 | |
| clarkb | pabelanger: for the fedora 21 slaves, there isn't anything critical relying on that right? | 19:05 |
| clarkb | pabelanger: I mean those tests basically check that we can run stuff on an EOL platform, there isn't much value there | 19:05 |
| *** ybathia has joined #openstack-infra | 19:06 | |
| *** dprince has quit IRC | 19:07 | |
| thingee | fungi: thanks | 19:07 |
| anteaya | SergeyLukjanov: how are you feeling about this patch? https://review.openstack.org/#/c/216275/ do you think we need to keep it open? | 19:08 |
| *** lucasagomes is now known as lucas-dinner | 19:08 | |
| pabelanger | clarkb: honestly, not sure. I mean, I can ask around if needed. Either way, I won't stand in the way if we remove them. Just adding a note to the review | 19:08 |
| *** thorst has joined #openstack-infra | 19:08 | |
| *** fawadkhaliq has quit IRC | 19:08 | |
| *** _nadya_ has quit IRC | 19:09 | |
| *** dkranz has quit IRC | 19:09 | |
| *** thorst has quit IRC | 19:10 | |
| *** thorst has joined #openstack-infra | 19:11 | |
| *** bpokorny has joined #openstack-infra | 19:11 | |
| jeblair | clarkb, zaro, fungi, mordred: i did some trials, and it looks like gc does do some things repack isn't doing for us. | 19:12 |
| *** otter768 has joined #openstack-infra | 19:13 | |
| fungi | i wonder whether git prune would also be useful | 19:14 |
| *** reaperhulk has left #openstack-infra | 19:14 | |
| jeblair | clarkb, zaro, fungi, mordred: nova is 7G on gerrit; git gc reduces it to 3.6G, git prune reduces it to 860M; git prune and git gc together reduce it to 850M; git prune and git gc and packing the refs reduce it to 394M. | 19:14 |
| jeblair | that last essentially means all the refs are in one file and all the objects are in another file. | 19:14 |
| fungi | jeblair: any better with pack -Ad? | 19:15 |
| *** mrmartin has quit IRC | 19:15 | |
| zaro | i believe puppet is doing -ad, correct? | 19:15 |
| jeblair | what's that? | 19:15 |
| *** ddieterly has quit IRC | 19:15 | |
| fungi | jeblair: should make a following prune more efficient since it moves unreachable objects out of the resulting pack | 19:16 |
| jeblair | fungi: do you mean 'git repack -afd'? | 19:16 |
| fungi | according to the manpage -A is like -a unless you add -d | 19:16 |
| zaro | here's the repack command: https://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/manifests/cron.pp#n13 | 19:17 |
| *** ddieterly has joined #openstack-infra | 19:17 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for manila dsvm horizon setup bug 1532300 https://review.openstack.org/265410 | 19:17 |
| openstack | bug 1532300 in Manila "manila dsvm jobs fail to setup horizon dashboard due to Permission Denied error on lock file" [Undecided,New] https://launchpad.net/bugs/1532300 | 19:17 |
| jeblair | fungi: i'll try "repack -Ad" followed by "prune" | 19:17 |
| fungi | and then it goes to extra lengths to get unreachable objects out of the repacking | 19:17 |
| *** aopincar has joined #openstack-infra | 19:17 | |
| *** otter768 has quit IRC | 19:18 | |
| *** thorst_ has joined #openstack-infra | 19:18 | |
| fungi | jeblair: there's also a `git prune-packed` referenced in the manpage | 19:18 |
| yolanda | hi pleia2, can you review https://review.openstack.org/256389 and https://review.openstack.org/260068 ? you already reviewed the one about bots, these are documentation patches | 19:18 |
| fungi | jeblair: which looks intruiging as well | 19:19 |
| jeblair | clarkb, zaro, fungi, mordred: at any rate -- those numbers sound great, but i'm not sure to what extent each is desirable -- it may be that having all of nova in one packfile might make things worse, especially considering all the refs that are only of historic interest. and having all the refs in one file means we can't edit them (though, after removing the zuul refs, hopefully we won't do that again) | 19:19 |
| zaro | fungi: more discussion, jrnieder is google guy, https://code.google.com/p/gerrit/issues/detail?id=3754 | 19:19 |
| *** bradjones has quit IRC | 19:19 | |
| jeblair | fungi: tried that -- had a minimal effect likely because of the "repack -afd" | 19:19 |
| fungi | got it | 19:19 |
| zaro | fungi: sorry took longer than i expected, i had to fix 3 bugs related to that issue. | 19:19 |
| *** bradjones has joined #openstack-infra | 19:19 | |
| *** bradjones has quit IRC | 19:19 | |
| *** bradjones has joined #openstack-infra | 19:19 | |
| anteaya | yolanda: I had a suggestion for 256389, did you see it? | 19:20 |
| *** thorst has quit IRC | 19:20 | |
| jeblair | clarkb, zaro, fungi, mordred: my feeling is that there is probably something to be gained by looking into gc and/or prune, but i think since we are in the position of running a public server based on this, we need to evaluate any changes very carefully based on that. | 19:21 |
| jeblair | clarkb, zaro, fungi, mordred: i think we should find a volunteer to really dig deeply into this and thoroughly evaluate the options and ramifications. | 19:21 |
| yolanda | no sorry, i just refreshed | 19:21 |
| zaro | jeblair: might be good to ask questions on the gerrit ML? | 19:22 |
| pleia2 | yolanda: yep, looking now | 19:22 |
| fungi | jeblair: agreed, most of those are focused on reducing backend storage, but very well could pose performance problems as a tradeoff | 19:23 |
| anteaya | yolanda: okay | 19:23 |
| clarkb | jeblair: that seems like areaosnable appraoch. | 19:23 |
| fungi | and yes, definitely sounds like an interesting research project which doesn't necessarily need a root admin involved | 19:24 |
| *** _diana_ has joined #openstack-infra | 19:24 | |
| clarkb | jeblair: what about for the web sessions maxAge? should we go ahead and just bump that to 7d? | 19:24 |
| *** trown|outttypeww is now known as trown | 19:24 | |
| fungi | i'm still dismayed that the session expiration is absolute from creation time and not relative to last use | 19:24 |
| jeblair | clarkb: that's probably the best approach | 19:24 |
| jeblair | fungi: yeah :( | 19:24 |
| fungi | so basically, your session takes a bath once a week whether it needs one or not | 19:25 |
| fungi | all jjb updates completed successfully except 02 (which hit a proxy error and had to be resumed) and 06 (which is still running and as noted needs a service restart soon) | 19:26 |
| fungi | continuing to keep tabs on those two stragglers | 19:27 |
| *** bradjones has quit IRC | 19:27 | |
| *** bradjones has joined #openstack-infra | 19:28 | |
| *** bradjones has quit IRC | 19:28 | |
| *** bradjones has joined #openstack-infra | 19:28 | |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: Revert "Add mimic to global requirements to use in tests" https://review.openstack.org/265416 | 19:28 |
| jeblair | fungi: the "git repack -Ad" resulted in a 2MB smaller repo than "git repack -afd" | 19:28 |
| zaro | clarkb, jeblair, fungi : once i'm free i can look deeper into what affects the git-gc and git-prune will have on gerrit performance. | 19:29 |
| clarkb | jeblair: should we make the nova repo as on review.o.o availalbe somewhere? | 19:29 |
| clarkb | jeblair: so that whoever works on it can compare like for like? its hard to get that just via clone | 19:29 |
| *** _diana_ has quit IRC | 19:29 | |
| jeblair | clarkb: yeah | 19:29 |
| *** ociuhandu has quit IRC | 19:29 | |
| jeblair | i'll do that and send an email | 19:29 |
| openstackgerrit | Merged openstack-infra/project-config: Mark swarm test as non-voting in Magnum https://review.openstack.org/264998 | 19:30 |
| zaro | what is the issue with having the repo in one packfile? | 19:30 |
| clarkb | zaro: it means that any time you need a ref you have to unpack that packfile | 19:30 |
| *** tonytan4ever has joined #openstack-infra | 19:30 | |
| clarkb | which may be huge if there is just one | 19:30 |
| jeblair | and maybe even transmit it under some circumstances? | 19:30 |
| jeblair | i'm not sure | 19:30 |
| clarkb | I think smart http makes a new packfile for all ops | 19:31 |
| clarkb | dumb http will transmit the entire packfile | 19:31 |
| jeblair | so maybe it's only the dumb http protocol where that's a worry | 19:31 |
| sdague | AJaeger done | 19:31 |
| jeblair | that may still be a worry for us though :) | 19:31 |
| openstackgerrit | Merged openstack-infra/project-config: Add solum-pythonclient to projects https://review.openstack.org/265088 | 19:31 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for manila dsvm horizon setup bug 1532300 https://review.openstack.org/265410 | 19:32 |
| openstack | bug 1532300 in Manila "manila dsvm jobs fail to setup horizon dashboard due to Permission Denied error on lock file" [Undecided,Confirmed] https://launchpad.net/bugs/1532300 | 19:32 |
| openstackgerrit | Merged openstack-infra/openstackid: Fix on Asymetric key thumbprint calculation: https://review.openstack.org/265397 | 19:34 |
| *** mrmartin has joined #openstack-infra | 19:35 | |
| fungi | electrical: oh, i just realized, i didn't use --flush-cache, should i have done for this update? it looks like in the end only reported about 15% of our job configs were updated | 19:35 |
| *** Sukhdev has joined #openstack-infra | 19:35 | |
| electrical | fungi: wrong auto complete? Lol | 19:36 |
| fungi | electrical: yep, sorry! | 19:36 |
| electrical | Haha np bud :) | 19:36 |
| anteaya | fungi: andreas did the last update on your patch to rename .log files to .log.txt files, do you want to review? https://review.openstack.org/#/c/203099/5 | 19:36 |
| fungi | looks like electrofelix dropped out of the channel | 19:36 |
| fungi | anyway, i suppose the concern was mainly that the change could have resulted in a very lengthy jjb update (which it did), but that the addition is a no-op so as long as the cache has updated for anything it thought was out of sync, we should be fine | 19:37 |
| *** Sukhdev has quit IRC | 19:37 | |
| fungi | anteaya: he mentioned it earlier--i'll take a look in a bit | 19:37 |
| *** Sukhdev has joined #openstack-infra | 19:38 | |
| clarkb | jeblair: what we really need is a `git optimize` which reduces repo size below some threshold while still contrucitng the repo in a way friendly for network transmission | 19:39 |
| jeblair | fungi: that's my thinking. | 19:39 |
| anteaya | fungi: oh sorry I didn't mean to bother you, thank you | 19:40 |
| fungi | jenkins06 finally tripped on a 502 from the api as well, so i've resumed it now | 19:40 |
| openstackgerrit | Merged openstack-infra/project-config: Add python-ironicclient to PROJECTS https://review.openstack.org/263335 | 19:41 |
| *** alkhodos has joined #openstack-infra | 19:42 | |
| zaro | clarkb: you think there's a way to optimize the repo without putting it all in one file? | 19:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Switch to templating for nodepool rackspace https://review.openstack.org/264994 | 19:43 |
| *** fawadkhaliq has joined #openstack-infra | 19:43 | |
| pabelanger | jeblair: ^ That should be what you were asking for | 19:44 |
| pabelanger | but still lowercase | 19:44 |
| jeblair | pabelanger: oh well, thanks :) | 19:45 |
| *** treaki_ has quit IRC | 19:46 | |
| *** yolanda has quit IRC | 19:46 | |
| *** eil397 has joined #openstack-infra | 19:46 | |
| *** bgmccollum has joined #openstack-infra | 19:48 | |
| EmilienM | hello, can someone point me to how to have os-loganalyze in our puppet openstack jobs? | 19:48 |
| *** Sukhdev has quit IRC | 19:48 | |
| EmilienM | I haven't found much documentation about that | 19:48 |
| jeblair | the tar of the nova repo is 5.1 while the directory is 7.0G. | 19:49 |
| jeblair | i'm guessing our 100k refs take up a couple of G of direntries. | 19:49 |
| *** ujuc has joined #openstack-infra | 19:50 | |
| openstackgerrit | Merged openstack-infra/system-config: Add instructions on how to troubleshoot bots https://review.openstack.org/260068 | 19:52 |
| jeblair | clarkb, fungi: http://tarballs.openstack.org/ci/ has nova.git.tar.bz2 from May 11 | 19:52 |
| jeblair | maybe something else related to our gerrit upgrade rollback? | 19:53 |
| jeblair | any ojebctions to me just overwriting that? | 19:53 |
| fungi | jeblair: yes, we put a copy there so others could help us figure out why the repo was "going bad" | 19:53 |
| fungi | it's fair game to clean up now | 19:53 |
| fungi | or replace/refresh, sure | 19:53 |
| *** yolanda has joined #openstack-infra | 19:54 | |
| jeblair | at least we're consistent with hosting locations :) | 19:54 |
| jeblair | while we're waiting for bzip: https://etherpad.openstack.org/p/1GtOYrmIGN | 19:55 |
| *** [1]Thelo has joined #openstack-infra | 19:55 | |
| *** ujuc has quit IRC | 19:55 | |
| pabelanger | jeblair: looks like depends-on for grafyaml not working as expected: https://jenkins07.openstack.org/job/gate-project-config-grafyaml/13/consoleText | 19:55 |
| jeblair | pabelanger: what change is that? | 19:56 |
| clarkb | jeblair: etherpad message lgtm | 19:56 |
| pabelanger | jeblair: 264994 | 19:56 |
| *** jsavak has joined #openstack-infra | 19:57 | |
| *** Thelo has quit IRC | 19:57 | |
| *** [1]Thelo is now known as Thelo | 19:57 | |
| fungi | jeblair: lgtm with one tiny clarification | 19:58 |
| fungi | jeblair: also, xz is what all the hipsters use these days. bzip2 is so last decade | 19:59 |
| *** _nadya_ has joined #openstack-infra | 19:59 | |
| jeblair | fungi: don't make me rar it. | 19:59 |
| fungi | bwahahaha | 19:59 |
| anteaya | email looks good to me | 20:00 |
| clarkb | ugh rar | 20:00 |
| jroll | please make it a split rar | 20:00 |
| clarkb | and host it on a warez site | 20:00 |
| jroll | ^^ | 20:00 |
| jeblair | jroll: i'll post it to usenet for you | 20:00 |
| jeblair | clarkb: like ftp.gnu.org? | 20:00 |
| jroll | your download will begin in 27 seconds... | 20:00 |
| Shrews | newsgroups ftw | 20:00 |
| fungi | jroll: i'll have uuencode split it before posting to the newsgroup instead | 20:00 |
| fungi | no need for a split archive | 20:01 |
| fungi | uudecode can stich just fine | 20:01 |
| fungi | er, stitch | 20:01 |
| jeblair | (apparently with twitter poised to accept >140 chars, this may be our last opportunity to download split tweets) | 20:02 |
| * jeblair really hopes that downloading split archives from tweets that say "[1/165478]" is a thing | 20:03 | |
| clarkb | jeblair: you cna base64 encode the tarball and post it to twiiter 140 bytes at a time | 20:03 |
| *** dkranz has joined #openstack-infra | 20:04 | |
| jeblair | pabelanger: i agree that didn't seem to work. the params look correct; i'm not sure why it didn't work. | 20:05 |
| *** Sukhdev has joined #openstack-infra | 20:06 | |
| *** EricGonczer_ has joined #openstack-infra | 20:07 | |
| pabelanger | jeblair: the sha1 zuul is using, is not correct | 20:07 |
| jeblair | doh! | 20:07 |
| pabelanger | jeblair: that appears to be current master | 20:07 |
| jeblair | it's space separated, not comma separated | 20:07 |
| pabelanger | ah | 20:07 |
| jeblair | fixing | 20:07 |
| pleia2 | yolanda: I really appreciate these documentation updates, inspired me to through my own notes to figure out what I'm storing locally that should be shared | 20:07 |
| anteaya | EmilienM: to be honest with you, I don't know the answer, sdague wrote most of it, perhaps he can suggest a pointer to have your logs use it | 20:08 |
| pleia2 | seeking lunch now | 20:08 |
| *** amuller is now known as amuller_afk | 20:08 | |
| anteaya | pleia2: may you find | 20:08 |
| clarkb | error lunch not found | 20:08 |
| clarkb | (this is my current isutation) | 20:08 |
| anteaya | clarkb: may you also find | 20:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Correct tox env override https://review.openstack.org/265427 | 20:08 |
| jeblair | anteaya, pabelanger: ^ | 20:08 |
| EmilienM | anteaya: clarkb is already helping me, thanks! | 20:09 |
| jeblair | pabelanger: mind if we merge that and recheck to make sure it works? | 20:09 |
| pabelanger | jeblair: WFM | 20:09 |
| anteaya | EmilienM: oh great, sorry I missed that | 20:09 |
| *** kzaitsev_mb has joined #openstack-infra | 20:09 | |
| anteaya | EmilienM: do share once you figure it out so I may learn, thank you | 20:10 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone-only uwsgi job https://review.openstack.org/264991 | 20:10 |
| EmilienM | anteaya: sure | 20:10 |
| anteaya | EmilienM: thanks | 20:10 |
| clarkb | fungi: there are a bunch of ready nodes on jenkins06 according to nodepool, we should maybe delete those to shift the load to othre masters? | 20:10 |
| clarkb | fungi: or do you plan on restarting soon? | 20:10 |
| yolanda | pleia2 thanks. In downstream we have that system to document every common task we have, and i missed that a bit upstream. So if you can add more of these bits, it will be really good | 20:10 |
| fungi | clarkb: great point, i forgot to blow away nodepool ready nodes for jenkins06. i'll do that asap | 20:11 |
| fungi | they're deleting now | 20:12 |
| fungi | thanks for catching that | 20:12 |
| *** EricGonc_ has joined #openstack-infra | 20:15 | |
| *** cbader has joined #openstack-infra | 20:16 | |
| *** fawadkhaliq has quit IRC | 20:17 | |
| *** fawadkhaliq has joined #openstack-infra | 20:18 | |
| clarkb | fungi: no problem I had just noticed the green portion of the status graph and saw that nodepool had a bunch of ready nodes on 06 | 20:18 |
| *** alivigni has quit IRC | 20:18 | |
| *** EricGonczer_ has quit IRC | 20:19 | |
| fungi | okay, jenkins06 updates completed successfully finally | 20:20 |
| *** jsavak has quit IRC | 20:21 | |
| fungi | it's due for a scheduled restart later today anyway, right? | 20:21 |
| docaedo | This is probably obvious, but where would I look to see results of periodic jobs? I realize I only know about jobs as the relate to specific changes, so not sure where to go :) | 20:21 |
| fungi | docaedo: http://logs.openstack.org/periodic/ | 20:21 |
| *** piet has joined #openstack-infra | 20:21 | |
| *** jsavak has joined #openstack-infra | 20:21 | |
| *** yonglihe has quit IRC | 20:21 | |
| docaedo | fungi: obviously ;) Thanks! | 20:21 |
| *** fawadkhaliq has quit IRC | 20:22 | |
| fungi | docaedo: well, it's embarrassingly non-obvious, but there are some log dashboard efforts underway to hopefully fix the discoverability issue in coming cycles | 20:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: clean up OVH graphs https://review.openstack.org/265431 | 20:24 |
| *** esp has joined #openstack-infra | 20:24 | |
| zaro | esp: check here: https://git.openstack.org/cgit/openstack/stackalytics/tree/etc/default_data.json | 20:24 |
| esp | nice! thx zaro | 20:24 |
| *** aopincar has quit IRC | 20:24 | |
| *** nelsnels_ has quit IRC | 20:25 | |
| docaedo | fungi: well, there's so many parts and pieces to the infra, making it all obvious is a tall order. Feel like I learn a little bit more about it every day | 20:25 |
| *** piet has quit IRC | 20:25 | |
| fungi | docaedo: me too! | 20:26 |
| fungi | clarkb: i think jenkins02 may also be overdue. i keep getting 502 errors trying to get the jjb update there to finish | 20:27 |
| fungi | i'm going to put it in prepare for shutdown as well (and delete the ready nodes for it from nodepool this time) | 20:27 |
| clarkb | I did check and it appeared they all successfully restarted lsat friday | 20:27 |
| clarkb | so a week may be too long of an internval | 20:27 |
| fungi | i'm checking the thread details on it now | 20:27 |
| *** hdd has quit IRC | 20:28 | |
| fungi | yep, lots of blocked sshlauncher.afterdisconnect threads | 20:29 |
| anteaya | I'm about to approve the f21 is EOL patch at 264378 if anyone else wants to review it quickly | 20:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: update bluebox dashboard like OVH https://review.openstack.org/265432 | 20:29 |
| openstackgerrit | Merged openstack-infra/project-config: Makes non-admin job to use pre provisioned cred https://review.openstack.org/263976 | 20:31 |
| fungi | that looping swift job seems to be tickling an odd situation in the scp publisher plugin https://jenkins01.openstack.org/job/gate-swift-python27/4944/consoleFull | 20:31 |
| *** salv-orlando has quit IRC | 20:32 | |
| *** salv-orlando has joined #openstack-infra | 20:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix cleanup of check-osc-plugins https://review.openstack.org/265085 | 20:34 |
| anteaya | can I get one more project-config core reviewer to agree that Sam-I-Am should have irc ops by +3'ing 265036 | 20:34 |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: enable test with ODL release of beryllium-snapshot https://review.openstack.org/265437 | 20:37 |
| *** Daviey has joined #openstack-infra | 20:38 | |
| *** ddieterly has quit IRC | 20:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Fedora 21 is EOL stop using it https://review.openstack.org/264378 | 20:40 |
| *** ddieterly has joined #openstack-infra | 20:40 | |
| *** jcoufal has joined #openstack-infra | 20:41 | |
| *** nelsnelson has joined #openstack-infra | 20:42 | |
| *** ddieterly has quit IRC | 20:42 | |
| *** jcoufal has quit IRC | 20:43 | |
| *** jcoufal has joined #openstack-infra | 20:44 | |
| *** kzaitsev_mb has quit IRC | 20:45 | |
| *** mrmartin has quit IRC | 20:46 | |
| *** sridhar_ram has quit IRC | 20:47 | |
| *** jcoufal has quit IRC | 20:49 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics https://review.openstack.org/263971 | 20:50 |
| *** ujuc has joined #openstack-infra | 20:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: update internap dashboard like OVH https://review.openstack.org/265438 | 20:51 |
| fungi | okay, jenkins02 finally finished its jjb updates, so that's the last of them | 20:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: update rackspace dashboard like OVH https://review.openstack.org/265439 | 20:52 |
| fungi | infra-root: all manual jjb updating has completed, with a confirmed successful result on each production jenkins master | 20:52 |
| jeblair | fungi: thanks! | 20:52 |
| fungi | no sweat | 20:53 |
| openstackgerrit | Merged openstack-infra/project-config: Correct tox env override https://review.openstack.org/265427 | 20:55 |
| jeblair | pabelanger: next zuul reload we should be able to recheck ^ | 20:55 |
| openstackgerrit | Merged openstack-infra/project-config: Removed testjobs and corresponding chef-jobs https://review.openstack.org/264130 | 20:56 |
| *** ujuc has quit IRC | 20:57 | |
| *** claudiub has quit IRC | 20:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: update bluebox dashboard like OVH https://review.openstack.org/265432 | 20:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: update internap dashboard like OVH https://review.openstack.org/265438 | 20:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: update rackspace dashboard like OVH https://review.openstack.org/265439 | 20:58 |
| jeblair | okay, those are ready for review if anyone wants to double check me ^ | 20:58 |
| jeblair | pabelanger: we might need to rebase your rax template change, sorry, but i think it's worth getting them all tidied up first. | 20:59 |
| *** daemontool has joined #openstack-infra | 20:59 | |
| openstackgerrit | Merged openstack-infra/puppet-askbot: Fix broken Pillow pip dependency https://review.openstack.org/265231 | 21:00 |
| pc_m | Hi infra folks! I have a question... | 21:00 |
| pc_m | I added constraints based jobs for neutron-vpnaas repo as experimental and have tested them. | 21:01 |
| pc_m | Should I now add them to check queue as non-voting, or add to check and gate queue as voting? | 21:01 |
| *** rfolco has quit IRC | 21:01 | |
| pleia2 | pc_m: that's a decision the team has to make, if the neutron-vpnaas folks think it's reliable enough to gate on, you can skip the non-voting step | 21:02 |
| *** bharathm has quit IRC | 21:02 | |
| *** Sukhdev has quit IRC | 21:02 | |
| pleia2 | pc_m: but many teams like to see how it goes for non-voting for a bit first | 21:02 |
| anteaya | pc_m: I would advocate for moving to check non-voting myself | 21:03 |
| pc_m | pleia2: anteaya: OK. Will do non-voting. One is a post (cover-constraints). Just hold on that, until voting? | 21:03 |
| anteaya | pc_m: I don't understand what you mean | 21:04 |
| anteaya | what is the name of the repo? | 21:04 |
| pc_m | anteaya: One of the new experimental jobs, is neutron-vpnaas-cover-constraints, which would replace the neutron-vpnaas-cover in the post queue. | 21:04 |
| pc_m | neutron-vpnaas | 21:05 |
| *** eharney has quit IRC | 21:06 | |
| anteaya | pc_m: yeah okay that post job, has that been exercised in the experimental queue at all? | 21:06 |
| pc_m | anteaya: yes | 21:06 |
| anteaya | it has | 21:06 |
| anteaya | oh right | 21:06 |
| *** hdd has joined #openstack-infra | 21:06 | |
| anteaya | okay so for everything else move to check non-voting | 21:06 |
| pc_m | anteaya: I've verified all the jobs in experimental queue and want to move forward. | 21:07 |
| anteaya | we don't have a non-voting situation for post jobs | 21:07 |
| anteaya | right | 21:07 |
| anteaya | that is a coverage job, I would say move that one job to the post queue | 21:07 |
| *** harlowja_at_home has joined #openstack-infra | 21:07 | |
| fungi | also, post jobs don't break merging since they happen "post"-merge | 21:07 |
| * anteaya nods | 21:07 | |
| pc_m | anteaya: fungi: Gotcha. Thanks. | 21:08 |
| fungi | and as docaedo discovered, blindly recommending running post jobs in the experimental pipeline doesn't always fit | 21:08 |
| fungi | in his case, a job which uploads the results of processing a merge as a proposed change in gerrit | 21:08 |
| *** _nadya_ has quit IRC | 21:08 | |
| anteaya | fungi: sorry I missed that, which job was that? | 21:09 |
| *** aopincar has joined #openstack-infra | 21:09 | |
| fungi | not really safe to try to make into an experimental job, which runs on unmerged changes (among other problems, it ends up with a divergence from the branch the job assumes it's basing new changes on) | 21:09 |
| docaedo | anteaya: this is the latest PR related to it, it was the dead-link checker script https://review.openstack.org/#/c/264978/ | 21:09 |
| *** amuller_afk is now known as amuller | 21:10 | |
| anteaya | docaedo: thanks | 21:10 |
| *** sridhar_ram has joined #openstack-infra | 21:10 | |
| docaedo | anteaya: no problem! | 21:10 |
| fungi | experimental is a change-based pipeline like check, gate and silent | 21:11 |
| fungi | it's often not suitable for "trying out" jobs intended to run in ref-based pipelines like post, pre-release, release or tag | 21:11 |
| fungi | or periodic | 21:12 |
| fungi | which is neither change-based nor ref-based | 21:12 |
| anteaya | ack, thanks | 21:13 |
| fungi | you're welcome! | 21:13 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/os-loganalyze: filter: add more coverage for puppet logs https://review.openstack.org/265447 | 21:13 |
| *** aopincar has quit IRC | 21:13 | |
| fungi | now to get back to making my owner-listing script increasingly inefficient | 21:14 |
| EmilienM | clarkb: I finally deciced to go that way ^ because log files not managed the same way in packaging than in devstack | 21:14 |
| *** otter768 has joined #openstack-infra | 21:14 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:17 | |
| *** edmondsw has quit IRC | 21:17 | |
| jeblair | does anyone object to me self-approving changes to the grafana dashboards? | 21:17 |
| clarkb | EmilienM: thats probably something we should work to make consistent by default as well | 21:18 |
| clarkb | jeblair: I don't | 21:18 |
| *** annegentle is now known as notsogentle | 21:18 | |
| Shrews | jeblair: if you get bored over the weekend, i'm proposing some new functionality to shade i'd love feedback on: https://review.openstack.org/264869 nothing urgent though | 21:18 |
| *** otter768 has quit IRC | 21:19 | |
| *** ganso has quit IRC | 21:19 | |
| anteaya | jeblair: I do not object | 21:19 |
| jeblair | Shrews: cool, we use functionality like that in both our launch script and in nodepool -- generally implemented as "sort the flavors by size and use the first that has at least X gb of ram" | 21:20 |
| jeblair | Shrews: we probably wouldn't use a max bound, but i could see having the ability to have one being a nice safety check, especially if you actually get billed for cloud usage. ;) | 21:21 |
| Shrews | ++ | 21:21 |
| clarkb | Shrews: might be good to at least confirm with nova that the scale/format/types of all the serachable flavors are consistent | 21:21 |
| Shrews | jeblair: yeah, that's there for balance, really | 21:21 |
| clarkb | I don't think we have had probablems with the RAM numbers | 21:21 |
| jeblair | clarkb: agreed, i don't remember any problems with ram. somehow i feel its only a matter of time :) | 21:22 |
| Shrews | clarkb: yeah, this is sort of limited to just integer fields (or convertable to integer), atm | 21:22 |
| clarkb | I just imagine that it could be possible for 8GB, 8192, and 8192MB to all be valid | 21:22 |
| clarkb | unless nova is explicit about it | 21:23 |
| Shrews | but, this is not supposed to be limited to flavors (though i can't come up with a better use case just now, either) | 21:23 |
| *** jsavak has quit IRC | 21:24 | |
| clarkb | sorting is always good fun | 21:26 |
| *** daemontool has quit IRC | 21:27 | |
| jeblair | wait, we don't have to implement our own quicksort in nodepool? | 21:27 |
| jeblair | pfft. there's my csc degree wasted. | 21:28 |
| openstackgerrit | Merged openstack-infra/project-config: Grafana: clean up OVH graphs https://review.openstack.org/265431 | 21:28 |
| Shrews | jeblair: well, my patch doesn't sort, so i could steal your codez and sort the datas :) | 21:28 |
| *** tlian has quit IRC | 21:28 | |
| Shrews | that may actually be useful... even as a separate review | 21:29 |
| openstackgerrit | Merged openstack-infra/project-config: Grafana: update bluebox dashboard like OVH https://review.openstack.org/265432 | 21:29 |
| openstackgerrit | Merged openstack-infra/project-config: Grafana: update internap dashboard like OVH https://review.openstack.org/265438 | 21:29 |
| openstackgerrit | Merged openstack-infra/project-config: Grafana: update rackspace dashboard like OVH https://review.openstack.org/265439 | 21:29 |
| *** tlian has joined #openstack-infra | 21:29 | |
| jeblair | well, i just called sort() so i probably failed the assignment. | 21:29 |
| clarkb | you'll already need it for min/max right? | 21:29 |
| fungi | jeblair: no objection to self-approving grafana dashboard additions/modifications as they're still sort of under construction anyway and non-impacting if they go terribly wrong | 21:29 |
| clarkb | I guess not | 21:29 |
| Shrews | clarkb: no. straight linear search | 21:29 |
| clarkb | difference between n and n log n | 21:30 |
| clarkb | ya | 21:30 |
| clarkb | next you'll be using a tree to log n | 21:30 |
| jeblair | you know what would be really cool in grafana -- if you could gang graphs together and force them to have the same y scale. | 21:31 |
| jeblair | (but still have that scale dynamic) | 21:31 |
| *** bharathm has joined #openstack-infra | 21:32 | |
| *** daemontool has joined #openstack-infra | 21:32 | |
| jeblair | oh there's a light theme: http://grafana.openstack.org/dashboard/db/nodepool-rackspace?theme=light | 21:35 |
| anteaya | jeblair: oh I like that | 21:36 |
| *** MarkAtwood has quit IRC | 21:38 | |
| *** ociuhandu has joined #openstack-infra | 21:38 | |
| *** devkulkarni has left #openstack-infra | 21:38 | |
| *** eharney has joined #openstack-infra | 21:39 | |
| jeblair | okay... why are the sparklines for node counts not integers? | 21:41 |
| jeblair | (ovh is apparently deleting 1.6 nodes right now) | 21:41 |
| jeblair | graphite itself says the much more sensible 1.0 for that query | 21:42 |
| jeblair | so it's a grafana mutation | 21:42 |
| *** rguillebert has quit IRC | 21:47 | |
| jeblair | oh. maxDataPoints. | 21:49 |
| jeblair | grafana is apparently not the tool to use if you are interested in accurate data | 21:50 |
| jeblair | by default it is more concerned that data that looks good be displayed quickly. | 21:50 |
| jeblair | we could increase maxdatapoints, but that may be excessive. | 21:59 |
| *** esker has joined #openstack-infra | 21:59 | |
| *** esker has quit IRC | 21:59 | |
| *** jsavak has joined #openstack-infra | 22:02 | |
| clarkb | jeblair: our data needs skinny jeans | 22:03 |
| fungi | also, beard | 22:03 |
| *** esker has joined #openstack-infra | 22:03 | |
| *** rlandy has quit IRC | 22:06 | |
| *** jsavak has quit IRC | 22:06 | |
| *** mriedem has quit IRC | 22:07 | |
| *** jsavak has joined #openstack-infra | 22:08 | |
| *** pradk has quit IRC | 22:08 | |
| *** salv-orl_ has joined #openstack-infra | 22:09 | |
| *** salv-orlando has quit IRC | 22:12 | |
| *** salv-orl_ has quit IRC | 22:12 | |
| *** salv-orlando has joined #openstack-infra | 22:12 | |
| *** kgiusti has left #openstack-infra | 22:14 | |
| *** ujuc has joined #openstack-infra | 22:15 | |
| amuller | armax: mtreinish: Finished mapping out all modifications to Neutron API tests since the fork: https://etherpad.openstack.org/p/neutron-tempest-defork-patches-since-initial-sync | 22:15 |
| amuller | linked to it from the main etherpad | 22:15 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Add a CONTRIBUTING.rst document https://review.openstack.org/265035 | 22:16 |
| *** nelsnelson has quit IRC | 22:16 | |
| *** trown is now known as trown|outttypeww | 22:16 | |
| *** esker has quit IRC | 22:17 | |
| * asselin watches first puppet-openstackci-infra-docs | 22:17 | |
| *** leakypipes has quit IRC | 22:20 | |
| anteaya | asselin: go go go | 22:21 |
| *** rossella_s has quit IRC | 22:22 | |
| *** peristeri has quit IRC | 22:22 | |
| *** esker has joined #openstack-infra | 22:22 | |
| *** rossella_s has joined #openstack-infra | 22:22 | |
| *** ujuc has quit IRC | 22:25 | |
| *** esker has quit IRC | 22:27 | |
| clarkb | gerrit GC has fallen off, not sure if related to firday or maybe the lack of thousands of comments comming in helped it | 22:27 |
| *** dkranz has quit IRC | 22:29 | |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: enable test with ODL release of beryllium-snapshot https://review.openstack.org/265437 | 22:31 |
| *** FallenPegasus has joined #openstack-infra | 22:31 | |
| alkhodos | Hi guys, I have a problem with CI: it connects to gerrit stream, adds events to queue, runs everything and posts results to web server - all good. But for some reason it stopped voting on gerrit. Was fine before. I can't figure out what is causing this issue, if you have any idea where should I like - I would really appreciate it. Thanks in advance. | 22:32 |
| *** amuller has quit IRC | 22:32 | |
| clarkb | alkhodos: it is probably one of two things, either your zuul layout file doesn't instruct it to vote anymore, or the project(s) you are posting to removed your accounts voting rights | 22:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/grafyaml: Add support for seriesOverrides https://review.openstack.org/265458 | 22:34 |
| *** Sukhdev has joined #openstack-infra | 22:35 | |
| *** erlon has quit IRC | 22:36 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: Add max server line to node graph https://review.openstack.org/265459 | 22:36 |
| jeblair | if anyone wants to review those 2 changes, they're pretty straightforward | 22:38 |
| jeblair | and we'll end up with a cool line on the test nodes graph corresponding to max_servers | 22:39 |
| *** yaume has joined #openstack-infra | 22:39 | |
| alkhodos | clarkb: Thanks! will look into those things | 22:41 |
| *** tlian has quit IRC | 22:42 | |
| anteaya | jeblair: alias is required in the overrides? | 22:43 |
| jeblair | anteaya: yeah, it's how you specify which thing you are overriding | 22:43 |
| anteaya | ah | 22:43 |
| jeblair | anteaya: so if you look at my usuage in the second patch, you'll see i'm saying "don't stack 'max'" | 22:43 |
| anteaya | I will look | 22:44 |
| anteaya | was just looking at the E501 line too long error that pep8 was offering | 22:44 |
| jeblair | <sigh> | 22:44 |
| anteaya | yeah | 22:44 |
| *** bharathm has quit IRC | 22:45 | |
| *** flepied has joined #openstack-infra | 22:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/grafyaml: Add support for seriesOverrides https://review.openstack.org/265458 | 22:48 |
| jeblair | updated the commit msg too | 22:48 |
| alkhodos | clarkb: sorry for a stupid question, but how can I see if my voting rights were removed? | 22:48 |
| clarkb | alkhodos: check the $project-ci group membership in gerrit. People -> list groups -> search -> membership | 22:49 |
| clarkb | you aren't there you shouldn't have voting rights | 22:49 |
| *** bharathm has joined #openstack-infra | 22:50 | |
| *** regXboi has quit IRC | 22:51 | |
| asselin | #success OpenStack Third Party CI documentation published http://docs.openstack.org/infra/openstackci/ | 22:51 |
| openstackstatus | asselin: Added success to Success page | 22:51 |
| *** bharathm has quit IRC | 22:52 | |
| pabelanger | jeblair: will test your suggestions later tonight | 22:53 |
| *** bharathm has joined #openstack-infra | 22:54 | |
| Thelo | Hi all, I have a problem with jenkins and nodepool, to do simple jenkins fails to ssh a slave ([01/09/16 06:43:39] SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 33 ms) however if I try to ssh the slave manually using the user jenkins and the key defined in the security in jekins it works fine, does anyone have clue of what could be wrong? | 22:55 |
| clarkb | Thelo: is that IP address reachable from nodepool? | 22:56 |
| clarkb | or is it jenkins failing to ssh? | 22:56 |
| *** yaume has quit IRC | 22:56 | |
| jeblair | pabelanger: thanks.. i added some more fields to graph: https://review.openstack.org/265458 and https://review.openstack.org/265459 | 22:56 |
| *** ddieterly has joined #openstack-infra | 22:56 | |
| clarkb | either way I would double check connectivity between that path if you haven't already testing ssh | 22:56 |
| *** jsavak has quit IRC | 22:58 | |
| clarkb | Thelo: another thing to check if it is between jenkins and the slave is that the credentials are configured properly on the slave | 22:58 |
| jeblair | pabelanger: and here's something to think about: the sparklines show the current value as a big number, but because they use "maxDataPoints=100" in the graphite query, graphite aggregates the data before returning it using the mean. | 22:58 |
| Thelo | clarkb: yes from nodepool I can conneect manually to the slave | 22:58 |
| jeblair | pabelanger: that means that it ends up as a rational number rather than integer, which doesn't make sense ("Deleting 1.8 nodes" ?) | 22:58 |
| Thelo | clarkb: only jenkins fails to ssh it | 22:59 |
| asselin | Thelo, check the jenkins log file | 22:59 |
| clarkb | Thelo: ok, in that case I would double check the slave credentials that are configured in Jenkins | 22:59 |
| jeblair | pabelanger: we could increase maxDataPoints (we can even set it to 'auto' which is just equal to the width of the graph in pixels) -- that's usually high enough that it doesn't need to consolidate | 22:59 |
| jeblair | pabelanger: but it also means shipping more data around and in some quick testing, does slow things down a little bit (not much) | 23:00 |
| jeblair | pabelanger: we could also add the 'consolidateBy' function to the query, but i'm not sure any of the other consolidation functions make much more sense -- the best would be 'max' | 23:00 |
| *** thorst_ has quit IRC | 23:00 | |
| *** thorst has joined #openstack-infra | 23:01 | |
| jeblair | pabelanger: but that's still inaccurate -- it means sometimes the numbers will lag a bit | 23:01 |
| anteaya | asselin: congratulations | 23:01 |
| asselin | anteaya, thanks :) | 23:02 |
| Thelo | clarkb: the log file only shows that "([01/09/16 06:43:39] SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 33 ms)" I am double checking the slave credential configuration | 23:02 |
| *** ZZelle has joined #openstack-infra | 23:04 | |
| asselin | Thelo, double check the ssh private key in /var/lib/jenkins/.ssh/id_rsa | 23:05 |
| asselin | try to login as jenkins user using that key file | 23:05 |
| *** thorst has quit IRC | 23:09 | |
| Thelo | asselin: I just did and I could ssh my node ( using jenkins user) | 23:12 |
| *** dimtruck is now known as zz_dimtruck | 23:12 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:13 | |
| *** notsogentle has quit IRC | 23:13 | |
| *** notsogentle has joined #openstack-infra | 23:13 | |
| asselin | Thelo, anything in the jenkins log file? | 23:14 |
| *** otter768 has joined #openstack-infra | 23:15 | |
| Thelo | INFO: Attempting to reconnect d-p-c-local_01-11 | 23:16 |
| Thelo | [01/09/16 07:07:39] SSH Launch of d-p-c-local_01-11 on 10.203.19.227 failed in 38 ms | 23:16 |
| Thelo | asselin: that message | 23:17 |
| *** tonytan4ever has quit IRC | 23:19 | |
| *** amitgandhinz has quit IRC | 23:19 | |
| *** otter768 has quit IRC | 23:20 | |
| *** _tzn has quit IRC | 23:21 | |
| *** vgridnev has quit IRC | 23:22 | |
| Thelo | asselin: hudson.AbortException: Cannot find SSH User credentials with id: 2f541bcd-a553-4614-9d61-ddc7d121d4aa do you know what this id refer to? | 23:22 |
| *** tzn has joined #openstack-infra | 23:23 | |
| asselin | Thelo, probably a credentials id....it's part of the docs | 23:23 |
| *** sridhar_ram1 has joined #openstack-infra | 23:26 | |
| *** notsogentle has quit IRC | 23:27 | |
| *** sridhar_ram has quit IRC | 23:28 | |
| *** sridhar_ram1 has quit IRC | 23:31 | |
| *** sridhar_ram has joined #openstack-infra | 23:32 | |
| *** tzn has quit IRC | 23:35 | |
| *** tzn has joined #openstack-infra | 23:43 | |
| Thelo | aselin: could it be something like the node ask the question "are you sure that you want to connect " and jenkins does not know how to anser to that? | 23:44 |
| asselin | Thelo, when you checked ssh, did you do as as the jenkins user? sudo su - jenkins | 23:46 |
| *** gordc has quit IRC | 23:47 | |
| *** Sukhdev has quit IRC | 23:49 | |
| *** mmedvede has joined #openstack-infra | 23:49 | |
| *** FallenPegasus has quit IRC | 23:50 | |
| *** Qiming has joined #openstack-infra | 23:54 | |
| Thelo | asselin: http://paste.openstack.org/show/483352/ | 23:55 |
| *** tzn has quit IRC | 23:56 | |
| *** Sukhdev has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!