| clarkb | jogo: yup and the interrupting deals with that (not necessarily well but it does make people aware and care) | 00:00 |
|---|---|---|
| lifeless | I don't think infra would be held in the same lights-on respect if its approach is to hold everything hostage until thing X is done | 00:00 |
| *** dannywilson has quit IRC | 00:00 | |
| jogo | clarkb: so as a nova-core this is the first time I am hearing about this ipv6 issue | 00:00 |
| clarkb | jogo: https://review.openstack.org/#/c/168701/ it just merged today | 00:00 |
| jogo | clarkb lifeless: I am not interested in having a theortical conversation about how to handle this aspect of dependency management. We can experiment once the bits are in place and just see | 00:01 |
| clarkb | jogo: but missed the kilo train | 00:01 |
| jogo | clarkb: oh I reviewed that this morning :) | 00:01 |
| jogo | clarkb: tell me more about the general issue | 00:01 |
| jogo | clarkb: we want to make devstack have ipv6 enabled in the gate by default? | 00:01 |
| jogo | what other bits are needed to test it etc. | 00:02 |
| clarkb | jogo: general issue is nova in < kilo will attach floating IPs to ipv6 private addrs | 00:02 |
| clarkb | jogo: so kilo ipv6 doesn't work with neutron and floating Ips | 00:02 |
| jogo | we can backport the fix right away | 00:02 |
| clarkb | jogo: this means we can't turn on tests for it (because they fail) and it is a bad user expereince | 00:02 |
| clarkb | jogo: or we could've met the problem head on and just fixed it | 00:02 |
| *** achanda has quit IRC | 00:02 | |
| *** zz_dimtruck is now known as dimtruck | 00:03 | |
| *** xyang1 has quit IRC | 00:03 | |
| asselin_ | is etherpad down? | 00:03 |
| clarkb | asselin_: its up for me | 00:03 |
| asselin_ | nevermind, just loaded...took a while | 00:03 |
| clarkb | asselin_: I think we do DB backups at 0000UTC | 00:04 |
| clarkb | which just happened | 00:04 |
| jogo | clarkb: so patch landed what is missing in tempest conf to turn it on? | 00:04 |
| asselin_ | ok I see | 00:04 |
| lifeless | clarkb: I understand your frustration, and as you know I'm a big fan of reducing concurrency and churn on project effort, | 00:04 |
| clarkb | jogo: we need to enable it by default in tempest now, that change has been rechecked with the depends on merged | 00:04 |
| *** mahito has joined #openstack-infra | 00:04 | |
| clarkb | jogo: then we should backport this fix and the devstack change to kilo | 00:04 |
| lifeless | clarkb: but I can't get beind stopping everyone elses genuine unrelated efforts unnecessarily | 00:04 |
| clarkb | jogo: sorry enable by default in devstack | 00:05 |
| clarkb | lifeless: ya I just worry that everyones unrelated efforts usually take precedence | 00:05 |
| clarkb | lifeless: and the actual user facing bugs get ignored | 00:05 |
| lifeless | stop the line is useful only because its stopping everyone to fix the flow | 00:05 |
| clarkb | lifeless: the gate is not a perfect analog to user facing issues but it does a pretty good job | 00:05 |
| jogo | clarkb: ahh https://review.openstack.org/#/c/160856/ | 00:05 |
| lifeless | things that are within the flow don't invoke that | 00:05 |
| jogo | clarkb: it failed bashate | 00:06 |
| *** sslypushenko has quit IRC | 00:06 | |
| jogo | https://jenkins05.openstack.org/job/gate-devstack-bashate/1186/ | 00:06 |
| clarkb | woo git problems | 00:06 |
| clarkb | I wonder if rax is generally having network weirdness | 00:06 |
| jogo | heh git | 00:06 |
| *** openstack has joined #openstack-infra | 00:09 | |
| clarkb | ya thats a different error but possibly related if its network trouble | 00:09 |
| jogo | lifeless: slight tangent. One of the things I am interested in seeing is better enabling smaller teams to own things they have an interest in. Such as a team of people who care about making OpenStack work with the latest dependencies | 00:09 |
| clarkb | (fungi has a support ticket open fwiw) | 00:10 |
| jogo | lifeless: versus making that the responsibility of the larger group etc | 00:10 |
| fungi | right, i added some timing data to the support ticket, but so far no fanatical takers | 00:11 |
| *** Swami has quit IRC | 00:11 | |
| * jogo wonders if AWS has the issues as frequently | 00:11 | |
| jogo | clarkb: support with rax or HP? | 00:11 |
| clarkb | jogo: rax | 00:12 |
| jogo | oh fanatical | 00:12 |
| *** jtriley has joined #openstack-infra | 00:12 | |
| fungi | fantastical | 00:12 |
| jogo | haha | 00:12 |
| *** unicell has quit IRC | 00:12 | |
| *** unicell has joined #openstack-infra | 00:13 | |
| fungi | okay, apparently it takes 3 bandersnatch mirror runs from a full sync to get the generation high enough that the todo is empty and it generates a status file | 00:14 |
| fungi | this explains the oddness i ran into after the refresh i did last night | 00:14 |
| *** sslypushenko has joined #openstack-infra | 00:15 | |
| *** samueldmq has quit IRC | 00:15 | |
| *** jogo has quit IRC | 00:17 | |
| *** tjones1 has quit IRC | 00:17 | |
| *** yamamoto has quit IRC | 00:18 | |
| *** rm_work is now known as rm_work|away | 00:20 | |
| *** Somay has quit IRC | 00:22 | |
| asselin_ | dsvm jobs always run on ubuntu or are other operating systems used? | 00:22 |
| *** markvoelker has quit IRC | 00:23 | |
| *** Somay has joined #openstack-infra | 00:23 | |
| *** Yingxin has left #openstack-infra | 00:24 | |
| *** baoli has joined #openstack-infra | 00:27 | |
| *** dims has joined #openstack-infra | 00:27 | |
| *** tjones1 has joined #openstack-infra | 00:28 | |
| *** ddieterly has joined #openstack-infra | 00:28 | |
| *** mtanino_ has joined #openstack-infra | 00:29 | |
| *** mtanino has quit IRC | 00:29 | |
| *** julim has quit IRC | 00:33 | |
| *** annegentle has joined #openstack-infra | 00:36 | |
| *** zhiwei has joined #openstack-infra | 00:37 | |
| *** annegentle has quit IRC | 00:42 | |
| *** davideagnello has quit IRC | 00:43 | |
| *** davideagnello has joined #openstack-infra | 00:44 | |
| clarkb | centos and fedora as well | 00:45 |
| *** baoli has quit IRC | 00:46 | |
| *** amotoki has joined #openstack-infra | 00:47 | |
| *** davideagnello has quit IRC | 00:49 | |
| *** kun_huang has joined #openstack-infra | 00:50 | |
| *** Somay has quit IRC | 00:52 | |
| asselin_ | clarkb, thanks | 00:53 |
| *** shashankhegde has quit IRC | 00:53 | |
| kun_huang | dear guys, is this link the latest one: http://ci.openstack.org/running-your-own.html | 00:54 |
| lifeless | yes | 00:54 |
| *** markvoelker has joined #openstack-infra | 00:54 | |
| kun_huang | lifeless: thanks ;-) | 00:54 |
| *** tjones1 has quit IRC | 00:55 | |
| *** markvoelker has quit IRC | 00:59 | |
| *** stevemar has joined #openstack-infra | 01:01 | |
| *** baoli has joined #openstack-infra | 01:01 | |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Use /opt/git directly https://review.openstack.org/177629 | 01:02 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Stop testing setup.py easy_install behaviour https://review.openstack.org/177505 | 01:03 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Test pip install -e of projects. https://review.openstack.org/177504 | 01:03 |
| *** krtaylor has quit IRC | 01:04 | |
| *** sputnik13 has quit IRC | 01:05 | |
| *** yamamoto has joined #openstack-infra | 01:06 | |
| *** david-lyle_ has joined #openstack-infra | 01:09 | |
| *** jogo has joined #openstack-infra | 01:10 | |
| *** tiswanso has joined #openstack-infra | 01:11 | |
| *** tiswanso has quit IRC | 01:16 | |
| *** tiswanso has joined #openstack-infra | 01:16 | |
| *** krtaylor has joined #openstack-infra | 01:17 | |
| jhesketh | clarkb: hmm, so I wonder if we need some kind of timeout catcher in zuul-swift-uploader? | 01:19 |
| jhesketh | the question also becomes, that if we turn off scp logs, how do we notify of this kind of error | 01:19 |
| clarkb | jhesketh: I think we fail the job then go look at jenkins? thats a bad answer | 01:21 |
| *** jtriley has quit IRC | 01:23 | |
| EmilienM | nibalizer: https://storyboard.openstack.org/#!/story/2000247 - let me know if it's wrong | 01:23 |
| *** signed8bit_ZZZzz is now known as signed8bit | 01:24 | |
| *** mriedem has quit IRC | 01:26 | |
| jhesketh | clarkb: we could look at making jenkins return more information than just FAILURE in the message back to gerrit | 01:26 |
| jhesketh | but that could be tricky | 01:26 |
| *** signed8bit has quit IRC | 01:30 | |
| *** otter768 has joined #openstack-infra | 01:30 | |
| *** fifieldt has joined #openstack-infra | 01:32 | |
| *** otter768 has quit IRC | 01:35 | |
| *** asettle has quit IRC | 01:36 | |
| *** dboik has joined #openstack-infra | 01:36 | |
| *** annegentle has joined #openstack-infra | 01:37 | |
| *** yamada-h has joined #openstack-infra | 01:39 | |
| *** dboik has quit IRC | 01:40 | |
| *** sarob has quit IRC | 01:43 | |
| *** yamada-h has quit IRC | 01:44 | |
| *** weshay has quit IRC | 01:44 | |
| *** dims has quit IRC | 01:46 | |
| *** patrickeast has quit IRC | 01:47 | |
| *** esker has joined #openstack-infra | 01:48 | |
| *** markvoelker has joined #openstack-infra | 01:55 | |
| *** tjones1 has joined #openstack-infra | 01:58 | |
| *** ayoung has quit IRC | 01:59 | |
| *** markvoelker has quit IRC | 01:59 | |
| *** ayoung has joined #openstack-infra | 02:00 | |
| *** shashankhegde has joined #openstack-infra | 02:00 | |
| *** hichihara has quit IRC | 02:02 | |
| *** spzala has quit IRC | 02:03 | |
| *** jtriley has joined #openstack-infra | 02:05 | |
| *** hichihara has joined #openstack-infra | 02:05 | |
| *** mtanino_ has quit IRC | 02:06 | |
| *** yamahata has quit IRC | 02:06 | |
| *** tjones1 has left #openstack-infra | 02:07 | |
| *** unicell has quit IRC | 02:09 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: Create rubygems mirror from rubygems.org https://review.openstack.org/178026 | 02:09 |
| EmilienM | nibalizer: PoC ^ | 02:09 |
| *** david-lyle_ has quit IRC | 02:09 | |
| *** annegentle has quit IRC | 02:12 | |
| *** sdake has joined #openstack-infra | 02:17 | |
| *** sdake_ has joined #openstack-infra | 02:20 | |
| *** sdake has quit IRC | 02:22 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: Create rubygems mirror from rubygems.org https://review.openstack.org/178026 | 02:23 |
| *** ivar-laz_ has joined #openstack-infra | 02:24 | |
| *** ivar-laz_ has quit IRC | 02:24 | |
| *** ivar-lazzaro has quit IRC | 02:27 | |
| *** mahito has quit IRC | 02:31 | |
| *** otter768 has joined #openstack-infra | 02:32 | |
| *** mahito has joined #openstack-infra | 02:35 | |
| *** yamada-h has joined #openstack-infra | 02:39 | |
| *** hichihara has quit IRC | 02:42 | |
| *** wenlock has joined #openstack-infra | 02:42 | |
| *** dboik has joined #openstack-infra | 02:43 | |
| *** baoli has quit IRC | 02:43 | |
| *** dboik_ has joined #openstack-infra | 02:44 | |
| *** yamada-h has quit IRC | 02:44 | |
| *** baoli has joined #openstack-infra | 02:46 | |
| *** dims has joined #openstack-infra | 02:47 | |
| *** dboik has quit IRC | 02:48 | |
| *** dims has quit IRC | 02:52 | |
| *** jamesmcarthur has joined #openstack-infra | 02:53 | |
| *** markvoelker has joined #openstack-infra | 02:56 | |
| *** markvoelker has quit IRC | 03:00 | |
| *** aswadr has joined #openstack-infra | 03:01 | |
| *** reed_ has joined #openstack-infra | 03:03 | |
| *** ddieterly has quit IRC | 03:04 | |
| *** jyuso1 has quit IRC | 03:06 | |
| *** asettle has joined #openstack-infra | 03:08 | |
| *** hichihara has joined #openstack-infra | 03:10 | |
| *** yamahata has joined #openstack-infra | 03:24 | |
| *** jtriley has quit IRC | 03:26 | |
| *** baoli has quit IRC | 03:28 | |
| *** mahito has quit IRC | 03:29 | |
| *** otter768 has quit IRC | 03:30 | |
| *** sdake_ has quit IRC | 03:30 | |
| *** otter768 has joined #openstack-infra | 03:33 | |
| *** mahito has joined #openstack-infra | 03:37 | |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config: new-project: stackforge/terracotta https://review.openstack.org/177747 | 03:37 |
| *** yamada-h has joined #openstack-infra | 03:40 | |
| *** reed_ has quit IRC | 03:41 | |
| *** tiswanso has quit IRC | 03:41 | |
| *** mahito has quit IRC | 03:41 | |
| *** xylan_kong has left #openstack-infra | 03:45 | |
| *** yamada-h has quit IRC | 03:45 | |
| *** yfried|afk is now known as yfried_ | 03:45 | |
| *** fedexo has joined #openstack-infra | 03:47 | |
| *** shashankhegde has quit IRC | 03:47 | |
| *** tjones1 has joined #openstack-infra | 03:48 | |
| *** dimtruck is now known as zz_dimtruck | 03:52 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: WIP: Add CLI tool to graph aggregate failure counts for tests https://review.openstack.org/178039 | 03:56 |
| *** markvoelker has joined #openstack-infra | 03:56 | |
| *** ildikov has quit IRC | 03:57 | |
| *** camunoz has quit IRC | 04:00 | |
| *** yfried_ is now known as yfried|afk | 04:00 | |
| *** otter768 has quit IRC | 04:01 | |
| *** tjones1 has quit IRC | 04:01 | |
| *** markvoelker has quit IRC | 04:01 | |
| *** ddieterly has joined #openstack-infra | 04:05 | |
| *** yamada-h has joined #openstack-infra | 04:06 | |
| *** subscope_ has joined #openstack-infra | 04:09 | |
| *** ddieterly has quit IRC | 04:09 | |
| *** unicell has joined #openstack-infra | 04:10 | |
| *** ildikov has joined #openstack-infra | 04:10 | |
| *** ayoung has quit IRC | 04:11 | |
| *** zz_dimtruck is now known as dimtruck | 04:13 | |
| *** unicell has quit IRC | 04:15 | |
| *** yfried|afk is now known as yfried_ | 04:15 | |
| *** camunoz has joined #openstack-infra | 04:16 | |
| *** shashankhegde has joined #openstack-infra | 04:16 | |
| *** unicell has joined #openstack-infra | 04:17 | |
| ianw | clarkb: is there a problem with https://review.openstack.org/#/c/167837/ ? | 04:20 |
| *** dimtruck is now known as zz_dimtruck | 04:23 | |
| *** isviridov_away is now known as isviridov | 04:25 | |
| clarkb | ianw: no? | 04:28 |
| ianw | clarkb: ok, well it has been there for a month now | 04:29 |
| *** yfried_ is now known as yfried|afk | 04:31 | |
| jhesketh | sdague: ping | 04:32 |
| *** ildikov has quit IRC | 04:33 | |
| *** Sukhdev has joined #openstack-infra | 04:37 | |
| *** jamesmcarthur has quit IRC | 04:39 | |
| *** isviridov is now known as isviridov_away | 04:39 | |
| *** yfried|afk is now known as yfried_ | 04:40 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: WIP: Add CLI tool to graph aggregate failure counts for tests https://review.openstack.org/178039 | 04:41 |
| *** mrmartin has joined #openstack-infra | 04:41 | |
| *** mahito has joined #openstack-infra | 04:42 | |
| *** achanda has joined #openstack-infra | 04:44 | |
| *** sks has joined #openstack-infra | 04:46 | |
| *** markvoelker has joined #openstack-infra | 04:47 | |
| *** btully has quit IRC | 04:49 | |
| *** mrmartin has quit IRC | 04:50 | |
| *** fedexo has quit IRC | 04:53 | |
| *** mrmartin has joined #openstack-infra | 04:54 | |
| *** fedexo has joined #openstack-infra | 04:54 | |
| *** prad has quit IRC | 04:56 | |
| *** subscope_ has quit IRC | 04:57 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/devstack-gate: Handle the case of REMAINING_TIME <= 0 https://review.openstack.org/178043 | 04:58 |
| *** yfried_ has quit IRC | 05:01 | |
| *** shashankhegde has quit IRC | 05:05 | |
| *** ddieterly has joined #openstack-infra | 05:06 | |
| *** wenlock has quit IRC | 05:06 | |
| *** prad has joined #openstack-infra | 05:09 | |
| *** yamada-h has quit IRC | 05:10 | |
| *** ddieterly has quit IRC | 05:11 | |
| *** jyuso1 has joined #openstack-infra | 05:13 | |
| *** xylan_kong has joined #openstack-infra | 05:14 | |
| *** mwagner_lap has quit IRC | 05:15 | |
| xylan_kong | hey, guys, I submitted a new project proposal https://review.openstack.org/#/c/177747/, and I hope it could be approved before May Day, so my team can continue our work when we are back in office. I don't know if the change is perfect, so, I came here, beg for reviewing and feecback, so I can improve it according to the community rules, and to make it happen | 05:19 |
| xylan_kong | finally. | 05:19 |
| *** BharatK has joined #openstack-infra | 05:23 | |
| *** ildikov has joined #openstack-infra | 05:25 | |
| *** camunoz has quit IRC | 05:26 | |
| *** rm_work|away is now known as rm_work | 05:32 | |
| *** mwagner_lap has joined #openstack-infra | 05:32 | |
| *** fedexo has quit IRC | 05:33 | |
| *** camunoz has joined #openstack-infra | 05:37 | |
| *** abregman has joined #openstack-infra | 05:40 | |
| *** ibiris_away is now known as ibiris | 05:41 | |
| jhesketh | xylan_kong: I'll take a look | 05:46 |
| jhesketh | xylan_kong: so you don't want to seed any repo now? | 05:46 |
| *** yamada-h has joined #openstack-infra | 05:47 | |
| xylan_kong | jhesketh: hi, I thought if the project is approved, a new repo will be created, and I can submit code to the repo, right? | 05:47 |
| jhesketh | yep, that's right | 05:47 |
| jhesketh | but it'll be empy | 05:47 |
| jhesketh | *empty | 05:47 |
| xylan_kong | jhesketh: ok, sure. I will submit code by myself, if the repo is there. | 05:48 |
| jhesketh | xylan_kong: if you wanted what's from https://github.com/beloglazov/openstack-neat you should do it at import | 05:48 |
| jhesketh | resubmitting the code will cause it to go through the review process which is not something you want to do | 05:48 |
| jhesketh | or else you'll lose all your history (also not great) | 05:48 |
| *** tnovacik has joined #openstack-infra | 05:49 | |
| xylan_kong | jhesketh: yep, i am ok with that. because i have do some code improvement to make it more suiable for an OpenStack related project. | 05:49 |
| *** shashankhegde has joined #openstack-infra | 05:49 | |
| xylan_kong | jhesketh: but thanks for your advise and the info! | 05:50 |
| *** hodos|2 has quit IRC | 05:50 | |
| jhesketh | xylan_kong: right, but it's reasonably important to keep the authors | 05:51 |
| jhesketh | and you never know when the history is going to be useful | 05:51 |
| jhesketh | wouldn't it be easier to just import the repo and then propose your changes on top of that? | 05:51 |
| *** hdd has joined #openstack-infra | 05:52 | |
| *** harlowja_ is now known as harlowja_away | 05:53 | |
| xylan_kong | jhesketh: actualy, it's what I intended to do, but as Andreas Jaeger said in my last patchset, the project contains a stale branch which will things complicated. | 05:53 |
| *** asrangne has joined #openstack-infra | 05:53 | |
| jhesketh | xylan_kong: you can't remove the stale branch? | 05:53 |
| *** aswadr has quit IRC | 05:54 | |
| xylan_kong | jhesketh: yes, i am not the author, i just get the permission of the author to put the project to stackforge, and make myself as the maintainer. the author will not work on that. | 05:54 |
| jhesketh | xylan_kong: the author is no longer involved? | 05:55 |
| xylan_kong | jhesketh: yep | 05:55 |
| jhesketh | xylan_kong: you could create a fork on github of the branch(es) you need and then use that as the seed for the new repo | 05:55 |
| jhesketh | I think it's important to keep the original commits and author details | 05:55 |
| xylan_kong | jhesketh: really? | 05:56 |
| xylan_kong | jhesketh: ok, I'll try | 05:56 |
| xylan_kong | jhesketh: give me a sec | 05:56 |
| jhesketh | thanks :-) | 05:57 |
| *** asrangne has quit IRC | 05:59 | |
| xylan_kong | jhesketh: btw, other parts of the patch is ok for you, right? | 05:59 |
| jhesketh | xylan_kong: I'll take a closer look, please hold | 05:59 |
| *** Krinkle is now known as Krinkle|detached | 05:59 | |
| jhesketh | xylan_kong: yep, otherwise looks good | 06:00 |
| *** yamada-h has quit IRC | 06:00 | |
| *** Darkwan has quit IRC | 06:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/178051 | 06:01 |
| *** scheuran has joined #openstack-infra | 06:01 | |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config: new-project: stackforge/terracotta https://review.openstack.org/177747 | 06:02 |
| *** otter768 has joined #openstack-infra | 06:02 | |
| *** yamada-h has joined #openstack-infra | 06:02 | |
| xylan_kong | jhesketh: i have updated the patch, please have a look when it pass the Jenkins, thanks very much! | 06:02 |
| *** hemnafk has quit IRC | 06:03 | |
| jhesketh | xylan_kong: thanks, looks good to me | 06:03 |
| *** hemnafk has joined #openstack-infra | 06:03 | |
| xylan_kong | jhesketh: your help is much appreciated! | 06:04 |
| *** stevemar has quit IRC | 06:04 | |
| jhesketh | no trouble :-) | 06:04 |
| *** SlickNik has quit IRC | 06:05 | |
| *** krotscheck has quit IRC | 06:05 | |
| *** krotscheck has joined #openstack-infra | 06:05 | |
| *** SlickNik has joined #openstack-infra | 06:06 | |
| *** otter768 has quit IRC | 06:06 | |
| *** ddieterly has joined #openstack-infra | 06:06 | |
| *** yamada-h has quit IRC | 06:06 | |
| *** samuelBartel has quit IRC | 06:10 | |
| *** ddieterly has quit IRC | 06:12 | |
| *** hyakuhei has joined #openstack-infra | 06:13 | |
| *** achanda has quit IRC | 06:13 | |
| *** sandywalsh has quit IRC | 06:14 | |
| *** btully has joined #openstack-infra | 06:14 | |
| *** sandywalsh_ has joined #openstack-infra | 06:14 | |
| *** yfried_ has joined #openstack-infra | 06:15 | |
| *** shardy_z is now known as shardy | 06:15 | |
| *** jamespage_ has joined #openstack-infra | 06:16 | |
| *** jamespage_ has quit IRC | 06:18 | |
| *** shashankhegde has quit IRC | 06:19 | |
| *** shashankhegde has joined #openstack-infra | 06:20 | |
| *** sandywalsh has joined #openstack-infra | 06:21 | |
| *** MaxV has joined #openstack-infra | 06:22 | |
| *** sandywalsh_ has quit IRC | 06:23 | |
| *** vlaza has joined #openstack-infra | 06:23 | |
| *** armax has quit IRC | 06:23 | |
| *** yfried_ is now known as yfried|afk | 06:24 | |
| *** mrunge has joined #openstack-infra | 06:26 | |
| *** yfried|afk is now known as yfried_ | 06:28 | |
| *** asettle has quit IRC | 06:31 | |
| *** zul has joined #openstack-infra | 06:31 | |
| *** MaxV has quit IRC | 06:33 | |
| *** zul has quit IRC | 06:36 | |
| *** hyakuhei has quit IRC | 06:36 | |
| *** Sukhdev has quit IRC | 06:37 | |
| *** hyakuhei has joined #openstack-infra | 06:38 | |
| *** e0ne has joined #openstack-infra | 06:39 | |
| *** jcoufal has joined #openstack-infra | 06:39 | |
| *** yfried_ is now known as yfried|afk | 06:40 | |
| *** hdd has quit IRC | 06:44 | |
| *** ociuhandu has joined #openstack-infra | 06:44 | |
| *** yfried|afk is now known as yfried_ | 06:44 | |
| *** hdd has joined #openstack-infra | 06:46 | |
| *** yamamoto has quit IRC | 06:48 | |
| *** samuelBartel has joined #openstack-infra | 06:50 | |
| *** sandywalsh_ has joined #openstack-infra | 06:51 | |
| *** sandywalsh has quit IRC | 06:51 | |
| *** ssam2 has joined #openstack-infra | 06:52 | |
| openstackgerrit | Yanis Guenane proposed openstack-infra/project-config: Add support for backport-potential commit flag https://review.openstack.org/175849 | 06:55 |
| *** ociuhandu has quit IRC | 06:57 | |
| *** sandywalsh has joined #openstack-infra | 06:58 | |
| *** sandywalsh_ has quit IRC | 06:59 | |
| *** zul has joined #openstack-infra | 06:59 | |
| *** zz_dimtruck is now known as dimtruck | 06:59 | |
| *** dtantsur|afk is now known as dtantsur | 07:01 | |
| *** [HeOS] has quit IRC | 07:04 | |
| *** hyakuhei has quit IRC | 07:06 | |
| *** ddieterly has joined #openstack-infra | 07:08 | |
| *** dimtruck is now known as zz_dimtruck | 07:09 | |
| *** samuelBartel has quit IRC | 07:10 | |
| *** ddieterly has quit IRC | 07:12 | |
| *** MaxV has joined #openstack-infra | 07:13 | |
| *** zz_dimtruck is now known as dimtruck | 07:14 | |
| *** Somay has joined #openstack-infra | 07:17 | |
| *** Somay has left #openstack-infra | 07:17 | |
| *** sergsh has joined #openstack-infra | 07:19 | |
| *** dimtruck is now known as zz_dimtruck | 07:23 | |
| jklare | hi, could some core give this one https://review.openstack.org/#/c/176674/ a quick push? it would be great if we could continue to work with these new gates for some time this week | 07:24 |
| *** funzo has quit IRC | 07:24 | |
| *** ildikov has quit IRC | 07:25 | |
| jklare | or fix any issues before the weekend | 07:25 |
| rakhmerov | hi, is there a way to delete 2015.1.X tags from stackforge/python-mistralclient repo? We'd like to start versioning our client in the same way as other projects (e.g. 0.2.x, 0.3.x) | 07:27 |
| *** shashankhegde has quit IRC | 07:28 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/system-config: Move hardcoded values into jenkins class params https://review.openstack.org/167288 | 07:28 |
| *** hdd has quit IRC | 07:30 | |
| openstackgerrit | Merged openstack-infra/project-config: new-project: stackforge/terracotta https://review.openstack.org/177747 | 07:31 |
| *** viktors|afk is now known as viktors | 07:32 | |
| *** Hal has joined #openstack-infra | 07:32 | |
| *** hichihara has quit IRC | 07:32 | |
| *** Hal is now known as Guest4607 | 07:32 | |
| *** pcaruana has joined #openstack-infra | 07:33 | |
| *** luqas has joined #openstack-infra | 07:33 | |
| *** hdd has joined #openstack-infra | 07:35 | |
| openstackgerrit | Huang Rui proposed openstack-infra/project-config: Create neutron-zvm-plugin project on StackForge https://review.openstack.org/171030 | 07:36 |
| *** samuelBartel has joined #openstack-infra | 07:37 | |
| *** ildikov has joined #openstack-infra | 07:37 | |
| *** jlanoux has joined #openstack-infra | 07:39 | |
| *** afazekas_ has joined #openstack-infra | 07:39 | |
| *** e0ne has quit IRC | 07:41 | |
| *** arxcruz has joined #openstack-infra | 07:41 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/system-config: Move server class call outside of jenkins*.pp class https://review.openstack.org/170487 | 07:41 |
| *** oomichi has joined #openstack-infra | 07:43 | |
| *** yfried_ is now known as yfried|afk | 07:44 | |
| *** markvoelker has quit IRC | 07:44 | |
| *** yfried|afk is now known as yfried_ | 07:46 | |
| *** hdd_ has joined #openstack-infra | 07:47 | |
| *** hdd has quit IRC | 07:49 | |
| *** mpavone has joined #openstack-infra | 07:50 | |
| *** isviridov_away is now known as isviridov | 07:53 | |
| *** sputnik13 has joined #openstack-infra | 07:53 | |
| *** jistr has joined #openstack-infra | 07:55 | |
| *** yamahata has quit IRC | 07:56 | |
| *** funzo has joined #openstack-infra | 07:58 | |
| *** mwagner_lap has quit IRC | 07:58 | |
| *** dhritishikhar_ has joined #openstack-infra | 07:59 | |
| *** yfried_ is now known as yfried|afk | 08:00 | |
| *** Longgeek has joined #openstack-infra | 08:00 | |
| *** dizquierdo has joined #openstack-infra | 08:00 | |
| *** Ala has joined #openstack-infra | 08:01 | |
| *** dhritishikhar_ has quit IRC | 08:01 | |
| *** dhritishikhar_ has joined #openstack-infra | 08:01 | |
| *** hyakuhei has joined #openstack-infra | 08:01 | |
| *** otter768 has joined #openstack-infra | 08:03 | |
| *** yfried|afk is now known as yfried_ | 08:03 | |
| *** hichihara has joined #openstack-infra | 08:04 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config: Use zaqar's devstack plugin https://review.openstack.org/178076 | 08:07 |
| *** otter768 has quit IRC | 08:08 | |
| *** ddieterly has joined #openstack-infra | 08:08 | |
| *** notnownikki has joined #openstack-infra | 08:09 | |
| *** Somay has joined #openstack-infra | 08:09 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/system-config: Move server class call outside of zuul_*.pp classes https://review.openstack.org/174258 | 08:10 |
| *** Ala has quit IRC | 08:10 | |
| *** Ala has joined #openstack-infra | 08:11 | |
| *** jogo has quit IRC | 08:11 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config: Use zaqar's devstack plugin https://review.openstack.org/178076 | 08:12 |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config: Use zaqar's devstack plugin https://review.openstack.org/178076 | 08:12 |
| *** ddieterly has quit IRC | 08:13 | |
| *** oomichi has quit IRC | 08:13 | |
| *** dhritishikhar_ has quit IRC | 08:14 | |
| *** markvoelker has joined #openstack-infra | 08:15 | |
| *** Longgeek has quit IRC | 08:16 | |
| *** Longgeek has joined #openstack-infra | 08:16 | |
| *** kaisers has quit IRC | 08:16 | |
| *** derekh has joined #openstack-infra | 08:17 | |
| *** btully has quit IRC | 08:19 | |
| *** markvoelker has quit IRC | 08:20 | |
| *** yfried__ has joined #openstack-infra | 08:20 | |
| *** openstackgerrit has quit IRC | 08:20 | |
| *** isviridov is now known as isviridov_away | 08:21 | |
| *** openstackgerrit has joined #openstack-infra | 08:21 | |
| *** esker has quit IRC | 08:21 | |
| *** hdd__ has joined #openstack-infra | 08:21 | |
| *** resker has joined #openstack-infra | 08:21 | |
| *** yfried__ has quit IRC | 08:21 | |
| *** yfried__ has joined #openstack-infra | 08:21 | |
| *** yfried_ has quit IRC | 08:22 | |
| *** zhiwei has quit IRC | 08:22 | |
| *** zhiwei has joined #openstack-infra | 08:23 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Create networking-zvm project on StackForge https://review.openstack.org/171030 | 08:23 |
| *** hdd_ has quit IRC | 08:24 | |
| *** lucap has joined #openstack-infra | 08:24 | |
| *** mrodden has quit IRC | 08:24 | |
| *** _nadya_ has joined #openstack-infra | 08:25 | |
| *** hashar has joined #openstack-infra | 08:25 | |
| *** mattymo has quit IRC | 08:28 | |
| *** mrodden has joined #openstack-infra | 08:28 | |
| *** mattymo has joined #openstack-infra | 08:28 | |
| *** [HeOS] has joined #openstack-infra | 08:29 | |
| *** kaisers has joined #openstack-infra | 08:31 | |
| *** ociuhandu has joined #openstack-infra | 08:32 | |
| *** mpaolino has joined #openstack-infra | 08:33 | |
| *** mpaolino has quit IRC | 08:33 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/puppet-openstackci: Add generic zuul manifests https://review.openstack.org/175970 | 08:37 |
| *** claudiub has joined #openstack-infra | 08:37 | |
| *** hichihara has quit IRC | 08:40 | |
| xylan_kong | jhesketh: ping | 08:43 |
| *** amotoki has quit IRC | 08:45 | |
| jhesketh | xylan_kong: pong | 08:46 |
| xylan_kong | jhesketh: https://review.openstack.org/177747 was approved. and then? | 08:47 |
| xylan_kong | jhesketh: accroding to http://docs.openstack.org/infra/manual/creators.html#update-the-gerrit-group-members, i need to contact someone | 08:47 |
| jhesketh | xylan_kong: yep, but unfortunately I don't have privs to do that | 08:49 |
| * jhesketh wonders if he should | 08:49 | |
| anteaya | jhesketh: I think you should | 08:49 |
| anteaya | if that counts | 08:49 |
| anteaya | then you can help folks like xylan_kong | 08:49 |
| jhesketh | xylan_kong: you can try pinging a infra/root person here, or otherwise open a bug | 08:49 |
| jhesketh | anteaya: well I like being helpful ;_) | 08:50 |
| openstackgerrit | Serg Melikyan proposed openstack/requirements: Add python-muranoclient to requirements https://review.openstack.org/177205 | 08:50 |
| xylan_kong | jhesketh: do you know who has the permission? and anteaya thanks! | 08:50 |
| anteaya | jhesketh: :) you excel at it | 08:50 |
| anteaya | xylan_kong: yes and they are asleep right now | 08:50 |
| anteaya | xylan_kong: did you submit the patch 177747? | 08:51 |
| anteaya | xylan_kong: are you the author of that patch? | 08:51 |
| xylan_kong | anteaya: yes | 08:51 |
| anteaya | great | 08:51 |
| anteaya | that is all they need | 08:51 |
| anteaya | check tomorrow | 08:51 |
| xylan_kong | anteaya: ok | 08:51 |
| anteaya | you should be in both -core and -release groups | 08:51 |
| anteaya | then you can add whoever else you need to, to the -core group | 08:52 |
| jhesketh | xylan_kong: in the mean time you should be able to clone the repo from git.openstack.org and push up your changes for review | 08:52 |
| xylan_kong | anteaya: well, see that from the doc.. | 08:52 |
| anteaya | what doc? | 08:53 |
| xylan_kong | jhesketh: yep, i find it, https://github.com/stackforge/terracotta | 08:53 |
| jhesketh | xylan_kong: that'll work, but the canonical repository is http://git.openstack.org/cgit/stackforge/terracotta/ | 08:53 |
| xylan_kong | anteaya: what you said is memtioned in http://docs.openstack.org/infra/manual/creators.html#update-the-gerrit-group-members | 08:54 |
| anteaya | xylan_kong: great | 08:54 |
| *** luqas has quit IRC | 08:54 | |
| anteaya | xylan_kong: thank you for reading the documentation | 08:54 |
| xylan_kong | anteaya: it's really helpful | 08:55 |
| anteaya | xylan_kong: I'm glad to hear that | 08:55 |
| anteaya | thanks for sharing that feedback | 08:55 |
| xylan_kong | anteaya: you're one of the author? | 08:56 |
| *** mahito has quit IRC | 08:56 | |
| anteaya | well I helped to review parts of it | 08:56 |
| anteaya | I don't think I wrote any part of the creator guide | 08:56 |
| anteaya | most of that was dhellmann | 08:56 |
| xylan_kong | anteaya: ok, good job! | 08:56 |
| anteaya | xylan_kong: thank you | 08:57 |
| anteaya | :) | 08:57 |
| anteaya | and I'm back to bed | 08:57 |
| anteaya | good night again | 08:58 |
| xylan_kong | anteaya: :) good night! | 08:58 |
| *** vlaza is now known as vlaza_brb | 09:02 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/puppet-openstackci: Add generic zuul manifests https://review.openstack.org/175970 | 09:03 |
| *** yfried__ has quit IRC | 09:03 | |
| *** yfried__ has joined #openstack-infra | 09:04 | |
| *** fhubik has joined #openstack-infra | 09:04 | |
| *** e0ne has joined #openstack-infra | 09:05 | |
| *** pelix has joined #openstack-infra | 09:06 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 09:08 |
| *** Ala has quit IRC | 09:08 | |
| *** ddieterly has joined #openstack-infra | 09:09 | |
| *** Ala has joined #openstack-infra | 09:09 | |
| *** yamamoto has joined #openstack-infra | 09:12 | |
| *** ddieterly has quit IRC | 09:14 | |
| *** mfmcdonagh has joined #openstack-infra | 09:14 | |
| *** vlaza_brb is now known as vlaza | 09:14 | |
| *** rlandy has joined #openstack-infra | 09:15 | |
| *** yamamoto has quit IRC | 09:16 | |
| *** markvoelker has joined #openstack-infra | 09:16 | |
| *** dguitarbite has joined #openstack-infra | 09:16 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:17 | |
| *** Somay has quit IRC | 09:18 | |
| *** markvoelker has quit IRC | 09:21 | |
| *** Somay has joined #openstack-infra | 09:21 | |
| *** fhubik is now known as fhubik_afk | 09:22 | |
| *** fhubik_afk is now known as fhubik | 09:22 | |
| *** yfried__ is now known as yfried|afk | 09:25 | |
| *** Somay has quit IRC | 09:26 | |
| *** Longgeek has quit IRC | 09:28 | |
| *** yamamoto has joined #openstack-infra | 09:29 | |
| *** luqas has joined #openstack-infra | 09:30 | |
| *** alexpilotti has joined #openstack-infra | 09:30 | |
| *** jamespage_ has joined #openstack-infra | 09:31 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add Neutron/Nova Floating IP support https://review.openstack.org/177036 | 09:31 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Fix exception re-raise during task execution for py34 https://review.openstack.org/178107 | 09:31 |
| *** mugsie has quit IRC | 09:31 | |
| *** jamespage_ has quit IRC | 09:36 | |
| openstackgerrit | Samuel BARTEL proposed openstack-infra/project-config: add project fuel-plugin-glance-nfs https://review.openstack.org/178109 | 09:39 |
| *** Longgeek has joined #openstack-infra | 09:40 | |
| *** mugsie has joined #openstack-infra | 09:40 | |
| *** jogo has joined #openstack-infra | 09:43 | |
| *** zhiwei has left #openstack-infra | 09:44 | |
| *** mugsie has quit IRC | 09:45 | |
| *** yfried|afk is now known as yfried__ | 09:45 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Fix exception re-raise during task execution for py34 https://review.openstack.org/178107 | 09:49 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add Neutron/Nova Floating IP support https://review.openstack.org/177036 | 09:49 |
| *** teran has quit IRC | 09:50 | |
| *** woodster_ has quit IRC | 09:50 | |
| *** mugsie has joined #openstack-infra | 09:52 | |
| *** Longgeek has quit IRC | 09:54 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 09:55 |
| *** yfried__ is now known as yfried|afk | 09:55 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/puppet-redis: Add redis 2.8 support https://review.openstack.org/178113 | 09:57 |
| *** fhubik is now known as fhubik_afk | 09:57 | |
| *** mugsie has quit IRC | 09:57 | |
| *** fifieldt has quit IRC | 09:58 | |
| *** mugsie has joined #openstack-infra | 10:00 | |
| *** Longgeek has joined #openstack-infra | 10:00 | |
| *** yamada-h has joined #openstack-infra | 10:02 | |
| *** otter768 has joined #openstack-infra | 10:03 | |
| *** mugsie has quit IRC | 10:04 | |
| *** xianghui has quit IRC | 10:04 | |
| *** xianghui has joined #openstack-infra | 10:04 | |
| *** yamada-h has quit IRC | 10:07 | |
| *** xianghui has quit IRC | 10:07 | |
| *** yfried|afk is now known as yfried__ | 10:08 | |
| *** otter768 has quit IRC | 10:08 | |
| *** Guest4607 has quit IRC | 10:08 | |
| *** jamespage_ has joined #openstack-infra | 10:09 | |
| *** ddieterly has joined #openstack-infra | 10:10 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 10:10 |
| *** kaisers has quit IRC | 10:12 | |
| *** dims has joined #openstack-infra | 10:12 | |
| *** ddieterly has quit IRC | 10:14 | |
| *** kaisers has joined #openstack-infra | 10:15 | |
| *** jamespage_ has quit IRC | 10:15 | |
| *** Longgeek has quit IRC | 10:16 | |
| *** markvoelker has joined #openstack-infra | 10:17 | |
| *** fhubik_afk is now known as fhubik | 10:20 | |
| *** hichihara has joined #openstack-infra | 10:21 | |
| *** markvoelker has quit IRC | 10:21 | |
| *** lucap1 has joined #openstack-infra | 10:23 | |
| *** lucap1 has quit IRC | 10:24 | |
| *** lucap1 has joined #openstack-infra | 10:24 | |
| *** lucap1 has quit IRC | 10:28 | |
| *** zigo has quit IRC | 10:29 | |
| *** mrmartin has quit IRC | 10:33 | |
| *** lucap has quit IRC | 10:33 | |
| *** cdent has joined #openstack-infra | 10:33 | |
| *** jlanoux_ has joined #openstack-infra | 10:33 | |
| *** Longgeek has joined #openstack-infra | 10:33 | |
| *** lucap has joined #openstack-infra | 10:33 | |
| *** yamada-h has joined #openstack-infra | 10:33 | |
| *** mugsie has joined #openstack-infra | 10:33 | |
| *** yamada-h has quit IRC | 10:33 | |
| *** gsagie has joined #openstack-infra | 10:33 | |
| *** jlanoux has quit IRC | 10:33 | |
| *** e0ne is now known as e0ne_ | 10:33 | |
| gsagie | Hello, i have a patch that fails Jenkins but i see eveyrthing as green so i am trying to understand why Jenkins added -1, https://review.openstack.org/#/c/178081/ , anyone might take a look please? | 10:33 |
| *** hyakuhei has quit IRC | 10:33 | |
| *** adrian_otto has quit IRC | 10:33 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Bump up delorean pinning to allow for the openstack-dashboard installation https://review.openstack.org/177176 | 10:33 |
| *** zigo_ has joined #openstack-infra | 10:33 | |
| *** BobH has quit IRC | 10:33 | |
| *** dtantsur is now known as dtantsur|brb | 10:33 | |
| *** BobH has joined #openstack-infra | 10:34 | |
| *** yamamoto has quit IRC | 10:34 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Puppet: don't manage /etc/hosts via cloud-init https://review.openstack.org/177722 | 10:35 |
| *** SergK_ has quit IRC | 10:35 | |
| *** e0ne_ has quit IRC | 10:35 | |
| *** SergK has joined #openstack-infra | 10:35 | |
| samuelBartel | hello, seems to have a problem similar as the gsagie'one | 10:36 |
| samuelBartel | jenkins put -1 during check but in console logs errors seems to be linked to ressources not related to my change and unavailable | 10:37 |
| samuelBartel | review is https://review.openstack.org/#/c/178109/ , if someont might take a look ti would be great thank you | 10:37 |
| *** e0ne has joined #openstack-infra | 10:39 | |
| *** sushilkm has joined #openstack-infra | 10:41 | |
| *** sushilkm has left #openstack-infra | 10:41 | |
| xylan_kong | ”After the review is approved and groups are created, ask the Infra team to add you to both groups in gerrit, and then you can add other members.“, I'm waiting for help from someone who has the previlege. | 10:43 |
| xylan_kong | https://review.openstack.org/177747 | 10:43 |
| *** ociuhandu has quit IRC | 10:43 | |
| *** teran has joined #openstack-infra | 10:44 | |
| AJaeger | xylan_kong, fungi or clarkb can do it later for you. Please tell us your ygerrit user name to add to terracota-release and terracota-core. | 10:45 |
| *** zul has quit IRC | 10:45 | |
| xylan_kong | AJaeger: hi, how are you, see you again | 10:45 |
| AJaeger | xylan_kong, doing fine, thanks! | 10:46 |
| *** alexpilotti has quit IRC | 10:46 | |
| xylan_kong | AJaeger: my gerrit full name: Lingxian Kong, short name: kong, email address: anlin.kong@gmail.com | 10:46 |
| *** spredzy_ is now known as spredzy_|afk | 10:46 | |
| *** teran has quit IRC | 10:47 | |
| *** jlanoux has joined #openstack-infra | 10:47 | |
| AJaeger | xylan_kong, thanks - I expect others to backscroll during US morning and do this for you. | 10:47 |
| *** teran has joined #openstack-infra | 10:47 | |
| xylan_kong | AJaeger: i hope so :) | 10:47 |
| *** marcusvrn1 has joined #openstack-infra | 10:49 | |
| *** jlanoux_ has quit IRC | 10:50 | |
| *** marcusvrn has quit IRC | 10:52 | |
| gsagie | Ajaeger : do you know what is the problem with Jenkins giving "-1" when every test pass? | 10:52 |
| *** fhubik is now known as fhubik_afk | 10:52 | |
| AJaeger | gsagie, do you have an example? | 10:53 |
| *** Longgeek has quit IRC | 10:56 | |
| *** hichihara has quit IRC | 10:56 | |
| *** nfedotov has joined #openstack-infra | 10:56 | |
| *** jamielennox is now known as jamielennox|away | 10:57 | |
| *** e0ne is now known as e0ne_ | 10:58 | |
| *** Longgeek has joined #openstack-infra | 11:00 | |
| gsagie | AJaeger: https://review.openstack.org/#/c/178081/ | 11:01 |
| AJaeger | gsagie, click on "toggle CI" to get the full information. | 11:02 |
| AJaeger | You have the -1 for "gate-dragonflow-requirements http://logs.openstack.org/81/178081/8/check/gate-dragonflow-requirements/be70cd3/ : Incompatible requirement found; see https://wiki.openstack.org/wiki/Requirements in 21s" | 11:02 |
| *** weshay has joined #openstack-infra | 11:07 | |
| *** e0ne_ has quit IRC | 11:09 | |
| *** weshay has quit IRC | 11:09 | |
| gsagie | AJaeger: thanks i see it now, but i see it fails on my requierment adding of ValueError: ('Expected version spec in', '-e git://git.openstack.org/openstack/neutron.git', 'at', ' git://git.openstack.org/openstack/neutron.git') | 11:09 |
| *** weshay has joined #openstack-infra | 11:10 | |
| gsagie | i saw this line used at another project where it works (i do imports from neutron) | 11:10 |
| AJaeger | gsagie, those might not have a requirements check enabled ;) | 11:10 |
| *** ddieterly has joined #openstack-infra | 11:10 | |
| gsagie | AJaeger : you add it per project? | 11:11 |
| gsagie | because it works in other project | 11:11 |
| AJaeger | gsagie, give me an example, please | 11:11 |
| *** mugsie has quit IRC | 11:11 | |
| gsagie | AJaeger: https://github.com/stackforge/networking-ovn/blob/master/requirements.txt | 11:12 |
| gsagie | set here and it works | 11:12 |
| AJaeger | and that project has no "check-requirements" job defined | 11:13 |
| AJaeger | gsagie, you could do it like neutron-vpnaas does it: | 11:13 |
| AJaeger | http://git.openstack.org/cgit/openstack/neutron-vpnaas/tree/tox.ini#n13 | 11:13 |
| gsagie | will try, thanks | 11:14 |
| AJaeger | see also http://git.openstack.org/cgit/openstack/neutron-vpnaas/tree/requirements.txt#n20 | 11:14 |
| gsagie | AJaeger: thanks! | 11:14 |
| *** ddieterly has quit IRC | 11:15 | |
| *** markvoelker has joined #openstack-infra | 11:18 | |
| sdague | :( | 11:20 |
| sdague | the top 3 recheck bugs are all infrastructure related | 11:20 |
| AJaeger | sdague ;( | 11:21 |
| sdague | the git servers are dropping connections | 11:21 |
| AJaeger | What are the problems? I see git failing quite often today | 11:21 |
| sdague | the apt repos just went bonkers | 11:21 |
| sdague | http://status.openstack.org//elastic-recheck/ | 11:22 |
| *** markvoelker has quit IRC | 11:23 | |
| AJaeger | ;( | 11:23 |
| *** e0ne has joined #openstack-infra | 11:26 | |
| ttx | sdague: I'll admit this is jeopardizing late RC respins. We basically cancelled Nova RC3 until the gate is not crazy anymore | 11:27 |
| *** rvasilets_ has joined #openstack-infra | 11:30 | |
| *** yamada-h has joined #openstack-infra | 11:30 | |
| rvasilets_ | Hi, I'm form Rally team. Our Murano job isn't working. And we need to merge this patch https://review.openstack.org/#/c/177746/ | 11:30 |
| rvasilets_ | Could you help with this? | 11:31 |
| sdague | ttx: yeh, well, tell rax to have a better network? | 11:32 |
| ttx | sdague: I tell them all the time. | 11:33 |
| *** jamespage_ has joined #openstack-infra | 11:33 | |
| sdague | I don't think anyone figured out why - Bug 1449136 - OpenStack pypi mirrors disconnecting connections stopped hitting, but that killed us yesterday | 11:33 |
| openstack | bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New] https://launchpad.net/bugs/1449136 | 11:33 |
| ttx | sdague: personally I blame our inability to detect release-critical bugs earlier in the process. We shouldn't need last-minute RCs | 11:33 |
| AJaeger | rvasilets_, I approved the patch, after merging it needs 30 mins until it's ready, so wait a bit before running "recheck" | 11:33 |
| *** jistr is now known as jistr|class | 11:34 | |
| rvasilets_ | AJaeger, ok, thank you | 11:34 |
| sdague | This - Bug 1282876 - git clone fails with "fatal: Not a git repository", "git remote update failed." - is apparently how rax implements bw limitting | 11:34 |
| openstack | bug 1282876 in OpenStack-Gate "git clone fails with "fatal: Not a git repository", "git remote update failed."" [Critical,Fix released] https://launchpad.net/bugs/1282876 - Assigned to Jeremy Stanley (fungi) | 11:34 |
| sdague | which, makes our git servers less reliable than github | 11:34 |
| sdague | and - Bug 1286818 - Ubuntu package archive periodically inconsistent causing gate build failures - is a thing that hits us all the time, because no one ever thought about apt mirroring correctly | 11:35 |
| openstack | bug 1286818 in OpenStack-Gate "Ubuntu package archive periodically inconsistent causing gate build failures" [Low,In progress] https://launchpad.net/bugs/1286818 - Assigned to Jeremy Stanley (fungi) | 11:35 |
| *** jamespage__ has joined #openstack-infra | 11:35 | |
| *** yamada-h has quit IRC | 11:35 | |
| *** zul has joined #openstack-infra | 11:35 | |
| *** yfried__ is now known as yfried|afk | 11:36 | |
| sdague | fungi: it would be nice to reopen this bug - https://bugs.launchpad.net/openstack-ci/+bug/1282876 - because it's not actually fixed | 11:37 |
| openstack | Launchpad bug 1282876 in OpenStack-Gate "git clone fails with "fatal: Not a git repository", "git remote update failed."" [Critical,Fix released] - Assigned to Jeremy Stanley (fungi) | 11:37 |
| jeblair | sdague, ttx: i'm not up to speed on what you're talking about. is there anything i can do to help? | 11:40 |
| *** jamespage_ has quit IRC | 11:40 | |
| sdague | jeblair: honestly, I think it's all structural | 11:40 |
| sdague | so we're getting big git mirror fails right now | 11:40 |
| *** ldnunes has joined #openstack-infra | 11:40 | |
| jeblair | sdague: on centos or otherwise? | 11:40 |
| sdague | 72 fails in 24 hrs | 11:41 |
| sdague | all over the place | 11:41 |
| sdague | we just had a giant apt-mirror fail spike on top of it | 11:41 |
| openstackgerrit | Merged openstack-infra/project-config: Using Neutron network in gate-rally-dsvm-murano-rally https://review.openstack.org/177746 | 11:41 |
| sdague | http://status.openstack.org//elastic-recheck/ | 11:41 |
| sdague | so that's basically managed to kill most things | 11:42 |
| *** skolekonov has joined #openstack-infra | 11:42 | |
| ttx | and create a healthy backlog | 11:42 |
| sdague | and the pypi mirror fail yesterday made anything with more than a couple of devstack jobs on it very difficult to get through | 11:43 |
| sdague | so things were pent up | 11:43 |
| jeblair | sdague: we may need to run our own apt mirrors then. we've talked about that, but these failures are usually rare and transient. maybe we should bump the priority of that. | 11:43 |
| jeblair | what do we know about the pypi issue? did running bandersnatch full resync fix it? | 11:44 |
| *** dims has quit IRC | 11:44 | |
| sdague | jeblair: I don't know, it went away. I don't know if fungi updated the bug with why | 11:44 |
| sdague | yeh, the bug seems to only contain my initial description - https://bugs.launchpad.net/openstack-gate/+bug/1449136 | 11:45 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New] | 11:45 |
| jeblair | i'm looking at the git servers | 11:46 |
| *** yfried|afk is now known as yfried__ | 11:46 | |
| sdague | great | 11:47 |
| jeblair | sdague: were the git failures temporally localized? | 11:48 |
| *** spredzy_|afk is now known as spredzy_ | 11:48 | |
| *** yamamoto has joined #openstack-infra | 11:48 | |
| sdague | it's the #2 bug on elastic-recheck, so you can see the graph there | 11:49 |
| jeblair | my connection is very bad. it should load eventually tho | 11:50 |
| *** rfolco has joined #openstack-infra | 11:50 | |
| *** dims has joined #openstack-infra | 11:51 | |
| sdague | wait... aren't you at an internet2 conference :) | 11:52 |
| jeblair | no i'm at the hotel wondering if i will get to attend the conference | 11:52 |
| *** sambetts has quit IRC | 11:52 | |
| *** sambetts has joined #openstack-infra | 11:53 | |
| *** skolekonov has quit IRC | 11:55 | |
| *** baoli has joined #openstack-infra | 11:56 | |
| sdague | it also looks like even when the git connection doesn't get killed, it gets really slow some times | 11:56 |
| sdague | http://logs.openstack.org/06/177306/1/gate/gate-devstack-dsvm-cells/dc5dc58//logs/devstack-gate-setup-workspace-new.txt.gz | 11:57 |
| sdague | the git remote updates there are taking about 2m per repo | 11:57 |
| *** claudiub has quit IRC | 11:57 | |
| openstackgerrit | Gal Sagie proposed openstack/requirements: Add Ryu to requierments https://review.openstack.org/178148 | 11:57 |
| sdague | the job was killed because it ran out of time as workspace setup took 48 minutes | 11:57 |
| *** che-arne has joined #openstack-infra | 11:58 | |
| *** mpavone has quit IRC | 11:59 | |
| jeblair | okay, for the apt mirror, we could override sources.list to use upstream | 12:00 |
| jeblair | it seems like all those errors are on rax and i beive on rax the nodes have sources.list using mirror.rackspace.com | 12:01 |
| sdague | yeh, the apt mirror is currently kind of melting everything | 12:01 |
| jeblair | (that should be double checked) | 12:01 |
| sdague | yep | 12:01 |
| sdague | so... not all are on rax | 12:01 |
| sdague | let me look at the logstash | 12:01 |
| sdague | yeh, the apt mirror seems to be nuking the world right now, we jumped to 174 failed jobs now | 12:02 |
| *** ildikov has quit IRC | 12:02 | |
| *** mugsie has joined #openstack-infra | 12:03 | |
| *** fhubik_afk is now known as fhubik | 12:03 | |
| *** dtantsur|brb is now known as dtantsur | 12:03 | |
| *** _nadya_ has quit IRC | 12:03 | |
| sdague | I see some hp cloud fails as well, let me see if I can get logstash to scope that a bit better | 12:04 |
| *** otter768 has joined #openstack-infra | 12:04 | |
| *** jamespage__ has quit IRC | 12:05 | |
| jhesketh | sdague: oh hiya. If you get a chance to talk os-loganalyze, let me know | 12:06 |
| sdague | so, there are a few hp cloud fails as well, I suspect that rax mirrored when the tcpdump was in the broken state | 12:06 |
| sdague | the fails seem to be all over a corrupt tcpdump package | 12:06 |
| sdague | or, a missing one, that is | 12:06 |
| *** yfried__ has quit IRC | 12:06 | |
| *** mpaolino has joined #openstack-infra | 12:06 | |
| *** yfried__ has joined #openstack-infra | 12:07 | |
| sdague | jeblair: so what would we need to do to flush the mirror setting on rax? | 12:07 |
| sdague | jhesketh: sure, what's up? | 12:07 |
| *** dprince has joined #openstack-infra | 12:08 | |
| jeblair | sdague: mordred is writing a change to do that now. it will require an image rebuild | 12:08 |
| gsagie | where do i change the project to not check requierments? | 12:08 |
| jhesketh | sdague: just wondering if you have any hints to what went wrong when serving up the console (that caused you to revert https://review.openstack.org/#/c/177221/) before I go digging | 12:08 |
| jhesketh | ie logs etc | 12:08 |
| gsagie | i need to add a requierment which is not yet in global-requierments (Ryu) | 12:09 |
| gsagie | but jenkins keeps failing | 12:09 |
| *** otter768 has quit IRC | 12:09 | |
| sdague | jhesketh: I don't remember, eventually fungi found an error log entry and put it in pastebin | 12:10 |
| sdague | jeblair: ok, great | 12:10 |
| jhesketh | okay | 12:10 |
| jhesketh | fungi: are you about per chance? | 12:10 |
| *** nmagnezi has joined #openstack-infra | 12:10 | |
| sdague | jhesketh: more importantly though, because only infra root can debug that, I think we should actually work out a devstack job to actually functionally test this. Because the cost of fail ends up being pretty high. | 12:11 |
| *** ildikov has joined #openstack-infra | 12:11 | |
| *** ddieterly has joined #openstack-infra | 12:11 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 12:11 | |
| jeblair | fungi, clarkb, jhesketh, pleia2, mordred: mordred is working on a change to switch to using upstream apt mirrors. hopefully that will ease the pain of the apt mirror bug, but it will take an image rebuild to take effect. | 12:12 |
| *** dprince has quit IRC | 12:12 | |
| fungi | sounds good | 12:12 |
| jhesketh | sdague: hmm, that's fair. I think maybe we need to improve the test suite first though.. | 12:12 |
| jeblair | fungi, clarkb, jhesketh, pleia2, mordred: afaict, the pypi mirror error is not happening now. i don't think it was related to bandwidth caps -- the cacti graphs look well under the specified allocation there | 12:13 |
| *** markvoelker has joined #openstack-infra | 12:13 | |
| *** mwagner_lap has joined #openstack-infra | 12:13 | |
| *** shardy_ has joined #openstack-infra | 12:13 | |
| jeblair | fungi, clarkb, jhesketh, pleia2, mordred: if it starts happening again, we may want to perform more intense debugging | 12:13 |
| fungi | yeah, i suspect a misbehaving neighbor on the same compute host as pypi.dfw | 12:13 |
| jhesketh | jeblair: okay. Let me know if there is anything I can do to help :-) | 12:14 |
| fungi | eventually rackspace may respond on the ticket i opened | 12:14 |
| *** woodster_ has joined #openstack-infra | 12:14 | |
| jeblair | fungi, clarkb, jhesketh, pleia2, mordred: i don't see us hitting limits on the git servers either. i do not know what the problem is there, but it seems to be ongoing and inspection of the mirror logs and/or traffic might be useful | 12:14 |
| jeblair | i can't do that kind of work from my current network location :( | 12:14 |
| *** shardy has quit IRC | 12:15 | |
| *** Adri2000 has quit IRC | 12:15 | |
| sdague | jhesketh: sure, but I think part of the issue is that unit tests are only going to get us so far. Making a devstack plugin for os-loganalyze where it actually runs in apache (and we can even use a swift) will make it much easier to know it's going to work on deploy | 12:15 |
| jeblair | fungi, clarkb, pleia2, mordred: so hopefully someone else can look into that | 12:15 |
| *** ddieterly has quit IRC | 12:16 | |
| *** dprince has joined #openstack-infra | 12:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Avoid vendor supplied apt mirrors https://review.openstack.org/178160 | 12:16 |
| *** nmagnezi has quit IRC | 12:16 | |
| mordred | ok - I think ^^ that's what we need there | 12:16 |
| *** Adri2000 has joined #openstack-infra | 12:16 | |
| *** nmagnezi has joined #openstack-infra | 12:16 | |
| jeblair | i think we sholud drop the rest of what we are doing and focus on these bugs now; i do not want to delay the release | 12:16 |
| sdague | so... the apt mirror failure is something we can very strongly fingerprint. Would it be possible to have zuul auto restart those jobs? | 12:16 |
| mordred | although I wonder if we should also put that into a ready script, so that we don' thave to wait for image rebuilds | 12:16 |
| jhesketh | sdague: I'm not sure devstack is the best place for that, but sure, integration testing is a good idea | 12:17 |
| sdague | jhesketh: I don't understand | 12:17 |
| jeblair | mordred: good idea | 12:17 |
| mordred | working on that now | 12:17 |
| jeblair | sdague: not easily and i'd rather avoid rube goldberging it | 12:17 |
| *** [HeOS] is now known as HeOS | 12:17 | |
| sdague | jeblair: ok, that's fair | 12:18 |
| jhesketh | sdague: we could have a test set up the instance it is on with os-loganalyze and serve up some content via swift and disk without needing to spin up a whole devstack cloud | 12:18 |
| jhesketh | sdague: the catch is we'd have to use switch credentials somewhere for it | 12:18 |
| jeblair | i'm going to transit to the conference now. i'll check in later. | 12:18 |
| jhesketh | (which I guess we'd get with devstack swift) | 12:18 |
| sdague | jhesketh: that seems like a lot of replicating devstack for no particularly good reason | 12:19 |
| *** shardy_ has quit IRC | 12:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Avoid vendor supplied apt mirrors https://review.openstack.org/178160 | 12:19 |
| sdague | also, if it was a devstack plugin, people could use os-loganalyze on their devstacks, which has been requested | 12:19 |
| *** shardy has joined #openstack-infra | 12:19 | |
| *** Hal has joined #openstack-infra | 12:19 | |
| *** Hal is now known as Guest69607 | 12:19 | |
| fungi | okay, so the one avenue i tried before with apt sources which seemed to work was to catch failures in apt-get update or apt-get install and then sed the sources.list with a different fallback url and do another apt-get update and retry the install | 12:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Avoid vendor supplied apt mirrors https://review.openstack.org/178160 | 12:21 |
| mordred | fungi, jhesketh: ok - that has readyscript in it too | 12:21 |
| fungi | there is a significant downside however | 12:21 |
| *** resker has quit IRC | 12:21 | |
| mordred | this is an area where I think yum has a better architecture, since it knows about mirror lists and how to sanly do fallbacks between them | 12:22 |
| fungi | which is that in performing a fallback we hide failures of the mirrors we're using, so that if they're decommissioned entirely we won't know about it until we start hitting failures on the fallback | 12:22 |
| jhesketh | sdague: I'm not sure what we'd be replicating of devstack? we'd still need to write the job to set up apache/os-loganlayze itself, but with the addition of spinning up devstack | 12:22 |
| jhesketh | mordred: looking | 12:22 |
| mordred | fungi, jhesketh: in any case- please check me on that patch - obvs testing this is mildly hard | 12:22 |
| notnownikki | on the openstack ci, do you ever have a problem with nodepool creating floating ips but failing to assign them to instances? | 12:23 |
| mordred | notnownikki: I'm not sure that I remember that being a pervassive problem - but we do have some floating ip leak problems that we've never really tracked down - it's possible that's a cause | 12:23 |
| fungi | jhesketh: alternatively, when developing changes to os-loganalyze run a local apache instance with it deployed there so you can simulate interactions. in this case all requests for console logs were causing wsgi tracebacks in the access logs | 12:24 |
| sdague | fungi: except swift | 12:24 |
| *** ChanServ sets mode: +o jeblair | 12:24 | |
| notnownikki | mordred, we've seen it a *lot* recently, I've written a script that gets run hourly that detects ips that have gone unassigned and frees them up | 12:24 |
| mordred | fungi, jhesketh: I suppose we could grab the ready script and put it in place manually on nodepool and see if failures stops | 12:24 |
| mordred | notnownikki: oh lovely | 12:25 |
| fungi | sdague: true. was this a problem only for console logs served via swift? | 12:25 |
| notnownikki | I can submit it to -infra if you're interested? | 12:25 |
| jhesketh | fungi: okay, good to know, thanks | 12:25 |
| mordred | notnownikki: yes please | 12:25 |
| notnownikki | cool :) | 12:25 |
| jeblair | i'm not sure how to communicate this effectively, but i think those three bugs are the only things we should be working on now. i would like for us to defer unrelated conversation to another time so that the people who can fix them are best able to do so | 12:25 |
| jhesketh | mordred: we'd still need to rebuild the images though right? | 12:25 |
| fungi | roger | 12:25 |
| sdague | jeblair: ok, no problem | 12:25 |
| mordred | jhesketh: nope - the ready script gets run by nodepool on boot | 12:25 |
| *** kgiusti has joined #openstack-infra | 12:26 | |
| jhesketh | oh sorry, ready, yes | 12:26 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack/requirements: Add ryu https://review.openstack.org/154354 | 12:26 |
| jhesketh | mordred: well I can't see it making things worse | 12:26 |
| mordred | anybody have a problem with me applying that ready-script by hand? | 12:26 |
| mordred | done | 12:27 |
| jhesketh | cool | 12:28 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/infra-specs: Specification proposal about system-config testing using containers https://review.openstack.org/172833 | 12:28 |
| mordred | have we sent out a status? | 12:28 |
| *** jeblair changes topic to "Bugs 1449136, 1282876, and 1286818 are critical and are affecting the release process" | 12:28 | |
| * jeblair transits | 12:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Avoid vendor supplied apt mirrors https://review.openstack.org/178160 | 12:29 |
| mordred | that is just updating the commit message to associate with the bug | 12:29 |
| AJaeger | mordred, did you run bashate on your script? I think it will fail | 12:30 |
| mordred | really? piddle | 12:30 |
| mordred | AJaeger: what did I get wrong? | 12:30 |
| AJaeger | mordred, indent the sudo. | 12:30 |
| AJaeger | Just from visual inspection... | 12:30 |
| mordred | oh - bashate will fail - but hte script will work | 12:31 |
| mordred | cool - I can handle that :) | 12:31 |
| *** bswartz has quit IRC | 12:31 | |
| AJaeger | Yep | 12:31 |
| AJaeger | mordred, just run it myself - bashate was happy. So, leave it as is ;) | 12:32 |
| fungi | mordred: out of curiosity why do we need to do it in the configure mirror script and the node prep script? | 12:32 |
| *** devvesa has joined #openstack-infra | 12:32 | |
| jklare | could i beg for a quick push for this one https://review.openstack.org/#/c/176674/ so we (the openstack chef people) can start playing with our new gates ;) ? | 12:32 |
| mordred | fungi: we don't - readyscript is purely to mitigate current problem without needing to respin nodes | 12:33 |
| fungi | jklare: we're suspending normal helpfulness while we focus on infra bugs | 12:33 |
| *** dboik_ has quit IRC | 12:33 | |
| fungi | mordred: oh, perfect. that makes great sense | 12:33 |
| *** davideagnello has joined #openstack-infra | 12:33 | |
| AJaeger | jklare, the team is working on fixing some release critical blocker, see topic | 12:33 |
| *** hyakuhei has joined #openstack-infra | 12:33 | |
| mordred | #status Gate is experiencing epic failures due to issues with mirrors, work is underway to mitigate and return to normal levels of sanity | 12:34 |
| BobBall | Yay | 12:34 |
| BobBall | I just noticed the epic failures in my CI too | 12:35 |
| jklare | AJaeger: fungi: oh sry didnt realize, nvm then and good luck | 12:35 |
| *** gordc has joined #openstack-infra | 12:35 | |
| *** mordred has quit IRC | 12:35 | |
| *** mordred has joined #openstack-infra | 12:35 | |
| jhesketh | fungi, mordred: so do we believe the git fails to be systemic to the apt mirrors borking? ie, is there any point trying to poke further there until we resolve the apt mirros | 12:35 |
| mordred | #status Gate is experiencing epic failures due to issues with mirrors, work is underway to mitigate and return to normal levels of sanity | 12:36 |
| openstackstatus | mordred: unknown command | 12:36 |
| mordred | gah | 12:36 |
| mordred | you'd think I'd know how to use that | 12:36 |
| mordred | but you'd apparently be wrong | 12:36 |
| sdague | so, the apt fix should make things much better, the git clone errors still exist though (at a lower failure rate) | 12:36 |
| sdague | http://logs.openstack.org/57/173357/5/check/gate-nova-docs/8155851/console.html is the most recent one | 12:36 |
| *** jtriley has joined #openstack-infra | 12:36 | |
| mordred | yeah - I think we should figure those out - we looked at the network and it doesn't seem to be bandwidth-cap related | 12:37 |
| mordred | so it may take looking at logs on the git servers and seeing what's up | 12:37 |
| sdague | that's from an hpcloud node | 12:37 |
| sdague | but it looks pretty spread around | 12:37 |
| jhesketh | okay | 12:38 |
| sdague | http://logstash.openstack.org/#eyJmaWVsZHMiOltdLCJzZWFyY2giOiJtZXNzYWdlOlwiZmF0YWw6IFRoZSByZW1vdGUgZW5kIGh1bmcgdXAgdW5leHBlY3RlZGx5XCIgQU5EIGZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIgQU5EIE5PVCBidWlsZF9xdWV1ZTpjaGVjay10cmlwbGVvIiwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJvZmZzZXQiOjAsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDMwMjI0NjU5NDQwfQ== | 12:38 |
| *** davideagnello has quit IRC | 12:38 | |
| sdague | last 48hrs, I also excluded the tripleo jobs that were failing because I don't know if they might be another issue or not | 12:38 |
| jhesketh | mordred: unfortunately I don't have access to the git morrors to check logs, but will see if there are any more clues in the jobs (although sdague seems to have done a good job of investigating0 | 12:38 |
| sdague | and they aren't preventing RC things from landing | 12:38 |
| fungi | jhesketh: i don't believe the apt mirrors and git failures to be in any way related (other than clouds don't do a good job of providing robust networks?) | 12:39 |
| mordred | #status alert Gate is experiencing epic failures due to issues with mirrors, work is underway to mitigate and return to normal levels of sanity | 12:39 |
| openstackstatus | mordred: sending alert | 12:39 |
| *** SotK_ is now known as SotK | 12:39 | |
| sdague | oh... hey... so all the current git failures are *non* dsvm jobs | 12:40 |
| sdague | now.... the apt fails could be masking that | 12:40 |
| sdague | however, in devstack we've got built in git retry logic | 12:40 |
| *** ildikov has quit IRC | 12:40 | |
| sdague | I wonder if that could be uplifted to help | 12:40 |
| fungi | we could in theory do something similar in the gerrit-git-prep macro | 12:40 |
| -openstackstatus- NOTICE: Gate is experiencing epic failures due to issues with mirrors, work is underway to mitigate and return to normal levels of sanity | 12:41 | |
| mordred | nod | 12:41 |
| *** ChanServ changes topic to "Gate is experiencing epic failures due to issues with mirrors, work is underway to mitigate and return to normal levels of sanity" | 12:41 | |
| *** jtriley has quit IRC | 12:41 | |
| sdague | https://github.com/openstack-dev/devstack/blob/master/functions-common#L489-L518 | 12:42 |
| sdague | that's the inner function we pass everything through | 12:42 |
| mordred | fungi: I can work on uplifting that into ggp if you want | 12:42 |
| mordred | although that _will_ take a node respin | 12:43 |
| fungi | mordred: alternatively we could retry the ggp script in the builder macro? | 12:43 |
| mordred | fungi: ah | 12:43 |
| openstackstatus | mordred: finished sending alert | 12:43 |
| fungi | that's about the only way i can think of to avoid rebuilding images | 12:43 |
| sdague | fungi: so it is the lesser fail | 12:44 |
| *** samueldmq has joined #openstack-infra | 12:44 | |
| fungi | sdague: i can't parse your last sentence | 12:44 |
| fungi | context? | 12:44 |
| sdague | sorry, this git clone disconnect is a less frequent failure | 12:45 |
| fungi | ahh, right | 12:45 |
| fungi | at the moment anyway | 12:45 |
| *** _nadya_ has joined #openstack-infra | 12:45 | |
| sdague | sure | 12:45 |
| mordred | sdague: yah - but we're waiting on the fix for the apt fail to help or not help | 12:46 |
| sdague | but that might mean that fixing it with something that requires an image rebuild might be acceptable, especially if it makes it largely go away | 12:46 |
| mordred | so might as well work on git fails | 12:46 |
| mordred | ah - yeah. gotcha | 12:46 |
| fungi | very good point | 12:46 |
| mordred | fungi, sdague, AJaeger: why can I not find the gerrit-git-prep macro with git grep? | 12:47 |
| sdague | http://logstash.openstack.org/#eyJmaWVsZHMiOltdLCJzZWFyY2giOiJtZXNzYWdlOlwiZmF0YWw6IFRoZSByZW1vdGUgZW5kIGh1bmcgdXAgdW5leHBlY3RlZGx5XCIgQU5EIGZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIgQU5EIE5PVCBidWlsZF9xdWV1ZTpjaGVjay10cmlwbGVvIiwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJvZmZzZXQiOjAsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDMwMjI0NjU5NDQwLCJtb2RlIjoic2NvcmUiLCJhbmFseXplX2ZpZWxkIjoiYnVpbGRfbmFtZSJ9 | 12:47 |
| fungi | still waking up and getting ready to coffee, so brain fuzzy | 12:47 |
| *** ddieterly has joined #openstack-infra | 12:47 | |
| AJaeger | mordred, jenkins/scripts/gerrit-git-prep.sh | 12:47 |
| mordred | oh. it's because I suck | 12:47 |
| sdague | is the job break down, I guess there still is one dsvm fail in there, but the majority is things like doc jobs | 12:47 |
| fungi | mordred: jenkins/jobs/macros.yaml | 12:48 |
| AJaeger | "git grep gerrit-git-prep" works for me | 12:48 |
| sdague | and based on how many more dsvm jobs we have, I think that points to the fact that this inner retry is quite effective | 12:48 |
| mordred | AJaeger: yeah - I just quit out of the search before and then forgot that I did | 12:48 |
| *** zz_dimtruck is now known as dimtruck | 12:50 | |
| *** ibiris is now known as ibiris_away | 12:50 | |
| fungi | i wonder if that would be useful to implement in zuul-cloner as well | 12:50 |
| *** sdake has joined #openstack-infra | 12:51 | |
| openstackgerrit | Andrey Pavlov proposed openstack/requirements: Add botocore to requirements https://review.openstack.org/172335 | 12:51 |
| *** alexpilotti has joined #openstack-infra | 12:52 | |
| sdague | damn, apparently we stick the retry message here into a weird log | 12:53 |
| *** sdake_ has joined #openstack-infra | 12:53 | |
| sdague | so we can't see how often we recover on normal devstack runs | 12:53 |
| *** hyakuhei has quit IRC | 12:53 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Put retry loop around gerrit-git-prep https://review.openstack.org/178173 | 12:53 |
| *** sushilkm has joined #openstack-infra | 12:54 | |
| *** sushilkm has left #openstack-infra | 12:54 | |
| mordred | fungi, sdague, AJaeger: ^^ there is one putting it into the builder macro, which should get us til a rebuild - I'll do one to ggp itself next | 12:54 |
| *** dboik has joined #openstack-infra | 12:54 | |
| *** marcusvrn1 has quit IRC | 12:54 | |
| *** marcusvrn has joined #openstack-infra | 12:55 | |
| sdague | mordred: you want to put a message in there when we fall into the retry loop as well so we can keep an eye on how often it happens? | 12:55 |
| *** julim has joined #openstack-infra | 12:56 | |
| *** sarob has joined #openstack-infra | 12:56 | |
| openstackgerrit | Ilia Meerovich proposed openstack-infra/jenkins-job-builder: Adding support for SSH bulider plugin https://review.openstack.org/178176 | 12:56 |
| *** sdake has quit IRC | 12:56 | |
| *** Somay has joined #openstack-infra | 12:57 | |
| *** ibiris_away is now known as ibiris | 12:57 | |
| AJaeger | mordred, no need to make the two changes dependend on each other | 12:57 |
| fungi | sdague: probably best to do that in the script modification | 12:58 |
| openstackgerrit | Ilia Meerovich proposed openstack-infra/jenkins-job-builder: Adding bulider for SSH plugin https://review.openstack.org/178176 | 12:58 |
| *** ildikov has joined #openstack-infra | 12:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Use git_timed function from devstack in ggp https://review.openstack.org/178177 | 12:59 |
| openstackgerrit | Ilia Meerovich proposed openstack-infra/jenkins-job-builder: Adding builder for SSH plugin https://review.openstack.org/178176 | 12:59 |
| mordred | sdague: sure! | 12:59 |
| mordred | AJaeger: good point - I'll decouple them next pass | 12:59 |
| *** bknudson has joined #openstack-infra | 12:59 | |
| *** gsagie has quit IRC | 13:00 | |
| sdague | fungi: ok, that's probably fine as well | 13:00 |
| sdague | AJaeger: well I approved the first patch | 13:00 |
| sdague | so dependencies won't really be an issue | 13:00 |
| AJaeger | sdague, ok ;) | 13:01 |
| *** jistr|class is now known as jistr | 13:01 | |
| *** yamamoto has quit IRC | 13:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Add retry message to gerrit-git-prep macro https://review.openstack.org/178180 | 13:02 |
| sdague | mordred: so... there is no GIT_TIMEOUT set in https://review.openstack.org/#/c/178177/1/jenkins/scripts/gerrit-git-prep.sh,cm right? | 13:02 |
| mordred | sdague: there ya go ^^ | 13:02 |
| *** zul has quit IRC | 13:02 | |
| mordred | sdague: nope | 13:02 |
| sdague | so... shouldn't we set one? | 13:02 |
| *** swat30 has quit IRC | 13:02 | |
| mordred | oh - heh | 13:02 |
| sdague | otherwise what does timeout do if timeout is set to 0 | 13:02 |
| mordred | good point | 13:02 |
| mordred | what does it default to in devstack? | 13:02 |
| mordred | oh - 0 defaults to waiting for forever | 13:03 |
| sdague | I think we set it in d-g | 13:03 |
| sdague | or.... not | 13:03 |
| sdague | yeh, I don't know | 13:03 |
| sdague | apparently we don't set it other places | 13:03 |
| sdague | so I guess it's fine | 13:03 |
| mordred | so - hangs aren't our problem - at least the mechanism is there so that we can set GIT_TIMEOUT in the g-g-p macro in the future | 13:04 |
| mordred | if it becomes an issue | 13:04 |
| *** _nadya_ has quit IRC | 13:04 | |
| sdague | maybe that's one of the things that ianw needed for the red hat network | 13:04 |
| *** yamamoto has joined #openstack-infra | 13:04 | |
| mordred | nod | 13:05 |
| sdague | https://review.openstack.org/#/c/74910/ - yeh, that's the original change | 13:05 |
| mordred | any chance it's been long enough for us to see if anyhting is working? | 13:05 |
| fungi | unlikely | 13:05 |
| fungi | i've finally gotten bandersnatch caught back up on all the mirrors and cron/puppet reenabled on them now | 13:06 |
| fungi | status files look fine this time so shouldn't have a repeat of yesterday | 13:06 |
| mordred | I should be able to see apt-get commands in a jenkins log on a dvsm job, yeah? | 13:06 |
| *** spzala has joined #openstack-infra | 13:06 | |
| fungi | well, in a devstack-gate setup log | 13:07 |
| *** swat30 has joined #openstack-infra | 13:07 | |
| fungi | you could ssh into a worker and look at its filesystem too | 13:07 |
| sdague | mordred: so... as of 2 minutes ago rax systems were still using their mirrors - https://jenkins02.openstack.org/job/check-tempest-dsvm-ironic-pxe_ssh/3604/console | 13:07 |
| jeblair | fungi: was the re-running of bandersnatch related to the disconnects, or were there two pypi problems? | 13:07 |
| mordred | oh wow | 13:07 |
| *** hichihara has joined #openstack-infra | 13:07 | |
| sdague | how soon should the hotfix have applied | 13:08 |
| sdague | ? | 13:08 |
| mordred | https://jenkins02.openstack.org/job/check-dg-tempest-dsvm-full/333/console | 13:08 |
| *** hichihara has quit IRC | 13:08 | |
| *** jeblair changes topic to "Bugs 1449136, 1282876, and 1286818 are critical and are affecting the release process" | 13:08 | |
| *** mriedem has joined #openstack-infra | 13:08 | |
| *** ZZelle has quit IRC | 13:08 | |
| *** raginbajin has quit IRC | 13:08 | |
| sdague | mordred: oh... that's a new one | 13:08 |
| mordred | jeblair: ready scripts run when the node becomes ready, not when a job starts, right? | 13:08 |
| sdague | fail on swift log upload | 13:08 |
| mordred | I'm thinking there are some issues at rackspace | 13:09 |
| openstackgerrit | Merged openstack-infra/project-config: Avoid vendor supplied apt mirrors https://review.openstack.org/178160 | 13:09 |
| *** ZZelle has joined #openstack-infra | 13:09 | |
| *** ddieterly has quit IRC | 13:09 | |
| fungi | jeblair: unrelated, though it was suspected they might be related early on yesterday | 13:09 |
| sdague | well, the apt mirror is the normal apt mirror issue | 13:09 |
| *** raginbajin has joined #openstack-infra | 13:09 | |
| mordred | yah | 13:09 |
| sdague | we actually saw the same package fail on an hpcloud node | 13:09 |
| mordred | wait - what? | 13:09 |
| sdague | rax just happened to lock their mirror to that mirror state | 13:10 |
| sdague | so in the giant pile of rax failures | 13:10 |
| mordred | gotcha | 13:10 |
| jeblair | mordred: yes, right before nodepool marks them ready | 13:10 |
| mordred | sdague: so - we need to wait for a node to move from building to ready state | 13:10 |
| sdague | I found 1 hp cloud node that tried to apt-get upgrade and hit the same missing tcpdump node | 13:10 |
| fungi | jeblair: it was ultimately caused by a problem on pypi itself (probably glusterfs-related) which caused bandersnatch to delete some packages from all our mirrors (but left them in the indexes), at least one of which we were using | 13:10 |
| sdague | mordred: ah, so we have to cycle out some ready nodes first then? | 13:10 |
| mordred | sdague: yes | 13:11 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkVSUk9SOnJvb3Q6RmlsZSBwb3N0aW5nIGVycm9yXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzAyMjY1NTgwNjh9 | 13:11 |
| *** tiswanso has joined #openstack-infra | 13:11 | |
| sdague | so... that swift fail, is a thing | 13:11 |
| jeblair | fungi: ack. that's exciting. | 13:11 |
| fungi | jeblair: but apparently when you do a full refresh of bandersnatch mirror now you need to keep running it over and over until it catches up, because the second pass is likely to still take longer than our timeouts | 13:11 |
| mordred | jeblair: we may have a _fourth_ problem | 13:11 |
| *** changbl has quit IRC | 13:11 | |
| sdague | 124 hits in the last 24 hours | 13:11 |
| sdague | let me ER that | 13:11 |
| jeblair | fungi: wow | 13:11 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add Neutron/Nova Floating IP support https://review.openstack.org/177036 | 13:11 |
| jhesketh | mordred: simple non-blocking comment on https://review.openstack.org/#/c/178177/ | 13:12 |
| *** zz_johnthetubagu is now known as johnthetubaguy | 13:12 | |
| *** annegentle has joined #openstack-infra | 13:13 | |
| jeblair | so "rax network problems" explains pypi, git, and swift errors :/ | 13:13 |
| fungi | oh yeah | 13:13 |
| *** zul has joined #openstack-infra | 13:13 | |
| sdague | mordred: though it looks like it's only a failure 25% of the time? | 13:14 |
| fungi | they did have notices up that they were doing network upgrade maintenance in dfw all week | 13:14 |
| mordred | jhesketh: yes. let me fix that | 13:14 |
| sdague | mordred: so... hmmm.... it's a thing, but it doesn't fail the job? | 13:14 |
| jeblair | fungi: they picked a good week for it | 13:14 |
| fungi | linked from https://status.rackspace.com/ | 13:14 |
| *** lucap has quit IRC | 13:15 | |
| jeblair | jhesketh: i wonder if we should put some retries in the swift upload? | 13:15 |
| fungi | oh, good point. we have it retry jenkins console log retrieval but we don't yet have it retry uploading to swift | 13:16 |
| *** dimtruck is now known as zz_dimtruck | 13:16 | |
| sdague | jeblair: so... it must be retrying | 13:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Use git_timed function from devstack in ggp https://review.openstack.org/178177 | 13:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Add retry message to gerrit-git-prep macro https://review.openstack.org/178180 | 13:16 |
| sdague | because that error is only 25% fatal | 13:16 |
| *** wenlock has joined #openstack-infra | 13:16 | |
| sdague | this is a fail - http://logs.openstack.org/72/177072/4/gate/gate-senlin-python27/fce63d8/console.html | 13:17 |
| fungi | i didn't see it in the script. looking again | 13:17 |
| sdague | but this is a success | 13:17 |
| sdague | http://logs.openstack.org/75/177675/1/check/gate-ceilometer-python34/b7770fc/console.html | 13:17 |
| sdague | 13:17 | |
| sdague | and I don't see any difference of note | 13:17 |
| *** sushilkm has joined #openstack-infra | 13:18 | |
| *** sushilkm has left #openstack-infra | 13:18 | |
| *** peristeri has joined #openstack-infra | 13:18 | |
| jeblair | sdague: i think it's never a failure actually | 13:18 |
| jeblair | sdague: the failure you posted looks like a git failure | 13:19 |
| jhesketh | jeblair: can do | 13:19 |
| sdague | jeblair: ok, so it's a red herring? | 13:19 |
| sdague | do we just need the tool to stop stack tracing | 13:19 |
| jeblair | sdague: i believe it really failed to upload the file, so i think we want to fix it, but i think it's not causing carnage | 13:20 |
| jeblair | jhesketh: ^ | 13:20 |
| openstackgerrit | Merged openstack-infra/project-config: Put retry loop around gerrit-git-prep https://review.openstack.org/178173 | 13:20 |
| jhesketh | jeblair: there are lot of errors in the query sdague posted.. it seems like zuul-swift-upload is having as much trouble with the network as anything else | 13:21 |
| jhesketh | difference is that the job doesn't fail on bad upload | 13:22 |
| jeblair | jhesketh: agreed | 13:22 |
| jeblair | jhesketh: so i don't think fixing it is in the critical path. just "nice" that we have "helpfully" been shown where it could be more resilient under adverse conditions :) | 13:23 |
| mordred | sdague: 2015-04-28 05:01:15.887 | fatal: unable to access 'http://zm01.openstack.org/p/openstack-infra/devstack-gate/': Failed to connect to zm01.openstack.org port 80: Connection timed out | 13:23 |
| mordred | sdague: that's one of the things we're tracking, yeah? | 13:23 |
| jhesketh | jeblair: well at the moment everything on the critical path appears to be network hardening/retrying anyway? | 13:24 |
| jeblair | sdague, mordred: that's a zuul merger, as was the git failure sdague posted in relation to the swift log thing. i think that is showing that the network errors are _very_ widespread | 13:24 |
| fungi | i saw a fetch failure from a zuul merger tank a job yesterday. retried it manually (same ref) and it worked. wasn't a timeout, but a short read or something | 13:24 |
| mordred | yah | 13:24 |
| *** dkranz has joined #openstack-infra | 13:24 | |
| fungi | seems likely to also be "rackspace network broken" | 13:24 |
| jeblair | so we're seeing network problems on zuul mergers, git servers, pypi mirrors, and individual nodes uploading to swift | 13:24 |
| fungi | oh! we also saw a multinode master lose ssh connectivity to a subnode in the middle of a job yesterday | 13:25 |
| mordred | well, hopefully the retry in g-g-p will help work us past network issues on git things | 13:25 |
| fungi | not sure what provider that was in though | 13:25 |
| jeblair | and by bad luck, there's the ubuntu mirror problem which is (probably) not network related | 13:26 |
| mordred | yah | 13:26 |
| mordred | jeblair: so - fwiw - in the past we've not had a good reason to run zuul mergers or git mirrors per-cloud | 13:26 |
| fungi | the distro mirror networks are usually volunteer operated, and i've seen them broken at random plenty. it's not just rackspace that has a hard time maintaining package mirrors | 13:26 |
| *** sks has quit IRC | 13:26 | |
| mordred | jeblair: this might be a reason to consider that in the future | 13:26 |
| mordred | fungi: totally | 13:27 |
| jhesketh | most failures do appear to be in the hpcloud (although I'm guessing y'all already knew that) | 13:27 |
| mordred | jhesketh: the swift errors? | 13:28 |
| jeblair | so i'm optimistic and glad we can mitigate the ubuntu mirror problem, and hopefully reduce the git errors. i'm also glad that it looks like we actually have an explanation for why everything broke at once. | 13:28 |
| jhesketh | mordred: yes | 13:28 |
| fungi | it's the whole "crossing the internet" problem | 13:28 |
| fungi | for swift anyway | 13:28 |
| mordred | jeblair: ++ | 13:28 |
| *** e0ne is now known as e0ne_ | 13:28 | |
| fungi | rax dfw seems to be broken enough internally right now that machines can't even communicate reliably within it though | 13:28 |
| mordred | it's too bad that we can't use a swift in each cloud and have it appear as one contiguous thing | 13:28 |
| jhesketh | fungi: ah, the swift ones probably aren't failing for rax as they are closer to the network | 13:28 |
| fungi | jhesketh: that would be my entirely unscientific conjecture, anyway | 13:29 |
| *** dustins has joined #openstack-infra | 13:29 | |
| *** dkranz has quit IRC | 13:29 | |
| *** adrian_otto has joined #openstack-infra | 13:30 | |
| sdague | fungi: yeh, it's hard to tell if it's hpcloud network being bonkers, or rax network being bonkers, or a 3rd one | 13:30 |
| sdague | but it looks like predominantly hpcloud fails in the swift log upload | 13:31 |
| sdague | I guess when/if we get a 3rd cloud, the finger pointing will be easier | 13:31 |
| fungi | sdague: breaking news! clouds fail at networking | 13:32 |
| fungi | story at 11 | 13:32 |
| sdague | and hence why people like to run their own, where they can control that :) | 13:32 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: Add a --stable-series argument to release_notes.py https://review.openstack.org/178194 | 13:32 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: Add option to format release notes for email https://review.openstack.org/178195 | 13:32 |
| sdague | jhesketh: I filed - https://bugs.launchpad.net/openstack-gate/+bug/1449570 | 13:32 |
| openstack | Launchpad bug 1449570 in OpenStack-Gate "raxspace swift sometimes fails to accept log uploads with file posting error" [Undecided,New] | 13:32 |
| jeblair | sdague: triangulation ftw | 13:32 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Retry log upload to swift https://review.openstack.org/178199 | 13:34 |
| jhesketh | sdague, jeblair: ^ | 13:34 |
| *** esker has joined #openstack-infra | 13:35 | |
| sdague | ok, I'm pushing a tempest patch series which will hopefully drain all the rest of the old nodes | 13:35 |
| *** jtriley has joined #openstack-infra | 13:35 | |
| sdague | http://dl.dropbox.com/u/6514884/screenshot_226.png (though that was an hour ago) | 13:36 |
| *** e0ne_ is now known as e0ne | 13:37 | |
| *** ddieterly has joined #openstack-infra | 13:38 | |
| fungi | it's marvellous how the rackspace maintenance notice says "We expect this maintenance to be non-impact to customers." | 13:38 |
| mordred | yeah - I spoke to johnthetubaguy about it | 13:38 |
| fungi | makes me wonder if they're wrong, or if we're wrong | 13:38 |
| mordred | and it was control-plane side maint | 13:38 |
| *** ociuhandu has joined #openstack-infra | 13:38 | |
| mordred | but he's helpfully talking to some people further | 13:39 |
| fungi | well, control plane side problems can certainly impact traffic flow | 13:39 |
| *** Ala has quit IRC | 13:39 | |
| *** _nadya_ has joined #openstack-infra | 13:40 | |
| sdague | jhesketh: that's an infinite loop, right? | 13:40 |
| *** esker has quit IRC | 13:40 | |
| sdague | there is no break out of the while loop | 13:40 |
| fungi | i'm less wondering about disruptiveness of the maintenance activity itself, and more whether what was changed by the maintenance yesterday is not quite operating as expected | 13:40 |
| jhesketh | sdague: the raise should break it yes? | 13:40 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:40 | |
| sdague | jhesketh: only if you get an exception | 13:40 |
| jhesketh | oh right, yes, my bad | 13:40 |
| *** Somay has quit IRC | 13:40 | |
| *** sarob has quit IRC | 13:41 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Retry log upload to swift https://review.openstack.org/178199 | 13:41 |
| jhesketh | sdague: okay, trying again ^ | 13:41 |
| sdague | so, honestly for stuff like this for x in xrange(3) is often what I use so that even if you screw up success you aren't in an infinite loop | 13:42 |
| johnthetubaguy | fungi: I am looking into what that change actually is | 13:42 |
| sdague | but this should be fine | 13:42 |
| fungi | johnthetubaguy: awesome--thanks and sorry to bother you! | 13:43 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Retry log upload to swift https://review.openstack.org/178199 | 13:44 |
| jhesketh | sdague: good idea, ^ | 13:44 |
| *** stevemar has joined #openstack-infra | 13:44 | |
| *** miqui has joined #openstack-infra | 13:44 | |
| sdague | jhesketh: s/while/for/ ? | 13:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Retry log upload to swift https://review.openstack.org/178199 | 13:45 |
| jhesketh | sigh, sorry, thanks | 13:45 |
| mordred | :) | 13:45 |
| *** vhoward has left #openstack-infra | 13:45 | |
| mordred | sdague, jhesketh: so - not for this instance- but we have an iterate_timeout function in nodepool and shade that we use for things like this - maybe it wants to become a micro-library? | 13:46 |
| mordred | s/instance/instant/ | 13:46 |
| jhesketh | yeah I've seen that reused in a number of places, so possibly not a bad idea | 13:46 |
| jhesketh | but multiple attempts like this is pretty trivial, it's just late and I'm rushing things (clearly a bad idea) | 13:47 |
| *** zz_jgrimm is now known as jgrimm | 13:47 | |
| mordred | yah - I certainly don't think we should block on that | 13:47 |
| jeblair | it is 8 lines | 13:47 |
| *** scheuran has quit IRC | 13:47 | |
| sdague | mordred: sure, though honestly retry logic is something that gets written all over the place, I think it's fine to just do it in place and not librarize it | 13:48 |
| *** esker has joined #openstack-infra | 13:48 | |
| *** mpavone has joined #openstack-infra | 13:48 | |
| jhesketh | mordred, sdague: where are we at with cycling out nodes to try the new ready script? | 13:49 |
| jhesketh | anything I can help with? | 13:49 |
| mordred | sdague: indeed | 13:49 |
| sdague | jhesketh: I do not know, I reordered a tempest patch series I had to try to sweep up any remaining bad nodes | 13:49 |
| openstackgerrit | Merged openstack-infra/project-config: Use git_timed function from devstack in ggp https://review.openstack.org/178177 | 13:50 |
| sdague | as that should have just consumed 100+ dsvm nodes | 13:50 |
| pabelanger | morning | 13:50 |
| sdague | those are still getting rax mirrors on rax nodes | 13:51 |
| sdague | https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-large-ops/46910/console | 13:51 |
| sdague | mordred: is there a signature for your change that we could see in the logs to know if it's running? | 13:51 |
| jhesketh | sdague: ah okay | 13:52 |
| jeblair | 2015-04-28 13:43:29.724 | + echo 'HTTP check of http://mirror.rackspace.com/ubuntu/dists/trusty/Release.gpg - attempt #1' | 13:52 |
| jeblair | i wonder if we should remove that from devstack-gate? | 13:53 |
| jeblair | sdague: does that read sources.list? | 13:53 |
| sdague | jeblair: I don't think it reads sources.list | 13:53 |
| fungi | it does not. its just an http ping basically | 13:54 |
| sdague | we should probably remove it, it was put in to try to debug this previously | 13:54 |
| fungi | grabs the release file i think | 13:54 |
| *** BharatK has quit IRC | 13:54 | |
| sdague | yep | 13:54 |
| jeblair | oh i think it's hardcoded for rax | 13:55 |
| jeblair | so it even does that on hpcloud | 13:55 |
| mordred | wow | 13:55 |
| fungi | heh | 13:55 |
| mordred | sdague: basically, you should not see mirror.rackspace.com in any of the apt interactions | 13:55 |
| mordred | sdague: if you do - my fix did not work | 13:56 |
| ttx | might have been not that smart to release after Ubuntu after all | 13:56 |
| sdague | mordred: ok, well still seeing it, but I want to know if your fix is not applied, or if it's not working | 13:56 |
| sdague | so I was wondering if there was a way to figure that out | 13:56 |
| ttx | maybe their apt mirrors are taking a 15.04 upgrade hit ? | 13:56 |
| jeblair | sdague: yeah, point me at a failed jobs | 13:56 |
| *** armax has joined #openstack-infra | 13:56 | |
| sdague | jeblair: https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-large-ops/46910/console | 13:57 |
| jeblair | ttx: we're seeing the problem mostly on rax mirrors, and we're in progress moving to upstream ubuntu mirrors to mitigate | 13:57 |
| ttx | jeblair: ack | 13:57 |
| sdague | ttx: yeh, so it's the same issue that happens from time to time, rax runs their mirror at a moment when canonical is updating theirs, and so gets a broken version | 13:58 |
| sdague | which remains broken until next mirror update | 13:58 |
| jeblair | 2015-04-28 13:40:06,707 DEBUG nodepool.NodeLauncher: Node id: 2345326 is running, ip: 104.130.134.112, testing ssh | 13:58 |
| sdague | but unlike hitting ubuntu servers directly, which recover in a couple minutes, this ends up being broken for a long window of time | 13:58 |
| ttx | sdague: I know there is a strict process to follow to avoid that -(- basically there is an "update in progress" lock file you can use to sync your own | 13:58 |
| *** Krinkle|detached is now known as Krinkle | 13:58 | |
| derekh | Can anybody take a look at this please, I'm trying to move the tripleo F20 job to F21 https://review.openstack.org/#/c/169778/ | 13:58 |
| ttx | Maybe rax didn't really follow the "official mirror" guidelines | 13:59 |
| jeblair | sdague, mordred: ^ that seems very recent, recent enough that i think it's not working | 13:59 |
| sdague | derekh: right now everyone is working on release impacting bugs in infra | 13:59 |
| sdague | jeblair: right, that's why I asked | 13:59 |
| derekh | sdague: ack | 13:59 |
| *** sdake_ has quit IRC | 13:59 | |
| sdague | that was one of the jobs in my 6 tempest series | 13:59 |
| jeblair | sdague: yeah, i mean that's the timestamp for when the ready script should have run. so confirming that node _should_ have gotten the fix. | 13:59 |
| mordred | jeblair: bleh | 14:00 |
| *** dprince has quit IRC | 14:00 | |
| *** dustins_ has joined #openstack-infra | 14:00 | |
| jeblair | mordred: i checked a random ready node and see mirror.rax in sources.list | 14:01 |
| mordred | if [ "$LSBDISTID" == "Ubuntu" ] ; then | 14:01 |
| jeblair | (_recently_ ready) | 14:01 |
| mordred | that should be a single = shouldn't it | 14:01 |
| *** dprince has joined #openstack-infra | 14:02 | |
| *** btully has joined #openstack-infra | 14:02 | |
| mordred | hrm. nope. == works too | 14:02 |
| sdague | so... at least on my 15.04 | 14:02 |
| sdague | DISTRIB_ID=Ubuntu | 14:02 |
| sdague | /etc/lsb-release | 14:02 |
| sdague | os1:~> cat /etc/lsb-release | 14:02 |
| sdague | DISTRIB_ID=Ubuntu | 14:02 |
| sdague | DISTRIB_RELEASE=14.04 | 14:02 |
| sdague | DISTRIB_CODENAME=trusty | 14:02 |
| sdague | DISTRIB_DESCRIPTION="Ubuntu 14.04 LTS" | 14:02 |
| mordred | yeah - I've checked that on precise, trusty and vivid | 14:02 |
| *** ildikov has quit IRC | 14:03 | |
| sdague | right, but where is LSBDISTID set? | 14:03 |
| *** ildikov has joined #openstack-infra | 14:03 | |
| jeblair | sdague: LSBDISTID=$(lsb_release -is) | 14:03 |
| mordred | sdague: the line before it | 14:03 |
| jeblair | in the script | 14:03 |
| sdague | oh | 14:03 |
| jeblair | mordred, sdague: i've run the contents of the if block and they seem to work | 14:03 |
| mordred | jeblair: I don't need to apt-get update after do I? | 14:03 |
| *** dustins has quit IRC | 14:03 | |
| *** asselin has joined #openstack-infra | 14:03 | |
| mordred | jeblair: I mean, we shoudl be doing an update before apt-get install commands across the board, right? | 14:04 |
| jeblair | mordred: no, i believe the problem is that isn't being run for some reason | 14:04 |
| jeblair | mordred: because the file did not appear to have been updated | 14:04 |
| fungi | nodepool.o.o _does_ seem to have the updated ready script at least | 14:04 |
| jroll | jeblair: so uh, you all want the rax mirror team poked to bump the mirrors? which region is this? | 14:05 |
| jeblair | sdague: ^ is it all regions? | 14:05 |
| *** otter768 has joined #openstack-infra | 14:05 | |
| *** ibiris is now known as ibiris_away | 14:06 | |
| jeblair | mordred: the pydistutils file _is_ being updated. i'm stumped. | 14:06 |
| *** shardy_ has joined #openstack-infra | 14:06 | |
| mordred | jeblair: my only hunch is that that if is resolving to false on the node | 14:07 |
| sdague | jeblair: I see at least iad and dfw in the list | 14:07 |
| sdague | mordred: so we're running under -x | 14:07 |
| fungi | "Exception: Unable to run ready script" | 14:07 |
| fungi | in the nodepool debug log | 14:07 |
| jeblair | jroll: 14:07 < sdague> jeblair: I see at least iad and dfw in the list | 14:07 |
| sdague | is it not +x or something? | 14:07 |
| fungi | oh, that's from much earlier | 14:07 |
| mordred | sdague: but we don't log the ready script output | 14:07 |
| *** mrmartin has joined #openstack-infra | 14:07 | |
| sdague | mordred: oh.... poo | 14:07 |
| fungi | apparently we only log the ready script when it fails to run, so it must be running | 14:08 |
| *** shardy has quit IRC | 14:08 | |
| jroll | jeblair: sdague: thanks, going to reproduce and email; ubuntu only yes? | 14:08 |
| jeblair | jroll: afaik | 14:08 |
| fungi | jroll: as far as we know, but we're not using your package mirrors for anything besides ubuntu | 14:08 |
| jroll | right, thanks | 14:09 |
| clarkb | I think d-g uses local git cache but ggp does not. devstack retries should not affect the gate due to error on clone | 14:09 |
| marcusvrn | krtaylor: ping | 14:09 |
| jeblair | mordred, fungi, sdague: the ready script has a sudo for the dd (it does not need it, but it should be okay). that means it gets logged in auth.log. i do not see it being run in auth.log. | 14:09 |
| *** yfried has joined #openstack-infra | 14:09 | |
| sdague | jroll: this particular issue is you have a snapshot of the mirror that misses the tcpdump package the meta files say you have | 14:09 |
| sdague | jroll: however, this happens every few weeks | 14:10 |
| fungi | jeblair: that suggests the conditional isn't matching | 14:10 |
| *** otter768 has quit IRC | 14:10 | |
| sdague | because the rax mirror scripts don't do mirroring correctly | 14:10 |
| jeblair | mordred, fungi, sdague: (i see my test runs of it in auth.log though, so i know it should show up there) | 14:10 |
| mordred | jeblair: you have a held node? | 14:10 |
| jroll | sdague: I was just going to ask which package, thanks. and yeah, aware of this issue and hate it. | 14:10 |
| jeblair | fungi: agreed | 14:10 |
| jroll | sdague: emailing folks | 14:10 |
| jeblair | mordred: not held, just unused. | 14:10 |
| mordred | jeblair: and you've run those commands on taht node and you don't get the skip? | 14:10 |
| jeblair | mordred: i did not run the conditional, only the contents of it. | 14:11 |
| jeblair | i will do that now | 14:11 |
| mordred | (this is going to wind up being something really stupid ultimately) | 14:11 |
| sdague | jroll: I see an ord node with the failure as well, so not region specific | 14:11 |
| krtaylor | marcusvrn, pong, high latency though in a meeting | 14:11 |
| clarkb | jroll supposedly reprepro will fix this issue | 14:11 |
| jeblair | mordred: the configure script is not correct | 14:11 |
| *** yfried__ has quit IRC | 14:12 | |
| jeblair | mordred: as in, it looks like the old version | 14:12 |
| mordred | oh! | 14:12 |
| jeblair | ends at pypi | 14:12 |
| *** shardy_ has quit IRC | 14:12 | |
| clarkb | jroll it does its own consistency checking when mirroring | 14:12 |
| *** dboik_ has joined #openstack-infra | 14:12 | |
| fungi | are we not running the configure script from /etc on the nodepool server? | 14:12 |
| *** shardy has joined #openstack-infra | 14:12 | |
| jeblair | i wonder if ready scripts are expected to be image-baked | 14:13 |
| mordred | they are | 14:13 |
| mordred | I just looked | 14:13 |
| fungi | crap | 14:13 |
| mordred | blastit | 14:13 |
| *** unicell1 has joined #openstack-infra | 14:13 | |
| jroll | clarkb: thanks for the info | 14:13 |
| *** unicell has quit IRC | 14:13 | |
| fungi | well, new images are kicking off now anyway | 14:13 |
| jroll | sdague: jeblair: email dropped, these folks are usually pretty responsive | 14:13 |
| mordred | fungi: hang on | 14:13 |
| mordred | fungi: we need to re-enable and re-run puppet on nodepool - it's disabled to put the ready script in place by hand from earlier | 14:13 |
| mordred | doing that now | 14:13 |
| fungi | hang onto what? 14:14 utc is when our image updates start | 14:13 |
| mordred | GAH | 14:14 |
| fungi | the daily ones | 14:14 |
| sdague | I think I'm going to have to start drinking early today | 14:14 |
| jeblair | mordred: it should be okay, yeah? the updates will put your hand-made ready script into place | 14:14 |
| mordred | indeed | 14:14 |
| mordred | I'm running puppet real quick to try to catch all of the script updates | 14:14 |
| mordred | done | 14:14 |
| *** dims has quit IRC | 14:15 | |
| fungi | right, it takes nodepoold a few minutes to get to the point where those files matter anyway | 14:15 |
| mordred | yah | 14:15 |
| *** ibiris_away is now known as ibiris | 14:15 | |
| *** Somay has joined #openstack-infra | 14:15 | |
| *** dims has joined #openstack-infra | 14:15 | |
| * jeblair transits | 14:15 | |
| *** dboik has quit IRC | 14:16 | |
| *** eharney has joined #openstack-infra | 14:16 | |
| fungi | nihilist arby's has it nailed today | 14:16 |
| sdague | heh | 14:17 |
| marcusvrn | krtaylor: I saw your email ("Announcing Third Party CI Tools Repo") then I discovered that there's a CI working group...hehe there's a channel for that group? or just the weekly meetings? | 14:17 |
| fungi | ooh! rackspace update on pypi.dfw. my suspicions confirmed... "After investigation of the physical host server I found another customer outbound spamming. They have been notified and appropriate actions have been taken. Please let us know if you see any other issues. | 14:18 |
| asselin | marcusvrn, just weekly meetings | 14:18 |
| *** amitgandhinz has joined #openstack-infra | 14:18 | |
| sdague | fungi: so... it would be nice if we didn't need to lose a day before they react there | 14:20 |
| sdague | I wonder if there is something we can do to get more early warning to them | 14:20 |
| *** sdake has joined #openstack-infra | 14:20 | |
| marcusvrn | asselin, hmm nice! I'm reading a few logs of the weekly meetings to get myself more involved and to know how can I help | 14:20 |
| *** rossella_s has quit IRC | 14:21 | |
| *** sdake_ has joined #openstack-infra | 14:22 | |
| jeblair | marcusvrn, asselin, krtaylor: 3rd party ci conversation normally happens here since we're all working with the same tools. we're just really busy focusing on some release-critical bugs right now. | 14:22 |
| *** rossella_s has joined #openstack-infra | 14:22 | |
| *** tonytan4ever has joined #openstack-infra | 14:22 | |
| *** sputnik13 has quit IRC | 14:22 | |
| *** fhubik has quit IRC | 14:23 | |
| *** bauzas has joined #openstack-infra | 14:23 | |
| johnthetubaguy | fungi: are things settling down at all now? | 14:24 |
| johnthetubaguy | fungi: are you still seeing swift issues, not the maintenance should completed, etc | 14:25 |
| fungi | sdague: agreed. acutally it was only a little over 11 hours for them to confirm, but i wonder if we should go back to pestering poor rackers in irc like johnthetubaguy | 14:25 |
| *** sdake has quit IRC | 14:25 | |
| sdague | johnthetubaguy: so, the swift thing is not currently fatal for us | 14:25 |
| johnthetubaguy | fungi: we need to get you some better contacts for this stuff | 14:25 |
| sdague | johnthetubaguy: ++ | 14:25 |
| *** ayoung has joined #openstack-infra | 14:25 | |
| fungi | johnthetubaguy: the pypi mirror in dfw turned out to be a noisy (spammer) neighbot on the same compute node, according to fanatical support | 14:26 |
| *** Somay has quit IRC | 14:26 | |
| johnthetubaguy | fungi: OK, interesting, what hardware is that on? | 14:26 |
| johnthetubaguy | fungi: performance or standard? | 14:26 |
| fungi | johnthetubaguy: good question... checking | 14:27 |
| clarkb | should be performance but do double check | 14:27 |
| fungi | johnthetubaguy: 4gb performance | 14:27 |
| clarkb | if it was standard we wouldnt need the cinder volume | 14:27 |
| johnthetubaguy | hmm, crazy stuff | 14:27 |
| *** mattfarina has joined #openstack-infra | 14:28 | |
| *** bswartz has joined #openstack-infra | 14:28 | |
| fungi | johnthetubaguy: for that one we were getting intermittent tcp read timeouts reaching that machine from other instances in the same region (it only serves machines in that same region) | 14:28 |
| *** BharatK has joined #openstack-infra | 14:28 | |
| sdague | clarkb: ... low priority item ... however in reading through a bunch of logs this morning it would be nice to tighten up the ansible output, it kind of spews throughout and makes following things a little hard | 14:28 |
| fungi | so someone on that node must have really, REALLY been generating a lot of traffic to have that sort of impact | 14:28 |
| johnthetubaguy | hmm, odd, I would have hoped QoS would have been enough for that… | 14:28 |
| johnthetubaguy | yeah… nuts | 14:29 |
| clarkb | sdague can you expand on that it writes a json blob with return codes iirc. is that a problem? | 14:29 |
| fungi | johnthetubaguy: the distro package mirror problem seems to be that they're either sometimes mirroring from other broken mirrors and then serving that broken state for a while. jroll is pestering the mirror operator there to fix it but we're in the process of switching to the normal ubuntu mirror network to work around it | 14:29 |
| clarkb | sdague it shouldnt really spew anything | 14:29 |
| sdague | well more things like: + /tmp/ansible/bin/ansible all -f 5 -i /home/jenkins/workspace/check-tempest-dsvm-full/inventory -m shell -a 'source '\''/home/jenkins/workspace/check-tempest-dsvm-full/test_env.sh'\'' && source '\''/home/jenkins/workspace/check-tempest-dsvm-full/devstack-gate/functions.sh'\'' && tsfilter setup_workspace '\''master'\'' '\''/opt/stack/new'\'' executable=/bin/bash' | 14:29 |
| sdague | as single line | 14:29 |
| clarkb | sdague ya thats how we run things | 14:30 |
| johnthetubaguy | fungi: ah, thats good to know | 14:30 |
| fungi | er, either sometimes mirroring from other broken mirrors, or mirroring incorrectly | 14:30 |
| sdague | sure, but really aggressively move xtracing things out of the console log for a while | 14:30 |
| sdague | hence all the sublogs | 14:30 |
| jroll | fungi: manual sync in progress | 14:30 |
| clarkb | sdague oh its the tracing nit ansible. I see | 14:30 |
| *** adrian_otto has quit IRC | 14:30 | |
| sdague | yeh | 14:30 |
| fungi | jroll: thanks for getting in touch with them | 14:31 |
| sdague | hence, low priority | 14:31 |
| *** mriedem has quit IRC | 14:31 | |
| fungi | johnthetubaguy: so the current remaining mystery is that we're also getting intermittent git remote failures reaching our git servers (which are also hosted in dfw) | 14:31 |
| *** notmyname has quit IRC | 14:31 | |
| fungi | i haven't yet been able to track down a cause | 14:31 |
| jeblair | fungi: and possibly related -- similar errors contacting zuul mergers | 14:32 |
| johnthetubaguy | fungi: makes me wonder if they have a noisy neighbour too | 14:32 |
| fungi | jeblair: oh, yep right | 14:32 |
| clarkb | fungi git is returning an error code thibg too right? maybe that is a clue? | 14:32 |
| johnthetubaguy | fungi: they don't back up to swift do they? | 14:33 |
| *** anthonyper has quit IRC | 14:33 | |
| fungi | johnthetubaguy: nope | 14:33 |
| *** afazekas_ has quit IRC | 14:33 | |
| johnthetubaguy | fungi: seems crazy it all happened at once | 14:33 |
| *** anthonyper has joined #openstack-infra | 14:33 | |
| fungi | johnthetubaguy: right, and at a particularly inconvenient time | 14:33 |
| *** bhunter71 has joined #openstack-infra | 14:34 | |
| johnthetubaguy | fungi: quite | 14:34 |
| *** tlbr has quit IRC | 14:34 | |
| *** notmyname has joined #openstack-infra | 14:34 | |
| *** tlbr has joined #openstack-infra | 14:34 | |
| *** jamesmcarthur has joined #openstack-infra | 14:34 | |
| sdague | johnthetubaguy: so, the apt thing is the one that's ruining the world and blocking all changes | 14:34 |
| fungi | clarkb: it sometimes returns a specific-looking git error which turns out to have any manner of possible causes, many of which can be network issues | 14:34 |
| sdague | the others have been bouncing around at a rando fail rate that was not good, but not killer | 14:35 |
| sdague | johnthetubaguy: see graphs here - http://status.openstack.org//elastic-recheck/ | 14:35 |
| *** annegentle has quit IRC | 14:35 | |
| fungi | well, the pypi mirror problem in dfw yesterday was pretty heinous for gate performance too, but it's thankfully addressed at this point | 14:35 |
| *** Somay has joined #openstack-infra | 14:36 | |
| *** mriedem has joined #openstack-infra | 14:36 | |
| sdague | fungi: true | 14:36 |
| jeblair | yeah, i still want to track down what's causing the git and/or swift failures | 14:36 |
| sdague | jeblair: agreed | 14:36 |
| *** nmagnezi has quit IRC | 14:37 | |
| johnthetubaguy | silly question, but what interface are you using to talk to git? public or snet? | 14:37 |
| fungi | public | 14:37 |
| fungi | because it's accessed from other regions and other service providers entirely | 14:37 |
| johnthetubaguy | oh yeah, of course, my bad | 14:38 |
| fungi | we don't (currently) have a per-region git mirror network | 14:38 |
| *** Somay has quit IRC | 14:39 | |
| *** vlaza has left #openstack-infra | 14:39 | |
| johnthetubaguy | I mean public vs snet shouldn't be any different I guess, its not an isolated network anyways | 14:39 |
| jhesketh | fungi, jeblair, mordred: seems like things are slowly getting under control. Anything else I can help with before I head off? | 14:40 |
| *** Ala has joined #openstack-infra | 14:40 | |
| *** Ala has quit IRC | 14:41 | |
| jeblair | sdague, fungi: gah! ER is tracking github failures in tripleo ci jobs | 14:41 |
| fungi | weird. when i try to click on console.html at http://logs.openstack.org/81/178181/2/check-tripleo/check-tripleo-ironic-overcloud-f20puppet-nonha/878edcb/ my browser wants to download and save it | 14:41 |
| sdague | jeblair: yeh, I can exclude those | 14:41 |
| ttx | hmm, gate is empty now but I think more because all checks fail than because everythingn is under control ? | 14:41 |
| jeblair | fungi: yes, also that | 14:41 |
| sdague | there are non github ones in there | 14:41 |
| sdague | jeblair: let me tweak the signature | 14:41 |
| fungi | looks like up until a few days ago we were just seeing this for bare-centos6-rax-dfw workers | 14:42 |
| jeblair | ttx: yeah, ubuntu mirror fix is still pending (two parallel fixes are in progress) | 14:42 |
| jeblair | jhesketh: i don't think so, thanks :) | 14:43 |
| ttx | jeblair: ok -- should I still recehck RC3-critical jobs, or that doesn't really help ? | 14:43 |
| jhesketh | jeblair: no trouble, sorry I can't help more! | 14:43 |
| jhesketh | see you guys in a few hours | 14:43 |
| sdague | ttx: not yet | 14:43 |
| fungi | oh, looks like that was some specific issue in rax dfw on the 23d | 14:43 |
| ttx | sdague: ok, I'll hold on further rechecks until you tell me we can use gate again. Some others might beat me to those though :) | 14:44 |
| *** Ala has joined #openstack-infra | 14:44 | |
| *** mattfarina has quit IRC | 14:44 | |
| sdague | ttx: yeh, well they'll just fail a lot | 14:44 |
| jeblair | fungi: i'm assuming the swath of tripleo failures over the past few hours are all github | 14:44 |
| fungi | so, ruling out the tripleo github failures, it looks like all the jobs matching the git error pattern are in hpcloud trying to reach our git mirrors in rackspace over the internet | 14:44 |
| jeblair | that's a very different characterization | 14:45 |
| fungi | so could be hpcloud network problems, could be an issue somewhere in the route between them on the open 'net, or could be something else i guess | 14:45 |
| mordred | fungi: oh. yeah - that's ... | 14:45 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: Remove tripleo from signature https://review.openstack.org/178220 | 14:46 |
| sdague | jeblair: maybe | 14:46 |
| sdague | jeblair: the following | 14:46 |
| fungi | this is just based on a quick visual scan of the logstash query results for that particular bug | 14:46 |
| sdague | message:"fatal: The remote end hung up unexpectedly" AND filename:"console.html" AND NOT build_queue:check-tripleo | 14:46 |
| *** annegentle has joined #openstack-infra | 14:46 | |
| sdague | will give you the non tripleo ones | 14:46 |
| sdague | it's still quite a few in the last 48 hours | 14:47 |
| sdague | 57 in 48 hrs | 14:47 |
| jeblair | sdague: yeah, as long as a stackforge project doesn't stick a github clone in there :) | 14:47 |
| fungi | most recent one which impacted a rackspace worker was ~26 hours ago | 14:47 |
| fungi | so there _are_ impacts to rackspace workers, but nowhere nearly the frequency of hits to hpcloud workers | 14:47 |
| *** masayukig_ has quit IRC | 14:48 | |
| sdague | gate-nova-docs is the most recent fail | 14:48 |
| sdague | in that query | 14:48 |
| jeblair | fungi, sdague: do the zuul merger errors share the same characterization? | 14:48 |
| sdague | I do not know | 14:48 |
| jeblair | (are they covered by this query and are they also hpcloud localized?) | 14:48 |
| sdague | fungi: ? | 14:48 |
| fungi | no clue yet | 14:49 |
| sdague | so this is not hpcloud localized | 14:49 |
| sdague | this is broadly impacting | 14:49 |
| fungi | i need to dig up some specific examples and see | 14:49 |
| fungi | sdague: broadly impacting but orders of magnitude more frequent in hpcloud today | 14:49 |
| *** sputnik13 has joined #openstack-infra | 14:49 | |
| jeblair | 2015-04-28 09:53:55.840 | error: RPC failed; result=7, HTTP code = 0 | 14:50 |
| jeblair | from http://logs.openstack.org/72/177072/4/gate/gate-senlin-python27/fce63d8/console.html earlier | 14:50 |
| sdague | fungi: ok | 14:50 |
| *** masayukig_ has joined #openstack-infra | 14:50 | |
| sdague | yeh, we still don't have cloud as broken out metadata, so it's visual scan to sort that out | 14:50 |
| fungi | sdague: as in logstash doesn't show a hit for its bug 1282876 query matching a rax worker for more than a day | 14:50 |
| openstack | bug 1282876 in OpenStack-Gate "git clone fails with "fatal: Not a git repository", "git remote update failed."" [Critical,Fix released] https://launchpad.net/bugs/1282876 - Assigned to Jeremy Stanley (fungi) | 14:50 |
| *** BharatK has quit IRC | 14:51 | |
| fungi | all the hits in the past 24 hours have been tripleo and hpcloud | 14:51 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Remove tripleo from signature https://review.openstack.org/178220 | 14:51 |
| jeblair | fungi, sdague: oh it emits the "remote end hung up" line, so zm failures should be included in the git failures query | 14:51 |
| fungi | jeblair: yep, looks like roughly the same set of jobs/workers/hits when i query for that | 14:52 |
| *** yamahata has joined #openstack-infra | 14:52 | |
| sdague | fungi / jeblair ok, so tripleo is purged from that query now, when ER updates again we'll see the updated list | 14:53 |
| *** nelsnelson has joined #openstack-infra | 14:53 | |
| *** _nadya_ has quit IRC | 14:54 | |
| fungi | what we've got here is a failure to communicate (over the internet using git) | 14:54 |
| *** annegentle has quit IRC | 14:54 | |
| *** _nadya_ has joined #openstack-infra | 14:54 | |
| jeblair | fungi, sdague: do we think we can stand down on the git failures then? | 14:54 |
| clarkb | fungi does it affect each distro/release? or is one being affected? | 14:55 |
| fungi | clarkb: looks like mostly bare trusty, but that may be because devstack is working around it and we don't run nearly as many jobs on other platforms | 14:55 |
| *** ajmiller has joined #openstack-infra | 14:55 | |
| clarkb | ya | 14:55 |
| *** bswartz has quit IRC | 14:55 | |
| sdague | jeblair: so, I think the work will mitigate it | 14:56 |
| clarkb | also note that devstack shouldnt actually eork around this | 14:56 |
| sdague | it's not a huge failure | 14:56 |
| jeblair | is there a ggp workaround change proposed for it? | 14:56 |
| jeblair | clarkb: ? | 14:56 |
| fungi | two | 14:56 |
| clarkb | we error on clone, I think the local git cache may make a difference instead | 14:56 |
| sdague | jeblair: https://review.openstack.org/#/c/178173/ | 14:56 |
| *** dkranz has joined #openstack-infra | 14:56 | |
| sdague | it's merged | 14:56 |
| sdague | and probably should be in new images | 14:56 |
| clarkb | jeblair devstack isnt going to retry any git clone ops for us. d-g does it all | 14:56 |
| jeblair | clarkb: this isn't specific to cloning, but also happens with any fetching | 14:57 |
| fungi | there's a short-term fix to do it in the ggp builder macro (retry the script if it exits nonzero) and a workaround in the script itself (which should end up in the new images) | 14:57 |
| jeblair | clarkb: we do fetch from git mirror to bring things up to date | 14:57 |
| clarkb | jeblair but thats d-g as well right? | 14:57 |
| jeblair | clarkb: yeah | 14:57 |
| jeblair | clarkb: i don't tthink anyone is actually talking about devstack | 14:57 |
| fungi | though if we think this is a useful pattern, then we may want to add something similar to zuul-cloner as well | 14:57 |
| clarkb | ok | 14:57 |
| jeblair | fungi: ++ | 14:57 |
| sdague | clarkb: correct, there are no dsvm fails here | 14:58 |
| *** jaypipes has quit IRC | 14:58 | |
| jeblair | clarkb: maybe we were just abbreviating :) | 14:58 |
| sdague | this was just about adding some retry logic to non dsvm jobs to see if that helps | 14:58 |
| *** shardy_ has joined #openstack-infra | 14:58 | |
| anteaya | github uses rackspace for servers, or did at one point | 14:58 |
| anteaya | not sure if they still do | 14:58 |
| *** _nadya_ has quit IRC | 14:59 | |
| clarkb | rigt I am just suugesting maybe d-g does something else to be more reliable? but if its normal fetches then retry is likely what we need | 14:59 |
| fungi | the in-script workaround for ggp sets a timeout on git operations too, which might be useful for the puppet apply jobs problem we have on centos6 (when coupled with the retry logic) | 14:59 |
| *** BobH has quit IRC | 14:59 | |
| *** nfedotov has quit IRC | 14:59 | |
| *** eharney has quit IRC | 14:59 | |
| fungi | that is, might be useful in those jobs if we do something similar in zuul-cloner | 14:59 |
| jeblair | okay, to recap: a) ubuntu mirror fixes in progress, b) git retry fixes in place, c) pypi problem beleived resolved (bad neighbor), d) swift problems ongoing, not critical, workaround in pipeline | 14:59 |
| *** shardy has quit IRC | 15:00 | |
| sdague | jeblair: yes | 15:00 |
| mordred | yup | 15:00 |
| sdague | that sounds correct | 15:00 |
| mordred | I agree with the recap | 15:00 |
| fungi | confirmed | 15:00 |
| sdague | resolution of a) should get the trains moving again | 15:00 |
| sdague | the rest just make them run better | 15:00 |
| fungi | and now to begin the day o' meetings | 15:00 |
| jeblair | w00t. and once we confirm a) we can status ok | 15:00 |
| *** claudiub has joined #openstack-infra | 15:01 | |
| mordred | I also agree with that w00t | 15:02 |
| *** sabeen has joined #openstack-infra | 15:02 | |
| *** BharatK has joined #openstack-infra | 15:03 | |
| *** sabeen2 has joined #openstack-infra | 15:03 | |
| *** mtanino has joined #openstack-infra | 15:03 | |
| *** shardy_ has quit IRC | 15:04 | |
| *** shardy has joined #openstack-infra | 15:04 | |
| *** erikmwilson is now known as Guest61272 | 15:04 | |
| *** erikmwil_ has joined #openstack-infra | 15:04 | |
| *** Guest61272 has quit IRC | 15:04 | |
| *** erikmwil_ is now known as erikmwilson | 15:04 | |
| *** e0ne is now known as e0ne_ | 15:04 | |
| *** erikmwilson_ has joined #openstack-infra | 15:04 | |
| *** e0ne_ is now known as e0ne | 15:05 | |
| jeblair | i think most image builds are complete | 15:05 |
| clarkb | is the swift problem related to the cloud? | 15:05 |
| *** smccully has joined #openstack-infra | 15:05 | |
| clarkb | or should we be looking at more than a workaround for ourselves? | 15:05 |
| *** emagana has joined #openstack-infra | 15:05 | |
| jeblair | clarkb: i don't know, we have very little data on that | 15:06 |
| fungi | yeah, needs more analysis. i suspect the frequency is low enough that we lack a great representative sample from which to draw conclusions | 15:06 |
| *** jamespage_ has joined #openstack-infra | 15:06 | |
| *** zul has quit IRC | 15:06 | |
| *** sabeen has quit IRC | 15:07 | |
| sdague | clarkb: https://bugs.launchpad.net/openstack-gate/+bug/1449570 has a query that will get you all the hits | 15:08 |
| openstack | Launchpad bug 1449570 in OpenStack-Gate "raxspace swift sometimes fails to accept log uploads with file posting error" [Undecided,New] | 15:08 |
| sdague | if you want to look at some to figure it out | 15:08 |
| sdague | mordred tripped over it by accident while looking at other logs | 15:08 |
| sdague | it's not in ER at the moment because it's apparently non-fatal atm | 15:08 |
| *** sputnik13 has quit IRC | 15:09 | |
| * mordred trips over many things | 15:09 | |
| sdague | reading random logs in openstack often turns up interesting issues that no one noticed yet | 15:09 |
| *** erikmwilson has quit IRC | 15:10 | |
| jeblair | are all the swift errors from hpcloud? | 15:10 |
| *** erikmwilson has joined #openstack-infra | 15:11 | |
| clarkb | jeblair: for the last 12 hours: yes | 15:12 |
| *** jamespage_ has quit IRC | 15:13 | |
| *** SergK has quit IRC | 15:13 | |
| clarkb | did a 48hour query, it started at 4/27 1900UTC ish | 15:13 |
| fungi | so... back in the beforetime, when we were actually running a lot of jobs in hpcloud, a disproportionate number of network-related job failures involved hpcloud workers. we didn't see it for a while because hpcloud was so broken that we effectively stopped running jobs there. but now... | 15:13 |
| jeblair | so that could be because of hpcloud network, internet1, or the rax-swift public network path (but _not_ the rax internal path) | 15:13 |
| fungi | absolutely | 15:14 |
| *** eharney has joined #openstack-infra | 15:14 | |
| fungi | time for internet2 already. srsly | 15:14 |
| clarkb | also looks like no rax failures at all since it started | 15:14 |
| *** peristeri has quit IRC | 15:14 | |
| clarkb | 123 matches of the query but some jobs matches it multiple times so thats not 1:1 job:failcount | 15:14 |
| *** signed8bit has joined #openstack-infra | 15:15 | |
| clarkb | message:"ERROR:root:File posting error" AND filename:"console.html" is my query | 15:15 |
| fungi | right, it doesn't (necessarily) cause job failures | 15:15 |
| clarkb | fungi: sorry I meant failcount meaning matching of that query | 15:15 |
| clarkb | basically one job can match multiple times | 15:15 |
| fungi | oh, right yep | 15:15 |
| fungi | we can (and do) encounter this error multiple times in a job | 15:15 |
| clarkb | expanding to a 7 day query there is one additional hit on the 24th | 15:16 |
| *** dangers_away is now known as dangers | 15:16 | |
| clarkb | that was also in hpcloud but thats it, started very recently and only affects hpcloud | 15:16 |
| sdague | did gerrit fall over? | 15:16 |
| sdague | I just ran rechecks on a bunch of jobs | 15:17 |
| *** changbl has joined #openstack-infra | 15:17 | |
| clarkb | its up for me | 15:17 |
| sdague | and .... not showing up in zuul | 15:17 |
| jeblair | sdague: works for me | 15:17 |
| sdague | sorry, gerrit event stream | 15:17 |
| jeblair | sdague: oh, did stream events get stuck? | 15:17 |
| jeblair | yes, it is stuck | 15:17 |
| jeblair | fungi: do you have things staged for a restart? | 15:18 |
| sdague | so... it's gotten stuck a lot recently, right? | 15:18 |
| jeblair | sdague: yes | 15:18 |
| jeblair | sdague: i have a change to help debug the problem which we have not been able to deploy yet | 15:18 |
| jeblair | fungi was working on staging that so that we might try deploying it again the next time it got stuck | 15:19 |
| sdague | gotcha | 15:19 |
| *** spredzy_ is now known as spredzy_|afk | 15:19 | |
| jeblair | we could just restart it, but if fungi or someone else up on the latest there is around, i'd like to see if we can slip it in | 15:20 |
| *** BobH has joined #openstack-infra | 15:20 | |
| fungi | jeblair: yep, i can pull from my env now | 15:20 |
| fungi | it's all set up and _should_ work (i tested it out as best i could) | 15:21 |
| *** sdake_ has quit IRC | 15:21 | |
| jeblair | i'm on review.o.o and can help out (v6 internet2 seems reliable enough) | 15:21 |
| fungi | at least, it should allow us to quickly recreate the bouncy castle error, hopefully see why that's happening, and then solve it or quickly switch back | 15:21 |
| *** sdake has joined #openstack-infra | 15:21 | |
| *** plaurin has left #openstack-infra | 15:21 | |
| *** emagana has quit IRC | 15:21 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 15:22 | |
| *** emagana has joined #openstack-infra | 15:22 | |
| fungi | okay, i'll pull from my env now... that should trigger a service stop but won't reindex lucene | 15:22 |
| jeblair | fungi: and don't forget to disable puppet to avoid flapping back | 15:22 |
| clarkb | I am around to help too | 15:22 |
| *** peristeri has joined #openstack-infra | 15:23 | |
| clarkb | fungi: and iirc your change wasto do everything but reindex right? | 15:23 |
| dansmith | gerrit is down? | 15:23 |
| clarkb | dansmith: emergency restart (event stream hung again) | 15:24 |
| sdake | in case you didn't know, review.openstack.org is experiencing downtime clarkb | 15:24 |
| sdake | sounds like you do know :) | 15:24 |
| sdake | thanks | 15:24 |
| fungi | pulled and puppet agent disabled for now | 15:24 |
| dansmith | clarkb: okay, it had been down long enough that it didn't seem like a restart, but then again, I have no idea how long it takes to restart a mega java app :) | 15:24 |
| fungi | Error in custom provider, java.lang.SecurityException: class "org.bouncycastle.util.io.TeeOutputStream"'s signer information does not match signer information of other classes in the same package | 15:24 |
| *** hdd__ has quit IRC | 15:25 | |
| *** lucap has joined #openstack-infra | 15:25 | |
| jeblair | while locating com.google.gerrit.server.contact.ContactStoreProvider | 15:25 |
| *** sushilkm has joined #openstack-infra | 15:25 | |
| clarkb | we have two bcprovs in place | 15:25 |
| *** sushilkm has left #openstack-infra | 15:25 | |
| fungi | that looks like the cause | 15:25 |
| jeblair | yeah, system and gerrit-local, right? | 15:25 |
| clarkb | one from today and the other is the symlink to the system package | 15:26 |
| clarkb | jeblair: yup | 15:26 |
| clarkb | same thing with the msql driver connector thing | 15:26 |
| fungi | looks like we've started bundling and unpacking bcprov-jdk16-144.jar | 15:26 |
| fungi | so they must be included in the builds and weren't previously? | 15:27 |
| fungi | i'll remove the symlinks | 15:27 |
| jeblair | could we have changed the job definitions? | 15:27 |
| jeblair | fungi: is that the right direction? | 15:27 |
| *** bswartz has joined #openstack-infra | 15:27 | |
| fungi | it's possible we've changed the build job to start bundling them | 15:28 |
| jeblair | the puppet module makes the system symlink | 15:28 |
| clarkb | I was going to suggest moving the non symlinks | 15:28 |
| jeblair | clarkb: i lean toward that; i think that may be the minimal change | 15:28 |
| jeblair | fungi: ^ | 15:28 |
| fungi | yeah, i wanted to see if the bundled ones work. they do not | 15:28 |
| fungi | removing them and restoring the symlinks now | 15:29 |
| clarkb | fungi: if ou need target paths I have those in my ls scrollback | 15:29 |
| jeblair | fungi: you have the old state? i have an ls if you need it. | 15:29 |
| clarkb | jeblair: :) | 15:29 |
| *** baoli has quit IRC | 15:29 | |
| fungi | i had the original state logged | 15:30 |
| fungi | that worked | 15:30 |
| clarkb | fungi: so it is starting with jeblair's patch in place? | 15:30 |
| *** baoli has joined #openstack-infra | 15:30 | |
| fungi | i've moved the unpacked versions to ~gerrit2 | 15:30 |
| fungi | i believe so | 15:30 |
| clarkb | awesome | 15:31 |
| *** ajmiller has quit IRC | 15:31 | |
| jeblair | yep, git ops are working | 15:31 |
| clarkb | I should probably go read the puppet now to try and sort out why we aren't cleaning those libs up properly | 15:31 |
| jeblair | web is up | 15:31 |
| *** krtaylor has quit IRC | 15:31 | |
| fungi | Powered by Gerrit Code Review (2.8.4-19-g4548330) | 15:31 |
| fungi | that's the right patched war | 15:31 |
| *** dizquierdo has quit IRC | 15:31 | |
| fungi | clarkb: it's also possible the names of the libs changed? the unpacked ones are bcprov-jdk16-144.jar and mysql-connector-java-5.1.21.jar | 15:32 |
| jeblair | sdague: rerecheck | 15:32 |
| *** tiswanso has quit IRC | 15:32 | |
| clarkb | fungi: i thought we were doing a puppet purge with a glob | 15:32 |
| clarkb | fungi: looking into it now | 15:32 |
| fungi | so if we were relying on puppet to delete those, then they might have been specified overly-specifically? | 15:32 |
| *** tiswanso has joined #openstack-infra | 15:32 | |
| clarkb | possible | 15:32 |
| *** erikmwilson has quit IRC | 15:33 | |
| clarkb | ok its the tidy at the end of the puppet-gerrit/manifests/init.pp class | 15:33 |
| clarkb | I wonder if its a regex and not a glob? | 15:34 |
| *** erikmwilson has joined #openstack-infra | 15:34 | |
| clarkb | to the puppet docs | 15:34 |
| *** sarob has joined #openstack-infra | 15:34 | |
| clarkb | nope https://docs.puppetlabs.com/references/3.stable/type.html#tidy-attribute-matches should be shell type file globs | 15:34 |
| jeblair | sdague: around? | 15:34 |
| *** erikmwilson has quit IRC | 15:35 | |
| *** TheJulia has quit IRC | 15:35 | |
| *** jamespage_ has joined #openstack-infra | 15:35 | |
| fungi | #status notice gerrit has been restarted to clear an issue with its event stream. any change events between 14:43-15:30 utc should be rechecked or have their approval votes reapplied to trigger jobs | 15:36 |
| openstackstatus | fungi: sending notice | 15:36 |
| jeblair | ttx: i think it might be worth rechecknig rc changes now | 15:36 |
| fungi | er, that was supposed to be 14:53-15:30 but close enough | 15:36 |
| *** ociuhandu has quit IRC | 15:36 | |
| *** zz_dimtruck is now known as dimtruck | 15:36 | |
| -openstackstatus- NOTICE: gerrit has been restarted to clear an issue with its event stream. any change events between 14:43-15:30 utc should be rechecked or have their approval votes reapplied to trigger jobs | 15:36 | |
| *** erikmwilson has joined #openstack-infra | 15:37 | |
| *** Swami has joined #openstack-infra | 15:37 | |
| openstackstatus | fungi: finished sending notice | 15:38 |
| ttx | jeblair: on my way | 15:38 |
| *** jamesmcarthur has quit IRC | 15:39 | |
| *** asselin has quit IRC | 15:39 | |
| *** jamespage_ has quit IRC | 15:39 | |
| *** dustins_ has quit IRC | 15:39 | |
| clarkb | more notes on the tidy. We don't install bcprov jar in review site, gerrit init seems to do that for us | 15:39 |
| sdague | jeblair: back, sorry was working on my linuxcon cfp while things were getting poked | 15:40 |
| jeblair | sdague: np, just wanted to alert you you can recheck | 15:41 |
| *** jamesmcarthur has joined #openstack-infra | 15:41 | |
| sdague | jeblair: thanks | 15:42 |
| *** sushilkm has joined #openstack-infra | 15:43 | |
| *** sushilkm has left #openstack-infra | 15:43 | |
| *** hdd__ has joined #openstack-infra | 15:43 | |
| jeblair | sdague: the results at the top of zuul status make me think the apt mirror problem is fixed | 15:44 |
| fungi | i'm going to take this opportunity to grab a quick shower, but will help come up with a puppet patch to bring review.o.o to sanity afterward so we can reenable puppet agent on it again | 15:44 |
| *** aarefiev has quit IRC | 15:44 | |
| jeblair | fungi: cool, thanks | 15:44 |
| jeblair | sdague: do you agree with that? | 15:44 |
| *** yolanda has quit IRC | 15:44 | |
| *** aarefiev has joined #openstack-infra | 15:44 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-gerrit: Run lib tidy after plugin install, before start https://review.openstack.org/178251 | 15:45 |
| clarkb | fungi: jeblair ^ I am sort of working on a hunch that that caused the issue. Either way I think my change is an improvment | 15:45 |
| *** yolanda has joined #openstack-infra | 15:45 | |
| jeblair | clarkb: do you think we saw the problem because we ended up with a differnt and incompatible version bundled with .19 but not in .7? | 15:46 |
| jeblair | er .17 | 15:46 |
| *** annegentle has joined #openstack-infra | 15:46 | |
| *** mpaolino has quit IRC | 15:47 | |
| clarkb | jeblair: I think the plugin installs may pull in the wrong stuff | 15:47 |
| sdague | jeblair: I'm not sure - https://jenkins06.openstack.org/job/gate-tempest-dsvm-neutron-src-python-glanceclient/111/console is in the gate now, and is a rax job, so if that passes stack.sh we're probably good | 15:47 |
| jeblair | oh, huh | 15:47 |
| clarkb | jeblair: so if we do gerrit-init, tidy libs, plugin installs, gerrit start we end up with the wrong stuff in the review site | 15:47 |
| *** hemnafk is now known as hemna | 15:47 | |
| *** MaxV has quit IRC | 15:47 | |
| clarkb | jeblair: but if we do gerrit-init, plugin installs, tidy, gerrit start we should avoid that | 15:47 |
| sdague | jeblair: so... it looks like that image bypassed the rax mirrors | 15:48 |
| clarkb | jeblair: though as I read more I am not sure we would tidy before doing a gerrit start | 15:48 |
| *** Guest69607 has quit IRC | 15:48 | |
| clarkb | jeblair: but we don't seem to have later run the tidy so I don't think that was the case | 15:49 |
| sdague | which means I think we're now isolated from rax mirror | 15:49 |
| jeblair | i'm going to 'status ok' then, sound good? | 15:49 |
| *** whoops has joined #openstack-infra | 15:49 | |
| clarkb | no opposition here | 15:50 |
| *** ajmiller has joined #openstack-infra | 15:50 | |
| sdague | jeblair: wfm | 15:50 |
| jeblair | #status ok | 15:51 |
| openstackstatus | jeblair: sending ok | 15:51 |
| mordred | sdague: woot! | 15:51 |
| jeblair | blank message since the last one was suggesting rechecks anyway | 15:52 |
| *** yfried has quit IRC | 15:52 | |
| clarkb | looking at http://puppetboard.openstack.org/report/review.openstack.org/afdb20cf974c2e2bca2bc47d3bbed678f019687e that doesn't seem to support my theory. I would've expected to see the gerrit-init and plugin install execs there | 15:53 |
| jeblair | and it sounds like we can declare the emergency over (i hope) | 15:53 |
| clarkb | fungi: did you disable those execs entirely? if so we may just not have properly cleaned up the old .17 env at all | 15:53 |
| openstackstatus | jeblair: finished sending ok | 15:53 |
| *** sergsh has quit IRC | 15:53 | |
| clarkb | also yay puppet and file permissions | 15:54 |
| *** adrian_otto has joined #openstack-infra | 15:55 | |
| mordred | jeblair: I love the smell of napalm in the morning etc etc | 15:55 |
| *** jeblair changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/manual/ http://ci.openstack.org/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/" | 15:55 | |
| *** jeblair sets mode: -o jeblair | 15:55 | |
| * jeblair lunches | 15:55 | |
| *** adrian_otto has left #openstack-infra | 15:55 | |
| *** krtaylor has joined #openstack-infra | 15:56 | |
| mordred | clarkb: while you're puppeting - https://review.openstack.org/#/c/178180/2 is one from this morning that we didnt' happen to get in | 15:56 |
| *** jamesmcarthur has quit IRC | 15:56 | |
| *** jcoufal_ has joined #openstack-infra | 15:57 | |
| *** cody-somerville has quit IRC | 15:58 | |
| *** doug-fish has quit IRC | 15:58 | |
| *** sdake_ has joined #openstack-infra | 15:59 | |
| *** julim has quit IRC | 16:00 | |
| *** jcoufal has quit IRC | 16:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate: Put input variables into ansible inventory https://review.openstack.org/177943 | 16:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate: Move all the ansible calls into playbooks https://review.openstack.org/177944 | 16:01 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Convert node_set_provision_state to task https://review.openstack.org/177987 | 16:01 |
| zaro | morning | 16:01 |
| *** alexpilotti has quit IRC | 16:01 | |
| *** tqtran has joined #openstack-infra | 16:01 | |
| *** jistr has quit IRC | 16:02 | |
| *** rbradfor has joined #openstack-infra | 16:03 | |
| *** sdake has quit IRC | 16:03 | |
| mordred | OH | 16:03 |
| *** sdake_ is now known as sdake | 16:03 | |
| *** annegentle has quit IRC | 16:03 | |
| *** hashar is now known as hasharMeeting | 16:04 | |
| fungi | clarkb: if you git diff in /etc/puppet/environments/fungi/modules/gerrit on the puppetmaster you'll see the very minor surgery i did to remove the lucene reindex command from the gerrit-init exec | 16:04 |
| mordred | clarkb: I'm reading some code elsewhere about neutron and floating ips ... and if I'm reading it right, all of our problems with how it works go away if we use neutronclient properly | 16:04 |
| *** jamesmcarthur has joined #openstack-infra | 16:04 | |
| mordred | clarkb: I'm going to verify | 16:04 |
| clarkb | fungi: ok, so I am not sure we ran the init at all then | 16:04 |
| fungi | clarkb: all i did was take out | 16:04 |
| clarkb | mordred: neutronclient does not solve the leak problem | 16:04 |
| fungi | ; /usr/bin/java -jar ${gerrit_war} reindex -d ${gerrit_site} | 16:04 |
| mordred | clarkb: depends on which problem we're talking about | 16:04 |
| clarkb | mordred: nor does it solve NAT, or the extra time it takes for another API round trip | 16:05 |
| mordred | nope | 16:05 |
| clarkb | mordred: so I don't know what problem you are talking about | 16:05 |
| *** harlowja_at_home has joined #openstack-infra | 16:05 | |
| mordred | clarkb: well, in this case doing the right thing with neutronclient would solve one of teh extra API calls (since we actually ahve to do 2 additional round trips, one for create, and one for attach) | 16:05 |
| clarkb | fungi: do you have the output from puppet on that run handy somewhere? | 16:05 |
| *** Ala has quit IRC | 16:05 | |
| *** jlanoux has quit IRC | 16:05 | |
| fungi | clarkb: what i find interesting though is that when i ran into this the first time i tried to upgrade it to the 2.8.4-19 war there were apparently those extra libs unpacked, but when i switched it back to 2.8.4-17 it cleaned them up properly | 16:06 |
| mordred | clarkb: in that you can apparently give neutron the fixed ip you want it associated with when you create it | 16:06 |
| *** peristeri has quit IRC | 16:06 | |
| *** otter768 has joined #openstack-infra | 16:06 | |
| mordred | clarkb: I'm going to hack up a quick test to see if the code I'm looking at a) works b) is useful to us - but it's hella cleaner than the thing we do now if it does | 16:06 |
| dansmith | jogo: sdague: can this go now? https://review.openstack.org/#/c/174567/ | 16:06 |
| dansmith | jogo: sdague: the nova patch that depends-on it is passing | 16:07 |
| sdague | dansmith: yes | 16:07 |
| fungi | clarkb: http://paste.openstack.org/show/210390 is from my terminal buffer | 16:07 |
| sdague | mtreinish: you want to look quick on that one? | 16:07 |
| *** ildikov has quit IRC | 16:07 | |
| *** jamesmcarthur_ has joined #openstack-infra | 16:07 | |
| dansmith | sdague: cool, thanks | 16:07 |
| clarkb | fungi: ok that shows the execs running, so puppetboard is just incomplate/wrong/whoknows | 16:07 |
| clarkb | fungi: however the tidy does not run | 16:08 |
| *** ddieterly has quit IRC | 16:08 | |
| fungi | clarkb: i wonder how/why it tidied again on downgrade last time though | 16:08 |
| jogo | dansmith: well mtreinish beat me too it | 16:09 |
| *** jamesmcarthur has quit IRC | 16:09 | |
| *** jamesmcarthur_ is now known as jamesmcarthur | 16:09 | |
| *** devvesa has quit IRC | 16:09 | |
| dansmith | jogo: better luck next time :) | 16:09 |
| dansmith | mtreinish: thanks | 16:09 |
| *** ddieterly has joined #openstack-infra | 16:10 | |
| clarkb | fungi: rereading tidy docs I don't see any problems with it. We recurse => true which is necessary to use matches. And the matches array items are OR'd not AND'd | 16:10 |
| *** unicell1 has quit IRC | 16:11 | |
| jogo | soo http://status.openstack.org/elastic-recheck/gate.html#1286818 | 16:11 |
| jogo | massive spike ^ | 16:11 |
| *** baoli has quit IRC | 16:11 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Revert "Revert "Update production gerrit to 2.8.4.19"" https://review.openstack.org/178269 | 16:11 |
| *** davideagnello has joined #openstack-infra | 16:11 | |
| *** otter768 has quit IRC | 16:11 | |
| clarkb | fungi: since we require both inits but I thoguht puppet did the correct thing with that. However in the case of going back to .17 that worked and I can't figure how to resolve that with requires being the problem | 16:11 |
| fungi | jogo: welcome to the fun! | 16:11 |
| sdague | jogo: dude, read the scrollback | 16:11 |
| jogo | o_O | 16:12 |
| *** yamahata has quit IRC | 16:12 | |
| fungi | clarkb: i agree. the only other possibility is that something is happening out of sequence, and the -17 war didn't actually bundle these libs | 16:13 |
| clarkb | fungi: and the base path looks correct /home/gerrit2/review_site/lib | 16:13 |
| openstackgerrit | Merged openstack-infra/project-config: Add retry message to gerrit-git-prep macro https://review.openstack.org/178180 | 16:13 |
| *** mpavone has quit IRC | 16:13 | |
| clarkb | fungi: I think next step is for me to run some puppet apply locally with a tidy manifest and see if I can reproduce | 16:14 |
| *** pcaruana has quit IRC | 16:14 | |
| jogo | fungi: ahh I see https://review.openstack.org/#/c/178160/4 | 16:15 |
| *** davideagnello has quit IRC | 16:15 | |
| *** mmmpork has joined #openstack-infra | 16:16 | |
| *** ajmiller has quit IRC | 16:17 | |
| clarkb | fungi: something like http://paste.openstack.org/show/210419/ | 16:17 |
| fungi | clarkb: that looks like a rough approximation yes' | 16:18 |
| *** claudiub has quit IRC | 16:19 | |
| clarkb | fungi: and that appears to work | 16:20 |
| clarkb | I see tidy say it is removing files then ls shows they are gone | 16:20 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Improve run_time graph formatting https://review.openstack.org/178276 | 16:21 |
| *** isviridov_away is now known as isviridov | 16:21 | |
| fungi | huh | 16:21 |
| fungi | this is puzzling | 16:21 |
| clarkb | added in the other matches to seeif that broke puppet too and it does not | 16:23 |
| zaro | yolanda: is change 75514 working for you? | 16:23 |
| *** lucap has quit IRC | 16:23 | |
| zaro | yolanda: i just tried running it and still getting the same error. | 16:23 |
| fungi | clarkb: so one theory is that requiring both gerrit-initial-init and gerrit-init even though we only exec the latter is the cause for not running the tidy after? | 16:23 |
| zaro | yolanda: ran with PS 30 | 16:23 |
| clarkb | fungi: correct | 16:23 |
| yolanda | zaro, i tested with jenkins-jobs test, it may be skipping some bits? | 16:24 |
| clarkb | fungi: I had thought that wouldn't matter because both execs are evaluated, its just that one does not actually fork | 16:24 |
| clarkb | fungi: and that had been sufficient to satisfy requires in the past | 16:24 |
| fungi | oh, right, i see that now | 16:24 |
| zaro | i'm getting the exact same error as i posted on PS 27 | 16:24 |
| clarkb | fungi: its possible newer puppet breaks that behavior? | 16:25 |
| clarkb | fungi: they may have optimized that node out of the graph because they know it won't exec | 16:25 |
| yolanda | zaro, let me try again, i can try with a real update instead of test | 16:25 |
| zaro | yolanda: maybe my local cache is messed up. let me try to kill my jjb cach | 16:25 |
| *** jamesmcarthur_ has joined #openstack-infra | 16:26 | |
| *** ivar-lazzaro has joined #openstack-infra | 16:26 | |
| yolanda | zaro, i'm using jenkins-jobs -l WARN test --workers 4 ../project-config/jenkins/jobs | 16:26 |
| *** ivar-lazzaro has quit IRC | 16:27 | |
| yolanda | and no errors | 16:27 |
| fungi | clarkb: interesting data point, on review-dev (which is running 2.10 so ymmv) there is a duplicate mysql-connector-java besides the symlink, but no bcprov symlink at all | 16:27 |
| *** jamesmcarthur has quit IRC | 16:27 | |
| *** jamesmcarthur_ is now known as jamesmcarthur | 16:27 | |
| *** ivar-lazzaro has joined #openstack-infra | 16:28 | |
| clarkb | fungi: for 2.10 and bcprov we had to stop using the system package | 16:28 |
| clarkb | fungi: system package was not new enough | 16:28 |
| zaro | yolanda: looks like that was it. deleted my local cache and it works now. | 16:28 |
| *** sputnik13 has joined #openstack-infra | 16:29 | |
| zaro | yolanda: hmm, will need to test that. | 16:29 |
| yolanda | zaro, darragh was writing some script for testing threads | 16:29 |
| cinerama | pleia2: hi there | 16:30 |
| *** mtanino_ has joined #openstack-infra | 16:30 | |
| *** isviridov is now known as isviridov_away | 16:31 | |
| zaro | yolanda: anyways, i ran some test for that change already mostly looks good. Although i did come across the same error i pointed out in PS 19, "ValueError: too many values to unpack" | 16:32 |
| *** mtanino has quit IRC | 16:33 | |
| *** ivar-laz_ has joined #openstack-infra | 16:33 | |
| *** harlowja_at_home has quit IRC | 16:33 | |
| zaro | yolanda: it only happened on one of my test runs so i wanted to test some before apprv | 16:33 |
| clarkb | fungi: my change should address the requires issue if it is an issue, and it will make sure we tidy before starting the service so thats also good | 16:33 |
| clarkb | fungi: I am just not convinced it solves all the problems here | 16:33 |
| zaro | yolanda: mostly i think it's just more testing at this point to make sure it's solid. | 16:34 |
| fungi | clarkb: oh, it's possible the service start failed and the tidy didn't happen until after i suppose | 16:36 |
| *** ssam2 has quit IRC | 16:36 | |
| *** ivar-lazzaro has quit IRC | 16:36 | |
| *** emagana_ has joined #openstack-infra | 16:36 | |
| yolanda | zaro, what are you using for testing? apart from unit tests and doing a test run on projects.yaml? | 16:37 |
| *** claudiub has joined #openstack-infra | 16:37 | |
| *** yamahata has joined #openstack-infra | 16:38 | |
| *** tiswanso has quit IRC | 16:38 | |
| fungi | hrm, only gets fired from a notify though, so nothing's depending on it in the puppet sense | 16:38 |
| pleia2 | cinerama: hey | 16:39 |
| *** emagana has quit IRC | 16:39 | |
| *** tiswanso_ has joined #openstack-infra | 16:39 | |
| clarkb | fungi: ya so even if that failed puppet should continue and run things that don't depend on it | 16:39 |
| zaro | yolanda: i'm just run a script that executes the update cmd with the following params: no worker specified, workers=0, workers=2, workers=4, workers=8 on a 4 cpu VM against a jenkins master. | 16:39 |
| zaro | yolanda: then i just validate the timestamp from the test run | 16:40 |
| cinerama | pleia2: hi there. so i'm having a think about how i want to structure the zanata client stuff. basically we need to run the client's 'stats' command to get what translations we have available & their percentage completion | 16:40 |
| yolanda | zaro, and in which case did you see that error? with some specific workers setting, or just happened on some random run? | 16:41 |
| cinerama | pleia2: so for the proposal scripts we have a file of common functions in bash & we can use some of those, but i'm doing the additional template generation and stats command result processing in python because that's a bit easier | 16:41 |
| pleia2 | cinerama: wfm | 16:41 |
| *** lucap has joined #openstack-infra | 16:42 | |
| clarkb | and the tidy doesn't appear wrapped in any conditionals | 16:42 |
| zaro | yolanda: the error i pointed out in PS 27 is from a messed up cache. | 16:43 |
| cinerama | pleia2: so i'm tossing up a bit on whether we should just create new bash scripts that call the python bits where needed | 16:43 |
| fungi | yeah, the only theory i have is that it's because of the require block | 16:43 |
| pleia2 | cinerama: I don't know how we currently track whether something has been downloaded and prepped for gerrit, so it doesn't keep proposing the same patches that are over 75% translated but does submit changes as they happen | 16:43 |
| clarkb | fungi: ya, so let me change my commit message since I basically know the theory is wrong there now, but the change itself is good for other reasons | 16:43 |
| yolanda | zaro, and the too many values to unpack from ps19? are you still seeing that? | 16:43 |
| fungi | also i missed your proposed change. i wonder if openstackgerrit is struggling | 16:44 |
| zaro | yolanda: not happening anymore because i cleaned out my cache. but will need to verify that cache from a run against master will work with this change. | 16:44 |
| cinerama | pleia2: unfortunately i need to use the processed data from stats in a couple places and it's a bit slow to connect & retrieve that. i could pickle it in a temp file & reuse it | 16:44 |
| yolanda | ah, good point | 16:44 |
| *** che-arne has quit IRC | 16:44 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-gerrit: Run lib tidy after plugin install, before start https://review.openstack.org/178251 | 16:44 |
| clarkb | fungi: ^ it was first pushed when you stepped out, the bot did report it here | 16:44 |
| fungi | ahh good. also zuul's still receiving gerrit events, so we're not stuck again (yet anyway) | 16:44 |
| zaro | yolanda: now i'm seeing this error: AttributeError: 'Namespace' object has no attribute 'name' | 16:45 |
| *** dboik_ has quit IRC | 16:45 | |
| jeblair | cinerama, pleia2: we can't count on keeping data around between jobs on the proposal slave (not sure that's what you were suggesting). partly because we want to eventually run those jobs on single-use slaves. | 16:46 |
| pleia2 | cinerama: and I think it's fine to create new scripts entirely for this | 16:46 |
| yolanda | zaro, that comes from a publisher.. is that happening only with that change, or also in the original jjb? | 16:46 |
| *** lucap has quit IRC | 16:46 | |
| pleia2 | jeblair: right, good to know, will have to poke into how state is remembered now | 16:46 |
| cinerama | jeblair: not between jobs, just within the context of that particular job | 16:46 |
| jeblair | cinerama: ok cool | 16:47 |
| jeblair | pleia2: i think state is remembered in gerrit | 16:47 |
| *** unicell has joined #openstack-infra | 16:47 | |
| jeblair | via queries for open changes -- if there is an open change for something, proposal bot updates it with a new patchset | 16:47 |
| yolanda | zaro, i need to setup a working jenkins to test live there myself, i had one but it's not working anymore | 16:47 |
| zaro | yolanda: it's coming from cmd.py | 16:47 |
| fungi | clarkb: interestingly, it's the gerrit-init exec which is downloading the files we want to tidy, but i guess install-core-plugins doesn't fire until notified by either gerrit-init or gerrit-initial-init so should work out | 16:48 |
| yolanda | zaro, can you paste it? | 16:48 |
| pleia2 | jeblair: ah, interesting | 16:48 |
| zaro | yolanda: take a look at the comment in gerrit. | 16:48 |
| clarkb | fungi: yup, it basically map reduces the two inits for us | 16:48 |
| fungi | as long as we keep an eye out for refactors that might change that relationship | 16:48 |
| yolanda | k | 16:49 |
| zaro | yolanda: yeah, since this is all about the update command i would suggest testing it because our CI won't | 16:49 |
| clarkb | sdague: any idea why https://jenkins02.openstack.org/job/gate-devstack-unit-tests/360/console failed? those unittests don't seem to say much | 16:50 |
| yolanda | zaro, can you pass me that cache file in some way? | 16:50 |
| zaro | pelix, yolanda : still wondering why --workers is a param for 'test' command? its a noop for test right? | 16:50 |
| *** tjones1 has joined #openstack-infra | 16:50 | |
| *** gyee has joined #openstack-infra | 16:51 | |
| zaro | yolanda: hmm maybe dropbox? let me take a look at how to do that. | 16:51 |
| yolanda | or email | 16:51 |
| *** yfried has joined #openstack-infra | 16:51 | |
| *** patrickeast has joined #openstack-infra | 16:51 | |
| *** e0ne has quit IRC | 16:52 | |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Install kpartx for DIB tests https://review.openstack.org/178283 | 16:53 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: WiP: Add keystone services/endpoints methods https://review.openstack.org/177621 | 16:54 |
| openstackgerrit | Louis Taylor proposed openstack-infra/project-config: Add functional test job for python-glanceclient https://review.openstack.org/178285 | 16:55 |
| pabelanger | Is anybody running a public (filtered) iCal feed for infra meetings? The iCal feed for all projects is a little noisy for me | 16:56 |
| clarkb | pabelanger: I just put 1900UTC tuesdays on my calendar directly. It hasn't changed since I started attending that meeting | 16:56 |
| fungi | pabelanger: hrm... we have one infra meeting at the same time every week and it hasn't changed in at least 3 years | 16:56 |
| fungi | so not sure what the benefit of an ical for that would be | 16:57 |
| *** julim has joined #openstack-infra | 16:57 | |
| *** lucap has joined #openstack-infra | 16:57 | |
| *** jamesmcarthur has quit IRC | 16:57 | |
| jeblair | pabelanger: i think that's a feature of the upcoming yaml2ical stuff | 16:58 |
| jeblair | pabelanger: so hopefully after the summit? | 16:58 |
| *** dboik has joined #openstack-infra | 16:58 | |
| *** baoli has joined #openstack-infra | 16:58 | |
| *** dboik has quit IRC | 16:58 | |
| clarkb | sdague: I see the error now, could not determine host ip address | 16:59 |
| clarkb | sdague: going to guess that is a difference with hpcloud left undetected by not testing there for a while | 16:59 |
| *** ildikov has joined #openstack-infra | 16:59 | |
| *** ZZelle_ has joined #openstack-infra | 17:00 | |
| *** dustins has joined #openstack-infra | 17:00 | |
| *** davideagnello has joined #openstack-infra | 17:01 | |
| fungi | clarkb: so if we get https://review.openstack.org/178269 approved i can reenable puppet agent on review.openstack.org soonish | 17:01 |
| clarkb | sdague: it looks like devstack is interferring with hpcloud networking on 10.0.0.0/24 | 17:01 |
| clarkb | fungi: looking | 17:02 |
| yolanda | thx zaro | 17:02 |
| fungi | zaro: thoughts on https://review.openstack.org/178251 before it gets approved? | 17:02 |
| fungi | nibalizer: ^ if you're around | 17:02 |
| *** hasharMeeting has quit IRC | 17:03 | |
| pabelanger | clarkb, fungi, jeblair: Cool, that is what I am doing too. Wanted to see if something public was available first. | 17:03 |
| fungi | seems like it should be safe enough, though we're not going to see it in action before the next time we update a gerrit war | 17:03 |
| sdague | clarkb: pointer? | 17:04 |
| *** dboik has joined #openstack-infra | 17:04 | |
| *** dboik has quit IRC | 17:04 | |
| sdague | I thought we forced fixed network to a different range | 17:04 |
| *** yfried is now known as yfried|prtially_ | 17:05 | |
| *** dboik has joined #openstack-infra | 17:05 | |
| *** ivar-laz_ has quit IRC | 17:05 | |
| nibalizer | hi | 17:05 |
| *** ivar-lazzaro has joined #openstack-infra | 17:05 | |
| nibalizer | ya looks good | 17:06 |
| *** unicell has quit IRC | 17:06 | |
| *** unicell1 has joined #openstack-infra | 17:06 | |
| *** marun has joined #openstack-infra | 17:07 | |
| *** rvasilets_ has left #openstack-infra | 17:07 | |
| *** jcoufal_ is now known as jcoufal | 17:07 | |
| *** nmagnezi has joined #openstack-infra | 17:08 | |
| *** ivar-laz_ has joined #openstack-infra | 17:09 | |
| *** ir2ivps has quit IRC | 17:09 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Set FIXED_RANGE in devstack unittests https://review.openstack.org/178294 | 17:10 |
| clarkb | sdague: ^ I don't think you do but that change should force it | 17:10 |
| clarkb | sdague: https://jenkins02.openstack.org/job/gate-devstack-unit-tests/360/console is the log from the failure | 17:10 |
| *** ivar-lazzaro has quit IRC | 17:11 | |
| sdague | clarkb: can we fix that in the unit tests instead? | 17:11 |
| clarkb | sdague: sure, its just that you likely won't have a range in unittests that will always work, but I can set a fixed range for our env that will | 17:12 |
| zaro | yolanda: email was sent to you | 17:12 |
| yolanda | yep, received it | 17:12 |
| yolanda | i'm fighting with my jenkinsmaster test now | 17:13 |
| sdague | clarkb: so we don't need to source openrc | 17:14 |
| zaro | fungi: i think i will -1 that change | 17:14 |
| *** dannywilson has joined #openstack-infra | 17:14 | |
| openstackgerrit | Merged openstack-infra/puppet-logstash: Modernize kibana vhost template https://review.openstack.org/153819 | 17:14 |
| *** tonytan4ever has quit IRC | 17:15 | |
| clarkb | sdague: gotcha, I have never looked at devstack unittests until about 5 minutes ago, so if that is possible I don't know | 17:15 |
| sdague | yeh, I removed a similar thing a few weeks ago | 17:16 |
| sdague | https://review.openstack.org/178297 | 17:16 |
| sdague | that should fix that issue in the fail | 17:16 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: remove cookbook-pacemaker from infra https://review.openstack.org/178298 | 17:16 |
| clarkb | sdague: oh I rechecked thinking that was several weeks old, thats a new change | 17:17 |
| sdague | ok, I'm going to get out for a bike ride before meetings galore | 17:17 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: move gate-.*-chef-rake job and run it branch specific https://review.openstack.org/176674 | 17:17 |
| sdague | clarkb: yes, I just wrote that one :) | 17:17 |
| sdague | but the other one | 17:17 |
| sdague | git grep HOST | 17:17 |
| sdague | test_libs_from_pypi.sh:# we don't actually care about the HOST_IP | 17:17 |
| sdague | test_libs_from_pypi.sh:HOST_IP="don't care" | 17:17 |
| *** markvoelker has quit IRC | 17:18 | |
| sdague | which let me drop openrc in that file | 17:18 |
| jklare | still firefighting? | 17:18 |
| *** harlowja_away is now known as harlowja_ | 17:21 | |
| *** sandywalsh has quit IRC | 17:23 | |
| *** dannywilson has quit IRC | 17:23 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: remove cookbook-pacemaker from infra https://review.openstack.org/178298 | 17:23 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Allow complex filtering with embedded dicts https://review.openstack.org/178299 | 17:23 |
| *** dannywilson has joined #openstack-infra | 17:24 | |
| *** yfried|prtially_ is now known as yfried | 17:25 | |
| *** jamesmcarthur has joined #openstack-infra | 17:26 | |
| *** wenlock has quit IRC | 17:27 | |
| *** HeOS has quit IRC | 17:27 | |
| *** e0ne has joined #openstack-infra | 17:28 | |
| *** hdd__ has quit IRC | 17:28 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Add keystoneauth library and testing infrastructure https://review.openstack.org/175596 | 17:29 |
| fungi | zaro: what problem did you spot on 178251? | 17:29 |
| *** claudiub has quit IRC | 17:29 | |
| *** luqas has quit IRC | 17:29 | |
| fungi | i don't see your -1 | 17:29 |
| yolanda | zaro, same error as you | 17:29 |
| *** luqas has joined #openstack-infra | 17:29 | |
| clarkb | jklare: the majority of fires are out now we are trying to sort out some remaining issues like why our gerrit upgrades don't work completely as expected | 17:30 |
| clarkb | we should also start collecting numbers on git fetch retries with mordreds patch in | 17:30 |
| morganfainberg | fungi, anteaya, clarkb, pleia2: ^ 175596, talked with jeblair and was told that should/can merge before governance change | 17:30 |
| morganfainberg | and sooner will go a long way to getting us moving on that. | 17:31 |
| morganfainberg | when you're not fighting fires that is | 17:31 |
| zaro | fungi: still reviewing. i think the use of both require & beofre is confusing | 17:31 |
| clarkb | morganfainberg: devils advocate, why wouldn't you just import keystoneclient.auth and have that be lightweight? | 17:32 |
| morganfainberg | clarkb: because keystoneclient has a lot of extra dependencies | 17:32 |
| morganfainberg | clarkb: keystoneauth is meant to be very light, trimmed dependencies that is relevant for servers, clients, or SDK w/o needing to import / depend on keystoneclient | 17:33 |
| *** jogo has quit IRC | 17:33 | |
| morganfainberg | eliminates pysaml2 for example as a dep | 17:33 |
| yolanda | zaro, it was a typo, testing again and i'll push it | 17:33 |
| yolanda | now testing on a live server | 17:33 |
| *** yfried is now known as yfried|away | 17:33 | |
| morganfainberg | and keystoneclient will continue to depend on extra things like oslo.serialization, where keystoneauth probabably wont (if we can avoid it) | 17:33 |
| zaro | fungi, nibalizer :commented on 178251 | 17:35 |
| *** ir2ivps has joined #openstack-infra | 17:35 | |
| zaro | yolanda: here's the scrip i used for testing: http://paste.openstack.org/show/210518/ | 17:36 |
| yolanda | nice, let me test with that | 17:37 |
| *** jamesmcarthur has quit IRC | 17:37 | |
| clarkb | morganfainberg: ok so you are concerned about dependencies | 17:37 |
| morganfainberg | clarkb: yes. and keeping separation of concerns. | 17:38 |
| *** melwitt has joined #openstack-infra | 17:38 | |
| morganfainberg | clarkb: ksc becomes more "management of IAM stuff" and keystoneauth is "authn/authz" specifc. | 17:38 |
| morganfainberg | but the dependencies is the #1 reason | 17:38 |
| adam_g | huh? when did gerrit start accepting changes without a Change-Id footer? | 17:39 |
| morganfainberg | adam_g: it shouldn't accept them - unless it was imported [initial import] | 17:39 |
| morganfainberg | adam_g: afaik | 17:39 |
| fungi | adam_g: it's a configurable option per project. have an example? | 17:39 |
| adam_g | morganfainberg, yeah thats what i thought | 17:39 |
| pleia2 | morganfainberg: had a comment/question inline | 17:39 |
| morganfainberg | pleia2: looking | 17:40 |
| adam_g | fungi, ive ended up with a bunch of dupes in akanda-appliance-builder, where is that toggled per project? | 17:40 |
| morganfainberg | pleia2: sure. | 17:40 |
| * mordred supports keystoneauth with low dependencies | 17:40 | |
| fungi | adam_g: i think it's turned on for all projects, but will check that one | 17:40 |
| morganfainberg | pleia2: can remove those. | 17:40 |
| morganfainberg | pleia2: let me respin addressing that | 17:40 |
| yolanda | wow, so zaro, reason of my typo | 17:41 |
| yolanda | http://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/tree/jenkins_jobs/cmd.py | 17:41 |
| yolanda | look | 17:41 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Add keystoneauth library and testing infrastructure https://review.openstack.org/175596 | 17:41 |
| yolanda | subparser: update has option "names" | 17:41 |
| morganfainberg | pleia2: ^ | 17:41 |
| yolanda | subparser: test has option "name" | 17:42 |
| pleia2 | morganfainberg: thanks | 17:42 |
| yolanda | i guess we should stick with "names" ? | 17:42 |
| morganfainberg | pleia2: np! | 17:42 |
| fungi | adam_g: looks like the acl needs fixing | 17:42 |
| clarkb | ok I am really stumped on the puppet thing and think we should give my change a go and if we still see this problem my change should make debugging easier | 17:42 |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Revert "Update production gerrit to 2.8.4.19"" https://review.openstack.org/178269 | 17:42 |
| *** luqas has quit IRC | 17:42 | |
| yolanda | zaro, that's a different problem, but is showing up on testing | 17:43 |
| *** yfried|away is now known as yfried | 17:43 | |
| fungi | adam_g: see unfortunate tyop at http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/stackforge/akanda.config#n11 | 17:43 |
| *** jogo has joined #openstack-infra | 17:44 | |
| clarkb | though if you use git-review it should always add one for you | 17:44 |
| clarkb | so protip use git-review? | 17:44 |
| clarkb | oh except maybe not for a stack | 17:44 |
| clarkb | we can fix that in git review though | 17:44 |
| adam_g | fungi, an unfortunate tyop indeed! | 17:44 |
| adam_g | clarkb, yeah | 17:44 |
| *** tjones1 has quit IRC | 17:45 | |
| fungi | adam_g: i'll patch it. i find it cargo-culted in three other acls too | 17:45 |
| adam_g | fungi, thanks | 17:45 |
| *** lucap has quit IRC | 17:46 | |
| clarkb | fungi: I am checking all-projects for that acl | 17:46 |
| clarkb | fungi: if its not there we should probably add it | 17:46 |
| fungi | clarkb: probably | 17:46 |
| fungi | it's inheriting false from all-projects | 17:46 |
| clarkb | ya its set to false in all-projects | 17:46 |
| clarkb | we can probably avoid this trouble by setting that to true | 17:46 |
| *** tonytan4ever has joined #openstack-infra | 17:47 | |
| clarkb | need to think about why that may be a bad idea | 17:47 |
| zaro | yolanda: i guess 'names' makes more sense. | 17:47 |
| mordred | ++ | 17:47 |
| zaro | yolanda: would be best to go with names and deprecate 'name' | 17:47 |
| zaro | yolanda: but should support both for now so it won't break when users update | 17:48 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Fix mistyped requrieChangeId in Gerrit ACLs https://review.openstack.org/178307 | 17:48 |
| openstackgerrit | Merged openstack/requirements: create a separate section for pinned requirements https://review.openstack.org/177193 | 17:48 |
| *** markvoelker has joined #openstack-infra | 17:49 | |
| *** peristeri has joined #openstack-infra | 17:49 | |
| yolanda | zaro, ok, it will need another change to fix that | 17:50 |
| *** jamesmcarthur has joined #openstack-infra | 17:50 | |
| *** tjones1 has joined #openstack-infra | 17:51 | |
| *** wenlock has joined #openstack-infra | 17:51 | |
| *** dustins has quit IRC | 17:53 | |
| *** arxcruz has quit IRC | 17:53 | |
| *** markvoelker has quit IRC | 17:54 | |
| *** wenlock_ has joined #openstack-infra | 17:54 | |
| *** annegentle has joined #openstack-infra | 17:54 | |
| *** wenlock has quit IRC | 17:54 | |
| fungi | zaro: clarkb: nibalizer: does my comment on 178251 about using a require in gerrit-start make sense at all? | 17:55 |
| clarkb | fungi: ya thats another option | 17:55 |
| *** SergK has joined #openstack-infra | 17:56 | |
| *** wenlock has joined #openstack-infra | 17:56 | |
| fungi | if i'm interpreting zaro's concerns correctly, using require instead of before/after makes it easier to follow what the intended order is | 17:57 |
| *** sushilkm has joined #openstack-infra | 17:57 | |
| nibalizer | zaro: 'before' means 'I go before this other thingg' and require means "that other thing goes before me" | 17:57 |
| nibalizer | fungi: if you want to use only require thats super ok with me | 17:57 |
| nibalizer | keeping it consisent has value i think | 17:57 |
| fungi | nibalizer: it's not bothering me, but i do take zaro's concerns seriously too | 17:58 |
| zaro | yes, valuable when you need to debug | 17:58 |
| *** achanda has joined #openstack-infra | 17:58 | |
| *** e0ne has quit IRC | 17:58 | |
| nibalizer | ya so switch to all requires | 17:58 |
| *** ir2ivps has quit IRC | 17:58 | |
| *** spredzy_|afk is now known as spredzy_ | 17:59 | |
| *** ildikov has quit IRC | 17:59 | |
| clarkb | I can update the change in a moment, finishing an email first | 17:59 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: Create rubygems mirror from rubygems.org https://review.openstack.org/178026 | 17:59 |
| *** sdake has quit IRC | 17:59 | |
| *** abregman has quit IRC | 17:59 | |
| *** sdake has joined #openstack-infra | 18:00 | |
| *** lucap has joined #openstack-infra | 18:00 | |
| *** tiswanso_ has quit IRC | 18:00 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Removed angular-eslint https://review.openstack.org/178312 | 18:00 |
| *** dboik_ has joined #openstack-infra | 18:01 | |
| *** tiswanso_ has joined #openstack-infra | 18:01 | |
| *** weshay has quit IRC | 18:02 | |
| *** melwitt has quit IRC | 18:02 | |
| *** melwitt has joined #openstack-infra | 18:03 | |
| *** melwitt has quit IRC | 18:03 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Update to search UI. https://review.openstack.org/178003 | 18:03 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Renamed result-set-size directive https://review.openstack.org/178004 | 18:03 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Result set paging update. https://review.openstack.org/178005 | 18:03 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Allow complex filtering with embedded dicts https://review.openstack.org/178299 | 18:04 |
| *** jamielennox|away is now known as jamielennox | 18:04 | |
| *** dboik has quit IRC | 18:04 | |
| *** MaxV has joined #openstack-infra | 18:04 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-gerrit: Run lib tidy after plugin install, before start https://review.openstack.org/178251 | 18:04 |
| clarkb | fungi: zaro ^ I moved the tidy too so that if you are doing a top to bottom read of the file it flows well | 18:05 |
| *** jogo has quit IRC | 18:06 | |
| *** melwitt has joined #openstack-infra | 18:06 | |
| *** dtantsur is now known as dtantsur|afk | 18:06 | |
| *** weshay has joined #openstack-infra | 18:07 | |
| *** otter768 has joined #openstack-infra | 18:07 | |
| *** jogo has joined #openstack-infra | 18:07 | |
| *** TheJulia has joined #openstack-infra | 18:07 | |
| *** annegentle has quit IRC | 18:09 | |
| *** jamesmcarthur has quit IRC | 18:09 | |
| pabelanger | So, are our DIBs accessible from a public URL before being moved into the cloud? Specifically, the devstack-centos7-dib? I'd rather just consume that directly from -infra, if possible, then build my own. | 18:10 |
| *** annegentle has joined #openstack-infra | 18:10 | |
| fungi | looking over a --noop puppet run on review.o.o it appears to be safe to reenable the agent on it, so i'm doing that now | 18:12 |
| *** otter768 has quit IRC | 18:12 | |
| fungi | pabelanger: not yet. we're looking into good solutions for publishing them | 18:12 |
| pabelanger | fungi, roger | 18:12 |
| *** ildikov has joined #openstack-infra | 18:12 | |
| fungi | pabelanger: we're also only using then for some images in some providers at the momenty | 18:13 |
| *** doug-fish has joined #openstack-infra | 18:13 | |
| clarkb | it should be trivial to build your own if you have about 8GB of disk, some network bandwidht and an hour of time | 18:13 |
| pabelanger | clarkb, agreed. Was more curious if anything was public before going down that path. | 18:13 |
| fungi | and really, it's your computer that needs the hour of time. you can go do something else while that runs ;_ | 18:13 |
| clarkb | in openstack-infra/project-config/tools/ there is a build-image.sh script which sets things up, you just have to override the default of ubuntu to get centos7 or fedora | 18:13 |
| fungi | for example, desk chair jousting in the office hallways | 18:14 |
| pabelanger | time to stock up on nerf supplies | 18:15 |
| fungi | okay, puppet on review.o.o is reenabled and up to date now | 18:16 |
| fungi | we return you to your regularly scheduled (computer) programming | 18:16 |
| *** cdent has quit IRC | 18:17 | |
| *** MaxV has quit IRC | 18:17 | |
| *** emagana_ has quit IRC | 18:18 | |
| zaro | clarkb: LGTM | 18:18 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: move gate-.*-chef-rake job and run it branch specific https://review.openstack.org/176674 | 18:18 |
| *** ivar-laz_ has quit IRC | 18:19 | |
| *** tqtran has quit IRC | 18:19 | |
| mordred | pabelanger: it probably does not matter - but please be aware that our SSH keys are on all of those images | 18:20 |
| mordred | pabelanger: so if/when we do start publishing to something other than glance and you do decide to directly reuse them ... just know that we'll be able to ssh in to them :) | 18:21 |
| *** ivar-lazzaro has joined #openstack-infra | 18:21 | |
| *** lucap has quit IRC | 18:21 | |
| *** emagana has joined #openstack-infra | 18:22 | |
| pabelanger | mordred, Ya, was thinking about that too. | 18:22 |
| *** pabelanger has quit IRC | 18:24 | |
| *** ir2ivps has joined #openstack-infra | 18:24 | |
| *** jcoufal has quit IRC | 18:24 | |
| *** pabelanger has joined #openstack-infra | 18:25 | |
| *** pabelanger has quit IRC | 18:25 | |
| *** pabelanger has joined #openstack-infra | 18:26 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Updated dependencies https://review.openstack.org/178312 | 18:26 |
| *** pabelanger has quit IRC | 18:26 | |
| *** pabelanger has joined #openstack-infra | 18:27 | |
| clarkb | sdague: can you comment on os-loganalyze functional testing as a devstack plugin (re 173328?) because I am not sure I udnerstand why devstack should be involved at all | 18:27 |
| clarkb | if we want functional tests we can just spin up an apache with some files | 18:28 |
| *** bswartz has quit IRC | 18:28 | |
| clarkb | or are you wanting swift from devstack? | 18:28 |
| mtreinish | clarkb: is that for devstack running os-loganalyze? (I thought that was added at one point) or is it something else? | 18:28 |
| clarkb | mtreinish: its to test os-loganalyze | 18:28 |
| *** guest_____ has joined #openstack-infra | 18:29 | |
| clarkb | trying to understand what devstack gives us and a swift install is the only thing I can think of | 18:29 |
| *** dustins has joined #openstack-infra | 18:29 | |
| mtreinish | ah, I see now. Misread your original comment. Yeah I think it would be just to provide swift (and probably keystone too) | 18:30 |
| krotscheck | clarkb: I have completely forgotten about the status of your vinz pitch. Any updates there? | 18:30 |
| clarkb | krotscheck: gave it last night think it went ok | 18:31 |
| clarkb | krotscheck: should know in a week or two if it will go beyond that | 18:31 |
| krotscheck | clarkb: What's the competition look like? | 18:31 |
| clarkb | krotscheck: mozilla had a firefoxos thing and a research group had some high performance db stuff | 18:31 |
| *** dustins has quit IRC | 18:34 | |
| *** achanda has quit IRC | 18:34 | |
| *** achanda has joined #openstack-infra | 18:35 | |
| *** guest_____ has quit IRC | 18:35 | |
| mtreinish | clarkb: http://logs.openstack.org/76/178276/1/check/gate-subunit2sql-python27/3f03cbc/console.html#_2015-04-28_16_38_00_670 that failed trying to run the first migration. Could we have failed a db cleanup on the test box before? | 18:36 |
| openstackgerrit | Merged openstack/requirements: Add cap for tempest-lib so reqs syncs don't break the world https://review.openstack.org/177306 | 18:36 |
| mtreinish | although I thought I made the tests drop existing dbs before it ran | 18:36 |
| clarkb | mtreinish: the test machine should be a pristine new VM just for you without any stale dbs | 18:36 |
| mtreinish | clarkb: hmm, then I have no idea | 18:37 |
| *** Longgeek has quit IRC | 18:37 | |
| mtreinish | oh, unless it's trying to upgrade twice. That would be something | 18:37 |
| mtreinish | oh, yep that's what it is, it's trying to use mysql on the postgres test | 18:38 |
| mtreinish | I bet it hasn't been running postgres this whole time and it was just racing, and if mysql was second it would delete the db from the postgres test... | 18:38 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Update to search UI. https://review.openstack.org/178003 | 18:39 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Renamed result-set-size directive https://review.openstack.org/178004 | 18:39 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Result set paging update. https://review.openstack.org/178005 | 18:39 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix mistyped requrieChangeId in Gerrit ACLs https://review.openstack.org/178307 | 18:39 |
| *** yfried is now known as yfried|afk | 18:40 | |
| *** tjones1 has quit IRC | 18:41 | |
| *** achanda has quit IRC | 18:41 | |
| *** pabelanger has quit IRC | 18:42 | |
| *** xyang1 has joined #openstack-infra | 18:44 | |
| *** krtaylor has quit IRC | 18:46 | |
| *** weshay has quit IRC | 18:48 | |
| *** Rockyg has joined #openstack-infra | 18:49 | |
| *** markvoelker has joined #openstack-infra | 18:49 | |
| mordred | AJaeger: I LOVE that requrieChangeId is misspelled in that commit message | 18:50 |
| *** med_ has quit IRC | 18:50 | |
| lifeless | is there some performance issue within nodes atm? my pbr changes are timing out | 18:50 |
| lifeless | they weren't while they were being developed | 18:50 |
| fungi | AJaeger: thanks for updating that | 18:51 |
| *** med_ has joined #openstack-infra | 18:51 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 18:51 |
| *** med_ has quit IRC | 18:51 | |
| *** med_ has joined #openstack-infra | 18:51 | |
| *** weshay has joined #openstack-infra | 18:52 | |
| *** pabelanger has joined #openstack-infra | 18:52 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Explicitly say when job times out https://review.openstack.org/178330 | 18:52 |
| *** pabelanger has quit IRC | 18:53 | |
| *** pabelanger has joined #openstack-infra | 18:53 | |
| *** markvoelker has quit IRC | 18:54 | |
| *** hdd has joined #openstack-infra | 18:54 | |
| *** lucap has joined #openstack-infra | 18:56 | |
| *** lucap has quit IRC | 18:56 | |
| jhesketh | Morning | 18:57 |
| *** nmagnezi has quit IRC | 18:57 | |
| *** lucap has joined #openstack-infra | 18:57 | |
| *** lucap has quit IRC | 18:58 | |
| clarkb | jhesketh: you were just here a few hours ago :) | 18:58 |
| jhesketh | yep, back for the infra meeting | 18:58 |
| clarkb | lifeless: looking at a job in #openstack-nova with jogo and if your jobs are timing out on hpcloud I think this is another symptom of hpcloud to rax connectivity not so great right now | 18:58 |
| jhesketh | clarkb: how are the systems going? | 18:58 |
| *** lucap has joined #openstack-infra | 18:58 | |
| *** dannywilson has quit IRC | 18:59 | |
| clarkb | jhesketh: things are much better but we are still seeing some hpcloud failures | 18:59 |
| *** e0ne has joined #openstack-infra | 18:59 | |
| clarkb | jhesketh: seem mostly due to performance to git.o.o leading to timeouts | 19:00 |
| *** tiswanso_ has quit IRC | 19:00 | |
| *** med_ has quit IRC | 19:00 | |
| fungi | meeting time | 19:00 |
| *** yfried|afk is now known as yfried | 19:00 | |
| *** tim_o has joined #openstack-infra | 19:00 | |
| jhesketh | clarkb: still? so the noisy neighbour hasn't been dealt with, or are there more issues | 19:00 |
| *** hyakuhei has joined #openstack-infra | 19:01 | |
| clarkb | jhesketh: noisy neighbor was pypi mirror, I think we are seeing something different with this | 19:01 |
| jhesketh | hmm, okay | 19:01 |
| jhesketh | clarkb: are we still leaning towards DoS'ing ourselves? | 19:02 |
| *** tonytan4ever has quit IRC | 19:02 | |
| yolanda | zaro, i was able to test that problem with parameter names, but i'm having some issues connecting with jjb to my master, it complains about json | 19:02 |
| yolanda | it just needs and admin user/pass, and jenkins url, right? | 19:02 |
| lifeless | clarkb: | 19:02 |
| lifeless | 2015-04-28 09:59:54.642 | Building remotely on devstack-trusty-hpcloud-b5-2341338 (devstack-trusty) in workspace /home/jenkins/workspace/check-pbr-installation-dsvm | 19:02 |
| lifeless | clarkb: sounds like | 19:03 |
| clarkb | jhesketh: we might be, I need to go look at cacti graphs | 19:04 |
| jhesketh | okay | 19:04 |
| openstackgerrit | Joe Gordon proposed openstack-infra/elastic-recheck: Expand fingerprint for git fetch error https://review.openstack.org/178338 | 19:04 |
| zaro | yolanda: yes, but i don't use user/password. i disable security on my jenkins master | 19:05 |
| jogo | clarkb mriedem: expanded fingerprint for that bug ^ | 19:05 |
| jogo | it has a *lot* of hits | 19:05 |
| mriedem | jogo: looking | 19:05 |
| zaro | yolanda: maybe you should try connecting and doing a simple get_job() with python-jenkins? | 19:06 |
| *** krtaylor has joined #openstack-infra | 19:06 | |
| *** bswartz has joined #openstack-infra | 19:06 | |
| *** jaypipes has joined #openstack-infra | 19:06 | |
| yolanda | zaro, yes, i need to dig more | 19:06 |
| yolanda | i had that working months ago | 19:06 |
| *** hdd has quit IRC | 19:07 | |
| *** erlon has joined #openstack-infra | 19:07 | |
| *** med_ has joined #openstack-infra | 19:08 | |
| *** med_ has quit IRC | 19:08 | |
| *** med_ has joined #openstack-infra | 19:08 | |
| mriedem | jogo: the first part of the query doesn't actually hit from what i'm seeing now, unless that's buried in results | 19:08 |
| *** lucap has quit IRC | 19:08 | |
| mriedem | nvm, just buried | 19:09 |
| *** lucap has joined #openstack-infra | 19:09 | |
| mriedem | goes from 693 -> ~3 million hits | 19:09 |
| jogo | the second part may just always happen | 19:10 |
| *** yfried is now known as yfried|afk | 19:10 | |
| jogo | better query coming soon | 19:11 |
| jhesketh | btw, I've never asked, what is etiquette in our meetings when providing a link... should one #link straight up? | 19:11 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Add Ironic machine power state pass-through https://review.openstack.org/172284 | 19:11 |
| jklare | AJaeger: ping, you got a moment? | 19:11 |
| openstackgerrit | Joe Gordon proposed openstack-infra/elastic-recheck: Expand fingerprint for git fetch error https://review.openstack.org/178338 | 19:11 |
| clarkb | jhesketh: usually yes, that way it ends up in the meeting note summary easy retrieval | 19:11 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Enhance error message in update_patch https://review.openstack.org/177985 | 19:11 |
| jogo | clarkb mriedem: that should be a bit better | 19:11 |
| *** emagana has quit IRC | 19:11 | |
| jhesketh | clarkb: right, but it's okay for non-chair people to do? | 19:12 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Update recent Ironic exceptions https://review.openstack.org/177986 | 19:12 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Convert node_set_provision_state to task https://review.openstack.org/177987 | 19:12 |
| clarkb | jhesketh: I think so, I have definitely done it when not chair | 19:12 |
| jhesketh | okay, good to know | 19:12 |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 19:12 |
| *** hyakuhei has quit IRC | 19:14 | |
| *** dprince has quit IRC | 19:15 | |
| *** dprince has joined #openstack-infra | 19:15 | |
| *** hyakuhei has joined #openstack-infra | 19:15 | |
| *** changbl has quit IRC | 19:15 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Add Ironic machine power state pass-through https://review.openstack.org/172284 | 19:16 |
| *** claudiub has joined #openstack-infra | 19:16 | |
| openstackgerrit | Merged openstack-infra/system-config: Adding openstack-horizon to statusbot channels https://review.openstack.org/174526 | 19:16 |
| *** med_ has quit IRC | 19:17 | |
| *** emagana has joined #openstack-infra | 19:18 | |
| *** tiswanso has joined #openstack-infra | 19:18 | |
| *** tonytan4ever has joined #openstack-infra | 19:18 | |
| *** emagana has quit IRC | 19:19 | |
| *** emagana has joined #openstack-infra | 19:19 | |
| *** bhunter71 has quit IRC | 19:20 | |
| *** dboik_ has quit IRC | 19:20 | |
| morganfainberg | mordred: how much revolting will I have if I try and add non-python language bindings into a lib we maintain (e.g. keystoneauth) once we have the python implementation super awesome. thinking future looking | 19:21 |
| *** dboik has joined #openstack-infra | 19:21 | |
| *** med_ has joined #openstack-infra | 19:21 | |
| *** med_ has quit IRC | 19:21 | |
| *** med_ has joined #openstack-infra | 19:21 | |
| mordred | morganfainberg: I'm not sure I understand your question | 19:21 |
| mordred | morganfainberg: do you mean "what if you created a repo that had a rust library for doing keystone auth?" | 19:22 |
| morganfainberg | mordred: thinking python keystoneauth is solid, we now build a mirror functionality for <Language X> | 19:22 |
| morganfainberg | and want to keep it in the same tree, because $reasons$ for release management | 19:22 |
| morganfainberg | and forcing functionality to be ... smart or something about language things | 19:22 |
| morganfainberg | rust, go, c++, node | 19:22 |
| mordred | morganfainberg: at a time that I'm not in the infra meeting | 19:23 |
| mordred | morganfainberg: I'd like to dig further into $reasons$ | 19:23 |
| morganfainberg | mordred: figured async ;) | 19:23 |
| morganfainberg | mordred: some other time. | 19:23 |
| mordred | because I'm not sure I agree that there the urge for co-located repo will outweigh the strangeness of such a beast | 19:23 |
| mordred | however, I could be wrong | 19:23 |
| mordred | morganfainberg: but I think it's a great idea and we should do it | 19:23 |
| mtreinish | morganfainberg: what wouldn't work about doing that? | 19:24 |
| mtreinish | I feel like it would just work | 19:24 |
| morganfainberg | mtreinish: was more of a "community revolt" thing not a "technology revolt" | 19:24 |
| morganfainberg | re the question | 19:24 |
| openstackgerrit | Sirushti Murugesan proposed openstack-infra/project-config: Add grenade jobs to Heat https://review.openstack.org/178352 | 19:24 |
| mordred | morganfainberg: the community is going to need to learn to not revolt | 19:24 |
| mordred | morganfainberg: it turns out there are other languages in the world | 19:24 |
| morganfainberg | i agree. | 19:24 |
| morganfainberg | anyway | 19:24 |
| mtreinish | oh, heh I wouldn't revolt, as long as you added fortran bindings :) | 19:25 |
| * morganfainberg will circle up later. | 19:25 | |
| mordred | morganfainberg: woot | 19:25 |
| * morganfainberg also waits till post -infra meeting to chase down the "get keystoneauth into gerrit" stuff :) | 19:25 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: move gate-.*-chef-rake job and run it branch specific https://review.openstack.org/176674 | 19:27 |
| greghaynes | yolanda: here is the DIB testing patch stack https://review.openstack.org/#/c/178040/ | 19:27 |
| yolanda | thx, looking | 19:27 |
| *** eharney has quit IRC | 19:28 | |
| *** markvoelker has joined #openstack-infra | 19:28 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Do not cache unsteady state images https://review.openstack.org/177494 | 19:28 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Add tests and invalidation for glance v2 upload https://review.openstack.org/176024 | 19:28 |
| yolanda | greghaynes, so that will cover each step of an element in dib? | 19:29 |
| SpamapS | clarkb: ^^ your concerns about container addressed | 19:29 |
| *** markvoelker has quit IRC | 19:29 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Do not cache unsteady state images https://review.openstack.org/177494 | 19:29 |
| *** markvoelker has joined #openstack-infra | 19:29 | |
| *** markvoelker has quit IRC | 19:29 | |
| greghaynes | yolanda: The tests can be made granular like that, yes. You just use elements to assert state during the image build process | 19:29 |
| *** markvoelker has joined #openstack-infra | 19:29 | |
| yolanda | greghaynes, that looks interesting | 19:30 |
| yolanda | i was looking at something more integrated like test devstack on the nodes | 19:31 |
| *** weshay has quit IRC | 19:31 | |
| lifeless | morganfainberg: I've not haad very satisfactory results with one-tree-N-languages before | 19:31 |
| lifeless | morganfainberg: just as a data point | 19:31 |
| yolanda | but this approach can be used for another kind of testing | 19:31 |
| morganfainberg | lifeless: yeah | 19:31 |
| morganfainberg | lifeless: will explore all that when we have a stable interface | 19:32 |
| morganfainberg | lifeless: alternative, rewrite in c and swig! (don't hurt me...) | 19:32 |
| *** rfolco has quit IRC | 19:32 | |
| *** mestery has quit IRC | 19:34 | |
| *** hashar has joined #openstack-infra | 19:35 | |
| *** mestery has joined #openstack-infra | 19:39 | |
| *** rfolco has joined #openstack-infra | 19:43 | |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Bump tempest-lib min version https://review.openstack.org/178362 | 19:43 |
| *** dustins has joined #openstack-infra | 19:45 | |
| sdague | clarkb: I want a working swift in there as well | 19:45 |
| sdague | given that the bulk of the current complexity involves the swift redirect | 19:45 |
| sdague | I also want this available in devstack just for people to use | 19:45 |
| sdague | I guess it's fine if I'm outvoted, but it seems like writing a devstack plugin for this is simpler than the rest of the setup for a bare node from scratch | 19:46 |
| *** asahlin is now known as asahlin_afk | 19:46 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: remove cookbook-pacemaker from infra https://review.openstack.org/178298 | 19:48 |
| *** marun has quit IRC | 19:48 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: remove cookbook-pacemaker from infra https://review.openstack.org/178298 | 19:48 |
| clarkb | sdague: ya it just seems odd since its not really an openstack service or driver or plugin | 19:49 |
| sdague | clarkb: so instead we'll have a different environment that configures apache & swift from scratch? | 19:50 |
| *** hyakuhei has quit IRC | 19:50 | |
| mordred | clarkb, sdague: I missed the context - what's it testing? | 19:51 |
| sdague | os-loganalyze | 19:51 |
| mordred | ah | 19:51 |
| sdague | because we had an epic fail deploy the other day | 19:51 |
| mordred | so like - dvsm functional tests for os-loganalyze | 19:51 |
| clarkb | sdague: no I am not suggesting that, I was trying to understand why you thought it would be a good devstack plugin | 19:51 |
| sdague | mordred: exactly | 19:51 |
| clarkb | sdague: I would actually suggest we have puppet deploy it | 19:51 |
| clarkb | since we already have that in place | 19:51 |
| clarkb | and we know it works | 19:51 |
| *** changbl has joined #openstack-infra | 19:52 | |
| clarkb | rather than write a new thing to have devstack do it | 19:52 |
| sdague | clarkb: we can pull that bit in | 19:52 |
| sdague | but I still don't see how that gets us a working swift to develop against | 19:52 |
| *** openstackstatus has quit IRC | 19:52 | |
| clarkb | it doesn't | 19:53 |
| *** openstackstatus has joined #openstack-infra | 19:53 | |
| *** ChanServ sets mode: +v openstackstatus | 19:53 | |
| clarkb | sdague: but we could do: `stack.sh` with only swift enabled then run puppet | 19:53 |
| *** fawadkhaliq has joined #openstack-infra | 19:54 | |
| sdague | but if we do it this other way, we also get this enabled for actual devstacks | 19:54 |
| sdague | which *has* been asked for a number of times | 19:54 |
| notmyname | hat's the goal you're looking for with a "working swift to develop against"? | 19:54 |
| EmilienM | nibalizer: here we go: http://logs.openstack.org/26/178026/3/check/gate-infra-puppet-apply-precise/d085597/console.html#_2015-04-28_18_01_34_653 | 19:54 |
| *** Krinkle is now known as Krinkle|detached | 19:55 | |
| *** gyee has quit IRC | 20:00 | |
| yolanda | zaro, looks as i need to fight a bit more with my jenkins master | 20:00 |
| clarkb | sdague: ok I wasn't aware of that | 20:00 |
| yolanda | would you mind testing on your environment? unit tests passing now, and changes addressed | 20:00 |
| clarkb | sdague: I thought people wanted a utility to operate on logs in a similar way but not a hosted service | 20:00 |
| clarkb | sdague: basically smart grep | 20:01 |
| zaro | yolanda: so how did you consolidate the name(s) param? | 20:01 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Add positive/negative tests exercising the parser https://review.openstack.org/178378 | 20:01 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Allow hyphens in profile strings https://review.openstack.org/178379 | 20:01 |
| sdague | so, there are those people as well | 20:01 |
| sdague | but the reason the actual wsgi toy server got added was for the devstack case, and it would be better to just be there | 20:01 |
| yolanda | zaro, left it for another change, i found it's a bit confusing | 20:01 |
| *** erikmwilson has quit IRC | 20:01 | |
| yolanda | don't want to mess in that paralelization change | 20:01 |
| clarkb | nibalizer: so I think template.pp is already naturally split into sections | 20:02 |
| yolanda | as some commands have "name", not only the test one, but the delete, for example | 20:02 |
| clarkb | nibalizer: it will be significantly easier to review if we move a section at a time | 20:02 |
| yolanda | zaro, i fixed my error on name/names for update and pushed again | 20:02 |
| clarkb | nibalizer: that way we can consider the impact of red hat specific code separate from everything else | 20:02 |
| zaro | yolanda: ok, good idea. | 20:02 |
| morganfainberg | pleia2: mind helping me understand what I am doing wrong in http://logs.openstack.org/96/175596/6/check/gate-project-config-layout/23853ce/console.html#_2015-04-28_17_46_18_436 | 20:02 |
| morganfainberg | pleia2: the error is... uhm | 20:02 |
| morganfainberg | not particularly specific/verbose | 20:03 |
| zaro | yolanda: will run my test again. | 20:03 |
| clarkb | nibalizer: and ssh keys (which were the trouble last time) | 20:03 |
| yolanda | thx | 20:03 |
| clarkb | nibalizer: and so on | 20:03 |
| *** dimtruck is now known as zz_dimtruck | 20:03 | |
| fungi | morganfainberg: Job keystoneauth-docs not defined | 20:03 |
| morganfainberg | fungi: huh | 20:03 |
| pleia2 | morganfainberg: 2015-04-28 17:46:18.436 | Job keystoneauth-docs not defined | 20:04 |
| morganfainberg | oh oh | 20:04 |
| pleia2 | oh, fungi beat me to it | 20:04 |
| pleia2 | :) | 20:04 |
| morganfainberg | zuul layout needs to be changed not just project | 20:04 |
| pleia2 | yeah, forgot to note that earlier, sorry about that | 20:04 |
| yolanda | asselin, so we should meet to collaborate on that for sure. I saw you are working on some changes and i don't want to overlap with you, is ok if i continue moving the functionality i can see, to modules? | 20:04 |
| fungi | morganfainberg: AJaeger's comment on that change tells you what's missing | 20:04 |
| *** ayoung is now known as ayoung-mtg | 20:05 | |
| asselin_ | yolanda, yes, we should coordiate that | 20:05 |
| morganfainberg | fungi: https://review.openstack.org/#/c/175596/ didnt see a comment from AJaeger | 20:05 |
| asselin_ | yolanda, https://storyboard.openstack.org/#!/story/2000101 are the stories | 20:06 |
| morganfainberg | ahhhh must be browser cache | 20:06 |
| morganfainberg | refreshed like 5 times and now it appeared | 20:06 |
| *** zz_dimtruck is now known as dimtruck | 20:06 | |
| morganfainberg | AJaeger, fungi: was asked to remove the doc publish jobs by pleia2 | 20:06 |
| asselin_ | yolanda, I'd like to either move to modules first, then move to openstackci, or move to openstackci, and move from there to the modules | 20:06 |
| morganfainberg | until we popilate the doc dir with the data | 20:07 |
| pleia2 | yeah, there are no docs in the doc/ directory | 20:07 |
| morganfainberg | i can roll it back | 20:07 |
| pleia2 | I don't actually know what the build job will do with nothing to build, but it seems silly to run it | 20:07 |
| *** otter768 has joined #openstack-infra | 20:08 | |
| asselin_ | yolanda, the other way to coordiate is for you to e.g. focus on the non-openstackci portions first. | 20:08 |
| yolanda | asselin, ok, i've been working more on the base items, but now i'm starting to move efforts to components, i'll work according to the stories | 20:08 |
| morganfainberg | pleia2: let me check to see what it does | 20:08 |
| morganfainberg | pleia2: i do expect to have docs populated as one of the first commits fwiw | 20:08 |
| pleia2 | morganfainberg: ah, good to know | 20:09 |
| morganfainberg | pleia2: it just was something i wanted through gerrit not by hand | 20:09 |
| *** fawadkhaliq has quit IRC | 20:09 | |
| * pleia2 nods | 20:09 | |
| morganfainberg | pleia2: since i need jamielennox 's brain for it | 20:09 |
| morganfainberg | pleia2: if we need to put stubby docs in to make it merge other code it is at least incentive to do so | 20:09 |
| yolanda | anyway, EOD for today, bye | 20:10 |
| * morganfainberg runs tox -edocs locally to see what happens | 20:10 | |
| *** thinrichs has joined #openstack-infra | 20:10 | |
| morganfainberg | yeah this would fail | 20:10 |
| pleia2 | thanks for checking | 20:11 |
| *** otter768 has quit IRC | 20:12 | |
| fungi | morganfainberg: pleia2: great point. if the project lacks docs, you don't need doc jobs | 20:12 |
| pleia2 | AJaeger: ^^ | 20:13 |
| fungi | i assumed you were trying to add them instead | 20:13 |
| morganfainberg | fungi: pleia2: putting stubby docs (copied from ksc) in | 20:13 |
| *** lucap has quit IRC | 20:13 | |
| morganfainberg | and we will add the docs back in [previous version of that review] | 20:13 |
| pleia2 | wfm | 20:13 |
| fungi | that'll teach me to read through all the review comments ;) | 20:14 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Add keystoneauth library and testing infrastructure https://review.openstack.org/175596 | 20:14 |
| *** ddieterly has quit IRC | 20:14 | |
| morganfainberg | pleia2, fungi, AJaeger: ^ that should be back to normal, and docs should now be populated in my github repo | 20:14 |
| *** ddieterly has joined #openstack-infra | 20:15 | |
| morganfainberg | and now... i need to go back to the hotel. | 20:15 |
| openstackgerrit | Joe Gordon proposed openstack/requirements: Require flake8 2.4.0 https://review.openstack.org/157985 | 20:15 |
| jamielennox | morganfainberg: i'd prefer you just did a stub for docs, i want to start scratch for keystoneauth | 20:16 |
| morganfainberg | jamielennox: we can just trash those ones once it is in gerrit. | 20:16 |
| jamielennox | morganfainberg: either way | 20:17 |
| *** lucap has joined #openstack-infra | 20:17 | |
| *** e0ne has quit IRC | 20:17 | |
| *** spredzy_ is now known as spredzy | 20:19 | |
| *** HeOS has joined #openstack-infra | 20:19 | |
| *** markvoelker has quit IRC | 20:20 | |
| *** _nadya_ has joined #openstack-infra | 20:20 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Add dstat to subnode in multinode mode https://review.openstack.org/177970 | 20:21 |
| clarkb | fungi: +2 on adding centos-6 images | 20:21 |
| clarkb | now to review swift upload retries | 20:21 |
| openstackgerrit | Gary W. Smith proposed openstack-infra/project-config: Add manila-ui to OpenStack https://review.openstack.org/175063 | 20:21 |
| *** markvoelker has joined #openstack-infra | 20:21 | |
| *** tjones1 has joined #openstack-infra | 20:21 | |
| *** jamesmcarthur has joined #openstack-infra | 20:22 | |
| clarkb | jhesketh: any reason you used xrange over range? (thinking about potential python3 compat, but we can cross that bridge if/when we get there) | 20:22 |
| fungi | clarkb: thanks | 20:22 |
| *** Krinkle|detached is now known as Krinkle | 20:23 | |
| fungi | xrange() is also a premature optimization for basically all but very large ranges | 20:23 |
| *** dimtruck is now known as zz_dimtruck | 20:25 | |
| clarkb | fungi: do you want to review the swift upload retries before I approve? | 20:25 |
| clarkb | (or anyone else pleia2 mordred jeblair SergeyLukjanov ) | 20:25 |
| fungi | which one was it again? | 20:25 |
| fungi | i should retry to review it ;) | 20:25 |
| *** rlandy has quit IRC | 20:25 | |
| *** markvoelker has quit IRC | 20:25 | |
| openstackgerrit | Louis Taylor proposed openstack-infra/project-config: Add functional test job for python-glanceclient https://review.openstack.org/178285 | 20:26 |
| clarkb | https://review.openstack.org/#/c/178199/4 | 20:26 |
| fungi | thanks | 20:26 |
| mordred | clarkb: go for it | 20:26 |
| pleia2 | lgtm | 20:27 |
| *** jamesmcarthur has quit IRC | 20:27 | |
| *** gary-smith_ has joined #openstack-infra | 20:27 | |
| *** jamesmcarthur has joined #openstack-infra | 20:28 | |
| *** teran_ has joined #openstack-infra | 20:29 | |
| *** samueldmq has quit IRC | 20:29 | |
| *** mrunge has quit IRC | 20:30 | |
| *** kgiusti has left #openstack-infra | 20:30 | |
| *** _nadya_ has quit IRC | 20:30 | |
| *** dustins_ has joined #openstack-infra | 20:30 | |
| *** mrmartin has quit IRC | 20:32 | |
| *** teran has quit IRC | 20:32 | |
| *** teran_ has quit IRC | 20:33 | |
| *** rmcall has joined #openstack-infra | 20:33 | |
| *** dustins has quit IRC | 20:34 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Add positive/negative tests exercising the parser https://review.openstack.org/178378 | 20:34 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Allow hyphens in profile strings https://review.openstack.org/178379 | 20:34 |
| fungi | i need to remember to pep8 my changes after _all_ my edits, not just in the middle of making them | 20:34 |
| openstackgerrit | Merged openstack-infra/project-config: Retry log upload to swift https://review.openstack.org/178199 | 20:35 |
| *** mrmartin has joined #openstack-infra | 20:36 | |
| *** e0ne has joined #openstack-infra | 20:36 | |
| *** peristeri has quit IRC | 20:37 | |
| *** changbl has quit IRC | 20:37 | |
| *** jamesmcarthur has quit IRC | 20:38 | |
| *** frobware_ has joined #openstack-infra | 20:38 | |
| *** dprince has quit IRC | 20:38 | |
| *** e0ne has quit IRC | 20:39 | |
| *** sarob has quit IRC | 20:44 | |
| *** lucap has quit IRC | 20:45 | |
| *** sarob has joined #openstack-infra | 20:45 | |
| *** thinrichs has quit IRC | 20:47 | |
| *** zigo_ is now known as zigo | 20:50 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Remove tracing since ansible seems to be working https://review.openstack.org/178398 | 20:50 |
| gary-smith_ | I received an error on http://logs.openstack.org/63/175063/4/check/project-config-gerrit/7916450/console.html saying my project was "not normalized". Does that mean that I cannot have multiple groups with the same access? | 20:50 |
| *** pabelanger has quit IRC | 20:51 | |
| *** markvoelker has joined #openstack-infra | 20:51 | |
| *** eharney has joined #openstack-infra | 20:52 | |
| *** sarob has quit IRC | 20:52 | |
| jhesketh | clarkb: no reason for xrange. I didn't realise it would be incompatible. Can redo it with range if you'd like | 20:53 |
| zaro | fungi: thanks for the info on how to use gnupg but i'm still not sure how to apply that to the gerrit contact store. i tried setting 'appsec' in gerrit config to your gpg key but gerrit still will not start for me. | 20:53 |
| *** tjones1 has quit IRC | 20:53 | |
| clarkb | jhesketh: nah we can worry about python3 all at once | 20:53 |
| clarkb | jhesketh: otherwise its a loosing battle | 20:53 |
| zaro | jhesketh: does zuul test work in your env? | 20:53 |
| pleia2 | fungi: there seems to be consensus on this patch from folks who know about such things, but is there a way I can see it worked (aside from merging and - hey, docs showed up!) or check paths myself? re: https://review.openstack.org/#/c/177839 | 20:54 |
| pleia2 | suppose it can't really hurt right now since they are broken, but will help for future reference :) | 20:54 |
| *** harlowja_ has quit IRC | 20:55 | |
| clarkb | pleia2: you can build the infra docs and check the source path easily | 20:55 |
| fungi | zaro: appsec is just a random string used to obscure the api call. i'll have to revisit the documentation for the contactstore feature to remind myself where pgp keys fit into this | 20:55 |
| zaro | pleia2: we don't have a fork of zanata do we? | 20:55 |
| clarkb | pleia2: tox -einfra-docs I think | 20:55 |
| pleia2 | clarkb: ah good, will do | 20:56 |
| pleia2 | zaro: no, we're running directly from upstream | 20:56 |
| jklare | clarkb: do you have a minute to take another look at this one https://review.openstack.org/#/c/176674/ so we (the openstack chef people) can move forward with the gates? | 20:56 |
| *** markvoelker has quit IRC | 20:56 | |
| *** emagana has quit IRC | 20:56 | |
| clarkb | jklare: done | 20:56 |
| jklare | clarkb: amazing, ty | 20:57 |
| *** emagana has joined #openstack-infra | 20:58 | |
| *** sarob has joined #openstack-infra | 20:58 | |
| *** bswartz has quit IRC | 20:58 | |
| zaro | fungi: ahh, re-reading jeblair's comment about gpg key for contact store. he asked if i provided *puppet* a valid gpg key, i thought he meant *gerrit*. | 20:58 |
| zaro | fungi: so i didn't use puppet to setup my test instance of gerrit at all. | 20:59 |
| *** tjones1 has joined #openstack-infra | 20:59 | |
| *** melwitt has quit IRC | 20:59 | |
| *** melwitt has joined #openstack-infra | 20:59 | |
| lifeless | grah 404 on tcpdump in the rax ubuntu mirror | 20:59 |
| lifeless | w t f | 20:59 |
| mordred | lifeless: yeah | 21:00 |
| mordred | lifeless: I mean, yeah | 21:00 |
| *** marun has joined #openstack-infra | 21:00 | |
| zaro | fungi: i'm guessing the puppet sets up the contact store somehow for gerrit somehow. so i think i'm missing that critical step and that's why contact store is a no-go for me :( | 21:00 |
| *** emagana has quit IRC | 21:01 | |
| *** melwitt has quit IRC | 21:01 | |
| *** melwitt has joined #openstack-infra | 21:01 | |
| fungi | zaro: probably. hopefully i'll have time to look over it again in a bit | 21:02 |
| *** gyee has joined #openstack-infra | 21:02 | |
| *** tiswanso has quit IRC | 21:03 | |
| gary-smith_ | clarkb: we talked last week about having multiple groups in an acl vs. creating a new group (see https://review.openstack.org/#/c/175063/). Are multiple groups prohibited? | 21:03 |
| clarkb | gary-smith_: apparently that check is over zealous pretty sure you can have multiples | 21:05 |
| *** hdd has joined #openstack-infra | 21:05 | |
| openstackgerrit | Merged openstack-infra/project-config: move gate-.*-chef-rake job and run it branch specific https://review.openstack.org/176674 | 21:05 |
| clarkb | fungi: AJaeger we have had a lot of problems with this script recently, should we maybe reevaluate what we are hving it do? | 21:05 |
| gary-smith_ | clarkb: can this change proceed with a -1 from jenkins? | 21:06 |
| *** mrmartin has quit IRC | 21:06 | |
| *** thinrichs has joined #openstack-infra | 21:06 | |
| *** dizquierdo has joined #openstack-infra | 21:06 | |
| fungi | clarkb: my opinion is that we shouldn't be testing for group name patterns in urls at all | 21:07 |
| *** baoli has quit IRC | 21:07 | |
| clarkb | gary-smith_: no, we will either need to fix the script or accomodate it in that chnage | 21:07 |
| gary-smith_ | clarkb: what are my chances of getting the script fixed :-) | 21:08 |
| zaro | hashar: hi | 21:08 |
| clarkb | gary-smith_: well I think you can fix it in that change | 21:08 |
| *** thinrichs has left #openstack-infra | 21:08 | |
| gary-smith_ | clarkb: ok, i'll look into that | 21:08 |
| clarkb | gary-smith_: I am trying to get a link to the file and where likely needs to be changed | 21:08 |
| *** thinrichs has joined #openstack-infra | 21:08 | |
| *** jgrimm is now known as zz_jgrimm | 21:09 | |
| gary-smith_ | clarkb: looks like it is tools/check_valid_gerrit_config.sh. | 21:10 |
| *** gyee has quit IRC | 21:11 | |
| clarkb | gary-smith_: I think it is calling https://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/normalize_acl.py#n137 | 21:11 |
| *** rfolco has quit IRC | 21:11 | |
| gary-smith_ | yup | 21:12 |
| clarkb | and we may be running into gerrit and python not agreeing what a valid ini file looks like there | 21:12 |
| clarkb | I think python is collapsing so that you have unique keys but gerrit allows duplicates iirc | 21:12 |
| gary-smith_ | right, gerrit allows it | 21:12 |
| hashar | zaro: hello! | 21:12 |
| hashar | zaro: I haven't been quite active on the jenkins / jjb reviews and coding recently :/ | 21:13 |
| *** dustins_ has quit IRC | 21:13 | |
| clarkb | gary-smith_: so the fix there may be to not use python's config parser for the output and instead construct a file to write out? | 21:13 |
| zaro | hashar: no, i'm not going to bother about that :) | 21:13 |
| clarkb | fungi: ^ any thoughts on that? | 21:13 |
| zaro | hashar: wondering if you've been hacking on zuul lately? | 21:13 |
| *** gyee has joined #openstack-infra | 21:14 | |
| *** tonytan4ever has quit IRC | 21:15 | |
| *** ajmiller has joined #openstack-infra | 21:17 | |
| *** teran has joined #openstack-infra | 21:18 | |
| *** teran has quit IRC | 21:19 | |
| fungi | clarkb: hrm... when i originally implemented that script it had its own parser because of that problem | 21:20 |
| fungi | does it not still? i'm in the middle of some release-related and vmt-related discussions at the moment and don't have time to check | 21:21 |
| clarkb | fungi: oh maybe thats just a dict collapsing it then | 21:21 |
| clarkb | fungi: would would still lead to the same problem | 21:21 |
| *** hdd has quit IRC | 21:21 | |
| clarkb | I may have completely misread this | 21:22 |
| gary-smith_ | clarkb: figured out a workaround: re-ordered the entries in the file | 21:22 |
| *** jtriley has quit IRC | 21:22 | |
| clarkb | gary-smith_: oh! ya its just wanting things in order | 21:22 |
| clarkb | I guess that is what the diff is showing? | 21:22 |
| clarkb | fungi: ^ | 21:23 |
| clarkb | http://logs.openstack.org/63/175063/4/check/project-config-gerrit/7916450/console.html#_2015-04-28_20_24_27_370 | 21:23 |
| fungi | yes, it does also alpha-order (and number-sequence) the lines in each section | 21:23 |
| *** ldnunes has quit IRC | 21:23 | |
| fungi | so the diff will include the reordering it wants to see | 21:23 |
| gary-smith_ | yup, I just sorted each section and now it's happy. | 21:23 |
| clarkb | I see | 21:23 |
| fungi | you should also be able to just run the script against the file and have it reformat it for you | 21:24 |
| *** julim has quit IRC | 21:24 | |
| openstackgerrit | Gary W. Smith proposed openstack-infra/project-config: Add manila-ui to OpenStack https://review.openstack.org/175063 | 21:24 |
| fungi | if you pass it the path to the acl followed by all the normalization rule numbers besides 0 it will edit the file in-place for you | 21:24 |
| fungi | 0 is a pseudo-rule which toggles modifying the file vs spitting the changed version to stdout | 21:25 |
| *** spzala has quit IRC | 21:25 | |
| gary-smith_ | fungi: that's good to know | 21:25 |
| gary-smith_ | while we're on the topic, anyone willing to review https://review.openstack.org/175063 ? | 21:25 |
| fungi | i originally wrote it as a tool for me to normalize all the acls in that repo so that we could see which ones were able to get collapsed into fewer acls, and also to perform needed cleanup of options we no longer needed in them in a way which could easily keep up with the inevitable rebase hell reviewing those changes were bound to encounter | 21:26 |
| fungi | it wasn't really designed to be a linter | 21:26 |
| gary-smith_ | The TC will be +2'ing the manila-ui project tomorrow morning, barring any unforeseen complaints: http://eavesdrop.openstack.org/meetings/tc/2015/tc.2015-04-28-20.02.log.txt @ 20:19 | 21:28 |
| *** HeOS has quit IRC | 21:29 | |
| gary-smith_ | gary-smith_: and we'd really appreciate having this move along. Thanks in advance! | 21:29 |
| *** HeOS has joined #openstack-infra | 21:29 | |
| hashar | zaro: since december the only zuul work I did was to package it for Debian :] | 21:29 |
| hashar | zaro: Wikimedia nows uses .deb packages to deploy zuul! | 21:29 |
| *** emagana has joined #openstack-infra | 21:30 | |
| mordred | hashar: wot | 21:30 |
| hashar | zaro: next steps for me are: integrate patches some patches pending reviews, catch up with all the changes that happened for the last few months, build a package for Debian Jessie | 21:30 |
| mordred | woot | 21:30 |
| mordred | I mean | 21:30 |
| hashar | or was it w00t ? | 21:31 |
| hashar | it is a bit challenging to package for distributions that potentially have old python modules :/ | 21:32 |
| openstackgerrit | Merged openstack/requirements: Updated oslo.config to 1.11.0 https://review.openstack.org/173449 | 21:32 |
| *** lucap has joined #openstack-infra | 21:33 | |
| *** viglesias has quit IRC | 21:33 | |
| *** ajmiller_ has joined #openstack-infra | 21:34 | |
| *** viglesias has joined #openstack-infra | 21:34 | |
| *** jklare has quit IRC | 21:34 | |
| *** frobware_ has quit IRC | 21:34 | |
| *** jklare has joined #openstack-infra | 21:35 | |
| hashar | mordred: also kudos on whoever maintains diskimage-builder :] I really like the elements concept | 21:35 |
| *** dkranz has quit IRC | 21:36 | |
| *** erikmwilson has joined #openstack-infra | 21:37 | |
| krotscheck | WOOOOO! | 21:37 |
| *** ajmiller has quit IRC | 21:37 | |
| *** jklare has quit IRC | 21:38 | |
| zaro | hashar: hmm, trying to get someone to help me figure out why zuul tests are failing for me. i'm at a loss. | 21:38 |
| hashar | zaro: zuul integration tests always looked like a magic tour to me. I am trying hard to figure out the trick being used but end up resorting on James to solve it :D | 21:39 |
| hashar | zaro: I would push the change to Gerrit (I guess it is done) and ask on openstack-infra list for clues | 21:40 |
| *** jklare has joined #openstack-infra | 21:40 | |
| *** Somay has joined #openstack-infra | 21:41 | |
| hashar | zaro: also if you are on Mac, the python version does not have poll() since the system poll() does not work properly (I think it does not work on sockets or something like that) | 21:41 |
| zaro | hashar: no change, it's failing from master branch. but it only fails when i run with tox. | 21:41 |
| fungi | nothing wrong with pushing a zuul patch you're working on into review and seeing if the tests do the same thing they're doing on your workstation | 21:41 |
| zaro | hashar: i'm running on trusty | 21:41 |
| hashar | zaro: oh have you tried rebuilding the tox env? | 21:41 |
| *** erikmwilson has quit IRC | 21:41 | |
| zaro | hashar: yes. | 21:42 |
| clarkb | mordred: ansible question in http://logs.openstack.org/98/178398/1/experimental/check-tempest-dsvm-neutron-multinode-full/c93ab05/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-04-28_21_03_11_699 ansible reports that it ran the setup_workspace and that returned 0, if you scroll to the bottom of that log it actuall says it exit 1'd | 21:42 |
| clarkb | mordred: any idea why that would happen? | 21:42 |
| hashar | zaro: what happens on the infra Jenkins slaves? | 21:42 |
| clarkb | oh I bet tsfilter does the wrong thing | 21:42 |
| fungi | i usually git clean -dfx before running tox locally just because there's all sorts of odd interactions some tests can have with stray files you've forgotten are there | 21:42 |
| clarkb | mordred: so probably not an ansible problem | 21:42 |
| clarkb | yup it doesn't set pipefail | 21:43 |
| zaro | hashar: always get the failure, "failure: process-returncode [ multipart","returncode 1420" | 21:43 |
| *** annegentle has quit IRC | 21:44 | |
| zaro | hashar: looks like it runs ok on jenkins slaves. | 21:44 |
| hashar | :( | 21:44 |
| zaro | hashar: setup with same tox version on my machine but still same failure | 21:44 |
| *** derekh has quit IRC | 21:45 | |
| *** jklare has quit IRC | 21:46 | |
| clarkb | zaro: I just reran zuul tests locally and they passed | 21:46 |
| clarkb | also I thought return codes were 8bits | 21:47 |
| *** jklare has joined #openstack-infra | 21:47 | |
| *** annegentle has joined #openstack-infra | 21:48 | |
| zaro | clarkb: when i run with tox ver 1.6.1 return code is 1420 when i run with tox 1.9.2 return code is 142 | 21:48 |
| clarkb | zaro: can you paste the output of the failing test(s)? | 21:49 |
| *** tiswanso has joined #openstack-infra | 21:49 | |
| *** tiswanso has quit IRC | 21:51 | |
| zaro | clarkb: http://paste.openstack.org/show/210864/ | 21:51 |
| *** tiswanso has joined #openstack-infra | 21:52 | |
| *** markvoelker has joined #openstack-infra | 21:52 | |
| zaro | clarkb: killing my .tox dir and retrying once again but i'm very sure i've tried this already. | 21:52 |
| clarkb | zaro: I think your tests may have hit a timeout, have you made changes to zuul? | 21:52 |
| zaro | clarkb: nope, none | 21:52 |
| clarkb | zaro: try bumping OS_TEST_TIMEOUT=30 to a bigger number in tox.ini | 21:53 |
| *** erikmwilson has joined #openstack-infra | 21:53 | |
| zaro | clarkb: but i am running it in an xsmall flavior | 21:53 |
| clarkb | zaro: maybe add a zero to go to 5 minutes | 21:54 |
| lifeless | so tomorrow is release day right? | 21:54 |
| lifeless | Right after that I want to cut a pbr release | 21:54 |
| clarkb | lifeless: ish, its thursday ttx time | 21:54 |
| clarkb | which is more like day after tomorrow | 21:54 |
| lifeless | ah | 21:54 |
| zaro | clarkb: alright, i'll give that a try | 21:54 |
| lifeless | friday bad day for releases | 21:54 |
| lifeless | next monday then | 21:54 |
| *** emagana has quit IRC | 21:55 | |
| *** spzala has joined #openstack-infra | 21:56 | |
| *** spzala has quit IRC | 21:56 | |
| *** markvoelker has quit IRC | 21:56 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 21:56 | |
| *** dboik has quit IRC | 21:56 | |
| *** spzala has joined #openstack-infra | 21:56 | |
| *** tnovacik has quit IRC | 21:57 | |
| hashar | zaro: sorry for not being of any help :( | 21:57 |
| *** jklare has quit IRC | 21:58 | |
| *** jklare has joined #openstack-infra | 21:59 | |
| *** lucap has quit IRC | 21:59 | |
| *** Somay has quit IRC | 21:59 | |
| *** Swami has quit IRC | 21:59 | |
| *** tim_o has quit IRC | 21:59 | |
| fungi | remember lifeless lives in the future | 22:01 |
| fungi | so for him, late tomorrow | 22:02 |
| *** spzala has quit IRC | 22:02 | |
| fungi | pretty sure he's already well into his wednesday at this point | 22:02 |
| lifeless | maybe you guys could cut a pbr release for me after ttx releases the servers? | 22:03 |
| *** jklare has quit IRC | 22:03 | |
| lifeless | yes, 1000 wed local time | 22:03 |
| *** miqui has quit IRC | 22:03 | |
| *** jklare has joined #openstack-infra | 22:03 | |
| fungi | dhellmann: ^ since it's an oslo lib would you be willing/around to take care of that? | 22:03 |
| fungi | dims: ^ ? | 22:03 |
| *** signed8bit_ZZZzz is now known as signed8bit | 22:03 | |
| fungi | i'll let the outgoing and incoming ptls battle to the death over it | 22:04 |
| *** spzala has joined #openstack-infra | 22:04 | |
| *** sarob has quit IRC | 22:04 | |
| *** harlowja has joined #openstack-infra | 22:04 | |
| dims | fungi: yes, i can. when should we cut it? (trying to parse "after ttx releases the servers") | 22:06 |
| *** esker has quit IRC | 22:07 | |
| *** sarob has joined #openstack-infra | 22:07 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Set pipefail when running tsfilter https://review.openstack.org/178437 | 22:08 |
| clarkb | greghaynes: ^ | 22:08 |
| lifeless | dims: right after kilo releases | 22:08 |
| clarkb | sdague: does devstack need the change in 178437 too? | 22:08 |
| clarkb | I want to say tsfilter comes from devstack? | 22:08 |
| dims | lifeless: cool | 22:08 |
| *** notnownikki has quit IRC | 22:08 | |
| lifeless | dims: doing it before could cause havoc if something goes wrong and there's a last minute OHFUCK for the release process | 22:08 |
| dims | right | 22:08 |
| *** dizquierdo has quit IRC | 22:09 | |
| lifeless | dims: 0.11 will be the next pbr tag - from master | 22:09 |
| dims | fungi: guess we'll need this? https://review.openstack.org/#/c/175369/ | 22:09 |
| greghaynes | clarkb: hah | 22:09 |
| greghaynes | clarkb: This is why I like it to just be on ;) | 22:09 |
| *** otter768 has joined #openstack-infra | 22:09 | |
| *** melwitt has quit IRC | 22:09 | |
| lifeless | dims: no | 22:09 |
| lifeless | dims: thats not to be landed until 0.11 is released | 22:09 |
| *** melwitt has joined #openstack-infra | 22:09 | |
| greghaynes | clarkb: easier though is usually to just make a subshell with it on | 22:09 |
| *** hashar has quit IRC | 22:09 | |
| lifeless | dims: (see my review comment on it) | 22:09 |
| dims | lifeless: workflow -1 got removed | 22:10 |
| lifeless | gnarh | 22:10 |
| lifeless | dims: thanks | 22:10 |
| clarkb | greghaynes: except I would have to preserve all the things a subshll doesn't right? | 22:11 |
| dims | lifeless: guessing this can wait too https://review.openstack.org/#/c/177504/ | 22:11 |
| dims | never mind, it's in the middle of a series | 22:11 |
| lifeless | dims: it can but doesn't need to - because its all testing | 22:11 |
| lifeless | dims: not changing the behaviour | 22:11 |
| clarkb | greghaynes: like -e | 22:11 |
| clarkb | greghaynes: if this wasn't supposed to be a flexible function I would do that | 22:11 |
| dims | lifeless: so looks like i just need to push a button when you or fungi ping me | 22:12 |
| greghaynes | clarkb: export SHELLOPTS | 22:12 |
| lifeless | dims: yes, but I'll be asleep | 22:12 |
| lifeless | dims: If I was going to be awake I could just hit the button myself :) | 22:12 |
| greghaynes | clarkb: that code looks fine to me though | 22:12 |
| dims | gotcha lifeless | 22:12 |
| clarkb | gah, shellopts is what I needed | 22:12 |
| clarkb | pretty sure that is not mentioned in the set help | 22:12 |
| lifeless | dims: my goal for 'just after ttx releases...' is to ensure we have enough debug time before the weekend *in case of fallout* | 22:13 |
| lifeless | dims: I'm thinking things like stable branches with setup.cfg set wrongly | 22:13 |
| *** zz_jgrimm has quit IRC | 22:13 | |
| lifeless | dims: specifyign releases in the past, that sort of thing | 22:13 |
| *** otter768 has quit IRC | 22:13 | |
| *** zz_ja has quit IRC | 22:13 | |
| lifeless | dims: I'm expecting a mild firedrill TBH | 22:13 |
| greghaynes | clarkb: yea, I missed that var too ;) | 22:14 |
| greghaynes | clarkb: horray for awesome docs | 22:14 |
| dims | lifeless: ok. i'll be ready for it :) | 22:14 |
| *** tiswanso has quit IRC | 22:14 | |
| lifeless | dims: if ttx releases very first thing his day, I'll still be up, but if its his late morning or afternoon I'll be out | 22:14 |
| lifeless | dims: I'll check in first thing my friday of course, to help with fixing up any projects with issues | 22:15 |
| dims | lifeless: understood. we'll tag team | 22:15 |
| lifeless | cool | 22:15 |
| *** dangers is now known as dangers_away | 22:16 | |
| *** zz_ja has joined #openstack-infra | 22:16 | |
| *** zz_jgrimm has joined #openstack-infra | 22:16 | |
| EmilienM | fungi: I think you can kill https://jenkins01.openstack.org/job/gate-puppet-vswitch-puppet-beaker-rspec/6/console | 22:17 |
| EmilienM | fungi: because it's testing puppet-vswitch with beaker and add eth0 to an OVS bridge | 22:17 |
| EmilienM | I'm not sure the job will timeout | 22:17 |
| *** jtriley has joined #openstack-infra | 22:18 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Set pipefail when running tsfilter https://review.openstack.org/178437 | 22:19 |
| *** bknudson has quit IRC | 22:19 | |
| clarkb | EmilienM: all of our jobs should timeout | 22:20 |
| *** peristeri has joined #openstack-infra | 22:20 | |
| clarkb | EmilienM: do you think we lost connectivity to the node because eth0 was drafted into service for something else? | 22:21 |
| EmilienM | clarkb: for sure | 22:21 |
| EmilienM | clarkb: you can stop it | 22:21 |
| clarkb | jenkins should notice that | 22:21 |
| nibalizer | haha | 22:21 |
| EmilienM | I'm trying to test puppet-vswitch | 22:21 |
| EmilienM | and I added eth0 to a virtual bridge | 22:21 |
| lifeless | morganfainberg: oh another angle (dunno if you touched on it) on multi-language one-repo is the impact on test matrices | 22:21 |
| EmilienM | I think i'll need to create a dummy interface | 22:21 |
| *** thinrichs has left #openstack-infra | 22:22 | |
| *** melwitt has quit IRC | 22:22 | |
| *** melwitt has joined #openstack-infra | 22:22 | |
| *** amitgandhinz has quit IRC | 22:22 | |
| EmilienM | clarkb: well, I think you can stop it, so we release resources | 22:22 |
| *** jtriley has quit IRC | 22:22 | |
| *** Somay has joined #openstack-infra | 22:22 | |
| *** gordc has quit IRC | 22:23 | |
| *** sarob has quit IRC | 22:23 | |
| clarkb | hrm everything is failing again, looks like git trouble | 22:24 |
| *** thinrichs has joined #openstack-infra | 22:24 | |
| greghaynes | git clone fail ( | 22:25 |
| greghaynes | :( | 22:25 |
| SpamapS | I didn't do it | 22:25 |
| SpamapS | it wasn't me | 22:25 |
| EmilienM | lol | 22:25 |
| * SpamapS just git pulled everything and had a moment of o_O | 22:25 | |
| * greghaynes moves SpamapS to top of suspect list | 22:26 | |
| fungi | EmilienM: jenkins just spotted your job doing badnez | 22:26 |
| SpamapS | I"m off the top?! | 22:26 |
| SpamapS | Or rather, I was? | 22:26 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=877&rra_id=all seems to be at fault | 22:26 |
| * SpamapS hasn't been trying hard enough | 22:26 | |
| EmilienM | fungi: cool | 22:26 |
| clarkb | EmilienM: I would prefer that jenkins kill the job on its own so that we have more time to debug issues like ^ | 22:26 |
| clarkb | EmilienM: and if jenkins does not notice then we should debug and fix that | 22:27 |
| *** Rockyg has quit IRC | 22:27 | |
| fungi | git04 looks pretty slammed | 22:27 |
| clarkb | ya | 22:27 |
| EmilienM | clarkb: makes sense | 22:27 |
| clarkb | fungi: I am tempted to run haproxy in more of a connection balancing mode | 22:28 |
| *** nelsnelson has quit IRC | 22:28 | |
| zaro | clarkb: wow! that was it, timeout caused that zuul error. thanks. | 22:28 |
| clarkb | fungi: we haven't in the past due to unsynchronized git replication, but I have a hunch errors related to that will be less than errors related to this | 22:28 |
| fungi | clarkb: yeah, after the gerrit upgrade i want to finish afs-backed git | 22:29 |
| clarkb | fungi: since a single git command should use the same backend, its only when we start running multiple commands | 22:29 |
| *** Somay has quit IRC | 22:29 | |
| clarkb | but I think what happens to a node like 04 is it gets stuck servicing an expensive request then a bunch of requests pile up behind it essentially dosing it | 22:29 |
| clarkb | whereas a connection balancing proxy would divert more connections to the other services while 04 fell over | 22:30 |
| fungi | it's seriously bogged down | 22:30 |
| fungi | 371 processes | 22:30 |
| clarkb | ya its pegged the cpus, swapping, and generally unhappy | 22:30 |
| clarkb | we should probably also double check that our routine git cleanups are running gc and repacking and all that | 22:31 |
| SpamapS | clarkb: whats the balancing mode now? | 22:31 |
| SpamapS | rrdns? | 22:31 |
| greghaynes | SpamapS: consistent | 22:31 |
| SpamapS | oh consistent hash of source IP? | 22:31 |
| greghaynes | think so | 22:31 |
| fungi | SpamapS: source hash | 22:31 |
| fungi | i'll have dinner coming off the stove momentarily and can dig deeper. guessing we have a bad actor | 22:31 |
| SpamapS | yeah so like, everybody at a single IBM campus gets one server? | 22:31 |
| fungi | like that, yeah | 22:32 |
| SpamapS | I mean, should be fine given the scale we are talking.. but we have so much automation going on... | 22:32 |
| mtreinish | SpamapS: heh, it's more likely everyone at ibm gets one server... | 22:32 |
| clarkb | SpamapS: the reason for that is gerrit replication is not atomic | 22:32 |
| clarkb | mtreinish: I would hope IBM has multiple output proxies | 22:32 |
| SpamapS | mtreinish: was hoping that wasn't the case. ;) | 22:32 |
| fungi | unfortunately we're going to need to analyze haproxy logs in multiple places to find out who's being balanced to git04 and making more requests | 22:33 |
| SpamapS | clarkb: yeah I figured as much. It's a good strategy but means we need to scale each node up and have spare space for expensive ops. | 22:33 |
| clarkb | SpamapS: not necessarily | 22:33 |
| clarkb | SpamapS: I am almost positive it isn't an IBM killing us | 22:33 |
| SpamapS | No I didn't mean to single out IBM | 22:33 |
| *** Swami has joined #openstack-infra | 22:33 | |
| mtreinish | clarkb: I'm not sure I saw some weird behavior with the outbound when at different sites | 22:33 |
| SpamapS | or single campus.. | 22:33 |
| SpamapS | I was thinking it's a single operation .. which I think you're saying too yes? | 22:34 |
| mtreinish | I think they should have it too | 22:34 |
| clarkb | mtreinish: when I ran a proxy setup we had something like 32 IPs per region we balanced through | 22:34 |
| SpamapS | as in, one big op slows all the ops on a box down.. | 22:34 |
| SpamapS | and soon everybody on that server is dos'ing everyone else. | 22:34 |
| clarkb | SpamapS: yes which is true with other connection methods too | 22:34 |
| SpamapS | no scale-back-on-unhealthy-service | 22:34 |
| clarkb | SpamapS: the specific problem here is we then throw more workload at it | 22:34 |
| clarkb | right | 22:34 |
| clarkb | SpamapS: importantly we may have to scale up regardless of the balancing method if we decide that expensive op is important | 22:35 |
| SpamapS | I wonder if there is a method that does consistent hashing-esque behavior, but rebalances anything > say, 5s idle. | 22:35 |
| *** stevemar has quit IRC | 22:35 | |
| *** ZZelle_ has quit IRC | 22:35 | |
| *** spzala_ has joined #openstack-infra | 22:36 | |
| SpamapS | clarkb: also makes me wonder if there would be a way to queue up those expensive ops and handle them in a more queue-like manner. | 22:36 |
| clarkb | SpamapS: thats what github tries to do and they fail at it based on their failure rate | 22:36 |
| SpamapS | like oh you're doing a full clone, you'll need to sit over there while that window is busy. | 22:36 |
| fungi | looks like they're probably coming through fe01 | 22:37 |
| fungi | i need to timeslice this a bit first to be sure | 22:37 |
| clarkb | ya, except I think the expensive thing here is harder to calculate, clones are cheap iirc. Expensive is when you have to construct an almost complete pack on the fly for a fetch | 22:37 |
| *** whoops has quit IRC | 22:38 | |
| clarkb | but basically we would have to implement our own git smart protocol proxy | 22:38 |
| *** spzala has quit IRC | 22:38 | |
| *** spzala_ is now known as spzala | 22:38 | |
| greghaynes | and if you did that youre kind of back where you started re: using balance with a health check | 22:38 |
| greghaynes | because you wont be consistent any longer | 22:39 |
| greghaynes | er, consistent based on client | 22:39 |
| *** zz_jgrimm has quit IRC | 22:39 | |
| fungi | ohyeah | 22:39 |
| clarkb | fungi: ? | 22:40 |
| fungi | http://paste.openstack.org/show/210916/ | 22:40 |
| fungi | 91.189.91.27 | 22:40 |
| clarkb | also looks like puppet is running which won't help anything but shouldn't be the cause either | 22:40 |
| fungi | em1.rapid.canonical.com | 22:41 |
| fungi | clarkb: are you in a position to be able to iptables block that? if not, i'll take care of it momentarily | 22:42 |
| *** davideagnello has quit IRC | 22:42 | |
| clarkb | SpamapS: I do think that possibly dropping requsets that we find to be bad is a not terrible idea | 22:42 |
| clarkb | SpamapS: should be able to do that without a special application proxy | 22:42 |
| SpamapS | what about dynamically adjusting server weight based on load? | 22:42 |
| clarkb | SpamapS: that doesn't solve the inconsistent data source problem | 22:43 |
| *** davideagnello has joined #openstack-infra | 22:43 | |
| clarkb | fungi: I can take a look in just a sec | 22:43 |
| SpamapS | clarkb: well.. it's solved when this isn't happening. | 22:43 |
| SpamapS | clarkb: and when this is happening, it becomes more likely, but still stays consistent on the servers that aren't screwed. | 22:43 |
| *** zz_ja has quit IRC | 22:43 | |
| clarkb | SpamapS: ya thats true | 22:43 |
| *** mriedem is now known as mriedem_away | 22:44 | |
| *** emagana has joined #openstack-infra | 22:44 | |
| *** heyongli has quit IRC | 22:45 | |
| *** maurosr has quit IRC | 22:45 | |
| *** annegentle has quit IRC | 22:46 | |
| fungi | adam_g: any idea what em1.rapid.canonical.com is? | 22:46 |
| clarkb | of course the file is in /etc/sysconfig | 22:46 |
| clarkb | fungi: I am going to disable puppet there so I can do this correctly and not have puppet reapply things for me in half an hour | 22:46 |
| *** zz_ja has joined #openstack-infra | 22:46 | |
| fungi | clarkb: thanks | 22:46 |
| fungi | good thinking | 22:46 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:46 | |
| *** maurosr has joined #openstack-infra | 22:47 | |
| adam_g | fungi, i feel like i did at one point but not anymore :\ | 22:48 |
| *** jtriley has joined #openstack-infra | 22:48 | |
| *** davideagnello has quit IRC | 22:48 | |
| *** heyongli has joined #openstack-infra | 22:49 | |
| clarkb | fungi: how does `-A openstack-INPUT -m tcp -p tcp -s 91.189.91.27 -j REJECT` look above the established connections rule look? | 22:49 |
| greghaynes | fungi: any idea if that src ip is doing a large number in parallel? | 22:49 |
| *** zz_jgrimm has joined #openstack-infra | 22:49 | |
| greghaynes | Seems like an easy thing to make stuff a little better is to just rate limit based on src ip also | 22:49 |
| clarkb | greghaynes: I don't think that helps | 22:49 |
| fungi | clarkb: lgtm | 22:49 |
| *** doug-fish has left #openstack-infra | 22:49 | |
| greghaynes | clarkb: oh? | 22:49 |
| *** Swami_ has joined #openstack-infra | 22:50 | |
| fungi | adam_g: yeah, didn't know if you brought any tribal knowledge when you moved | 22:50 |
| clarkb | greghaynes: because connection request comes in costs almost no bytes, then you ask git daemon to do X and that causes CPU to go crazy | 22:50 |
| adam_g | fungi, what are you seeing coming from there? | 22:50 |
| clarkb | greghaynes: its not the number of connections or data transferred, its the specific request | 22:50 |
| SpamapS | You could limit to something super low, like 2-3 active conns. | 22:50 |
| clarkb | greghaynes: because git makes a custom pack file for you | 22:50 |
| *** peristeri has quit IRC | 22:50 | |
| fungi | adam_g: a denial of service attack against our git server farm | 22:50 |
| adam_g | fungi, it might be the gateway that the canonical auto package build system sits behind? | 22:50 |
| *** zz_ja has quit IRC | 22:50 | |
| fungi | Daviey: jamespage: if either of you are around, what is em1.rapid.canonical.com? we're blocking it from accessing our git servers | 22:50 |
| greghaynes | clarkb: sure, im just looking at the paste, and theres a huge number for that ip in haproxy logs | 22:51 |
| SpamapS | adam_g: might also be outgoing SNAT for cloud instances. ? | 22:51 |
| adam_g | SpamapS, could be. em1 definitely sounds familiar | 22:51 |
| SpamapS | adam_g: as does rapid. | 22:51 |
| clarkb | fungi: ok applying that rule on fe01 now | 22:52 |
| adam_g | fungi, elmo is the guy to talk to, im sure he'd be eager to get to the bottom of it | 22:52 |
| lifeless | adam_g: em1 is a specific ethernet port | 22:52 |
| *** markvoelker has joined #openstack-infra | 22:52 | |
| lifeless | adam_g: rapid.c.c is probably the thing to identify | 22:52 |
| fungi | adam_g: oh! thanks for the pointer | 22:53 |
| lifeless | adam_g: its 91.189.91.27 | 22:53 |
| *** dboik has joined #openstack-infra | 22:53 | |
| *** Swami has quit IRC | 22:53 | |
| * mordred would help, but just boarded a plane | 22:53 | |
| fungi | clarkb: be aware this seems to be hitting both fe01 and fe02 roughly evenly (that tells me it's a nat address because those are round-robin dns entries so even distribution is otherwise unlikely) | 22:53 |
| lifeless | oh, brad might know | 22:53 |
| *** jtriley has quit IRC | 22:53 | |
| *** gary-smith_ has quit IRC | 22:53 | |
| lifeless | he should be up nowish | 22:54 |
| clarkb | fungi: ya doing fe02 now | 22:54 |
| *** dboik_ has joined #openstack-infra | 22:54 | |
| lifeless | clarkb: fungi: also - #canonical-sysadmin is the public channel to reach the canoncial sysadmins | 22:54 |
| lifeless | its a bit late to be ringing elmo, but they have follow-the-sun coverage these days | 22:55 |
| clarkb | and fe02 is done now | 22:55 |
| fungi | though they're probably going to be popping in here any moment now that they're getting tcp resets | 22:55 |
| lifeless | I've pinged bradm in #canonical-sysadmin, and a couple of likely names in launchpad-dev on the offchance | 22:55 |
| clarkb | whoops I lied forgot to turn off puppet on 02, fixing | 22:56 |
| *** zz_ja has joined #openstack-infra | 22:56 | |
| *** davideagnello has joined #openstack-infra | 22:56 | |
| *** blahdeblah has joined #openstack-infra | 22:56 | |
| clarkb | done | 22:56 |
| blahdeblah | \o lifeless | 22:57 |
| lifeless | clarkb: fungi: meet blahdeblah the Canonical sysop on vanguard atm | 22:57 |
| *** ddieterly has quit IRC | 22:57 | |
| fungi | load average: 101.87, 100.56, 104.23 | 22:57 |
| *** sabeen2 has quit IRC | 22:57 | |
| fungi | welcome blahdeblah! | 22:57 |
| SpamapS | #winning | 22:57 |
| lifeless | blahdeblah: clarkb and fungi are the ops on the openstack side | 22:57 |
| *** markvoelker has quit IRC | 22:57 | |
| blahdeblah | \h clarkb, fungi | 22:57 |
| blahdeblah | Or \o, even | 22:57 |
| clarkb | fungi: ya I don't know that this was sufficient to kill the established connections since they would already have been accepted and in the state table right? | 22:57 |
| bradm | I'm about now, but blahdeblah will be able to handle it I'm sure :) | 22:57 |
| *** dboik has quit IRC | 22:57 | |
| blahdeblah | Let me see what's going on on our end | 22:58 |
| clarkb | fungi: but I think haproxy can do that for us, looking now | 22:58 |
| SpamapS | clarkb: you didn't use state in the REJECT rule you pasted | 22:58 |
| SpamapS | clarkb: so they'd get an icmp reject no matter what state they were in | 22:58 |
| fungi | blahdeblah: bradm: in summary, we're seeing a ton of git requests bound for git.openstack.org from 91.189.91.27 | 22:58 |
| greghaynes | clarkb: conntrack -D can too | 22:58 |
| greghaynes | oh, SpamapS is right I think | 22:58 |
| lifeless | clarkb: fungi: the git processes will eventually block on full tcp buffers and then stop causing IO | 22:58 |
| clarkb | SpamapS: oh good I am not as bad at iptables as I thought | 22:58 |
| lifeless | until tcp times out they won't actually go away though | 22:58 |
| fungi | yeah | 22:59 |
| blahdeblah | fungi: I'm pretty sure that's the firewall behind which our OS lab lives | 22:59 |
| lifeless | you need an outbound reject rule to cause them to die early | 22:59 |
| blahdeblah | Just checking that out now | 22:59 |
| lifeless | blahdeblah: cjwatson said a VPN endpoint for misc stuff amongst other things | 22:59 |
| fungi | blahdeblah: thanks. we're in the process of (or already have) blocked it at our load balancer to lessen the impact we're seeing for now | 22:59 |
| lifeless | blahdeblah: so fw sounds plausible | 22:59 |
| bradm | rapid.canonical.com is a firewall with lots of labby type stuff behind it | 22:59 |
| SpamapS | as usual.. no matter the safeguards.. evil escapes the lab to rampage in the village | 23:00 |
| lifeless | evvvvil | 23:00 |
| bradm | I think the OIL stuff is behind it, that seems like a possiblity | 23:00 |
| mordred | eeeeeevil | 23:00 |
| SpamapS | EVIL | 23:00 |
| blahdeblah | When did you see the excessive traffic start? | 23:00 |
| bradm | but I'll just make random unconfirmed observations and let blahdeblah do the actual work. ;) | 23:01 |
| blahdeblah | bradm: ssshhhh | 23:01 |
| lifeless | ok so my job is done, folk hooked up, I'm -> pip internals | 23:01 |
| *** wenlock_ has quit IRC | 23:01 | |
| clarkb | netstat says my iptables rule was plenty | 23:01 |
| fungi | bradm: blahdeblah: rough bisection of our log analysis suggests it started up at 21:50 utc today (a little over 2 hours ago) | 23:01 |
| blahdeblah | OK | 23:01 |
| blahdeblah | I'm just going to reset counters on our end so I can see what's causing it | 23:02 |
| lifeless | clarkb: sure, just means that load, which is 'blocked processes' will be held somewhat higher for up to 30m, even though actual /io/ and /cpu/ use will drop almost immediately | 23:02 |
| lifeless | clarkb: I wasn't suggesting your rule was wrong, more explaining why load could still be high | 23:03 |
| fungi | for reference, eating a salad and typing are more or less mutually exclusive activities. i did not know this | 23:03 |
| lifeless | fungi: you need the salad mounted au natural near your head | 23:03 |
| fungi | lifeless: i need to invent a typing trough | 23:04 |
| lifeless | fungi: ewwwww | 23:04 |
| mordred | mmm. trough | 23:05 |
| blahdeblah | clarkb, fungi: | 23:05 |
| blahdeblah | what's the IP address we're hitting? | 23:05 |
| blahdeblah | (sorry for the linebreak there...) | 23:06 |
| *** peristeri has joined #openstack-infra | 23:06 | |
| clarkb | blahdeblah: 23.253.252.78 and 23.253.252.15 | 23:06 |
| fungi | blahdeblah: rr dns between | 23:06 |
| fungi | yeah those | 23:06 |
| clarkb | blahdeblah: its a DNS round robin (git.openstack.org) | 23:06 |
| blahdeblah | ta | 23:06 |
| fungi | seems to be pretty evenly distributed, so i'm assuming multiple sources behind that nat | 23:06 |
| *** bswartz has joined #openstack-infra | 23:06 | |
| *** signed8bit has quit IRC | 23:08 | |
| *** weshay has joined #openstack-infra | 23:09 | |
| fungi | also i mathed badly. 21:50 utc was a little over an hour ago | 23:09 |
| clarkb | 2050 according to cacti | 23:09 |
| *** Somay has joined #openstack-infra | 23:10 | |
| fungi | yes, indeed | 23:10 |
| fungi | it looks like it was mostly coming through fe01 originally and then split | 23:11 |
| *** dboik_ has quit IRC | 23:11 | |
| fungi | so a little over two hours is right | 23:11 |
| fungi | i should finish this salad. i hear it's brain food | 23:11 |
| *** dboik has joined #openstack-infra | 23:11 | |
| *** spzala has quit IRC | 23:12 | |
| lifeless | fungi: EWUT | 23:13 |
| lifeless | fungi: if its a fish and egg salad, then yes.. | 23:14 |
| fungi | i think it's just people who want me to eat salad claiming that | 23:14 |
| lifeless | fungi: oils and proteins are brain food :) | 23:14 |
| fungi | clarkb: the cacti graph says we either got really lucky or that block rule has fixed us | 23:14 |
| clarkb | fungi: pretty sure the block rule is working | 23:16 |
| mordred | mmm. oil salad | 23:16 |
| fungi | lifeless: i'll tell my wife that's why i prefer to eat tasty dead animals | 23:16 |
| *** zz_ja has quit IRC | 23:17 | |
| fungi | load average: 0.75, 4.19, 35.24 | 23:17 |
| fungi | that's pretty quick | 23:17 |
| *** tjones1 has left #openstack-infra | 23:18 | |
| fungi | and the 5-min avg hasn't even bottomed out yet | 23:18 |
| *** zz_jgrimm has quit IRC | 23:18 | |
| clarkb | it looks like we could use haproxy's httpchk, run a http server on a separate port (or mod rewrite it I suppose), then 404 whenever load exceeds some threshold | 23:19 |
| clarkb | that should take the server out of the rotation based on health checks | 23:19 |
| fungi | doesn't really help for something like this though | 23:19 |
| fungi | it just ends up strafing the load across the farm server by server knocking them offline as it goes | 23:19 |
| *** zz_ja has joined #openstack-infra | 23:19 | |
| clarkb | yup | 23:20 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Add functional tests for create_image https://review.openstack.org/178452 | 23:20 |
| fungi | been there, implemented that, explained to customers why it was a bad idea, helped them clean up the mess afterwards while biting my tongue | 23:20 |
| SpamapS | mordred: ^ works against devstack | 23:20 |
| *** ayoung-mtg has quit IRC | 23:21 | |
| SpamapS | mordred: have not tried against rax because I just realized I don't have an account to play with on rax | 23:21 |
| mordred | sweet | 23:21 |
| mordred | v1 and v2? | 23:21 |
| SpamapS | do they even give you image upload there? | 23:21 |
| mordred | get one | 23:21 |
| mordred | yup | 23:21 |
| *** zz_jgrimm has joined #openstack-infra | 23:21 | |
| SpamapS | k | 23:21 |
| mordred | they're v2 | 23:21 |
| mordred | HP is v1 | 23:21 |
| SpamapS | mordred: I am just now looking at how to make devstack do v2 | 23:21 |
| greghaynes | SpamapS: youll need a swift | 23:21 |
| greghaynes | for added fun | 23:22 |
| mordred | rax v2 specifically is 'upload to swift and do task-create import' | 23:22 |
| lifeless | omg | 23:22 |
| lifeless | mailman 3 released | 23:22 |
| clarkb | fungi: but we can also rate limit with haproxy, not sure if our haproxy is new enough though | 23:22 |
| fungi | mordred has apparently figured out the magic words needed to explain to hpcloud why you need to expense an account with their competitor | 23:22 |
| SpamapS | lifeless: prepare for the apocalypse? | 23:22 |
| fungi | lifeless: no! | 23:22 |
| lifeless | fungi: yes! | 23:22 |
| mordred | wow | 23:22 |
| * fungi looks around for horsemen | 23:22 | |
| greghaynes | fungi: well, I havent submitted the expense report yet | 23:23 |
| mordred | upgrade all the things now!!! | 23:23 |
| greghaynes | fungi: so the jury is still out on whether he has ;) | 23:23 |
| fungi | we get a debian release, an openbsd release, an openstack release _and_ the fabled mailman 3 release all in one week | 23:23 |
| blahdeblah | clarkb, fungi: found the culprit; blocking it momentarily on our end, and will work out whom to contact thereafter | 23:23 |
| fungi | blahdeblah: much appreciated | 23:23 |
| *** armax has quit IRC | 23:23 | |
| mtreinish | fungi: wait I can do that? mordred me want | 23:23 |
| pleia2 | lifeless: their mailman3.org website has been an amusing journey | 23:24 |
| fungi | blahdeblah: feel free to send them here and we can quite possibly help them work out a solution to whatever it is they're trying to implement which will be less impactful | 23:24 |
| blahdeblah | fungi: will do | 23:24 |
| pleia2 | but looks like they stopped maintaining it, so sad | 23:24 |
| *** hemna is now known as hemnafk | 23:25 | |
| lifeless | blahdeblah: or perhaps nat them across a wider set of IPs ? | 23:25 |
| lifeless | blahdeblah: AIUI it wasn't total load, it was load-from-one-IP that caused us grief | 23:25 |
| blahdeblah | lifeless: I'm sure we can convince them just to be better behaved | 23:25 |
| clarkb | I think it may have even be load from a single request | 23:26 |
| clarkb | because you'll see git processes servicing the request use all the memory then things dig into swap and we all have a sad time | 23:26 |
| fungi | lifeless: well, in this case it was also a lot of requests in total even if we did spread them across the whole farm. it may not have been as destructuve if that were to happen, but we'd be staring down danger there regardless | 23:26 |
| *** bknudson has joined #openstack-infra | 23:27 | |
| fungi | in this case they were performance-bottlenecked on the one server they were being served from. if they had all 5 to answer requests we might have seen their workload expand to fill teh entire cluster | 23:28 |
| lifeless | the shadow knows | 23:28 |
| fungi | the weed of crime bears bitter fruit--crime does not pay! | 23:28 |
| *** dims_ has joined #openstack-infra | 23:29 | |
| mtreinish | fungi: I dunno, tv seems to tell me otherwise | 23:30 |
| blahdeblah | clarkb, fungi: OK - that IP is blocked on our end now; when you are comfortable doing so, please open up access again - feel free to rate-limit, or I can enforce a rate-limit on our end if you prefer | 23:30 |
| SpamapS | http://logs.openstack.org/02/177002/2/check/gate-dib-dsvm-functests-devstack-trusty/cbf441e/console.html#_2015-04-28_22_24_08_540 | 23:30 |
| SpamapS | greghaynes: ^ | 23:31 |
| fungi | mtreinish: well, they didn't have television when those serials originally ran | 23:31 |
| fungi | time has proven them wrong | 23:31 |
| mtreinish | fungi: hehe | 23:31 |
| *** erlon has quit IRC | 23:31 | |
| *** dims has quit IRC | 23:31 | |
| fungi | blahdeblah: thanks! we'll do so soon probably | 23:31 |
| greghaynes | SpamapS: wah | 23:31 |
| *** jogo has quit IRC | 23:33 | |
| *** jogo has joined #openstack-infra | 23:34 | |
| *** wenlock has quit IRC | 23:36 | |
| *** thinrichs has left #openstack-infra | 23:36 | |
| mordred | WOOT | 23:37 |
| *** pblaho_ has joined #openstack-infra | 23:37 | |
| mordred | in-flight transatlantic wifi FTW | 23:37 |
| fungi | mordred: nifty--which airline did you manage that on? | 23:38 |
| *** dtantsur|afk has quit IRC | 23:39 | |
| *** pblaho has quit IRC | 23:41 | |
| *** hloeung has joined #openstack-infra | 23:41 | |
| mordred | fungi: delta | 23:42 |
| mordred | fungi: they're rolling it out fleet-wide | 23:42 |
| *** ashleighfarnham has quit IRC | 23:42 | |
| clarkb | so we already do set maxconn on the haproxy listen directive for git protocol | 23:43 |
| clarkb | its set to a very conservative 32 | 23:43 |
| clarkb | this isn't a per server setting though so doesn't help a ton when we direct load to a specific server | 23:43 |
| clarkb | so while the canonical IP may have made a lot of requests only a total of 32 should've been serviced at one time aiui | 23:44 |
| clarkb | whcih does make me think more that its costly requests hurting us here | 23:44 |
| *** pblaho__ has joined #openstack-infra | 23:45 | |
| *** pblaho_ has quit IRC | 23:45 | |
| *** dtantsur has joined #openstack-infra | 23:46 | |
| clarkb | in fact http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=880&rra_id=all confirms that | 23:46 |
| blahdeblah | clarkb, fungi: Logging a ticket with the internal owners of this system - just to clarify, was it excessive traffic causing the issue, or the nature of the git activity? | 23:46 |
| clarkb | blahdeblah: I am beginning to think it was the nature of the git activity | 23:46 |
| clarkb | blahdeblah: the above graph shows that we only had just over 30 connections at one time | 23:47 |
| blahdeblah | Did I see someone mention it exhausted swap? | 23:47 |
| clarkb | blahdeblah: not exhausted swap, but swapping | 23:47 |
| blahdeblah | OK | 23:47 |
| fungi | well, the sort of git requests being made, but also the proportion of requests from that ip address was nearly an order of magnitude higher than any other single source | 23:47 |
| clarkb | blahdeblah: I think what happens is that in some circumstances git reuqests are far more expensive than normal, when that happens we chew up the available memory on our mirros and swap | 23:47 |
| clarkb | fungi: yup, but even then we never exceeded 40 concurrent connections | 23:48 |
| clarkb | (its an average though) | 23:48 |
| blahdeblah | cool | 23:48 |
| clarkb | so I think it was sustained expensive operations | 23:49 |
| clarkb | maybe because they were failing so retries happened? | 23:49 |
| clarkb | would be curious to know what the other end was attempting to do (yay git and its terrible logging) | 23:49 |
| fungi | right, this was via git:// protocol in this case, so we don't really even have apache access logs to go on | 23:50 |
| blahdeblah | Oh, nice; it definitely seems like it was playing nasty: http://cacti.openstack.org/cacti/graph.php?local_graph_id=878&rra_id=all | 23:50 |
| nibalizer | does this look right? http://puppetboard.openstack.org/report/lists.openstack.org/ddd0f8e63111e5e0b8588cbdb3f527fd6b39dcd2 pleia2 ? | 23:50 |
| fungi | yep. did a doozy on the git04 server | 23:50 |
| clarkb | nibalizer: I want to say exim doesn't use aliases like that? | 23:51 |
| fungi | blahdeblah: the 5-minute load average graph is even more impressive | 23:51 |
| *** ddieterly has joined #openstack-infra | 23:51 | |
| blahdeblah | quite distinctive, no? :-) | 23:52 |
| clarkb | nibalizer: was there a recent change tha went in related to that? | 23:52 |
| pleia2 | nibalizer: I think clarkb is right, so it shouldn't matter, but how did that happen? | 23:52 |
| clarkb | pleia2: ya curious to know why it changed, I wonder if the puppet provider for mailing lists was updated/ | 23:52 |
| mordred | did we land the puppet provider patch? | 23:53 |
| clarkb | mordred: I do not know what patch that is | 23:53 |
| mordred | oh - this is actually what that patch is intended to fix IIRC | 23:54 |
| clarkb | mordred: have any more hints so I can go looking for it? | 23:54 |
| mordred | one sec - link coming | 23:54 |
| clarkb | ty | 23:54 |
| mordred | https://review.openstack.org/160343 | 23:54 |
| mordred | the problem is - the puppet maillist provider thinks it should add an alias to /etc/alaisaes | 23:55 |
| mordred | but our exim config does not do that | 23:55 |
| *** esker has joined #openstack-infra | 23:55 | |
| mordred | and manages that file itself | 23:55 |
| mordred | so we wrote a provider to make it stop fighting | 23:55 |
| clarkb | oh I thought exim ignored aliases completel | 23:55 |
| clarkb | and had its own mapping | 23:55 |
| fungi | clarkb: mordred: we've got an exim transport that routes based on what mailman db files it finds, yeah? | 23:55 |
| mordred | fungi: that's right | 23:55 |
| mordred | clarkb: just for mailman | 23:55 |
| clarkb | mordred: gotcha | 23:56 |
| mordred | clarkb: it uses them for non-mailman | 23:56 |
| mordred | thus the confuse | 23:56 |
| pleia2 | aha | 23:56 |
| mordred | nibalizer: we had talked about fixing upstream at some point too, iirc | 23:56 |
| *** imcsk8 is now known as imcsk8|afk | 23:56 | |
| clarkb | mordred: so basically exim updates to what it wants then puppet updates an they go back and forth | 23:56 |
| fungi | that's, like, the recommended way to do mailman+exim. whereas /etc/aliases is the usual deployment for something like mailman+sendmail | 23:56 |
| mordred | yup | 23:56 |
| clarkb | doesn't affect mailing lists but is annoying in puppet | 23:56 |
| mordred | yup | 23:56 |
| mordred | I haven't landed it because I haven't wanted to watch it to make sure it doesn't break | 23:57 |
| mordred | and it's not THAT important | 23:57 |
| fungi | though you could probably emulate the mailman exim transport configuration with a sendmail milter | 23:57 |
| fungi | but i wouldn't want to | 23:57 |
| *** markvoelker has joined #openstack-infra | 23:58 | |
| mordred | good golly no | 23:58 |
| pleia2 | hehe | 23:58 |
| clarkb | mordred: also you have feedback to address on that change | 23:58 |
| mordred | I do? | 23:58 |
| * fungi hung up his sendmail hat along, LONG time ago | 23:58 | |
| clarkb | mordred: you do | 23:58 |
| blahdeblah | Well, that was a fun start to the day - thanks clarkb, fungi, lifeless; I'll let you know if I hear anything further from the team running this system. | 23:58 |
| clarkb | blahdeblah: thank your for the quick response | 23:59 |
| mordred | nibalizer: any chance you want to just fix it? your comments make snese to me, but we're already WELL past my ruby comfort zone | 23:59 |
| clarkb | fungi: should I go ahead and remove our rule? | 23:59 |
| *** emagana has quit IRC | 23:59 | |
| mordred | nibalizer: so I'll be just blindly doing whatever you and crinkle tell me | 23:59 |
| clarkb | I can just turn puppet back on | 23:59 |
| lifeless | blahdeblah: thanks | 23:59 |
| mordred | thanks blahdeblah ! | 23:59 |
| clarkb | mordred: I think you can just remove that function | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
