| *** david_lyle_ has quit IRC | 00:00 | |
| clarkb | jogo: no hits? | 00:00 |
|---|---|---|
| clarkb | error in page generation? | 00:00 |
| jogo | tons of hits message:"savanna-dashboard/info/refs not found" AND message: "fatal\: https://git.openstack.org/openstack" AND filename:"logs/devstack-gate-setup-workspace-new.txt" | 00:01 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use unbound https://review.openstack.org/80092 | 00:01 |
| jogo | I am trying to geneate the pages locally | 00:01 |
| jogo | didn't see any errors | 00:02 |
| jogo | and the query works | 00:02 |
| clarkb | could be an error in the js | 00:02 |
| *** mrodden has quit IRC | 00:02 | |
| jogo | I don't think so because saw hits make it to the unclassifed page as well | 00:03 |
| *** markmcclain has quit IRC | 00:04 | |
| *** flaper87 is now known as flaper87|afk | 00:05 | |
| jogo | clarkb: just generted the unclassifed page locally without a hitch | 00:06 |
| jogo | perhaps my timeline of things is off | 00:06 |
| jogo | or the new code didn't get pushed out | 00:06 |
| jogo | Generated at: 2014-03-12T23:40 | 00:07 |
| clarkb | jogo: I thougt you were looking at classified stuff? | 00:07 |
| jogo | clarkb: that too | 00:07 |
| jogo | all the pages | 00:07 |
| clarkb | we generate pages once an hour | 00:07 |
| jogo | https://review.openstack.org/#/c/80080/ | 00:07 |
| clarkb | with a 20 minute offset between the 3 things | 00:07 |
| jogo | http://status.openstack.org/elastic-recheck/data/uncategorized.html | 00:07 |
| *** alex-afk is now known as Alexandra | 00:07 | |
| clarkb | jogo: right so what do you expect there? | 00:07 |
| clarkb | jogo: I am confused about what you expect to see | 00:08 |
| clarkb | 1291440 shouldn't be there right? because it is classified | 00:08 |
| jogo | I expect to see fewer hits in uncategorized | 00:08 |
| jogo | and 1291440 to be on the classified page | 00:08 |
| jogo | since this landed over 2 hours ago | 00:08 |
| *** malini_afk is now known as malini | 00:10 | |
| clarkb | jogo: 00:03 all.json, 23:22 gate.json, 23:40 uncategorized.html | 00:10 |
| jogo | clarkb: I saw that | 00:10 |
| jogo | but 1440 isn't there | 00:11 |
| clarkb | d626db2021954c207824638036d3baa17587bcc3 is the commit of e-r | 00:11 |
| jogo | soDate: Tue Mar 11 17:27:16 2014 +0000 | 00:11 |
| jogo | yup that be old | 00:11 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test caching-forwarding nameserver https://review.openstack.org/80045 | 00:11 |
| *** SumitNaiksatam has quit IRC | 00:11 | |
| *** CaptTofu has joined #openstack-infra | 00:12 | |
| jogo | clarkb: think we found the problem | 00:12 |
| clarkb | jogo: yeah, I am going to dry run puppet see what it says | 00:12 |
| jogo | clarkb: thanks | 00:12 |
| jeblair | (puppet dashbord soon!) | 00:12 |
| clarkb | puppet is stuck that will do it | 00:12 |
| *** malini is now known as malini_afk | 00:14 | |
| clarkb | I restarted it, all is well | 00:16 |
| jogo | clarkb: thanks | 00:16 |
| jogo | can you regenerate the e-r pages? | 00:17 |
| clarkb | jogo: the cron will get them | 00:17 |
| jogo | kk | 00:17 |
| *** dkliban has joined #openstack-infra | 00:17 | |
| *** rpodolyaka has quit IRC | 00:17 | |
| *** malini_afk is now known as malini | 00:18 | |
| *** ildikov_ has quit IRC | 00:18 | |
| *** SumitNaiksatam has joined #openstack-infra | 00:19 | |
| *** thuc has quit IRC | 00:22 | |
| *** jasondotstar has joined #openstack-infra | 00:22 | |
| *** thuc has joined #openstack-infra | 00:22 | |
| *** dkliban has quit IRC | 00:24 | |
| *** MarkAtwood has quit IRC | 00:24 | |
| *** gokrokve has joined #openstack-infra | 00:24 | |
| *** thuc_ has joined #openstack-infra | 00:25 | |
| *** gokrokve_ has joined #openstack-infra | 00:25 | |
| *** dkliban has joined #openstack-infra | 00:25 | |
| *** maxbit has quit IRC | 00:25 | |
| *** maxbit_ has quit IRC | 00:25 | |
| *** mgagne has joined #openstack-infra | 00:25 | |
| *** reed has quit IRC | 00:26 | |
| *** maxbit has joined #openstack-infra | 00:26 | |
| bknudson | there's essentially no difference between "recheck" and "reverify" now? | 00:26 |
| fungi | bknudson: shh... don't shoult or everyone will want one | 00:26 |
| *** maxbit_ has joined #openstack-infra | 00:26 | |
| jhesketh_ | jeblair: what do you think of my idea to overlay zuul boxes onto gerrit reviews (https://review.openstack.org/#/c/79836/) | 00:26 |
| *** thuc has quit IRC | 00:26 | |
| *** matsuhashi has joined #openstack-infra | 00:27 | |
| *** rpodolyaka has joined #openstack-infra | 00:27 | |
| bknudson | I want to make sure things are working right... I recheck and it goes to gate. | 00:27 |
| fungi | bknudson: basically correct. now that we check before gating, approval can end in either a -1 or a -2, so we basically wanted to account for either possibility without people needing to remember which to use when they can both happen post-approve now | 00:28 |
| *** thuc_ has quit IRC | 00:29 | |
| *** gokrokve has quit IRC | 00:29 | |
| *** EmilienM has quit IRC | 00:30 | |
| *** sdague has quit IRC | 00:30 | |
| lifeless | fungi: jeblair: mordred: https://review.openstack.org/#/c/76588/1 - we'd *really* like this in to fix leaks in our CI environment | 00:31 |
| *** nati_uen_ has joined #openstack-infra | 00:31 | |
| *** esker has joined #openstack-infra | 00:33 | |
| fungi | lifeless: lgtm, but that looks like one jeblair probably also wants to take a quick peek at | 00:33 |
| mordred | I agree | 00:33 |
| mordred | it seems simple/safe enough - but I'd like jeblair to verify there isn't something I'm not thinking of | 00:33 |
| fungi | it seems so simple, but i've been bitten by missing threading big-picture before | 00:34 |
| jhesketh_ | lifeless: +1 | 00:34 |
| *** nati_ueno has quit IRC | 00:34 | |
| *** EmilienM has joined #openstack-infra | 00:35 | |
| fungi | i mean, it's at the end of the cleanup() cleanup method and it's only closing self.things so... i can't imagine how that could go bad but... | 00:35 |
| *** sdague has joined #openstack-infra | 00:35 | |
| *** jasondotstar has quit IRC | 00:36 | |
| *** wchrisj has quit IRC | 00:37 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1291694 https://review.openstack.org/80129 | 00:44 |
| *** jasondotstar has joined #openstack-infra | 00:45 | |
| *** zns has quit IRC | 00:47 | |
| *** Alex_Gaynor has quit IRC | 00:48 | |
| *** melwitt has quit IRC | 00:49 | |
| *** sarob has joined #openstack-infra | 00:49 | |
| *** dolphm has quit IRC | 00:49 | |
| *** simonmcc has quit IRC | 00:49 | |
| *** JoshNang has quit IRC | 00:50 | |
| *** nati_uen_ has quit IRC | 00:50 | |
| *** simonmcc has joined #openstack-infra | 00:50 | |
| *** JoshNang has joined #openstack-infra | 00:50 | |
| *** nati_ueno has joined #openstack-infra | 00:50 | |
| *** wchrisj has joined #openstack-infra | 00:51 | |
| *** mkoderer has quit IRC | 00:52 | |
| *** JayF has quit IRC | 00:52 | |
| *** JayF has joined #openstack-infra | 00:52 | |
| *** cyeoh has quit IRC | 00:53 | |
| *** weiwei has quit IRC | 00:53 | |
| *** sarob has quit IRC | 00:53 | |
| *** cyeoh has joined #openstack-infra | 00:54 | |
| *** Alex_Gaynor has joined #openstack-infra | 00:54 | |
| *** mkoderer has joined #openstack-infra | 00:54 | |
| *** dolphm has joined #openstack-infra | 00:54 | |
| openstackgerrit | Kei YAMAZAKI proposed a change to openstack-infra/jenkins-job-builder: Added support for CI Game plugin https://review.openstack.org/79244 | 00:54 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Kazoo has moved onto 1.3.1 https://review.openstack.org/80134 | 00:55 |
| *** wchrisj has quit IRC | 00:56 | |
| *** julim has quit IRC | 00:56 | |
| mordred | fungi: but exactly | 00:56 |
| *** esker has quit IRC | 00:56 | |
| *** esker has joined #openstack-infra | 00:57 | |
| *** wchrisj has joined #openstack-infra | 01:00 | |
| clarkb | and they claim to have tested it | 01:00 |
| *** sweston has joined #openstack-infra | 01:00 | |
| *** esker has quit IRC | 01:01 | |
| *** Daisy_ has joined #openstack-infra | 01:04 | |
| *** weiwei has joined #openstack-infra | 01:06 | |
| jeblair | jhesketh_: i think the idea is fantastic (code unreviewed) | 01:09 |
| *** matsuhashi has quit IRC | 01:11 | |
| jeblair | lifeless: i will review that tomorrow (a review at this point in my day is not meaningful) | 01:11 |
| lifeless | jeblair: thank you! | 01:12 |
| *** wchrisj has quit IRC | 01:12 | |
| *** caleb_` has joined #openstack-infra | 01:13 | |
| *** VijayTripathi has quit IRC | 01:13 | |
| *** matsuhashi has joined #openstack-infra | 01:14 | |
| mriedem | clarkb: your suggestion has been taken to heart https://review.openstack.org/#/c/80120/ | 01:15 |
| *** caleb_ has quit IRC | 01:15 | |
| *** maxbit_ has quit IRC | 01:17 | |
| *** jhesketh_ has quit IRC | 01:17 | |
| *** dkranz has joined #openstack-infra | 01:17 | |
| clarkb | \o/ | 01:17 |
| *** maxbit has quit IRC | 01:17 | |
| *** maxbit has joined #openstack-infra | 01:18 | |
| *** wchrisj has joined #openstack-infra | 01:18 | |
| *** maxbit has quit IRC | 01:18 | |
| mordred | clarkb wins | 01:19 |
| *** caleb_` has quit IRC | 01:19 | |
| *** dstanek has quit IRC | 01:19 | |
| *** maxbit has joined #openstack-infra | 01:22 | |
| *** CaptTofu has quit IRC | 01:24 | |
| *** CaptTofu has joined #openstack-infra | 01:24 | |
| *** jp_at_hp has joined #openstack-infra | 01:28 | |
| *** CaptTofu has quit IRC | 01:29 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test caching-forwarding nameserver https://review.openstack.org/80045 | 01:29 |
| *** maxbit has quit IRC | 01:29 | |
| *** maxbit has joined #openstack-infra | 01:29 | |
| *** fungi has quit IRC | 01:29 | |
| *** maxbit has quit IRC | 01:31 | |
| *** matsuhashi has quit IRC | 01:31 | |
| *** maxbit has joined #openstack-infra | 01:31 | |
| *** matsuhashi has joined #openstack-infra | 01:31 | |
| *** jepoy has joined #openstack-infra | 01:31 | |
| *** matsuhashi has quit IRC | 01:32 | |
| *** dkliban has quit IRC | 01:34 | |
| *** vhoward- has left #openstack-infra | 01:35 | |
| *** matsuhashi has joined #openstack-infra | 01:35 | |
| *** jhesketh has joined #openstack-infra | 01:36 | |
| *** jhesketh__ has quit IRC | 01:36 | |
| *** wchrisj has quit IRC | 01:36 | |
| *** jhesketh__ has joined #openstack-infra | 01:37 | |
| *** mspreitz has joined #openstack-infra | 01:40 | |
| *** jp_at_hp1 has joined #openstack-infra | 01:40 | |
| *** blamar has quit IRC | 01:41 | |
| *** talluri has joined #openstack-infra | 01:41 | |
| *** dkranz has quit IRC | 01:41 | |
| *** jp_at_hp has quit IRC | 01:42 | |
| *** wchrisj has joined #openstack-infra | 01:43 | |
| *** alexpilotti has quit IRC | 01:43 | |
| *** zhiwei has joined #openstack-infra | 01:43 | |
| *** wenlock has quit IRC | 01:43 | |
| openstackgerrit | A change was merged to openstack/requirements: Rename Savanna to Sahara https://review.openstack.org/79994 | 01:45 |
| *** maxbit has quit IRC | 01:46 | |
| *** nosnos has joined #openstack-infra | 01:46 | |
| *** miqui has quit IRC | 01:49 | |
| *** esker has joined #openstack-infra | 01:50 | |
| *** sarob has joined #openstack-infra | 01:50 | |
| mspreitz | I have just installed DevStack; if I want to run tempest do I have to do it now or can I do some other work first? That is, does tempest have pre-conditions that I might violate by working on something else first? | 01:51 |
| *** adalbas has joined #openstack-infra | 01:52 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Logging in with a pre-existing user saves details https://review.openstack.org/79705 | 01:53 |
| *** Ryan_Lane has quit IRC | 01:55 | |
| clarkb | mspreitz: the -qa channel would know | 01:55 |
| mspreitz | They may know but they are not saying. | 01:55 |
| clarkb | mspreitz: for usptream testing we ran exercises then tempest in the past so I think doing stuff before hand should be ok | 01:55 |
| clarkb | but don't know for certain and we stopped running exercises first a long time ago | 01:55 |
| mspreitz | thanks for the clues | 01:56 |
| *** amcrn has quit IRC | 01:56 | |
| *** sarob has quit IRC | 01:56 | |
| *** rpodolyaka has quit IRC | 01:58 | |
| *** khyati has quit IRC | 01:59 | |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Return the real class in VersionInfo __repr__. https://review.openstack.org/79856 | 02:00 |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Fix up some docstrings. https://review.openstack.org/79855 | 02:00 |
| lifeless | mordred: got a sec? | 02:00 |
| *** wchrisj has quit IRC | 02:01 | |
| *** rfolco has joined #openstack-infra | 02:04 | |
| *** mriedem has quit IRC | 02:05 | |
| *** rpodolyaka has joined #openstack-infra | 02:08 | |
| *** weshay has quit IRC | 02:10 | |
| *** CaptTofu has joined #openstack-infra | 02:10 | |
| *** wchrisj has joined #openstack-infra | 02:11 | |
| *** adalbas has quit IRC | 02:15 | |
| *** talluri has quit IRC | 02:15 | |
| *** talluri has joined #openstack-infra | 02:15 | |
| mordred | lifeless: sure | 02:18 |
| mordred | sup? | 02:18 |
| *** talluri has quit IRC | 02:19 | |
| lifeless | mordred: semver and pbr | 02:20 |
| *** malini is now known as malini_afk | 02:20 | |
| *** Alexandra is now known as alex-afk | 02:21 | |
| *** rpodolyaka has quit IRC | 02:21 | |
| lifeless | mordred: I think what I want to do is add a marker in tree somehow to signal 'last breaking API change', and then use that to teach pbr about dev/alpha versions and when to bump various versions | 02:21 |
| lifeless | mordred: I was wondering if you had any thoughts or constraints on that | 02:21 |
| *** thuc has joined #openstack-infra | 02:22 | |
| mordred | lifeless: I'm not sure about all of the words you just said - but the sentiment seems like a good one | 02:22 |
| mordred | lifeless: are you talking about adding ability to do like a "pbr give-me-next-blah" sort of command? | 02:23 |
| mordred | (I'm getting that from the word "bump") | 02:23 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Make zuul more worker agnostic https://review.openstack.org/80152 | 02:23 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add turbo-hipster to gearman-launchers https://review.openstack.org/80153 | 02:23 |
| *** khyati has joined #openstack-infra | 02:25 | |
| *** hdd_ has joined #openstack-infra | 02:25 | |
| mordred | jhesketh__: I haven't read the patches, but I like the vibe | 02:25 |
| *** vkozhukalov has quit IRC | 02:25 | |
| jhesketh__ | mordred: :-) | 02:25 |
| jhesketh__ | mordred: the next thing to see would be this: http://summit.openstack.org/cfp/details/51 | 02:26 |
| *** thuc has quit IRC | 02:26 | |
| lifeless | mordred: I'm proposing to make setup.py sdist useful for testrepository | 02:27 |
| lifeless | mordred: right now it creates a version 0.0.18.5.gHASH | 02:28 |
| lifeless | mordred: in trunk | 02:28 |
| lifeless | mordred: which is wrong - thats not the version of that commit. | 02:28 |
| mordred | jhesketh__: ++ | 02:28 |
| clarkb | lifeless: isn't it? | 02:28 |
| lifeless | clarkb: its not. | 02:28 |
| clarkb | lifeless: its the fifth commit after 0.0.18 with hash foo | 02:28 |
| clarkb | lifeless: what should it be? | 02:28 |
| lifeless | clarkb: thats a final version number, it should have a dev/alpha/beta/rc marker | 02:28 |
| mordred | lifeless: what do you want it to do? | 02:28 |
| lifeless | clarkb: per http://legacy.python.org/dev/peps/pep-0440/#pre-releases and http://docs.openstack.org/developer/pbr/semver.html | 02:29 |
| *** mgagne has quit IRC | 02:29 | |
| lifeless | mordred: I want it to emit a dev version number on untagged revisions. | 02:29 |
| mordred | lifeless: well, dev is not right, because dev implies version leading towards a thing | 02:30 |
| mordred | so, 1.0.0.dev1 is a pre-version towards 1.0.0 | 02:30 |
| mordred | wait - no, I'm wrong | 02:30 |
| mordred | hangon. let me actually engage my brain with the subject properly | 02:31 |
| mordred | lifeless: I support adding the dev prefix. I honestly had thought I'd done that already :) | 02:32 |
| clarkb | mordred: you do it for pre version with the .a | 02:32 |
| clarkb | iirc | 02:32 |
| clarkb | but post version doesn't | 02:32 |
| mordred | right. 1.0.0.dev1.g234235 is what the pbr semver doc discusses is the PEP440 compliant way to do what we do with post-release versioning | 02:33 |
| mordred | so lifeless version should be coming out as 0.0.18.dev5.gHASH | 02:34 |
| mordred | which shoudl sort 0.0.18 < 0.0.18.dev5.gHASH < 0.0.19.a1 < 0.0.19 | 02:34 |
| mordred | although in a post-versioning repo, you'd have to do 0.0.19.a1 with a tag | 02:35 |
| mordred | and the next untagged commit after it would likely be 0.0.19.a1.dev1.gHASH - which is a little weird | 02:35 |
| clarkb | so reading semver there are MAYs | 02:35 |
| clarkb | I don't think you are required to do ^ | 02:35 |
| mordred | right | 02:35 |
| clarkb | not that it hurts just pointing out this isn't in conflict with semver | 02:35 |
| mordred | it's not | 02:36 |
| jamielennox | if grenade fails with unstable (and there is no log file) is there a reverify bug for that? | 02:36 |
| mordred | (be sure to call out pbr semver - this is an extension to semver) | 02:36 |
| clarkb | jamielennox: why did it fail unstable? | 02:36 |
| jamielennox | no idea - no log file | 02:36 |
| clarkb | jamielennox: have a link to where it reported? | 02:36 |
| jamielennox | i'm up to about the 7th reverify for this review | 02:36 |
| jamielennox | (it's not my fault) | 02:36 |
| jamielennox | https://review.openstack.org/#/c/79474/ | 02:36 |
| mordred | lifeless: in any case - I'm fairly spent brainwise at the moment - I support the general thing you are discussing- I feel like we might want to hash out details sometime soon when I'm less brain-burned | 02:37 |
| clarkb | jamielennox: hrm no logs at all, that indicates to me the test went really sideways. I don't think we have an open bug for that | 02:37 |
| mordred | lifeless: or, if you want to sketch out something and throw it at me for comment, that's fine too | 02:38 |
| clarkb | jamielennox: but submitting a bug against openstack-ci with a link to that change and the link given by jenkins/zuul indicating the build failed unstable and failed to upload any logs is appropriate | 02:38 |
| jamielennox | clarkb: i've had to reverify that one a lot - but the files i've changed shouldn't even be imported | 02:38 |
| clarkb | jamielennox: welcome to the jungle | 02:38 |
| jamielennox | ok, so that's 'expected' at the moment? | 02:38 |
| lifeless | sorry, ECYNTHIA there | 02:39 |
| clarkb | jamielennox: not expected but certainly not abnormal | 02:39 |
| lifeless | mordred: 0.0.18.dev5 is leading up to 0.0.18 | 02:39 |
| clarkb | jamielennox: the UNSTABLE thing is abnormal | 02:39 |
| jamielennox | clarkb: ok, thanks - i just don't want to keep reverifying if it's something i'm doing | 02:39 |
| clarkb | but I don't think something you did caused the unstable. I think jenkins really derped there | 02:40 |
| clarkb | usually we at least get a jenkins traceback saying it failed for some reason in the log | 02:40 |
| lifeless | mordred: thats why I need the API break marker so I can do 0.0.19.dev.g123124 or 0.1.0.dev.g123412 | 02:40 |
| lifeless | ( Pre-release versions have a lower precedence than the associated normal version. ) -> 0.0.18.dev is < 0.0.18 | 02:41 |
| clarkb | lifeless: but it shouldn't be | 02:41 |
| lifeless | clarkb: yes, it should be | 02:41 |
| clarkb | not in a post release world | 02:41 |
| clarkb | let me rephrase | 02:41 |
| lifeless | clarkb: I know the words you say but not how you're using them. | 02:41 |
| clarkb | the old version the one you don't want | 02:41 |
| clarkb | that one is > 0.0.18 | 02:41 |
| lifeless | here is the scenario | 02:41 |
| lifeless | 0.0.18 is tagged | 02:41 |
| lifeless | I have a commit in some arbitrary branch some arbitrary distance in commits from 0.0.18. | 02:42 |
| lifeless | there are no dagwise newer tags than 0.0.18 | 02:42 |
| lifeless | 0.0.18 is a semver | 02:42 |
| lifeless | fin | 02:42 |
| clarkb | 0.0.18.+somenumberofcommits is the newer version | 02:42 |
| lifeless | clarkb: if it was a release, I would agree. Its not. Its not tagged. | 02:43 |
| lifeless | clarkb: Thus its not a release. | 02:43 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add titles to the times on zuul status's page https://review.openstack.org/80160 | 02:43 |
| clarkb | jamielennox: after a quick scan of zuul's status I don't see other jobs in that state so I don't think this is a symptom of al arger issue | 02:44 |
| clarkb | lifeless: I think you want preversioning | 02:44 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add zuul overlay to review page https://review.openstack.org/79836 | 02:44 |
| clarkb | 0.0.18.devX implies this is some version in lead up to 0.0.18 | 02:44 |
| lifeless | clarkb: I don't think I do. | 02:44 |
| jamielennox | clarkb: filed: https://launchpad.net/bugs/1291742 but there isn't a lot of information i can put about it | 02:45 |
| lifeless | clarkb: no, it doesn't | 02:45 |
| lifeless | clarkb: sorry, yes, it does. | 02:45 |
| lifeless | clarkb: which is why I don't want 0.0.18.devX | 02:45 |
| *** matsuhashi has quit IRC | 02:45 | |
| clarkb | which is possibly why pbr does the thing for preversioning and not post versioningin | 02:45 |
| lifeless | clarkb: but no, I don't know the version I want. | 02:45 |
| *** matsuhashi has joined #openstack-infra | 02:45 | |
| clarkb | lifeless: you want 0.0.19.dev5 | 02:45 |
| lifeless | clarkb: I know the rules for determining the version. PBR exists to do the legwork for me. | 02:45 |
| lifeless | clarkb: or 0.1.0.dev5 | 02:46 |
| lifeless | clarkb: or 1.0.0.dev5 | 02:46 |
| clarkb | yeah | 02:46 |
| clarkb | which is preversioning | 02:46 |
| *** matsuhashi has quit IRC | 02:46 | |
| lifeless | clarkb: no | 02:46 |
| clarkb | as currently implemented by pbr | 02:46 |
| *** mwagner_lap has joined #openstack-infra | 02:46 | |
| lifeless | clarkb: preversioning is me making a decision about the next version | 02:46 |
| *** matsuhashi has joined #openstack-infra | 02:46 | |
| lifeless | clarkb: I don't want to make a decision about the next version. | 02:46 |
| clarkb | lifeless: I don't see how you get around that | 02:46 |
| lifeless | clarkb: semver | 02:46 |
| lifeless | clarkb: which was my intial discussion point with mordred. | 02:47 |
| *** matsuhashi has quit IRC | 02:47 | |
| clarkb | I see | 02:47 |
| *** mgagne has joined #openstack-infra | 02:47 | |
| *** thuc_ has joined #openstack-infra | 02:47 | |
| lifeless | clarkb: there are precisely two rules needed to determine the version of a new release. | 02:47 |
| clarkb | so you want preversioning with automagic incremements based on some metadata around the significance of changes made | 02:47 |
| lifeless | a) will it be a stable release (or have we ever done one in the past) | 02:47 |
| *** matsuhashi has joined #openstack-infra | 02:47 | |
| lifeless | b) have we broken the API since the last release | 02:47 |
| lifeless | arguably there is c) is this a cosmetic fix only (point/micro) | 02:48 |
| *** matsuhashi has quit IRC | 02:48 | |
| *** matsuhashi has joined #openstack-infra | 02:49 | |
| clarkb | so you can sort of have that today if you chagne the version you are going up to in conjunction with the changes you describe above | 02:49 |
| clarkb | but making it automagic would be nice | 02:49 |
| lifeless | clarkb: right. e.g. If I could say similar to what Nova does with RPC apis, that we last broken the API vs (not in) version X, then semver can say 'we released something since, so there have been no further breaks, so its a minor release) | 02:49 |
| *** matsuhashi has quit IRC | 02:49 | |
| openstackgerrit | Jamie Lennox proposed a change to openstack-infra/config: Add the kite (KDS) project and kiteclient project to gerrit/jenkins https://review.openstack.org/73074 | 02:50 |
| *** matsuhashi has joined #openstack-infra | 02:50 | |
| lifeless | clarkb: btw I can't find any use of the pre_version stuff in pbr | 02:50 |
| lifeless | clarkb: AFAICT its dead code | 02:50 |
| clarkb | jamielennox: fwiw, one of us with root on zuul will probably need to log into that server and dig through logs to try and find out why that job was marked unstable. I won't do that tonight but will try to do that tomorrow | 02:50 |
| clarkb | lifeless: its not, all of the server projects use it | 02:51 |
| lifeless | clarkb: can you point me at how they use it ? | 02:51 |
| jamielennox | clarkb: no worries | 02:51 |
| clarkb | lifeless: sure | 02:51 |
| lifeless | I see metadata.version | 02:51 |
| lifeless | but that overrides the version | 02:51 |
| clarkb | lifeless: https://git.openstack.org/cgit/openstack/nova/tree/setup.cfg#n3 | 02:51 |
| lifeless | clarkb: specifically, look at pbr/packaging.py | 02:52 |
| lifeless | def get_version(package_name, pre_version=None) | 02:52 |
| lifeless | clarkb: I can't find any callers passing pre_version in | 02:52 |
| *** sarob has joined #openstack-infra | 02:53 | |
| *** rpodolyaka has joined #openstack-infra | 02:53 | |
| *** rpodolyaka has quit IRC | 02:54 | |
| *** rpodolyaka has joined #openstack-infra | 02:54 | |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: TEST: Deleting thought-to-be-dead code. https://review.openstack.org/80161 | 02:55 |
| clarkb | lifeless: https://git.openstack.org/cgit/openstack-dev/pbr/tree/pbr/hooks/metadata.py#n28 the hook(s) do it | 02:55 |
| lifeless | ah | 02:55 |
| *** sarob has quit IRC | 02:56 | |
| *** sarob has joined #openstack-infra | 02:57 | |
| *** sarob__ has joined #openstack-infra | 02:58 | |
| *** sarob has quit IRC | 03:01 | |
| *** rcleere has joined #openstack-infra | 03:04 | |
| lifeless | clarkb: so anyhow, the point is that dev support currently is only done when the user has asserted a next version, which is frankly awkward | 03:04 |
| lifeless | clarkb: *because* its broken on the commit that becomes a new version. | 03:05 |
| *** sarob__ has quit IRC | 03:05 | |
| lifeless | I think. /me putters about | 03:05 |
| *** sarob has joined #openstack-infra | 03:06 | |
| clarkb | you would have to bundle the setup.cfg change with any change that changes the step you need to take to next version | 03:06 |
| clarkb | which is definitely clunky | 03:06 |
| lifeless | once I put 0.0.19 in setup.cfg, I'll get 0.0.19pre; if I tag a commit, I'll get 0.0.19. but once I've released 0.0.19 | 03:06 |
| lifeless | the next commit *must* change the version or it will error | 03:06 |
| *** zhiyan_ is now known as zhiyan | 03:06 | |
| clarkb | yeah there are definitely things you must do | 03:06 |
| lifeless | because the next version given will go back to a dev | 03:06 |
| lifeless | even though 0.0.19 has been released. | 03:06 |
| clarkb | yup | 03:07 |
| lifeless | ok, so tl;dr the postversion stuff is basically broken - no dev, it needs to add dev | 03:07 |
| clarkb | so you have to do a dance, I personalyl would change the version in the version file then do the tag on the previous commit | 03:08 |
| clarkb | https://github.com/uberVU/elasticboard looks interesting, I wish it wasn't tied to github so specifically | 03:08 |
| clarkb | I wonder if we can shove data into ES from gerrit that looks like what it wants | 03:09 |
| *** sarob has quit IRC | 03:10 | |
| *** mgagne has quit IRC | 03:14 | |
| *** sarob has joined #openstack-infra | 03:16 | |
| *** nati_ueno has quit IRC | 03:21 | |
| mspreitz | How can this be: Hyper-V CI complains "This change was unable to be automatically merged with the current state of the repository", but the other CI does not? | 03:22 |
| clarkb | mspreitz: its possible they have a bug somewhere | 03:27 |
| mspreitz | clarkb: thanks | 03:27 |
| *** ryanpetrello has quit IRC | 03:27 | |
| *** fungi has joined #openstack-infra | 03:32 | |
| *** changbl has quit IRC | 03:33 | |
| openstackgerrit | Cyril Roelandt proposed a change to openstack-dev/hacking: Add a six-related checker https://review.openstack.org/80171 | 03:34 |
| *** Daisy_ has quit IRC | 03:36 | |
| * fungi is back, presumably for good now. host server crash coupled with filesystem corruption on nfs machine. uglllly | 03:38 | |
| clarkb | fungi: o_O | 03:39 |
| clarkb | fungi: if it helps all appears to be well here | 03:39 |
| fungi | yay! | 03:39 |
| clarkb | we did have one job fail with UNSTABLE that will need to be looked at tomorrow | 03:39 |
| *** ArxCruz has joined #openstack-infra | 03:39 | |
| clarkb | but I am not worrying about it tonight because it doesn't seem to be affecting other things | 03:39 |
| fungi | k | 03:40 |
| fungi | in that case i will get back to cleaning up the remainder of my post-system-disaster mess | 03:40 |
| *** alex-afk is now known as Alexandra | 03:41 | |
| clarkb | lifeless: StevenK http://lists.openstack.org/pipermail/openstack-dev/2014-March/029846.html customizing stuff like that on launchpad is not possible right? | 03:45 |
| StevenK | Bug tags are up to the project | 03:46 |
| lifeless | thats basic help | 03:46 |
| *** sarob has quit IRC | 03:46 | |
| lifeless | the official tags will autocomplete as typed | 03:46 |
| clarkb | lifeless: right so the pop up can't be changed | 03:46 |
| clarkb | but the tags themselves are arbitrary | 03:46 |
| lifeless | and should (if not its really a bug) hyperlink after that so they can be clicked through | 03:46 |
| clarkb | and will autocomplee | 03:46 |
| *** sarob has joined #openstack-infra | 03:46 | |
| lifeless | clarkb: there's an official list per project maintained by the project | 03:46 |
| lifeless | clarkb: those ones will typecomplete | 03:47 |
| clarkb | gotcha thanks | 03:47 |
| lifeless | clarkb: users can type additional tags in, of course. | 03:47 |
| lifeless | clarkb: but if its not autocompleting, its not official. | 03:47 |
| clarkb | roger | 03:47 |
| lifeless | clarkb: customising the help or associating metadata descriptions with official tags sounds like a good core feature | 03:47 |
| lifeless | by customising the help I mean 'also show the list of official tags and their descriptions' | 03:47 |
| lifeless | there may be a description already, I don't recall | 03:48 |
| *** wchrisj has quit IRC | 03:48 | |
| StevenK | I think there is a list of offical tags somewhere already | 03:48 |
| StevenK | But I suspect there will be some pushback to including the list in the help | 03:48 |
| *** jp_at_hp1 has quit IRC | 03:48 | |
| * clarkb writes up a response | 03:49 | |
| *** matsuhashi has quit IRC | 03:49 | |
| lifeless | there most definitely is an official list | 03:49 |
| lifeless | w.r.t. help, who knows, ask a dev :P | 03:49 |
| clarkb | looks like the official list shows up on the main bugs page on the right hand side | 03:50 |
| *** sarob has quit IRC | 03:51 | |
| *** gokrokve_ has quit IRC | 03:51 | |
| *** vkozhukalov has joined #openstack-infra | 03:51 | |
| *** rpodolyaka has quit IRC | 03:51 | |
| *** thuc has joined #openstack-infra | 03:53 | |
| *** thuc_ has quit IRC | 03:56 | |
| *** rfolco has quit IRC | 03:56 | |
| *** thuc has quit IRC | 03:57 | |
| *** wchrisj has joined #openstack-infra | 04:00 | |
| *** CaptTofu has quit IRC | 04:03 | |
| *** thuc has joined #openstack-infra | 04:03 | |
| *** esker has quit IRC | 04:15 | |
| *** esker has joined #openstack-infra | 04:16 | |
| *** harlowja is now known as harlowja_away | 04:20 | |
| *** thuc_ has joined #openstack-infra | 04:20 | |
| *** afazekas has joined #openstack-infra | 04:21 | |
| *** rpodolyaka has joined #openstack-infra | 04:22 | |
| *** Daisy_ has joined #openstack-infra | 04:22 | |
| *** thuc has quit IRC | 04:23 | |
| *** harlowja_away is now known as harlowja | 04:24 | |
| *** thuc_ has quit IRC | 04:25 | |
| *** dstanek has joined #openstack-infra | 04:25 | |
| *** thuc has joined #openstack-infra | 04:27 | |
| *** wchrisj has quit IRC | 04:33 | |
| *** pcrews has quit IRC | 04:33 | |
| *** matsuhashi has joined #openstack-infra | 04:47 | |
| *** sarob has joined #openstack-infra | 04:47 | |
| *** sarob has quit IRC | 04:53 | |
| *** zns has joined #openstack-infra | 04:55 | |
| *** gokrokve has joined #openstack-infra | 04:55 | |
| *** resker has joined #openstack-infra | 04:57 | |
| *** jasondotstar has quit IRC | 04:57 | |
| *** esker has quit IRC | 05:00 | |
| *** resker has quit IRC | 05:02 | |
| *** hogepodge has quit IRC | 05:03 | |
| *** amotoki has joined #openstack-infra | 05:06 | |
| *** SumitNaiksatam has quit IRC | 05:07 | |
| *** SumitNaiksatam has joined #openstack-infra | 05:10 | |
| nibalizer | clarkb: did you resolve your issues with inifile? | 05:11 |
| clarkb | nibalizer: ya, I think puppet forge was having a hard time. When I got back from lunch it just worked | 05:12 |
| nibalizer | ok | 05:13 |
| *** thuc has quit IRC | 05:13 | |
| *** blamar has joined #openstack-infra | 05:15 | |
| *** blamar has quit IRC | 05:19 | |
| *** blamar has joined #openstack-infra | 05:20 | |
| mordred | lifeless: if you want 0.0.18.dev5 to be leading up to 0.0.18 then what you want is pre-versioning and that's alraedy supported | 05:21 |
| mordred | lifeless: just put "version=0.0.18" in your setup.cfg | 05:22 |
| mordred | lifeless: (I'm responding as I read the scrollback, so it's possible clarkb has already answered you) | 05:23 |
| mordred | lifeless, clarkb: ok. it looks like you guys have covered some good background | 05:26 |
| mordred | lifeless: tbt - I think we should do post-versioning as implemented by pbr currently, except people should semver the first three numbers | 05:27 |
| mordred | and be WAY less stingy with tags/releases | 05:27 |
| mordred | and realize that the days of commits between "releases" not being de facto releases is over and a theory and a myth and go aawy | 05:27 |
| mordred | away | 05:27 |
| mordred | but - I've been drinking | 05:28 |
| clarkb | mordred: no way, me too <_< | 05:28 |
| mordred | clarkb: it happens. sometimes .. there is wine | 05:28 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add PuppetBoard to openstack infra https://review.openstack.org/77928 | 05:29 |
| *** emagana has joined #openstack-infra | 05:29 | |
| mordred | nibalizer: is that ... a patch? | 05:30 |
| nibalizer | mordred: hopefuly | 05:33 |
| nibalizer | are you okay with all the code duplication? | 05:33 |
| clarkb | the test failure looks like a zuul merger derping | 05:33 |
| clarkb | nibalizer: yes | 05:33 |
| clarkb | hopefully one day we can r10k that | 05:33 |
| nibalizer | okay | 05:33 |
| nibalizer | that bash loop is actually beyond where i stop bashing, so adding a loop around that was like 'nope' | 05:33 |
| nibalizer | clarkb: how retrigger run? | 05:34 |
| clarkb | its only bash beacuse its pre having anything else | 05:34 |
| clarkb | nibalizer: comment that says 'recheck no bug' | 05:34 |
| nibalizer | do i have to say that, or can i say snarky things? | 05:35 |
| clarkb | you have to say that. snarky things won't match the regex | 05:35 |
| nibalizer | ok | 05:36 |
| *** mrda is now known as mrda_away | 05:36 | |
| mordred | nibalizer: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type python::virtualenv at /etc/puppet/modules/puppetboard/manifests/init.pp:182 on node puppetdb.openstack.org | 05:36 |
| nibalizer | oh right | 05:36 |
| nibalizer | haha | 05:36 |
| clarkb | silly dependencies | 05:37 |
| mordred | :) | 05:37 |
| nibalizer | its almost like i turned off dependency resolution | 05:37 |
| nibalizer | weird | 05:37 |
| mordred | haha | 05:37 |
| nibalizer | also that puppet module is siiiick, it can create virtualenvs and do almost anything under the sun with them | 05:37 |
| *** VijayTripathi has joined #openstack-infra | 05:39 | |
| clarkb | nibalizer: so it installs puppet board in a virtualenv with all of its dependencies? | 05:40 |
| clarkb | we should probably do that for more of the things we do | 05:40 |
| mordred | nibalizer: can is create virtualenvs, install software in them, and then have the software in them operate normall? | 05:40 |
| mordred | because what clarkb said if so | 05:40 |
| clarkb | eg zuul, gerritbot, statusbot, graphite, and so on | 05:40 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add PuppetBoard to openstack infra https://review.openstack.org/77928 | 05:40 |
| nibalizer | clarkb: mordred yuup | 05:42 |
| nibalizer | all is in the readme https://github.com/stankevich/puppet-python | 05:42 |
| nibalizer | and yes puppetboard runs it out of a virtualenv | 05:42 |
| lifeless | mordred: so, I think pbr can be a lot better with what I have planned. | 05:42 |
| lifeless | mordred: I will write it up and you can gaze in awe when I'm drunk and you're not. | 05:43 |
| mordred | lifeless: it might can - but first and foremost pbr is here to serve openstack - so I need to make sure it does that | 05:43 |
| *** nicedice has quit IRC | 05:43 | |
| mordred | lifeless: I look forward to reading your thoughts though - I do strive for completeness | 05:44 |
| mordred | nibalizer: I'm VERY excited about looking at how you use virtualenv there | 05:45 |
| clarkb | mordred: that may just solve the pip package problems | 05:46 |
| clarkb | but I need sleep | 05:46 |
| clarkb | so I will look at it later | 05:46 |
| mordred | clarkb: +100 | 05:48 |
| *** sarob has joined #openstack-infra | 05:50 | |
| *** Ryan_Lane1 has quit IRC | 05:51 | |
| *** talluri has joined #openstack-infra | 05:51 | |
| *** gyee has quit IRC | 05:51 | |
| *** skraynev_afk is now known as skraynev | 05:52 | |
| *** zns has quit IRC | 05:53 | |
| *** harlowja is now known as harlowja_away | 05:53 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add PuppetBoard to openstack infra https://review.openstack.org/77928 | 05:53 |
| *** zns has joined #openstack-infra | 05:53 | |
| SergeyLukjanov | hi folks | 05:54 |
| *** sarob has quit IRC | 05:54 | |
| *** hdd_ has quit IRC | 05:55 | |
| *** gokrokve has quit IRC | 05:57 | |
| *** david-lyle has joined #openstack-infra | 05:57 | |
| lifeless | mordred: What I want for testr is also what I want for os-*-config in OpenStack :) | 05:58 |
| lifeless | mordred: Imma verra simple | 05:59 |
| *** blamar has quit IRC | 06:00 | |
| *** blamar has joined #openstack-infra | 06:01 | |
| *** dstanek has quit IRC | 06:01 | |
| *** coolsvap has joined #openstack-infra | 06:02 | |
| *** david-lyle has quit IRC | 06:02 | |
| coolsvap | infra guys, need a small help | 06:03 |
| coolsvap | I have recreated my devstack and after that my git review is not working | 06:03 |
| *** dstanek has joined #openstack-infra | 06:03 | |
| mordred | lifeless: I look forward to reading your literature | 06:04 |
| lifeless | mordred: :) | 06:04 |
| *** CaptTofu has joined #openstack-infra | 06:04 | |
| lifeless | mordred: code really | 06:04 |
| mordred | lifeless: I look forward to reading your literature | 06:05 |
| lifeless | mordred: ok now you're trolling :) | 06:05 |
| clarkb | coolsvap can you paste the failure? | 06:06 |
| coolsvap | clarkb: http://paste.openstack.org/show/73340/ | 06:07 |
| coolsvap | I think its not able to ssh review.openstack.org | 06:08 |
| *** CaptTofu has quit IRC | 06:08 | |
| coolsvap | clarkb I tried manually creating remote, disabling firewall still the same error | 06:08 |
| coolsvap | clarkb earlier devstack was working and other projects in /opt/stack were not working, not I cloned devstack again | 06:09 |
| coolsvap | its also not working | 06:09 |
| *** unicell has quit IRC | 06:09 | |
| clarkb | coolsvap yup lookslike ssh doesnt work to review.openstack.org | 06:09 |
| clarkb | are you behind a firewall? | 06:10 |
| coolsvap | no | 06:10 |
| clarkb | or is virtualbox not using NAT? can tou get to google from that host? | 06:10 |
| coolsvap | clarkb yes I can ping google.com | 06:10 |
| clarkb | traceroute and/or mtr may help | 06:11 |
| clarkb | is your ssh key on that node? | 06:11 |
| coolsvap | yes | 06:11 |
| coolsvap | clarkb the stragne thing is I was able to do it before, just after I recreated devstack I am not able to do so | 06:12 |
| coolsvap | will try spinning a new node and recheck | 06:12 |
| clarkb | can you try manually ssh'ing? ssh -p 29418 username@review.openstack.org gerrit ls-projects | 06:13 |
| clarkb | also make sure the username is correct | 06:13 |
| coolsvap | yup | 06:13 |
| jhesketh__ | coolsvap: have you tried disabling IPv6? | 06:14 |
| coolsvap | the ssh is also not working i guess | 06:14 |
| jhesketh__ | we were having issues talking to review.openstack.org over v6 yesterday | 06:14 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add PuppetBoard to openstack infra https://review.openstack.org/77928 | 06:14 |
| coolsvap | jhesketh__ okay let me check that | 06:14 |
| jhesketh__ | coolsvap: also try the verbose flag on ssh to see if it's the connection or command that's failing | 06:14 |
| jhesketh__ | (-v) | 06:14 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add PuppetBoard to openstack infra https://review.openstack.org/77928 | 06:16 |
| nibalizer | sweet, we're down to just tweaking the apache configs, a pretty cool place to be | 06:17 |
| nibalizer | if anyone has any input on that i'd love to hear it | 06:17 |
| nibalizer | im off for now | 06:17 |
| jhesketh__ | nice work nibalizer | 06:19 |
| *** maxbit has joined #openstack-infra | 06:20 | |
| *** yamahata has quit IRC | 06:21 | |
| *** khyati has quit IRC | 06:22 | |
| *** rcarrillocruz1 has joined #openstack-infra | 06:23 | |
| nibalizer | ty | 06:23 |
| *** nati_ueno has joined #openstack-infra | 06:24 | |
| *** maxbit has quit IRC | 06:25 | |
| *** rcarrillocruz has quit IRC | 06:25 | |
| *** yaguang has joined #openstack-infra | 06:26 | |
| *** lukego has joined #openstack-infra | 06:27 | |
| *** yaguang has quit IRC | 06:30 | |
| *** rcarrillocruz1 has quit IRC | 06:30 | |
| *** rcarrillocruz has joined #openstack-infra | 06:30 | |
| *** pblaho has joined #openstack-infra | 06:32 | |
| coolsvap | jhesketh looks like its connection failure, i disabled ipv6 and rebooted, lets see | 06:35 |
| jhesketh__ | coolsvap: okay, let me know how it goes | 06:36 |
| *** vogxn has joined #openstack-infra | 06:37 | |
| *** coolsvap_ has joined #openstack-infra | 06:38 | |
| *** coolsvap has quit IRC | 06:40 | |
| *** rcarrillocruz1 has joined #openstack-infra | 06:43 | |
| *** yaguang has joined #openstack-infra | 06:43 | |
| *** rcarrillocruz has quit IRC | 06:44 | |
| *** maxbit has joined #openstack-infra | 06:44 | |
| *** maxbit has quit IRC | 06:49 | |
| *** maxbit has joined #openstack-infra | 06:49 | |
| *** maxbit has quit IRC | 06:50 | |
| *** sarob has joined #openstack-infra | 06:51 | |
| *** Alexandra is now known as alex-gone | 06:56 | |
| *** sarob has quit IRC | 06:56 | |
| *** sarob has joined #openstack-infra | 07:01 | |
| *** VijayTripathi has quit IRC | 07:02 | |
| *** sarob has quit IRC | 07:05 | |
| *** chandan_kumar has joined #openstack-infra | 07:10 | |
| *** rpodolyaka has quit IRC | 07:12 | |
| *** zns has quit IRC | 07:13 | |
| *** vkozhukalov has quit IRC | 07:16 | |
| *** yolanda has joined #openstack-infra | 07:17 | |
| *** ildikov_ has joined #openstack-infra | 07:18 | |
| *** morganfainberg is now known as morganfainberg_Z | 07:18 | |
| jhesketh__ | coolsvap_: how did you go without ipv6? | 07:27 |
| *** tian has joined #openstack-infra | 07:30 | |
| *** vogxn has quit IRC | 07:31 | |
| *** matsuhashi has quit IRC | 07:32 | |
| *** matsuhashi has joined #openstack-infra | 07:32 | |
| *** nosnos has quit IRC | 07:32 | |
| *** nosnos_ has joined #openstack-infra | 07:32 | |
| *** jlibosva has joined #openstack-infra | 07:39 | |
| *** mspreitz has left #openstack-infra | 07:42 | |
| *** BobBall has joined #openstack-infra | 07:46 | |
| *** jcoufal has joined #openstack-infra | 07:47 | |
| *** e0ne has joined #openstack-infra | 07:47 | |
| *** sarob has joined #openstack-infra | 07:54 | |
| *** sarob has quit IRC | 07:58 | |
| *** ArxCruz has quit IRC | 07:59 | |
| *** dizquierdo has joined #openstack-infra | 08:02 | |
| *** CaptTofu has joined #openstack-infra | 08:05 | |
| *** jgallard has joined #openstack-infra | 08:06 | |
| *** vogxn has joined #openstack-infra | 08:08 | |
| *** emagana has quit IRC | 08:08 | |
| *** eharney has joined #openstack-infra | 08:09 | |
| *** CaptTofu has quit IRC | 08:10 | |
| *** saju_m has joined #openstack-infra | 08:12 | |
| *** flaper87|afk is now known as flaper87 | 08:13 | |
| *** unicell has joined #openstack-infra | 08:19 | |
| *** andreaf has joined #openstack-infra | 08:30 | |
| *** hashar has joined #openstack-infra | 08:45 | |
| *** jpich has joined #openstack-infra | 08:45 | |
| *** sarob has joined #openstack-infra | 08:55 | |
| *** rlandy has joined #openstack-infra | 08:56 | |
| *** derekh has joined #openstack-infra | 09:00 | |
| *** jasondotstar has joined #openstack-infra | 09:02 | |
| *** sarob has quit IRC | 09:02 | |
| *** matsuhashi has quit IRC | 09:07 | |
| *** jp_at_hp has joined #openstack-infra | 09:10 | |
| *** eharney has quit IRC | 09:10 | |
| *** nosnos has joined #openstack-infra | 09:11 | |
| *** nosnos_ has quit IRC | 09:11 | |
| *** matsuhashi has joined #openstack-infra | 09:12 | |
| *** coolsvap_ has quit IRC | 09:12 | |
| *** thuc has joined #openstack-infra | 09:18 | |
| openstackgerrit | Sylvain Baubeau proposed a change to openstack-infra/jenkins-job-builder: Add workspace scm https://review.openstack.org/79345 | 09:20 |
| *** nati_ueno has quit IRC | 09:21 | |
| *** rossella_s has joined #openstack-infra | 09:21 | |
| *** yaguang has quit IRC | 09:22 | |
| *** thuc has quit IRC | 09:22 | |
| *** johnthetubaguy has joined #openstack-infra | 09:24 | |
| *** Daisy_ has quit IRC | 09:25 | |
| *** yassine has joined #openstack-infra | 09:27 | |
| *** rossella_ has joined #openstack-infra | 09:28 | |
| *** yassine has quit IRC | 09:28 | |
| *** yassine has joined #openstack-infra | 09:28 | |
| *** rossella_s has quit IRC | 09:28 | |
| *** chandan_kumar has quit IRC | 09:30 | |
| *** eharney has joined #openstack-infra | 09:30 | |
| openstackgerrit | Victor Stinner proposed a change to openstack/requirements: Allow trollius 0.2 https://review.openstack.org/79901 | 09:38 |
| *** jooools has joined #openstack-infra | 09:38 | |
| *** eharney has quit IRC | 09:43 | |
| *** matsuhashi has quit IRC | 09:44 | |
| *** chandan_kumar has joined #openstack-infra | 09:46 | |
| *** matsuhashi has joined #openstack-infra | 09:47 | |
| *** bada has joined #openstack-infra | 09:53 | |
| *** rcarrillocruz has joined #openstack-infra | 09:58 | |
| *** sarob has joined #openstack-infra | 09:58 | |
| *** rcarrillocruz1 has quit IRC | 09:59 | |
| *** jasondotstar has quit IRC | 10:00 | |
| *** sarob has quit IRC | 10:02 | |
| *** CaptTofu has joined #openstack-infra | 10:05 | |
| *** zhiwei has quit IRC | 10:08 | |
| *** CaptTofu has quit IRC | 10:10 | |
| *** bada_ has joined #openstack-infra | 10:15 | |
| *** bada has quit IRC | 10:16 | |
| *** kmartin has quit IRC | 10:20 | |
| *** yolanda has quit IRC | 10:23 | |
| *** hashar has quit IRC | 10:24 | |
| *** blamar has quit IRC | 10:24 | |
| *** markwash_ has quit IRC | 10:25 | |
| *** hashar has joined #openstack-infra | 10:26 | |
| *** markwash has joined #openstack-infra | 10:28 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Fix migration tests https://review.openstack.org/80236 | 10:35 |
| *** zhiwei has joined #openstack-infra | 10:35 | |
| *** markmc has joined #openstack-infra | 10:36 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Temp enable savanna service for sahara tests. https://review.openstack.org/80237 | 10:36 |
| *** ociuhandu has quit IRC | 10:39 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/devstack-gate: Support XenServer in the gate https://review.openstack.org/80241 | 10:50 |
| *** e0ne has quit IRC | 10:52 | |
| *** mwagner_lap has quit IRC | 10:52 | |
| *** sarob has joined #openstack-infra | 10:59 | |
| *** yaguang has joined #openstack-infra | 11:02 | |
| *** sarob has quit IRC | 11:05 | |
| *** jgallard has quit IRC | 11:08 | |
| *** matsuhashi has quit IRC | 11:11 | |
| *** matsuhashi has joined #openstack-infra | 11:12 | |
| *** johnthetubaguy is now known as johnthebpguy | 11:15 | |
| *** mbacchi has joined #openstack-infra | 11:17 | |
| *** jasondotstar has joined #openstack-infra | 11:22 | |
| *** jamielennox is now known as jamielennox|away | 11:22 | |
| *** vogxn has quit IRC | 11:28 | |
| *** amcrn has joined #openstack-infra | 11:31 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: only care about unclassifieds in gate queue https://review.openstack.org/80066 | 11:32 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: fix formatting on 'and' https://review.openstack.org/80083 | 11:32 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1291694 https://review.openstack.org/80129 | 11:33 |
| *** e0ne has joined #openstack-infra | 11:35 | |
| *** jgrimm has joined #openstack-infra | 11:39 | |
| *** e0ne has quit IRC | 11:40 | |
| *** alff has joined #openstack-infra | 11:45 | |
| *** pdmars has joined #openstack-infra | 11:49 | |
| *** _sweston_ has joined #openstack-infra | 11:50 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for nova unit test bug 1291620 https://review.openstack.org/80091 | 11:52 |
| *** sweston has quit IRC | 11:52 | |
| *** yaguang has quit IRC | 11:53 | |
| *** dkranz has joined #openstack-infra | 11:55 | |
| *** dkranz has quit IRC | 11:57 | |
| *** dkranz has joined #openstack-infra | 11:58 | |
| *** sarob has joined #openstack-infra | 12:02 | |
| *** weshay has joined #openstack-infra | 12:03 | |
| openstackgerrit | Serg Melikyan proposed a change to openstack/requirements: Updated oslo.messaging requirement https://review.openstack.org/80259 | 12:04 |
| *** e0ne has joined #openstack-infra | 12:05 | |
| *** CaptTofu has joined #openstack-infra | 12:06 | |
| *** sarob has quit IRC | 12:06 | |
| *** johnthetubaguy has joined #openstack-infra | 12:08 | |
| *** johnthebpguy has quit IRC | 12:09 | |
| *** e0ne has quit IRC | 12:10 | |
| *** mbacchi has quit IRC | 12:10 | |
| *** matsuhashi has quit IRC | 12:11 | |
| *** CaptTofu has quit IRC | 12:11 | |
| *** matsuhashi has joined #openstack-infra | 12:12 | |
| *** nosnos has quit IRC | 12:15 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add a footprint for Bug 1291957 https://review.openstack.org/80262 | 12:15 |
| *** matsuhashi has quit IRC | 12:16 | |
| *** mbacchi has joined #openstack-infra | 12:16 | |
| *** jeckersb_gone is now known as jeckersb | 12:16 | |
| *** hashar has quit IRC | 12:20 | |
| *** hashar has joined #openstack-infra | 12:22 | |
| *** aysyd has joined #openstack-infra | 12:24 | |
| *** matsuhashi has joined #openstack-infra | 12:26 | |
| *** yolanda has joined #openstack-infra | 12:27 | |
| *** dims_ has quit IRC | 12:30 | |
| *** e0ne has joined #openstack-infra | 12:31 | |
| *** dims_ has joined #openstack-infra | 12:33 | |
| *** ArxCruz has joined #openstack-infra | 12:36 | |
| openstackgerrit | Kei YAMAZAKI proposed a change to openstack-infra/jenkins-job-builder: Added support for CI Game plugin https://review.openstack.org/79244 | 12:38 |
| lukego | Howdy! I would love some help with three devstack-gate/ci setup issues: KEEP_LOCALRC not working for me; DEVSTACK_GATE_TEMPEST_REGEX not working for me; leaking loopback mounts on /opt/stack/data/swift/drives/images/swift.img (ultimately causing test failures). | 12:38 |
| lukego | anybody keen to help? | 12:38 |
| *** CaptTofu has joined #openstack-infra | 12:39 | |
| *** pdmars has quit IRC | 12:43 | |
| *** andreaf has quit IRC | 12:44 | |
| *** jlibosva has quit IRC | 12:46 | |
| *** jlibosva has joined #openstack-infra | 12:46 | |
| katyafervent | Hi! Does anyone know about standards for the guid format? | 12:46 |
| *** fifieldt has quit IRC | 12:47 | |
| *** pdmars has joined #openstack-infra | 12:47 | |
| *** exz has quit IRC | 12:48 | |
| *** jcoufal has quit IRC | 12:50 | |
| *** jcoufal has joined #openstack-infra | 12:51 | |
| *** smarcet has joined #openstack-infra | 12:52 | |
| *** jhesketh__ has quit IRC | 12:53 | |
| fungi | lukego: it looks like gate-manila-tempest-dsvm-neutron uses KEEP_LOCALRC: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/manila.yaml#n23 | 12:54 |
| lukego | I looked at that example, but couldn't make it work for myself. | 12:54 |
| fungi | lukego: we don't seem to be using the DEVSTACK_GATE_TEMPEST_REGEX variable anywhere in upstream jobs | 12:54 |
| fungi | but i'll take a peek at the d-g scripts and see if it does what i think it does | 12:55 |
| lukego | thanks! | 12:55 |
| *** dprince has joined #openstack-infra | 12:55 | |
| *** thuc has joined #openstack-infra | 12:56 | |
| *** unicell has quit IRC | 12:56 | |
| lukego | fungi: with KEEP_LOCALRC the part that's most unclear to me is what directly the localrc is in, and how that relates to what directory my Jenkins script (that kicks off safe-devstack-vm-gate.sh) is running in. i.e. likely I create a localrc file in the wrong place? | 12:56 |
| *** thuc_ has joined #openstack-infra | 12:56 | |
| lukego | perhaps that would break for me but work for manilla, if we are calling devstack-vm-gate.sh in different ways. | 12:57 |
| *** jhesketh has quit IRC | 12:57 | |
| lukego | I also don't really understand how to experiment with hacks to devstack-vm-gate.sh for troubleshooting. I've tried many different ways and somehow it keeps getting overwritten by a fresh checkout from git. (I've tried disable that in various ways and it never does what I hope.) | 12:57 |
| fungi | lukego: entirely possible that the manila job is a work in progress. i'm going to look at one of their logs and see whether it's actually preserving that | 12:58 |
| lukego | thank you | 12:58 |
| lukego | (gotta pop into a meeting now, will continue troubleshooting afterwards) | 12:58 |
| fungi | lukego: for experimenting with files inside of the devstack git repos, you really want to do something like keep separate clones which devstack won't touch, and then tell devstack to pull from there | 12:59 |
| lukego | I tried that, but somehow it finds the openstack-infra project on github anyway for me :-\ | 12:59 |
| lukego | my infrastructure is setup via Jay Pipes' puppet scripts so I don't really understand the whole workflow | 12:59 |
| fungi | it should only ever look on github if it doesn't have a local cache | 12:59 |
| lukego | (bbiab) | 13:00 |
| fungi | k | 13:00 |
| *** thuc has quit IRC | 13:00 | |
| *** sarob has joined #openstack-infra | 13:03 | |
| fungi | katyafervent: wikipedia mentions a few... http://en.wikipedia.org/wiki/Globally_unique_identifier#Text_encoding | 13:03 |
| *** ociuhandu has joined #openstack-infra | 13:03 | |
| *** yamahata has joined #openstack-infra | 13:03 | |
| *** exz has joined #openstack-infra | 13:04 | |
| katyafervent | fungi, thanks so just regular format is using | 13:05 |
| fungi | katyafervent: closet thing i can find to a standard for a guid though is the uuid described in ietf rfc 4122... http://ietf.org/rfc/rfc4122.txt | 13:05 |
| katyafervent | fungi, thanks a lot) | 13:06 |
| fungi | katyafervent: you're welcome! | 13:06 |
| *** beagles has joined #openstack-infra | 13:07 | |
| *** dkranz has quit IRC | 13:07 | |
| *** rlandy_ has joined #openstack-infra | 13:08 | |
| *** sarob has quit IRC | 13:08 | |
| *** adalbas has joined #openstack-infra | 13:09 | |
| *** beagles has left #openstack-infra | 13:09 | |
| *** rfolco has joined #openstack-infra | 13:09 | |
| *** rlandy has quit IRC | 13:10 | |
| SergeyLukjanov | fungi, morning | 13:12 |
| *** mriedem has joined #openstack-infra | 13:12 | |
| fungi | morning SergeyLukjanov | 13:12 |
| SergeyLukjanov | fungi, how are you? | 13:12 |
| fungi | er, evening i guess | 13:13 |
| fungi | SergeyLukjanov: doing well--trying to catch up on e-mail | 13:13 |
| fungi | SergeyLukjanov: how goes the renaming? | 13:13 |
| SergeyLukjanov | fungi, gate is still failing | 13:13 |
| fungi | :( | 13:13 |
| fungi | any hints on what's broken? | 13:13 |
| *** talluri_ has joined #openstack-infra | 13:13 | |
| *** talluri has quit IRC | 13:13 | |
| SergeyLukjanov | fungi, one of the problems that we see yesterday was Django 16. | 13:13 |
| SergeyLukjanov | fungi, and I've already fixed it :) | 13:14 |
| SergeyLukjanov | fungi, and I've forget to update d-g yesterday, so, waiting when it'll be merged... | 13:14 |
| SergeyLukjanov | fungi, https://review.openstack.org/#/c/80237/ | 13:15 |
| SergeyLukjanov | fungi, it's needed to allow run tempest tests for savanna | 13:15 |
| SergeyLukjanov | and we could update them to be sahara after client release ;) | 13:15 |
| SergeyLukjanov | fungi, circular dependencies, 90 lvl | 13:16 |
| fungi | yeah, i remember when we renamed quantum... it was pretty messy | 13:16 |
| fungi | and our integration testing has only gotten more complex since | 13:17 |
| *** julim has joined #openstack-infra | 13:17 | |
| SergeyLukjanov | fungi, https://review.openstack.org/#/c/80237/ is passing voting jenkins, very appreciate if you can push it to make me not wait for 10 hours | 13:17 |
| *** jgallard has joined #openstack-infra | 13:17 | |
| SergeyLukjanov | fungi, yup, I find some unexpected conditionals in devstack | 13:18 |
| fungi | looks like quite the pile-up in the gate too... i keep seeing tempest-neutron jobs failing | 13:18 |
| *** markwash_ has joined #openstack-infra | 13:18 | |
| fungi | i haven't gotten to -dev list e-mail... has anybody looked into that yet? | 13:18 |
| *** changbl has joined #openstack-infra | 13:19 | |
| SergeyLukjanov | fungi, don't see anything about it | 13:19 |
| SergeyLukjanov | fungi, oh, just update mail page | 13:19 |
| SergeyLukjanov | [openstack-dev] [qa] [neutron] Neutron Full Parallel job - Last 48 hours failures | 13:19 |
| fungi | ha | 13:19 |
| SergeyLukjanov | fungi, some analytics about neutron failing reasons | 13:20 |
| *** dizquierdo is now known as dizquierdo_afk | 13:20 | |
| fungi | yeah, reading it now | 13:20 |
| *** e0ne_ has joined #openstack-infra | 13:20 | |
| *** markwash has quit IRC | 13:20 | |
| *** markwash_ is now known as markwash | 13:20 | |
| fungi | looks like that may not necessarily be about something which has recently spiked up | 13:21 |
| fungi | pretty big resurgence of bug 1257885 today | 13:21 |
| *** dkranz has joined #openstack-infra | 13:21 | |
| SergeyLukjanov | fungi, it's about sahara | 13:21 |
| fungi | oh! so that was sahara re-encountering the django 1.6 issues. got it | 13:22 |
| SergeyLukjanov | fungi, yup, our dashboard was incompatible with dj 1.6 | 13:22 |
| SergeyLukjanov | fungi, and it's support was merged only two days ago, so, that's the thing we saw before renaming | 13:23 |
| *** e0ne has quit IRC | 13:23 | |
| SergeyLukjanov | fungi, when we force pushing commit, will it reset gate? | 13:23 |
| *** dkliban has joined #openstack-infra | 13:24 | |
| *** alff has quit IRC | 13:26 | |
| fungi | it wouldn't cause a gate reset, but i think it would be better to promote it than force-merge. is it impacting any other projects? | 13:26 |
| *** mfer has joined #openstack-infra | 13:26 | |
| SergeyLukjanov | fungi, promote? | 13:26 |
| SergeyLukjanov | fungi, it affects only savanna | 13:26 |
| SergeyLukjanov | sahara* | 13:26 |
| fungi | we should reserve the manual merge option for situations where the change won't pass on its own, or is causing such mass destruction that we can't wait for one set of jobs to complete at the head of the gate | 13:27 |
| fungi | in this case, if we just promote it, it will get tested once in the gate before merging | 13:27 |
| SergeyLukjanov | fungi, nope, it breaks only sahara testing | 13:28 |
| SergeyLukjanov | fungi, promote == it'll be tested earlier? | 13:28 |
| *** unicell has joined #openstack-infra | 13:29 | |
| fungi | SergeyLukjanov: yeah, promote means the gate changes get reshuffled with that one at the top and all tests restarted at that point, but i'm looking to see whether there's any proposed patches to help whatever's caused the 10-hour backlog so i can promote them at the same time | 13:29 |
| SergeyLukjanov | fungi, it'll be really great... | 13:29 |
| openstackgerrit | Nikola Dipanov proposed a change to openstack-dev/hacking: Add TestCase assertion methods for testing checks https://review.openstack.org/80278 | 13:30 |
| SergeyLukjanov | fungi, because current gate state looks like we'll wait for unblocking sahara renaming for dozens of hours | 13:30 |
| openstackgerrit | Nikola Dipanov proposed a change to openstack-dev/hacking: Add an import whitelist helper factory https://review.openstack.org/80018 | 13:30 |
| fungi | SergeyLukjanov: quite possibly. i just want to make sure i'm correctly prioritizing productivity for the whole of openstack, so if there's something i should be doing to help move things along in the gate, i can also help unblock sahara at the same time | 13:31 |
| SergeyLukjanov | fungi, thank you | 13:31 |
| *** zns has joined #openstack-infra | 13:31 | |
| *** hashar_ has joined #openstack-infra | 13:37 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add a footprint for Bug 1291992 https://review.openstack.org/80281 | 13:38 |
| *** jasondotstar has quit IRC | 13:38 | |
| *** hashar has quit IRC | 13:40 | |
| *** matsuhashi has quit IRC | 13:42 | |
| *** e0ne_ has quit IRC | 13:43 | |
| *** dkranz has quit IRC | 13:47 | |
| sdague | fungi: more timeouts on zm02 | 13:48 |
| sdague | actually never mind | 13:49 |
| fungi | sdague: which job? | 13:49 |
| sdague | the issue is the mirror | 13:49 |
| sdague | http://logs.openstack.org/01/80001/1/check/check-grenade-dsvm/50d5a77/logs/grenade.sh.log.2014-03-13-115232 | 13:49 |
| *** dkranz has joined #openstack-infra | 13:49 | |
| fungi | grr we should be using apache mod_mime_magic or something. my browser wants to download that | 13:50 |
| sdague | hm, mine doesn't | 13:51 |
| fungi | okay, _that_ mirror | 13:51 |
| *** dkliban has quit IRC | 13:51 | |
| fungi | didn't know if you meant the git mirror or rackspace's deb mirror or... | 13:52 |
| sdague | I think the issue is mod_mime_magic is enabled | 13:52 |
| sdague | Content-Type: text/x-sh | 13:52 |
| fungi | "Cannot fetch index base URL http://pypi.openstack.org/openstack/" | 13:52 |
| fungi | so yes, looks like it... | 13:52 |
| fungi | and in rax-dfw, same region where that server is hosted (just in a separate tenant). depressing | 13:53 |
| fungi | checking to make sure static.o.o which serves that is healthy | 13:54 |
| *** hashar_ has quit IRC | 13:54 | |
| lukego | fungi: back and ready to attack again :-). rereading, it's not devstack code that I want to try hacking, it's devstack-gate code i.e. the scripts that run devstack/tempest. | 13:54 |
| fungi | lukego: yeah, i took a quick look at archived logs for that manila job, and it's not doing what they expect | 13:55 |
| lukego | fungi: do you have any tip for how to splice a line or two into the devstack localrc config? | 13:55 |
| fungi | lukego: i agree with you, the localrc needs to go wherever devstack is being run from, not in the current working directory for the job itself | 13:55 |
| *** talluri_ has quit IRC | 13:56 | |
| *** talluri has joined #openstack-infra | 13:56 | |
| fungi | lukego: i think that's the intent of the KEEP_LOCALRC variable, just need to put your initial localrc in the correct place | 13:56 |
| lukego | fungi: I tried to create a localrc elsewhere but it just got overwritten instead of appending. so either KEEP_LOCALRC didn't work or I picked the wrong path | 13:57 |
| fungi | there's also the possibility that in recent refactoring how the git repository caches for devstack-gate work, the directory where the localrc needs to go is getting wiped and replaced after the wrap script starts. should probably figure out whether that is actually happening | 13:57 |
| *** wchrisj has joined #openstack-infra | 13:57 | |
| lukego | that would be consistent with what I see. I can't find the localrc path with 'find / -name localrc' | 13:58 |
| fungi | and if so, patch devstack-gate to move and restore that file if it already exists | 13:58 |
| lukego | I'm not sure how to patch devstack-gate.. it keeps magically downloading the official upstream version from github and overwriting my changes :( | 13:58 |
| fungi | lukego: i mean submit a patch upstream to devstack-gate | 13:58 |
| lukego | would like to be able to test it first? | 13:58 |
| fungi | or maybe KEEP_LOCALRC behavior should be to copy that file from a specific safe location | 13:58 |
| *** malini_afk is now known as malini | 14:00 | |
| lukego | I have tried about 30 different tweaks to various scripts today and so far nothing has done what I expect, so I feel like there's a low chance to successfully sending changes upstream without testing them locally :) | 14:00 |
| *** talluri has quit IRC | 14:01 | |
| lukego | (it would be great to have devstack-gate run from a different repository. I forked it on github and put changes there, but it seems to only partially be used, and other stuff was automatically pulled from the upstream version for reasons I couldn't quite deduce) | 14:01 |
| lukego | i.e. it wasn't enough to hack my jenkins job definitions to clone/pull my fork | 14:01 |
| fungi | well, part of the challenge is that we want devstack-gate to test itself. one way to do that is to set up environment variables for the project and ref to test and make that devstack-gate and your change ref/repository url | 14:02 |
| fungi | i haven't done that for a few months so am not completely certain the readme is still up to date on the steps needed for that | 14:03 |
| lukego | I'll have a bit of a dig into the REGEX issue.. try to see how the other people doing Neutron plugins manage to skip non-neutron-related tests to get faster turnaround | 14:04 |
| *** mbacchi has quit IRC | 14:04 | |
| AJaeger | fungi, once you have time to do a few reviews, I would appreciate if you could approve https://review.openstack.org/#/c/79866/ to move us one step forward with translation of manuals | 14:05 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add a footprint for bug1283522 https://review.openstack.org/80286 | 14:05 |
| *** lcestari_ has joined #openstack-infra | 14:05 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/elastic-recheck: Improve fingerprint for Django 1.6 not supported issue https://review.openstack.org/80287 | 14:05 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1291953 https://review.openstack.org/80288 | 14:05 |
| sdague | fungi: also - http://logs.openstack.org/87/80087/1/gate/gate-elastic-recheck-python27/0f78a2f/console.html | 14:06 |
| sdague | tests pass, node went offline | 14:06 |
| *** dstufft has quit IRC | 14:06 | |
| *** sarob has joined #openstack-infra | 14:06 | |
| *** thomasem has joined #openstack-infra | 14:06 | |
| *** ryanpetrello has joined #openstack-infra | 14:07 | |
| *** e0ne has joined #openstack-infra | 14:07 | |
| *** dstufft has joined #openstack-infra | 14:08 | |
| *** amotoki has quit IRC | 14:09 | |
| *** caleb_ has joined #openstack-infra | 14:09 | |
| fungi | sdague: yeah, we've been tracking that one. jeblair still seems to think it's something happening in concert with changes in job registration events. on the jenkins master side of things it fails to write to the slave while starting to processing the job logs and thinks the slave's disk has filled up, so marks the slave offline (resulting in that stacktrace in the console) | 14:10 |
| *** hashar has joined #openstack-infra | 14:10 | |
| sdague | ok | 14:10 |
| *** sarob has quit IRC | 14:10 | |
| sdague | so we should reopen that? | 14:11 |
| sdague | because the infra bug is fix-released | 14:11 |
| fungi | however, almost all of them seem to happen in hpcloud-az2 that i've seen (found one incident for it in rax a while back), so not sure what's up with that correlation | 14:11 |
| devananda | good morning! got a question -- what version of fedora should we be testing our devstack patch on? | 14:11 |
| fungi | sdague: which infra bug? there was a separate infra bug from when we actually *were* running out of space on slaves in hpcloud, because they have a 10gb filesystem | 14:11 |
| sdague | we actually seem to have 2 bugs here | 14:12 |
| fungi | so the outward-facing errors are unfortunately the same | 14:12 |
| fungi | because supposedly jenkins pretty much assumes any write failure is a full filesystem, and takes the same actions | 14:12 |
| sdague | Bug 1268732 - Slave went offline during log upload | 14:13 |
| fungi | devananda: what devstack patch? | 14:13 |
| sdague | Bug 1284371 - gate error: Looks like the node went offline during the build. Check the slave log for the details.FATAL: /var/lib/jenkins/logs/slaves/*/slave.log (No such file or directory) | 14:13 |
| sdague | so 1268732 is marked as fixed, and 1284371 is open | 14:13 |
| *** adalbas has quit IRC | 14:13 | |
| sdague | and we have 2 er fingerprints that are the same :) | 14:13 |
| fungi | sdague: yeah, the open one is probably for the current round of issues | 14:13 |
| fungi | but yes, in situations where the public logs are identical for two entirely separate classes of problem, i'm not sure how you intend to fix that on the classification side of things | 14:14 |
| sdague | well, I'll just remove the old one | 14:14 |
| *** bknudson has left #openstack-infra | 14:14 | |
| sdague | and the classification will be whatever level of accuracy it is | 14:14 |
| devananda | fungi: https://review.openstack.org/#/c/70348/ <-- add ironic testing to devstack | 14:14 |
| fungi | unless we stop treating bugs as a way to track specific issues and start using them purely as a unique key for a failure classification system | 14:15 |
| *** mbacchi has joined #openstack-infra | 14:15 | |
| devananda | fungi: tldr; we've been testing on precise, since (afaik) taht is what the gate runs | 14:15 |
| fungi | devananda: well, we have precise and centos 6.5 | 14:15 |
| devananda | perfect, thanks! | 14:15 |
| devananda | cent 6.5 it is | 14:16 |
| sdague | fungi: so I get that the system isn't perfect, but having these frequency numbers, even if imperfect is useful in knowing what our top issues are | 14:16 |
| fungi | devananda: but you were asking about fedora | 14:16 |
| fungi | sdague: agreed | 14:16 |
| devananda | fungi: my bad. you're correct :) | 14:16 |
| fungi | devananda: did you want to know about fedora specifically, or were you just asking about distributions in general? | 14:16 |
| devananda | fungi: distributions in the gate | 14:17 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: remove duplicate bug https://review.openstack.org/80295 | 14:17 |
| fungi | for fedora, i think pleia2 and dprince are working on fedora 20 support, but not sure what the current state of that work is | 14:17 |
| *** markmcclain has joined #openstack-infra | 14:17 | |
| *** markmcclain has quit IRC | 14:17 | |
| *** rcleere has quit IRC | 14:18 | |
| sdague | devananda: honestly, I don't expect to see fedora for a while. I would not block that on fedora issues. | 14:18 |
| *** vhoward- has joined #openstack-infra | 14:19 | |
| devananda | sdague: ack. and I dont plan on it, but conversely, I dont want to stop someone who's interested in fixing issues there | 14:19 |
| *** markmcclain has joined #openstack-infra | 14:19 | |
| fungi | sdague: so for that log you linked a minute ago, the jenkins log once again has... | 14:20 |
| fungi | Mar 13, 2014 11:51:02 AM hudson.node_monitors.AbstractDiskSpaceMonitor markNodeOfflineIfDiskspaceIsTooLow | 14:20 |
| fungi | 0WARNING: Making bare-precise-hpcloud-az2-2696529 offline temporarily due to the lack of disk spacel | 14:20 |
| *** saju_m has quit IRC | 14:22 | |
| fungi | interestingly, if timestamps are accurate, this happened during "py27 installdeps" before "py27 develop-inst" started | 14:23 |
| *** hdd_ has joined #openstack-infra | 14:23 | |
| fungi | would be interested to get more correlation there when i find time | 14:23 |
| sdague | in what log ? | 14:23 |
| sdague | http://logs.openstack.org/87/80087/1/gate/gate-elastic-recheck-python27/0f78a2f/console.html doesn't seem to have it | 14:23 |
| fungi | sdague: matching up the jenkins service log on the corresponding master | 14:24 |
| sdague | ah | 14:24 |
| sdague | we don't have any external visibility on that, do we? | 14:24 |
| fungi | no, i think because it may contain all sorts of danger | 14:24 |
| *** adalbas has joined #openstack-infra | 14:25 | |
| *** jasondotstar has joined #openstack-infra | 14:25 | |
| dims_ | jogo, sdague, jamespage, clarkb - 0.9.8+ is good added a comment in the bug (https://bugs.launchpad.net/nova/+bug/1254872/comments/18) | 14:25 |
| fungi | full of java backtraces containing who knows what sensitive data | 14:25 |
| jamespage | dims_, ack | 14:25 |
| dims_ | jamespage, thanks again! | 14:26 |
| jamespage | dims_, np - still trying to figure out the 1.2.2 test failure on 12.04 | 14:26 |
| jamespage | its fine on 14.04 | 14:26 |
| * jamespage sighs | 14:26 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add a footprint for bug 1283522 https://review.openstack.org/80286 | 14:27 |
| dims_ | jamespage, have a link to the test runs? (just curious, may not be able to help) | 14:27 |
| jamespage | dims_, its quite possible the least helpful message ever | 14:27 |
| dims_ | jamespage, lol | 14:28 |
| jamespage | dims_, https://launchpadlibrarian.net/168625536/buildlog_ubuntu-precise-amd64.libvirt_1.2.2-0ubuntu1~cloud0_FAILEDTOBUILD.txt.gz | 14:28 |
| jamespage | libvirt: error : An error occurred, but the cause is unknown | 14:29 |
| *** zns has quit IRC | 14:29 | |
| *** thuc_ has quit IRC | 14:30 | |
| *** thuc has joined #openstack-infra | 14:30 | |
| *** mrodden has joined #openstack-infra | 14:32 | |
| *** thuc has quit IRC | 14:35 | |
| fungi | jgriffith: if you're around, i see a couple of cinder changes failing gating on snapshot mismatches in python26 unit test jobs. do you happen to know whether there's a fix in flight for that which we should prioritize in the gate? | 14:37 |
| *** gokrokve has joined #openstack-infra | 14:37 | |
| *** andreaf has joined #openstack-infra | 14:37 | |
| *** dizquierdo_afk is now known as dizquierdo | 14:39 | |
| fungi | VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Volume device file path /tmp/tmp.2O0RjVocYA/tmplaZloE/tmpEeXSxw/tmpqd0RS-cow does not exist. | 14:41 |
| *** nati_ueno has joined #openstack-infra | 14:43 | |
| *** CaptTofu has quit IRC | 14:43 | |
| *** CaptTofu has joined #openstack-infra | 14:44 | |
| fungi | aha, looks like https://launchpad.net/bugs/1291108 (not yet triaged) | 14:46 |
| *** hdd_ has quit IRC | 14:46 | |
| jeblair | fungi: good morning. does anything need my immediate attention? | 14:46 |
| *** rcleere has joined #openstack-infra | 14:47 | |
| fungi | jeblair: i don't think so. the gate's turned to tar, but it doesn't look like it's infra-related for the most part | 14:47 |
| jeblair | fungi: targate? | 14:47 |
| *** markwash has quit IRC | 14:47 | |
| fungi | so it seems | 14:48 |
| *** wenlock has joined #openstack-infra | 14:48 | |
| *** alff has joined #openstack-infra | 14:50 | |
| *** johnthetubaguy1 has joined #openstack-infra | 14:52 | |
| *** johnthetubaguy1 has quit IRC | 14:53 | |
| *** johnthetubaguy1 has joined #openstack-infra | 14:53 | |
| *** reed has joined #openstack-infra | 14:54 | |
| *** reed has quit IRC | 14:54 | |
| *** sdake has quit IRC | 14:55 | |
| *** maxbit has joined #openstack-infra | 14:55 | |
| *** sdake has joined #openstack-infra | 14:56 | |
| *** thedodd has joined #openstack-infra | 14:57 | |
| *** alff has quit IRC | 14:58 | |
| *** thuc_ has joined #openstack-infra | 14:59 | |
| *** chandan_kumar has quit IRC | 15:02 | |
| *** thomasem has quit IRC | 15:02 | |
| *** julim has quit IRC | 15:02 | |
| *** thomasem has joined #openstack-infra | 15:02 | |
| *** krotscheck has quit IRC | 15:02 | |
| *** cody-somerville- has quit IRC | 15:02 | |
| *** dstufft has quit IRC | 15:02 | |
| *** cody-somerville- has joined #openstack-infra | 15:03 | |
| *** chandan_kumar has joined #openstack-infra | 15:03 | |
| *** hashar has quit IRC | 15:03 | |
| *** jgallard has quit IRC | 15:03 | |
| *** julim has joined #openstack-infra | 15:03 | |
| *** dstufft has joined #openstack-infra | 15:03 | |
| *** prad_ has joined #openstack-infra | 15:03 | |
| *** jgallard has joined #openstack-infra | 15:03 | |
| *** hashar has joined #openstack-infra | 15:03 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Close server connect pipes on cleanup https://review.openstack.org/80304 | 15:05 |
| *** krotscheck has joined #openstack-infra | 15:05 | |
| *** CaptTofu has quit IRC | 15:05 | |
| *** johnthetubaguy has quit IRC | 15:05 | |
| *** dstanek has quit IRC | 15:06 | |
| fungi | actually, i don't suppose the gate is too tar-like... we've merged about 70 changes in the past 24 hours in the integrated gate queue. looks like someone simply approved a substantial number of changes right around 00:00 utc from the look of various graphs | 15:06 |
| *** sarob has joined #openstack-infra | 15:07 | |
| *** mkerrin has quit IRC | 15:07 | |
| *** mwagner_lap has joined #openstack-infra | 15:07 | |
| fungi | zuul spiked from ~0-1750 jobs running in a matter of minutes | 15:07 |
| jeblair | fungi: that might be the periodic jobs | 15:08 |
| fungi | oh, interesting. we run that many periodic jobs? | 15:08 |
| jeblair | fungi: yeah, periodic triggers at 0 6 * * * | 15:09 |
| jeblair | then stable at 1 6, and qa at 2 6 | 15:09 |
| fungi | ahh, right, that's the 06:00 line not the 00:00 line i was looking at | 15:09 |
| *** krotscheck has quit IRC | 15:10 | |
| *** SumitNaiksatam has quit IRC | 15:10 | |
| fungi | i wonder why we registered a spike in gerrit comment-added events around that time though | 15:10 |
| fungi | not that it was a massive spike there | 15:11 |
| *** sarob has quit IRC | 15:12 | |
| jeblair | fungi: good question; same spike shows up yesterday | 15:12 |
| fungi | and interestingly, the periodic jobs spike yesterday at 06:00 is maybe a third as big as today's | 15:13 |
| *** maxbit has left #openstack-infra | 15:13 | |
| *** maxbit has joined #openstack-infra | 15:13 | |
| fungi | so i think something must have coincided with the periodic jobs | 15:13 |
| fungi | the 06:00 spike two days ago was even shorter still | 15:14 |
| jogo | dims_: awesome! | 15:15 |
| *** jepoy has quit IRC | 15:15 | |
| jogo | dims_: now to get that rolled out in CI | 15:15 |
| jogo | sdague: https://review.openstack.org/#/c/80132/ | 15:16 |
| *** zhiwei has quit IRC | 15:16 | |
| *** caleb_ has quit IRC | 15:16 | |
| jogo | sdague: tempest patch to get rid of a bug in gate, 20 gate hits in last 48 hours | 15:16 |
| sdague | jogo: there is an actual fix in queue | 15:16 |
| jogo | sdague: ohh even better | 15:17 |
| *** kmartin has joined #openstack-infra | 15:17 | |
| fungi | sdague: any benefit to promoting the fix up to the top? | 15:17 |
| sdague | https://review.openstack.org/#/c/80301/ | 15:17 |
| sdague | fungi: it needs to pass check | 15:17 |
| jogo | sdague: in that case I will abandon | 15:17 |
| sdague | then, sure | 15:17 |
| jeblair | u'reason': u'code review expired after 1 week of no activity after a negative review, it can be restored using the `Restore Change` button under the Patch Set on the web interface', | 15:17 |
| fungi | oh, by in queue i thought you meant in the gate, got it | 15:17 |
| sdague | jogo: please look at https://review.openstack.org/#/c/80301/ | 15:18 |
| jeblair | fungi: ^ i think that might be the reason | 15:18 |
| sdague | just to see if you think it's fair | 15:18 |
| fungi | jeblair: aha! makes sense--that runs periodically as well | 15:18 |
| *** mgagne has joined #openstack-infra | 15:18 | |
| fungi | jeblair: interesting that we didn't account for skipping checks for abandoned changes--zuul wouldn't be able to leave votes on them anyway right? | 15:18 |
| *** rpodolyaka has joined #openstack-infra | 15:18 | |
| jeblair | fungi: 588 instances of that, all between 6:00 and 6:05 | 15:18 |
| fungi | yeah, that would definitely match what the graph is showing | 15:19 |
| jeblair | fungi: launchpadsync emits both a 'change-abandoned' and a 'comment-added' event | 15:20 |
| jeblair | fungi: i wonder if it has a bug | 15:20 |
| jeblair | i love that it's called launchpadsync | 15:20 |
| fungi | jeblair: i think it may. it also comments twice... https://review.openstack.org/#/c/62209/ | 15:21 |
| jeblair | we should rename it to "Evil Patch Hating Robot" | 15:21 |
| *** jergerber has joined #openstack-infra | 15:21 | |
| fungi | and the comment can be changed to "stomps all over your dreams" | 15:21 |
| *** caleb_ has joined #openstack-infra | 15:21 | |
| *** caleb_ has quit IRC | 15:22 | |
| *** caleb_ has joined #openstack-infra | 15:22 | |
| fungi | i'll see if i can spot/fix the double-comment/event bug | 15:22 |
| *** nati_ueno has quit IRC | 15:23 | |
| *** tristanC has quit IRC | 15:24 | |
| fungi | though first i need to fix yesterday's nodepool script changes which broke image updates for rax | 15:24 |
| *** tristanC has joined #openstack-infra | 15:25 | |
| *** zns has joined #openstack-infra | 15:25 | |
| *** SumitNaiksatam has joined #openstack-infra | 15:26 | |
| *** sarob has joined #openstack-infra | 15:28 | |
| jeblair | clarkb, mordred: i'm fairly strongly opposed to installing our servers outside of a venv | 15:29 |
| jeblair | s/outside/inside/ | 15:30 |
| *** markmcclain has quit IRC | 15:30 | |
| *** caleb_ has quit IRC | 15:30 | |
| jeblair | clarkb, mordred: i don't think we should have to activate a venv to run "zuul foo" or "nodepool list" | 15:30 |
| jeblair | clarkb, mordred: besides, i thought mordred fixed the known pip problems | 15:31 |
| jeblair | nibalizer: fyi ^ | 15:31 |
| *** mkerrin has joined #openstack-infra | 15:33 | |
| sdague | fungi / jeblair: there was just a spelling fix for this patch - https://review.openstack.org/#/c/80301/3 - however, given that it only deletes asserts, I'm fairly confident it can only make things better | 15:33 |
| *** odyssey4me has joined #openstack-infra | 15:34 | |
| sdague | so I'd be pro pushing it to the gate queue and promoting | 15:34 |
| openstackgerrit | A change was merged to openstack-infra/gear: Close poll loop pipes on cleanup https://review.openstack.org/76588 | 15:34 |
| *** AJaeger has quit IRC | 15:34 | |
| fungi | sdague: okay, i'll promote that one and the savanna/sahara d-g patch right behind it | 15:35 |
| sdague | cool | 15:35 |
| *** rpodolyaka has left #openstack-infra | 15:35 | |
| *** jcoufal has quit IRC | 15:35 | |
| fungi | since they're also completely blocked without that, but i didn't want to promote a change just for that | 15:35 |
| fungi | but if we're already fixing tempest failures, might as well take the opportunity | 15:35 |
| sdague | yeh, agreed | 15:36 |
| jaypipes | lukego: still around? | 15:36 |
| lukego | yep! | 15:36 |
| lukego | "Jay, thank GOD you're here!" -- do you get that a lot? | 15:36 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: remove duplicate bug https://review.openstack.org/80295 | 15:37 |
| jaypipes | lukego: lol, no not really :) but I'm sure most of the folks on this channel do get that :) want to move this convo to #openstack-neutron? | 15:37 |
| lukego | sure | 15:37 |
| *** jasondotstar has quit IRC | 15:38 | |
| *** jdurgin has joined #openstack-infra | 15:38 | |
| mtreinish | sdague: how is removing "not" from a comment a spelling fix? | 15:40 |
| mriedem | clarkb: added you to this to see what you think about the wildcards used in the query | 15:40 |
| mriedem | https://review.openstack.org/#/c/80287/1/queries/1257885.yaml | 15:40 |
| jeblair | sdague: fyi my testing for fixing https://bugs.launchpad.net/openstack-ci/+bug/1286963 is causing (and will continue to cause) a lot of false hits of that bug | 15:40 |
| mriedem | there are 12 jobs between savanna/saharra though which sucks - you could list out all 12 though | 15:40 |
| *** pcrews has joined #openstack-infra | 15:40 | |
| jeblair | sdague: (dns issues) | 15:41 |
| *** vhoward- has left #openstack-infra | 15:41 | |
| *** vkozhukalov has joined #openstack-infra | 15:41 | |
| fungi | jeblair: you can't make an omelet without breaking a few jobs | 15:42 |
| *** bada_ has quit IRC | 15:42 | |
| *** bada_ has joined #openstack-infra | 15:42 | |
| mordred | jeblair: key word "known" ... but in general, I agree that you should not need to activate a venv to run zuul foo of nodepool list | 15:42 |
| mordred | jeblair: however, I wouldn't mind having a global venv that was automatically added to the path globally if it helped pip not crap the bed (there are several ifs in there) | 15:43 |
| jeblair | mordred: yep. i think we previously agreed that your fixes (not all of which are merged, i believe) was our desired short term solution, and auto-building packages was the long term one | 15:44 |
| *** dprince has quit IRC | 15:44 | |
| *** david-lyle has joined #openstack-infra | 15:44 | |
| mordred | jeblair: nod | 15:45 |
| jeblair | mordred: also, "zuul foo of nodepool" is really funny. sounds right out of a sci-fi b-movie. :) | 15:46 |
| *** mark0n has joined #openstack-infra | 15:46 | |
| mordred | jeblair: can I change my name to zuul foo of nodepool? | 15:47 |
| jeblair | i think you just did, zuul foo of nodepool. | 15:47 |
| *** Ryan_Lane has joined #openstack-infra | 15:48 | |
| jeblair | mordred: i hope you know that title comes with a hat which you will be expected to wear at the next summit. | 15:48 |
| malini | We are trying to troubleshoot an issue at the gate, where mongo db is really slow to start up. Is there any chance we can get access to the node where the tests run ? | 15:51 |
| SergeyLukjanov | mriedem, so, it's better to explicitly list jobs? | 15:51 |
| mriedem | SergeyLukjanov: well we don't want to use wildcards because while they work, they put load on ES | 15:52 |
| fungi | malini: what job? | 15:52 |
| *** smarcet has quit IRC | 15:53 | |
| malini | fungi: check-tempest-dsvm-marconi | 15:53 |
| malini | its the experimental job in devstack | 15:53 |
| SergeyLukjanov | mriedem, okay, I'll update it later | 15:53 |
| *** thingee has joined #openstack-infra | 15:54 | |
| *** vkozhukalov has quit IRC | 15:54 | |
| fungi | malini: looks like it works some times but not others? | 15:55 |
| thingee | I was told infra registered the different irc channels recently. Who can I talk to about op rights in one of the channels or how that is handled? | 15:55 |
| malini | fungi: I have been troubleshooting around it to make sure the mongo startup is the issue | 15:56 |
| *** alexpilotti has joined #openstack-infra | 15:56 | |
| jeblair | thingee: any of us can give you ops on a channel now, and then you should be able to manage it after that | 15:56 |
| jeblair | thingee: which channel? | 15:56 |
| thingee | #openstack-cinder | 15:56 |
| pleia2 | devananda, sdague fedora work is being tracked here: https://etherpad.openstack.org/p/fedora-on-gate last review outstanding to mostly get us there is here (needs reviews) https://review.openstack.org/#/c/69510/ | 15:56 |
| thingee | jeblair: thanks | 15:56 |
| malini | fungi: the working jobs have a diff backend or sleep..but we dont see the mongo issue locally. | 15:57 |
| *** smarcet has joined #openstack-infra | 15:57 | |
| malini | fungi: it always fails at the gate with mongo backend | 15:57 |
| fungi | malini: so on this run, as a point of reference, it started successfully just after package installation... http://logs.openstack.org/94/78294/7/check/check-tempest-dsvm-marconi/415ec4a/logs/devstacklog.txt.gz#_2014-03-13_12_58_53_083 | 15:58 |
| *** jepoy has joined #openstack-infra | 15:58 | |
| malini | fungi: & this is a failed one http://logs.openstack.org/88/74088/5/experimental/check-tempest-dsvm-marconi/cb7b183/ | 15:59 |
| *** johnthetubaguy has joined #openstack-infra | 15:59 | |
| devananda | pleia2: thanks! | 15:59 |
| *** johnthetubaguy1 has quit IRC | 16:00 | |
| *** johnthetubaguy has quit IRC | 16:00 | |
| *** zns has quit IRC | 16:00 | |
| fungi | malini: so on that one the initscript also seems to successfully return control, suggesting that mongodb did in fact start | 16:01 |
| fungi | at 2014-03-12 22:57:13.527 | 16:01 |
| jeblair | thingee: I did "/msg chanserv access #openstack-cinder add thingee +Afortv" | 16:01 |
| jeblair | thingee: i also added jgriffith | 16:01 |
| malini | It starts up, but does not immediately become available | 16:01 |
| jeblair | thingee: the two of you can add other ops with the same command now | 16:02 |
| malini | fungi: ^ | 16:02 |
| *** che-arne has joined #openstack-infra | 16:02 | |
| thingee | jeblair: thank you | 16:02 |
| fungi | malini: where is the first sign of failure in the log? | 16:02 |
| fungi | malini: just wondering what the time difference is from when the service supposedly started | 16:02 |
| *** johnthetubaguy has joined #openstack-infra | 16:02 | |
| fungi | malini: "start_marconi seems to be called around 2014-03-12 23:02:41.176 | 16:03 |
| malini | fungi: I an hunting for the failed logs..1 min | 16:03 |
| *** zns has joined #openstack-infra | 16:03 | |
| *** jcoufal-mobile has joined #openstack-infra | 16:03 | |
| fungi | malini: so you're suggesting that mongodb is being slow to start... that's more than 5 minutes difference. i suspect if mongodb isn't usable 5 minutes after starting that's not just being slow | 16:04 |
| malini | fungi: See http://logs.openstack.org/88/74088/4/experimental/check-tempest-dsvm-marconi/5400c48/logs/ error.txt.gz | 16:04 |
| malini | fungi: I dont think its a 5 min difference..I can get around the issue with a sleep of 2s | 16:05 |
| fungi | malini: okay, so you're talking about "mongodb starting" in a different sense than the way it's being started by its initscript when it gets installed on the server | 16:05 |
| fungi | malini: does mongodb log its startup process? | 16:06 |
| zaro | fungi: i still have two accounts on review.o.o? | 16:06 |
| malini | fungi: I couldnt find tht in the logs | 16:06 |
| fungi | zaro: probably. we don't delete accounts | 16:06 |
| malini | fungi: & To clarify, I am refering to mongo being available after the restart | 16:07 |
| fungi | malini: if you know where the logs should be on the system, we can configure your job to collect them and add them to the set of logs it's already putting on logs.openstack.org | 16:07 |
| zaro | fungi: ohh that could be confusing. | 16:07 |
| fungi | malini: oh, okay... so you restart mongodb as part of start_marconi? | 16:07 |
| *** dripton is now known as dripton_lunch | 16:07 | |
| *** khyati_ has joined #openstack-infra | 16:08 | |
| fungi | zaro: gerrit doesn't give us a way to merge accounts, so if someone has activity (reviews, comments) under one account and then later ends up using a different account, we need to keep both | 16:08 |
| malini | fungi: yes ..it is done in configure_marconi | 16:08 |
| fungi | zaro: deleting the account would cause all their old activity to be orphaned and show comments from "anonymous" | 16:08 |
| *** markmcclain has joined #openstack-infra | 16:08 | |
| malini | flaper87: /var/log/mongodb.log is what we need, rt? | 16:08 |
| openstackgerrit | A change was merged to openstack-infra/gear: Throttle geard connect loop https://review.openstack.org/76589 | 16:09 |
| malini | flaper87: sorry /var/log/mongodb/mongodb.log | 16:09 |
| flaper87 | malini: yeah, I'd say so | 16:09 |
| flaper87 | hopefully, that'll give more insights of what's happening there | 16:10 |
| zaro | fungi: i see, maybe i'll just change the name on it so it's the same on both accounts. | 16:10 |
| *** odyssey4me has quit IRC | 16:11 | |
| *** jepoy has quit IRC | 16:11 | |
| fungi | malini: so the easiest way is at the end of the job, make sure that the mongodb log gets copied to the same logs directory that devstack-gate sticks the others in, and it will get grabbed along with them. should i think be able to use a post hook for that? | 16:12 |
| malini | fungi: We dont have a posthook already..Do you want us to add one? | 16:13 |
| zaro | opps, i can't seem to do that because it's tied to my LP account and i believe i only have 1 LP account. | 16:13 |
| malini | fungi: also do you happen to know what version of mongo db is used in the gate? | 16:13 |
| *** yassine has quit IRC | 16:14 | |
| *** caleb_ has joined #openstack-infra | 16:14 | |
| fungi | malini: i'm looking to see if there's a good example hook for that already. as for mongodb, it's whatever comes in ubuntu 12.04 lts | 16:14 |
| malini | fungi: I dont think ubuntu comes with a default mongo db..flaper87, can you confirm? | 16:15 |
| fungi | malini: http://packages.ubuntu.com/precise-updates/mongodb | 16:15 |
| flaper87 | malini: it does | 16:16 |
| *** johnthetubaguy has quit IRC | 16:16 | |
| malini | aah..ok..thts why I always check with smart ppl ;) | 16:16 |
| *** johnthetubaguy has joined #openstack-infra | 16:16 | |
| *** MarkAtwood has joined #openstack-infra | 16:17 | |
| fungi | malini: yep, we're just installing the distro-provided package according to https://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/manifests/params.pp#n129 | 16:17 |
| *** jcoufal-mobile has quit IRC | 16:17 | |
| *** mrodden has quit IRC | 16:18 | |
| *** jcoufal-mobile has joined #openstack-infra | 16:18 | |
| malini | thanks for confirming fungi! | 16:19 |
| *** rpodolyaka has joined #openstack-infra | 16:20 | |
| *** miqui has joined #openstack-infra | 16:21 | |
| *** mgagne has quit IRC | 16:22 | |
| *** bogdando has quit IRC | 16:22 | |
| *** sarob has quit IRC | 16:22 | |
| *** sarob has joined #openstack-infra | 16:23 | |
| fungi | malini: so, i'm not really seeing a great example of a post hook which copies particular logs into the right place for collection but an example inline post hook which doesn't call out into an external script within you project would look like https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#n81 | 16:23 |
| *** thuc_ has quit IRC | 16:23 | |
| *** bogdando has joined #openstack-infra | 16:24 | |
| malini | thanks fungi! I'll chk tht out | 16:24 |
| *** thuc has joined #openstack-infra | 16:24 | |
| malini | I am going to give a final try with the journal stuff & will get straight to this | 16:24 |
| *** ociuhandu has quit IRC | 16:24 | |
| *** akerr has joined #openstack-infra | 16:25 | |
| *** vkozhukalov has joined #openstack-infra | 16:26 | |
| *** akerr has left #openstack-infra | 16:26 | |
| fungi | malini: and looking at http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh you're just going to want to have your hook copy those into /opt/stack/logs/ | 16:27 |
| *** sarob has quit IRC | 16:27 | |
| *** thuc has quit IRC | 16:28 | |
| *** jcoufal-mobile_ has joined #openstack-infra | 16:28 | |
| *** jcoufal-mobile has quit IRC | 16:28 | |
| *** jcoufal-mobile_ has quit IRC | 16:28 | |
| *** jcoufal-mobile has joined #openstack-infra | 16:29 | |
| jeblair | malini: make sure it gets renamed so it has a .txt extension, then gzip it too. | 16:29 |
| *** yassine has joined #openstack-infra | 16:29 | |
| malini | jeblair: sure..will do! | 16:29 |
| malini | thanks fungi ..I might have more questions for you soon | 16:29 |
| *** gokrokve has quit IRC | 16:30 | |
| fungi | malini: good luck | 16:31 |
| jeblair | fungi, clarkb, mordred: so, looking at my test change to see how a caching-forwarding nameserver affects devstack jobs, i noticed that it seems to work fine on rax but not on hpcloud (1.0) | 16:31 |
| *** jcoufal-mobile_ has joined #openstack-infra | 16:31 | |
| malini | fungi: thanks! I need a lot of that :) | 16:31 |
| *** mrodden has joined #openstack-infra | 16:31 | |
| jeblair | fungi, clarkb, mordred: here's why: | 16:31 |
| jeblair | Mar 13 16:14:28 devstack-precise-hpcloud-az2-2709387 named[2377]: error (no valid RRSIG) resolving 'org/DS/IN': 10.6.0.1#53 | 16:31 |
| jeblair | Mar 13 16:14:28 devstack-precise-hpcloud-az2-2709387 named[2377]: error (no valid DS) resolving 'git.openstack.org/AAAA/IN': 10.6.0.1#53 | 16:31 |
| *** jasondotstar has joined #openstack-infra | 16:32 | |
| jeblair | fungi, clarkb, mordred: it kinda looks like the nameservers in hpcloud 1.0 break dnssec. | 16:32 |
| fungi | oh, so very, very nice! | 16:32 |
| *** gyee has joined #openstack-infra | 16:33 | |
| *** jcoufal-mobile_ has quit IRC | 16:33 | |
| *** jcoufal-mobile has quit IRC | 16:33 | |
| jeblair | i'm not really excited about the idea of having _recursive resolvers_ on every test node (because we should see a huge benefit from the cache memory of regional resolvers)... | 16:35 |
| *** skraynev is now known as skraynev_afk | 16:36 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Temp enable savanna service for sahara tests. https://review.openstack.org/80237 | 16:36 |
| *** sarob has joined #openstack-infra | 16:37 | |
| jeblair | but it kind of looks like our options are that or google or run our own. | 16:37 |
| *** maxbit has quit IRC | 16:37 | |
| *** rcarrillocruz has quit IRC | 16:37 | |
| *** gokrokve has joined #openstack-infra | 16:37 | |
| jeblair | or turn off dnssec | 16:38 |
| fungi | jeblair: didn't you say before that hpcloud was configuring resolv.conf to use google resolver ip addresses anyway? | 16:38 |
| jeblair | fungi: hpcloud 1.1 | 16:38 |
| fungi | ahh | 16:38 |
| jeblair | fungi: which, granted, we're about to move to | 16:39 |
| jeblair | so if we solve the 1.0 problem by forwarding to google, then we've at least made 1.0 and 1.1 the same. | 16:39 |
| *** dprince has joined #openstack-infra | 16:40 | |
| jeblair | presumably dnssec will provide some extra assurance while using google's servers. except for our own domains of course, which aren't dnssec signed. | 16:40 |
| fungi | but could be, if only we didn't use our provider's dns hosting services, since they don't support serving dnssec signed zones either | 16:41 |
| fungi | it's so disappointing all around | 16:41 |
| jeblair | :( | 16:41 |
| *** changbl has quit IRC | 16:41 | |
| *** mgagne has joined #openstack-infra | 16:42 | |
| *** sarob has quit IRC | 16:42 | |
| fungi | and the other other alternative there is running our own nameservers. we already run plenty of things, so i'm not sure it adds much new headache, but it's still more work that we already don't have time to do | 16:42 |
| *** Ryan_Lane1 has joined #openstack-infra | 16:42 | |
| jeblair | fungi: i'm continuning to dig; i don't think i fully understand the hpcloud problem yet | 16:42 |
| fungi | okay | 16:42 |
| *** krotscheck has joined #openstack-infra | 16:44 | |
| *** jepoy has joined #openstack-infra | 16:48 | |
| *** miqui has quit IRC | 16:50 | |
| clarkb | morning | 16:51 |
| fungi | anybody have any idea how often mediawiki updates its search indexes? i removed mention of openstack-ci-admins@lists.launchpad.net from GerritJenkinsGit and GerritJenkinsGith/ja yesterday, but they still show up in https://wiki.openstack.org/w/index.php?title=Special%3ASearch&profile=default&search=openstack-ci-admins&fulltext=Search | 16:51 |
| fungi | Ryan_Lane1: ^ ? | 16:51 |
| *** dripton_lunch is now known as dripton | 16:51 | |
| Ryan_Lane1 | hm. good question. we're using the CIrrusSearch stuff. it's possible that I broke it by upgrading | 16:52 |
| clarkb | jeblair: let me know what you find. I can submit a ticket or whatever if appropriate | 16:52 |
| Ryan_Lane1 | when I get home I'll take a look | 16:52 |
| *** dstanek has joined #openstack-infra | 16:52 | |
| fungi | Ryan_Lane1: okay, cool. no rush, was just curious | 16:52 |
| *** maxbit has joined #openstack-infra | 16:52 | |
| * Ryan_Lane1 nods | 16:52 | |
| *** miqui has joined #openstack-infra | 16:53 | |
| *** sarob has joined #openstack-infra | 16:53 | |
| *** mgagne has quit IRC | 16:55 | |
| fungi | clarkb: i've been digging into this, but not sure what we're missing... http://paste.openstack.org/show/73426/ | 16:55 |
| jeblair | clarkb, fungi: i installed unbound to get a second opinion, and even without configuring forwarding mode, it reported this: | 16:56 |
| jeblair | Mar 13 16:50:42 devstack-precise-hpcloud-az2-2709387 unbound: [5087:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN | 16:56 |
| *** dkliban has joined #openstack-infra | 16:56 | |
| Ryan_Lane1 | ls | 16:56 |
| jeblair | that's _theoretically_ straight from the root nameservers | 16:56 |
| jeblair | i'm reading tcpdumps now | 16:56 |
| *** zns has quit IRC | 16:57 | |
| *** caleb_ has quit IRC | 16:57 | |
| fungi | clarkb: i inserted a little extra debugging into the restrict script, and it says grub-legacy-ec2 0.6.3-0ubuntu1.10 is installed but confirms that /etc/default/grub is nonexistent when that script is running | 16:57 |
| clarkb | fungi: huh so we have precise with grub 1 and precise with grub 2? | 16:57 |
| fungi | clarkb: i'm seeing those in at least iad and ord, probably dfw as well | 16:57 |
| *** hashar has quit IRC | 16:57 | |
| clarkb | fungi: I should probably rewrite that to [ -f /etc/default/grub ] ; then do grub 2 elif [ -f /boot/grub/grub.conf ] ; then do grub 1 fi | 16:58 |
| *** khyati_ has quit IRC | 16:58 | |
| clarkb | because I think the pvhvm image is grub 2 if I remember correctly from jeblairs testing | 16:58 |
| fungi | clarkb: that might make more sense. i have a similar patch mostly drafted, but wasn't sure if i was missing something obvious | 16:58 |
| clarkb | fungi: no, I think the images just differ in ways I didn't expect | 16:58 |
| clarkb | eg all precise don't use the same major version of grub apparently | 16:59 |
| sdague | pleia2: so as soon as that is getting anywhere close, it would be good to get it non voting on devstack | 16:59 |
| *** thuc has joined #openstack-infra | 16:59 | |
| sdague | because that info would be incredibly useful | 16:59 |
| fungi | right. again this is provider stuff. i think grub-legacy-ec2 is basically a stub to let you include configuration within your filesystem for a bootloader which is getting invoked external from the vm itself | 16:59 |
| fungi | by the host | 16:59 |
| sdague | we break fedora all the time because no feedback | 17:00 |
| pleia2 | sdague: so my focus was tripleo at first, haven't really talked much about nodepool in regular infra using fedora | 17:00 |
| clarkb | fungi: gotcha, then ya direct tests for config seem like a much better way to go | 17:00 |
| clarkb | fungi: you are writing that patch? | 17:02 |
| fungi | clarkb: incoming | 17:02 |
| clarkb | great, I am going to try and get through reviews again | 17:03 |
| *** denis_makogon_ has joined #openstack-infra | 17:03 | |
| *** jepoy has quit IRC | 17:03 | |
| *** dstufft_ has joined #openstack-infra | 17:05 | |
| *** maxbit has quit IRC | 17:05 | |
| *** maxbit has joined #openstack-infra | 17:05 | |
| *** gokrokve has quit IRC | 17:06 | |
| *** sakhome has joined #openstack-infra | 17:06 | |
| *** dstufft_ has quit IRC | 17:08 | |
| *** dstufft_ has joined #openstack-infra | 17:08 | |
| *** markmc has quit IRC | 17:08 | |
| *** e0ne has quit IRC | 17:09 | |
| *** rcarrillocruz has joined #openstack-infra | 17:09 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:09 | |
| *** markwash has joined #openstack-infra | 17:09 | |
| sdague | pleia2: ok, I'll express similar concerns to you that I did to SpamapS yesterday. There is a ton of shoring up of behavior of components that could be done in the main gate that will take a while to get to tripleo | 17:09 |
| sdague | so I'd be careful about jumping that step. In a lot of ways the main gate is the unit tests for tripleo | 17:10 |
| *** harlowja_away is now known as harlowja | 17:10 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Restrict nodepool memory only when possible https://review.openstack.org/80345 | 17:12 |
| *** BobBall is now known as BobBall_Away | 17:14 | |
| *** vogxn has joined #openstack-infra | 17:16 | |
| *** dstufft_ has quit IRC | 17:17 | |
| *** dstufft_ has joined #openstack-infra | 17:18 | |
| clarkb | http://summit.openstack.org/cfp/details/59 btw | 17:18 |
| pleia2 | sdague: so redhat is bringing up their cloud (in addition to hp's) for tripleo testing, and we wanted full fedora and full ubuntu story there, in some ways it's easier to start with tripleo on things like fedora because the scope is still limited and won't impact many projects | 17:18 |
| *** sakhome has quit IRC | 17:19 | |
| *** dstanek has quit IRC | 17:19 | |
| pleia2 | clarkb: nice | 17:19 |
| jeblair | ah | 17:23 |
| jeblair | clarkb, fungi: https://bugs.launchpad.net/ubuntu/+source/unbound/+bug/988513 | 17:24 |
| jeblair | clarkb, fungi: now i hate resolvconf again | 17:24 |
| clarkb | huh | 17:24 |
| jeblair | clarkb, fungi: the reason i was still seeing the broken dnssec behavior even without a forwarding configuration is because i was, unknowingly, using a forwarding configuration | 17:24 |
| *** sarob has quit IRC | 17:25 | |
| clarkb | :( | 17:25 |
| *** derekh has quit IRC | 17:25 | |
| jeblair | clarkb, fungi: so i think we're back to "hpcloud 1.0 dns servers break dnssec" | 17:25 |
| fungi | jeblair: so you're getting dnskey errors in hpcloud but not in rax? | 17:25 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make IRC bot list which failures were seen in which job. https://review.openstack.org/78790 | 17:25 |
| jeblair | fungi: yeah | 17:26 |
| *** khyati has joined #openstack-infra | 17:26 | |
| *** gokrokve has joined #openstack-infra | 17:26 | |
| clarkb | if we drop an /etc/default/unbound in before the package install will that work? | 17:28 |
| clarkb | I suppose that would only help to disable dnssec which we don't want | 17:28 |
| jeblair | clarkb: i'm not focused on solving the problems in that bug atm; | 17:29 |
| jeblair | clarkb: (we might need to, but i'd like to set it aside for the moment and figure out how we actually want things configured) | 17:30 |
| *** maxbit has left #openstack-infra | 17:30 | |
| jeblair | clarkb, fungi: i have confirmed that if i set RESOLVCONF_FORWARDERS=false dnssec works | 17:31 |
| jeblair | (so we can at least bypass the hp resolvers and run a recursive resolver) | 17:31 |
| fungi | it would be nice if we could have a magical pony which used dnssec when available and otherwise just used traditional dns queries, but i don't see that at the moment without hard-coding provider-specific behavior | 17:32 |
| *** mgagne has joined #openstack-infra | 17:32 | |
| *** mgagne has quit IRC | 17:32 | |
| *** mgagne has joined #openstack-infra | 17:32 | |
| *** blamar has joined #openstack-infra | 17:32 | |
| *** changbl has joined #openstack-infra | 17:33 | |
| jeblair | forwarding to 8.8.8.8 from hpcloud also works | 17:33 |
| clarkb | pleia2: re https://review.openstack.org/#/c/69510/8/modules/openstack_project/files/nodepool/scripts/prepare_tripleo.sh can you expand a bit on how tripleo is different? I mean they are differnet but I am trying to come up with a scenario were prepare_devstack would interfere in a bad way | 17:34 |
| jeblair | clarkb, fungi: so should we aim to configure hpcloud to forward to 8.8.8.8, and rax to their local resolvers? | 17:36 |
| jeblair | clarkb, fungi: (and then, having decided that, see if https://bugs.launchpad.net/ubuntu/+source/unbound/+bug/988513 affects us) | 17:36 |
| clarkb | jeblair: I think that wfm since the other half of hpcloud is doing it for all vms anyways | 17:37 |
| fungi | i had some vague memory, from before the root was signed, that isc maintained an altroot with resolvers available you could recurse through. hunting now to see whether they still maintain that resolver network, but maybe point it at the global root trust instead hopefully | 17:37 |
| *** atiwari has joined #openstack-infra | 17:37 | |
| pleia2 | clarkb: cached repo dependencies are determined differently | 17:38 |
| *** yassine has quit IRC | 17:38 | |
| *** dstufft_ has quit IRC | 17:38 | |
| pleia2 | clarkb: it might be ok to use/include devstack's script for doing the repo pulls, then just adding in specific ones tripleo needs in its script | 17:38 |
| *** dstufft_ has joined #openstack-infra | 17:39 | |
| *** dstufft_ is now known as Guest72267 | 17:39 | |
| *** _sweston_ has quit IRC | 17:40 | |
| clarkb | pleia2: but keeping that straight is probably harder than a shared install dependencies script | 17:40 |
| clarkb | pleia2: I think I prefer the current patchset | 17:40 |
| clarkb | pleia2: thanks | 17:40 |
| *** nithyag has joined #openstack-infra | 17:41 | |
| *** Guest72267 has quit IRC | 17:43 | |
| openstackgerrit | Stefano Maffulli proposed a change to openstack-infra/config: Added list for Content team https://review.openstack.org/80351 | 17:44 |
| fungi | yeah, i think with the rash of ddos attacks using open resolvers for traffic amplification, any open resolvers the isc or other organizations may have maintained were pulled down once the dlv was no longer relevant | 17:44 |
| *** dprince_ has joined #openstack-infra | 17:45 | |
| clarkb | what is the openstack content team? | 17:45 |
| fungi | so at this point, either forwarding to teh goog' or running our own recursive resolvers in each provider/zone (likely restricting them to use by our systems) seems like the available options | 17:46 |
| fungi | clarkb: volunteer copy writers assembling openstack-specific marketing collateral | 17:47 |
| *** sarob has joined #openstack-infra | 17:48 | |
| *** Hefeweizen has quit IRC | 17:48 | |
| *** Hefeweizen has joined #openstack-infra | 17:48 | |
| jeblair | fungi: restricting them to use by our systems is likely tricky -- probably needs neutron or vpn | 17:50 |
| jeblair | clarkb, fungi: just to throw this out there, should we forward to goog on all our single-use nodes (ie, instead of rax resolvers?) | 17:50 |
| *** atiwari has quit IRC | 17:51 | |
| clarkb | jeblair: isn't that a bit mean? | 17:51 |
| fungi | jeblair: i agree on the tricky part (that was sort of my point in mentioning it) | 17:51 |
| jeblair | clarkb: what's mean? | 17:51 |
| clarkb | jeblair: sending 900 machines to google all at once | 17:51 |
| clarkb | it is google, but still | 17:51 |
| jeblair | clarkb: that's different than 450? | 17:51 |
| *** rcarrillocruz has quit IRC | 17:51 | |
| clarkb | jeblair: we weren't responsible for the 450 :) (and actually its only about 5 right now because we only use 5 nodes in region b) | 17:52 |
| jeblair | clarkb: right, but we just decided that 1.0 needs to forward to google | 17:52 |
| fungi | clarkb: do you know anyone involved in that decision, btw, so that you can find out whether hp has some sort of agreement with them not to throttle/blacklist their cloud at the drop of a hat when the volume rises? | 17:53 |
| clarkb | fungi: I don't | 17:53 |
| clarkb | fungi: best I could tell it is an oversight apparently there are DNS servers that should be used as in 1.0 | 17:53 |
| *** gyee has quit IRC | 17:53 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for nova unit test bug 1291605 https://review.openstack.org/80087 | 17:53 |
| jeblair | clarkb: oh, ha! | 17:53 |
| *** dprince has quit IRC | 17:54 | |
| *** rossella_ has quit IRC | 17:54 | |
| fungi | so we very well may find we get similar lookup problems if all our systems stampede 8.8.8.8 and google begins throttling access from hpcloud | 17:54 |
| *** pblaho has quit IRC | 17:54 | |
| clarkb | but it doesn't help that those servers aren't working properly either | 17:54 |
| jeblair | https://developers.google.com/speed/public-dns/docs/security | 17:54 |
| jeblair | the bit at the bottom pertains to this | 17:55 |
| fungi | https://developers.google.com/speed/public-dns/docs/security#rate_limit yeah | 17:55 |
| *** dkranz has quit IRC | 17:56 | |
| *** jpich has quit IRC | 17:56 | |
| jeblair | honestly, i don't expect it to be a problem. i don't think we do _that_ many dns queries in jobs, and we're talking about forwarding cache misses | 17:56 |
| fungi | "per-client-IP QPS and average bandwidth limits" is nice and vague | 17:56 |
| *** dkranz has joined #openstack-infra | 17:57 | |
| clarkb | jeblair: this is true the slaves themselves don't talk to much outside themselves | 17:57 |
| fungi | i agree. i expect the actual number of queries to be very, very low once we're caching on each machine | 17:57 |
| *** rpodolyaka has quit IRC | 17:57 | |
| *** rcarrillocruz has joined #openstack-infra | 17:57 | |
| jeblair | so a nameserver is going to see a bunch of *.openstack.org, *.ubuntu.com, *.python.org queries, then nothing from that ip for the next 20-60 minutes. | 17:57 |
| clarkb | they will need to lookup the jenkins master (because ssh), anything serving refs, and thats about it right? | 17:57 |
| *** jooools has quit IRC | 17:57 | |
| clarkb | oh right package repo locations too | 17:57 |
| *** rcarrillocruz has quit IRC | 17:57 | |
| jeblair | (i mean, assuming we reset our ttls back to 3600) | 17:58 |
| fungi | so here's another (possibly distasteful) option... | 17:58 |
| *** changbl has quit IRC | 17:58 | |
| *** rcarrillocruz has joined #openstack-infra | 17:58 | |
| fungi | since we're not running long-lived servers in hpcloud anyway, and we probably don't actually care that nodepool nodes get dnssec validation on lookups, turn off dnssec validation on nodepool nodes but turn it on for long-lived servers (where rax is at least usable) | 17:59 |
| *** markwash has quit IRC | 17:59 | |
| fungi | and hope that hpcloud fixes their dnssec issues soon, so we can go ahead and enable it everywhere after that | 18:00 |
| *** rpodolyaka has joined #openstack-infra | 18:00 | |
| jeblair | fungi: aside from the dnssec issues, hpcloud1.0 is going away so i don't feel like it's worth too much to support their broken resolvers | 18:00 |
| jeblair | clarkb: how do you know that hpcloud 1.1 has resolvers of its own? | 18:00 |
| clarkb | jeblair: I semi loudly mentioned that all of hpcloud 1.1 was using google to resolve dns because dhcp and was told by someone that he has DNS servers there but doesn't manage dhcp so isn't sure why the servers aren't used | 18:01 |
| *** jgallard has quit IRC | 18:01 | |
| jeblair | clarkb: is it worth a ticket to get the dhcp thing fixed? | 18:02 |
| *** reed has joined #openstack-infra | 18:02 | |
| clarkb | jeblair: maybe? for us probably not since we are working around broken DNS in hpcloud | 18:02 |
| jeblair | clarkb: can you ask that person if the resolvers work with dnssec? | 18:02 |
| clarkb | jeblair: I can | 18:02 |
| jeblair | clarkb: i'm just thinking about making hpcloud better. :) | 18:02 |
| clarkb | yup | 18:02 |
| jeblair | clarkb: i expect none of this will get resolved before we want to put unbound in | 18:03 |
| clarkb | agreed | 18:03 |
| jeblair | clarkb: also, if you get an ip for those servers, you can check with "dig com. SOA +dnssec" (look for the ad flag to be set) | 18:03 |
| clarkb | let me walk down to that side of the building to see if I can learn more | 18:03 |
| clarkb | biab | 18:03 |
| openstackgerrit | A change was merged to openstack-infra/gerrit: Add a Change Owner group https://review.openstack.org/65133 | 18:04 |
| *** flaper87 is now known as flaper87IsNotAwa | 18:04 | |
| *** flaper87IsNotAwa is now known as flaper87DoesNotK | 18:04 | |
| *** flaper87DoesNotK is now known as flaper87WTF | 18:05 | |
| *** flaper87WTF is now known as flaper87 | 18:05 | |
| jeblair | clarkb, fungi: it sounds like our preference would be to use a non-google local resolver when possible? so we should leave rax using its resolvers and special case hpcloud to use google until the rest gets worked out? | 18:06 |
| jeblair | clarkb, fungi: (as opposed to just using google both places) | 18:06 |
| *** hogepodge has joined #openstack-infra | 18:06 | |
| fungi | i'm just always shaky about using google for things, but also think that the provider should be providing functional recursive dns resolution too | 18:08 |
| fungi | so yeah, that seems like a pragmatic, if moderately distasteful, compromise | 18:08 |
| *** zns has joined #openstack-infra | 18:08 | |
| clarkb | gah | 18:08 |
| clarkb | ok so after more digging | 18:08 |
| clarkb | we might be ok (at least from a why google) | 18:09 |
| *** sarob__ has joined #openstack-infra | 18:09 | |
| clarkb | in 1.1 neutron is used | 18:09 |
| *** zhiyan is now known as zhiyan_ | 18:10 | |
| clarkb | in my west instances the default network gives me nameserver 10.0.0.2 | 18:10 |
| *** sarob has quit IRC | 18:10 | |
| clarkb | in my much older east network google was used | 18:11 |
| fungi | ohhh | 18:11 |
| fungi | so this is stale configuration basically | 18:11 |
| clarkb | So we should see what we get on an actual slave | 18:11 |
| clarkb | fungi ya I think so | 18:11 |
| clarkb | it is so hard to remember this neutron thing is magical like that | 18:12 |
| *** lcestari_ has quit IRC | 18:12 | |
| *** lcestari has quit IRC | 18:12 | |
| *** reed has quit IRC | 18:12 | |
| *** dizquierdo has quit IRC | 18:12 | |
| *** reed has joined #openstack-infra | 18:12 | |
| clarkb | however the 10.0.0.2 server seems to have the same trouble as 1.0 which means we are back to google | 18:13 |
| clarkb | `dig com. SOA +dnssec` gives me flags: qr rd ra; | 18:13 |
| clarkb | jeblair: ^ | 18:13 |
| *** nati_ueno has joined #openstack-infra | 18:14 | |
| jeblair | clarkb: want to open anoter ticket to ask them to fix their resolvers? | 18:15 |
| clarkb | jeblair: yup can do | 18:15 |
| *** bknudson has joined #openstack-infra | 18:15 | |
| jeblair | clarkb: you got 10.0.0.2 as a nameserver? i spot checked an hpcloud-region-b node and see 206.164.176.34 | 18:15 |
| clarkb | jeblair: huh | 18:15 |
| clarkb | is that dyndns? | 18:16 |
| jeblair | clarkb: dnssec is still broken | 18:16 |
| clarkb | I wonder if they offloaded by default in the past but new networks get the local neutron resolver | 18:16 |
| clarkb | which is why I see google on my account's old networks | 18:16 |
| jeblair | clarkb: i think that's an hp address | 18:16 |
| clarkb | eds.com according to dig -x | 18:16 |
| jeblair | so, yep. | 18:17 |
| clarkb | dig com. SOA +dnssec @206.164.176.34 doesn't resolve anything very quickly for me | 18:18 |
| clarkb | apparently can't hit that from where I sit | 18:18 |
| *** atiwari has joined #openstack-infra | 18:18 | |
| jeblair | clarkb: yeah. it's fast on the region b node i'm on. | 18:18 |
| fungi | might be restricted by clients | 18:19 |
| clarkb | so I think there are a couple things here. 1. 1.1 default nameserver as provided by neutron in hpcloud 2. dnssec support on those defaults | 18:19 |
| clarkb | jeblair: ns1.eds.com is not dnssec happy either? | 18:19 |
| *** caleb_ has joined #openstack-infra | 18:19 | |
| jeblair | clarkb: what's ns1.eds.com? | 18:19 |
| clarkb | jeblair: 206.164.176.34 | 18:19 |
| *** caleb_ has quit IRC | 18:20 | |
| jeblair | clarkb: ns1.eds.com.86270INA192.85.154.78 | 18:20 |
| clarkb | oh gah thats an SOA not an A record | 18:20 |
| jeblair | clarkb: clarkb ns1.eds.com looks like the authoritative server for the domain | 18:20 |
| *** caleb_ has joined #openstack-infra | 18:20 | |
| clarkb | yeah I read the SOA record as an A record because special | 18:20 |
| jeblair | clarkb: i'm not sure we care whether they ... ok. | 18:20 |
| clarkb | so 206.164.176.34 does that do dnssec? | 18:21 |
| jeblair | no | 18:21 |
| *** caleb_ has quit IRC | 18:21 | |
| clarkb | kiall: ^ is there anything you can add? | 18:21 |
| * clarkb just remembered that kiall does dns stuff for hpcloud | 18:21 | |
| *** caleb_ has joined #openstack-infra | 18:22 | |
| *** caleb_ has quit IRC | 18:22 | |
| *** caleb_ has joined #openstack-infra | 18:22 | |
| *** caleb_ has quit IRC | 18:22 | |
| openstackgerrit | Luke Gorrie proposed a change to openstack-infra/devstack-gate: Fixed DEVSTACK_GATE_TEMPEST_REGEX. https://review.openstack.org/80359 | 18:23 |
| *** dangers_away has joined #openstack-infra | 18:23 | |
| *** caleb_ has joined #openstack-infra | 18:23 | |
| fungi | lukego: ^ you figured out what was going wrong there i take it? | 18:24 |
| jeblair | clarkb: i'm not sure that rax servers do dnssec validation, but they at least pass the rrsig records; perhaps that's enough for unbound to do it itself. | 18:24 |
| *** caleb_ has quit IRC | 18:24 | |
| jeblair | clarkb: it looks like hpcloud doesn't do validation and truncates responses | 18:24 |
| lukego | fungi: Looks like it. (I couldn't test "for real" because my installation doesn't seem to serve as development environment) | 18:24 |
| *** caleb_ has joined #openstack-infra | 18:25 | |
| lukego | my edit-compile-run loop for devstack-gate actually involves getting changes merged upstream at the moment, which I realize it not optimal, but don't know how to fix.. | 18:25 |
| *** johnthetubaguy has quit IRC | 18:26 | |
| jeblair | lukego: hrm. there used to be documentation about it | 18:26 |
| jeblair | lukego: there's a variable called SKIP_DEVSTACK_GATE_PROJECT | 18:26 |
| jeblair | lukego: but that seems to have gone missing from the README | 18:26 |
| *** jlibosva has quit IRC | 18:26 | |
| sdague | jeblair: so a question on gate-noop, what is it exactly, and can we get away without it? | 18:27 |
| *** johnthetubaguy has joined #openstack-infra | 18:27 | |
| jeblair | lukego: but in the source it says "Set this variable to skip updating the devstack-gate project itself. Useful in development so you can edit scripts in place and run them directly." | 18:27 |
| sdague | because I realize this is an area I'm fuzzy on | 18:27 |
| lukego | jeblair: !!! thanks :-) | 18:27 |
| *** johnthetubaguy has quit IRC | 18:27 | |
| jeblair | sdague: i believe that zuul needs a job in the gate pipeline in order for a change to trigger the pipeline for the project (and of course, you need to trigger the gate pipeline if you want something merged) | 18:28 |
| lukego | So I am sorry if this is a stupid question, but is it really a good idea for random guys like me to be required to run all of this infrastructure? I feel like these are internal tools of the openstack infrastructure community, and I'm just a guy who commited a 200-line extension that nobody except my customer's customers cares about working | 18:29 |
| sdague | jeblair: so does gate-noop actually allocate physical resources at this point? | 18:29 |
| sdague | that was something that came up during the discussion of job templates, which was new to me | 18:29 |
| jeblair | sdague: yes, it wastes an entire bare-precise node. it's ridiculous. | 18:30 |
| notmyname | is logstash happy? I keep getting 502 Bad Gateway responses to queries | 18:30 |
| jeblair | sdague: zuul was designed for openstack which has tests for all of its projects. the stackforge project with no tests thing was a bit of a surprise. | 18:31 |
| fungi | lukego: well, devstack-gate is intended to be a tool developers can use to perform integration testing on their own, and lots of organizations outside of upstream openstack are using it for that purpose. it also has somewhat of a test suite now, so if you have features you want to keep working in it, add validation tests for them | 18:31 |
| *** vogxn has quit IRC | 18:31 | |
| sdague | jeblair: well I was also thinking about the -specs repos | 18:31 |
| *** velker has joined #openstack-infra | 18:31 | |
| sdague | because we want to make one for qa as well | 18:31 |
| jeblair | sdague: (heh, those should have tests too probably :) | 18:31 |
| jeblair | sdague: at any rate, it should not be to difficult to have zuul handle it internally | 18:32 |
| sdague | ok, so you think it's a zuul internal change vs. something else on the gate-noop definition | 18:32 |
| jeblair | sdague: either with some special casing, or by adding a new kind of trigger, or even having zuul itself register a 'noop' gearman worker. | 18:32 |
| *** dstanek has joined #openstack-infra | 18:32 | |
| jeblair | s/trigger/launcher/ | 18:33 |
| fungi | notmyname: i'm testing it now and get query results | 18:33 |
| fungi | notmyname: is it intermittent, or repeatable for you? | 18:33 |
| jaypipes | anyone else noticed paste.openstack.org is sluggish in last couple weeks? | 18:33 |
| jeblair | sdague: yes, the only way to run a job for zuul right now is via jenkins | 18:34 |
| *** velker has quit IRC | 18:34 | |
| clarkb | jeblair: have the 1.0 nameserver handy? | 18:34 |
| sdague | a bare precise is unit test node, or is that now also single use? | 18:34 |
| * clarkb is writing up a thing to send to support@hp | 18:34 | |
| jeblair | sdague: bare-precise is a single-use unit test node | 18:34 |
| fungi | jaypipes: it's seemed that way to me too, but the cacti graphs suggest it's not resource-starved | 18:34 |
| notmyname | fungi: repeatable, it seems. try this query: project:"openstack/swift" AND build_status:"FAILURE" AND build_name:"check-tempest-dsvm-full" | 18:34 |
| clarkb | fungi: jaypipes: I think the old version of drizzle may be to blame | 18:34 |
| jaypipes | yeah? | 18:35 |
| jaypipes | interesting. | 18:35 |
| sdague | notmyname: in what time window? | 18:35 |
| clarkb | yeah its running what precise has which is ancient | 18:35 |
| notmyname | last 24 hours | 18:35 |
| jeblair | clarkb: 10.6.0.1 | 18:35 |
| clarkb | and linuxjedi has alawys said it should be upgraded for all the reasons | 18:35 |
| sdague | notmyname: so that's going to be all lines on all failures | 18:35 |
| *** sarob__ has quit IRC | 18:35 | |
| sdague | it would not surprise me if that overwhelmed it | 18:35 |
| pleia2 | clarkb: if the summit doesn't overlap, might be nice to do a CI overview talk here: https://www.usenix.org/conference/lisa14/call-for-participation/papers-posters (I thought about submitting one myself, but you're in Seattle!) | 18:36 |
| pleia2 | it overlaped last year, sad sad | 18:37 |
| fungi | notmyname: i do actually get results, but the result is so huge my browser thinks the javascript is running too long and wants me to kill it | 18:37 |
| notmyname | fungi: ok, thanks | 18:37 |
| *** amcrn has quit IRC | 18:37 | |
| fungi | and oh wow is it ever dragging down my browser with it now | 18:37 |
| rcarrillocruz | helo helo, any core reviewer with some time to review https://bugs.launchpad.net/openstack-ci/+bug/1227985 ? | 18:38 |
| *** jp_at_hp has quit IRC | 18:38 | |
| notmyname | sdague: I'm happy for someone else to narrow it down more efficiently. (I'm doing this for you, actually.) this is what I need: http://logs.openstack.org/04/71704/6/check/check-tempest-dsvm-full/d268267/console.html#_2014-03-13_01_02_02_560 | 18:38 |
| clarkb | pleia2: ya but overlapping is what I am worried about | 18:38 |
| clarkb | jeblair: fungi https://etherpad.openstack.org/p/hpcloud-dns-issues I will fill in the account details when I send the email (and not on the etherpad, not sure how sensitive that data is) | 18:38 |
| pleia2 | clarkb: hopefully we'll have dates before CFP closes :) | 18:38 |
| rcarrillocruz | puhlease :-) | 18:38 |
| jeblair | rcarrillocruz: on it | 18:40 |
| sdague | notmyname: so this is actually a better starting point - http://logstash.openstack.org/#eyJzZWFyY2giOiJwcm9qZWN0Olwib3BlbnN0YWNrL3N3aWZ0XCIgQU5EIG1lc3NhZ2U6XCJGaW5pc2hlZDogXCIgQU5EIGJ1aWxkX25hbWU6XCJjaGVjay10ZW1wZXN0LWRzdm0tZnVsbFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzk0NzM1OTAyNTU4fQ== | 18:41 |
| sdague | that's all the Finished results on swift tests in those jobs | 18:42 |
| rcarrillocruz | thx jeblair | 18:43 |
| *** yolanda has quit IRC | 18:43 | |
| jeblair | rcarrillocruz: done! | 18:43 |
| clarkb | jeblair: fungi can I get feedback on that etherpad? | 18:44 |
| clarkb | if it looks good I will submit it as a ticket | 18:44 |
| fungi | clarkb: yeah, sorry, was restarting my browser after that kibana... ahem... experiment | 18:44 |
| *** thuc has quit IRC | 18:45 | |
| jeblair | clarkb: yeah. the only thing i'm wondering is if we should suggest that just passing the dnssec records through would be nice, even if their resolvers don't validate. but we could just reserve that as a fallback. | 18:46 |
| *** thuc has joined #openstack-infra | 18:46 | |
| fungi | clarkb: lgtm, also agree with jeblair on providing suggestions as to *what* we'd consider "support dnssec" to mean in this context | 18:48 |
| clarkb | fungi: jeblair: can one of you add that? | 18:48 |
| rcarrillocruz | jeblair: sorry, i didn't give you the review link: https://review.openstack.org/#/c/79205/ , zaro and SergeyLukjanov already put some comments on it | 18:48 |
| jeblair | rcarrillocruz: oh i thought you were asking me to review the bug | 18:48 |
| clarkb | I'm not sure how I wouldword the out iwthout making it too tempting | 18:49 |
| rcarrillocruz | yeah, my bad, had two tabs opened, gave you the bug report instead of the review | 18:49 |
| *** thuc has quit IRC | 18:50 | |
| lukego | jeblair: I added to my Jenkins job "export SKIP_DEVSTACK_GATE_PROJECT=1" but on the slave /home/jenkins/workspaces/workspace/dsvm-tempest-full/devstack-gate/devstack-vm-gate.sh was overwritten with a pristine copy anyway. do you see what I did wrong? | 18:53 |
| *** mrmartin has joined #openstack-infra | 18:54 | |
| fungi | lukego: are you running it from a jenkins job? | 18:56 |
| *** thuc has joined #openstack-infra | 18:56 | |
| lukego | yep | 18:56 |
| *** thuc has quit IRC | 18:56 | |
| fungi | lukego: and is that jenkins job using something akin to our gerrit-git-prep builder, or maybe the jenkins git plugin or something which might reset the copy of the devstack-gate repository in that workspace? | 18:56 |
| lukego | should I actually be using devstack-vm-gate-dev.sh ? | 18:56 |
| *** thuc has joined #openstack-infra | 18:57 | |
| lukego | fungi: I've set it up according to Jay's blog walkthrough, jenkins-jobs slurping .yaml files and such | 18:57 |
| fungi | lukego: right, your jenkins job is most likely resetting that repository clone in the workspace | 18:57 |
| lukego | oh! | 18:57 |
| lukego | right :) | 18:57 |
| lukego | I will try disabling that bit of the job. thanks for catching :) | 18:58 |
| *** rcarrillocruz has quit IRC | 18:58 | |
| *** dstufft_ has joined #openstack-infra | 18:59 | |
| *** dstufft_ is now known as Guest42225 | 18:59 | |
| clarkb | jeblair: fungi: happy with the etherpad's current state? if so I will send that off now | 19:00 |
| clarkb | oh wait the out was removed | 19:00 |
| jeblair | fungi's sentence just got deleted | 19:00 |
| fungi | clarkb: yeah, i think jeblair's parenthetical is sufficient | 19:00 |
| * fungi deleted his sentence | 19:00 | |
| *** Guest42225 has quit IRC | 19:00 | |
| *** dstufft- has joined #openstack-infra | 19:01 | |
| jeblair | yeah, i think the first thing (validation) is something that would be good to have. i think the second thing (truncating responses) is broken. | 19:01 |
| fungi | i can put it back if you think the parenthetical there needs reiterating | 19:01 |
| clarkb | fungi: I don't think it hurts | 19:01 |
| fungi | putting back, improved | 19:01 |
| *** changbl has joined #openstack-infra | 19:02 | |
| fungi | clarkb: jeblair: yeah? | 19:02 |
| jeblair | ++ | 19:02 |
| clarkb | yup looks good. /me throws that in an email with account details | 19:03 |
| fungi | great | 19:03 |
| *** dstufft- has quit IRC | 19:05 | |
| *** dstufft- has joined #openstack-infra | 19:06 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: Create qa-specs git repo https://review.openstack.org/80371 | 19:07 |
| sdague | are we still only doing project creates on fridays? | 19:07 |
| clarkb | sdague: yes | 19:08 |
| *** dstufft- has quit IRC | 19:08 | |
| clarkb | also I think the only real outstanding issue with those is the race between gerrit and cgit | 19:08 |
| clarkb | fungi: ^ last week went pretty well, the jeepyb patches seem to make jeepyb happy | 19:09 |
| sdague | ok, so anything else I need to do to get that queued for tomorrow's create batch? | 19:09 |
| *** dstufft- has joined #openstack-infra | 19:09 | |
| jeblair | sdague: set the topic to new-project | 19:09 |
| jeblair | sdague: also, erm, are we sure we want to extend the experiment past nova? | 19:09 |
| *** dstufft is now known as dstufft-- | 19:09 | |
| fungi | clarkb: excellent! sorry i had to bail in the middle of it and missed the excitement | 19:09 |
| sdague | jeblair: yes | 19:09 |
| jeblair | sdague: can we maybe at least try it with one project a little bit first maybe? | 19:10 |
| *** marun has quit IRC | 19:10 | |
| sdague | so we hit 3 items in today's qa meeting that we need something like this for | 19:10 |
| *** dstufft- is now known as dstufft | 19:10 | |
| clarkb | fungi: speaking of did we approev the config change change? | 19:10 |
| clarkb | fungi: if not maybe we should since it was tested at least with manage-projects | 19:10 |
| fungi | clarkb: which one? | 19:10 |
| *** dstufft is now known as Guest37858 | 19:11 | |
| jeblair | russellb: please set the topic to 'new-project' for https://review.openstack.org/#/c/79363/ | 19:11 |
| jeblair | cause right now, kite is the only one on the docket to be created | 19:11 |
| sdague | ok, how do I set topic for an already pushed review? | 19:11 |
| *** Guest37858 has quit IRC | 19:11 | |
| clarkb | fungi: the one that makes jeepyb.util read the config. I don't see it in the list of open changes so it must've been approved. /me checks git | 19:12 |
| jeblair | sdague: push a new patchset and add the '-t' option? | 19:12 |
| *** dstufft- has joined #openstack-infra | 19:12 | |
| fungi | clarkb: ahh, that, yes it got merged | 19:12 |
| clarkb | fungi: yup merged I5bc2e81388420ce03d5379b8d75063481887243b | 19:12 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: Create qa-specs git repo https://review.openstack.org/80371 | 19:12 |
| sdague | ok, had to force a rebase to make it different enough to push | 19:13 |
| *** dstufft- has quit IRC | 19:13 | |
| *** adalbas has quit IRC | 19:13 | |
| *** dstufft- has joined #openstack-infra | 19:13 | |
| jeblair | sdague: would you not be willing to wait even a week or two? | 19:13 |
| clarkb | jeblair: fungi mail sent | 19:13 |
| fungi | sdague: you could have put a period on the end of the commit title--then it would have let you change the topic <evil grin> | 19:13 |
| *** rpodolyaka has quit IRC | 19:13 | |
| jeblair | fungi: i often do that! :) | 19:14 |
| sdague | jeblair: I think if we wait the reality is that it that 3 blueprints just stall until we get this | 19:14 |
| *** dstufft- is now known as dstufft | 19:14 | |
| jeblair | sdague: i'm mostly thinking about whether the initial attempt might run into anything like "oh, it should be a branch, not a repo" or other similar assumptions turn out to be invalid and are hard to correct | 19:14 |
| sdague | because existing communication channels are not working | 19:14 |
| sdague | what do you mean on branch vs. repo? | 19:15 |
| jeblair | sdague: as in, should this be a branch of an existing project? | 19:16 |
| jeblair | (it came up in the discussion) | 19:16 |
| sdague | ok | 19:16 |
| sdague | I'm generally pretty anti branch if the branch ends up as radically different content | 19:16 |
| sdague | because it's tough on discovery | 19:17 |
| sdague | jeblair: which discussion did that come up in? I don't remember seeing it on the thread | 19:17 |
| jeblair | yeah. i'm not actually arguing that. i'm only advocating caution. if all the projects create -specs repos before anyone has even tried it once, i fear we'll have a mess to clean up. | 19:18 |
| sdague | jeblair: sure | 19:18 |
| *** vkozhukalov has quit IRC | 19:18 | |
| sdague | we just hit 3 very clear issues where we needed someone to write up a design that we could review this way in today's meeting | 19:18 |
| *** mwagner_lap has quit IRC | 19:19 | |
| sdague | so, honestly, I would rather get started here, because if we wait, the people interested in this are all going to go off to other things and the discussion will die out | 19:19 |
| sdague | strike while the iron is hot | 19:19 |
| *** marun has joined #openstack-infra | 19:21 | |
| *** zns has quit IRC | 19:22 | |
| jaypipes | SergeyLukjanov: your email sig still says Savanna :) | 19:24 |
| SergeyLukjanov | jaypipes, :( | 19:25 |
| *** adalbas has joined #openstack-infra | 19:25 | |
| * jaypipes writes a check dated 2013 to SergeyLukjanov :) | 19:25 | |
| SergeyLukjanov | jaypipes, I have a bunch of t-shirts with savanna on em | 19:25 |
| SergeyLukjanov | jaypipes, and I have a sticker savanna on my laptop | 19:25 |
| jaypipes | SergeyLukjanov: they will be worth a huge amount on antiques roadshow in 100 years :) | 19:26 |
| SergeyLukjanov | jaypipes, quite difficult to rename everything ;) | 19:26 |
| jaypipes | indeed | 19:26 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Create nova-specs git repo https://review.openstack.org/79363 | 19:26 |
| *** achuprin has joined #openstack-infra | 19:26 | |
| *** sarob has joined #openstack-infra | 19:27 | |
| *** malini has left #openstack-infra | 19:28 | |
| *** nati_ueno has quit IRC | 19:33 | |
| *** atiwari has quit IRC | 19:35 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Create a nova-docker repo https://review.openstack.org/79900 | 19:35 |
| *** zns has joined #openstack-infra | 19:36 | |
| *** maxbit has joined #openstack-infra | 19:40 | |
| *** sarob has quit IRC | 19:40 | |
| *** amcrn has joined #openstack-infra | 19:41 | |
| *** sarob has joined #openstack-infra | 19:41 | |
| *** maxbit has quit IRC | 19:42 | |
| *** sarob has quit IRC | 19:42 | |
| *** e0ne has joined #openstack-infra | 19:45 | |
| *** mark0n has quit IRC | 19:46 | |
| *** andreaf has quit IRC | 19:47 | |
| *** gokrokve has quit IRC | 19:50 | |
| *** gokrokve has joined #openstack-infra | 19:50 | |
| *** nicedice has joined #openstack-infra | 19:52 | |
| *** gokrokve has quit IRC | 19:52 | |
| *** smarcet has left #openstack-infra | 19:54 | |
| *** ildikov_ has quit IRC | 19:55 | |
| *** jcoufal has joined #openstack-infra | 19:56 | |
| *** maxbit has joined #openstack-infra | 20:00 | |
| *** maxbit has quit IRC | 20:01 | |
| *** caleb_ has quit IRC | 20:02 | |
| *** adalbas has quit IRC | 20:06 | |
| *** atiwari has joined #openstack-infra | 20:07 | |
| ryanpetrello | anybody have time to peek at https://review.openstack.org/#/c/79839/ ? | 20:08 |
| ryanpetrello | hoping to make a new pecan release soon-ish w/ official pypy support | 20:08 |
| achuprin | Hi Infra! | 20:11 |
| achuprin | I'm working on deploying a CI environment for MagnetoDB! | 20:12 |
| achuprin | I have some questions about integration with OpenStack Infra. | 20:12 |
| achuprin | before I sent an email to openstack-infra@lists.openstack.org. link here http://www.mail-archive.com/openstack-infra@lists.openstack.org/msg01013.html | 20:13 |
| achuprin | For testing MagnetoDB project we need specific environment (for example instaled cassandra claster and so on), so we cannot use devstack gate for this. | 20:14 |
| *** rpodolyaka has joined #openstack-infra | 20:14 | |
| achuprin | I am interested in the following way of integration: | 20:14 |
| achuprin | (from mail) Adding own public cloud to Infra's NodePool as new provider which will be used to provide the specific environment (jenkins slaves) for Magnetodb project. Adding own Jenkins job and script to Infra for this. For all tasks use Openstack Infra infrastructure (Jenkins, Zuul etc.) | 20:15 |
| *** julim has quit IRC | 20:15 | |
| achuprin | I would like to clarify the procedure of organization of this type of integration | 20:16 |
| achuprin | Now I see it as: | 20:16 |
| *** jcoufal has quit IRC | 20:16 | |
| achuprin | 1. Adding own public cloud to Infra's NodePool as new provider | 20:16 |
| achuprin | 2. Adding new scripts for preparing image | 20:16 |
| achuprin | 3. Adding own job to Infra's via Jenkins Job Builder ) | 20:17 |
| *** mrmartin has quit IRC | 20:17 | |
| achuprin | 4. Adding infomaition about new jobs to zool | 20:17 |
| achuprin | What do you think about this? | 20:17 |
| *** rpodolyaka has quit IRC | 20:19 | |
| *** adalbas has joined #openstack-infra | 20:19 | |
| achuprin | *3. Adding own job to Infra's Jenkisns via Jenkins Job Builder | 20:20 |
| achuprin | *4. Adding infomaition about new jobs to Infra's Zuul | 20:23 |
| openstackgerrit | Maria Nita proposed a change to openstack/requirements: Add xvfbwrapper package for running Selenium tests headless https://review.openstack.org/80393 | 20:25 |
| openstackgerrit | Maria Nita proposed a change to openstack/requirements: Add xvfbwrapper package for running Selenium tests headless https://review.openstack.org/80393 | 20:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Add pypy jobs to pecan. https://review.openstack.org/79839 | 20:29 |
| *** rpodolyaka has joined #openstack-infra | 20:32 | |
| *** vkozhukalov has joined #openstack-infra | 20:32 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Hide "Create Project" for non-superusers https://review.openstack.org/79939 | 20:32 |
| *** rcarrillocruz has joined #openstack-infra | 20:34 | |
| *** achuprin has quit IRC | 20:34 | |
| *** dkliban has quit IRC | 20:36 | |
| *** achuprin has joined #openstack-infra | 20:37 | |
| *** achuprin has quit IRC | 20:38 | |
| *** rpodolyaka has quit IRC | 20:39 | |
| *** sahumada has quit IRC | 20:39 | |
| *** achuprin has joined #openstack-infra | 20:39 | |
| *** rpodolyaka has joined #openstack-infra | 20:40 | |
| clarkb | ok back from lunch | 20:40 |
| *** caleb_ has joined #openstack-infra | 20:41 | |
| achuprin | I will describe the same thing (about integrations with infra) in an email | 20:43 |
| achuprin | thanks, bye | 20:43 |
| clarkb | achuprin: hold on | 20:44 |
| clarkb | I amtyping up a resposne now :) | 20:44 |
| lifeless | clarkb: how does one work around | 20:44 |
| achuprin | ок | 20:44 |
| lifeless | tox -epy27 | 20:44 |
| lifeless | Traceback (most recent call last): | 20:44 |
| lifeless | ... | 20:44 |
| lifeless | tox.ConfigError: ConfigError: substitution key 'posargs' not found | 20:44 |
| clarkb | achuprin: we like adding clouds. Quick question about that though. Are you wanting to add your cloud so that it can run jobs on specialized hardware or so it can supplement the existing test slaves? | 20:44 |
| clarkb | lifeless: downgrade tox to 1.6.1. The 1.7.0 release is not backward compatible | 20:44 |
| lifeless | thanks | 20:45 |
| lifeless | semver fail | 20:45 |
| achuprin | for testing we need specific slave | 20:46 |
| clarkb | achuprin: if your goal is to supplement the existing test slaves you should be able to submit a JJB change to begin testing on the existing slaves as soon as that goes in | 20:46 |
| clarkb | achuprin: gotcha, in that case we probably need to talk more about redundancy and all of that. The tripleo cloud is currently working on bringing up a second AZ from red hat so that they are redundant and can do gating | 20:47 |
| clarkb | jeblair: ^ I haven't said anything wrong yet have I? | 20:47 |
| *** caleb_ has quit IRC | 20:48 | |
| clarkb | achuprin: so it may help to talk more about what you would like to do from a testing standpoint. Forget that Zuul, Jenkins, and nodepool exist and describe what the test requirements are | 20:49 |
| clarkb | achuprin: then from that we can figure out the best way to do it | 20:49 |
| achuprin | I understood now I'll describe task more specifically | 20:51 |
| *** nati_ueno has joined #openstack-infra | 20:51 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerrit: Fix Schema_86 to use correct UUID for Change Owner group https://review.openstack.org/65134 | 20:54 |
| achuprin | clarkb: we need the environment with installed cassandra claster (for example), devstack all-in-one and MagnetoDB. | 20:54 |
| achuprin | clarkb: all this can live in one slave | 20:55 |
| *** jamielennox|away is now known as jamielennox | 20:55 | |
| *** sarob has joined #openstack-infra | 20:55 | |
| *** arosen2 has joined #openstack-infra | 20:55 | |
| arosen2 | Hi, I'm curious why does jenkins run: gate-tempest-dsvm-full on neutron patches? that doesn't use neutron so it seems running that doesn't do us any good? | 20:57 |
| achuprin | clarkb: I intended to use his prepared image for this | 20:57 |
| clarkb | arosen2: beacuse sdague is working to have standard list of tests for everyone which in some cases includes tests that may not always apply | 20:57 |
| achuprin | clarkb: image which will be used by nodepool | 20:57 |
| clarkb | achuprin: isn't that something you could do on the existing slaves? | 20:58 |
| arosen2 | clarkb: I'm not sure i see the benefit for running it though. It seem unnecessary. | 20:58 |
| clarkb | you could install cassandra and magnetodb then use the existing devstack-gate machinery to get devstack | 20:58 |
| arosen2 | clarkb: my patch has been bumped from the gate a few times because of failures in that :/ | 20:59 |
| * arosen2 my neutron patch | 20:59 | |
| clarkb | arosen2: you will need to take it up with sdague | 20:59 |
| arosen2 | clarkb: k , just curious | 20:59 |
| clarkb | arosen2: the tl;dr as I understand it is we need to start somewhere in normalizing the tests | 20:59 |
| sdague | arosen2: it's an artifact of the normalization that happens | 20:59 |
| sdague | I was pondering some kind of idea of an exclude, but we didn't get there yet | 21:00 |
| sdague | arosen2: yeh, so a high reset gate means a high reset gate, and until the various races get addressed, this is going to remain this painful for everyone | 21:01 |
| *** mbacchi has quit IRC | 21:02 | |
| *** pdmars has quit IRC | 21:02 | |
| achuprin | clarkb: how we can do it | 21:03 |
| *** caleb_ has joined #openstack-infra | 21:04 | |
| achuprin | clarkb: i understood, thank you very much | 21:05 |
| achuprin | clarkb: now we need only create own job | 21:06 |
| achuprin | clarkb: and run this job on existing slave | 21:07 |
| *** dprince_ has quit IRC | 21:07 | |
| clarkb | achuprin: yup. I think we should still talk about adding clouds if that is somethign you want to do, but you should be able to run your tests on the existing clouds and get that going in parallel | 21:07 |
| arosen2 | sdague: I understand. Though it seems gating with nova-network on a neutron change would only add more potential resets to the gate unnecessarily | 21:08 |
| *** lukego has quit IRC | 21:08 | |
| sdague | arosen2: sure, in fairness I get people coming at me with pitch forks all the time that neutron is reseting their non networking changes | 21:10 |
| sdague | so I get that it's not great, but we're either all in this together, or we aren't | 21:11 |
| arosen2 | sdague: though in those cases they are uses neutron as well? | 21:11 |
| sdague | how does cinder use neutron? | 21:11 |
| *** yassine has joined #openstack-infra | 21:13 | |
| sdague | the reality is in the flat job list we are running code that covers more than the straight subset of what the code being changed could impact | 21:13 |
| achuprin | clarkb: we'll think about it, thanks for the help | 21:13 |
| sdague | however, every time we hand tuned all this, we actually screwed up in the other direction and missed an interaction | 21:13 |
| sdague | which caused gate wedges | 21:13 |
| *** gyee has joined #openstack-infra | 21:14 | |
| arosen2 | sdague: The vms that nova is launching are on neturon networks. I imagine it's possible for nova to take a special code path when using neutron that could potentially have cinder stuff in it where a cinder change could break neutron support. Unlikely i agree. | 21:14 |
| mattoliverau | Morning all | 21:14 |
| sdague | arosen2: if you want to spend the next week going through all those interactions and sorting, feel free. The far more useful approach is help nail some of these bugs so we're back to the gate we had a week ago | 21:14 |
| sdague | which was sailing things through | 21:15 |
| thingee | sdague, arosen2: I'm not aware of cinder interacting directly with neutron fwiw. As arosen2, it would have to be some path in which it's making calls to cinder. | 21:15 |
| thingee | also I'm missing a lot of context and just jumping in with that info | 21:15 |
| arosen2 | thingee: right i'm just giving an example.. | 21:16 |
| arosen2 | in the neutron case there will never be a time when one is running neutron with nova network so it seems like a different scenario | 21:16 |
| arosen2 | sdague: I'm not coming at you with a pitch fork btw just curious the reasoning :) | 21:16 |
| *** dstufft-- has quit IRC | 21:17 | |
| *** rcarrillocruz1 has joined #openstack-infra | 21:17 | |
| arosen2 | I hear where you are coming from though | 21:17 |
| sdague | arosen2: yeh, so the ideal world neutron is able to replace nova-network and we only have one networking path | 21:17 |
| sdague | in the non ideal world that we are in, neutron will probably keep running this job | 21:17 |
| sdague | consider it incentive :) | 21:18 |
| arosen2 | sdague: :) sounds good. Thanks for the explanation. | 21:18 |
| *** e0ne has quit IRC | 21:18 | |
| *** rcarrillocruz has quit IRC | 21:19 | |
| *** dstanek is now known as dstanek_afk | 21:19 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test caching-forwarding nameserver https://review.openstack.org/80045 | 21:21 |
| *** thuc has quit IRC | 21:26 | |
| *** achuprin has quit IRC | 21:26 | |
| *** thuc has joined #openstack-infra | 21:26 | |
| *** adalbas has quit IRC | 21:28 | |
| openstackgerrit | Ziad Sawalha proposed a change to openstack-infra/config: satori: Drop requirements, pypi, and old upstream https://review.openstack.org/79975 | 21:29 |
| *** ryanpetrello has quit IRC | 21:30 | |
| *** sarob has quit IRC | 21:30 | |
| *** thuc has quit IRC | 21:30 | |
| *** sarob has joined #openstack-infra | 21:31 | |
| *** markmcclain has quit IRC | 21:31 | |
| jeblair | clarkb, fungi: in testing the unbound change with nodepool; what set of servers do you think i need to do? should one each of bare-precise and bare-centos6 in rax, hpcloud 1.0, and hpcloud 1.1, along with one tripleo-precise in tripleo do it? | 21:31 |
| fungi | jeblair: yes, the precise variants shouldn't have any impact on that test | 21:31 |
| fungi | jeblair: and i think (read:HOPE) it's safe to assume images/regions in rax will behave consistently, same among hpcloud 1.0, and then among hpcloud 1.1 | 21:32 |
| jogo | clarkb: care to chime in on the '*' in ES discussion https://review.openstack.org/#/c/80286/ | 21:33 |
| clarkb | fungi: I wouldn't count on 1.1 and 1.0 being consistent as they are very different | 21:33 |
| clarkb | fungi: jeblair but that list looks good | 21:33 |
| fungi | clarkb: i mean consistent within themselves | 21:33 |
| clarkb | fungi: oh right | 21:33 |
| clarkb | fungi: the exception is rax-iad devstack-precise-check because it is the pvhvm image | 21:33 |
| fungi | 1.0 az1, az2 and az3 consistent with one another | 21:33 |
| clarkb | jeblair: list looks good otherwise | 21:34 |
| clarkb | jogo: sure | 21:34 |
| clarkb | mordred: https://review.openstack.org/#/c/55721/ is related to the novnc discussion happening on the ml. Maybe someone should bring that up there too | 21:34 |
| jogo | thanks, as that whole thing is to appease the ES gods re:you | 21:34 |
| fungi | clarkb: do you think the pvhvm image is likely to differ for this? i guess it's remotely possible they put those in a different network or something | 21:34 |
| clarkb | jogo: done | 21:36 |
| *** Ryan_Lane has quit IRC | 21:36 | |
| clarkb | fungi: it has differed in weird ways | 21:36 |
| clarkb | eg grub, curl, and so on | 21:37 |
| *** Ryan_Lane has joined #openstack-infra | 21:37 | |
| *** Ryan_Lane has quit IRC | 21:37 | |
| *** Ryan_Lane has joined #openstack-infra | 21:37 | |
| *** dizquierdo has joined #openstack-infra | 21:37 | |
| *** thomasem has quit IRC | 21:37 | |
| jogo | clarkb: thanks | 21:38 |
| fungi | clarkb: it's definitely differed in list of packages installed by default (add lvm and gettext to that list) | 21:38 |
| clarkb | jogo: fungi on the ES note reviews for https://review.openstack.org/#/c/79447/1 would be great | 21:38 |
| fungi | er, lvm2 | 21:38 |
| *** thuc has joined #openstack-infra | 21:39 | |
| clarkb | oh you know what let me update that chang really fast so that we can CD it | 21:39 |
| *** maxbit has joined #openstack-infra | 21:42 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Better logstash field data. https://review.openstack.org/79447 | 21:43 |
| clarkb | fungi: jogo jeblair ^ that should make the logstash field data change less of a coordinated effort | 21:43 |
| fungi | wow... my internet connection just decided to go on hiatus for almost 5 minutes | 21:43 |
| fungi | surprised i didn't time out | 21:43 |
| jogo | clarkb: ahh the console change yeah | 21:44 |
| jogo | why the name change for console anyway? | 21:44 |
| clarkb | jogo: because there is a console format that devstack uses now to | 21:44 |
| clarkb | so we want to make it less console.html specific and more this is shell console output | 21:44 |
| jogo | ahh | 21:45 |
| jogo | so more then just jenkins console output? | 21:45 |
| clarkb | yup | 21:45 |
| jogo | gotcha | 21:45 |
| clarkb | and with the changes like that I am pretty sure any of those files and services can be restarted in any order | 21:46 |
| *** maxbit has quit IRC | 21:47 | |
| jeblair | clarkb: lgtm | 21:47 |
| *** dstufft_ has joined #openstack-infra | 21:47 | |
| ekarlso | are there not any tests running xenserver anymore ? | 21:47 |
| *** ildikov_ has joined #openstack-infra | 21:48 | |
| *** sarob has quit IRC | 21:49 | |
| *** sweston has joined #openstack-infra | 21:50 | |
| *** dkliban has joined #openstack-infra | 21:51 | |
| *** sarob has joined #openstack-infra | 21:51 | |
| *** ryanpetrello has joined #openstack-infra | 21:51 | |
| *** ryanpetrello has quit IRC | 21:52 | |
| *** alff has joined #openstack-infra | 21:53 | |
| *** mriedem has quit IRC | 21:54 | |
| *** sarob has quit IRC | 21:55 | |
| clarkb | jeblair: fungi I am going to approve that logstash change will babysit it | 21:56 |
| fungi | clarkb: excellent | 21:56 |
| clarkb | but queues are small right now | 21:56 |
| jeblair | assuming you can visually demultiplex 7 log streams in real time, running nodepoold locally is a great way to debug the build scripts! | 21:58 |
| jeblair | make a change, and odds are, one of the 7 builds will be just about to upload it and start running it | 21:58 |
| *** david-lyle has quit IRC | 21:58 | |
| fungi | jeblair: well, it helps that the loglines say the name/location of the image, so you cound filter/split the output | 21:59 |
| fungi | i wonder if it would be beneficial to actually separate out the logfiles for those different builds rather than interleaving them in one log | 21:59 |
| jeblair | fungi: yes but i didn't think of that when i started it :) | 22:00 |
| fungi | heh | 22:00 |
| jeblair | fungi: i would love that, but i'm not immediately sure how. | 22:00 |
| *** dstufft has quit IRC | 22:00 | |
| *** dstufft_ has quit IRC | 22:00 | |
| jeblair | fungi: (with python-logging) | 22:00 |
| sdague | jeblair: so interesting thing - https://review.openstack.org/#/c/79949/ | 22:00 |
| fungi | right, we'd need dynamic log targets based on the image name, which could get hairy | 22:00 |
| sdague | apparently reverify bug ##### | 22:00 |
| sdague | spawns the job in check and gate at the same time | 22:01 |
| *** dstufft has joined #openstack-infra | 22:01 | |
| jeblair | sdague: it's probably related to the fact that a change depends on it | 22:01 |
| sdague | ok | 22:01 |
| fungi | which got approved 2 minutes after the reverify, yeah | 22:02 |
| jeblair | sdague: zuul walked down the tree and enqueued the child, then walked back up and enqued the parent | 22:02 |
| sdague | oh, so that dragged it across | 22:02 |
| *** dstufft has quit IRC | 22:02 | |
| sdague | interesting edge condition | 22:02 |
| jeblair | sdague: fix is to add something like the "require-approval" thing to the pipeline itself (rather than the trigger) | 22:02 |
| jeblair | sdague: (is a code change to zuul) | 22:02 |
| fungi | since parent/child enqueuing is not based on a trigger for the affected change in that case | 22:03 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add a footprint for bug 1283522 https://review.openstack.org/80286 | 22:03 |
| jeblair | fungi: yup | 22:03 |
| *** dstufft has joined #openstack-infra | 22:03 | |
| *** denis_makogon_ has quit IRC | 22:05 | |
| *** maxbit has joined #openstack-infra | 22:05 | |
| *** dstufft has quit IRC | 22:05 | |
| openstackgerrit | A change was merged to openstack-infra/gerrit: Fix Schema_86 to use correct UUID for Change Owner group https://review.openstack.org/65134 | 22:05 |
| *** maxbit has left #openstack-infra | 22:06 | |
| *** dstufft has joined #openstack-infra | 22:07 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Fixed mobile footer padding. https://review.openstack.org/80410 | 22:07 |
| openstackgerrit | A change was merged to openstack-infra/config: Better logstash field data. https://review.openstack.org/79447 | 22:09 |
| *** sdake has quit IRC | 22:11 | |
| *** dstufft has quit IRC | 22:13 | |
| *** dstufft has joined #openstack-infra | 22:14 | |
| *** rlandy__ has joined #openstack-infra | 22:15 | |
| *** dstufft has quit IRC | 22:15 | |
| *** rpodolyaka has quit IRC | 22:15 | |
| jeblair | clarkb: i think we are going to run into that ubuntu bug with unbound; were you thinking that having puppet drop the file into place before installing the package might work around it? | 22:16 |
| *** dstufft has joined #openstack-infra | 22:16 | |
| jeblair | (package requires file requires directory)? | 22:16 |
| *** fifieldt has joined #openstack-infra | 22:16 | |
| *** changbl has quit IRC | 22:16 | |
| clarkb | jeblair: yup | 22:16 |
| *** alff has quit IRC | 22:16 | |
| *** rlandy_ has quit IRC | 22:17 | |
| *** dcramer__ has quit IRC | 22:17 | |
| *** rpodolyaka has joined #openstack-infra | 22:18 | |
| *** dstufft has quit IRC | 22:19 | |
| jeblair | separately, on centos, puppet decided to write /etc/resolv.conf about halfway through, before starting unbound | 22:19 |
| *** dstufft has joined #openstack-infra | 22:19 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: remove the check-devstack-dsvm-neutron job https://review.openstack.org/80414 | 22:19 |
| jeblair | so it got about halfway through and then bombed out | 22:19 |
| clarkb | :/ | 22:21 |
| jeblair | i don't really know what to do about that | 22:22 |
| clarkb | jogo: ok, build_short_uuid is a thing now, as is filename less path in tags | 22:22 |
| jeblair | mattoliverau, nibalizer: ping? | 22:23 |
| *** caleb_ has quit IRC | 22:23 | |
| clarkb | jogo: sdague so if you want you can switche everything now and ignore older data, or wait two weeks then change the queries | 22:23 |
| *** alex-gone is now known as alex-meetingz | 22:23 | |
| fungi | sometimes it feels like puppet puntd a bit on bootstrapping problems | 22:23 |
| clarkb | jogo: I think the uuid stuff can be updated now because that is only used to correlate current failures to current logs | 22:23 |
| jogo | clarkb: excellent, I don't want to loose any history in the short term | 22:23 |
| clarkb | jogo: but the filename: to tags: will probably have to wait | 22:24 |
| jogo | clarkb: I was just going to say that | 22:24 |
| sdague | clarkb: yeh, loosing data right now would be bad | 22:25 |
| clarkb | sdague: yup agreed, just giving you options :) | 22:25 |
| *** rlandy__ has quit IRC | 22:25 | |
| *** mrda_away is now known as mrda | 22:26 | |
| *** dizquierdo has quit IRC | 22:29 | |
| mattoliverau | jeblair: pong | 22:29 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Task list now shows associated project https://review.openstack.org/80417 | 22:29 |
| clarkb | jeblair: is there a require between resolv.conf and the unbound service? | 22:30 |
| clarkb | jeblair: are we being strict about order there or playing the odds | 22:30 |
| mattoliverau | jeblair: what clarkb said, if it's puppet then a a require is needed in the resov.conf part and the Service["unbound"], if that is how you've got it working, sorry stuck reading scroll back :) | 22:31 |
| *** rcleere has quit IRC | 22:34 | |
| *** jergerber has quit IRC | 22:34 | |
| *** rcarrillocruz has joined #openstack-infra | 22:35 | |
| jeblair | mattoliverau, clarkb: really strict: resolv.conf requires service requires package | 22:35 |
| clarkb | jeblair: oh is this a badly behaved daemon? | 22:36 |
| jeblair | mattoliverau, clarkb: but since it's a _huge_ puppet operation, it installs the package then installs 500 other packages, then does some more stuff then installs the resolv.conf | 22:36 |
| clarkb | where you start it and it tells init it is ready right away? | 22:36 |
| jeblair | mattoliverau, clarkb: do we need to set service to ensure=>running? | 22:37 |
| clarkb | jeblair: possibly | 22:37 |
| clarkb | to force it up | 22:37 |
| *** rcarrillocruz1 has quit IRC | 22:37 | |
| jeblair | i'll give that a shot | 22:37 |
| mattoliverau | jeblair: doesn't hurt to be more specific. | 22:39 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use unbound https://review.openstack.org/80092 | 22:40 |
| jeblair | if that doesn't work, i'm afraid we may need to do this all outside of puppet. | 22:42 |
| jeblair | (like we do with pip) | 22:43 |
| nibalizer | jeblair: hi | 22:44 |
| *** prad_ has quit IRC | 22:44 | |
| jeblair | nibalizer: was wondering if you or mattoliverau had any crazy ideas about the above problem | 22:45 |
| jeblair | nibalizer: (short version is that puppet installs a dns server, then writes /etc/resolv.conf to use it before getting around to actually starting it) | 22:45 |
| jeblair | nibalizer: i'm trying again with ensure=>running on the service | 22:46 |
| *** CaptTofu has joined #openstack-infra | 22:46 | |
| *** wchrisj has quit IRC | 22:46 | |
| jeblair | nibalizer: (but i'm not sure that means "ensure running before doing anything that depends on this" or "ensure it's running before puppet exits") | 22:46 |
| *** fifieldt has quit IRC | 22:46 | |
| clarkb | wow I was going to try updating e-r to use the build_short_uuid field and it is a bit more complicated than I thought | 22:47 |
| *** jgrimm has quit IRC | 22:47 | |
| clarkb | there is a bunch of test data that uses it too and so on | 22:47 |
| * clarkb will leave it to someone that understand what is going on there | 22:47 | |
| nibalizer | jeblair: code? | 22:48 |
| jeblair | nibalizer: https://review.openstack.org/80092 | 22:49 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: drop neutron-full (non voting job) on most projects https://review.openstack.org/80419 | 22:49 |
| *** dkliban has quit IRC | 22:50 | |
| sdague | jeblair / clarkb: https://review.openstack.org/#/c/80414 and https://review.openstack.org/80419 should save us some nodes | 22:50 |
| sdague | given that we've been flat out on quota for most of the week | 22:50 |
| *** julim has joined #openstack-infra | 22:51 | |
| jeblair | nibalizer, clarkb, mattoliverau: i think it made it through... | 22:51 |
| nibalizer | jeblair: yea that code looks good | 22:51 |
| clarkb | jeblair: that was with ensure running? | 22:51 |
| *** ryanpetrello has joined #openstack-infra | 22:51 | |
| nibalizer | also redhat does the whole 'services off by default' thing so maybe thats whats biting you? | 22:51 |
| jeblair | nibalizer, clarkb, mattoliverau: interestingly, it installed unbound, started it, did lots of other stuff, updated the config to add the forwarder bit, then restarted it. | 22:51 |
| nibalizer | you would /have/ to specify ensure => running to get be on | 22:51 |
| jeblair | nibalizer: ah, right, possibly so. | 22:52 |
| jeblair | sdague: isn't neutron-full "any day now"? | 22:52 |
| *** aysyd has quit IRC | 22:52 | |
| sdague | jeblair: not with the current fail rates | 22:53 |
| dansmith | do we expect to see DNS failures for openstack.org names? | 22:53 |
| jeblair | sdague: k. both lgtm | 22:54 |
| jeblair | dansmith: on rackspace workers yes, it's a known bug (is in e-r) and i'm working on fixing it | 22:54 |
| *** sarob has joined #openstack-infra | 22:54 | |
| dansmith | jeblair: okay | 22:54 |
| mattoliverau | jeblair: seeing as unbound is dns related it might have had a trigger to rewrite resolv.conf which required a restart or something | 22:54 |
| *** sweston has quit IRC | 22:55 | |
| *** geekinutah has left #openstack-infra | 22:55 | |
| dansmith | jeblair: could almost deliver an /etc/hosts via puppet without much trouble right? although I presume expecting DNS to work is not unrealistic :) | 22:55 |
| jeblair | mattoliverau: i subscribed it to the unbound config file, so it's right that it restarted; just interesting that it happened so much later | 22:56 |
| mattoliverau | jeblair: so long as it works | 22:56 |
| mattoliverau | Time for my team stand up, brb. | 22:57 |
| *** dims_ has quit IRC | 22:58 | |
| openstackgerrit | A change was merged to openstack-infra/config: remove the check-devstack-dsvm-neutron job https://review.openstack.org/80414 | 22:58 |
| *** julim has quit IRC | 22:59 | |
| *** thuc has quit IRC | 22:59 | |
| *** rpodolyaka has quit IRC | 22:59 | |
| jeblair | dansmith: you'd be surprised how much trouble writing /etc/hosts would be (puppet doesn't append to files very nicely); our fix is going to be to run a caching/forwarding resolver on all the workers to reduce the outgoing dns traffic | 22:59 |
| *** thuc has joined #openstack-infra | 22:59 | |
| *** sarob has quit IRC | 23:00 | |
| dansmith | jeblair: that works | 23:00 |
| jeblair | dansmith: (of course getting puppet to do that is at least as complicated as writing /etc/hosts would be, but at least we're still using dns) | 23:00 |
| clarkb | mordred: https://review.openstack.org/#/c/75691/ depends on an abandoned change. Can we abandoned 75691 as well? (/me is trying to clean up the queues) | 23:00 |
| mordred | clarkb: looking | 23:00 |
| mordred | clarkb: yes | 23:00 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add PuppetBoard to openstack infra https://review.openstack.org/77928 | 23:00 |
| mordred | clarkb: abandoned | 23:00 |
| clarkb | ty | 23:00 |
| jeblair | dansmith: and if we still have trouble with that, it's a simple switch to make it a recursive resolver | 23:00 |
| dansmith | jeblair: I figured your hosts file would be the template, plus one thing for localhost, but yeah, local dns works | 23:01 |
| dansmith | jeblair: dnsmasq can easily read from another hosts file if you wanted to hard-code those openstack.org things in place, but maybe not worth it at that point | 23:01 |
| jeblair | dansmith: yeah, we move things around a lot, and the huge lead time to make new images with updated hosts files makes that difficult | 23:02 |
| *** emagana has joined #openstack-infra | 23:02 | |
| clarkb | also DNS is pretty good | 23:03 |
| dansmith | jeblair: okay, didn't know if things like pypi.o.o would, but yeah.. do the workers live long enough to cache those names for a while? | 23:03 |
| dansmith | clarkb: I'm not trying to be ridiculous, I'm just thinking of things | 23:03 |
| *** adalbas has joined #openstack-infra | 23:03 | |
| clarkb | dansmith: yup yp | 23:03 |
| *** thuc has quit IRC | 23:03 | |
| jeblair | dansmith: 10-60 minutes, which is why we would like to have them be caching-forwarding | 23:04 |
| *** rpodolyaka has joined #openstack-infra | 23:04 | |
| jeblair | dansmith: (so the forward servers have a warm cache to help populate the local one) | 23:04 |
| dansmith | jeblair: so that's one job, which could fail early if dnsmasq fails to look up the first pypi.o.o right? | 23:04 |
| *** mriedem has joined #openstack-infra | 23:04 | |
| dansmith | it helps as long as the first lookup works, I mean | 23:05 |
| *** mriedem has quit IRC | 23:05 | |
| jeblair | dansmith: right. if the provider-supplied resolvers turn out to be so unreliable we can't even do that, then we can switch our workers to have recursive resolvers and regroup | 23:05 |
| *** dkliban has joined #openstack-infra | 23:06 | |
| jeblair | dansmith: (possibly consider running our own recursive resolvers in each region, etc) | 23:06 |
| dansmith | oh, right, recursive, yes | 23:06 |
| dansmith | jeblair: or mesh dns among all the workers :D | 23:06 |
| dansmith | or maybe just expect the damned provider DNS to be reasonable :) | 23:06 |
| fungi | crowdsourced p2p dns! | 23:06 |
| jeblair | dansmith: (because it would be silly to have a recursive resolver on every worker. but possible) | 23:06 |
| fungi | dansmith: right, preferably that last one | 23:07 |
| *** mriedem has joined #openstack-infra | 23:07 | |
| dansmith | fungi: it's funny we're even discussing that, given how robust DNS is and that it's 2014 and all | 23:07 |
| fungi | where "reasonable" is apparently a fairly vague term (for example, working dnssec?) | 23:07 |
| jeblair | dansmith: ha! we have one provider that likes to reject queries (i think they have a new blacklisting system that may not be working right), and another that strips dnssec info. | 23:08 |
| fungi | rackspace likes to just ignore our queries sometimes, while hpcloud on the other hand is stripping out signing data | 23:08 |
| SpamapS | why do we have a hacking rule that won't let me put a period at the end of the short commit message.. but we don't have one that limits to 50 chars? :-P | 23:08 |
| dansmith | sheesh | 23:08 |
| fungi | what he said | 23:08 |
| fungi | service providers ftw! | 23:08 |
| clarkb | atleast both have responded to our questions about it | 23:08 |
| dansmith | but, with the free quota and all... LP | 23:08 |
| dansmith | er, -> :P | 23:09 |
| clarkb | though so far one is only "this has been escalated" | 23:09 |
| fungi | we get what we pay for, yes | 23:09 |
| clarkb | SpamapS: I have no idea | 23:09 |
| fungi | SpamapS: is that 50 with or without the trailing period? | 23:09 |
| jeblair | hahaha | 23:09 |
| openstackgerrit | A change was merged to openstack-infra/config: drop neutron-full (non voting job) on most projects https://review.openstack.org/80419 | 23:10 |
| *** dims_ has joined #openstack-infra | 23:11 | |
| *** jgrimm has joined #openstack-infra | 23:15 | |
| sdague | SpamapS: I'll tell you over drinks some time :) | 23:15 |
| SpamapS | fungi: gnar!! | 23:15 |
| SpamapS | sdague: 'nuff said. | 23:15 |
| * SpamapS is no stranger to the gentleman's agreement | 23:16 | |
| *** rpodolyaka has quit IRC | 23:17 | |
| *** bada_ has quit IRC | 23:17 | |
| *** atiwari_ has joined #openstack-infra | 23:17 | |
| *** atiwari_ has quit IRC | 23:18 | |
| *** atiwari_ has joined #openstack-infra | 23:18 | |
| *** jasondotstar has quit IRC | 23:18 | |
| jeblair | clarkb, fungi: another quirk: rax centos is going to repopulate resolv.conf on boot with the static values that are written into the image | 23:19 |
| *** atiwari has quit IRC | 23:19 | |
| *** bada_ has joined #openstack-infra | 23:19 | |
| jeblair | sorry, i mean the static values that it writes to the fs when it boots a new machine | 23:19 |
| clarkb | jeblair: :( | 23:19 |
| clarkb | so its like dhcp but without a clear override mechanism? | 23:19 |
| *** mfer has quit IRC | 23:20 | |
| jeblair | (nova file injection -> /etc/whatever -> something in boot writes /etc/resolv.conf) | 23:20 |
| jeblair | clarkb: yeah, i don't recall the full mechanism; starting to dig | 23:20 |
| *** rpodolyaka has joined #openstack-infra | 23:20 | |
| mattoliverau | Wow, go RAX! :P | 23:21 |
| *** thedodd has quit IRC | 23:21 | |
| mattoliverau | can you mark it as imutible? | 23:21 |
| mattoliverau | in puppet | 23:21 |
| jeblair | mattoliverau: yeah, we've got 6 variations which means we have 6 problems! :) | 23:21 |
| *** jasondotstar has joined #openstack-infra | 23:22 | |
| jeblair | mattoliverau: so part of the challenge here is that when nodepool launches a node from an image it creates, it expects it to be ready to go at boot, and puppet never runs on it | 23:22 |
| jeblair | mattoliverau: (puppet only runs on the template node that is used to build the image) | 23:22 |
| mattoliverau | ahh of course, I just read that code so I should know :P | 23:23 |
| fungi | jeblair: worth thinking about though, we already test ssh on a node boot. maybe we should also run puppet apply? that might solve some of our race issue with slave scrips being present for jobs which need them and so on | 23:25 |
| jeblair | fungi: possibly, and i think there's a patch out there to add the feature to nodepool | 23:26 |
| jeblair | fungi: though i kinda think it shouldn't be necessary for this. | 23:26 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix doc error in the testng publisher https://review.openstack.org/79012 | 23:26 |
| fungi | fair enough | 23:26 |
| clarkb | jeblair: fungi: that could significantly slow node ready times depending on deltas | 23:26 |
| jeblair | clarkb: also dos puppetmaster | 23:26 |
| fungi | clarkb: jeblair: both very good points | 23:27 |
| jeblair | though hopefully it shouldn't even talk to puppetmaster... | 23:27 |
| mattoliverau | resolv.conf will just point to localhost right, so there is no difference between the template and booted nodes right? | 23:27 |
| jeblair | mattoliverau: the booted nodes get rax dns servers because of the fileinjection of the network config | 23:27 |
| *** masayukig has joined #openstack-infra | 23:27 | |
| jeblair | apparently if you set RESOLV_MODS="no" then it won't mess with it. | 23:27 |
| mattoliverau | Yeah but setting resolv.conf to immutable should stop it from being over written | 23:28 |
| jeblair | though i think you set that in the file that gets injected. :( | 23:28 |
| mattoliverau | on the template before an image is made i mean | 23:28 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: ZMQ Event Publisher plugin support added https://review.openstack.org/77561 | 23:28 |
| jeblair | mattoliverau: you mean with unix file perms? | 23:28 |
| mattoliverau | yeah, if its ext3/4 you can use setattr | 23:29 |
| jeblair | mattoliverau: it uses "cp -af" so i don't think unix perms will work | 23:30 |
| *** thuc has joined #openstack-infra | 23:30 | |
| jeblair | not sure about setattr | 23:30 |
| mattoliverau | not if it's attempting to write to an ext3/4 FS. It can't overwrite it. It isn't unix perms it is an EXT FS extended attribute | 23:31 |
| jeblair | mattoliverau: i'm a little worried about what that would do to have a failure at that point in the networking startup | 23:32 |
| jeblair | we could try it though | 23:32 |
| *** atiwari_ has quit IRC | 23:32 | |
| mattoliverau | sorry I meant chattr +i <file> | 23:33 |
| mattoliverau | Then to turn it off chattr -i <file> | 23:33 |
| *** CaptTofu_ has joined #openstack-infra | 23:36 | |
| *** ryanpetrello_ has joined #openstack-infra | 23:36 | |
| *** dstanek_zzz has joined #openstack-infra | 23:36 | |
| *** dstanek_zzz is now known as dstanek | 23:37 | |
| *** flaper87 is now known as flaper87|afk | 23:37 | |
| *** jasondot_ has joined #openstack-infra | 23:38 | |
| *** dcramer_ has joined #openstack-infra | 23:38 | |
| *** thuc has quit IRC | 23:38 | |
| *** KurtMartin has joined #openstack-infra | 23:39 | |
| *** mkerrin1 has joined #openstack-infra | 23:40 | |
| *** jasondotstar has quit IRC | 23:41 | |
| jeblair | mattoliverau: yeah, that should cause it to fail | 23:41 |
| clarkb | would node boot succeed in that case? | 23:41 |
| *** julim has joined #openstack-infra | 23:42 | |
| jeblair | clarkb: no idea | 23:43 |
| jeblair | i just realized why my test node booted without my being able to sudo, so i'm a bit away from being able to test that | 23:44 |
| *** bada_ has quit IRC | 23:45 | |
| *** markmcclain has joined #openstack-infra | 23:45 | |
| *** CaptTofu has quit IRC | 23:45 | |
| *** ryanpetrello has quit IRC | 23:45 | |
| *** dstanek_afk has quit IRC | 23:45 | |
| *** khyati has quit IRC | 23:45 | |
| *** mkerrin has quit IRC | 23:45 | |
| *** kmartin has quit IRC | 23:45 | |
| *** ryanpetrello_ is now known as ryanpetrello | 23:45 | |
| *** bada_ has joined #openstack-infra | 23:46 | |
| *** zhiwei has joined #openstack-infra | 23:46 | |
| *** adalbas has quit IRC | 23:46 | |
| jeblair | mattoliverau, clarkb: i think it would continue; i don't think it sets -e in the script | 23:49 |
| jeblair | is there a nice puppet way to set +i on a file? | 23:49 |
| clarkb | jeblair: I think with a module off the forge | 23:50 |
| jeblair | it's not worth that :/ | 23:50 |
| mattoliverau | I don't know, you could just run the command. | 23:50 |
| zns | Would it be possible to get https://review.openstack.org/#/c/79975/ through? It's now blocking us from getting some of our patches through. | 23:50 |
| jeblair | maybe we should just drop this in the nodepool script since that's the only place we need it | 23:50 |
| *** thuc has joined #openstack-infra | 23:50 | |
| mattoliverau | yeah good idea | 23:51 |
| *** thuc has quit IRC | 23:51 | |
| *** spredzy has quit IRC | 23:51 | |
| *** dkliban has quit IRC | 23:52 | |
| *** thuc has joined #openstack-infra | 23:52 | |
| jeblair | zns: done | 23:52 |
| *** dkliban has joined #openstack-infra | 23:52 | |
| *** spredzy has joined #openstack-infra | 23:52 | |
| mgagne | jeblair: could be a challenge having immutable attribute mixed with puppet management. =) | 23:52 |
| zns | jeblair: thank you! | 23:53 |
| openstackgerrit | A change was merged to openstack-infra/config: satori: Drop requirements, pypi, and old upstream https://review.openstack.org/79975 | 23:53 |
| *** sdake has joined #openstack-infra | 23:55 | |
| *** sweston has joined #openstack-infra | 23:57 | |
| *** sarob has joined #openstack-infra | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!